It is not harmless. Or rather, Hydrogen Selenide is the real danger, which is the product of the reaction of ZnSe with any acid (eg. like that in your stomach). It is toxic in very low concentrations ie. 0.3 ppm.
ZnSe safety sheet:
https://www.thorlabs.com/images/tabimages/Zinc-Selenide_MSDS.pdf
H2Se safety sheet:
https://www.chemadvisor.com/matheson/database/msds/00244337000800003.pdf
So take precautions. Don’t sand or scrape the lens to make it fit. Wear gloves when handling damaged lenses. Don’t lick your fingers! H2Se smells strongly of garlic/rotten eggs so you’ll know about it if it’s around.