Technically it can be done but risk of contamination with environmental debris, this explains the use of clean rooms in headset assembly factories.
Generally, the display screen, eye tube and lens housing are bonded with adhesive to provide a water resistant sealed unit. Small variations require calibration after headset assembly to ensure optimum performance and user comfort.
It possible to replace the entire unit (one for each eye) with relative ease providing you have access to tools and some disassembly information.