The e-ink display has a working lifespan of about 50,000 hours, translating to about 90,000,000 refreshes at 2 seconds each. (source)
I liked your guess about the direct sun light causing the fading. More unreliable sources (reddit) seem to have similar experiences.
So maybe just a UV light protected enclosure to the rescue?