NEVER LEAVE YOUR STAMP IN THE SUN Don't leave it by the window. Don't leave it out on the patio. Don't leave it on the table where the sun can touch it. Make sure your clear plastic storage trays are not facing the sun. I'm guessing you shouldn't leave them in the tanning bed either, but I can't imagine a reason to do so in the first place! The reason for this is that the rubber will get hard and brittle. The image will degrade. It will not stamp as easily. Eventually, the rubber will just crumble.
DO NOT USE OIL ON YOUR STAMP Oil *seems* like it ought to work to rejuvenate stamps--after all its nice on our skin, right? WRONG! Oil on stamps makes them slimy and icky and degrades the rubber. It will never be the same ever again.
As far as I know there is no cure for either of these mistakes. The only thing I could suggest is to rip the rubber off the mount, have a little funeral for the demise of your lovely stamp, then buy an unmounted stamp that will fit your mount and start fresh.