Penumbra jump started today's first person suspenseful horror game style, although it was borrowed more directly from Amnesia: The Dark Descent which uses Penumbra's formula. Penumbra is one of very few games I consider scary, and it is one of few Lovecraftian games and perhaps the only great one.
It can be terrifying on a surface level, being alone and near-defenseless in a dark, detailed/believable and spooky location with dangerous creatures surrounding you. It also attempts to be horrifying on an existential level. The existential horror is done well especially for a game, but the best writing aspects come from its characters, even characters you never actually meet. It's very tough to write unique, memorable characters that you never meet, but this game is a master at it. Then there is Clarence... one of the best characters and antagonists in video game history.
Obviously, play Overture first. It's around 4 hours. Then play Black Plague, which is about the same length but ramps up the intensity tenfold yet never loses coherence or craft. You might as well give Requiem a try after it all but beware that it is absolutely nothing like the rest, and it is a completely unnecessary expansion created only to try and deliver a less ambiguous ending. But Black Plague's ending is wonderfully ambiguous, Requiem actually damages the story slightly and it isn't even a horror game, it's a puzzle game but a good one actually.