Archetypes are important to telling stories, and the nemesis creates an interesting, compelling figure in many stories. While they may serve as a means of examining the protagonist–of showing off the things that make them good–they mostly do this in traditionally heroic stories where the protagonist eventually overcomes their flaws.
But what then of the stories where our hero has a tragic bent, or where we are stuck with an anti-hero?
Then, we have the nemesis as a tragic impetus: the force that causes the hero to fall from glory, or to be stuck in the dirt where they began.
The method by which the nemesis does this will vary from story to story, but the fundamental premise is the same: as a force nearly equal to and in direct opposition to the hero, the nemesis holds great power over the supernatural world that the hero seeks their ultimate reward in (this supernatural world, of course, being part of the Hero's Journey) and may actually overcome the hero.
In this case, we refer to the hero as a tragic hero, or anti-hero, depending on how they started at the beginning of the story.
But what in particular happens with this sort of nemesis?

Illustration of Othello and Iago, by Charles and Mary Lamb

This nemesis, for whatever reason, is directly destructive to our hero and focuses on their destruction (this is not unique to this sort of nemesis, however) but is able to find some particular weakness that takes down the hero.
Shakespeare's Othello is an example of a tragic hero who is destroyed by his nemesis: his lieutenant Iago convinces Othello that his wife is cheating on him, producing false, but inconclusive, evidence against her. At this point Othello falls to his rage and gullibility, killing his wife before killing himself.
Similar nemeses can be found in Julius Caesar, with Cassius who deceives Brutus, leading to his joining the conspiracy against Caesar, which he is ultimately killed for. This is interesting because Cassius wants no harm to come to Brutus, but he is hoping to use Brutus for his own personal gain. If Brutus had pursued his own path to resisting Caesar, there is a chance that events would have turned out differently, as his more noble traits like honor and patriotism would have overcome his gullibility.
Cassius is also the nemesis of Caesar himself, who is also a tragic hero figure in the play. Cassius manages to use the would-be emperor's vanity and self-importance against him as part of the conspiracy, showing an aptitude for deception and manipulation as he builds a web of conspirators.
Note that the fact that Cassius is a nemesis doesn't necessarily make him evil; Caesar was a threat to Rome, and Brutus' fall on account of Cassius was by no means a clear act of malice. Rather, he is a destructive force acting upon the flaws of other characters in the play.
1984's O'Brien is likewise a nemesis who seeks to destroy the hero, but in this case he is paired with an anti-hero in the form of Winston Smith. Much like Othello, Smith is naive, but he has nothing to lose (where Othello was a great general, Smith holds a low-level job in the Ministry of Truth).
The anti-hero's nemesis can take a different turn than that of the tragic hero, however. The tragic hero has great strength in their domain, but has areas of weakness that can be exploited. O'Brien, however, is able to attack Smith on his point of strength–his individuality–targeting him with formulated torture and getting him to confess to his treason through ploys intentionally designed to foster his open rebellion against the state and induce his destruction.
Victor Frankenstein faces this in Mary Shelley's novel Frankenstein. He is able to acquire great knowledge, even to the point of replicating the spark of life so that he can breathe life into his creature, but he cannot actually conquer the fundamental goal of creating something better than a person: the Creature does become something that is in many ways superhuman, but because of his disconnect from his own creation and his lack of wisdom during its making, he finds himself confronted by a Creature that is deeply concerned with morality and its role in society–things that Victor never considered until after he had made the Creature.
The connecting factor between these nemeses is that they have an innate connection to the flaw of the tragic hero in their work.
A good storyteller can use a nemesis to heighten the weaknesses of the hero and create meaning by highlighting their flaws. Whether these flaws are moral or practical can depend on the story, but the destruction brought to a hero who cannot withstand the trials of the world is compounded by a nemesis who is capable of mastering the supernatural world themselves (like how O'Brien is a member of the Inner Party and has great authority in Oceania), creating a bold reflection for the audience to consider.
It's worth noting that for this reason Cassius is relatively weak as a nemesis; he is not only a potentially sympathetic character if you overlook some of the smaller scenes where he is deceiving Brutus and engaging in scandalous behavior, but he is not the master of his world and ultimately kills himself by the time the play has run its course.

Application

How can you use a nemesis in your stories to highlight the flaws in a tragic hero or an anti-hero?
Keep in mind that the nemesis is often a malevolent apotheosis or antithesis of the hero: is their flaw that they lack guile, as Shakespeare's Othello and Brutus, or is the flaw that they do not have the ability to confront their society, like Winston Smith? Do they, like Frankenstein, only realize their path was one of dissolution and abomination after they succeed?
The nemesis has the answer to these problems; they are cunning where the tragic hero is gullible, they are powerful where the anti-hero is weak.
As a storyteller, the nemesis serves as an answer to these questions, but only by counter-example. They do not demonstrate virtue themselves, but highlight the lack of virtue in the tragic hero or anti-hero.
In a game, as a designer, you can create a nemesis that does this via attacking the heroes on something that they need to master. If your game focuses on the creation of tragic heroes or anti-heroes, where the end result is guaranteed to be death or loss, then you're going to have great effects this way. The alternative is to have the protagonists' downfall come at the hands of an inscrutable or natural force. This is a decision that you can make based on the story you want to tell, but a nemesis lets you highlight the meaning of your story.
The greatest examples of this typically come in games inspired by Lovecraftian or Gothic fiction: the nemesis attacks the characters' sanity or essence, draining away parts of their life and highlighting their weaknesses.
However, it is worth noting that this sort of nemesis can be frustrating for the player characters, and you need to be aware of the limitations of this format, especially in group-based roleplaying. It may be worthwhile to plan a particular nemesis for each player character in a game, or otherwise focus on a relationship between one character and their nemesis instead of orienting the nemesis toward the whole party.

Wrapping Up

The nemesis serves as an excellent catalyst for the fall of the tragic hero or destruction of the anti-hero. They provide an opportunity to illustrate the ways in which the protagonist's flaws cause them to fail in their goals, and illustrate the meaning of the story.
When you include a nemesis that specializes in attacking heroes' flaws, it provides a suitable reason for the protagonist to fail, allowing you to avoid some of the common pitfalls of anti-heroes and tragic heroes where they are perceived as unworthy as models because of glaring flaws. The tragic nemesis allows even a minor flaw to become a rationale for destruction.