Vlad Zamfir explaining Casper (Ethereum's Proof of Stake)

Vlad Zamfir has given a lot of thought into this and is on the forefront of research into collaborative game theory + blockchain consensus. The thinking works similar to this, generally speaking human behavior (or any animal) is directed by rewards and punishments. Formalized this would be risks and incentives. Casper works by making it so that people have more to gain by following the protocol than by not following it, and goes further where by breaking the rules can be punished.

This is in essence a type of "trust by collateral" where people put up collateral and you trust them because you know if they mess up their collateral is lost. In general this is what governs human behavior in general because reputation works in a similar way where people don't do certain things because they have something to lose in terms of how other people will see them which will make other desired behaviors much more costly in the future. Collateral is a way to know someone has "skin in the game" and something at stake and you can quantify how much they are risking.

I do think the incentive structures behind something like Casper can work in many ways and can even include intangible "psychic" credits, or non-monetary rewards and punishments. Reputation for example can be included to add even more trust to the system beyond just the collateral so that the validators can not only lose their money but also reputation. Of course that would require pseudo-anonymity or a known identity for reputation to be attached to but it is an example where you can create reputation incentives on top of monetary incentives. Economic incentives and disincentives work even for completely anonymous validators.