How to solve the Translation Category flood and poor quality contributions problem - Rethinking the Process

The problem:

On the last days, utopian.io saw an increase of users and contributions, reaching almost 1 contribution per minute.

A big part of these contributions is being sent on the Translation Category.

The Translation category on Utopian.io is probably the most accessible one, because a big part of the internet users know their native language and english. But knowing a language do not necessarily means that someone can do quality translations.

And as an aditional problem, everyone have acess to a translation websites, and some people are using them to create poor quality translation.

These two features alone may attract users that have more interest in the rewards paid by the Utopian bot than to contribute with high quality work.

The consequence is a flooded review feed where the moderators have a hard time reviewing all the contributions.

What i have observed is that each contribution category have have its own characteristics, and each one of the must have an specific workflow.

Here i present a possiible solution to the Translations:

Temporary Immediate action to create space to the team

It's difficult to stop and think about solutions when there is more and more urgent matters that need immediate action.

For the Utopian team to have enough time avaiable to discuss about the problem and find a solution, translation contributions need to be slowed down.

This isn't an ideal scenario, but it is needed, or else the problems that are happening in the translation category (5 days old posts, poor quality translations being approved and others) may start to appear on other categories, creating an unstopabble snowball effect.

Action Sugestion:

Make an temporary change of rules, adding that 12h (or 24h) from the implementation of this rule, the Translators may only contribute with one post each 24h. Any other contribution by the same user before the 24h period between posts will be rejected immediately.

To mitigate the damage some users may feel for this sudden change, maybe increase to 1000 the minimum amount of words translated and increase the utopian bot vote strength (if possible for the bot to know wich posts is made after the stt of this change).

This will give the team time to clear the feed and focus on finding solutions for the translation workflow.

Permanent solution proposal - a new and better way to think about the Work Flow.

First we need to figure out wich is the work flow of the translation category. This is how i see how the process is right now:

The contributions end up in a big pile, that grows faster than the moderators can properly handle.

The quality control only happens at the end of the flow, at the moment of approval/rejection of the contribution, where in the most extreme cases (like what is happening now), the moderators will have a small time frame to make a good review. With less time to review, the quality of the moderation work suffers.

Besides that, the biggest effort of the moderators lies on the review of new users contributions, since they have a tendency to make more mistakes because most of them don't yet fully understand utopian rules, why they exist, and how the work is done.

Moderators spend a lot of time explaining to newcomers what they must do to have their contributions approved.

In the end, there is too much data getting on the system, and not enough power to process them.

What is needed is to control the influx of data to a size that the current moderators are able to handle. To do that, we need to create checkpoints.

First checkpoint - The door

• The new users (and maybe all/some existing users from this point on) should be allowed to make only one contribution.
• The new user would be listed in a document, only acessible by the team, as a Tier 0 Translator after he post his first contribution
• If the user post any new contribution after the first one, they will be rejected immediatly, and the moderator reply stating these rules, and the consequence of not following this rule.
• If after being warned by the moderator that he is not allowed to make any new contributions before the first one is reviewed, he will be banned.
• Moderator proceeds with the review (There is also a suggestion to a new review system below).
• If rejected, the user is told why he was rejected, and is allowed to make another contributions
• The new user would have a X number (to be determined) of chances to get a contribution right. If he doenst meet the quality criteria in the X contributions, he will get a temporary ban, so he may have the chance develop better quality skills and try again.
• If the contribution if approved, the user is moved to a tier 1 status.

This solves 2 problems:

1 - It filters out those who want to only make easy money, since it will be harder to make too much contributions, and the worst ones will be stopped at the door.
2 - Slow down the feed for moderators, giving them time to proper evaluate the quality.

Second Checkpoint - Continuous Improvement

• On Tier 1, the user is allow to make more contributions each 24h. Number of contributions allowed must be defined by the team, but i think the max number of contributions per day on this tier should be 4
• Contribution and reviews proceeds as usual, but the quality standards must be higher at this point.
• On each post, the moderator would work with the user, talking on chat with the user as much as possible, to help the user improve the quality of his work.
• If the user contribution reached the highest quality possible, he would receive a "point"
• When the user reach a set number of "points" (10?,40?, 100?), he is ready to move to the next Tier

Third Checkpoint - User evolution

• At Tier 2, the user is allowed to do as many contributions as he want per day (or may as high possible, like 10 posts per 24h), as long as they are on the same project.
• With this, the translator could reach a high productivity, and the moderator would be able to review all contributions as one, since they are on the same project.
• With high quality contributions over time, two options may be presented to the user: Become a Moderator (if there is a spot avaiable), or become a proof-reader, to help the moderators with the reviews, if the moderators find themselves in need of an extra help.

This system would help to filter out the bad contributions at the start of the process, and help the good users improve over time, reaching a pretty high standard.

Reviewing the Translation Review Process (or, how Proof Reading Contributions should work)

There is indeed the need to create a Proof Reading category, but it should be a limited category.

Only moderators (or the proof reading users, as stated above) should be allowed to contribute on this category.

To the moderators, their job already is proof reading, so this would work as an extra incentive to do a good proof-reading work inside the CrowdIn platform (Don't know how it would work directly on github though). There may be exceptions, like if there is someone of an specific open source project development team already doing this and he wants to join Utopian.

To the high level users, this would be an incentive to help only high quality work to be approved.

The review of this category would only be done by supervisors.

Some points about the rewards on this category should be discussed:

• Higher pay than translator - The logic one, since proof readers and moderators have a higher responsibility than the translator. But on the other end, it may create unbalanced rewards, since moderators already are paid by the moderation work, wich in the end, is a proof reading work.
• Lower pay than translator - This would work on the moderator side, but may hurt the proof reading users, that may find its better to just keep on translation.

How the Proof Reading category should work

First of all, a list must be created with all crowdin projects already translated through Utopian.

The moderators/supervisors would then try to reach the Crowdin Project Managers to give Utopian moderators/supervisors proof reader status. This is needed to provide proof-of-work for the proof reading contributions and to make an actual impact in the project, expanding the utopian brand.

If a new project appears on the Review feed, the moderators/supervisors would try to get the proof reader status within the 48h of pending review.

In the projects that the moderators were able to reach the proof reader status inside Crowdin, the proof reading would a help with the review, since it will create an statistic about how many of an user contribution were rejected.

If the mods weren't able to get the proof reader status in time, they would proceed with the utopian review as usual.

Besides the above suggestion, some small improvements may also help with the flow:

• The second Tag of the contribution must always be the language on the translation (PT, FR, GE, etc)
• Add a secondary tag filter to use the language tag
• Define a standard template for Translation Contributions. A stardard contribution post improve the efficience of the moderators in finding the information that matters quickly.

---

Posted on Utopian.io - Rewarding Open Source Contributors

---