Thoughts about the questionnaire and new proposals
A few days ago a draft of the questionnaire was released and some users provided some feedback and also some critics. We considered and discussed the points that were made in the comments but it was obvious that some users misunderstood how the scoring system worked. In our opinion when releasing the questionnaire it should have been specified that because of devs requirements, the questionnaires started from a value of 100 and we had to substract points instead of adding, that is the reason why there were no positive questions. Probably that point caused the most outrage and it was just a perception issue.
Others mentioned that there were too many questions regarding the contribution post and that it was a bad idea to start from these questions. Some users asked to be more punitive on average to mediocre translations that contain several errors but reward more work well done and error free.
We came out with two possible questionnaires, the first option (Q1), is something implementable in the current utopian framework. The option (Q2) may not be currently implementable that it's what we feel should be done.
First example questionaire (Q1):
Q1 Google Sheet example link
Second example questionaire (Q2):
Q2 Google Sheet example link
The two examples are in the same file, but on different tabs
Q1
The first option is compatible with the current framework in Utopian: it starts from a maximum score of 100 and it subtracts or adds values based on the bonus / malus obtained from the answers to the questions. Now we know that the final score can go beyond the value of 100 so we added also positive questions.
Limits:
In order to reward quality, we must assign greater weight (score) to the questions regarding the quality of translation. Thus, it is not possible to differentiate in a balanced way the economic remuneration for translations of different volumes. For example, a translation of 2000 words should not reward twice as much as a 1000 words translation. On the other hand, if we assign more points to the volume of the translation, the quality loses weight and a long but average translation could receive more than an excellent but short translation.
Meaning of the questions
1. How do you rate the overall accuracy of the translated text?
2. How many major mistakes were found in the translated text? (that can be change the meaning of the text)
3. How many minor mistakes were found in the translated text? (that do not change the meaning of the text)
4. Has the same style of translation been maintained for this project? If more translators are working on this project, are they respecting the original style?
5. How would you rate the formatting, grammar, writing style and overall presentation of the contribution post?
6. Did the translator provide in the contribution post all the information needed to fully evaluate the translation? (For example, did he specify if he needed to research the definition of unfamiliar terms and which tools he used in the translation?)
7. What was the total volume (words) of the translated text in this contribution (excluding duplicate phrases and non-translatable content)?
Question 1: It represents an overall judgment of the translation also taking into account factors not specified in the following questions.
Questions 2 e 3: This is a more specific question, it requires a minimum count to identify the proportion between the errors committed and the length of the processed text, distinguishing the weights in serious and minor errors.
Question 4: It is important that the style is consistent between the different parts translated, such as when you decide not to translate technical or medical words. The consistency of methods and style is also important when different translators work on the same project or on one already started by others. Please note that if you find issues in the previously translated strings you should translate them again avoiding the same mistakes.
Question 5: This question refers to the post presentation. It has less weight than the questions relative to the quality of the translation but it gives more granularity to the final score. Nonetheless, a clumsy post, with several grammatical mistakes, would seriously undermine the professionalism of the translator.
Question 6: Second and last question about the contribution post. Here we are asking if the translator has provided all the necessary information for the correct evaluation of his work, such as: number of translated words, project name, github link, crowdin link, name of translated files (if applicable) and other if necessary.
Question 7: The length of the translation is another important parameter, a standard post is minimum 1000 words. We have introduced bonuses for longer posts and panalities for shorter ones.
Note: The scoring system is a short blanket, it is difficult to give due weight to both quality and length, we tried to mediate.
Q2
Our second proposed questionnaire may not be implementable (it would require some changes in the utopian voting bot). But, we believe it would provide a score that reflects better the quality of the translations and it multiplies the result by a coefficient proportional to the volume of translated words. We would like to highlight two values: the score, which indicates the quality of the translation regardless of its volume and the actual vote that it will receive, which considers also to the amount of translated text. This allows us to discriminate between quality represented by the score and to tune the reward taking into consideration the volume of text translated.
Why is this important? To avoid that short but excellent translations are skipped by utopian voting bot only because there were longer translations in the cue.
Please:
When providing feedback/criticism it would be helpful if you could also provide constructive proposals instead of just saying "I don't like it". We will have a chance to discuss this on Monday in the utopian discord server.
 CC0 Image - Creative Commons, thanks to @mrazura for ITASTEM logo.CLICK HERE AND VOTE FOR DAVINCI.WITNESS DAVINCI WITNESS TRAIL DAVINCI WITNESS DISCORD SERVER If you would like to delegate to Davinci.Witness you can do so by clicking on the following links: 50 SP, 100SP, 250SP, 500SP, 1000SP, 5000SP. Be sure to leave at least 50SP undelegated on your account. |
|---|
After going through all the comments on elear's post, and reading about the questionnaire issue, I think finding a consensus is impossible since everyone has different standards and point of views, there are strict people who emphasize a lot on the quality and accuracy of the contribution and I tend to be one of them, and there are other people who tend to have a minimalistic approach, others are more permissive …. On the other hand, we have the standards that utopian wants to implement. There are just many ideas, and many contradictions going on here, there is also a huge lack of information about what utopian wants and what people want because from what I saw, many people don’t know that utopian requested many things before about the presentation, word count, granularity and so forth.
I think people need to understand that there is no perfect solution to this, and at the need of the day, a decision will have to be made, not because it’s the right and best solution, but just because it’s the fairest possible solution. I believe that this is not the end, and we can keep working for a better model in the future, to make it even more accurate and professional. But the sure thing is that if we want to take a step farther in this, more resources will be needed, we will also need developers and people well versed in mathematics to do the proper equations for scaling … for now, we just need to work with what we have and can control, but as I already said, I am pretty sure that a better solution(new project) will be set in the future, a solution that equals the professional existing tools that are available online for companies.
Whoever told you this is the case is misinformed. The score does start as 100, but you can have both positive and negative values for the answers. For example, for the question "How would you rate the overall value of this contribution on the open source community and ecosystem?" in the development category we have the values 5, 0, -12 and -20 for each answer respectively. Other categories also have this, so you could do the same if you want to.
You are right, we did not know that the score could go beyond 100 so that it was possible to add positive points.
As it is rare for everything to be perfect, a score beyond 100 will probably never happen.
Hey, @davinci.witness!
Thank you for your contribution to the crowd. We are the Steemit project dedicated to empowering The Wisdom of Crowds. You can find more about us on our official blog or whitepaper and you can support us by voting for our witness and joining our curation trail on Steemauto . We are also inviting you to join Crowdmind Discord server. Don't forget to use the #crowdmind hashtag and happy crowdsourcing!
Aditional comment: Besides the fact that you are already a great project with core crowdsourcing values implemented in your work, your decision to source the questionary from your community was absolutely brilliant. It will not only help you keep the high level of engagement, but also produce better final solutions because you decided to tap the wisdom of the crowd. As from our beginning, crowdmind supports you!
Best regards
@hidden84
My suggestion is to have a "silver bullet", that will mark that the post must be upvoted, not matter the score.
Let me explain:
Imagine that there is a project with 1.152 words
It gets - 0 points
Another translator is working on a project with 10.000 words, and it's easy to split it in parts with 2.000 words
It gets - 14 points
The first translation, although with the lower score...
Is one more finished project. It's translated well and it deserves the upvote, no matter the score.
I am glad you brought it up, that's exactly why we made a second proposal where the volume does not affect the score but it's used as a coefficient to change the voting weight. I am just not sure if it could be implemented. There are also manual votes that can be requested in special cases.
Utopian and DaVinci
Thanks @davinci.witness for your awesome work with the Utopian translations category!
Contribute to Open Source with utopian.io
Learn how to contribute on our website and join the new open source economy.
Want to chat? Join the Utopian Community on Discord https://discord.gg/h52nFrV