A Look At The Review Process | Analysis Questionnaire Ideas and Suggestions

Repository

https://github.com/utopian-io/utopian.io

Introduction

The goal of this series of posts is to provide input into the sets of guidelines for a @utopian-io contributor, moderator, and the community as a whole. Additionally, this research would also then feed into the moderation questionnaire which is used to evaluate and score an analysis contribution to @utopian-io.

The first post in the series can be found here

In the first post, I researched into how a company/organisation would evaluate a piece of data analysis and uncovered many over-lapping definitions and pointers regarding what to do and what not to do when producing a piece of analysis work. In this post, I will take a look at how the current reviewer questionnaire is structured, and offer suggestions for improvement based on the findings detailed in series post 1. Previous @utopian-io contributions may also be used as examples to highlight particular points.

The opinions given in this post are my own and not representative of @utopian-io

---

The review questionnaire - Analysis

The following is the current version of the reviewer questionnaire used to evaluate analytical contributions to @utopian-io. An initial score is granted to each contribution considered for review and in the main, points are deducted when the optimum answer is not chosen for each question. The questions have been numbered for easier reference in the following sections.

1) Were all relevant aspects or metrics related to the objective analyzed?

• All relevant metrics were covered
  -Only selected metrics were chosen; including more may have provided additional insights
• Only a single or narrow aspect was chosen
• No metric was chosen

2) How would you rate the complexity (of) data extraction for this analysis?

• Gathering the data required complex queries and post-processing
• The method of extracting data was moderately challenging
• The data can be directly imported for visualization - no additional data transformation was needed
• No data was extracted

3) How would you rate the quality of the visualization of the findings?

• Visualizations presented were superb and beyond expectation
• Appropriate and sufficient visualizations were used to present the results
• Visualizations were included but lacked in quality and/or quantity
• Visualizations included were irrelevant to the objective

4) Was the analysis reproducible through the use of the contribution content?

• All queries or data gathering methods and all data processing scripts were included
• The core query or data gathering method was included and the data processing steps were described
• Data gathering methods and processing steps were sketched
• Data gathering methods were not included

5) Was it a new and unique analysis?

• Yes, it was a unique analysis
• It’s similar to another contribution but covers deeper or additional aspects
• It’s similar to another contribution but covers a different time period
• It’s a recurring analysis covering too short a time-frame (i.e., daily)

6) How would you describe the formatting, language and overall presentation of the post?

• Good
• Average
• Below Average
• Low Quality

7) How would you rate the overall value of this contribution on the open source community and ecosystem?

• This contribution brings great and impactful value and can be used for applications outside the specific project.
• This contribution adds significant value to the open source community and ecosystem or is of critical importance to the specific project
• This contribution adds some value to the open source community and ecosystem or is only valuable to the specific project
• This contribution adds nearly no value to the open source community and ecosystem or the specific project

---

Review of existing questionnaire

First, I would like to state that I believe the 7 questions above to be a solid starting point for reviewing an analysis contribution and that they have been a good guide in evaluating the varying quality of submissions. The questionnaire has been a good guide, but in light of future plans for @utopian-io, these questions and the guidelines need reworking somewhat. I'll take a look at each question and the answers and then put forward some additional ideas for future questions. These suggestions will be in relation to the findings of the first post in this series.

Question 1 relates to the range of metrics that could be analysed in relation to an open source project. It is quite tricky to assess if 'all' metrics have been covered and this depends on the knowledge level of the reviewer with regard to the project. However, if the contribution covers a number of metrics and nothing obvious is missing, then it will likely achieve a top mark here. Using this analysis of @actifit, the level of detail and assessment of the progress so far, earned this contribution the top answer 'all relevant metrics were covered.

A potential drawback to this question is the 4th answer which states 'No metric was chosen'. It is highly unlikely that no metric will be chosen, and I suggest the following changes to the answers

• All relevant metrics were covered
• Most of the relevant metrics were covered in supporting the analysis
• Only selected metrics were chosen; including more may have provided additional insights
• Only a single or narrow aspect was chosen

This change should allow more opportunity to use the final selection. In almost all cases there is a metric but in some cases, it is single or narrow and I think that this should be the lowest option/score.

Question 2 attempts to evaluate the complexity of the data sourcing and modelling. This question relates well to the definition of a 'data analysis' and I think has an important part in assessing the quality of a contribution, particularly with regards to modelling the data to find useful information.

The 3rd answer, in my opinion, is there for if the contributor has used a simple 'select' statement in SQL for example, requiring no joins to other tables, and producing a base table which can be easily charted. The fourth answer, in my opinion, is for data that has been gathered manually from a web-page, without the requirement of coding.

Question 3 clearly guides the reviewer in assessing the quality, detail, and accuracy of the visualisations. If these are plentiful, appropriate, accurate, and readable (e.g. the legends and axis text is large enough), answer one would be chosen here. It has been discussed if an analysis without any visualisations could still be of excellent value, and I think the answer here could be 'yes' but also think that visualisations are a great way to explain findings and so would expect to see at least one or two in all outstanding contributions. And indeed, in the analysis linked in above in the discussion around question one, the contributor supplied 15 visualisations to help support his findings.

Question 4 highlights the requirement to include scripts as part of the contribution. The idea around Open Source is that the project can be taken, and amended (forked), to hopefully improve on the work done by the previous contributors. If the code/scripts are not supplied in full, then this would be difficult or impossible to do and is not in keeping with the open source movement. Also, an interested party and/or a reviewer may just wish to reproduce and test the work, and this will only be possible with all the original scripts available.

Despite reservations with regards to, for example, a user picking up an analysis script, changing the dates, executing the script and producing their 'own' analysis, I think that supplying all the work done to support the contribution shows the effort involved and is appreciated by @utopian-io and the wider community as a whole.

Question 5 is there to help guide the reviewer in evaluating the originality and frequency of the analysis contribution. From my own experience, it is tough to come up with a new and unique analysis. (However, This may change when Steem grows and other open source projects appear, and could also be easier to find if the contributor looks outside the scope of Steem - which I will discuss a little more in the next section.) Weekly and monthly reports are common for the category, but this question seems to suggest that daily reports (with a reduced scoring) are also acceptable. I am weary of this, but the future vision of @utopian-io is that there could potentially be too many contributions to evaluate, and only the best will be picked for review. Personally, I would be unlikely to select 'daily' analysis contributions as being among the 'best', and attempt to seek those with original findings or covering a larger time period.

Question 6 is simple to understand, but still requires consistency amongst the reviewers. I think the question has its place, particularly with regards to spelling and grammar, as an important part of all contributions involve explaining what you are presenting in a way the consumer can understand.

Question 7 is perhaps the most important in the list above, and ties in closely with a definition of 'data analysis' provided in the previous post in this series.

Data analysis is a process of inspecting, cleansing, transforming, and modelling data with the goal of discovering useful information, informing conclusions, and supporting decision-making

The contribution should provide value and be useful to as many consumers as possible, particularly project leads and keen contributors to Open Source projects. With regards to the answer options, we (as analysis reviewers) have found that the wording of the final answer has not sat well with contributors in the past. A subtle change to this answer option could be:

• This contribution adds only a small amount of value to the open source community and ecosystem or the specific project.

The reason being is that the wording ‘adds nearly no value’ is felt to be pretty tough on the contributors (even though it might actually be the case!), and that changing this to ‘only a little value’, is not so harsh and may be less of a deterrent to future contributions. Other than this small suggestion, I think the question is a valuable part of the reviewer questionnaire and is indeed present across all contribution categories available at @utopian-io.

---

Ideas for future questions

To hopefully assist the reviewer in their assessment of an analysis contribution, keeping in mind that not all contributions will potentially be rewarded in future due to an increase in contributions, I would like to offer the following suggestions which may aid the process. These suggestions are again based around the findings presented in the first post in this series.

Introducing the analysis contribution with clear goals and focus will provide the consumer with valuable insight on what the analysis is there to provide.

• Is the analysis clear in defining purpose and goals?

---

Taking a reasonably unbiased stance when approaching a piece of analysis work should allow for a fairer and more open assessment of findings, and the conclusions provided. It is worth pointing out that a 'soft' bias could lead to a more interesting discussion around the work.

• Has the contributor shown any obvious bias in evaluating and concluding the results found?

---

Using 'trusted' data sources that are well-managed and respected should allow for an increased reliability and validity of the data, which should, in turn, raise the quality and respectability of the analysis.

• Is the data sourced from established and recognisable sources?

---

A question relating to the potential of the analysis to be able to contribute to future decisions made by project owners - How well the analysis is concluded?

• How valuable do you think the findings and conclusions of the analysis could be in their potential support towards future decision-making for the Open Source project?

---

The potential inclusion of a question relating to Open Source projects outside the scope of Steem (with a positive weighting towards contributions focusing outside the Steem ecosystem) could also be added to encourage a wider range of open source contributions? I am a little unsure as to the appropriateness of this idea but thought I'd mention it anyway.

---

Summary

As an ex-reviewer, contributor, and now Community Manager of the 'Analysis' category at @utopian-io, I am reasonably happy with the state of the questionnaire in its current format. It is difficult to predict the future climate and quantity of contributions to Open Source projects, with regards to analysis and as a whole at present, but the general thinking is that growth will come in time.

Developing the questionnaire to cover deeper aspects of 'data analysis', in light of the definitions and from what has been contributed so far, is not without merit but is also difficult, and I expect that multiple iterations will take place as we move forward. I hope that this work will at least spark ideas as to how the questionnaire could look in time.

For those of you still awake, thank you for reading and please give any feedback below.

Cheers

Asher