Open source program managers must show ROI for their efforts. This guide provides an overview of some standard ways for organizations to evaluate their open source programs, projects and contributions.

Learn what to measure, how to determine success, and how best to use this information to advance your open source program goals, demonstrate effectiveness and gain support.

table of contents

• How to define success
• Why set goals?
• How to set goals
• Common goals
• What to trace
• Other metrics to track
• The final word
• Thank-you note

How to define success

Intelligent organizations understand the value of investing in open source development and set goals related to the use and participation of open source. But every open source program defines success slightly differently. The goals you set, and the metrics you track, will vary according to why you are investing in open source - whether it's to hire developers, bring new ideas and technologies through open innovation, reach faster times to market, lower development costs, or a myriad of other reasons.

It is important to set goals according to your unique strategy - and look for buy-ins from the executive team to ensure an open source strategy fits the overall business strategy. That said, there are some standard ways for open source program managers to measure success, regardless of their industry, product, or business strategy. These include:

Participation and level of participation of their developers in external open source projects
Their organization's reputation in the open source community
Their ability to recruit and retain talented developers
Public health projects open source organizations and important business projects donated by developers
How well they manage open source licensing compliance
Why set goals?
Before we dive into what is an open source program program and how to do it, let's talk for a moment about what you can accomplish by setting goals and measuring against them.

First and foremost, tracking progress against your goals helps you ensure that the open source projects you invest (whether external or internal) stay healthy - that they answer the community, represent the company well, and help meet the broader goal requirements of the open source program . Periodic tracking can help set benchmarks for open source projects and serve as an early warning system that allows program corrections if a project defects off track, falls from legal compliance, or just retires.

Careful (and strategic) measurements also make a great feed for reports to top management. Regular reporting helps ensure the program stays in line with overall business goals and strategies and helps program managers gain internal support for programs amongst the chief executive (especially if you meet or exceed those goals!)

The Office of the Facebook open source program, for example, periodically posts month-by-month results from open source projects internally and sends executive reports to management.

"Reports are just a great way to raise awareness.Although Facebook puts a high-value open source (as an organization), it's still good to market yourself internally all the time and show your worth." - Christine Abernathy, Open Source Developer Developer on Facebook.

Publicly publishing results on a regular basis also helps raise awareness of your organization's open source activities among potential partners, users, and developers.

Getting word about the results - the good, the bad, and the bad - increase the transparency, accountability, and credibility of your program, in the open source community. See examples of open source reporters from Facebook and Google.

How to set goals

It's okay to have a noble purpose for your open source program, but set reasonable expectations for how you'll get there and on what timeline. Firstly, it is helpful just to start measuring to define the baseline of performance. Prepare the right tools for collecting data and make sure the source of the data is clean and in a format that (and your manager) can understand. Many organizations create a metric dashboard for their open source program, to track all data in one place and provide project snippets that can help assess progress in an instant. (See our guide on Tools for Managing Open Source Programs).

Next, get all program managers and open source stakeholders - on Facebook, this includes technical prospects and project managers - and decide as a group where you want to go for the next 3-6 months (small and achievable improvement). At the end of that time period, review how you did it and adjust your goals and tactics for your next pass, based on your previous feed.

"I tend to find metrics based on what people perceive as pain, and try to change those metrics better to improve people's health." - Sarah Novotny, Community Kubernetes Manager at Google.

In addition to basic performance metrics, here are some things to consider when you set goals for your program:

• Strategy alignment: Do your goals match your core business strategy, product goals, and other internal business goals?
• Level of control: Does your program manager have direct control over the results, or is it in accordance with techniques, laws, or other functions? Set an achievable goal, within your control.
• Project variations: Goals can and should vary by project, depending on the objectives, community composition, technology piles, and other variables. For example, Facebook has noticed that Javascript projects tend to branch more frequently. They have learned (after many tracking cycles!) This metric is not necessarily the best indicator of project health for this type of project.
• Quantity vs. quality: Not all goals should be tied to numbers. Improvement processes that improve the quality of your project are just as important, if not more. Just because you hit all the numbers, does not mean your project is healthy. Conversely, very small projects that do not develop can also be extremely unhealthy.

"You can have half a dozen core contributors and dozens of active but non-managers, but there's healthy discussion, and pull requests are handled in a very short way, and sluggish feature discussions have a clear beginning, middle, and end. which is very healthy, but will not have a star or a fork in GitHub, because this might be a niche project, "Novotny said. "So I tend to see how society interacts with itself, how new leadership grows and is guided, and how each pain point develops."

Common goals

When it comes to measuring the success of your open source program, it's tempting to focus on the quantitative metrics for your project: the total number of contributors, the code line, the number of projects, etc. We will discuss what to measure to assess the health of your project. in the next section. But first there are many other important ways to measure the success of your program rather than strictly by these numbers.

"I think using metrics as a way to inform the trend is good.Using it as the only method of success takes you to trouble." - Joe Beda, the founding engineer of Kubernetes at Google and Co-founder and CTO at Heptio.

Kubernetes is one of the highest-speed open source projects in GitHub, which attracts over 80,000 commits from 2,760 developers in 1,181 companies over the past three years. But from the beginning, the project has been successful in terms of whether its users are interested in technology and using it, not by "some list of open source metrics," says Beda.

Here are some general purposes behind the open source program office, and the best way that program managers measure against this goal to track the overall progress of the program. Some of these objectives can not be measured, per se, but are related to process improvement, efficiency and quality. Others can be measured by conducting surveys or other assessment methods such as regularly, and actively requesting oral or written feedback. (Talk to your team!)

Objective 1: Ensure the efficient and lawful use of open source code.
This is where organizations usually start when they are involved in open source. They realize that engineering consumes a lot of open source software either in their infrastructure, or their products and services, or both. A program office helps to centralize policy and decision making around open source consumption, track its use, and ensure that the organization does not violate its legal obligations under various open source licenses. Programs can also track how well they help developers solve legal problems they may encounter.

Some of the most common ways of measuring against this first goal are to ensure that policies and processes work as they should and that the organization remains in compliance with the law:

• How much open source code do you consume?
• How well is the consumption tracked?
• The policy to use open source code is clear and developers are aware of it.
• Processes and tools to bring the code clear and developers follow it.
• Which third party code products and services are used?
• How many compliance issues are you facing and how quickly are they solved? (You have a compliance program, right? See our legal source from the Open Compliance Program for this topic more.)

Goal 2: Increase developer productivity.
Once you track and manage open source usage, you'll want to make it easier for developers to contribute to open source projects. If your technician has to pass through the red ribbon layer to send bug fixes or new features to projects that depend on your business, you waste a lot of valuable time and resources. Developers also save a lot of time in the long run by contributing upstream, rather than maintaining a fork apart from projects that generate technical debt over time.

"We are trying to fit in to be like the people on the marathon track who are giving water to runners We encourage developers to take a few extra steps toward us knowing that by doing so they will actually achieve their goals Long term We really trying to make the program to be a support service, not the speed of a bump. "- Gil Yehuda, Senior Director of Open Source at Oath (Yahoo + AOL)

Metrics associated with this goal are intended to block the wheel for the developer to contribute back to the open source project, as well as increase the overall amount of code that your organization contributes upstream. Once you remove the contribution barriers and make the approval process clear and fast, you can expect more contribution and efficiency. Things that need to be tracked include:

• The number of commitments made to external projects is identified as a strategy for the organization
• Number of contributed developers. Also, who are they and which projects do they rely on?
• Number of project managers employed by the organization (leased and planted)
• Health projects for the projects you contribute
• Sentiment analysis: your organization's reputation in the open source community
• Do developers know the policies to contribute? (You have one, right?)
• Are they following the process to contribute? (ie should they sign CLA, etc)
• Did they ask for help and did you immediately give it away?
• Amount of time between software releases - is it increasing or decreasing?
• What is the engineering cost associated with upstream contribution vs. maintaining branched code?

Goal 3: Create and develop an open source project.
This is the main goal of many open source programs in general, organizations that focus on techniques like Facebook, Google, Microsoft, Twitter, and many others. They create hundreds (or even thousands) of open source projects that aim to solve the problem of hard technology. The goal is to attract outside users and contributors who bring new ideas and help advance technology more quickly - a University of California, Berkeley, professor Henry Chesbrough's concept of open innovation.

"How do you really get the smartest people in the world working in your company? Well, you open-source stuff and then you convince them to contribute to your project!" - Chris Aniszczyk, Executive Director of the Open Container Initiative and COO of the Cloud Native Computing Foundation

Many data points available to measure project health are key to tracking these goals (see top 5 in the next section). But there are also other considerations:

• Is there a clear policy for creating new open source projects and developers knowing it?
• Is there a clear and easy process for creating new projects and does the developer follow them?
• How easy is it for outsiders to contribute to your organization's project?
• Project keepers are friendly and helpful
• Projects are well maintained and supported
• Code is well documented
• How to contribute well defined
• Other quantitative metrics such as the number of new contributors, the number of issues created, the * amount of time it takes to close the issue, etc. (See next section)
• The amount and diversity of external contributions your project receives
• The popularity of your organization's projects: GitHub stars, social media followers, etc.
• Number of users in deployment and / or production
• The number, breadth, and quality of projects your organization launches. For example, infrastructure * * projects related to mobile data or data, etc.
• Improved performance in your project and related products
• Time between releases

"For one-and-a-half we decided that we wanted to have more excellent projects launched and make it a little more of a tight process in terms of what we opened, but there was no number behind it, it just meant reviewing the process about how the project open source launches. "- Christine Abernathy, Facebook.

Goal 4: Recruit and retain developers.
Participating in and creating open source projects as an organization is a great way to attract developers - and instantly install them quickly, with fewer resources devoted to training. Developers who use or contribute to your project are familiar with your processes, tools and technology when they join the organization. (See our guide on Recruiting Open Source Developers.)

But your chances as a program manager will not have a direct role in recruiting developers, and it may not be clear what is the direct effect of your organization's open source participation on hiring. To help make a more direct connection between effort and program recruitment, Facebook conducted a biennial survey that asked several new employees for three basic questions:

• Do they know the company's open source program?
• How does awareness affect their decision to join the company?
• Does their experience with open source apply to the work they do now?

"We use surveys to measure the health of our open source culture and it talks about the overall effectiveness of how people view our open source projects." It's nice to know the upward trend figure. " - Christine Abernathy, Open Source Developer Developer on Facebook.

Other common metrics for hiring developers include:

• Which open source projects employed and donated by employees
• How new recruits hear about the organization
• The number of developers you follow through an open source project
• How many project managers do you recruit (and grow)
• How long does it take for the new hires onboard
• How open source developers advanced in their careers
• Developers' contributions are valued as part of work performance
• Developers are recognized and rewarded for contributions
• Developers receive help and support in contributing

Goal 5: Promote open source culture.
Much how open source programs contribute to technical talent also leads to the cultivation of open source culture and practices within your organization. That's because organizations that embrace open source are known as good places for developers to work and innovate. Open source program managers are often ambassadors for the open source ethos within their organizations, as well as overseeing collaborative policies and practices.

It is important to track the development of open source culture within your organization to measure the effectiveness of your program. Some common ways to measure adoption of open source culture include:

• Awareness and support for open source strategies and programs between individual management and contributors in all departments, from engineering to marketing and public relations
• Branding and awareness of the open source community - how your organization is perceived
• Participation - You actively participate in the open source community in a positive way.
• Training and guidance - You work with developers to improve their contributions and open source projects, find opportunities to contribute, and learn about the tools and processes of the open source community, ensuring that contributors receive support from their peers and managers.
• Adoption of common tools
• Code quality is acceptable for open source / external consumption
• Advocacy on behalf of the organization - attending and speaking at conferences, writing articles or * * * tutorials, etc.
• Sponsoring foundations, groups, or hackathons

Goal 6: Align the interests of the open source community with product interest.
Community advocacy is a fairly new role, but is increasingly popular in open source programs. You will often act as a liaison between the project developer and the adopter community - representing the voice of external users who build open source code and funnel information back to the product management team.

This is an important role that ensures your products and services benefit from your open source community, so your open source program remains in line with the organization's broader business strategy and objectives. Some of the metrics for tracking success in your advocacy include:

• How many contributions come from outside the organization?
• How many full-time contributors are outside your organization?
• How much code that contributes externally makes it back into the product?
• How many employees come from open source contributions?
  What to trace
  There are many ways to measure success and track the progress of open source programs. Project health is not the only thing that can be traced, but it's still important. The problem is, there is so much data available around open source projects. Whatever data you can get, you can collect and track. Again, the metrics tracked by each organization - and what they do with the data - rely heavily on their own goals for the program, and its unique challenges in the open source market and community.

"We collect the data we can because the data is available but we do not live in numbers.We live in making sure we have the right results." - Gil Yehuda, Senior Director of Open Source at Oath (Yahoo + AOL)

For some program managers (crazy or fully automated) the answer is just to keep track of all things. But for large organizations in particular, there are so many projects that it is impossible to track them down and make sense from them. So what are the real indicators of open source project health?

Here are the top metrics to assess the overall health of the project in your open source program. This is only the starting point for more rigorous and prudent analysis. Remember that this is a tip to help program managers responsible for ensuring the health of multiple projects. The project itself must also track their own metrics for health. The GitHub guide to open source metrics provides a good idea of ​​what project managers should pay attention to.

These numbers are easy to collect from GitHub using free and open source tools, as well as commercial offerings. Measure them periodically (monthly, quarterly, and annually) to help benchmark progress for individual projects, and rolled into the overall number for the program as a whole. Use them in reports to management, and to help your developers improve your project.

"We periodically just try to check and see whether the project is healthy or not, and just give them advice on what to do better, but we do not manage it directly We just give them data and then nudge them when we can, or when we should do it. "- Christine Abernathy, Facebook.

Number of contributors (and external ratio to internal contribution)

The project starts with most contributions coming from internal developers and evolves to include more outside contributions because the source code is used or branched. The longest sustainable healthcare projects of all time have a very diverse community with the largest contribution coming from other companies in the project ecosystem that have used commercial dependence on the code. (Remember the 1,000+ companies that contributed to Kubernetes?)

Projects that consistently attract new external contributors are likely to do a good job of defending projects, welcoming contributors, and incorporating community feedback. (Note: This may still apply to projects that do not grow their contributors base!)

Number of pull requests filed, opened and accepted (and the length of time they remain open)

If a contributor finds a bug or has a feature request so they can (and have permission to) patch or write it themselves, they do so and submit it as a pull request (PR). Tracking the number of pull requests, and what happens to them, shows how many codes are filed by contributors outside of your work and thus an indicator of the level of importance outside your project.

The length of time the PR stays open also shows how responsive and friendly your project manager is outside the contributors. If a PR is waiting too long without a response, potential contributors might take their good ideas elsewhere.

"If we have a good project, maybe we do not have an open demand for more than that, at least I say, at least two to three months, and that's actually a lot." - Christine Abernathy, Facebook.

Keep in mind that this metric depends heavily on the size of the project. The smaller Facebook projects will try to keep the number of open pull requests at 10 or less. But keeping PR on this boundary will challenge a larger project that has a lot of input from the community compared to the number of managers. Reviewing the pull request takes time so that larger projects tend to have longer open PR.

The Facebook open source office often runs queries in the database and takes the top five projects with the most open PR. They point out some issues and then take the opportunity to open a dialogue with the project manager. They ask some questions to them to get the root of the problem and see what might help solve the problem. More often than not, it's just a matter of refocusing their attention and reminding them that it's important to keep people happy. But sometimes, digging numbers points to deeper problems with a project. A lot of open PR, or old PR, could mean just one or two people who defended the project - a potential red flag.

The number of issues raised (and the length of time they remain open)

Users who do not have the time, permission or ability to submit a pull request, but encounter problems with your code may submit bugs and feature requests as a problem. The number of issues, and how they can be handled, can indicate the level of user adoption of your project as well as how responsive the manager is to the user's needs.

This amount of course depends on how the issue is tracked. For projects that only use GitHub to track bugs, the problem may remain open for a much shorter timeframe than projects that use GitHub for issues that include feature requests. This consideration pulls down, or drags, an era of trouble.

Number of commits per contributor (external vs internal)

The amount of external conducting a project relative to the whole is another indication of how effectively the project is innovating in openness - bringing in new ideas from the outside. Healthy projects will see the ratio of external contributors increasing over time. Measuring the number of commits per contributors also helps assess whether your project attracts new contributors and if the new contributor persists.

Number of external adopters

Any open source project should have a way of tracking organizations that choose to adopt software in a production environment. Whether it's through the ADOPTERS.md file or a simple list in README, it's important to keep track of this list and make sure it evolves over time. If the number of external adopters stops growing or shrinking, it can signal everything from project maturity to project obsolescence.

Number of projects created or donated to (overall program)

Track this metric for every project your organization releases, but also projects that are actively contributing to your developers. In the process of creating your open source strategy, you should have identified the important business projects your organization is using and allocating some investment to contribute to those projects. It is important to measure the success of your open source organization not only by the health of your own open source project, but also because of the overall open source activity. This includes the health of projects you rely on for product development and business operations, and ensuring your organization legally comply with open source licenses of any project you use or release. (See our Open Compliance Program publication.)

Other metrics to track

Basic project metrics are a good starting point for getting your thumbs on the pulse of your open source contributions. But successful program managers require deeper dive into other key metrics.

Here are many other things that you can and may have to track, depending on your goals. Remember that the number itself is not the goal - this is their process of tracking over time and finding patterns in the data that can inform project and process improvements. Measure for each project, and across projects to see comprehensively the outputs and results of your program.

• Popularity / awareness
• Visitors to the project site
• Total number of followers in GitHub / GitLab
• Number of followers on social media accounts like Twitter, Facebook or LinkedIn
• News clips and media mentions
• Number of meetings held and hosted (eg via meetup.com)
• Influence
• The number of employees in the manager / manager role in your strategic project
• The diversity of your project contributors
• Patch rejected, and why
• Adoption
• Number of downloads
• The number of forks made
• Number of external company contributions
• Stages of adoption (# placement in PoC and production)
• Number and quality of commercial dependence (product) - It can be tracked by looking at companies that contribute to your project, as well as following news and trades.
• Program costs
• Staff: engineering, PR & marketing, legal
• Infrastructure and support
• Tool
• Attendance and travel conference
• Exercise
• Membership and donations

The final word

Organizations evaluate their open source programs, projects, and contributions in whatever way makes the most sense for their needs. The most important thing to remember is to set additional strategies and goals to achieve them. What you are tracking, and how, will naturally follow.

Thank-you note

Contributors:

Christine Abernathy, Open Source Developer Advocate on Facebook.
Chris Aniszczyk, COO of the Cloud Computing Foundation.
Joe Beda, Founder Engineer Kubernetes at Google and Co-founder and CTO at Heptio.
Sarah Novotny, Community Kubernetes Manager at Google.
Gil Yehuda, Senior Director of Open Source at Oath (Yahoo + AOL)

This resource was created in partnership with the TODO Group (Talk Open, Develop Openly) - a professional open source network group program at The Linux Foundation. Special thanks go to open source program managers who contribute their time and knowledge to create this comprehensive guide. Participating companies include Autodesk, Comcast, Dropbox, Facebook, Google, Intel, Microsoft, Netflix, Sumpah (Yahoo + AOL), Red Hat, Salesforce, Samsung and VMware. To learn more, visit: http://todogroup.org/

---

Posted on Utopian.io - Rewarding Open Source Contributors

---