How to bootstrap an Open Source project from scratch

Great ideas can be often quickly prototyped but turning a quick gist into a full fledged project can be a cumbersome task. The same rule applies for software development and for any type of a real life project.

The truth is that even though most of the people learn the mechanisms of the proper software development during professional career we tend to skip the kickstart phase of the project and usually there is only one or two persons that are experts in the initial integration of the workspace and tools needed to build, test and deploy the release package. Those activities require a steep learning curve. Some people read books, some learn from colleagues and some prefer to watch online videos to learn it. Do whatever you like but bear in mind that this is a slow process. You will not learn it in one week.

During professional development you often have to do what the company asks you for but you gain knowledge. Having taken part in Waterfall, V-model, Agile, Scaled Agile as well as in greenfield projects with no software development philosophy at all I recently discovered that I have now experience that allows me to kickstart my own Open Source project. It was difficult though since I did not have a checklist that I could use to boostrap the project and automate most of the tasks. That is why I decided to write this post. A few years ago most of the tools and the whole toolchains that I used were already installed locally at company's premises but nowadays almost everything that we need can set up in the cloud. Unless you do not do embedded software engineering, with a proper project bootstrap, you can automate the whole software project so that it can be maintained by a skilled person working remotely.

A brief introduction to Open Source

Many of you do not realize how much of the current software products use Open Source libraries. Open Source software is software that can be freely used, changed, and shared by anyone. I have been always an enthusiast of an Open Source software. I use it extensively on a daily basis and decided recently that perhaps it is time to give back something to the community.

Wikipedia has a good article on Open Source software and I recommend to read more there if the subject brings your attention. The good news is that excellent tools are sometimes available for free when used for Open Source software development. The current state of the art allows to bootstrap an Open Source project completely for free in the cloud. The concept is great. You can use a fully distributed set of free tools installed on remote servers running in data centers all over the world.

Software development process simplified

A software in most simplified terms consists of a set of source code files, libraries, resource files etc. that is compiled and bundled into a package. The package can be delivered to an end user or become part of a larger system.

There are many books on the software development process and different software development methodologies to choose from that are out of scope of this introduction due to complexity of available solutions. They all have something in common though. A picture is worth a thousand words, therefore I decided to draw a picture.

Open source project development life cycle.

Usually it all starts with one person, that hacks around and creates a proof-of-concept library or an application. Sometimes it's two or more people but for the clarity let's assume it's one person.

The software can be build and run on the local development machine. Then if source code needs to be shared the author can push the code one of the version control systems.

At some point the software is getting more and more complex and it is noticed by a community. People want new features and the feature backlog may get too big for one person to manage. The original developer(s) become maintainers of the project allowing more people to add their own contribution in form of pull requests.

The original author wants to keep his code base coherent, expects it to be tested and automatically deployed. This is where the correct bootstrap of the project in the initial phase pays off. The original author can still contribute to the project and become the maintainer. At some point the decision is made to take a current snapshot of the software and announce a new software release. In order to make the whole process robust and automated a few tools need to be selected.

How to choose your technology stack

Since I post on STEEM and I actually want to set up an Open Source STEEM-related project I will use the tools and frameworks that will allow me to access STEEM blockchain. The stack may be different for you depending on your needs but you can easily adapt the overall strategy and fit the tools that you need.

The toolchain technology stack that I compiled is using all tools that can be used for free:

You may want to ask why I chose Python based stack and not a Node.js based one. I chose to develop in Python, because I love this language. If you do not like the toolchain choose the tools that are convenient for you. I did not realize that I am Python evangelist until three jobs ago when I led a small project with a few students and we were allowed to use any tool and any language that we wanted. I chose then Python for the backend and Java/Spring for the frontend. I was hooked on Python since then. I left the company shorty after the project completed but Python has been still used there. When I found out that the Piston tool and written by @xeroc is implemented in Python I immediately picked this up. I also noticed that he was using almost the same set of tools that I was using and I thought that it would be useful for many people to create this post so that they can benefit.

Bootstrapping the project

A few steps are necessary to be checked to make project automation successful. I compiled a project bootstrap checklist for you. Feel free to adapt it to your needs.

Source code

• Store the source code repository in a version control system. Create a GitHub account if you have not done it yet.
• Use well adopted coding style. This will make your life easier in the future. I try to follow PEP8 for Python, recommend AirBnb style guide for JavaScript and MISRA for C/C++.
• Pick and use a version control workflow. I tend to use a successful GIT branching model or at least create a 'master' and 'develop' branch.
• Use well defined commit guidelines. See How to Write a Git Commit Message for an example.

Build locally

• Make sure you can build the project locally and describe the build process in a readme file.
• Make sure that you development environment is consistent. I am using Docker containers for all my local development. With Docker you make sure that you use the same environment no matter what is installed on your local machine. This also makes life easier for people working on different machines.

Build remotely in the cloud

• Set up a continuous integration system account and make sure that you can build the project when you push your changes to the version control system.

Test locally

• Add test infrastructure and write a few test cases. Check that you can run tests on your local development machine and evaluate test results in a human readable format.
• Use TDD or BDD if you can. It is easier to do acceptance driven testing in a later stage of the project.

Test remotely in the cloud

• Make sure that your continuous integration server can execute the same tests that your run on your local development machine.
• Ensure that you have a feedback loop enabled for broken tests. You must be notified when anything goes wrong.

Deploy to a staging area

• Use a staging area that resembles the production environment before your project can be deployed to production
• Make sure that you have an automated way to deploy to production environment

Update documentation

• Make sure that you maintain project documentation and that it is up-to-date. The documentation can be a single text file in the beginning but it may be beneficial to make it convertible to a nice looking human readable format.
• Deploy the documentation online. Users of your release will be grateful.

Release the project

• Make sure that you have an automated way to create a current software snapshot as a release on your local development machine.
• Make sure that you provide a release version. Semantic versioning is a preferred way to bump version numbers for many projects.
• Make sure that you can push the release to the outside world. Let people know that improvements in the next exciting thing that you are working on get noticed and the new release can be downloaded.

The checklist will be the same or very similar for all distributed software projects that want to follow continuous integration and continuous delivery principles. Have fun with your future projects!