Prospective users: If you don't have time to read this, just skip to Use Cases.
The Ansible Story
Michael DeHaan started out writing datacenter management software, and came to get involved in creating open source projects, including projects such as Cobbler, which was used to provision renderfarms, banks, gaming platforms, and even major social networks. In 2012, he launched Ansible, which grew to be one of the largest open source projects in the world, eventually growing to have over 10,000 contributors.
When managing such a large project, it is very important to know who is doing what, and who your star contributors are. Who is coming and going? What do contributors work on and how often do they stop by? What are the hot spots of the application?
Acting on metrics of community health can help strengthen communities, and a strong community leads to more happy users who help spread a product, and in turn builds even more contribution. Community quality is self-reinforcing, and as a result, critically important.
Despite building various earlier prototype tools to explore these community statistics, there was always a feeling it could be done better. To do it properly would require time and a much more rigorous approach.
(above: Historic and very-serious authentic whiteboard drawing from the Ansible Era)
The Microservices Story
After leaving Ansible, Michael worked for a communications software company with hundreds of microservices spread across 7 datacenter regions, with each of these microservices managed by different teams. To keep this organized, Michael helped build a dashboard that tracked every software repository and allowed recording of the owners and some compliance statistics.
It it became evident that the average software company needed tools like this application. With hundreds of services, how do they know what is actively maintained, or how to allocate hiring? How does a new employee find out who the best person to consult about a given software component is?
While this one company was being very proactive at building a dashboard to answer some of these questions, I knew most companies did not have time to build such a system. Could we generalize this idea? Could we combine it with ideas from the Ansible experience?
The NC State Story
Shortly after Ansible sold to Red Hat (which is now a part of IBM) in 2015, Michael started working with undergraduate student teams at NC State University in their CSC492: Senior Design course, helping them with topics such as software architecture. As we typically mentored 8-12 teams at once per section, it was often difficult to keep track know how each team was doing. We weren't merely interested in performance metrics, but identifying areas where we could and should share advice. Instead of the typical "How's it going? / Fine" conversation with teams, could we really get a better understanding of who is working on what, week to week, and be able to query that data in very specific time slices? How do we ensure leadership is well shared? Could software help us teach the course better? Of course it could.
The natural solution was to build the tool we wanted using teams in the CSC492 course itself. SourceOptics begins!
Our first version was built in the Fall of 2019, further refined over the summer in 2020, and extended heavily in the Spring of 2020. It was informed during the entire process by using SourceOptics against our own student data. Results were encouraging, and we have continued to refine SourceOptics to provide an ideal way to keep track of these diverse projects -- never quite in the same problem domain, programming language, or even timetables. It is now used to manage 5 sections of the course from a single instance and multi-tenancy options are being explored to host it to other undergraduate courses at the university.
SourceOptics is currently a fully usable, production-grade application. It can be used as a business planning tool, team building assistant, and a research platform -- and is not limited to a particular industry or vertical. Development remains very active, with refinements by student teams and from open source contributions.
As the stories above show, source control history contains a wealth of historically untapped information. The level of time-series based statistics we collect and display are merely scratching the surface of what is possible. We can continue to pull in outside data sources to our time series database, gather new statistics, and find new ways to present the data we collect. Essentially the future of SourceOptics is limitless.
Future exploration will continue to explore what other patterns we can find in individual repositories, or their aggregrate organizational collectives. As we call software engineering an engineering, and monitor our infrastructure, we too, must monitor the act of developing software better ourselves. Tools like SourceOptics can help us do that. In inspiring curiosity in those patterns hiding in our data, we can encourage continual improvement in the way we work together.
There's really no stopping how this application can evolve. We're adding features for pulling in (optional) time-tracking data, and also looking into how to graph other statistics and metrics in the time-series system.
If you have ideas for futures or if you have questions if SourceOptics is right for you, you can email Michael DeHaan here. Conversations are quite welcome!