Measuring developer productivity through their own product measurements

Stef: I am Stef CEO and Co-founder of avo. And we help companies ship fast without constantly compromising their data quality for their feature updates. And today I wanted to talk about how we measure our customer's developer productivity when they are doing their own product measurements basically, and sort of our journey as a company to getting to that qualitative metrics for our customer productivity. So the schedule that I had in mind is a quick intro talking about our qualitative learnings of our customer value and our customer success and how that was an input into how we measure this quantitatively. And then take that forward and then go into the ask me anything. So super excited about that and keep your questions written down so that we can have a fantastic conversation. In general, I like giving talks, but I also just love seeing audience and I quite miss that this time. But it's a good, an exciting experience.

So I hope we'll be able to make up for it in written comment communications.

Yeah, so let's dive into it. So I am a mathematician and philosopher by training and I have been in data science for around 10 years now, starting in genetics, doing and writing software and doing algorithms, writing algorithms and distributed computation and analysis, correlating DNA to physical traits. Super exciting when a mobile game called QuizUp reached 1 million users in their first week, which was the fastest growing app at the app store at that time. And they decided to look for their first analyst asked me to apply and I decided to do that and eventually became then head of data science and Growth Lead. Also got the opportunity to open up a B2B office for QuizUp in New York with a fantastic team that was very fun. And through this sort of data experience at QuizUp, this mobile game, which eventually reached 100 million users around the world and was backed by Sequoia and Tencent and we grew the company into 100 people, I learned a lot about data quality and developer efficiency and productivity around making sure we measure our products well.

And so what I have in mind by that is, for example, all of these different versions of how we tracked when a game was started, basically because there are maybe 20 different code paths for how you can start a game in a mobile game app.

And the typical way to track these things and log this user behaviour is by using a generic SDK and typing in the strings that describe the user interaction that the user interacted with. And as you can imagine, it's horror to work with data that looks like this, but it was also horrible for all of the developers involved and just figuring out what they should write in their code to log data. And so basically we decided to build a lot of tools internally to increase developer efficiency so that they wouldn't have to be concerned about how to spell the user interaction and so that we could have better data quality.

Built a lot of tools for this internally. And then fast forward a few years later when QuizUp had been acquired, me and a couple of friends started another company and it was only five months into that company when we actually shipped a feature update based on incorrect data. And that was a bummer for me personally, but also a huge realisation because I realised the amount of amazing work we had built internally at QuizUp to prevent situations like this. And I also realised that it was going to be a really long time until we would have the throughput to build that infrastructure again. And so fast forward a few more months when we had actually had some discussions with colleagues in the industry talking to data leaders at amazing companies like Twitch and Spotify and Airbnb.

And everyone had built a bunch of internal tools to solve this problem problem mostly from the perspective of increasing data quality and increasing developer efficiency.

And so basically we decided to start doing that instead of the thing that we started the company to do. And because before, basically everything now, if you are about to release a feature, developers now not only have to write great code to get product into the hands of users, they actually have to write great code to get data points into a database for each important user interaction that the customer may be doing. And to solve this, we are trying to solve the fact that teams have been constantly forced to choose between product delivery, speed and reliable insights. That's what we solved at QuizUp. And now I'm so lucky to have this amazing team working with me at AVO on building a solution to this problem for way more people in the world. And obviously what we are doing is, and it's built into this, we are trying to increase product delivery speed.

That means developer efficiency and developer productivity.

And so I wanted to take you through the journey of how we went from qualitative to quantitative and measuring this today. So keep your questions coming and also some fantastic feedback if you have any on how we can do better. So some of our qualitative learnings in our customer value and success had two themes, very clear themes, operational efficiency. That's what our customers kept talking about as the value of AVO and increased data quality and reliability. And I'm going to show you a few quotes that speak to this point, but we were really glad to hear that feedback from our customers because this is exactly our mission. So for example, the director of data science at Patreon told us that instrumentation of analytics used to take one to four days, but now takes 30 minutes to maybe a couple of hours.

That's amazing. That's huge operational efficiency, increase that right there.

Head of analytics at a daughter company of TripAdvisor finished setting up analytics in weeks for a new product and it would have taken at least half a year without avo, but based on his previous experience. So that's also amazing operational efficiency. Head of engineering at rappi, within their first week, AVO was already helping them discover tracking issues like missing properties and type mismatches on the user event that described by user action. And now we know what to fix and we can do that with vo. So this is data quality as well as operational efficiency. And a senior product manager for business intelligence at one of our customer's business of fashion said before vo, we would have a meeting plan things in a Google sheet and then manually do what Ava automatically does. Now we don't meet the meetings and across a year AVA saves at least weeks of work per product, developer, manager and analyst. So that's again operational efficiency.

And with alvo, we could have be a small non-specialized team and still have sophisticated data workflows. So that's sort of data reliability with way less efforts.

And we had a few more quotes like these operational efficiency and now the question was how do we measure this quantitatively? Because definitely it's definitely a part of an early startup to stay or any company, any company building any product to stay very close to the customer, listen to the customer hearing the user feedback, but ideally we want to create an environment where we can actually measure those indicators, the qualitative indicators quantitatively, and then monitor how the features that we release actually impact those quantitative metrics. And so we decided that and realised basically that we care about the turnaround time of wanting to implement analytics until you have actually implemented the analytics and released them into production. And so to give some context for that, for the metric that we then decided to implement or use to measure this turnaround time, I'm going to quickly walk through the workflow of ava.

So the AVA workflow is a branched workflow for updating your tracking plan. Your tracking plan describes all of the user actions that your users have with your product so that you can be measuring those important conversion rates.

And before avo, just like the product manager for business of fashion was saying, the way to plan your tracking updates would be through a meeting, then a Google sheet, and then a bunch of QA processes. But with a, there's a loop that evolves around sort of a product manager creating a branch from their tracking plan within vo, just like developers create branches in their code base for their feature release and those branches sort of follow each other through the release process of a feature there they define a metric that they want to measure the success of the feature, and they define also the specific user actions that go with that.

They then get a peer review from someone on the data team or a product leader or something like that. The developer then implements this change using actually type safe code rather than these generic analytics track functions that I showed in the beginning of the presentation. And that entirely eliminates these endless versions of the game started event. And instead of typing all that stuff out, they would just type AVO game started and use a function for that. And then the avo SDK sends the data eventually into whatever analytics platform you're using. Then the branch gets merged and then you do this loop over and over again and this is how product teams work.

They release features intuitively measuring the success of the features. And so what we wanted to measure and thought we would measure was the conversion rate to merge those branches because that would be an indicator of how quickly they would be able to finish their loops and how efficient we were making their operations around product.

The definition of this would then be the user action of opening a branch to, until in the conversion funnel from opening a branch until the conversion funnel of closing a branch, we found room for improvement when looking at the data as well as in our customer feedback. The specific improvement that we were excited about is that we learned that customers were desire pathing, an extended branch workflow. They were building their internal Slack channels where they were sort of assigning collaborators at mentioning the people who they wanted to specifically get for review. All of those things, everything that typically happens in GitHub, for example, when you're doing code reviews. And so we decided to ship something that we called branch collaborators with improved notifications because we learned from our customers that they really wanted to be able to efficiently communicate within the tracking plan, change suggestion and what we learned using our metric.

Or before we go into the learning, we extended the definition of the metric.

So instead of talking only about the conversion to merge a branch, we talked about the conversion to merge collaborative branches within one day. So how high can we get that conversion rate was our thought because we want to increase this shipping efficiency. And the definition for that then extended a little bit. It went into being the branch open to branch merge events. We also added that it had to have at least one comment on it, so it was collaborative and that these two events would to happen within the same day. And with shipping this, using these quantitative metrics for measuring developer productivity, we actually saw a 70% increase in this conversion in the first week after release. So obviously the entire team is super happy with that. Also, I'm so glad to be able to quantitatively measure the success of our feature releases.

That really increases our operational efficiency. And so this is where we are now at, this is how we measure developer productivity of our customers. We are then also measuring exciting things like workload balance, sort of workspace activation and all sorts of exciting things, qualitative things like that, and quantitative. And yeah, we're growing our, right now we have built our product development cycles to specifically raise metrics, quantitative metrics that we set for measuring developer productivity. So.