Measuring dev rel programs far beyond marketing activities

So many of the ways that we measure dev rel are borrowed from marketing. In a team where dev rel is as much about product as marketing, those numbers might not be relevant at all.

In this talk from DevRelCon London 2019, Ana Jimenez Santamaria of Bitergia shares metrics and KPIs for dev rel programs that focus on developer engagement, developer acquisition, and developer satisfaction.

Transcript

Ana: Before starting with the main topic, I think it’s quite important to get a little bit of context, ’cause I think everyone knows that if you want to know the State of Developer Relations from this year you can have a look at this. I think there is really cool insights here. And you can get more knowledge.

I think one of the main issues that I think also that’s safe during this morning it was also mentioned that is that dev rel roles are really diverse. A dev rel can be hired by a soft foundation. They can be hired by a company. They can be hired because they want to manage open source projects. Or just proprietary software. And that makes really, really difficult to bring up a standard way of metrics. And moreover, if you adding that when people hire dev rel roles, they can be hiring to different departments. This is from “The State of Developer Relations”, you already have all the information here. So, it turns out that dev rel roles can report to many different departments and those departments have different ways, and different KPIs to measure success.

There’s an evidence that it’s really, really difficult to bring up a standard ways of measuring the breadth, just because of that example. However, we shouldn’t forget one important thing about developer relations. dev rel roles are connected even though they’re hired by a foundation, for a company, they report to marketing, or they report to engineering. The mission of a dev rel is always build relationships with developers. And developers can be… seen in many different profiles or personas. For instance, this is some differentiations I usually use in my company. So, you can see your developers as users, so mainly they will be asking questions. You can see your developers as contributors, that normally have answers. Or you can even see your developers as maintainers, that actually create code. This is more typical in open source communities, for instance, and, well, this is from the “Open Source Contributors Funnel.” And I’m not saying we should do a funnel, like trying to get users to contributors to maintainers. That’s not the idea because you can just see your developers as I just want to get users and that’s fine, but you need to know the type of your community you have.

Then at the end, everything is related with community issues and ways to measure community. Because if developers are everywhere, a dev rel should be there too. And I think it’s also in this report, that the main things that dev rels focus more is on marketing. I understand that ’cause the 35% of dev rel teams report to marketing. But that’s not the only thing. Remember that we are here to connect other developers and to build relationships, and that’s community. And they can be doing code review, they can be on mailing lists, they can be asking questions in forums, they also can be on meetings and events, search for documentation, or be in charge.

It’s not just focus on half of those channels. That is what marketing is providing. It’s focusing on the whole picture to see what’s going on, actually, with your dev rel community. We came to this situation that we need to measure because we need to know what’s going on. And during this conference, I’ve been listening a lot of ways to measure dev rel actions in terms of acquisitions, engagement, and satisfaction. But the thing here is that I’ve always listened that from a marketing point of view. I have documentation and I measure, for instance, signups, or something like that that is really marketing focused. So in today’s presentation I want to share with you some ideas, some metrics, and ways to measure your developer programs in terms of acquisition, engagement and satisfaction, but far away from marketing. Like more focused in a community perspective. So let’s get started.

The first point is measuring acquisition. Before beginning with this, just to let you know, these are just some examples and at the end of the talk I would love to talk with you and share some other metrics you usually use or you would like to know more about, so we can complete this picture better. One of the ideas we can measure is the developer growth by software product/project. I’m showing you some examples, some real examples, this comes from the CHAOSS community. If you’re not familiar with that, this is an open source project from the Linux Foundation, and it’s basically focused on creating analytics and metrics to help define community health.

So they wanted to know what was going on with a community, basically. And this screenshot, you can see the activity over time from different data sources. That it’s basically where that community works. So they’re measuring Git, GitHub issues, I don’t know if you can see it well but it’s Git, GitHub issues, GitHub pull requests, and mailing list. So this is just basic stuff, basic metrics, but the idea of this picture is that you don’t just go ahead and see what’s going on on GitHub, ’cause it’s not the only channel your community is using. You also need to know what are the different challenges where my community is, and then get the whole picture with that. One of the other things we can try to analyze is the developer retention rate or the developer bounce rate.

In order to understand that, I’m gonna tell you a little story and then I will give you some context. This is a software that’s used; GrimoireLab. It’s open source and you can just try it your own and you will like. If you just go to the activity, sorry, community and go to attraction and retention. And imagine we want to measure Git, ’cause we are fond of our community, it’s really engaged, it does a lot of Git code, for instance, and that’s a lot of Git commits. So in here you can see the attracted developers and the last commit developers over a period of time. This still is the CHAOSS community, as I said, so if we go to attracted developers, we zoom in a little bit, we see here there was high peak. It was around March. What this is measuring is Git commits, the first Git commit from a developer.

So this is the attracted developers, 26. So we go in and we can see, okay, from this month the 26 developers when those developers made their last commit over a period of time. To get insight from that, we need to get a domain knowledge, that’s what the dev rel is for, so we can define that. For instance, a logical developer for us is a developer that made their last commit six months ago, for instance. So we will see that even though we got 26 developers, only two of them remained until today. ‘Cause we are talking here it’s July and this is well, yeah, just one developer remained until today ’cause the other ones, for instance, the most of them, the China developers that contributed to Git during March left the next month. So we can with that information, we can get the developer retention rate and also the developer bounce rate, going month by month. And this is just for Git, but we can do the same in other many different data sources. So then, about measuring engagement, I think this is the most fun part.

We have been talking also about profiling developer communities and getting personas. And I think if we take a look about the data sources about all the different challenge where the developer community is, they can help us a lot to find that. So if we take a look about all the different data sources aggregated, all the different channels where our community is. We can define if my community is a contributors community, is a user community, is a maintainer community, or any other type of community you would like to profile. This is just my example but as I said this depends a lot of domain knowledge. This is what dev rel is for, to interpret that data.

So in this case, this also still being CHAOSS as I don’t know if you remember CHAOSS is an open source project. So normally the type of community they will have will be a contributors community. Developers that actually create code. And that’s why the main activity they are showing over a period of time is Git. The blue stuff is Git. So in a different community, maybe they can find more GitHub issues activity. That will mean that maybe that community is not a contributors community; it’s a user community. There are users that ask questions and that’s why they have a lot of issues. But they don’t create code and they don’t submit code, for instance. And just to show you the example, this is from last 90 days, so we can filter for last five years. This is the same picture. This was from the last five years and this was from the last five years. But as you can see, this community is mainly Git and mainly a contributors community. And using this same visualization, a kind of visualization, the main concept here is try to aggregate all the channels of the knowledge you have about your community.

You can even try to better define a personas pattern. And I will go again to my dashboard…  Okay, here we can filter by data source, so we have Git, you have pull requests, etc. That’s what the community wanted to track. We can also filter by project, organizations involved, ’cause this is open source community, and then we have contributors. So we can search for a specific developer… Okay so, I choosing him because he is my CEO. That’s why. I always like to see his evolution, I think it’s really visual. So it’s here. This is the important visualization. As you can see, this is from last five years maybe? Yeah. When he was hired as a CEO, he began to add a lot of Git, add a lot of contributions to the project. But then he stopped, suddenly stopped, and he started to ask questions and had a lot of GitHub issues. So just imagine if I’m the dev rel and I say I’m gonna track Git. Just only Git because I think it’s the most important channel in my community. So if I only measure Git, well I cannot do that, but I would just see that this developer, for me is dead on January 18th. Because I cannot see anything else. I’m not getting the overall picture of my community. I’m just tracking just part of the channels. And that’s not true! He just started to get more responsibilities and that’s why most of the CEOs has started adding code and then they get into more managing things and getting more questions instead of actually getting into coding stuff. So with that we can even add more insights to our personas, the personas we are building.

Another thing we can analyze is developer engagement by software product or project depending. So for instance in the CHAOSS community, they have, they are running different projects. GrimoireLab is the one that we also use in our company, Bitergia, just to let you know. So we can see the most active community and the most engaged community project among the other ones. And we can also identify which projects had at some point some activity and then were left like the one on prospector. So it started to get some activity and then it suddenly didn’t get so much so activity. And again, I don’t know what this data means ’cause I’m not into the CHAOSS community but the dev rel should be have those knowledge, like what this exactly means. What happened then? What did we do wrong, or what can we do better in order to improve that engagement with the project itself? And finally, things related with measuring developer satisfaction. For that I will like to tell you a story.

This story is related with Uber OPSO use case ’cause we have been working with Uber OSPO. And for those who doesn’t know what OPSO means, comes from open source program office. So it’s another department or places where open source and all the related activities related with open source are centralized and growing inside a company. Well, this started because open source right now is eating the software industry. And I’m not saying this, the data is saying that. And the companies, most of the companies that owns an open source program office are these one. This comes from the TODO Group organization, so this is an open community, open group, for all those companies that has open source program office and just you get into one single community. And, well, I just had like new department for a dev rel role, I probably some dev rel roles are within OPSO? I don’t know if anyone of you in this room know? You? Okay there’s one. And I think that’s gonna be increasing during the next months. So be aware.

So coming back to the use case, Uber wanted to know how efficient are they being in answering, enclosing, well in merging pull requests. Okay? So they wanted to measure GitHub in this case. But not only that, they wanted to know how fast are they being when measuring Uber developers versus non-Uber developers. And it turns out that they tend to merge non-Uber developer took them more time when, rather than when, they wanted to merit a Uber developer pull request. So this data is… So we go here, this is GitHub pull requests and well this is the dashboard from Uber. If you go to its public dashboard you can go to Uber.peter.io and just take a quick look at if you would like to. To find more data. So in here we can see the time to merge. So we can filter by organizations. So these are the organizations involved, so we can say, okay I wanna know the data with only Uber. So this work from the last 90 days, let’s go to the last two years, for instance, and here we can see the median time to merge in days. That’s in days. If you just want to see the averages plus, multiply by 24 and that’s all. And then we can go and say, okay, I want to know the non-Uber users. So you just filter this, and you get the median time to merge for non-Uber users. And I think that’s really important to measuring satisfaction.

And finally, to sum up everything I said. The most important thing I wanted you to take from this talk is that dev rel is about building relationships. And it doesn’t matter, well it matters, the department you came from but always have in mind the real mission and the why you were hired as a dev rel and not as a marketing specialist. ‘Cause dev rel is about building relationship, and it’s about community. And it’s really really nice that you track things related with marketing but don’t only do that. Go more beyond that. The second thing is be near your developers and go to the same places with they are. Don’t just say okay the main channels are these ones and I’m just gonna track, I don’t know, GitHub and Slack and that’s all.

Okay, maybe your developer community is somewhere else. Maybe they also they can be in social media. They can be interacting with internal communication and so on but try to get the whole picture. And finally, but this is really, really, really important, understand that metrics takes a lot of time and for those who maybe are looking for the Uber.peter.io data maybe you see that at the first glance and say, okay this is really cool dashboard and nice but I don’t understand that. And the reason because of that is that it takes time and also needs to get the domain knowledge and you’re a dev rel because you have that domain knowledge. You are the ones that can interpret those data and tell to your boss, hey this is what is happening. You might not know what’s going on with just a bunch of numbers, but you are the ones that are gonna be there to interpret that and to get insight for your community. And that’s that. That’s everything. So if you have any questions, fell free to ask. You can follow me on Twitter, you can drop me an email, also LinkedIn… And that’s all. Hope you liked it.