Getting the measure of developer relations

At Elastic and, before that, Rackspace, Shaunak Kashyap devised ways of measuring the value and impact of developer relations.

Here, at DevRelCon London 2015, he shares his experience and advice on measuring dev rel.

Transcript

Yeah. Hi, everyone. I’m Shaunak from Elastic. We build Elasticsearch and some other products. I’m not here to talk about that. What I want to talk about today is how we use metrics and data to run our developer relations program at Elastic. And specifically, I want to talk about just three metrics because, as any good developer evangelist knows, you talk about things in threes.

So the first one is how we measure community engagement on GitHub. We have a lot of our products on GitHub. So I’ll talk about how we measure our community engagement there. Then I’m going to talk about Meetup groups. We sponsor a lot of Meetup groups around the world, and managing them at scale becomes difficult. So I want to talk about how we use data to do that.

And then I’m going to switch gears a little bit and talk at a higher level, a more strategic level, how we use data to even decide where in the world we want to engage and at what levels. So let’s dig in. So this is a really simple chart that just shows on the X-axis there’s time. It’s really small.

The X-axis is time. The Y-axis is the average first response duration. And I’ll explain what that is. Basically, we have many projects on GitHub, and our community is very active. Our developers are very active, engaging with the community over there. And primarily, the way the community engages with us on GitHub is in two ways. They will finally choose, submit bug reports, etc., and they will also submit contributions like pull requests, documentation, etc.

So what I thought might be interesting to measure is how long does it take for them to get a response from someone, time to first response because I feel like that’s a reasonable measure of engagement, you know, how well are we doing. And what you don’t want to see is these spikes, right. It’s okay if you have these occasional spikes.

You don’t want to see this thing going up and then staying up there because that means something is wrong. People aren’t responding. So we defined a first response very simple. We defined it as any time someone other than the author comments on an issue, or they assign a milestone, or they assign an assignee, or a label, etc.

So if any of those things happen and they’re done by someone other than the original author, we say that’s a first response. And then we just measure them over time and see if there’s any, you know, any bumps that stay up there. So far, this is not a problem. Right now, we’re not really using this for anything per se, we’re trying to establish a baseline just trying to see where each of these projects, you know, sort of end up.

By the way, each of these lines is a different GitHub project. So we want to know where things are, and then we can… Oops. So yes, we just measure these over time for various projects, and we’re just trying to establish a baseline. There’s obviously a lot of noise here, a lot of shakiness, but we can still kind of get an idea.

And the idea is if you see something that is steadily moving up or is moved up and then stayed up there, that’s not good. So we’re going to try to use this to find out what’s going on. And we really just want to have conversations with the developers at that point, what’s going on. There’s probably never going to be any hard goals here. Like no one is going to get fired for not responding, you know, but we want to know what’s going on. Maybe everyone went on vacation or a conference or something, and that’s probably okay.

The next one is about Meetup groups. So at Elastic, we sponsor about 70 or so Meetup groups around the world, and just to put that in perspective, what that works out to is about four Meetups a week, three to four Meetups a week among Elastic that are happening somewhere around the world.

That’s a lot of events to keep track of and what tends to happen when you have that sort of scale is there’s a lot of out of sight out of mind. We found that there were certain Meetup groups that were really active and then ones that are not. And the ones that are not, we tended to just forget about them, which is not great.

That’s not a great way to build a community. So we came up with this really simple chart. X access is just all the Meetup groups that we sponsor. Y-axis is time since their last Meetup in weeks. That’s it. So what you’re seeing on your right, yes, on your right, is our very active Meetup groups. Groups that have had Meetups in, you know, recent weeks.

And then the opposite on the left, which is groups that haven’t met like 95 weeks, for example. That’s a long time. Right? Almost two years. So how do we use this data? Well, what we do with this is we then reach out to, maybe we’ll pick a few of these, and we reach out to them and the organizers or the core organizers and see if we can help them, you know, just have a conversation.

Again, like, what’s going on? Are you having trouble finding speakers or venues, or would it help for us to send you some swag to brag? I mean, to get people to talk, etc. So, by doing that, you know, we can do this in a data-driven fashion. Sometimes we also have to make the hard decisions and kill a Meetup group. That doesn’t happen very often, and we don’t like doing it, but if there’s one all the way over here and there’s just nothing going on there, maybe it’s time to shut it down.

And we’re nice about it. We email all the members and tell them, “There’s these other Meetup groups that are related. Maybe you can have Elastic nights every once in a while over there.” We don’t have to stop talking about Elastic or Elasicsearch at all. Just maybe there’s not enough appetite for a Meetup group there in that region. And the last thing I want to talk about is this global event strategy, fancy word.

Right? So we’re a very small team. We’re are about six of us and with a relatively small budget as well. And so, we had to figure out a way to prioritize where we’re going to go and be active in the world. And I’ll talk about two things.

I’ll talk about the results first, like what we ended up with, and then I’ll talk about how we got to those results. So what we ended up with is three buckets. We ended up with a list of what we call mature regions, and we defined those as these are regions where there is a lot of awareness and adoption of our products, and whatever we’re doing there is working, so let’s just keep doing it.

Just keep that the same sort of standing rate. The next bucket was what we called emerging regions, and this is where there’s a little bit of an option, possibly very little awareness as well. This is where we want to actively engage all around the web. And by the way, when I say engage, I’m talking about sponsoring conferences, speaking at conferences, attending Meetups, etc.

Of course, we want to be all over the world at every single event, it’s just not possible. So we do have to prioritize. And then the third bucket that we ended up with was every other region of the world. This is where they don’t fall in any of the first two buckets. What do we do with this third bucket of regions? We don’t completely stop.

We don’t refuse. We just lean on the community a lot more heavily. We sponsor them. We support them. We’ll ship these what we call “Meetup in a Boxes” to them. So we try to do things that are lighter touch, are less effort for us, but support the community and get them to help out. So we ended up with these three regions.

I can’t really reveal what the regions are, sort of might reveal some private information about our financials. I don’t want to go there, but I will explain how we came up with these lists. So the first thing we looked at is we have this little form on our website whenever you want to watch a webinar.

So we use that as a leading indicator. We say that if you’re signing up to watch a webinar, we’re going to assume you’re interested in learning about us. And we can use this to figure out where people are signing up from around the world. So we use that as one factor. The next one we use was Google Trends. We just looked up what people are searching for, terms like Elasticserch, or Logstash, or .

These are our products, and where are they searching from, so we just looked at those. And then finally, on the other side, we looked at revenues, you know, we looked at where are our sale activities, where are we doing well and we used that as a lagging indicator.

So just to clarify, the first two are leading indicators, the last one is a lagging indicator of developer relation activities. And then we sort of did this clever, weighted intersection of these lists, sprinkled some intuition on top from our leadership, so you know there’s always going to be someone in leadership saying, “Wait. I don’t see Brazil on that list. I think Brazil is a good bet.”

That’s okay. I think having intuition isn’t necessarily, you know, a bad thing. So mix all that, and then we came up with those three buckets. And that’s how we drive developer relations at Elastic with data. And that was, I guess, perfect timing. Thank you.