Made by
Made by
Mark Boyd
Measuring the impact of a developer evangelism program can be a tricky business. In this DevRelCon London 2017 talk, writer and API industry analyst Mark Boyd offers four practical strategies. Along the way, he offers a look at what aspects of data can be most useful as evidence of success, the ‘YAAARRRP’ model, and how you can make a positive argument for evangelism.
It’s great to have you all here. Thanks for attending this session. I hope you’ve enjoyed the day. I’ve followed along on Twitter, so I don’t give everyone my flu, and it seems pretty exciting as far as the engagement with the talks and everything today, so I hope you’ve gotten a lot out of it. So my talk is around “Storytelling with Data” because I was trying to think up a good title that would get you in here. But, really, what I would have called it would be “Building a Data Evaluation Framework for Developer Evangelism,” which, probably, I wasn’t sure would get as many people to come along to it.
So a little bit about me, so there’s my Twitter handle at the top because I’ll make sure I put these slides up on Twitter because there are some links in them, so you don’t have to worry about that. So I do some industry writing and analysis around the API industry. I work with businesses around helping them open their APIs, and then I’m working with John Musser, the founder of ProgrammableWeb and API Science, to build a tool to measure platform intelligence. And also my background is, actually, in community health data evaluation, so I was doing a lot of work with cities designing data evaluation systems around health and well-being, for local populations, and I use a lot of that sort of background in the work I do today.
So what I’m gonna do is talk through what stories do you need to tell, a little bit of the research, how to build in developer evangelism evaluation framework, the YAAARRRP model, which everyone seems to be referencing Phil Leggetter, and that’s one of his ideas. And then an example, you know, just starting some ideas around what sort of things you should measure. But can I just ask quickly, how many people here are working for, like, API providers where your central product is an API?
Yeah. And then how many are working with, like, SAS tools where you’ve got an API? Okay, yeah. Is there anyone working for a more traditional business who is then opening APIs? Yeah, we’ve got a few of everyone. That’s great. Cool. And so some of the stories that you might wanna tell with your API. So you might wanna, for example, you might wanna be able to tell people that your STKs are actually bringing in new business to your company, and then talking about how that’s happening, and the contribution those STKs are making as far as bringing in new developers who are then building new products that are then selling.
You might be wanting to show that there’s…that you’ve got a stronger developer market share from your…because of your APIs and that that’s, again, increasing revenue, and then, as a result, you might be wanting to ask your CFO, or some higher-ups, to be able to allocate more budget to your developer evangelism program because it’s sort of turning the wheels of your company’s budget. Or you might wanna be trying to push the idea that you actually need a broader range of API products because you’ve had success with one API, and then these are meant to be the golden eggs there of the multiple APIs. Apologies for my bad sketch drawing.
So they’re the sorts of stories, and then data will back all of that up. How many people are sort of measuring elements of their developer program at the moment? Just a couple? Yeah. Okay, cool. So then, hopefully, then this will be useful as far as getting started with some of those ideas. Here’s some ratings. Like I say, I’ve got…I’ll put it up on Twitter, the slides, so you can click through to those, but they’re a really good start for understanding some of the ideas around “How do you measure your developer engagement program.”
We’ve also…I’ve also been looking at speaking with a lot of developer programs and asking about what people are measuring at the moment. So monthly active users is quite a common measure that people are using. Number of API calls sort of demonstrates the overall usage of your APIs. I quite like…I’ve heard that some people are measuring the specific endpoint usage because that…from out of that they were then been able to tell some of the use cases that the end developers are having with their APIs.
Net Promoter Score is a big one. I think that this one you sort of see more from companies that are either moving to IPO, or have, actually…have then sort of moved to be on the public market because that’s something that their boards understand, and their shareholders understand, so that’s sort of a big one. Personally, I find it too much of a vanity metric, and I think that there are cleverer ways to measure Net Promoter Score.
Like, so going back to my community health background, one of the things we used to do would be interview people at this community centre around what did they wanna do in their free time because we had a bus access, and we wanted to offer that bus to these community centre members.
And the community centre members all said, “Oh, we wanna go to the gallery and we wanna go to museums.” So we organised a bus and no one came on the bus to the gallery or the museums. And then we asked them, you know, “Why didn’t you come?” And it was like, “Well, that’s what we expected to say every time. Actually, we just wanna go to the new Westfield shopping mall that’s just been set up.” And, you know, so we ended up doing the bus out to there. So sometimes when you ask people, you know, if they…some stuff, like, the Net Promoter, you know, like ask them, they say, “Sure sure sure,” or, “Yeah yeah, I would definitely recommend this to their friends…oh, to my friends,” and, actually, they probably haven’t recommended it at all.
And I think it’s much more useful to look at, for that sort of thing, it’s much more useful to look at say, for example, contributions to a GitHub repo, or people writing blogs on their own blog site that are talking about how to use your API. That sort of thing will be a better metric, I think, for that sort of thing. Or by giving away the discount codes. So if someone signs up via a discount code, and they get a certain number of free calls, or months or whatever then, again, that sort of…that’s better proof, I think, of Net Promoter Score.
And then revenue generated from integrated apps. I’ll come back to this point in a bit, but so mostly the SAS spenders are starting to do this, so insured who do FreshBooks measure the number of apps that are built that integrate with their FreshBooks, and then they measure the revenue that those apps are bringing to them. Overall, so far, John Musser and I have identified 63 different metrics for measuring developer evangelism success, and at the moment we’re going through and working out, you know, what’s the pros and cons of using each one? You know, what can they really tell us, and where do they fall over and all of that sort of stuff.
So we’ll have that released shortly, so through my contact details that I gave earlier, you can find out about that, too, if you’re interested, and I’ll pass on to the good people at DevRel.net. Okay, what’s more difficult to measure, one of the big ones is how do you measure your involvement in conference activities? Like if you’ve got a booth that you’ve paid money for at a conference, or if you’re running a workshop at a conference, how do you track the developers who come along to there, and then end up going on to be your regular developers?
When I’ve asked developer evangelists about this in the past, and they normally tend to say, they look at Google Analytics, like, the week after the event, and then just sort of try to see if there’s been a spike in people coming to their website. But I think there’s…or maybe there’s other ways that we can start measuring that customer journey or that developer journey as far as how they come on board, without having to do that horrible experience of when you get to the door there’s someone standing there trying to do the QR code for you.
So, yeah, so I think that will be…that’s one that I constantly come across as far as API providers are interested in measuring. The…I think which programming language resources are converting more developers is an interesting question, and then…because that gives you some look into your developer community and what languages they’re using. But then there’s…it’s also great to do this whole two year, five year sort of look, so you’d be looking at, “Okay, what languages are people using now? What’s the fastest growing languages that people might be using in our field in a couple of years? And are we ready? Are we starting to build resources for those languages as well?”
Time to first API call is really difficult to measure because you’re not, actually peer programming with developers to be able to sit next to them and sort of see how often…see how quickly they onboard. But I think there’s a number of metrics that you can use to sort of get you to that point of time to first API call. So using cookies, I guess, on your website, so people who come to your website. And then looking at the difference between the number of people who come to your developer portal and those who signup, who register. And then the time length between the people who register and them getting an API key. And then, you know, then looking at that API key, and when it was sort of used in some sort of authentification process or some sort of actual making of an API call. So but, you know, like that…I think we need some workaround being able to do some of that. I’ve also heard…I’ve got a model for being able to sort of segment your developers as they’re coming on board, which I’ll show in a minute.
But I’ve also heard that some API providers, while that’s all well and good for certainly…for follow-up with those developers in different…oh, you know, once they’re first signed up for a developer portal…for, you know, on the developer portal that for some API providers it all comes down to time to first API call. Like, the developers who make that first API call, that’s the time to increase engagement because they’re the ones who are, actually, have sorted out that they do need your API and are willing to use it.
And then there’s revenue impact of developer evangelism, which is surprising that it’s so difficult to measure, but I’m hearing, constantly, it’s still one that’s…that people are having a struggle with. So all up it’s a sort of a bit like a multiplayer online gaming. Look it up.
Mark: So in that…within an API provider…within your company, or within your business, there’s going to be a whole range of people who all need different types of data in order to play. So they’re…all of them have different needs as far as what they’re, actually, wanting from proof around developer evangelism and the impact that it’s having.
So one model that helps us get to be able to share all of that different sort of data is the YAAARRRP model, which is really sort of like…it’s looking at the developer funnel. So it’s the sort of standard in marketing where you have…by the way, so, originally, Phil Leggetter had called it “The AAARRRP model” and then he was asking, “Could someone try to figure out a way to put a Y in front of it because then there’s a Simon Pegg “Hot Fuzz” reference to YAAARRRP, so I’ve tried to help out where I could.
So then the model really goes through…First of all, I think, the starting point needs to be your goals or the “Y?”. You know, so, like, looking broadly…more broadly rather than just at your developer evangelism efforts. What’s the goal of the business? Why have you got APIs? Like sometimes it’s pretty straightforward because it’s that your API is your business product, but other times it’s, you know because you’re opening up APIs like the people who were saying they’re from traditional business. Maybe it’s in order to make partners relationships stronger and to outsource the onboarding effort so the partners have to do all the integration costs.
Or it might be that you’re trying to…you’re a SAS spender and you’re trying to reach new markets by releasing APIs and having other people create apps, or because of the customer stickiness that will come because customers have a broader range of apps that they can use or via your APIs. So I think that is a starting point for…So here I’ve got it…does this…Oh, yeah. Here I’ve got it. So this is sort of the context if you like. So I think it’s important to always start with that. Then we’ve got the sort of the traditional developer, or the traditional sort of marketing funnel if you like.
So, you know, like awareness of, like, where did people find out about the platform, about the developer portal and what it does? How many are signing up? How many are then active? Who’s now actually regularly using your APIs? Are they actually turning into commercial developers where they’re actually building commercial products and selling those commercial products or those integrations with your APIs? Are they telling other people or sharing their successes around using your API? And then I like the one that Phil added here, too, which is product.
So really it’s about, like…then there are those developers who are contributing back to the product. So, for example, by writing new API wrappers, by contributing to any GitHub repos that you might have, by sort of providing feedback on your product, and sort of suggesting how to improve it, all of those sorts of measures. So, ideally, you’d actually have, I guess, an arrow that would go from the product back up to the top because it’s sort of, like, bringing in new awareness even. So there are the sort of buckets that you could have a metric around for each of those.
So then coming back to my health background. In Australia, we had this thing called “HARP.” So in HARP, it was Hospital Admission Risk Prevention. So, basically, there’s a whole heap of data sets that you can have that would start to flag and identify that people were going to be readmitted to hospital, but then the trick is that if you can…you really wanna distill that down to, like, three core metrics. And it was, basically, have they been already been admitted once in the past six months? Did they have someone who they could go home to support with?
And was it cardiovascular or pulmonary disease. So if it was one…if they had those three things, tick, tick, tick, then they would get this additional support in the home so that they don’t get returned into hospital if you like. And I think that’s the thing with the YAAARRRP because, you know, like, having a bucket or having metrics around each of those is just overkilling it when you’re coming from a place where you don’t…where you’re just starting on the metrics journey. So, I think, take the HARP approach and just look at one, two, three, or four.
So I would suggest, as a minimum data evaluation framework, first of all, think about the stories that you wanna tell with your data. Like, what’s the frustrating proof that you need to give as to your existence? Like, when…you know, like if you’re trying to argue for a greater increase of budget, or to attend a conference, to prioritise a particular set of resources, what’s the sticking point there as far as what data will help you make that case more strongly? You know, so then maybe even it’s the first step is just keeping a track of some of those, “If only they understood me,” sort of moments, you know?
Then the second thing I would suggest is choosing four areas depending on the maturity of your API program. And so sometimes you’ll be measuring outputs instead of outcomes. Ideally, you’re measuring outcomes. Like, what’s the actual impact that it’s having? But for some cases, it might be too early along the journey to be able to measure those outcomes, so you’d then just be measuring outputs where you’ve got an assumption that if you create these things then it’s gonna lead to these outcomes. I’ll give you an example of that in a minute.
And then, also, I’ve said choose four areas, but make sure that the wider business goals is one of the four. So choose three areas I guess. Okay, so for example, the “Y?” for this one is so their business goal is entering new markets, or extending customer reach through third-party providers. So then you’re wanting to come up with a metric that’s going to measure that. So, again, I can share this with you if you like, but I’ve come up with a process where you can start segmenting your developer audience once they come onboard, so that, basically, you’d sign up to your developer portal say through MailChimp. Then you’d use Zapier to be able to pull those MailChimp emails into a Google spreadsheet. Then you use the Clearbit API to be able to then populate all of the data. So you’ve got their email, and then you put in there Workroll, their Twitter feeds, their LinkedIn, or whatever. So you’ve got that detail. Then you can actually segment by industry that they’re working in, or you could segment by…what I was suggesting is you segment them by their organisational role. So if they’re, like, clearly in a position that sort of smells like decision maker, then maybe you’ve put them off to…you give those names to a particular developer-evangelist for more one-to-one followup.
And if it’s…but if it’s more hobbyist user, then maybe you keep them going with the self-serve sort of stuff. So that’s so, you know, so then you might be able to get further along with your developer evangelism by more…by quickly identifying who’s your commercial opportunities. But at the same time, it’s giving you some data on the industry backgrounds. So if your goal is to show you’re entering new markets, then you could, actually, slice that sort of data by the different industry groups that those developers represent.
And so here’s the…that output rather than an outcome that say…because you’re sort of showing that, “Okay, of all the developers that are signing up, X are coming from this new market that we, you know, either geographic or industry that we hadn’t previous…that we had wanted to reach.” So you sort of can start to show that there’s some traction there, and then maybe that pushes for being able to ask for the sort of resources or prioritisation to do more work for those industries. Developing resources for those sort of industry developers.
So then an R, so retaining developers and keeping them activated, so for here, one metric that might work would be support calls that do not escalate, so you’re wanting to really check that your documentation has this feedback mechanism. So starting…so here, you know, like are…is our documentation team…are we giving the sort of documentation that is allowing people to solve their own problems, you know? So the good thing about this sort of…having this as one of your metrics is that you also get to start seeing, “Well, why don’t you get to play more of a relationship across your organisation?”
So you start talking to sales engineers and support agents as well and thinking things. And then thinking about what you have to do as far as what they need in their job for the API to continue retaining developers and growing. So you can sort of…so it sort of brings you into greater internal team discussions, which is awesome for the people who I’ve heard that have done that. But then also it means that you’re there at the front end of hearing what are the use cases of your APIs. So you start to hear how your customers are actually integrating the APIs, what they’re building with them, what’s interesting for them, and what they wanna try to do in what order. That sort of stuff.
So even though it’s the sort of metric that’s just, you know, about support calls that do not escalate, you can also…and I think Matthew Barnier had a talk about this earlier today about that role between product management and developer relations. So…and it’s really, you know, this is a good example of that. So you’ve got your support calls, which is then helping show that, you know, your developer portal and all of the rest is actually making it…is reducing costs of managing support because developers can help themselves, but then also out of it, you’re building stronger relationships across the business, so that you can identify what some of those new use cases that customers have, and what sort of resources that they might need to build.
So in some cases I’ve heard things like support agents say that they constantly are answering the same questions, so then the developer engagement team has been able to build new resources that particularly speak to…you know, so that, in future, the support agent can say, “Oh, read this,” you know, instead of, like, answer, oh, you’re writing that long email each time, and then the revenue. So one of the…one of the ones I like to look at here is revenue is generated from third-party apps. So, for example, Atlassian, they have a really simple share…of revenue share model with any developers that use their APIs to build, like, extensions for Atlassian products.
And that’s, basically, 25% revenue share model. So if you build any sort of HipChat extension, or Confluence extension, or Jira or whatever, and then you make it available in the Atlassian marketplace and you charge for it, the in customer gets access to that, and Atlassian…and you get paid for it, and then Atlassian gets 25% of whatever your cover price was. So over five years, Atlassian has made $250 million in sales, which for them means sixty-two and a half million. So then…so, you know, like just been able to show that revenue benefit back to the organisation through the APIs.
None of that would have been available if the APIs weren’t available, and, in fact, you could say more than that because those customers who are using Atlassian, they would then…they might be looking at other solutions that do have those complementary products. So you could, actually, lose some of your existing customer base as well, so. And then the for the product developers or supporting developers. So here are…what I really like is this SendGrid…SendGrid did a Hacktoberfest, where they just promoted to encourage people to actually contribute to their GitHub repos over the month.
And I don’t think you can see it very clearly here, but it says, “Three hundred and forty-seven contributors to their docs.” So during that month, a big swag of those contributors were people who were like helping them refine their documentation and improve it via GitHub. So, you know, one of the…and I think, again, like I was saying earlier, I think this is one of those better measures than Net Promoter Score. Gives you a bit more real-world number of developers writing API wrappers or tutorials on their own sites and number of developers contributing to GitHub repos.
Okay, that was…that’s to really just try to get you started so that, you know, hopefully, that you could think up three or four metrics to start measuring the impact of your developer engagement effort, and then you’ve got those going forward to be able to talk about why your API developer evangelism efforts are so great, and why you should need more of it. Okay, thanks.