Made by
Made by
Adam FitzGerald
DevRel is awash with data – from daily users to twitter followers, conversion events to customer value – but which measures are the most important? In this talk from DevXCon San Francisco 2017, Amazon Web Service’s Adam FitzGerald looks at which metrics matter, how to compute them and why we should heed the lessons from The Hitchhiker’s Guide to the Galaxy.
Hi. Thank you. So I’m not sure metrics is, I am an expert at or in any kind of sense, but I used to do a lot of mathematics a long, long time ago so I’m not afraid of them. I think it makes…kind of qualifies me in some regard. So this is really subjectivity applied to objectivity, metrics objective things. We can measure, we know what a number is, we know what a value is, and I’m here to tell you which ones matter and which ones don’t. And I’m not sure I’ve got a good answer for you, I apologise in advance.
So, I’ve been doing a little bit of the DevRel stuff for a while. I started out at BA systems, and I went to a SpringSource source, then VMware, then Pivotal, and I spent the last three and a half years at Amazon Web Services. We got anybody that uses Amazon Web Services in the room? I want to see your hands. Okay. Yeah.
We got a server or two, got a couple of spare ones if you need them. We’re happy to help you use the Cloud effectively. It’s been some remarkable growth at Amazon over the last, well, I’ve been there three and a half years but over the last 10 years or so. So a lot of things that we focus on and trying to improve things. But the thing that Amazon focused on most is our customers. So we’ve got some. Sounds like there’s a few in the room, we’ve got a couple of these guys, customers, some of these guys are customers too, actually, all of them. And then these guys are customers as well. And now, I’m done with the official corporate pitch. Thank you very much.
So, when I thought about what is the question we’re trying to answer here, what metrics matter the most for developer relations? I was reminded of the management guru, Peter Drucker. I wasn’t reminded, I might have googled it. Peter Drucker said, “If you can’t measure it, you can’t improve it.” Well, I think he said that because when I googled it, I also found out that Lord Kelvin, the creator of modern thermodynamics in the 1890s in Scotland, also said, “If you can’t measure it, you can’t improve it.” I think is around, before Peter Drucker so it doesn’t really matter. The point is we want to measure things.
So we’re going to try and measure things, and there’s somebody else who keeps telling you you want to measure things, and as this guy, most of the time, he’s telling you he wants to measure things because he doesn’t believe you’re doing your job. Or, as Alex pointed out, he doesn’t believe that what you’re doing is the thing that’s the most impactful, and that’s actually the bigger lesson to take away from here is, if you’re spending your time doing things and you’re not doing the things that are most impactful, then you’re wasting a massive amount of opportunity, you’re wasting a mass amount of energy. And so, we want to make sure that we do the things that make the biggest difference. And this is an important thing to takeaway here is, there’s things we can do all the time to make developers happy, but what are the things that make the biggest difference?
So I went to Twitter, of course, I had no idea. I mean, I googled the first part of my talk, I had to go to Twitter for the rest of it, and I said, “Can you tell me what you think the most critical developer relations metric is?”
Of course, no, nobody knew that I was feeding this poll information into this talk. And here we had 27% said, “Oh, it’s all about the users,” and then 7% it was all about the love from the community, 20% said it was all about the volume, and 46% said it was all about the money. So I’ll walk through each one of these and we’ll talk about each of these different types and try to give you some sort of context for it.
So let’s start with daily active users, or weekly active users, or monthly active users, or whatever you want to call it. And I do want to thank Phillip Freier from Twitter. He pointed out that daily active users is an acronym in German that actually stands for Dümmster anzunehmender User which is, apparently translates into stupid possible user. So that’s pretty handy to remind ourselves, but the truth is that lots of people care about users, they are actually really important part of understanding whether your platform is really being used. It’s also really important for other people. I mean, we even this is AWS, we talk in our public slides here about the number of active users on the platform. You have gotta read some really fine print at the bottom to work out what an active user is, okay?
This is a user that doesn’t work at Amazon that has had an active action on AWS in the last month. So that’s a fine print. This is basically a monthly active user from our perspective. Daily Active Users was a episode, it’s the first episode in Silicon Valley, in season four, last month, okay? And Richard Hendricks kidnaps a…faked himself into an Uber kidnap to VC, and put him in the back of Uber and said, “Piper Chat.” Yeah, its got, what it say, 120,000 daily active users growing at 18% a week organically. And the VC wasn’t very happy.
You guys watched Silicon Valley, right? Okay. The VC wasn’t very happy, he got out of the car and then he came back and said, “When you hit a million daily active users then everybody will try to kidnap you.” So we know the daily active users and user count is important, but there are some things we get to think about, like how to actually count what a daily active user is. Well, there’s a couple of places I’ve got some math here, so I don’t want to scare you too much. But an active user is the cardinality of the set of IDs uniquely across all events that have happened in a time period that you’re considering, whether it’s 24 hours, or a month, or a week, or whatever it is, okay?
It’s actually not that hard to compute if you know what all your events are and you’re logging, right? You guys log, right? You log user identity and your log trials, and you trial your logs, and you keep track of that information, and you stuff it in your data warehouse. This is just a simple query. But if you got a lot of activity and you got a lot of users, this could run to the billions of events in a small period of time, and you are extracting unique identifiers out of those billion events that can get expensive pretty quickly. So there is a great blog on Helpshift from a couple years ago, where we talk about different models for computing this. And so, if you can compute it rawly, that’s okay. If you do it just very, not very smart, you can go up to about a billion users in less than like two gig of RAM, you do it all in memory.
If you want to do it with higher values than that, you can actually do it with a different counting techniques that actually have an error rate to them, it depends upon the size of the way you do it, and you can do them in a linear style, you can do in a log-log style, you can do it in a hyper log style, and I encourage you to read this blog. It has actually fantastic information on how to do that counting.
But the truth is that daily active users or weekly active users, they’ve got a couple of problems with them. First of all, they’re a great thing to count, they get lots of people’s attention, but they’re affected very adversely by externalities. They’re a measure of a value, not a measure of an input, or measure of an output, okay?
So, for example, you get featured on TechCrunch, maybe you’ll get more daily active users, but you don’t know what your activity was that caused you to get more daily active users. You could have some kind of disaster that causes bad press and all of a sudden, there are more people looking at your products or looking your services. You got increasing daily active users, you don’t know whether it’s good or bad.
Daily active users, you can’t actually have increasing values of daily active users if you are churning through your users high enough, those people can only stay for 24 hours, then you get new users right afterwards. It’s like a Ponzi scheme for daily active users. So, you could be masking, churning your user base if you’re just focused on the daily active users. And lastly, how do you know what event you’re measuring? Most of the times when people think about daily active users, you’re thinking about either app open or app log in or service log in or API events. You’ve got to make sure that events set that you are collecting your ID from, you got to make sure is the set that actually makes sense for your business. So, you can’t just naively go around measuring daily active users. You’ve got to use some smarts to think about “What does a daily active user mean for me, my app, and my service?” Okay? So, that’s my rant about daily active users.
So, you got to measure them. More is better, usually. But you got to think about them and not just think about them as a metric, okay? So, let’s talk about love. Well, I’m not going to really talk about love or GitHub stars, because Jono is the man, okay? If you hope you watched Jono talk earlier today, I recommend The Art of Community to everybody, it’s a fantastic book. He’s the expert talking about these things. I loved him mentioning accepted pull requests, merge pull request is a great measure of engagement in open source projects. It is a fantastic place to look at it.
I’ve got one data point I can talk about on users versus contributors versus committers. I used to work for a company called SpringSource. We had a little Java application framework that was pretty popular. We may have used it a little bit. So this is actually from my time when I was at SpringSource about six, seven years ago and this is the population information from our community. And when you look here, you can see the top number of user base, numbers of subscribers, that was the number of people that were engaged with us, subscribed to our newsletter, actives, the number of accounts, and user accounts on our community forums. Answers was the number of people that actually answered questions on the forum, look at that ratio. 400 people answered all the questions of 12,000 people on the forum. And there’s only 20 committers in the whole team that committed to that open source project. And in fact, the most important ratio we considered when we were part of that company was the ratio between the committers to the experts, the call to experts. And every committer, every project lead on Spring, was encouraged to make friends and build relationships in the community, with the people that answered questions in their forum. The goal here was to get the community to be self-serving, be self-sufficient, and find those community members that would answer other people’s questions instead of having the engineers answer those questions all the time themselves.
So that for us was a critical metric. The user base was a big metric for the business because that was the top of funnel for conversion and so on. But it was really for us, it is about the committers to answers ratio. Those are the important ones. All right.
So I’m not an expert on love. I’ve been accused of being pretty loud sometimes, so maybe I’m no expert on volume. But I’m not even sure I’m an expert on that, because Tamar Owen and Andy Piper, Andy is still around? You and you guys are going to laugh at me because my Twitter is so weak, my Twitter game is just weak, very weak. But you’ll find no shortage of companies that want to talk to you about, “Oh, I can measure your Twitter. I can measure social media engagement. I can give you your impression. I can give you your reach, I can give you the total number of people you are talking to.” And you’ll get your Google Analytics and you’ll be like, my page views and that was my unique page. How many people are touching it. You’ve got all this information and I’m not sure. I don’t know if that data actually means anything to me. I don’t know how to measure what the value is of an impression on Twitter. Some drive by viewing of whatever I said two months ago. I don’t know what it means. So, what we’ve done at AWS, is we thought a little bit more about what our standard unit of measure is, when we talk about engaging with the community. And our evangelism team, which I’m responsible for, along with a developer marketing and startup marketing team. Our evangelism team, we think about measuring engagement using a unit. A single unit is, one hour of in-person time.
Now, in a forum like this, we got a 100 odd people, one hour of in-person time, that’s 100 units. You get to speak for one hour to 100 people, that’s in person, that’s one unit. That’s something that’s long enough to have substantial interaction, a real conversation around technology, to actually really think about things. But speaking at conferences, and the only thing a lot of evangelists do. Evangelists talk on webinars, they write content, they broadcast on Twitch, it’s amazing that they broadcast on Twitch. They write blog posts, they brief the press, they talk to analysts. They do all kinds of things. So there’s a collection of activities that our evangelists do. And for each type of activity, we ask them to measure the audience reach in which they talk to them, and then we sum them over all of the activities they do and then we apply a weighted measure, so that each one has a value that’s related to this single unit as one unit hour. So webinar time, it’s worth about 43% of one unit. Because webinar, what do you do when you’re on the webinar? Nobody multitasks, right? You’re not on email, you’re not on Twitter, you’re not doing anything else on a webinar, right? So we actually did a pretty nice like regression to get these weights to work out what they should be, and we’ve got this little formula that calculates these things for us, and now we got an idea of what our evangelists are doing. Now, we’ve got an idea of the kind of reach and impact they have. And don’t be afraid of the math. It’s a double summation.
This is a vector multiplied by matrix. It’s easy. Still get a scalar at the end, you get a numeric value at the end is great. It’s great. It’s great. All right. So. I’ll talk to you a little bit about volume and how we think about volume at AWS, but you guys told me that the thing you cared about was money. You said money was the thing that mattered. That was what you said in the poll. So let me talk about money. Money is conversion to paying action. Now, for all of you, you should have an understanding of what that means for your business. There needs to be an event which you are measuring your conversion on. Now, for most people, it’s handing over the credit card details, right?
Sometimes, it might be sign up, if you have to give a credit card when you sign up or sometimes, it might actually be activity in the account that causes some kind of billing activity. If you’ve got some free tier that you have to get over. So kind of depends on your business, which one of these things is actually a conversion event, but doesn’t matter what it is. I don’t know what it is. You know what it is. I don’t know what it is. But what matters is that the event downstream gives you information about the spend of your customer.
Now, last year, I talked about customer lifetime value and how to compute it, and that thought scared a few people a little bit of statistics, again. But it’s pretty easy stuff to compute. but it’s really worth doing. You should understand what the value of a customer is to you, on your platform. But we’re not talking about the things that happen after you start paying. We want to know about what activity did you do that produced that paying customer. So we talked about this by Alex. He said, “You got to do your thing. You got to measure what the most impactful thing is.” How do you measure that thing? How do you know what thing impacted this conversion? Well, there are lots of things. Maybe someone searched for you and found you, and so you’re really proud about your SEO because your SEO game really worked.
Your SEO game got you to the top of the search rankings and wound up at the conversion event. Or maybe you were very smart, you bought some AdWords, and they clicked on the AdWord and that converted. So that’s the thing that brought onto the conversion event. Or maybe you had a webinar and they really loved the webinar, and they came out of the webinar and then said, “I’m going to sign up for your service.” Or maybe they saw you on Twitter, said, “That thing solves my problem. I’m going to go sign up.” Again, conversion event. Maybe they said, “Well, I don’t know. I got your email somehow, and I sent a direct email to you and you loved what I said in that email and you signed up. You reminded me that I should sign up.” Or maybe you saw all the cool stuff we were doing on Twitch and you said, “Okay. Yeah. No, I see what you’re doing on Twitch. I like what you’re doing on Twitch. I’m going to go sign up now.” So there are lots of different paths to this conversion event.
So how do you know what path is the one that caused the person to convert? Now, it’s even worse if you think about what they actually probably did. So they probably googled the problem they had, and they wound up on your website because you did great tech talk. So you wrote a great blog that said, “This is how you solve problem X.”, and then they were like, but they didn’t really convert when that happened. “Yeah, I’m going have to remember that thing, it was really good.” And then, after they remembered that thing they said, “Oh, yeah. There’s a webinar. I’m going to go sign up for that webinar. I’m going to see actually have they do that thing.” And then once you do that thing, they were like, “Very interesting, very interesting. Okay. All right. I signed up for the webinar. “So you got my email? “Oh, yeah. Yeah. Send me an email. “Okay. And when will I get an email from you?”.”Oh yeah, I remember that thing.” And all of a sudden, their body on social media says, “I’m using product X to solve your problem.” And you go, “Yeah, that’s the thing. I’m going to go sign up now. I’m signing up now because the thing I saw on Twitter.”
So which one of these things gets the credit for the thing that drove you to the conversion event? Okay. Now, does it matter? Well, I don’t know. Is it natural search? Is it the first impression that matters the most? Is it the last impression that matters the most? Is it the one you had the most control over that matters the most? The point is they all matter. They all matter to differing amounts. So in order to work out which one matters, you really, really should be doing some kind of backtrack analysis about what these things are doing, and that is called a multi touch attribution model. Don’t be afraid. Be afraid. Don’t be afraid. Don’t be afraid. Don’t be afraid.
So P is the probability you’re going to convert. Beta is the probability you’re going to convert at step I after engaging at J, at initiative J, action J, which is maybe it’s the webinar or maybe it’s the email. F sub-I sub-J is the incidence of following I with J for that activity, okay? All right. You can work all those things out with a little bit of awe. It’s actually pretty easy. And what is this? It’s a double sum of two index curvy. This is matrix multiplication. It’s not scary at all. It’s always very, very easy. You can go on to your friendly data scientists, they’ll do it in their sleep time, okay?
So what does this give you? This actually gives you information about what channel, which one of those activities has the most relative impact on the conversion. It tells you whether email was the thing that was most likely to convert people. It tells you whether the webinar was the most likely thing to convert people.
You can get all this information about what the most valuable channel was to your conversion event by just doing a little bit of mathematics. And it’s a lot more straightforward than it looks in this picture. So I’m wrapping up. Sorry, I’m going to wrap up early. So the question is what metrics matter in developer relations? What are the key things you’ve got to watch out for her and about developer relations? Well, he’s honestly being mindful. It’s all of them.
Every metric matters, okay? But I’m reminded that kind of it’s not, maybe we weren’t asking the right question. In fact, I think there’s been the situation I’ve read about whereas seven and a half million years were spent by deep thought working out the answer to life, the meaning of life, the universe, and everything. And the answer was 42. Anybody else who had read Hitchhiker’s Guide to the Galaxy? Couple of you, hopefully. And what did deep thought says at the end of that entire long experiment, when he said 42, yep, but you probably didn’t really know what question you are asking, and that’s what’s happening.
That’s what happens in developer relations. It’s not what metric matters the most, it is what problem you’re trying to solve, and what metric can you use to work out, whether you’re doing the right thing. So you’ve got to apply some thought. You’ve got to apply some knowledge. You got to apply some logic. Is it particular to your business environment, your use case, your product? Is it about conversion? Is it about getting people into the top of the funnel? Is it about understanding which activity was the thing that caused conversion and what is going on with your daily active users? How do I know that customers are interested in what I’m doing? Is it about my community engagement? I don’t know. You know. You know already.
You know what your product is, you know what your community looks like, you’re just looking for the measure that will help you identify the things you already know. And these tools, the things we talked about here, they are simple tools to help you understand those things. But you know what was right for me, thinking about what an engagement metric is for evangelist for AWS or me looking at the segmentation model for attribution model for our audience input, for customer sign for AWS. That doesn’t help you any because your problem is not my problem. Your problem’s different from my problem and you should be solving your problems and using simple metrics that you can take control of, though indicate what the right result is for you.
I strongly encourage you guys to think about things being metric-driven. It makes it much easier to make those conversations with your leadership, with your CEO, and with your manager about I am doing the right thing for helping the community. Now, you can do all the friendly, outreach, soft touch, intangible stuff as well because that’s what we get super validated on. But if you want to meet justify your operations, justify the things you’re doing, you’ve got to make a stronger case to your business about what is you’re trying to do. So that’s it.
Thank you for indulging me with a few pieces of mathematics. I work for Amazon Web Services. Gina Clemente’s here with me from Amazon as well. We are hiring development marketing lead, development marketing program managers, developer advocates in serverless AI. And I’m thinking of someone trying to come from somewhere else, containers. We’re hiring technical evangelists in Japan, Singapore, Germany, London, US, Canada, Mexico, and Brazil. We got some job openings. I hope you guys come join us. Happy to talk to you about metrics or about working with AWS at any point. Thank you guys for listening.