Measuring progress in developer relations

Matt Makai: Thank you so much. Hey folks, my name is Matt Makai. I want to talk about measuring developer relations and experience progress, the metrics, and also I want to show you a tool that might be handy for you all. And so it's going to be a little bit of a mix of how I think about measuring since I've jumped around so much to different companies and I have conversations at different levels from CEO board all the way down to individual contributors and how I just have to get up to speed on the metrics that I need to talk about in that moment. So I think this is one of the big things that I have seen in DevRel particularly, is we talk about metrics. There has been a lot of focus on measuring things, particularly as operational efficiency has become more important over the past few years, and these metrics get lumped together.

So I want to give you my mental framework. This is one of many, but for me, I think about things from input metrics, the things that are truly under your control. I'll give you some examples in a moment here. The output metrics, the things that are indirectly in your control, and then the financial metrics and the business goals. And there's different categories of what people are thinking about at any given moment. And you want to make sure that what you're talking about, the metrics that you're talking about actually match the mental mindset of the person, the position that you're talking to. There's a lot of tactics. This is a very busy slide in part because there's so many different tactics.

What you're looking at here are the different developer relations tactics between online engagement, offline engagement, and then split between how I define developer relations and developer experience.

And you're going to be asked to do a lot as you start to take on leadership roles in developer relations. So we have a mixed audience here, a lot of individual contributors, managers, executives, and a lot of times you're going to be asked about something that you don't run and you really just need to have a mental mindset for what metrics matter for which executions, and not just the metrics that matter as far as measuring input metrics, but also the things that those are going to affect when you're actually executing on 'em. So when I talk about the difference between input metrics, output metrics, financial metrics and business objectives, this is often what gets measured as far as like, oh, we published X number of blog posts. We did 10 this month. We want to do 20 and six months from now as we add one or two people to the team.

And this is great if you're an individual contributor, it's often good to roll up these metrics on a team basis, and these are directly under your control. So the way that I look at input metrics is these are the things that you actually like the number of blog posts that you published, and it's up to you to make sure that the quality remains the same. But this is something you can directly control that number based on the amount of work that you do with output metrics. It's an indirect control. So it's something like the web traffic that your blog posts get. So you may go for that hacker news hit, you think you've got a really great blog post, but it doesn't pop. So you've taken your best shot to publish that blog post, but the results out the other side are not completely in your control.

You do the best you can with your intuition. You may have some percentage success rate, but you're never really sure what the output metric is going to be. On the other side, this is a lot of when we talk about, okay, what are we measuring with DevRel? A lot of times it's just the input metrics and the output metrics. The thing is though, that that is a very different world from what you're looking at here, which is Datadog's, investor relations statement of cash flows, balance sheet, the financial metrics that ultimately run a business. Now, if you're working for a publicly traded company like I was at Twilio, I was there for about two, three years before we IPO'ed. I love this stuff because I very fortunately undergrad and my first master's in computer science, but then I went to business school and I went to business school, not because I wanted to do business, but because I wanted to actually be a better software developer.

I wanted to be able to understand this language that you see in front of you, these numbers, what does this actually mean? And so I actually think that if you look at this as learning another dev tool, this is what for many of you could complete your journey as far as where you want to go in your career trajectory. I find that there is very much a hard cap on folks that do not ever learn about what these business metrics and the business objectives are. They just simply look at it and they're like, well, I can't read a financial spreadsheet, I can't read a balance sheet. And we don't really talk about this stuff in de a lot because, and what I've noticed is as I've listened to a lot of the other talks and they're very good, they're getting to a lot of important pieces, but I feel like almost sometimes we're trying to reinvent business.

This has been around for a hundred years. This is a study discipline and we may crap on it sometimes because like, oh, those business people, they don't get the technology. But if you complete your journey by knowing the technology, being a software developer, which I very much believe, if you want to do DevRel at the best highest level, you really need to have technical credibility. If you really want to be able to then impact the business to the full extent, you need to know a lot of this stuff. So I put this stuff up here because this is some of the things that have helped me the most. Now I'll say my mental mindset, when I learned this stuff, I was a heads down developer for many years and I was programming in Java and Python and c plus plus, and I was like, if I learned Ruby beyond Rails, is that going to be more valuable to me than learning how to read an income statement?

Probably not. And so it was just a mental mindset for myself where I said, the next dev tool that I learn is going to allow me to speak to everyone in business. So I will not only have the computer science background, the programming, the technical know-how I will also be able to break down a financial spreadsheet. Now, that may not be the career journey that all of you want to go on, but I'm saying that this really is what allows you to get past a ceiling as someone in a company to make a true impact because the things that you do on a daily basis, the number of blog posts that you publish per week and the web traffic you get out the other side ultimately is towards a business objective. And I think that there was a blog post recently that I skimmed, which was like swix, like DevRel was Azer phenomenon or whatever.

And I think that kind of, well, I disagree with a lot of what was in there because I think it was kind of a little bit hand wavy in some of the ports. I think part of the point was basically we forgot that at the end of the day, there's financial metrics that are getting impacted by the work that we're doing. So lemme just go over this real quick. I think there's actually, if you take nothing else away from this, there's three resources that really help me learn. The first is this book, financial Intelligence for IT Professionals. It is written for people who are practising in, I mean IT professionals is kind of like a super formal term, but it's written for people who are practising technologists, software developers to learn how to read financial statements. And again, mental mindset of this is a new developer tool for me to use in my career.

There's also, I actually really like this book, the 10 day MBA. I think it's good as a refresher, if you've studied any of the business concepts before, it's going to go over a lot of different stuff. I don't think you necessarily have to learn every part of business. I think it's helpful, but this will go over everything that you would actually learn in an MBA programme. And then the thing that you could sign up for right now, I'd love this newsletter only CFO, this Guy, guy or Galax, you don't know who the author is, is incredible. Clearly an incredible level of depth in what they understand about the software industry and in particular the financial performance of companies and particularly why software companies have had such difficulties with their stock prices recently. So this is a free newsletter I would definitely sign up for. It'll teach you how to read balance sheets.

Again, the mindset as this is my next developer tool. Alright, so let's connect this, right? Why is this important? And then let's get into this tool that I built and why that's connected to this. So for me, having conversations with our CEO or chief product officer, I'm speaking about the financial metrics and the business objectives, and I'll give you a few examples as we go through a little bit later. But for me, that's really where I'm living a lot of my time. But I have to connect that back with a narrative. And so the input metrics that I've talked about, like the blog posts and we look at the blog posts published are going to create signups for developer signups.

And that is going to allow us to get that next fundraising round. And this is actually situations that I've been in in the past, and you have to connect that narrative.

That is what allows us to truly be meaningful to the business. We cannot just hand wave about the metrics. Okay, this all probably makes sense if you learn some of the financial metrics, but what does this have to, what's the connection here to a tool or something like that? So for me, when I was at Twilio, a lot of the stuff we were doing was fairly new. We were spinning up tutorials programmes and we weren't really sure what was working. I spent a lot of time just getting the data and I had a spreadsheet of 5,000 blog posts that we had published on the Twilio blog. I hand wrote that entire spreadsheet. It was maybe a little bit of a waste of time, but the output of that was really valuable because I could say, yeah, we published a dozen PHP blog posts.

This is actually a real example.

We published a dozen PHP blog posts and then some of the earlier developer evangelists left and we didn't publish any PHP blog posts. So Nexmo Vonage I think fills in the audience here. They were going to town on PHP blog posts. It was a total gap for us. Didn't have anybody doing that stuff. So I was like, Hey, we got to hire somebody to do PHP content because here's an area in which we used to be successful. We have almost no signups. So you're starting to connect the dots, Hey, we didn't do the tutorials, we're not getting signups in that programming language.

And that's leading us to have a competitive gap against competitors in a specific programming language. So you start to thread the narrative and you can only really get to that if you have those input metrics just really fast at hand. So that is why I built this tool.

Y'all can access it. It's plush cap. com. Plush cap is a bird that has a really interesting kind of design to it. And I got the domain and I was like, one day I'm going to build a really cool tool with this. So that is why it's called Plush Cap. That is the naming history of this tool and it kind of rolls off the tongue. I kind of like, it doesn't have anything to do with DevRel, but it's a good domain name.

Alright, so if you go to Plush Cap, the thing is I have a database that I've curated of about 450 developer focused companies with self-service motions only. That's it. The hypothesis here is that if you have a self-service motion and you're focused on developers, you basically have to do everything, almost everything in public. Your docs have to be out there, you have to do technical content.

There's only so many playbooks that you can run, maybe YouTube videos, things like that. And it's totally different from a talk to sales motion. So if some of you work for talk to sales companies, that's not what I'm trying to do here. I want to see what every single company is doing and just immediately have access to it. So someone in one of their talks today said, oh yeah, we had to do something on Twitter because our CEO EO is on Twitter. Well, it's the same way for anybody who's running a DevRel programme where it's like, I saw this other company doing this thing, why aren't we doing that? And so if I actually have all the data already, I can say, yeah, they're doing that, but they're not doing these other things. Do we want to invest in that versus what we're doing?

And here's the trade-offs of what it would take to do that work, the output we would expect from it, and then actually the financial metrics that would be impacted by this.

That's a completely different conversation than just saying, no, no, we can't do that. We don't have capacity. And so that's really why I built this. So having access to data really fast is just a complete difference in how you're able to operate because at all these different levels of conversations, if you're just looking at how many blog posts somebody published or what type of blog posts they published, it's really helpful just to be able to have all that data at your fingertips. So why don't we go through some examples of how I've used this in the past and hopefully all of you who work for self-service developer focused companies, you'll find your company on here. If not, please let me know and I'd be happy to pull your data. I don't have data for every single company. I scrape everything from as many companies as I can, but it's only about 40 or 50 companies.

So if your company's not on here, I can add a scraper in like 10 to 15 minutes for it. Alright, so if we go to plush cap. com, this is the first page is just a list of recent blog posts by different companies. But really where I think some of the power comes in is, okay, so we have all these different stages of companies. So you may want to look at peers. If you're a seed stage company, what are your peers doing in various categories? How much blog content are they putting out? The way that I actually used this most recently was just showing for LaunchDarkly where I've joined about a month ago, just our blog content output and the size of our docs in comparison to other benchmark companies that are series D round companies.

So just having some comparison points is really helpful in a lot of these conversations.

But lemme give you a really specific example of a timeline that happened for my previous employer Assembly ai. So with assembly ai, we can click in and get number of blog posts. We're going to zoom out to all the blog posts they've ever published. I want to tell you the story of assembly AI in this chart. Assembly AI was founded in 2017. They were figuring out product market fit. So this is AI models. They train AI models on large amounts of data and that creates much higher accuracy.

Speech detect models than previous, if you'd use Dragon Nuance or whatever, there's some previous technology. So a step function change in the accuracy and the quality of the outputs of speech to text. They were figuring out a lot of the product market fit and then they were trying to figure out how do we go. Their business objective was like we got to raise a Series A.

Well, investors are looking for certain metrics and so this actually cleanly ties together blog content, the signups that blog content, particularly the developer tutorials will output with the actual raising of a series A round because you can show customer acquisition, look at all these customers that are signing up to use our API in series A. A lot of times you're raising on this future potential. And so I think that's one thing about this is if you work at a seed stage company, the input metrics and the output metrics may be the same that you're looking for, but you have to understand what the business objectives are that you're going for. So what happened with Assembly was they realised we got to hire some initial DevRel folks. So this is a story of developer relations driving business objectives. If you ever want different stories of driving business objectives, driving financial metrics, I actually have quite a few of them just because I talk to people behind the scenes.

If I see something interesting in the data that I've scraping of company, I just reach out. I'm like, Hey, what was the story behind this? What happened here? If I see an interesting angle to something. So around late 2021 Assembly hired a few de folks and made them really consistently publish high quality developer content. And so you can see this large 16, 17 blog posts published per month for several months and their signups skyrocketed. They were getting hacker news hits, getting SEO, it was working really well. And so that was so clearly in support of we have to get customer acquisition, we raise this series A.

And so actually they raised series A and then a fast follow same investor said, Hey, well when we do the series BA few months later, then they kind of trailed off. They were doing different marketing activities and things like that.

And that's actually where when I joined, I'd been an advisor for them for a long time When I joined, they had really gotten off being consistent. And so actually what I've used this chart for with many different companies is actually just showing them like, Hey, you say you want to drive signups or you say you want to get customers into using your product API usage up, but you're only publishing one blog post a month. The next month you're publishing 10 and then you don't publish anything for a while. You have to be consistent about your output. Just showing a chart like this to many people really helps them to understand like, oh, I never visualised, I never thought about it in that way. And so when I came to assembly, I was like, we got to get back to consistency. And that's really what allowed a lot of the more recent results in the raising of the series C round.

Lemme show you another example here. I'm going to go over to one of my favourite companies. We've got a few people from CloudFlare here and in CloudFlare, this is a little bit of a different story, but you can see the way that they launch products. So some companies are like, Hey, we do a winner release spring release. I'm going to go into all their blog content output and then we'll go back to 2009. Now they have a very different model here. You're going to see some months are crazy, like 65 blog published. They do launch weeks.

So this is how they want to operate their business. And so you can see they've operated on a different cadence. And so just to be able to visualise this, they're actually deliberately not going for a consistent month in month out blog post output to try to drive signups or something like that.

They're using this to drive a steady drumbeat of product announcements off of specific weeks, an AI week, a developer focus week, some other networking week, any sort of different theme to that week. And that's why you see a different spike here, but that's actually intentional. And so that's part of why I built this tool is to be able to visualise different go-to-market motions for developer focused companies and actually see, well, is this intentional or not? And I'll show you another piece to this and I'll just go back to assembly. But if we click on this is all YouTube data. So I scrape all the YouTube data for all 450 companies, not scrape, use the YouTube API and I collect some of the YouTube data. So what you're looking at here is company name, the YouTube channel that they have, the number of subscribers that they have, the number of videos, the number of views all time, and then the different stages.

So you can compare like, hey, we're a publicly traded company, what does that look like? You can just sort by or search by publicly traded company, but you'll see different things that are in here. So you can just very clearly tell this is basically my leaderboard for developer focused companies. And the fun thing here is if you're starting a YouTube channel, I don't know, let's just kind of scroll down the list. Let's take a look at, I dunno, there's somebody, I saw somebody with a Fastly T-shirt. So let's take a look at Fastly, right? So Fastly is in here. They got 2000 subscribers, they got 178 videos that are posted and they're publicly traded company.

So this is kind of putting them very much middle of the pack if they really want to execute well on YouTube. YouTube is a particularly interesting example because you either, it's a bimodal distribution of outcomes, so you either get tonnes of results or you get zero results.

And so I would expect that regardless of whatever input that they put here, they're getting almost no output out of the results that they want out the other side. Now let's say they invested in it, well, they could climb up the leaderboards. So then the nice part is you can tie this back to some of the business objectives. We really want to grow our brand awareness, things like that. And so you can start talking about, well, we passed these companies in subscribers, so we passed these companies in views. And I actually used this very successfully. We had very talented team at a very talented team at Assembly ai. And I can tell you going into quarterly business reviews saying, yeah, we passed Twilio, we passed Palantir, we passed MongoDB and subscribers, and we could see the subscribers out the other side, we could see the business results.

And that was just so much momentum that was behind it. And so that's really what you want is you want to just very quickly be able to access the input metrics and then be able to tell the narrative out the other side. So that's really what connects the different pieces together between the input metrics, output metrics, financial metrics and business objectives. So the other thing is you can also see over time, OpenAI obviously huge number of subscribers, but they jumped up recently. So every time that they do one of those live streams, they went from 800,000 subscribers up to over a million. So just from that, when they had the her thing and the whole Scarlet Johansson controversy about using her voice by live streaming, that they got a tonne of subscribers on YouTube from that. So different styles, but you want to be able to then tell the narrative around the metrics that you have.

So this is why I built this tool. If your company is not on here and it's a self-service developer focused company, let me know. If your data is not on here, I'm happy to just go run a scraper so you can access that data. What's funny is I have other teams that are using this tool, I just have it up online and then that actually helps them to run their Devra programmes. And then all I ask in turn is just like, tell me how you're operating. What are you doing and is it working for you and what is there an interesting motion? Then I can tell more stories about the ways that we should be investing and the wages we should be executing. And I can tell those at different levels.

I can talk to the CEO about it in business terms, and then I can bring it down to the individual contributors that are executing and really wrap a whole narrative around it.

So that is roughly what this tool plus gap is, and that's how my framing as a VP for just aligning the narrative and the metrics, and hopefully this has been a helpful just framing for you all of, if you're thinking about you want to just continue your career in DevRel and be either an executive or you want to go into some sort of related function like in marketing or in product, these are just the types of things that you're going to want to have on hand. Now, maybe they're going to be a little bit different depending on the function that you're operating, but I found that it is just transformative to have the data at my hands and then I can work with our internal teams to really combine that with the financial metrics and tell a complete story. Because I found that the best executives that I've worked with, they're constantly hammering you on getting more data. And if you already bring that data to the table, they're like, oh, okay, you know what you're doing? And they stop questioning you. They're like, okay, I trust you. Just go and do that. So that's a little bit about what I had for you all today.

My name is Matt Makai. Thank you all very much. I'll take some questions.

MC: Thank you so much. We're now going to go around with questions for Matt.

Audience member 1:

Firstly, great talk and out of curiosity, how do you run four scrapers without getting rate limited or blog and how often do you run them?

Matt Makai: The thing is, once you have the data, you actually don't really need to run it that frequently, so I just run it periodically. So that's basically how I get around it. Now, I also scrape, I scrape the blog posts and I'm looking at scraping all the docs, but I don't scrape absolutely every piece of every website. In fact, you find some really weird edge cases that actually are a little scary. This company called Lambda Test, they auto generate like 400,000 pages and just storing that just because I store the raw HTML in case I need it later. And actually I guess the part that I didn't really talk about is I run LLMs on this stuff to summarise the data, give me some insights, and that amount of data from just companies that auto generate pages is actually pretty hard to keep up with but's roughly it. I am, I don't go for, I want to capture all the data. I go for targeted collection and then I just store that and then I accumulate that over time.

And that's basically how I can do it in sort of a reasonable way that is, I think not going over the line or something like that. Got it. Thank you.

Audience member 2:

Great talk. I have had the 10 day MBA on my desk for probably 12 years, so I don't have an MBA, but every time one of the financial emails comes in and I don't understand a word, I go right to my 10 day MBA. So you said interesting motion versus I have a two part question. One, what's an interesting motion versus an uninteresting motion, right? You mentioned that and I'm curious as to what it means. And then the second thing is a lot of this DevRel and I have been in DevRel for about five minutes, so excuse the ignorant nature of the question, but a lot of it is not directly tied to revenue, right? It's indirectly tied to revenue. Do you have a model to measure the indirect tie to revenue to actual dollars?

Matt Makai: Yeah. Okay, great questions. Also, the first thing is I'll go with the zeroth questions. So the 10 day MBA, you don't necessarily have to go and read that. I actually think that if you make friends with somebody in finance, they'll teach you a lot of this stuff. And some of my best friend coworkers are often people in finance that can explain stuff. And I also will do this for my teams. I was talking to some folks last night about how I used to do Twilio.

I would just, every time I do a quarterly earnings report, I'd be like, Hey, I'm just going to do an optional one hour session. We can break down the financial metrics. I'll talk about how our work impacts these financial metrics. Now I've forgotten the other two questions. I don't, no, let's do the indirect to revenue. I don't know.

I mean, look, I think here's a really important point. Those of us who are technical work in a deterministic world, except maybe now a little bit with the LLMs people in the business world work in a non-deterministic world in which they're making a lot of assumptions all the time. So think about it from, if you're a developer and you're changing a variable, you're going to look for what the change in output is. But in business, it doesn't really work like that, and so you just have to become comfortable. I think this is actually one of the biggest challenges that is not, I don't know, I haven't really heard a call it out, but I haven't heard every talk. If you're a developer, going from a deterministic world to a non-deterministic world is actually a little bit scary. And so I think that's the thing is you actually, it's your job, especially as you become more of a leader in DevRel, to actually tie that direct revenue.

Now, that is also how I position things. A lot of this comes down to maybe a bigger point of positioning, which is I drive growth at the company. I drive revenue growth, full stop. I'm not doing sports stuff. We drive revenue growth and we do that through customer acquisition. We're going to double customer acquisition, we're going to double it again. And that's why I care about self-service developer companies. I don't do non self-service.

I don't do talk to sales stuff, right? There's weird incentives. I don't do ecosystem companies. I know for a lot of you in here, you probably do, and I don't mean that to be like, I don't know, it's scary or something like that. But you just have to be comfortable with the decisions you're making of what choice you want to work for. I work for self-service companies so I can drive the revenue.

And I am very careful about telling a story of these are the things that we're doing. And I don't say I'm very good at not saying no, but I'll say, if we want to do that, we require this investment in order to get there. And then you're not stretching your team to into a zillion things. Focus is almost always going to triumph over just peanut buttering over a bunch of different stuff. And so I actually look at it as directly tying to revenue. I directly impact the company's top and bottom lines. And that's the story that that's the story you have to tell. That's the narrative that you have to create.

And you can do that through the metrics is a big part of my point. I think there was another question in there. Interesting versus uninteresting. Oh yeah, interesting motion. I'll say that. The interesting part is just, I'll just discover weird stuff and I actually post occasionally about it on LinkedIn or YouTube or whatever, and I'll just be like, I found a company, created a comic book about their product, and I was like, I don't know.

This is fascinating. I don't think I would necessarily do that, but tell me the story behind it. So stuff pops up when you start scraping these sites. The other piece is a lot of times I'll see interesting stuff as far as integrations. So I'm like, okay, they did a bunch of integrations, they created a bunch of pages for it. Is this working? And then I just go talk to people just like, Hey, just reach out to 'em. Like, Hey, does this work for you?

And a lot of times they'll be like, oh yeah, it really works. And then I'm like, all right, I'm definitely stealing that or it doesn't work. I'm not going to do that. I'll do a different tactic. So it's all about driving some of those conversations.

MC: Alright, well thank you again so much. Matt made another round of applause for Matt. I.