DevRel metrics with product goals

Derric Gilling: Awesome. And thank you, for being here, and always love to talk about DevRel and and product. And, you know, one of the passions of myself, what I always believe in is how do you think about the right metrics dependent on the team that you're on and the product goals? And this is really dependent on where DevRel sits sometimes within the marketing team, sometimes within the product team or engineering itself. And, you know, when you see DevRel, it's really more cross functional than that, and you're thinking about devolved experience and how you make sure that those metrics are aligned for what DevRel, cares about and what DevRel should be demonstrating value to the organization versus what marketing or other areas, are focusing on.

Yeah. So just before jumping into things, a quick overview of myself. I'm the CEO of MoSEF, the API analytics company. I love discussing API strategy, platform growth, and analytics. And a little known fact about myself, I love IPAs.

So if you're in San Francisco, you can find myself, drinking one in Zeitgeist. Sadly, it's a little rainy here, so can't do that today. But why we talk about metrics? You know, a lot of the DevRel metrics you see today are very focused on top of the funnel, and this is just by nature of DevRel sometimes being born out of the marketing side of things. So things like hours read time, you know, how many page views you have on various content pieces, how many plays you have, you know, for different YouTube videos, or maybe it's how many folks are signing up and and listening to a webinar.

You know, metrics are great. However, you know, the product team has very different goals. We're just thinking about things like engagement, usage, and true adoption on the platform. And so speaking through these two three things, this is what I'm gonna speak on today is adoption, engagement, and retention. If you go look at the product teams, usually, their KPIs or OKRs can be tied to one of these three areas.

Some folks might be focused on adoption side. Some folks might be focused on retention side, but it's always important to tie everything back to these three goals. And why does it matter? Well, getting developers who truly adopt the platform can be hard. Right?

It's one thing to have them, you know, read through a blog post, but it's a whole different story for them to actually adopt and integrate your platform, whether that's a API or open source project, and they have to get started with it. They're always context switching. You know, they don't trust nontechnical people. That's what we love, you know, within the DevRel communities to be more empathetic towards developers. But even if you're the most empathetic DevRel person out there, you know, those engineers and and other folks, you know, they're they're focusing on other things.

You know, they they have a road map. You know, their manager is telling them to do this and that. So how do they have time to, you know, play around and and adopt your your platform? In addition to that, you know, there's other stakeholders that you now have to consider. So on one side, you have the developer adopting your API, you know, the checking out documentation, but then there's the engineering leadership.

Right? And what does engineering get out of it? Whether it's, DevOps or some other areas, you know, you have various business teams, engineering leaders, and, you know, they may care less about whether developer is having a great experience or able to, you know, discover this content. What they do care about is the impact to the the top line or bottom line of the company or the organization. So let's first walk through, how do we truly understand product adoption.

One of the things I always recommend is taking a look at your your cross platform funnel. We love talking about things like time to first hello world, time to first working app. In this case, what we're looking at is now that they signed up or that they, view that blog page, you know, how many folks actually end up signing up for this platform and then getting to that first hello world? That can mean making a first API call. It could mean, you know, getting a Opus projects up and running.

Maybe it's running a Docker image or something like that. At least something that allows them to have a best interest in moving forward with this project and and spending more time there. They're also able to see some initial signs of of value. And then there's the last step here, which is truly seeing value from the platform, whether that is deploying an integration, it could be getting a working app out there in in, say, the App Store or something like that where there's real users or real revenue coming in. Until you get to that point, you really don't have adoption yet.

But what what the tricky thing here too, though, is you're looking at two different platforms at the same time. On the left side here, which is the, blue bars, that's what's happening within your marketing pages, blog pages, or any other, you know, top of funnel content that you have. What we're really trying to do is get people into the product itself, get their get inside the product. And that's the orange boxes here, which is what we always recommend tracking. And eventually, hopefully, you can loop that back to the first page, which is, how do you, you know, drive that viral loop.

Right? So when you set up these funnels, time to press hello world, one important thing is to also look at the volume of engagement. Right? Because it's one thing to make one single curl command through, say, and and hit a API. It's a whole different story to say, you know, someone sent a 100 text messages or did, know, a 100 different payment transactions.

And so in this case, what we're doing here is setting up a three stage funnel. Our first step is, you know, signing up for the application. Second step is making the very first API call. So that's our hello world example. Okay?

So they're able to at least play around with Postman, or maybe they're able to you know, you have some interactive documentation that able to make that API call with. And then it's the green box, which is the main goal here, which is how they made over a 100 payment transactions. And we can, of course, time box this or look at this in in different ways. But, again, the main goal here is is the volume of of activity. Just a quick tip here.

If you don't see people getting to that green box, you can always, you know, add those in app guidance, email guidance. Just nudge them in the right direction. You know, if they haven't made that first day of the call within a seven day period, you know, just let them know, you know, what is the next step here. Because many times, you know, developer could be confused on on what they should be doing. But let's walk through product engagement, which is the meat of you know, we, start thinking about, the usage of a platform.

So how do you actually measure, engagement? Well, one of the easiest ways, but usually a bad metric, is something called page views, right, that we we're all track You can see how many folks are, either viewing the home page, maybe view particular app pages. But in reality, they could be logging in and not get any value out of it. Right? Are they do they have data?

Do they, are they coring the API and getting good results? Have they even integrated or not? Right? So we can a little bit of a better metric here, which is weekly active tokens. In this case, at least we're tracking people are fully integrated with the platform and getting some value out of it.

Right? They're not just, going to your landing page. Then there's also another problem here, which is, you know, especially within a a developer tool space, you know, a single developer might create five API tokens. It might be a 100 API tokens depending on how those around. So if you're able to tie that back to, you know, account or user or maybe organization, that's really the main metric, to be tracking engagement.

How many weekly active users you have? And if you wanna, see this from a activity standpoint, you even add additional layers such as, you know, weekly active users over monthly active users to see how many people are returning. Now when it comes to API itself, you know, there are some other bad metrics that you could be tracking, and here are some recommendations. So it's one thing to be looking at, you know, API calls per minute, but what if you have a batch endpoint? Right?

So is it better for a developer to send 10 separate transactions or 10 separate API calls to try and pass that altogether into one single transaction? Many times, you know, the best developer experiences try to create, you know, one single API call for one single business transaction even though there might be many different, underlying steps to complete that, transaction. Right? It's still only a down to a transaction that's either made or not made. Right?

So many times, API calls per minute is usually a bad metric. You may also have, like, health probes in there, failing calls, and then a lot of other stuff. So how do you how do you get a little better? Well, then tie it to the business transaction. In this case, if, you know, your SMS API, instead of tracking how many requests they're making, it's more about how many, text messages are being sent through the API.

You know, if you're a a payments API, how many payment transactions are made? Right? But even then, you know, that that may not be the best metric because, you know, you could have 20 different communication channels for telecommunications platform. You have video. You have SMS.

You have a a voice. And then you also have how much are they actually sending. Right? Is it, you know, $10 in in payments, or is it, you you know, know, a $100,000,000 of payments going through the API? And so the best metrics are if you're able to really deep dig into the the breadth and the depth of usage of the API by your customers.

In this case, how many different features are they using? Are they using both the SMS API and the voice API? Are they sending, you know, doing, you know, thirty minute phone calls versus just, you know, twenty second, test clips or audio clips. And this might be dependent on the domain and and, industry that you're in. So here's a couple examples of how do you think about different, business transactions.

You know, in in the ecommerce platform, it might be the number of items listed or viewed, with the platform, you know, for for a typical SaaS app. It might be number of integrations being used. Is it just one integration, or is it 20 different connectors or integrations? You know, if you have a data API, think about things like match rate. You know, were they able to access that information in your database, or whether there's zero real adults.

Right? Maybe, you know, it just wasn't able to come in your database. For logistics API, it might be, you know, how long does it take to, complete, that shipment? Was it on time or not on time? Just because they, you know, place the order for a shipment, but then it's always late, you that know, can be actually a pretty bad experience here and and not really aligned to, you know, product success.

One last thing here is you can start adding things like user attributes, truly understand who your customers are. It's one thing to see developers signing up for our platform. It it's a whole different story to see. You know, you have engineering leaders. You have, you know, different a wide variety of different backgrounds, whether it's senior versus junior engineer, trying to find you know, how did they discover your your platform?

Was it through, you know, ads versus, you know, attending a webinar or some other activities that you did, you know, on the marketing and DevRel side? Once you add all these pieces together, we can start formulating a better picture of engagement, you know, with the platform, whether it's broken on my customer, demographics, that type of stuff. But more importantly, now we can start looking into what they actually did and were they successful. You know, example that I'm showing here, we can see, you know, the unavailable, which is the red line, is actually quite high, and it's almost the same as available. So even though we're tracking, you know, daily API users, we see that our our true you know, what we consider a active user should be only half of what we're seeing here.

Right? Because we might wanna remove the available and out of stock labels and only consider, you know, folks who saw, you know, item for sale as available as as being successful. And to show a little differently here, now we're able to even break that down by, those customer demographics and truly see, you know, who is seeing, you know, success, which in this case, we're saying, you know, available inventory, for this ecommerce platform. One thing we recommend is is start, scoring, your leads as they come in. And so this is what we call p q PQL scoring.

Easy way to do this is this red, yellow, green. And usually four different criterias we recommend, but you can always adjust this based off of your your company and and your product. You know, number one is the different APIs and features. So that's going back to the the breadth and depth of features. Number two is, you know, how many transactions have they sent?

Is it just one, or is it, you know, thousands? Does it look like it's, deployed to production or not? Then also, there there's the, per you know, after all, it's it's folks that are signing up for your platform. So is it the right person? You know, what size of company do they work at?

What industry are they in? And are they doing things like adding seats, adding users, adding licenses, especially if there's some type of UI, component for for your platform? And all four of these can be combined together to create some type of, lead scoring. And notice that it's, you know, looking at both with having with the product itself along with who the person is. Right?

You look at more traditional sales processes, many times you're only looking at the person without, you know, considering are they actually using the product. So last bit here is let's walk through product retention. In this case, we're looking at what user are doing and are they sticking around and actively using your product. Right? So this is a little bit different than, you know, subscription retention or or revenue retention, which is just looking at the dollars and and and logo growth and and churn.

In this case, what we really care about are people coming back and actively using the API or the platform. When you build these retention curves, in this case, we're looking at a daily retention graph. So day zero should always be at 100% because in this case, you know, anyone that let's say, you know, the first step here is is signing into a platform. The second step is making a call. We would hope that, you know, 100% of people are are in that day zero, but it's gonna eventually taper down.

You know, good retention curve is gonna be roughly flat. It could even, go up a little bit at the very end. So some people call that the smile retention curve. If this is always going down to zero, that might mean you have a leaky boat. So why spend, you know, more efforts on DevRel and webinars and and and acquisition when the product itself, you know, is leaky?

That may mean you need to focus more around the developer experience side of things such as, you know, confusing documentation or maybe, you know, your reference material or or your onboarding experience. Right? Things that are you can be done within, you know, that that middle of funnel stage versus just continuing to invest into, you know, conferences like that. We can also dig into these retention curves a little further. Does, you know, which programming languages or SDKs slice and dice by, you know, versions of API that a customer might be using or even things like the customer persona?

Are are they a engineer? Are they back end versus front end? Are they senior versus junior? You know, where do they live in the world? Do they work at a large company versus small company and so on?

Example of how we can leverage these retention curves. In this case, you know, we're breaking it down by different programming languages. So the top one here, see Node. Js. We see that's actually a very healthy retention curve.

Starts off at a 100%, like I was mentioning earlier, but then it tapers around that, you know, 20 to 30% range, which is pretty good. Then if we go down to PHP in in the red box here, we can see that actually drops down to, 0% on day two. So so this implies something is wrong with PHP, whether, you know, the SDK itself is buggy, the documentation or or something else is is is lacking. It may just require you to to create some more, content and tutorials and quick starts around PHP, but it does warrant some investment and focus there. Lastly, when it comes to DevRel, you know, it it's always great to share some of these insights and metrics directly with your customers.

You know, try and make them successful. Right? It's it's really important as as a developer to to feel that you're empowered to figure things, you know, by yourself versus, you know, always following that support ticket and trying to get help from someone else, whether that's things like SLA and performance metrics, usage metrics, especially if you have some type of, you know, pay as you go or usage based billing model, even things like logs, whether, you know, if it's a API as a service, you might wanna embed some API logs. You know, if it's open source tool, you know, if you're able to find a way for them to, you know, see when when the project's initialized or maybe a way to see the output there. One last tip that we usually see, happening is, you know, keep your customers informed or keep your development informed.

Those are things like quota limits, rate limits, errors with the API. You wanna avoid, you know, surprises as much as possible. And if you're able to pull in some of these other metrics around, you know, spikes in usage, you know, that can help, especially on the more developed relations and CSM efforts. There's one takeaway. You know, optimize for product value.

You know, what are they actually doing with form? Are they integrated? How long did it take for them to integrate and and get set up with the platform versus just page views, by itself? Again, that doesn't mean you shouldn't look at page views, but it's always you know, should be taken within the context of the overall, devolved experience or product experience.