Baby steps for dev rel metrics

Jason: Hey everybody. Welcome to Baby Steps for Metrics. I am going to be showing you some very ugly reporting today. So my name's Jason. I work over at Sitecore as a manager for tech evangelists. We're kind of a developer marketing arm of developer relations, but we do a lot of advocacy work and a lot of hats. But today I want to talk about the really dirty, ugly part of metrics, which is getting started. When you are sitting there and you've got nothing and you see all these awesome things like all the speakers have been talking about today, like was said, it's the masterclass.

Felipe was just showing these awesome data sets. What if you don't have that? What if you are getting started? How do you do that? And so I want to take that through our journey of how we went going and doing that, and my priorities were I got no time to do this. I need to show something and I need to make our team look good. How do I get there quickly?

Now, when I first started looking into trying to do metrics, the really hard part for us was how do we even know what we're supposed to do? I came into Sitecore over three years ago now, but at the time I'd never held a DevRel position before. I'd always been in the developer side of things and I'd definitely not done product marketing before. So I was walking in and I'd been hearing about how the marketing department really likes data-driven marketing. Okay, that's great. So I need to show how well we are doing and I want to track what we're doing accurately, but I don't really know about this whole developer marketing thing. So I grabbed hold of the enablement programme. We were a small team when I joined and everybody was kind of doing everything, but I kind of understood the enablement side of things.

It matched up with what I'd done before. So I figured I'm going to talk about enablement, I'm going to talk about how we can go and help people. So where am I going to start? I had absolutely nothing. We had no dataset, we had no dashboards, we had zero knowledge of what was going on. We didn't even know what people were working on. Now, I mentioned marketing because I sit inside of a product marketing department rolling up into marketing, and the reason that's important is because different departments see developer relations differently of what they want to get out of it. So we have some folks who do developer experience and documentation and stuff on the product side, and then our side is sitting in the marketing department, and I knew when we go and look at how we represent our team, I have to report on ways that they're going to understand things and metrics that will make sense to them.

So what is it that I'm going to put on this report? I'm just like you staring at this blank screen going, what am I going to write? So I started by thinking, what is it that we do? Let's just figure out all those hats. So the big thing that I knew was we help developers, but we're not a developer SaaS, API company. We build this massive platform that can go on-prem as I as PaaS, SaaS, you name it, somebody's got a way of getting it. But as a company, the purpose is to help marketing teams deliver great customer experiences. That sounds great, but then as somebody trying to target developers, that sounds like the marketing team.

So where are my developers at? They're sitting there in the customer. Maybe it's an IT team. They might be influencing the decisions on the purchase, maybe they're operating it.

We have a lot of developers sitting over implementation agencies, so they're working with the product to make it do what the customer wants it to do. So we're here helping freelancers and partners and developers at the customer, and we're also doing sales enablement because the sales engineers need to know what's going on inside the box. And we also have to help out with customer success because we want that customer community to feel good, feel appreciated. We're out there on social getting into Slack, trying to help everybody out, and then the product team, well, we've got documentation. We want to make some help there, maybe help out the training team, and we love content marketing. We're doing the videos, the blog articles, get people excited doing events, things like this. So we've got a lot of hats. It's a lot going on.

So where are we going to start?

Now I come from a continuous delivery advocacy background. So I started putting on a dev hat of like, how do I iterate this? I knew I needed to start small, I had to do something that I could get done quick. So let's count some things and then move into some basic analytics. If I can get to that stage, then at least I'm seeing some of what's happening after that. I figured I needed to start getting some insights. So popularity metrics are usually pretty easy to try to get some kind of insights into it, and I knew that I had to start telling a story. So that's the journey we're going to go on is how did we go about doing these baby steps and what are the ways that maybe you can get started doing your own metrics in a similar way?

Now I mentioned starting small and coming from that continuous delivery background, wanting to always iterate, do better than we did before. I knew we needed an early win, something we could grab hold of, say, yay, we did something and go from there, build on it, and I said, what are we going to do? Let's count something. Anything. It doesn't matter. Let's just measure something. But anybody who tells you anything about metrics is going to tell you that counting things is a terrible metric. It doesn't measure the right thing.

You don't want to measure developers by the lines of code they create, and you don't want to measure re experts based on the number of videos they produce. But we had to start somewhere. We had nothing. How do we get started? So how do I even know what it is that we are doing? I didn't even know how to figure that out.

So the first time I ever got any type of a metric on any quarterly report was these two little bullets embedded in a slide up in a corner in this box. The emphasis here is purely for your own purposes, so you could actually read what it was. It's not impressive. There is no value shown in these metrics, but it was the first time we got something, and while this particular report was not very impressive, what it did was something much bigger because I didn't know what to do to get those numbers. How do I even know that we did 32 videos that made me have to go back to how we worked. I had already started us on this kind of content planning board where we could put things through review and publishing, kind of track what was going on. So we started tagging stuff.

What type of stuff are we doing? Then when we got things to certain places, we started tracking where we finished that. What quarter was it done? So when we got to an end of a quarter, it was much easier to report. So the biggest value we got out of counting something was actually enforcing us to do tracking. If we had never actually made that decision of like, yes, we have to count this, we would've never done the work to track the work we were doing, we would've just been doing whatever the next fire was. So this allowed us to really start getting to more planning, getting a better tracking. Then over time, we get the good stuff.

Now we're doing this all year long, we can start looking for trends, things that are changing like here. For some reason, our video count suddenly dropped this last quarter, docs skyrocketed, and we're not doing in-person events anymore.

Fancy that. These are things that I can just look at an easy graph and say, oh, something changed. Maybe I should go look into that. Now, this is comparing recent data, but the great thing about doing this even longer because we've kept counting these things, is I can start counting compared against last year. Now, marketing teams love the year over year comparison because it eliminates the seasonal variance. So let's say you've got a big community event that happens at, maybe it's the summer, maybe it's the fall for you. If it's happening in that timeframe, every year you have certain activities that are going to happen around that every year. By doing the comparison year over year, you eliminate that as a variant.

So for example, here, I've got videos actually showing as they're up versus last year, but on the last graph it showed it was down versus last quarter. Well, that helps me know that maybe there's a seasonal issue here. Maybe we just do fewer videos in general in Q4 versus Q3. So I can probably eliminate that as something to investigate. But the docs are definitely out of whack with what would happen last year, and we're not doing personal events. So between the two, I started getting a of, oh, here's something I can actually start going and looking into and something I can start showing to management about changes that have been happening with the team's effort.

Now at this point, like I said, counting metrics, terrible, terrible metric, but we got somewhere that was our first step. We finally had something that we could build on, and the next thing I wanted to do was be able to say, how do we go about figuring out if all this work we're doing, anybody's even looking at? This is where I started leaning on the learning of the marketing side of things. Because content marketing has this figured out. They've got this concept of eyeball metrics, which are super easy. They're not great, but they're easy, very introductory way for you to get something that kind of starts measuring what it is. The impact of your content is. Now for video, you can go to YouTube really easily, find the views, subscribers, everything you're getting every month.

If you're using Google Analytics, it's easy for you to figure out how your content's behaving.

If you have NPM downloads, that's a good way to see adoption going out. It's another easy metric to grab. And Slack was an interesting one for us because channel growth shows whether or not different topics are growing in trends. So for example, with us, our community likes to create new channels. When a topic starts getting popular or a product starts getting a lot of discussion. So we can start charting the growth of those particular channels and compare that against the growth of the community as a whole. So we can start seeing, well, are certain topics going up or certain topics going down? Where's the popularity coming from?

Fairly simple eyeball metrics to grab. It's a great way to get started. Eventually, you're probably going to look at time on page or watch time or something like that, but this is a good place to start.

And from there, charting over time, trends are the way that you can start seeing is 10,000 views good? Is it bad? I don't know, this month was 10,000. Is it better? You have to look at it over time, and one of the things I like to do is set a goal. So I look at what did we do last year? Add 10 or 20% as a goal, like a target to shoot for, and then we can see if our efforts that we're changing things, are we getting better at what we're doing? Are we reaching more people?

Are we seeing us go towards our goal or away from our goal? So this was a very easy way for us to adapt very simple metrics, start getting some insights, start getting an idea of whether or not what we're doing works. Now, I mentioned earlier counting is terrible, but it's not lost investment for us.

See, we can start looking at return on investment here because we can start looking at, oh, look, we started doing more videos as the quarters went on and then we had a sudden drop. Well, how did that impact our subscriptions per month? And we can see in our subscriptions, oh yes, as we did more videos, we started seeing some more activity, more subscribers, and as we dropped, it went down. Very obvious. Anyone could have made that conclusion. But the return on investment piece here is notice how the amount we went up in the winter when we did a whole lot more is not as equivalent as the amount of extra effort we did. So it's probably somewhere between the fall and the winter effort. That's probably our sweet spot where we're going to get the best return on investment for our time there.

Then we can take that time, put it somewhere else.

This is the type of thing that you can get by leveraging that counting data you did with these simple metrics. At this point, I'm happy. I've got something. We're a few quarters into my reports now and I'm looking good, but I don't really have insights yet into what is actually making up all these views. What are people actually wanting from us? What do people want to see? Popularity metrics help with that does a very basic way of you finding out what is going on in your community. Now, one of the things that's really easy is most of the analytics tools you would've had to set up to do the previous step have these types of things built in.

So you can go in, pick a time period and say, what's the top stuff? YouTube is what I'm showing right here, and it is very simple.

You might grab 10, whatever. I like grabbing five, and then I go to each one of our channels and I take each one and I say, okay, these are the popular ones. Now, I don't use any fancy algorithm for pattern matching or anything, just the good old human eyeballs and brain. And I can see here that there is a pattern that's emerging in this quarter. I know what we put out and it wasn't all of this. So why is it all of a sudden this SXA module that we have is popping up in every single channel? Clearly we're seeing some adoption and some interest in the field, but it also means that they're looking for this much content on it. People need help.

So this is a great place that we can start focusing on creating a tutorial or going and doing some more presentations or trying to dive one level deeper because clearly there's a demand for this.

This was an easy way to get very simple insights into what we're doing. Now at this point, I had to admit that early on, the hardest part of all of this is how do I tell a story? When I was first entering into the developer realm, I did not believe in metrics. I in fact thought that they lied that numbers. You could tell anything you wanted with it and anyone would be impacted by it and try to gain the system and try to do whatever it was to make the number go up. So 20 years later, I look back on it and say, you know what? I was right. You can influence people with numbers.

You can tell anything you want backed by numbers, and that was why it was important. The story was the important part. Now, this ridiculous picture I have up here, the data, if we look at it here, there's a desert, there's a shadow going on, probably the light. We can see where the sun's coming in. Dude's got some flippers and snorkelling gear on, and for some reason a floaty tube, that's the data, that's the metrics. That's how it looks like to someone else when you just give them the raw data.

We don't know why this person is out there. Now, we might make you feel something when you look at this data, but that's what you bring to the situation. Someone else is coming in and they look at this data and they make an interpretation based on whatever they bring to the table. But you want to control the narrative. Why is this person dressed like they are going to go and jump in the pool when there doesn't seem to be any pool there? Why wear a flotation device when you want to go underwater? Clearly for snorkelling, I don't know, but that's the story I want to have. I want to know why that's happening.

How does all this fit together? We have to drive that context. We have to wrap our metrics around in a store. Now, here's an example. Really basic numbers, a couple numbers. They went up a little bit. Big deal. The value in this report was not the numbers, it was the story we wrapped around it, the insights.

Why did these numbers go up? Why did we do something to make those skills? A lot of the time, we have to also speak in ways that the management chain is going to value. We want to make people see that the investment in our teams is worth it and that we're paying attention and we're reacting and we're keeping up. Things don't always go well.

Here's an example. When things used to be going well, we were having great partner views, and then all of a sudden July was great, and then bam, just a sudden drop complete collapse of our analytics. Now we thought, maybe it's the summer, we'll let a ride for a quarter. No, keep going. We realised we have to start telling a story about this. We don't know what happened. We didn't do anything differently, and all of a sudden our analytics changed. So we started saying, well, maybe it's these possible issues.

This is what we're going to investigate. We're going to iteratively try different things. So let's try something out in January that didn't work. Okay, numbers didn't go up. Let's try something else. Oh, started seeing a little bit of a spike, but not really where we were last year still. So we're going to have to keep trying things.

Telling the story and framing all these numbers incredibly important. It influences people. People don't like to make mistakes. They don't like to go on hunches. They want to have data that makes it seem like what you're deciding is backed up on research. So it's incredibly influential to do this with your management chain because you want to be able to tell them, we know what we're doing. We know how to get there, and we're watching. We're doing the research.

We're doing what's necessary to make sure that what we're doing, our experiments are working. So when people say that they're data driven, really they're data supported, they're not making the decision on the data, they make the decision on the story around the data. So that's my key piece of advice for these first baby steps. Get through those pieces, get that story, and then figure out value.

So we saw lots of cool things like DevRel qualified leads, the Orbit model. There's funnels. There's a lot of different ways that you can go figuring out what your team does in terms of delivering value, and that's where you need to go next. Because when you go and you want to make your boss look good to their boss and their boss look good to their boss, they have some kind of priority already set. They already know what's valuable to them. They know what makes them look successful. So how do you tell your story so that you can make them look successful? That's going to help you ultimately in the end, and it may not change what it is that you're doing, but it might help you in how you tell that story.

What you track. So when we started here was how do you get started?

How do you get something? How do you get basic analytics? Start learning how to tell a story. Figure out some basic insights. But then you have to start asking yourself, well, what are the management of business part of this? Who is it that our team is supposed to be helping? And what is it that our team does that makes it valuable, and where should we go and do more? If you start answering those questions, you'll be able to see, oh, here's a place that I can go.

Here's a place that I can start telling my story. And just remember in the end, most of us do this job because we want to help people, right? It's not about the numbers, it's not about the metrics. Make sure you're telling the story of how you're helping people. Ultimately, that's going to be the best thing and it's most fulfilling work that you're going to be able to do. So keep helping people. Thank you very much. Make sure to reach out on et your thoughts on.