Iterating on Developer Marketing ... with Metrics!

Jon Gottfried: Welcome everyone. I'm going to be talking today about iterating on developer marketing with metrics. All of us here are investing a lot of time and energy and often budget in creating really compelling experiences and marketing campaigns for developers. And it can be incredibly difficult to assess not only areas of improvement, but also what might be working well in a lot of these marketing campaigns. And so in this talk, I'm going to cover a couple of key areas here. First, I want to give you a little introduction on myself and our community, just so you have some background. Then we're going to be spending the majority of our time going over three real world case studies of actual developer marketing campaigns that we worked on and what the takeaways are from that. And then I'll leave you at the end with some action items and lessons learned that you can hopefully bring back and apply to your own teams.

So to dive right in, I'd love to introduce myself and our community. I'm Jon Gottfried. I am the co-founder of Major League Hacking. Previously I was a developer evangelist at Twilio and worked on a lot of other hacker community organisations like Startup Bus and Hacker Union and others. I've been doing DevRel for 10 plus years now and I truly love it. And yes, since I'm sure a lot of people are wondering, I do have a sourdough starter. It does not have a name yet, so if any commenters want to help me brainstorm, I would truly appreciate that, but it's been getting a lot of usage over the last few months. So Major League Hacking is the global community for student developers.

We work with about a hundred thousand up and coming developers every single year, primarily through events, both in-person and virtual. And our goal is to help students learn practical skills that compliment their academic curriculum.

And one of the biggest ways of doing this is by exposing them to real world tooling and technology that they hopefully will be using when they go off into the industry. Oftentimes curriculum lags behind the industry, and so we help them get their hands on practical skills and projects that set them up for success. Now, the reason I say this is because as we're thinking about these developer marketing case studies I'm about to share with you, these are based on a foundation of being exposed personally and as a team to many of the world's leading developer platforms and also many up and coming developer platforms and looking at a lot of the patterns and repeated problems that we've observed by working with many different companies. Now, I'm really grateful that Tamo covered a lot of this in the introduction, but I assume many people watching this might deal more with enterprise or professional developers.

And I wanted to give a caveat before we dive in of why you should even care about students as a sample here. Well, the truth is that students have an immense willingness to learn minimal established technical preferences and deliver raw feedback.

This is often in contrast to professional developers who are dealing with organisational politics or years of expertise and bias that they've built up. And so I've found even for the most well-established developer products, you can learn a lot by exposing students to your platform and asking them what they think. And also you set yourself up for long-term brand awareness by making your platform a part of the tool belt of the next generation of developers. So shameless plug over, I would love to dive in. And now we're going to talk about our three case studies. So these case studies fall into three core categories.

The first is developer onboarding. Next we'll talk about documentation.

And last we're going to talk about messaging and digital marketing. So for developer onboarding, this case study set out around a promotional campaign with a goal to get developers signed up for a specific onboarding flow. Many of you probably have a conference specific or vertical specific onboarding flows. In this case, it was focused on students, but I've seen analogues for startups or Microsoft developers or Android developers. This is a really common tactic for getting developers onboarded. And we launched this campaign with a fairly simple premise. We would give out promo credit at events and via email and see how developers use the platform. So really simple copy and email.

Literally we were going to people saying, Hey, here's a hundred dollars of free credit. Try this out. And it worked. It was a really successful campaign in all of the top of funnel metrics.

We had about 60% open rate and 20% click-through rate on these pre-event emails. And that's something any marketing person will be proud of. And in addition to that, the audience was increasing by 80% year over year. So we had a constant influx of new users to test this on.

So that's fantastic, right? We start off strong, this is looking good, people want what we're giving them, but here's where things started to go wrong. Out of those thousands of developers who are clicking through this email campaign, less than 50% of them were actually able to sign up for an account. That's kind of wild, right? When you think about it, that is the core most base level thing someone has to do to use a developer platform. And so we set out to analyse and measure their experience, and we identified a couple of key problems here by looking at different metrics.

So the first problem is that we had a really bad verification flow. Oftentimes in promos like this, there's a lot of anti-fraud measures because they get posted around.

And what was happening was developers would fill out this form, find out they were rejected, but not find out why for many hours. And oftentimes the explanations were pretty ambiguous. We looked at metrics like how many times people were retrying a signup form, how many support tickets we were getting for bad verification, and how able people were to self-correct problems. That seemed obvious to us, but maybe not to them. And this is a huge problem because everyone is competing for a developer's attention. The more friction you have, the more likely they are to simply forget and move on because there's a lot of different platforms out there. There's a lot of things to experiment with and limited time in the day.

And so we set out to fix this by simply adding real time feedback to the promotional form.

It seems obvious in retrospect, but didn't exist at the time. As you filled it out, it would say, Hey, you're using a Gmail, maybe you should use an EDU email instead. Things like that. The second problem we identified here was that the signup process was incredibly long in arduous. It had almost 30 required questions. People were getting bored and giving up. We looked at things like how long it took someone to successfully fill out the form, what the churn rate was as you go on a question by question basis and whether someone could actually complete it in a reasonable amount of time. And I understand why it's important to know your customers really well and collect a lot of information, but that can be at odds with the ease of their experience.

You want to inspire people as they're starting to use your platform, not drain their energy. And so we started eliminating questions and making a higher percentage of questions optional, which really did help this problem and may have disappointed some of the marketing analytics folks, but certainly increased the developer experience. And lastly, the account types that developers had the option of choosing from were really confusing. A lot of people who did successfully complete the form turned immediately because they had selected the wrong account type and it turned out they couldn't do what they intended. And we discovered this by looking at how long does it take someone to make a first API call, and how likely were they to continue making API calls once they had signed up? If we saw that someone never made an API call or made one and then never came back, it signalled to us that people were churning after success.

Developers really do just want to solve a problem, right? Whether it's for a personal project or a business use case, that's their core purpose here.

And as developer relations folks, I think we often focus on a lot of the technical details, but billing workflows and marketing workflows matter just as much to developer experience as how the API is designed. So we provide clearer signup, playout paths and better explanations. And it helped this problem quite a bit. And what we saw after making these couple of changes is that the conversion rate from this forum grew by 50%. That's a huge impact just by assessing how developers were experiencing that first touch onboarding. And it really gave us this perspective for one, that signing up should never be the hard part, which I think sounds really obvious to a lot of us, but is an incredibly frequent problem. And the fact that DevRel needs to touch all of these different aspects of a product and informed decisions that are being made regardless of whether it's technical or not. So the next example we're going to talk about here is documentation.

This is a bit different from the first platform because we were promoting a hardware device, and that prevents presents a lot of issues. You can't add analytics tooling to every step of the process because oftentimes these devices are completely disconnected from the internet. And similar to the first case, we had a lot of excitement. These were bleeding edge hardware devices and it had a cornucopia of features, Bluetooth, wifi, ar, Arduino, Linux, you name it, it had it. And a lot of developers were really excited to try it, but as you might guess, it didn't pan out how we won. Only about 15% of people who got their hands on this device were able to make a working project that's really low. That's really not great, especially because they were so excited about it. And so we decided that we needed to dive in and get a little bit more qualitative feedback because we knew that people were failing.

I actually think that qualitative feedback and surveying is one of the most underutilised derail tools that we have. A lot of times we focus on a lot of quantitative metrics, timing, different event tracking on pages, but sending a really well-crafted survey is often as effective if not more. And we started to see this in the raw feedback that we got, and this one probably hits home the hardest, but I'll give you a couple of more examples. One developer spent almost seven hours trying to set up this board and couldn't get it to work. That's heartbreaking when they were so excited, they spent an entire day and still were unsuccessful. And it revealed to us a lot of smaller, more specific problems with how we were introducing developers to this platform. So the first is that it was a brand new system. This launched to the world and developers immediately got their hands on it, which is great, but the documentation was almost non-existent.

We started asking people in our surveys, tell us about your first experience with the platform. What resources did you use to get started? Things along those lines to reveal what was going on when they first got their hands on it. And developers told us kind of what we assumed here, that it was really hard to find accessible guides. They couldn't find comprehensive sources for troubleshooting, and oftentimes they even blamed themselves for that problem for not being knowledgeable enough. There's immense amount of imposter syndrome in tech, and you don't want to reinforce that with your platforms. And I think we often assume that if you build something with the best features and the best use cases, that developers will just come and use it, but they won't, right? They'll move on if it lacks clarity or is difficult to use.

And so we realised that we needed to create a lot more documentation here.

That was a lot more 1 0 1 level Before this, we had generated docs for the entire code base. Engineers had spent hours commenting on their code to make it really obvious how to use it, but no one had written a 1 0 1 level tutorial That made a huge difference. The second problem is that the platform seemed really simple, but it was a lot harder than that. We started asking these questions like, where did you get stuck? What took the most time getting it to work? Things like that. And developers told us that there were not even docs on what certain ports on this board did, and they want to do more intricate and complex things, but they couldn't figure it out.

And this goes back to partially what Tamal is saying, where developers are technical and they're often highly skilled in a specific area, but they can't read your mind.

Things that might be obvious to an experienced electrical engineer might be really cryptic for someone who's never touched hardware before, even if they have all of these dreams about how they're going to use it. And so we realised that we needed to handhold them a little bit more because this is a teaching moment. If your platform can teach a developer something, it opens up a lot of possibilities. So we started creating visuals for hardware. What does it mean to have A-G-P-I-O port that doesn't necessarily make sense to someone who's brand new? And so we created these really explicit guides that showed them where do you plug something in? What does that do, what not to do?

And that helped immensely in them being able to simply get the board up and running. The last problem we identified here is that software was not the hard part.

A lot of the developers using this board were able to deploy their software to the board and run it, and it was really complex and interesting and well architected, but it still didn't accomplish what they want because they couldn't figure out the hardware portion. And as we asked them, what were you trying to do? How did it turn out expectation versus reality? They told us straight up, a lot of the problems were hardware related. We couldn't figure out how to get the power source moving. And so that ruined our project.

Often the problems with developer platforms are really hard to investigate. How do you as someone behind the scenes know that someone failed to get a power source working and that's what ruined their project, not the API design. And I think that this often applies to cloud platforms as well, where you have all of these libraries, you have the API itself, and then developer has their own stack.

And debugging where the friction points are with all of those different things in play can be incredibly difficult. And I find that the best way to figure it out is often just to ask explicitly. So we started to create pre-configured operating system images that pre-configured most of the hardware compatibility, so installing the best case scenario, libraries, things like that that made someone's life way easier even if they didn't understand electrical engineering. And the conversion rate for this usage doubled. Obviously this is just a starting point, but simply by creating better resources and not assuming so much in a developer's ability or perspective, we're able to increase their success rate.

And so I would encourage everyone to look at qualitative metrics with a really careful eye. I think as organisations grow, we often split out product versus marketing versus documentation versus all of these other disciplines here. DevRel frequently has their hands in all of those different pots, and you have to be taking notes and observing and listening to developers in their experience so that you can improve it. And it's not all about the numbers. You could have a million people signing up and if they're not able to figure it out, or if they do figure it out and then fail 50% of the way through their project, you need to know.

The last example we have here is around messaging. And the goal here was to introduce developers to a new type of tooling. What I mean by that is it's not a cloud platform, it's not a hardware device.

It's kind of a enterprise-y set of tools here where our customer's goal was to build more community, right? Get hobbyist developers excited about it, build long-term brand awareness so that maybe someone learns the platform and gets a job and brings that with them to their job. That's a really common case for students. And we rolled this out with a lot of information. We learned a tonne from previous campaigns. And so we had a lot of metrics from the get go. We had tonnes of different social metrics, tonnes of different email metrics. We looked at projects at events, feedback surveying, all of this stuff, more metrics than you could reasonably look at on a day-to-day basis.

But we found that the metrics were often obscuring or revealing deeper problems with how we actually crafted the marketing here. So the first problem we identified with this was that the jargon that we were using for marketing was really confusing, and I just made this up. This has nothing to do with the actual customer, but oftentimes you end up with these really technical or corporate-y jargon that can confuse new developers. And this was revealed to us by things like the email open rate, how many people were actually signing up? What did the survey results say about whether or not someone tried this platform? I think that oftentimes as we're designing marketing copy and jargon, it's about what does the enterprise customer want? What Gartner quadrant do you fit into? And I think it's equally important to inspire and excite developers.

And this is not students specific.

Certainly students have a lot of passion for their projects, but even someone in the enterprise who's been in their job for 30 years, they want to have fun, right? They want to use interesting, cool technology, and I think we often lose sight of that, and we often drive people away because they feel a little alienated by the complexity of the language that's used in developer platforms. And so we started to simplify it, turning something like a multi-tenancy, whatever database, and to just build a better database into your application. The goal of marketing copy is to hook someone, not reveal every potential possibility of your platform. The second problem was wasted visuals. This is specifically a problem with social media, which obviously is a valuable channel for DevRel. And oftentimes when you're linking to pages that weren't designed with visuals, this one from our own Twitter is a perfect example.

It pulls in something useless like a logo, and these posts are not shared a lot. No one clicks on them if they're a blog post or email. People don't view them for very long. People's attention spans are really, really short. And even in a presentation like this, the design of the presentation matters immensely in how engaged someone is in it. And that's even compounded by social media or digital marketing. So you fix this by having fun and shareable and explanatory graphics. That image there is so much more engaging and informative than the previous screenshot that I had because it gets the developer's attention and says, oh, I should give this a second look.

And the last problem with this one is that we actually missed out on a lot of opportunities. Oftentimes we had a very clear call to action but didn't go deeper. We got some survey results, but maybe not as many as we wanted. We didn't really get anyone emailing us feedback or replies. And developers do have strong opinions, and this is really evidenced by that previous case study, but sometimes all you have to do to ask. And I think the important thing here is that our calls to action from a marketing standpoint are frequently metrics driven in the sense that we want people to sign up. That is the core KPI here. But what happens if you ask people to email you?

Instead, we would add buttons to an email where it said, sign up now for an account. And below that, we'd add a second button that said, email us your feedback.

And tonnes of people did that simply because we presented it as an option. And it not only allowed us to build a relationship, it also increased brand preference and usage because people felt like we actually cared about their opinion. And so by making all of these changes, the brand preference of this new platform that students had never seen before increased by 40% over the time of the promotion, which is really great. That's exactly what you want to see when you're building that long-term love. And so I think that we need to have our hands in an influence marketing copy. And in many cases, you have to look at this from many different perspectives.

It's not just about your core customer, but also all of the different segments that you might talk to. If I was giving this talk to a group of students versus a group of DevRel professionals, I would change almost everything and how I describe what we're talking about, because you all have a different perspective than students do. Same thing is true of your developer marketing. So let's reflect on some lessons learned and what you can do to take away some action items for your team.

So we have these three case studies, developer onboarding, documentation messaging, and with developer onboarding. I would encourage you to do these three things at a bare minimum. The first is literally watch your developers. In every role I've ever had, I learned an immense amount from simply sitting there and watching someone go through a process or use an API almost like pair programming and just taking notes.

And that drastically improved our ability to understand customers. The second is to track time spent. It's not just about how many people sign up or what the conversion rate is. Often it's about how long it takes too. And there can kind of be an inverse relationship between desire to use a product and how long it takes to use. And then the last thing is to engage developers personally. And this might sound really common and obvious for developer relations folks, but it means a lot if you meet someone at the conference, introduce 'em to your platform and then follow up with them personally when they start using it and ask about it because they are then willing to be transparent with you about what problems they're actually facing.

The next is when we're talking about documentation survey developers extensively, I really think this is underutilised.

A well-crafted survey reveals a tonne of valuable information and supplement that with simple one-on-one level resources. Don't assume all your users are at the same skill level. I have seen many cases where enterprise platforms were adopted by someone in a non-technical role and then roll out to the engineering team. You need to think about those potential avenues of adoption. And the last thing is look for patterns. Oftentimes we get caught up in creating these really compelling quantitative ROI reports, but a well-crafted qualitative report with interesting patterns in it can be just as valuable, if not more so. And lastly, think about metrics in a different way as we're compiling all of these reports and saying, okay, this is our open rate. This is how many views we got, et cetera.

You have to ask why, right? You have to separate vanity metrics from real world KPIs. And a lot of this can actually be informed by core marketing principles. I think that there's always almost like a tense relationship between developers. Are they engineers in developer relations? Are they marketing people? Is it a mix? I think that it's only a value add to learn marketing principles as well as engineering principles so that you understand best practises and analytics and tooling and can use that to improve your developer's experience.

And the last thing is to invest in copywriting. It's not a skill that's super focused on for technical folks, but it is incredibly valuable. Understanding your reader or customer or user and how you explain concepts to them can make a massive difference in their success. Is this a multi-tenancy crazy cloud platform or is it a database, right? You have to really think that through and learning good copywriting practise can make a huge difference there. So thank you all for listening. I hope that it was informative and valuable. I'll post the slides online later on SlideShare and now I would love to take some questions if anyone has them.

Tamao Nakahara: Excellent. Thank you so much. That's great. So I'm going to be looking to the right, because I am looking at our Slack channel, so reminder to everybody if you have questions, please post 'em there, but I'll get started with my own. I have so many thank you so much for that. I really reminder to people who come to Decon, it's something that I've impressed upon every year is that we really love not just the concepts and practises, but having them really be grounded in actual experiences. So really hearing these experiences from you are so helpful, and I really appreciate your willingness to share all these mistakes made and things how we learned not anymore. Yes, yes, same here.

Same here. So thanks so much for that. So yeah, let's start in order because this is a bit of a personal thing for me in terms of the dev marketing, I've held different roles.

I've held roles in different teams currently. I mean, I roll up directly to the CEO. We're a small company, so we're not within marketing. Before I was in a role where I was not only in marketing, but at some point within field marketing. So there's all these different parts.

And so I've used the term dev marketing, but in the last couple months I've kind of been reflecting on that, excuse me a second, and wondering, because we've had some issues about over tracking and I thought about maybe really there isn't a term we joke about, oh, I market to people who don't like to be marketed too. And I run dev marketing and I thought, well, maybe it kind of rolls back to this concept of developer relations and why we've had this term because we're building relations as opposed to doing this tracking so much. So anyway, without talking too much, I guess my question is, first of all, is it really just a dev issue? I think there are plenty of customers who don't like to have to sign up for something right at the beginning. So I guess I'll kind of pose it to you. In what ways do you think it's specifically developer and what things do you think it's a corporate marketing thing that maybe has just as much friction and issues in consumer experiences?

Jon Gottfried: In some ways, I think developers are indicative of a larger trend with consumers. I've always heard like, oh yeah, developers hate to be marketed to, but that's not a hundred percent true.

What that actually means is developers often care more about getting their hands on something and their experience with it than they do about what you say it can do. And I think the perfect example of this is if you ever go to a website where in order to try the product, you have to get on the phone with a salesperson, that drives me bonkers. And I think it drives developers crazy too when they have similar experiences. And I think that that's actually a larger consumer trend. People want to use something immediately. They need that instant gratification. And for developers that can mean making their first API request or getting an account or starting to use something.

And they may not be ready to be a buyer, they may not be ready to think about it in the context of their work.

And I think that marketing is often about, here's how you can use this for business, but a lot of really compelling business use cases start because someone becomes familiar with a product for fun. That's a really common path for developers, for anyone creative. And I think that that is actually where a lot of consumer marketing is going is it's really about experiences and feeling and how you use something not about how cool it makes you look to your friends.

Tamao Nakahara: And then what about that signup? I don't know if I represent a certain population, but if I download a phone app and it won't let me even try it without a signup, I'm often just going to forget about it. How do you feel about balancing that part? Is there a way to still have marketing be happy or even marketing within your own org and saying, we might even just remove the initial signup process.

Jon Gottfried: I don't know.

It kind of depends because there are a lot of factors that go into that. Is there a high fraud rate in your industry? Does it literally cost you money when someone signs up and starts using something for spam purposes? I think it's a difficult question to answer. I generally believe that you should make signup processes as simple as possible while still accomplishing your goals. And often that can be like name, email, password, maybe. But I've seen some like Streamy yard, which is what we're using for this conference is a great example where they don't even have you make a password, they just collect your email and you get going. And that's wild.

That's a great user experience. And I think that more people are moving towards that kind of thing. And I dunno, it's a balancing act. You still have to prove marketing ROI without making someone's life miserable. And I think that people are way more willing to give you demographic or use case information once they've been using your platform for a while rather than making it a barrier to entry.

Tamao Nakahara: Yeah, I guess I gave a talk recently in which I shared a past experience where there's an education team that was sort of saying, oh, Dell's giving away all this free stuff that we would normally charge for. And I was saying, well, what if we found a shared metrics in which our goal was to grow the user base by five two x, 10 x, whatever, then people will be knocking on your door for more training. So how do you think of it this way where you're like, well, maybe if we remove the signup, we may grow the user base by 10 x and then we'll have more people engaged and maybe the responses, but I won't know who they are, so it doesn't matter.

But what could be some balance there? Because in some ways, one of our key metrics around DevRel is around just growing the community, and then that's part of what goes through the funnel.

Jon Gottfried: I mean, in some ways, DevRel is taking a consumer marketing approach to B2B products, like you're saying, I want volume, I want people in the community, and then somehow we'll make money on the other end, which is really hard to stomach because it's a long tail conversion that's often really difficult to track. I remember when I was at Twilio, and granted this was seven or eight years ago now, but frequently you meet someone at a hackathon or a conference who makes a totally novelty product using your API, and six months later they end up in a sales funnel and you only hear about it through a secondhand source. They didn't say they met you at that hackathon, they don't remember that. You don't remember that. It was just this little touch point that results in something down the line. And I think that's incredibly valuable and probably one of the biggest benefits of a community first developer relations approach.

But it's super, super hard to, except with a lot of personal relationships because oftentimes those people would email me personally and say, Hey, help me do this thing for work now. And that was not how I initially engaged with 'em, but it's what it turned into.

Tamao Nakahara: So on your next area, you really emphasise the importance of qualitative feedback and surveys. I really appreciate that. So I put my little academic hat on. I always wonder, I overthink surveys

Jon Gottfried: Because

Tamao Nakahara: I think, oh, well, I had some friends who are sociologists and the years of academic experience of how does craft the particular wording for a survey, how there isn't bias in it and all that. And I'm sure there are people business who've worked on that a lot, do you think to that level or do you just say, I'm not going to worry about it. We just want to get some feedback.

And maybe there's bias and maybe it's not perfect, but at least it'll get us to a place. How do you craft your surveys?

Jon Gottfried: We think about that now, and there are people on the team that are very expert in designing well-crafted, unbiased surveys. I'm not personally one of them, but that expertise does exist now. But I think even in the early days when we were figuring it out as we go, some surveying is better than nothing. Even open-ended questions that are totally open to interpretation by whoever looks at them, they're not really easy to quantify, gives you a lot of valuable touch points. And what I found is when something is particularly interesting or maybe reveals a larger pattern, get on the phone with that person, email them and say, Hey, I saw your survey result, that was super helpful. Can we talk for 20 minutes about what you meant here?

And that can reveal a lot more depth, especially when the surveys are not super well crafted. I dunno, I think that some surveying is better than the perfect surveying. We don't need to be perfectionist, even if that's the right way to do it. And at scale that matters way more. But if you have a hundred or a thousand developers or even 10,000 developers, it may not matter as much. Maybe just having the information coming in from that source is valuable enough.