September 9, 2020
DevRelCon founder and CEO of Hoopy, the content agency for the developer economy.
Borrowing from Voice of the Customer, the dev rel team at Asana created Voice of the Developer (VoD). It’s a methodology to collect and organise developer feedback into a ranked list. They then use the resulting list of developer priorities to help decide what their team should work on. Here Asana’s Jeff Schneider describes how that works for them.
Hello, everyone. Thanks for joining my talk. My name is Jeff, and I’m a developer advocate at Asana. Our team has created a process that we call Voice of the Developer or VoD for short.
It’s used to shape both the platform and broader company roadmap. The process creates an artifact that we call VoD Priorities, and it looks like this. So it consists of a ranked list a developer needs that are organized by developer personas, which are supported by numerous user stories that we’ve collected over a variety of channels.
So when the DevRel or API fixes one of these developer needs, we can measure its success when it moves down or it’s completely removed from the list. The ranked list feeds into a virtuous flywheel of platform improvement where we gather developer feedback, which feeds into that VoD ranked list. This is then used to kind of shape our roadmap.
We solicit more dev feedback on the proposed features. And then we make the necessary revisions before implementing the final design. So this process kind of continuously gives us confidence that we’re building a platform that our developers want. Done. Now, but it wasn’t always this way.
When I joined Asana, we had no idea who our developers were, much less who they wanted. So for the next 20 minutes, I’m going to tell you about our journey to get where we are today, from knowing almost nothing about our developers to being reasonably certain that we’re building the platform that they want. I’ll highlight mistakes that we made along the way so that you guys can avoid them. For brevity, I’m going to consolidate my story to seven mistakes, and because I’m an aspiring optimist, I’m going to call them learnings.
The story begins about four years ago when my colleague and I, Matt, joined Asana Engineering and started DevRel programming. To provide a bit of context, Asana is a work management platform. And unlike a Twilio or a Stripe where the API is the product, our API is a feature of Asana.
So third party developers use the API to build apps and our customers use it to make workflows and integrate with our internal systems. I mention this because it’s important to tailor your DevRel initiatives to your company’s business model and your team structure. Okay, back to four years ago. So after getting onboarded and addressing some of the immediate needs, Matt and I eagerly set about coming up with objectives for our team.
This is when we realized one of the biggest challenges in DevRel is determining which of the numerous activities is the most leveraged, should we improve our docs, move to the OpenAPI spec, write some code examples, create SDKs, speak at conferences, foster a dev community, answer technical questions, integrate with partners.
The possibilities seemed endless. And you know, how do you choose where to spend your limited time? What’s the 80-20 that would deliver the most value for our developers? So our initial thought was, you know, we’re developers so we know what developers want. We’ll just choose the things that seemed useful to us. Wrong. After some honest self-reflection and taking the advice of an insightful PM, we realized that we had no idea who 99% of our developers were, much less what they wanted.
And Ygritte here is standing on for that helpful PM. This brings us to the first lesson, assuming you know what developers want. To be fair, we knew some of our developers. We did have a third party app directory and we have relationships with a few of those third party developers, but that was the extent of it. We knew even less about our customers that were using the API.
We could see in the logs that many were using it, but we didn’t know why and we didn’t know anything about their experience. It was really a black box. We didn’t realize it at the time, but I guess acknowledging that we didn’t have the answers was an important first step in this kind of ongoing multiyear journey that we’re still on to understand our developers and to build a platform they want.
So having a small team, there were only two of us, we took a somewhat scrappy old school approach and decided to venture into the meatspace to talk with our developers and meet them face to face. Because it’s confusing to talk about developers that are also our customers, we came up with some labels to differentiate our personas. So we called developers that work at our customers 2nd party developers.
They’re generally building bots or scripts or workflows that only affect their domain. We’re probably all familiar with 3rd party developers. So for us, those are developers that build apps or integrations with Asana that are meant to be used by many customers. And then we use the label 1st party to talk about Asana software engineers that work on our internal apps team.
So they use our API to build like what we call Asana build integrations. So over the years, we’ve kind of, like, fleshed out deeper characteristics of these different groups, but at the time, we just had those names. So back to the story, we started emailing our Bay Area customers and trying to meet with them, and it was really slow going at first.
It was difficult to actually track down the developers and get those first handful of meetings. But after our word got out and our customer success team heard that we are solving our customer’s problems, we begin getting introductions and soon we had more than enough people to talk with. So all told, we probably spoke with about 20 developers, most of which were in person, and we learned so much.
It really was a truly humbling experience. So it’s just really, really hard to understand the behavior of your users until you have one-on-one conversations and meet with them and listen in person. And ideally, you can actually build up a personal relationship. So one of the surprising things we discovered was that many of our developers didn’t actually use the Asana web product, which leads us to our second lesson, don’t assume your developers know your product.
I know it’s really hard to imagine, but not everybody loves your product as much as you do. Many of our developers were actually assigned to build our API. So they weren’t all jazzed about having to learn a new product or the new API on top of their regular work. So for example, like, a marketer or a designer whose team loves Asana may have put a request into a developer on IT or biz tech who’d never actually heard of Asana.
So this realization made us reevaluate our developer experience. We’d wrongly assume that developers understood the basic building blocks of Asana. So one of the ways we addressed this was by writing an object hierarchy doc to explain the elements of Asana, and this has now become one of our most popular docs. Another early discovery was that many of the folks weren’t actually professional developers, which leads to our second lesson, don’t assume all of your developers are the same.
So unlike our third party app developers, we discovered that many of our second party devs were on business or IT teams and were not software engineers by trade. The docs and examples that this group craved were different than what a professional engineer finds helpful. So kind of based on these learnings, we’ve created distinct personas that I’ve mentioned earlier, the first, second, third party, started to flesh those out a little bit more.
So this allowed us to create different developer tracks for each group. So for example, we made Getting Started guides and coding tutorials to help the nonprofessional devs. So it would be very difficult for us to have understood these nuances without actually sitting down and meeting these folks one on one. So we packaged all of these developer interviews into case studies and presented them to numerous teams across Asana.
So as an aside, this kind of, like, internal roadshow that we did, it really taught us the importance of internal platform evangelism. So if you can’t get your colleagues really excited about your API, it’s going to be very difficult to cultivate a thriving developer ecosystem. So it was a great lesson for us and kind of, like, allowed us to hone our pitch and messaging and figure out what resonated about our API.
So anyway, these presentations, they were well received, but nothing really changed, which leads us to our next lesson, don’t just capture cool developer case studies, you need to ask the hard questions to a broader set of developers. This is essentially what our case studies captured. The API team saw our findings as encouraging anecdotes, but the case studies didn’t really communicate how the API needed to be improved.
The customer success, sales, marketing teams we showed, they were excited to see how our customers were using the API, but they had their own goals, and the API and DX was not one of them. So it was clear that we needed to go deeper by asking better questions, like what’s missing, what’s confusing, how can we do better. So, yeah, thankfully, Asana is a customer-centric company, and we had a UXR team that was willing to speak with me.
So I sat down with one of our researchers and got a crash course on the basics. And here are some of the takeaways that I’ve got from kind of that crash course. So you want to choose your research methods and goals before you start talking to users. So for instance, we decided to take a qualitative versus quant approach, and we looked into understanding user behavior versus user attitudes.
Ask open-ended questions. So this might sound obvious, but it was helpful for us. So you want to give people the space to speak their minds, and this kind of, like, led to insights that we wouldn’t have known to ask about specifically. So just kind of get out of your own way, let your developers tell you what they experience.
And then, finally, come in with an open mind. So you got to fight the urge to form a hypothesis until your research is complete. So if you come in with an opinion, you risk only hearing the evidence that supports that hypothesis. So after completing our crash course and earning my UXR hat, we set about finding a more structured approach to organizing and develop our dev feedback.
So we decided to emulate Asana’s Voice of Customer process because it was battle-tested and people were already bought into it. So we dubbed our version of the process, Voice of the Developer, highly original, I know, but it was clear that we needed to continue listening and gathering more developer stories.
So we used a variety of channels. We have initially just kind of interviewed folks. So we moved to a much broader set of ways to get in touch with other developers. So we sent out our first developer survey. We’ve launched a developer community on our website. We added a feedback form to the docs.
We worked with our support team to aggregate API-related support tickets. We worked with customer success and sales to capture any customer API-related stories they heard. We met with our internal apps team, the first party developers I mentioned, to capture their API feedback. And then we did research sessions, like we had with our second party developers.
We did research sessions with our third party app developers. So after collecting all this data, we meet our next learning. So it was clear that we needed to present our findings in a more actionable format than we had in the past. So all of these developer stories collected across all these different channels fed into a single Asana project.
So within this project, we could organize the feedback into themes so each section was a different theme, and then we used custom feels so we could track user stories by dev personas. So if we wanted to see, like, you know, what’s affecting all of our second party developers or third party developers, what’s affecting all of them, it was easy to see that data. So here is the first iteration of that ranked VoD properties list.
So I showed a later version when I began. Here’s kind of the first one that came out, and we went and presented our findings again. So this time, things went better because we’ve presented the data in a much more actionable format than just, you know, “Yeah, here are some cool case studies.” So we knew that we were on the right track because the API team actually started using this list to inform their roadmap.
And I’ll share an example of what I mean so you would kind of know how this fits into that flywheel I referenced. So as you can see here, advanced task filtering is the number one developer need. It was affecting both second and third party app developers. So because it was the number one need, the API team prioritized building a fix for it.
So they created a design doc for a proposed /search endpoint so the developers could build complex filters to find and retrieve the exact data they needed. Because we now actually have relationships with these developers, we could share our design doc with them and get their thoughts before we build it. So we had a group of developers sign an NDA, and we felt comfortable, you know, sharing our roadmap, sharing design docs, they could get feedback, and then we brought this feedback back to the API team, they were able to incorporate it into the design, and then we were able to ship the final revised design kind of based on that feedback.
So once that task /search endpoint was shipped, we actually stopped seeing developer requests for task filtering, and we were able to remove it entirely from subsequent versions of the VoD. So while things were definitely improving, this was much better than our first go-around, we still needed to iterate on the process. So one of our mistakes was actually not involving those API engineers and their PM in the ranking process.
So next learning, we learned that to get the best results, you got to include your stakeholders. So assuming we knew developers best because we were DevRel, we met in seclusion to rank developer priorities and then essentially told the API team what to build. So you know, to their credit, they were still receptive, but rightfully, they had a lot of clarifying questions.
They wanted to dig in, they didn’t really know what was being meant based on our little sentence about it, they wanted to know why things were ranked the way they were, all very legitimate questions. And so moving forward, we invited those API engineers and their PM to the ranking process. So this resulted not only in more buy-in by that team, but it also created more clarity in the VoD itself.
Another challenge was getting broader reach within the Asana organization. So our findings were now shaping the API roadmap, but how can we incorporate those into the product roadmap? For example, if customers are willing to devote their precious internal injury sources to building a bot or a script to enhance some aspect of Asana, isn’t that a clear signal we should consider making that enhancement a native feature in Asana?
We thought so, so we began to rephrase some of those developer needs and themes in a way that would help all customers, not simply those developers. So this shift was our final major lesson in this journey. So while our primary constituents are our developers, we also serve our customers, the business, and ultimately, the company mission.
So we started reframing some of the developer needs to encompass some of the broader customer and business needs. So this eventually led us to be invited over, you know, maybe a year process, doing a lot of presenting, that’s eventually led us to join the official Asana Voice of Customer process. So now, our developers are represented at company planning.
So kind of, you know, we went from just presenting case studies to slowly kind of shaping the API roadmap and now kind of getting invited to help shape the actual Asana web product. So before I end, I’ll do a quick recap of the lessons that we learned. Anyway, if you take just one thing from this talk, I encourage everyone who hasn’t to see the value and actually connecting with your developers one-on-one so you can learn from their stories and feel their pain.
You know, in-person is fine. We did that with some of our second party developers. In this COVID world, obviously, that’s harder, and also with developers spread around the world. So when we did our UXR sessions with our third party developers, that was almost entirely through video. So it doesn’t really matter if it’s on the phone or through video or in-person, as long as you’re kind of connecting one-on-one and asking them open-ended questions.
There’s a ton of value there. So once you take that first step, remember that just because you’re a developer doesn’t mean you know what developers want. Your developers are not all the same, so consider organizing them into personas. Your developers and probably no one loves your product as much as you do, so take that into account. Make a research plan so that you’re asking the right questions.
And then if you’re going to take the time to do that research, make sure you present your findings in an actionable format. Don’t get overly protective or precious with the process, so include your stakeholders early and often. And finally, if you want to maximize the impact of your feedback, frame those needs in a way that supports your customers and your business.
Thank you so much for your time.