Jesse Davis started out as a developer but these days he’s an executive at DZone Software. He spoke at DevXcon in San Francisco about how to prove the value of your developer community to your company.

Transcript of Jesse’s talk

You know, the business guy is standing up at the front of the room, my name’s Jesse, I am Chief Operating Officer at DZone. It’s super interesting though, I have a secret in my past that I’ll let you in on. I’m a developer. Yes, woah, a developer. I’ve been working in the Java space since the late 90s at the turn of the century, for those of you who are old enough to know back then, and worked on the JDK a little bit and coincidentally, that’s when DZone first got started with one of the first communities.

A 20 year-old developer community

The idea of DZone.com has been in there for 20 years. 1997 was when Javalobby was founded, which was one of the first communities set up to help Java developers get and share knowledge about programming in Java. And so, what’s really interesting about this is now I get up every single day and get to think about how to build the world’s greatest developer communities. One of them is at DZone.com, and we’re also gonna talk about some of the others. DZone is big and there’s a lot of things that go on there, but the software that we used…because again, it was founded by developers, so we built the software that runs that side, we productized, and now we sell that to other great companies who use it to run their sites, so companies like Pixar, and IBM, and Amazon and Microsoft, and Epic Games.

So, we’re gonna be talking a little bit about some of these customers as well, some of the things that we did to get their communities up and running, some of the metric and analytics that we use to measure the success of a site. And the fun thing is, it doesn’t matter the size of your community. If it’s a 14 million member community or if you got, you know two or three hundred, the metrics that you should be paying attention to, to drive the value in your company are the same and so, we’ll get talking about those. So a couple of the things that I wanna talk a little bit about is life cycle, right? Whenever you get asked for ROI there’s really two ways that people interact with your site, right? There’s the life cycle of the community itself and then there’s the life cycle of a developer as he comes or she comes to visit your site and figure out how to get the information that they need and how do they move forward, how do you get through each step in that life cycle and what are the analytics that you need to be looking at? How many of you know where this reference comes from? Thank you, awesome. I feel validated.

Value in the community lifecycle

So, to get started, we’ll talk a little bit about Value in the Community Life Cycle, right? So when you first bring up your community, it’s really new, the first thing you wanna do is launch that community, right? How many of you have launched to community? How many of you are launching a community? Fantastic. So in the beginning, what you’re really looking for is for people to come and visit your site, right? I think it was Jonah this morning, he’s in the room, put up a slide that said,…it looked like a huge dashboard of things that you could flip and look at in terms of metrics, and there’s a hundred things you can look at when starting up a site. And while I get excited because my past is all about data, when I see big dashboards, he was dead on. The most important thing is to focus on the right metrics. What do those metrics do? Each phase of the life cycle that’s gonna prove that value? What is your boss gonna ask you for, when they come walking in the room and they say, “Hey, Matthew, we just launched this site. How’s it doing?” What do you gonna say? Well, a couple of the things, here are the number of new members, right? We’re gaining people into the site. Here are the activity metrics, right? The number of new posts, the number of engagements, the reach of the community. These are the things that you’re gonna want to talk about early on in the life cycle of a community as you are launching, these are the things that you wanna focus on.

And as your community starts to grow a little bit, you wanna look at what I will call the vibrancy metrics. We had a really great conversation during the break with the yellow team, so we were talking a little bit about when your community is in the growth phase, what are the things that you look for? And we talked about vibrancy, how do you tell is a community is vibrant? And so, one of the things we looked at was, the percentage of active vs. contributing members, right? The people that are coming to your site to consume information or to ask the questions vs. those coming to your site that are actively contributing information to the site, right? Are they answering questions? Are they contributing articles? How are they helping? So as your site continues to grow, these are some practical things that you wanna look at because it isn’t indicator of vibrancy and engagement within your community. So developers really want to come and find something out. If you have a sweet new API, or a hot product that they’re using internally, they want to come and they want to sometimes solve a problem or figure out how to implement something themselves. And by making sure that, you know it’s not just your staff but it’s other people coming in and contributing, you can really drive up the level of engagement as you look for those champions and advocates within the community.

Measuring satisfaction

We talked a little bit about customer satisfaction or CSAT this morning as well. So as you go through, if you’re using this for support, you can ask on a per-question basis as you answered the question, “How did we do? How are we doing in supporting you, our developers? How are we doing ourselves?” And take those measurements back and show, “Hey, boss man, here is the customer satisfaction score that’s going up, proving a return on that investment as they’re putting it to you, your team, and the software you may have purchased.” These are all things that they’re gonna be looking for to prove value to the company through the community, right?

Last one, time to first response and then time to solution. So time to first response is a really good metric, right? It indicates, you know how fast your community’s moving, and we’ve stood up some communities, time to first responses less than a minute on some of these things, super active, super engaged communities like SAP. And we’ll talk a little bit about SAP as a use case here after the slides. But one of the reasons they drive that so high, is through high quality content and high quality answers coming from both their staff those contributing members within the community. Because there’s something about developer to developer knowledge sharing that’s almost magic. When we build developer communities, we like to…we have a saying and it’s actually on the top of our internal community, it says, you know, “Knowledge sharing and building these communities is really 90% culture, it’s 5% tools, and it’s 5% magic.” And you guy, each one of you guys in here, developer relations, community managers, you guys are the ones that have to figure out the magic. What is the magic? What is that thing that makes your community unique? What is the thing that makes your project unique, that value proposition to your developer? What’s gonna keep them coming back? All of those are things that you need to be thinking through.

Community maturity

Then we go into maturity, right? So you’ve launched to your community, it’s growing, and now you’re moving into maturity. We had a really nice curve on one of the other presentations earlier this morning, and as you get into this mature thing, you’re gonna wanna start looking at how this maps back into your business processes, right? How are you doing on your support savings? We talked a little bit about case deflection. I think it was in Joshua’s speech just before this, right? How many cases are you deflecting? And if you can pick how many cases you’re deflecting and the average number of cases that a support engineer can handle, you can actually get a head count number and go to your boss and say that, “My community that I am running is saving you two head count in support per year.” That’s pretty powerful, how much does a support engineer cost? Multiply that number by two. That is a real dollar value that you can take back to the business and say, “Hey, this is what I’m saving you, just for your support savings.” This is how these business guys are gonna think and how they’re gonna see return.

Referrals through advocates, you know we’re gonna talk about the developer life cycle here in a minute, and when you get a developer in your community to the advocacy stage where they are bringing other people in. They’re saying, “Hey, this site is really cool.” or “This product is really great.” You’ve seen it a few times in the presentations here where one of the speakers would say, “This product, this product is really cool.” We just had one then with Josh? Yeah. It was really neat to see that because now he is an advocate. And as those people come in, those referrals, that is real return on your investment and those are some of the metrics.

So, I wanna get into a use case here really quick. We’ll talk a little bit about Hortonworks. They have a…how many of you are familiar with Hortonworks? Big data, data stuff? Awesome, love Hortonworks. So they have a… they have built a really interesting website using AnswerHub, that people can come and ask questions and kind to get support, and they track their top metrics here. Unique visitors, active Horton workers by department, the number of contributors, internal vs. external, or staff vs. the community and that’s a really, really interesting metric, right? They also do top contributors by quarter and number of pieces of content. And to show you and example, this is not real data, it’s completely made up as a data guide, kinda hurts my feelings but I feel like I’ve got to tell you that right out the gate. Here’s what a users vs. staff kinda chart would look like. So if you take a look here, kind of a relatively young or new community, getting in the double digits here a little bit and this is…whoop, where did you go…ha, there you go. So you got…this is over time, September through August and then you get answers to questions or posts, users vs. staff. And in this case, you can see that the users are participating but in most cases the staff is doing the lot.

So, what does this mean? This means that you have a high support load, right? So if you’re gonna scale this model, what do you need? You need more people, right? However, what you want it to do, through the programs that you create and by working with your community, is a chart that looks a little more like this. This is what you’re going for, guys. This is real return on investment at the business level for a support community saying, “Hey, out of all the hundreds of questions that were asked in September, 390 or 389 of them were answered by people already within the community, so your support team didn’t even need to get engaged.” What this means is, you’ve started that fusion reactor. You’ve started that community that is going to build on itself and it will grow. And as your product grows and you know continues to reach more and more people that this community will start to sustain itself, and that’s what you’re really looking for.

So, another example, I mentioned Hybris. Sylvia, a good friend of mine, you now uses this. Their answers community is really pretty active. They’ve really reduced call volume and sign up about 600 new users every single month. And so, they’ve got some of their top metrics up here, I’ll let you read those. But we talked a little bit about how to gain quality metrics of the content that’s posted within a community, right? As you have your community launched and grown, as you get people contributing, you have to have high quality content. Developers really, really want high quality content. Things that help them out, that answer their question the first time. So, you know Sylvia down here, she notices a trend and downvotes, right? This is an indicator that quality is not what it should be and so they’ll go through and they’ll moderate and they’ll curate the content, right? So, if you’ve got your moderators, your curation team, or of that’s you yourself, this is a way that a metric that you can use to go back and say, “I might not be meeting the community’s needs.” and to address it before it becomes a problem. So, you know these are just a couple of examples of some real-world things that you can do.

And I wanna try to turn to finding value in the developer life cycle. So as a developer, self proclaimed developer, I even wear the shirt and I have a couple for you other developers if you would like some in the back, but as a developer comes into your site, the first thing they’re gonna do is, they’re just gonna kinda poke at it a little bit. They’re gonna be in discovery mode, right? I’m trying to find out if I can trust you as a reliable source of information about a product or a problem that I am interested in solving, right? And so, the numbers of page views, the number of sign ups, and any types of referrals are things that you’re gonna want to look for as someone new is visiting your site, a new unique visitor. Then, once you’ve kinda got a hook, you want to start figuring out ways to make sure these developers come back. How do you onboard them? You know, how do I get on board with your product or your offering, right? How often am I engaging with you, right? The number of downloads that I do of your site, if you have white papers or you have, you know a security brief or something like that.

Indicators of engagement

And then, the number of questions that this person asks on your community is an indicator of engagement, right? Sometimes we focus on those indicators of engagement earlier on in the cycle and then lose it, but these things build on themselves, you’re always gonna be looking at the number of page views per unique user. Always, always, always, and then whenever you get into the support phase, this is where you’re starting to drive the business values, so the later in the cycle you get, the more business value you get out of someone. So moving them through these earlier phases faster actually delivers that value back to the business at a higher rate. So case deflection, response times, and then correct answers. So you can tell if someone’s enjoying your site, again as we go to some more of these softer metrics, which are very, very hard to measure, you can look at these very tangible metrics to figure out of the intangible metrics matter, right?

Measuring developer happiness

So how do you measure developer happiness? It’s very hard. It’s very, very hard. But what we can do is we can measure the response time and case deflection and did this answer my question, right? Thumbs up or thumbs down. That is still a measure of sentiment in some case. And that’s one of the things that you want to look at, so don’t make that mistake. One of the common pitfalls is, you’ll go through phases of analytics, and I’ve seen this before, and you don’t want to do that. You don’t wanna look at this group of analytics this month and then this group of analytics this month because you’re gonna end up chasing your tail. And sometimes, you know as Jonah was saying earlier, is you can get lost in all the different dashboards you can create. Pick the things that are very valuable to you that show that you’re meeting your goals and focus on those things to drive value within the business.

The last phase is the advocate. How many of you have advocates for your community you could name to me right now? Fantastic. Everybody who has an advocate knows them by name. Because they’re important to you, they’re almost like part of your team, right? They are actually an extension of your team helping you grow your community. They’re using the product, they are loving it, they’re giving you feedback, your product management team knows them because they’ve suggested feature X, Y, or Z. So you wanna measure the number of interactions and you wanna reward those people to continue to get them to be advocates for your product. So as they share things to your site, you know one of the best practices that we were talking about earlier is sharing swag and letting them know that you care, all of these things are really important. But when it comes down to it, when someone says, “Show me the money”, right? How are you delivering ROI? Focus on these types of things later in the cycle to really drive home the value in a numeric or an economic level, but then, you know you’ll see that magic piece as the community starts to grow and it will be become apparent, right?

So I wanna, you know kind end here, I know my time is short, I’ve got like a minute or two, but talk a little bit about DZone. DZone’s are pretty big community, right? It gets about five million page views a month, 10,000 new users and we, you know we publish a lot of things, right? “The Guide to Big Data”, “Guide to IOT”, a lot of Refcards, but one of the most interesting things is, one of the reasons that DZone grew so fast is because we gave everything away for free. DZone was founded by believing that sharing knowledge makes life better. Developers want to share knowledge. Even when they’re competing with one another, they’ll still sit down and argue about, you know Java vs. C#. Always, and the merits of each, and the best way to formulate this algorithm because at their heart, developers really just want the world’s problems to get better, you know? And so by providing a site that gives away all of it’s, you know of its assets and things for free to the developers, it allows us to help grow a community. And that’s one of the things that, you know we’ve got to think about in delivering value is giving them something for coming, meet their need where they are, gain their trust and they’ll keep coming back for more.

So if you have any questions, my time is out, but I’ll put up of some contact information up here, I’ll be around for the rest of the day. And I would love to talk to you about your communities, about DZone or any of the other communities, or anything else that you need. I’ve got a Wolf Pack fan up here, so I gotta give it back to you, for those of you who don’t know. Oh yeah, Adam too, fantastic. So anyway, as you guys go through the day, you’ll hear a lot of similarities between these talks on analytics and they say that a human sometimes needs to hear something seven times before it sets in. But I hope that you’re seeing a common theme throughout all the different speakers on ways to deliver value back to your communities and grow them because we want to see them all successful. Thank you so much.