Made by
Made by
Steve Pousty
Engaging in a discussion of ROI will never end well for most dev rel activities. DigitalGlobe’s Steve Pousty explains why ROI is a trap and gives some ways to steer the discussion to a more useful focus. Steve was speaking at DevRelCon London 2018.
This is the video link for the talk if you want to get to the slides, it’s also will be at the end of the show so you don’t have to get, you know, put all that pressure on yourself to say, “Should I take a picture now?” You can be like, “Oh, that talk totally sucked. I’m not wasting the digital bits for that.” It’ll be on the last slide as well.
This is me, that’s a zero because I’m l33t like that. And the talk is about ROI is a trap. Everybody know what I mean when I say ROI or return on investment?
Audience member: We don’t know what the trap means.
Steve: You don’t know what trap means? Yeah, okay. That was a trap right there.
So context for this. I’ve been doing this for about 14 years, developer relations and I’ve worked at companies of all sorts of sizes. And I have heard this discussion a lot. How many of you have actually even experienced the, “What’s the ROI on something?”
A bunch of you have, yeah. And then I’m going to talk about that today. So let’s get going. The other thing is, and this is no judging whatsoever. There is no judgment here. On a scale of passive-aggressive to aggressive, both are not good terms.
So I’m not judging. I use negative terms for both. Here we could say the Bay Area is passive-aggressive and New York City is aggressive. Is that a fair approximation for most people? Yeah. Okay, so here’s where I lie. Do you want me to sing you a song of my people?
Audience member: Yeah.
Steve: Okay, it’s a participatory song. So if you know it, sing along. ♪ We’ll give it a shot, Ohh, halfway there ♪- [Together] ♪ Oh-oh living on a prayer ♪- ♪ Take my hand and we’ll make it I swear ♪- ♪ Oh-oh living on a prayer ♪- Thank you, Chloe take that and put in your pipe.
I can do show tunes too. So that’s where I’m from and that’s the song of my people. And, this is where I live. I live in a very passive-aggressive, I live in Santa Cruz, which is part of the Bay Area for those of you who don’t know. It’s basically San Francisco and it is one of the…for me, it is torture.
Like, every time I go back to New York or I go to any Mediterranean country, I’m like, “Oh, my people.” And so, I get…if you’ve worked in the Bay Area which a bunch of you probably have, or at least have colleagues in the Bay Area, you’ve experienced this passive-aggressiveness. And so this is going to be part of what I’m going to bring up in the talk later. No judging though, it’s just different cultural manners. So let’s set up the scenario. You and your team come up with some, you know you should do an event, OSCON, DevRelCon, you know you should do the event, everybody on your team has agreed you should do the event, or you come up with some great schwag idea.
You’re like, “This is fucking awesome, developers are going to love this. Oh, God, we are the most brilliant team in the entire universe.” And then, you know you’re going to have to go to marketing, or brand, or some other corporate group, not in the DevRel group. And then you steel yourself for the imminent death of your idea.
You know that you’re going to end up in this place at the end of the talk. This is just where, but you’re like, “I will live the dream. I’m going…” You’ve got to try. So you get to the meeting and then you hear, “Yeah. So what’s the ROI on that event?Or what’s the ROI on that item?” And you know what’s happening, right? It’s not going to happen anymore.
That’s the way that they’re going to kill it. And so you mount a valiant effort to maybe discuss the ROI. And I say, “No, it’s a trap.”
So let’s do the aggressive, 30-year-old response that I want to give when I hear it’s a trap. Do you guys want to hear that? Now, the other song of my people, there’s two dominant accents from where I come from.
One is, “Yo, I’m going to fucking talk to you about this idea.” And then the other one is, “Let’s go to the court. Let’s go to the mall and have a cup of coffee.” Now which one do you want? Do you want the Italian or the Jewish accent? Those are the two accents I was just doing.
Audience member: Italian.
Steve:The Italian. Yeah, it’s easier to get into the, I’m upset one. Thank you for picking that one. Fucking really, you think that’s what we need to talk about. What is the fucking ROI on all the MacBooks the engineers are using? Did you do an ROI calculation on that? The VP’s and the CEOs flying business class, what’s the ROI for them?
I’m sure you did an extensive calculation. The president club for sales. Oh, really? They get to go to the Caribbean and just hang around with their wife because they did good in sales. But how much of the engineers actually did drive up value? What’s the ROI on that?
And this one hits a little close to home. I’m glad Chloe’s not here to hear this one because I know her fascination with, and it’s not LaCroy, it’s LaCroix. I took French in high school, I took French in college. It’s LaCroix, not LaCroy, people. So this next, snacks in the LaCroix, what’s the ROI on that? I know we all love it, but they didn’t calculate ROI on that.
None of these things do they actually calculate ROI on but somehow when it comes to your one event that’s maybe $5,000, $7,000, suddenly ROI’s a huge, big deal. And so I’ll give you another way to… And then I’m like, “Yeah, see, I just schooled you.” And then the rest of the room goes, “Oh, shit, he’s going to get fired.”
So, I’ve developed another technique to use is your possible mature response, shall we say. And that would be, now I’ll do my California voice. “Yeah, that’s really interesting. Let’s talk about how we approach this ROI in a collaborative manner. Let’s get some working groups to look at ROI for how the PR firm you paid to do those articles and papers. Let’s look at the ROI of that. Maybe we can table some of the other discussions and put them up on the whiteboard and then that T-shirt that you produced in marketing that was all covered in words that nobody wants to…some people might not find they want to wear.”
So I almost went to them, “Nobody wants to fucking wear that thing they produce, that you paid a bunch of money for. And oh, how about all that money you spent on marketing experts that came up with a non-developer friendly slogan that you now want to plaster over everything. That means nothing to developers, let’s do the ROI on those calculations first, and then once we figure out what methodology we should use, then we can approach ROI on the events.”
So that’s the more mature passive-aggressive response. I actually believe that anytime someone in general brings up ROI with you, it is a passive-aggressive, “I don’t agree.” And why I call it passive aggressive is, because instead of just saying, “That’s a fucking dumb idea, why would we do that show?” We get the, “Oh, well, let’s calculate the ROI.” And then what can you do with that?
Nothing, it’s a limp rope and you can’t come back and there’s no intelligent discussion that you can actually have about the event. And there’s a bigger problem with the ROI. It’s a trick question. Asking about ROI is a trick question if you start to engage in it, so don’t go down the path, because you’re now justifying numbers that you don’t even believe are good things.
You suddenly have gone from, “I really think we should do this event. It’s an important event,” to “Well, if we scan badges, ba, ba, ba, ba, ba, ba, ba, ba, ba,” and scanning badges may work for someone, no offence is intended. But you’re now having a discussion about things you don’t think are right, because you’ve been asked to come up with some sort of number, which is bullshit in this case. So, now that we’ve served the idea about like, that’s been totally served, we’re not going to actually discuss ROI anymore, that doesn’t get us out of the… We’re not off the hook.
We can’t just say, “Well, I want to do this event because I think it’s a great event and you should send me to London to hang out by Westminster and drink beers with all my friends because it’s a great event.” That’s not going to work. It’s going to take trust and good intentions on everyone part, and this is one of the things I think we forget.
You have to build trust beforehand. You can’t have these discussions without some sort of trust in the room. I mean, you can, they just generally tend not to go very well. And the problem is that usually what I find is that the group that’s saying the ROI thing, fears you’re going to do something that’s going to mess their life up, especially marketing and brand.
They’re very fearful of us because we could go off-brand, and I think it’s really important to establish the trust. Without that, you can’t have real rational discussions. You can keep talking behind each other’s back and figuring out other ways to get budget. But if you want to solve it, you’re going to have to identify trust. So that’s easy. You can do that, no problem. And then… Oh, I forgot to bring up my little thing.
I had a whole other, I had a little whiteboard that said, “tangent” on it. For those of you who’ve seen me speak before, it’s very easy for me to speak. Matt is evil for giving us a 20-minute spot, that’s like the hardest speaking slot to do, is like 20 minutes. So that was a tangent right there about building trust. I just want to remind us of three goals as developer advocates or in the DevRel community, one, make developers excited and successful on our product or platform.
Would everybody agree, that’s basically our reason that we are around. And developers, that could be data scientists, that could be operator, like systems operators, it could be whatever, but technical people. Two, grow our company’s likability, visibility, thought leadership our je ne sais quoi, we’re trying to make our company look good. That’s part of why I’m speaking here supposedly or I might go speak at another industry event or be on a panel or something.
And then the third is to grow our own personal brand. And that’s good for us, but it is also good for our company. Growing our brand is important. Don’t worry that… You have the link to the slides, later, you don’t have to… I mean, if you’re tweeting it, fine. Okay, good, so then my talk is a success even without stalling in it. So, use these to guide all the discussions. That’s the rational framework I want us to use.
What goal are we trying to solve and then how does this thing fit into that goal? So like me coming here, I should not say this is about getting users excited about our platform. That would be a dumb one because have I even talked about my company? Not really, but it is building my personal brand, and maybe by association, the brand, my company’s brand because you saw it in the beginning.
Warning, not everything of value can be measured. And I’m saying that as a statistician. I have a Ph.D. in Ecology and basically a master’s in Statistics. So I love measuring things and doing inference on them and like measuring my house and measuring how many birds are coming to my feeder all the time.
I love measuring things, but not everything of value can be measured. And if someone challenges you on that say, “Okay, quantify how much you love your kids. And what’s the value on how much you love that kid versus that kid? Is that a 67 versus a 68?” Because really, we’ve got to get it nailed down. We’ve got to nail that shit down. And so there’s a whole bunch of things… Or trust, quantify trust, you can’t. But we do know these things are important. And we still work to build them because they’re important in our lives.
So to tell someone to come to you and say, “Well, you have to come up with a metric, otherwise we can’t do it.” Sometimes I think we tend to, under trying to quantify things because it’s hard for a lot of us, but on the other hand, doesn’t mean everything has to be to say that, “Oh, it’s worth it or a value.”
It’s extremely difficult in our field to get accurate numbers. And here’s, I gave at least three reasons. I’m sure you can come up with some of your own. We have a very suspicious audience, especially if you work in continental Europe. How many of you work in continental Europe? It is almost impossible to get them to sign up for things. You say, “I just want your email so I can give you free servers.” “Really? What are you going to do with that email? How often are you going to email me?” It’s very hard, it’s hard for us to engage, that’s why they’re even using us rather than regular marketing people.
Like why did dev rel come into existence? Because marketers couldn’t actually reach developers. So they asked…there’s a whole field around it. It’s a really hard audience. We have little control over our user behaviour. We need them more than they need us. It’s not one of these things like the iPod, you’re not selling an iPod.
We need that and we can’t really say, “Well, if you want to be cool,” we’re trying to create things that they do it. The other one, and I think this is important from a numbers perspective, is most of our outcomes are not direct. If I go and talk at a show for the show reasons it’s very hard for me to associate that, despite what Baruch says, it’s very hard in most cases to associate that user with that show and then track them down the funnel.
Baruch: Now you know how to do it.
Steve: We can talk about, I’ve done mark and recapture studies, we can talk about your methodology later.
But to really get real numbers, it’s hard. It’s really hard, and you brought up some of the reasons why it’s hard even at a show. People don’t want to go to a link and then there goes your number. Or there was another good talk apparently during mine, that everybody else felt they needed to see.
So, I think these are some of the things you can bring up as real justifications, but that doesn’t still get us off the hook. One other thing I want to keep up is remember to keep the phase of your company or your product in mind because you may be a product within a larger company.
There are different metrics you want at different phases of your product or company lifecycle. So there was… Five minutes, no problem.
We got a… No, I fucking got it, we’re good. There was one partner company when we launched a new product, not DigitalGlobe, another company I worked for, and someone started ragging on another developer evangelist team because they’re like, “All they’re doing is getting sign-ups.” And I was like, “Well, actually, that makes a lot of sense.” Like they’re going to boots and really just trying to get as many sign-ups as they could. And I was like, “Well, that actually makes sense because of where they are in the market.”
They’re trying, they’re behind, way behind and all they’re trying to do right now is raise awareness. So getting sign-ups and nothing else coming from it, that is perfectly fine. So they can have bragging numbers, so they can seem to be a competitor, like that’s fine. It just depends on where you are in the lifecycle, so don’t judge any metric. Think about it in relative to where you are. More mature companies would not want that. They wouldn’t want someone further down the funnel.
Things you can offer as a way forward, open discussion about return. This is what I was talking about before, what does return and return on investment actually mean? Is it the money, numbers of developers signing up, number of people doing X? Maybe it’s something else that we haven’t thought of. Comparison with peers. I actually think this is quite effective.
We have to be at Monki Gras. We have to have some presence at Monki Gras because so and so and so and so are there and if we’re not there, then it’s making a statement about us. Like that Oracle Code. Well, I think it’s called Oracle, back when it was JavaOne, if one of the big Java players didn’t show up at JavaOne, it was like, “Oh, fuck, are they going out of business? Have they given up on…”
Like you had to show up and that was because your peers were doing it and that was a justification. Comparison to year over year. So we may not be able to get good, accurate, numbers in that year, but if your company is lucky enough to stay around long enough, you can actually watch trends and decide on events and schwag items based on what’s happening. Like an example that probably most of us have dealt with was, what’s the big one in Portland from O’Reilly?
Why I can’t think of it. OSCON, like that, if you watched discussions in the channel on the Slack channel, a lot of people were really down on OSCON for a while. We were huge supporters and then it just kind of went down and we chose not to do it anymore but that was based on a shared group of values and what it was providing for us. Now we’re back to maybe doing it again. And then, reports, you owe your company a report if you go to an event, kind of, at least to somebody saying, “Yeah, that was a good event for us. That wasn’t a good event for us. Here’s three reasons why,” even just something like that so that you can make a good decision the next year.
Selective badge scanning, I don’t think badge scanning is always bad. He had an example before but one other reason could be, at companies I’ve been, we badge scan for people who want to be contacted. They opt-in to be scanned. They want to talk about something. Insert your creative ideas here.
If you do find some agreement on metrics, this is just a cautionary note, be aware of optimizing to the metric. Does everybody know what that means? Just because that’s the new number. Suddenly, that’s all you’re focusing on when it’s not actually the most important thing.
You’ve replaced your actual goal with the metric. The metric is never the goal. The metric is the reflection of how you’re doing towards your goal, should keep checking in on that. So, bringing it all home, avoid the term ROI because it’s nonsense with what we do. Talk about value. Be open and honest.
Know this is hard for all of us so don’t feel bad. That’s the other thing that’s an inspirational message to all of you. It’s just, if you notice a lot, if you talk to your peers, every one of them will say, “This is really fucking hard.We are having a really hard time getting a handle on how we measure that event, or how we do this.” I think this is something as a field we can mature on. I don’t actually think it will necessarily get all that much better. And then the last is learn and adapt. Don’t hold onto things too tightly.
My plea to event organisers
And with that, theme. So, again, there’s my Twitter handle. Now you can decide whether you want to take a picture or not, whether the slides were worth it, and then this, I put some stuff up in GitHub.
Oh, any of you who do event organization, you will go to this list because I’ve started compiling a list of religious holidays. I’m tired of not being able to attend event because you schedule an event during the High Holiest Days of my religion’s event and you don’t look.
It’s not that hard, and there’s not that many religions in the world. I mean, there are, but not that many major ones. There’s a ton, but for most of us, we can handle maybe three, five, somewhere between three and five. It’s not that hard to look at the calendar. “Oh, but I will never be able to find a date.” That’s bullshit. That’s like taking one of those things out of your butt and saying, “Oh, mimimimmiii!” Actually look on a calendar, it’s not that hard, okay.
Please go there and contribute if you want to put pull requests in for other dates we should do. So, I’m done, thank you.