The grand unifying dev rel theory

Steve: All right, so let's get started. My name is Steve. That's where you can find me, that's a zero on the end 'cause I'm leet like that. I work at Crunchy Data. We're Postgres people. I'm gonna talk today, I did a little bait and switch, but I figured this audience would be the most forgiving of oh my gosh, I said I was gonna talk about this, I started putting the talk together. I'm really not happy with that, but I can stay somewhat close to it, but it needs a better title. Can I get forgiveness on that from this audience? Yeah, okay.

Moving on, the main point of the whole talk is we keep talking about metrics, and I don't even think we have a decent model for why we're collecting metrics or what we're going to do with them. I've been at this for about 15 years now, developer relations, and we'll get to the main point of that at the end. Let's get right to it. What are these? Tools, nice job. What's a tool? Tools are things, something that helps you in a particular activity. I'm cheap, so I didn't pay for the expensive British university I guess, the OED Dictionary. Instead, I figured I got the Cambridge one, that must be the lesser one down the street. Are there any Brits, is there really a difference between Oxford and Cambridge?

Did you go to Oxford? Yeah, I thought so. Is that like a Harvard versus Yale thing? Is someone in the back giving him a finger that went to Cambridge? Is that's what's going on over there? That's the Cambridge dictionary, right? I figured that would be good, 'cause at least it's from the right continent. So, what is that?

[Audience Member]: It's a word.

It's a word, but what's a word? It's also a tool. Words are actually tools, because I put some lines on the screen, and every single one of you in here now has a mental picture of some dog. I didn't have to show you my two beautiful dogs. Right, exactly. My girls, but I put that up there to say, look, a word is a tool, because I could put up squiggly lines on a page or say the word dogs to you and you immediately get the picture of what we're communicating about. It's a particular type of tool that also fits in with this. A map and the equation for a line. What kind of tools are these, all three of them? They're models, right? They're representations that aren't the actual thing, but helps us get our work done when we're talking about stuff. So models are a type of tool. All models are tools, not all tools are models, okay?

And then what I'm gonna get to with this is a peculiar dilemma about models that we are afflicted by as humans. Fact, our brains are constantly making and using models. All the time, even while you're listening to me speak, you're making models. When you walked over here this morning, you had models processing through your brain of all the people in front of you, right? Where are they moving, I don't want to collide with that person. You're not consciously thinking of the model, but your brain is churning through models all the time. Reality is too much for us.

Has everybody seen those videos about how we actual filter reality? Even what we think is reality is actually a model of reality that our brain can comprehend. You've seen that video where they're all like, did you see the gorilla come through the room? Someone's giving a speech, you're watching the speech, you didn't see the gorilla, then they show you the rewind of the video and there was a gorilla in the room. It was actually really a person in a gorilla suit, but still, or there was a banana on the wall that the gorilla ate, but still there was a model that we used that didn't allow us to perceive that gorilla in the room because we were busy focusing on the talk. We actually filtered reality so that our brain could comprehend it.

So that gets us to, George Box is a British statistician. He says, all models are wrong, but some are useful. So here is the dilemma. We're always making models, and all the models we make are by definition wrong. A map is wrong, right? There are some parts of it that are useful. You've simplified the world to make it so that certain things are not true, but other things that were useful to you are true on the map. Your models are always wrong, kind of like there was no gorilla in the room and there was a gorilla in the room. Models also constrain your world view.

If we go back to language as an example, I'll say it this way. Did you know, more cocktail party trivia coming throughout the rest of the talk, but did you know that the Ancient Greeks had no word for blue? The word blue did not exist in their language, so if you asked them what the sky looked like, there was no color to sky. It was just sky. Russians, in their native language, have two words for blue. Because of that, they actually can perceive differences in blue shades faster than non-Russian speakers. If you want to hear the whole thing on that, that was on a Radiolab podcast, where they talk about words and perceptions and the functioning of the mind. The constraints are usual, at least for me growing up as a Gen Xer, constraints are bad. How dare you constrain me? But actually, it turns out that in some ways, constraints are good in that they allow us to do things maybe faster or communicate things better. If we had no constraints in language, how well would that work? It wouldn't, right? Part of what makes language work is that we constrain what we're gonna talk about and what words mean so that we can share communication.

So we get all models are wrong and models constrain how we think about the world. What's a model that we use a lot in developer relations? Personas. We just saw one before with Aisha, the persona of the user. That's a definite model. Does that person actually exist that she put up on the screen? That's not a hard question, it's really not. I know you're all introverts. You can turn it on for a second, right? Turn it on for a second and just answer the question, did that person really exist, or when we do personas, is the person usually some exact person? Do they usually exist? No, it's a model of who that type of programmer or developer is, right? So, by definition, is that model correct? No. When you say a Java developer, then you've got some model in your head, but then you go to interview a Java developer like Baruth, and some parts of it are correct and some parts of it are not. That model was useful, but it also constrains what we think that person can do. So you say, okay fine, models suck, I'm gonna opt out. That's it, but you can't. I wore the belt in honor. If you can't see it, this goes out to all the Canadians in the audience as well. Okay, do you want to me to sing it? I sang last year at this conference, do you want me to sing it again?

[Audience Members]: Yes.

Steve: All right. If you choose not to decide, you still have made a choice. Thank you, thank you. That's Free Will by Rush. Here they're talking about free will, but here I'm talking about models. You can't choose not to make a model. All that you chose by not making a model is that you're gonna use a crappy model or you're just gonna use some default model. You don't have to make personas, but then what that means is you're just gonna treat your developers as an unwashed mass, and that's not really good, either. You say, thanks a lot Steve, I can't opt out. No matter what, I'm making models, so what's your point?

So my point is, I would like us as a profession to bring attention and intention to the models that we choose. We're always gonna be making models. Let's actually get better about picking our models and be intentional about which model we're using in which situation.

Let's look at one of the models that we use a lot, and this is probably the most common model we talk about today in DevRel. What's that? Apologies to Phil. I know that someone put in the AAARRRP into there. That doesn't require the AAARRRP stuff in the diagram, but what kind of model is that? Funnel, exactly. It's a funnel model. It's probably the most common one that we talk about. We borrowed it from social media SEO stuff, and I think it actually is not a great model for what we do. I think some parts of it are nice. It gets us to think about transitions between different states, but the problem is it constrains our view to be a linear flow for developers.

Remember how I said models constrain? When you think about a funnel model, it imposes on that view. The developer starts with seeing you at a show. From there, she moves on to trying out your API. From there, she builds a demo app. From there, and that's the only way you can conceive of how your activities fit together. I think that actually imposes a false narrative on the way that we actually interact with a lot of our developers and how they become our customers, or whatever you want to call it. The other problem with the funnel model is it's actually not quantitative. You can't parameterize it, making it very hard to test your hypothesis in it. There's no hypothesis testing in it, as far as I can tell. So, let me tell you a little story. In a former life, I was an ecologist. For my dissertation work, I studied this beautiful animal. This is Hemilepistus reaumuri. It is found in the Negev deserts and the deserts of Asia. What do you call it in England?

[Audience Member] A bug.

Steve: Okay, good. It's not exactly a bug, it's not an insect. I would imagine some of you call it a sow bug. Has anyone heard of a sow bug in England? Pill bug? Okay, a roly poly? Woodlice, yes, woodlice. Having had kids, any time you say the word lice, immediate hard cancel.

More cocktail party trivia. Do you want to know how to say this in Hebrew? Uri is a boy's name or a man's name. Kadur is ball, so they're called Uri Kaduri, just to let you know. So I was interested in this organism for a whole bunch of reasons in the desert of Israel. I'm telling you, you go up to somebody in a cocktail party and you're like, hey, do you want to know how to say woodlice in Hebrew? Instant in. Hitting lice and all together. I was studying this as part of my dissertation, and they live in family groups.

This individual here, this is where you go oh, that's so cute. This is the one of the parents. There's actually a burrow inside of here, and these are all the offspring, and this is all their feces that they pile up around their burrow. Okay, calm down about feces. We're all adults here, and it's just little pieces of mud. Mary's like, I gotta go, triggered. They all live like this. They have a lot of really interesting characteristics for ecological study.

The one that I was interested in is what influences where they make their burrows? Basically, they're born, they form a pair, well actually, one finds a burrow, they make a pair, they have babies, they live there for the year, the parents die in August, and then everybody leaves the next year. And I was like, what makes them pick where they want to make their burrows? So I said, okay, let's go ahead and use our funnel model to figure out what they're doing. So how would we do this using the funnel model? I just want to show how this model can constrain things. You find a lot of individual isopods, you follow them around a bit, you watch them die or form a burrow, you then follow that burrow until everybody leaves and all along you send them Net Promoter surveys. How are you feeling about this burrow location, would you recommend this burrow location to other isopods, are you happy with your burrow location, all that kind of stuff. That actually doesn't work very well.

Here's how we actually solved it, and I'm putting up an equation, so for mathophobes, just breathe, okay? It's just addition is all we're gonna do here. I'm gonna say the probability of settling is equal to some number times the amount of shrubs, plus some number towards the amount of rocks, plus error due to my measurement, plus error due to stuff I didn't think about that's unexplained. Does everybody recognize that as a regression? Anybody who took basic stats, that should just look like a regression model to you. This is basic regression. The way I actually estimated is way more than basic regression, but I'm making it really easy here. The point of how I would say this in real life is the probability that an isopod selects a spot on the ground is some linear relationship between the amount of rocks there, the amount of shrubs there, and then there's some error that's messing up my estimate of the probability. That's basically what that's saying.

The fun parts here are, there's a whole field called statistics that has spent a lot of time how to estimate these numbers. Why is that important? Because I can say, well if I increase the amount of shrubs, I expect to see this much more probability of settling, as opposed to this is the transition from this state to this state, 10% are making it from our blog post to our signup. I can actually estimate stuff, and I'll go into this in a bit, on this.

Second, it is explicit that there is error. This helps us to avoid the very common, which we are not immune to, I put numbers into a computer and a number came out. It is truth, right? That is what most people, have you had that with any of your family or friends where you put something in a spreadsheet, a number comes out, and like oh, that's obviously the truth. We believe that number, it came out of a computer. It helps us get away from that, and finally, it's a way to conceptualize what's actually happening out there. No, not finally.

The final thing is I don't actually have to follow any individuals around, and this is what's important for us. I didn't do any following of individuals to estimate this equation. I looked at who was settling where, what was the characteristics that was in that place, and then I estimated the model. So back to our problem, what does this look like for us? What's our main goal? That's our main goal, right? As developer relations, our goal is to make making users successful and happy on our product, project, whatever our thing is.

So, what does our equation look like? The probability of success or happiness or whatever you want to say is equal to some number times awareness plus some number times user experience plus some number towards ease of adoption plus some error we can control like things that we do wrong plus stuff that we don't control, like sales making prices too expensive or something like that.

That's not under our control, but we can actually estimate something like this. The slides will be up. You don't have to really memorize this. You can also change that to the number of successful or happy users. It doesn't even have to be the probability. We could just say the number of successful or happy users. The point of this is, this one I don't think is very estimatable the way it's written write now. I don't think I could crank this out and estimate the numbers around this. I could with a lot of work. It would take a long time, there's hierarchical modelling techniques that we could do this with, but I think the bigger question you would say to me is like, one, the nice part, Steve, thanks, is that we've gotten away from following individuals. I think that's what a lot of us get hung up on. How do I track this individual through the flow, right? They started here and then I lost track of them, or I don't know how many times they saw my blog post or I don't know how many came to my talk. I don't have to worry about that so much anymore.

The next part is, we can actually do things like this where we can take this awareness piece and break that down into something like this. I can say awareness is some function of the number of blog posts I produce, the amount of money I spend on ads, the amount of social media effort I put in, some number of talks and whatever. This is where the AAARRRP, am I pronouncing that correctly, Phil? Thank you. That AAARRRP model can actually help, because if you look at Phil's AAARRRP matrix where he says things filter into awareness, become all of these different pieces of the equation.

We as a profession, if we got good around figuring out what this number meant, we as a profession could actually get good at looking at this, which is what we actually all really care about. Rather than saying, well, my transition was 10%, I still have no idea if the number of blog posts I produced, how many people in your dev rel program is, we need blog posts, you guys should be producing blog posts.

Why are you not producing more blog posts, right? That's what we hear all the time. Some of us like producing blog posts, so that's great. If you're like me and you hate it, then you're like, we can't have a rational argument about it because we can't actually estimate what that number looks like. Is it worth it to actually just spend more money on ads? Might we get more output if we just spent more money on ads and stop bugging dev rel? Maybe not, maybe it is the number of blog posts. The issues that are still left after this though, it's still hard to get the numbers. We still have an observability problem. It's not like with my isopod where I could go walk around the desert and go, there's a burrow, there's a burrow, take a photograph from an airplane and say there's the amount of shrubs, there's the amount of soil, there's the amount of rock. We can't do that same kind of thing, so we're still gonna have problems estimating some of the numbers. This equation doesn't help, or using this framework doesn't help.

A lot of success metrics like awareness, happiness, ease of use, those things, are ill-defined by us. If I asked everybody here, how do you measure awareness? I'd imagine I'd either get a lot of blank looks, or we'd get very disparate examples. This to me is a good place to put our efforts as a profession. I'll come back to where we should be doing as our profession in a little bit, I think. There are other models we could use as well. I'm just giving you an example of another model, kind of to free up some of the constraints you had, because all you're thinking is funnel maybe.

Other fields create models all the time, so we saw a model from product design early this morning from Aisha, and then we've seen the funnel model. There's a whole host of models, even if you just look in statistics. One example that we could probably use a lot is survivorship functions. This basically says what is the time to an event, or what's the probability of an event happening, given certain things happening along the way. This is what they use in drug trials all the time.

If you think of survivorship instead of for us, time until they adopt an API. Time until they use a certain function in your API. You could actually say, how long does it take for them to get to this next level in our API, and you can estimate it and then you can say when I change things, how does that change how long it takes?

This brings up the other problem that estimating these models will take careful thought on how we do our treatments. What I mean by this is suppose we wanted to estimate this model. You as yourself probably only have one population that you're dealing with, just your users that you have. You're gonna have to do one of two things, which is either pick a new market. Let's say you're mostly North American. You could say oh, we're moving into APAC. I'm gonna try something different in APAC to see if when I do something different with these, do I get a difference in the response rate? Or we're gonna have to get better at sharing statistics, sharing data along the way to estimate these models.

We can have a longer discussion about this later. I'm sure you're all dying to. I just want to point out that it's not gonna be easy necessarily to parametrize, but the point of those models necessarily wasn't to parameterize anyway. It was also just to get us to think about things differently. I'm gonna go back again.

The other part with this model that's nice is I'm no longer thinking about the funnel, and I'm thinking about things differently. This can actually be the some number of blog posts I produce over a certain time period. I'm not tracking individuals. I'm just looking at numbers of blog posts, right? And so I don't have to go, oh, I have to make sure that they make this transition to this place and I'm focusing on where they are. I can think actually in aggregates. So it may clear up our thinking or think of different ways to deal with our stuff. Let's bring it home.

So, the take homes: One, you are always making wrong models, always, which should actually be liberating, because everybody else is making wrong models as well, everybody is. There is no doubt, so when you're in an argument with somebody and they say something, there is something about their model that is wrong as well, unless you're dealing in pure math, or Boolean logic or something. Other than that, their model is probably wrong in one way or another.

So the question is, is the model that you're actually arguing about, is it useful or helpful to figure out what you want to figure out? We can have other ways of thinking about our efforts, and we should be mining other disciplines, looking for other models to help us think about what we want to do. Using analogies, doing all the other model extrapolations. As a profession, we need to start doing more formal work on this, I think.

Dev rel you could say is about 15, 20 years old right now. I think we've done quite well in establishing ourselves as fun. I think what we really need right now in our profession is more effort on professionalism. I think it's time to self-correct perhaps a little bit. If you look at the early days of software engineering, there were tons of arguments within the profession about what was the right way to evaluate productivity. There was a vigorous debate about it, and I think they actually came up with there is no right way to evaluate productivity, and that's not really the point. The point was, they had frameworks and models and they discussed it and they worked on it and figured that out, rather than just keep saying we're gonna use the funnel model and isn't that awesome, and then arguing about our funnel is better if we did this and we produce blog posts and we're getting better transitions or something.

I think we actually need to start thinking about ourselves as a formal profession, and I'd really like to discuss it more, but I'm not gonna do it today 'cause it's a third rail. And then, let's identify common metrics and how to measure them. I'd really like to see this happen, at least. What does awareness mean? Go through Phil's AAARRRP, thank you. Do you know the hard part for, I don't know how many other Americans feel this, but you know, in America we have the American Association for Retired People, the AARP, so every time I see it, I'm like, damn it, I'm not that old yet. That's the last step, thank you, where I get discounts and I eat lunch at three o'clock, and dinner at three o'clock.

The final part, and I know I talked about metrics and models, and I always bring this up 'cause I did it last year with the ROI thing, but we can justify our existence without numbers. I would like us to stop also navel gazing so much on numbers and justifying our existence around numbers. At the point at which Dev rel justifies its existence based on numbers, we've already lost. The person who made the really cool iPod advertisements, I bet you there was no discussion about numbers when they made that. You know which one I'm talking about, like the person dancing and there's all this, created this whole lifestyle feeling around the iPod and Apple and music. Look at the software engineers in your company. How many of them are arguing about what numbers, bugs per line count? Anybody actually have software engineers at their company taking that seriously? No, but do we say oh, that means we need to get rid of software engineering? The only group I've seen that's so focused on numbers because it's really easy for them because their life is all about numbers is sales, just sales. Everybody else, and SEO people, because it's easy for them.

Basically, we become focused on numbers where it's easy, and I don't think that works for us. As Mary's pointed out in her book, there's a lot of squishiness around what we do. We do need to talk about value and things we bring to the company, and we should work on formal models so that we get better at our discipline where we can in terms of numbers, but we bring value in bringing developers to our product or project, regardless of the numbers.

I think we could be a bit more professional in how we present ourselves to the outside world. This is within the family, so in the video, if you're not a dev rel person, please ignore this next part of the discussion. This is an in-family discussion here, but I do think it's come time for us to actually be a bit more overcorrect a little bit towards, dare I say it as I wear this shirt, stuffiness or professionalism or committees or blah, blah, blah, whatever you want to put around it and suffer that pain for a bit. We don't have an academic discipline to back us up, and usually that comes through oh, I studied this in school. We don't have that. Anyway, that's not about the model part. That's my thing I would like to talk to people about after, but what I'm trying to bring home here is I think there's other models we could use. I would like us to explore the other models. I'd actually like us to start parameterizing the models, and I think by doing this, we can actually get to better understandings of how we reach out to developers. Thank you.