Takeaways from 500 virtual events since the pandemic’s start

Philipp Krenn: Let's see what I can take away from 500 virtual events. And that's not just me. That's, like, the entire team at Elastic doing that. So I work for Elastic, the company behind Elasticsearch and a couple of other things. I'm one of the many developer advocates that we have.

So we have a a larger team, and we do a lot of talks. And I try to, like I I manage a lot of these, like or mangle with these statistics that we have, and I wanted to take a quantitative look like what can we take away from a lot of the events that we have done over the last twenty months or so. So why are we even looking into, like, what is the event covering and why is it doing that? So at last second, I think this is kind of, like, the right way to approach this. Our goal is reach.

It's not leads or revenue because if you do leads, you're more marketing. And if you're revenue based, you're more sales. And maybe there are some ways we can help them and kind of integrate with that to some degree. I had discussions about that where some people would say that I if you ignore the business, you will not be successful. That is true.

But our commitment is to the community. So what we want to have is a reach. So I always say the goal that we want to reach is a successful user, and then maybe they become the paying customer or a heavier user or an advocate kind of like in their own right for the products. So that's why we focus on reach and education, and that's what we want to get out to our user base. So what are we doing, or how do we approach this?

We have a global network of meetups, which up until twenty months ago was mostly in person. So I'm normally based in Vienna. I was running it from or I was running the Vienna Meetup Group, and we had every couple of months. We had an in person meetup, and we did that in lots and lots of cities worldwide. Yeah.

We also do external meetups. I think the the conceptual changes like internal meetups are for your own audience that you have already captured, and external meetups are more to gather new crowds and get the word out. Of course, there are conferences like lots of great virtual conferences like today. We also do things like lunch and learn sessions where we might present something as an overview at a company. We do student courses.

We do workshops. All of these kind of filter back into these many events that we have collected. So, working for Elastic and one of our products being Kibana, obviously, we throw all the statistics that we have in Kibana, and we have, like, a bit of a workflow there. So we track everything in Airtable, and then I have an application running in GitHub actions that is kind of like taking that data every night and combines it and prepares it. So we end up with a big dashboard like this.

And, actually, it turns out that I have to change the title for today because you can see we have crossed the 1,000 events. So the proper title for this event would now be takeaways from, like, a thousand events or so. So what can we see from those or learn from those events? Where do we run those? For our own meetups, for example, we use Bevy, which is also used by CNCF, MongoDB, Atlassian, Docker.

I'm not sure if meetup.com is dead. At least in the pandemic, I had the feeling it was very sleepy. I'm also not sure that Debbie is the the greatest tool and the way forward, but it seems to be what many are using today and what works pretty well in the virtual world. I know that some groups are also more or less active on Meetup in the virtual world, but I always had the feeling Meetup's big strength was the or these in person events. And it's kind of like the virtual world didn't transition so well for that.

Now where are we running those? We run a lot of these events in, the three geographic regions that we have. So Asia Pacific, The Americas and Europe and Middle East and Africa. And looking at those numbers, that's kind of like one of the first things that was very interesting looking at our own numbers was that EBITDA is much larger in terms of attendees than the other regions. And once you dive into the events, you will actually figure out that we had some very large events in China, like we had a single meetup, with one of the local cloud providers there, which had 20,000 attendees, which I'm afraid I will never manage in Europe.

We will probably never reach an one meter with 20,000 attendees. That's why the sum of attendees in APJ is much higher. Now coming from Europe, I was a bit like, okay. But why are we doing so much more poorly than The Americas? Like, I want to be at least as good as America or The Americas because it's north and south.

And when we started looking into those numbers, at least for us, and I'm not sure if that translates so well to everybody else, but since we have all the meetup groups, we looked into, like, the regions, like, what is working well and what is not working so well. And then when we started to look into those regions, basically, in Tibana, filtered out Brazil as one of the subregions that we target there. And it turns out that why is how The Americas having so many more attendees was because Brazil was doing or working so well for us. And it turns out that Brazil hasn't had that big infrastructure as we have, for example, in North America or India, where people are kind of like almost saturated or more already by meetups, and you were used to having it in your city and you we are used to getting pizza and beers every evening. In Brazil, it was like a new market where there was a lot of enthusiasm when we switched to virtual because suddenly you could get that event to your home all over Brazil, and we had a huge inflow of people joining our events from Brazil seeing the new opportunities there.

And it's just by adding that new market. Like, the the American numbers used to be, like you can see America is almost or 50%, 60%, 70% more than EMEA event attendance in in total. Once I take out Brazil, suddenly America is quite a bit behind EMEA. So that's kind of like the Brazil effect that we kind of figured out. And of course, then there is the the China effect in APJ that I would need to filter out, and then we're kind of, like, getting more even.

So regions are different, and you kind of need to embrace those differences and work with what you have and what makes sense for each region. This was one thing that was very interesting to see, like, how different regions behave virtually and in person because I always have the feeling like the Austrians and Germans, for example, are kind of lazy, they don't want to join another Zoom call because of Zoom fatigue, but they would probably go to a beer in the evening. So that's how you could draw them out first. Virtually, it's very hard to motivate them to come to your events. So that's kind of one thing that you just need to accept, like, what works for what medium.

The next thing is, like, where do we stream? We tried various things. So Bevy, that white label meter platform basically has its own built in stream, which, in my opinion, is pretty poor because the streaming quality is only seven twenty p, and it's very pixelated and sometimes cuts out things. So it's it was not a great experience, and we recorded some stuff there that we basically had to throw away. We did use Zoom, and it's nice for the interaction, but most people have their video turned off anyway, so you didn't get much interaction.

And then you always have this problem that people might be afraid of Zoom or maybe that's, again, the the German speaking region that is afraid of their data privacy, and so Zoom was a bit of an issue. And so we have, like, switched mostly to YouTube live streaming. You only have chat. Like, the attendees are not live, and we use StreamYard in the background to stream live stream on YouTube. But that works pretty well, and we have the recording on YouTube right away so everybody can access that.

And it turns out the live attendance is often, like, tenth of what we get in YouTube views within twenty four or forty eight hours of our events. And we have built a sizable YouTube channel over the last twenty months or so just by producing a lot of relevant content there over time. So in terms of YouTube views, you can see in the last twenty months or so with all our virtual talks, we have brought in more than half half a million of YouTube views on all videos, and we have tracked approximately 500 YouTube videos right now. This could be from external conferences, external meetups, or our own. Basically, how this is tied together, we have this Airtable sheet, which is glorious big spreadsheet.

We paste in the YouTube link, and then, like, the little script that is running every night is just extracting the views from every single video, and then combines that. And you can see there is one single video here. It is our most successful video that has, like, 90 or 95,000 views alone. And it's, like, presented as, like, when was the video added, and we we show the views on that video when it was added. So this is, for example, the most successful video.

You can see here we had, like, our own virtual conference, and these are all the talks we have from that conference. And you can kind of, like, see that blip, where the conference Elastic CC as as in CC email, has been running, and the views from that. And you can see that the average attendees, like, these are Chinese outliers. Basically, these are events with, like, 20,000 people or so and then average over two or three events, so that's why it's 10,000. But these are kind of, like, outliers, and most of these single events, like live attendance, at the event is much, much smaller, so in the range of, like, 50 people or so.

And then you could break that down on the specific event just to see how well one is working. Who and what is doing what? So we, for example, have the concept of a host for everything, and we have a presenter so we know what kind of content is popular or who is doing a lot of talks. We also extract, for example, all the the the words or tokens from each title so we can kind of, like, see who is talking about what and kind of, like, what are the main terms that we're using. And then you could also, like, increase the size based on the the viewers so you know kind of, like, what is popular and what is not popular.

So we can slice and dice a lot of the data that we have collected and combined. What I already teased it before, virtual talks are great because you have this scale and, like, the very limited cost factor and also accessibility. Anybody can join for pretty much nothing. On the other hand, what is the downside is that interaction for or with the attendees and everybody at the event is just totally different. Like, being at an in person event, you have a thirty minute talk with somebody that would just not have happened in the virtual world.

You have tried so many things, it just doesn't feel safe. Also, as a speaker, feedback at a virtual event is invaluable, and you really see what is going on. Like, if I have a live audience, I see who is falling asleep. I see who is on their phone, and I see who is kind of paying attention or maybe fully paying attention. Whereas here, I'm not sure, do you have that my talk running in the fifth window somewhere on the side and you're not paying attention?

Are you asleep? I'm not getting any cues, so I cannot really react to you. It's also very draining to have this, like, talking into the void and not getting much feedback back from a talk. One other thing that I found interesting that we took away that sponsoring costs were in EMEA and The Americas. So North And South America were mostly down because virtual events are cheaper, mostly because what is expensive is food and venues and the people working at the event.

And in APJ, we almost saw the opposite effect that there were a lot of local events that were run pretty cheaply, for example, in India. And then once people switch to those international platforms with standardized standardized prices, it kind of, like, all evened out a bit more. So the EMEA and North And South America events kind of, like, went a bit down in cost, and APJ went a bit up because everybody has to pay the same price for the platform in many parts. If you get much value out of virtual sponsorships, our opinion is no. So unless we have some good reason to sponsor, like combining with a talk, for example, we will not sponsor virtual conferences anymore because we just did not see a lot of interaction or traction, which is a very harsh statement for, organizers of virtual conferences, I'm I'm aware, but that is just the learning that we took out of that.

By the way, the steps for this event, of course, I will ping Matt afterwards, like, how many live viewers did we have, where's the link for the event. If it's YouTube, I will get the steps kind of like going forward, And I will feed all of that back into the dashboard. So we will slightly bump up the views again on that. So this was the very quick run through through 1,000 virtual talks. And I think we still have a couple of minutes left.

And I can just slice and dice a bit more into the actual data that we have because, like, screenshots are always nice, but, making it a bit more interactive is good. Matt, stop me when we are out of time, but we have, all of this running live here. So I don't want to push Kibana too much. On the other hand, it's it is a nice tool for this. And once you have kind of, like, prepared the data correctly, you can see a lot of stuff.

So I have picked all the events since March 1 when we basically locked down and went into virtual mode. I could filter this down by region, subregion, status of event event type, like a meetup, a conference, an external meetup, all of those. But I'll just look at the global view. So as of kind of, like, now, we are at more than a thousand events. We have a 130 something or so average attendees.

By the way, the the average of the attendees is tracked. Like, only if and only if we have attendees for an event, we will use that for the average. So if we don't have a number, the event doesn't count to the average because sometimes people try to do that calculation and it doesn't work out. For some events, when we just cannot get numbers, we will skip them in the statistics here. You can see we have a more or less even, like, distribution of how many events we are running, approximately every couple of weeks here.

We had one spike here because, at our own conference, we to make tracking easier, we used every single talk as its own event, which might be a bit of an anomaly, but that's why you have this jump here. But otherwise, you can see some seasonality. Summer is lighter at least in North America and Europe. Christmas or winter or New Year's is also lighter, and then you have, like, spring and fall peaks where you have more events. And you can see, we started kind of slow, and then we ramped up to, a decent amount of events every week or so.

So this is each little slice is one week, and you can see we run 20 ish events or so a week, like 10 to 20, unless it's, summer and everybody takes some time off. As I said, you can see the the summer of attendance, and you can really see, like, where where were large events. So here you can see this is kind of like an outlier. We had the large external meter, and the large external meter had 20,000 attendees, and that kind of throws off your statistics. To take that out, I have actually filtered down the statistics to exclude events with more than a thousand attendees just to have, like, a bit fewer outliers.

And then the sum of attendees is kind of, like, much more even, and it correlates much more strongly, like, with the number of events we have been running. So this is the same thing here. And you can see how many attendees we approximately get a month, on these events. You can also see the sum of YouTube videos, so we need to think about that. Like, we need to produce a couple of more hit videos because per week, we only get, yeah, this amount of of views right now.

But we have a couple of older and very popular videos, and those are drawing in the the crowd, basically. You can also see, like, what is the average of attendees. And while the perception has been that attendance has been down in a lot of virtual events, the actual numbers are not that strict, actually. So some events are larger and others are smaller. So you can kind of like yeah.

You need to take the good and the bad, I think. Like, there is no way to say, like, virtual is broken and we'll never do it again or virtual is great if we always do that. It's just finding the right middle ground for that. We also have, like, how we kind of, like, from a business perspective, we have broken topics into different areas, and we track those, like, where do we focus our effort and is that working out. We also track things like if there are RSVP numbers, how many attendees actually attend to RSVP to our events.

We track, like, what kind of event types we have, how many events did we have, average attendee per event type, etcetera. Obviously, there are always outliers in those, but it's still an interesting indicator of, like, what works well and what doesn't work so well and where do you want to focus your effort once you have collected all of that. We know, for example, what languages are working well, and you can see we have a strong focus on English talks. But like I said, Brazil was like this focus area, which gave us a lot of attendees. So we did more Portuguese talks, mostly for Brazil actually, and less so for Portugal, sorry, Portugal.

We do have a fair amount of events in Mandarin for China as well. And then the French always prefer French. So we kind of, like, try to compartmentalize and see what kind of topics work well. Also seeing, like, which regions is how active and how many events we're doing per week there. We also even know internally, who is doing the talks at our events, especially for meetups.

This is interesting. Like, it's largely our team. The community is also doing a lot of talks, and then, like, which other teams within the company are doing talks. So where should we promote that we want a new talk from you? Who can actually jump in and give a compelling talk?

You could even correlate that with views. Like, if the engineers do a talk, like, does it get more attract or does it attract more viewers, because it might be the audience might be keener on that type of content. So all of that is kind of like something that we can drill down, filter by reach, and just see how it's going. Also tracking who is doing a lot of talks, who is doing fewer talks. And this is not a benchmark that you have to do the most talks.

It's also like an an impact factor of what is in there. So for example, we can track, like, these are the YouTube views of the some of the videos, so we can figure out, like, what are the topics that we should probably do more of because they seem to have, like, large viewing on YouTube and be going well. Or what is going well in terms of live attendees at conferences? By the way, since I filtered out more than a thousand attendees, this might be totally skewed again as well. What we track, something that's been filtered out at the bottom here.

We have started to track, like, on the YouTube channel, like, how many views are we gathering and how many subscribers are we getting to the channel. And you can see that the growth is very promising. So we will definitely need to do more virtual events even when we can go back fully in person. So that's it kind of what what we're tracking and how we're trying to figure out, like, what makes sense, what doesn't make sense. And going forward, how will be the right split between virtual and in person.

And with that, I think I'm not quite out of time, but this is pretty much the the amount of content that I have today that I want to share. I'm happy to discuss afterwards. If you have any questions, let me know. I've built all the plumbing behind this, so I should be able to answer pretty much any question that you have. One final thing is data quality is always tricky.

So for example, we have some internal alerts on an event like, is there any data missing or looks wrong so we can actually fix it at? There's quite a few things that we should be fixing still at past events. For example, here, it looks like somebody created an empty event that we probably just need to delete. But data cleanup is always a tricky thing. Your stats are always as good as the data that you have behind it.

So it's kind of like an ongoing task that we have, like, a monthly cleanup process to work with that.