Made by
Made by
Shilla Saebi
In the second part of this talk on open source communities from DevRelCon San Francisco 2019, Shilla Saebi from Comcast walks through a variety of internal and external metrics for community growth and open source contributions from an extensive program of events.
See part one from Indeed’s Alison Yu.
Okay. So, this is part two of the talks that you just heard from Alison. She talked a lot about executive buy-in, especially leadership buy-in, as well as branding, and dove into internal metrics. I’m going to go ahead and dive a little bit deeper into the metrics that we put out, internal ones for our communities and external.
A little bit about me before I begin, my name is Shilla Saebi, and I’ve been with Comcast for a little bit over nine years. I am an open source program manager. Prior to this role, I was on the cloud team, and I was an operations engineer.
You might be wondering why Comcast has an Open Source Program Office or what they’re doing with open source. So, I will give you all just a little recap of the journey that we’ve been through.
I worked with a number of folks that have been at Comcast longer than I have. They’ve been there for much, much longer, and they’re open source contributors. And I reached out to a bunch of folks from different orgs as well, to find out, when did open source really begin at Comcast? 2006 was the date that was consistent.
We know that in 2006, we were large consumers of open source. In 2012, we launched our first two open source projects. And in 2013, the Open Source Advisory Council was formed.
The Open Source Advisory Council consists of folks from our legal team as well as security and engineering teams, well-respected engineers across the company. They are a group who decide whether open source contributions should go through or should not go through. And so, 2013 is when that was launched. In 2015, we were now large contributors to open source. We won the OpenStack Superuser Award.
In 2017, the Open Source Program Office was created. And that’s where you’ll see we joined a bunch of different foundations, and did sponsorships like the Linux Foundation, CNCF, Apache Software Foundation, etc. We also launched comcast.github.io, which is a showcase of our open source projects for external community folks.
Fast forward to 2018, last year, we had our first open source conference that was available for, not only Comcast employees but to the community. We had about 70% external attendees and 30% internal to Comcast.
And we also launched the Open Source Fellowship pilot. This is a pilot program that was launched, specifically for our engineers to work on open source contributions. We have seven open source projects that we’ve selected, that we know that we’re large consumers of, and our engineers get to work 20% of their time, originally, it was Open Source Fridays, but it’s 20% of your time, whenever that may be, to work on open source, and that is 100% supported by our leadership. So, we’re really proud of that.
And then in 2019, which is today, we have several hundred open source projects. Our team has grown tremendously. And we’re looking at putting more strategy around open source. We also have a partnership with our talent acquisition team, so that when we go to conferences, we’re showing up with a strategy and organized.
So, the team that I work on, I work in the Open Source Program Office. Like I mentioned, we opened our doors in 2017. We support over 7,000 developers. We have 224, maybe more as of today, repos on GitHub.
And we are a seven-person team. Just in the last month, we’ve hired two folks. We have two software engineers on the team. We have two program managers. We also have our leader as well as a summer intern that’s come on board to help us with metrics.
So, what’s brewing in our open source communities? So, a little bit about how we kind of got into metrics, why they’re even important.
So, originally, once my colleague and I started on the team, we knew that open source was happening, we knew that people were contributing to open source, but we didn’t really know who those teams were or where it was coming from, and there was no organization around it.
So, what we started doing was, we started meeting with different engineering teams each week. Some weeks, we were meeting with two or three teams, to the point where we were exhausted from meeting with so many engineering teams. You know, I don’t know exactly how many, but I want to say maybe 30 or 40 engineering teams, and we have so many more that we haven’t even touched.
But once we started meeting with these different teams, we started getting a lot of open source contribution requests. We’re telling people, “Hey, we love what you’re doing with open source, please just come to us, and talk to us about it. We want to partner with you. We don’t want to necessarily police you or anything. We want to make sure that you’re using the right licenses and that we’re also showing gratitude when you are doing open source contributions, and that we know what’s going on.”
So, the communities that we report out on. We report out on our internal community, achievements, and progress that we’ve made, and also on our external communities.
Our internal communities are the 7,000 developers that I mentioned, the thousands of developers, plus, we also have other program managers, project managers. It’s not just necessarily software development engineers. We have a ton of people in the open source community.
Also, the open source projects that we put out, we want to make sure that we’re watching what’s happening with those communities. VinylDNS is a one-stop shop portal for DNS configuration management. Trickster is an open source dashboard accelerator that plugs into Grafana or Prometheus. Kuberhealthy is easy synthetic testing for Kubernetes. So, these are some examples of some of the projects that we’ve put out. So, we want to look at what’s happening there.
So, the internal community metrics that we track. We look at who our top open source contributors are in the last year. We want to reward them. We also showcase them on our blog. We also showcase them on this exact report that goes out.
Our community growth is another internal metric that we look at. The place that the entire internal open source community mainly hangs out is in Slack. And so, we want to make sure that, that community is growing. We’re making people aware that there is a place to congregate, and hang out, and ask questions, and chat about open source.
Also, our internal presentations. We also do a bunch of internal presentations across the company. We look for opportunities to speak with those engineering teams that we’ve met, because now we’ve built this partnership with them, we work with them. And if we know that there’s an event happening, we’ll ask them, “Hey, do you all have, like, 20 minutes in your schedule? We’d love to talk about the open source program.”
Incentives and gratitude that is given to the OSPO from our developers, we definitely collect and share our kudos with the team. We started doing that a few months back and actually reported out on it. And it was really nice to see all of the thank yous that we’re getting from engineers for simplifying the whole open source process.
Prior to our team joining, the engineers had to kind of talk with legal, and go through why something should go open source. And that wasn’t really the best experience for them. So, it’s really nice to see that they’re thankful for the work that our team is doing.
And we do the same thing for them. So, anytime any open source contributors make a contribution, we make sure to thank them. We have a number of different ways to do that. We have portals. We do handwritten notes. We do cards, but we track.
And then the internal events that we host. We do open source days twice per year. Last year, we went to Seattle and Denver. And the reason why we have to travel for them is because they asked us to come. So, they asked us to come and do a mini open source day. And so, we came because Philadelphia is where our headquarters is. And we have a very large auditorium there where we do a lot of conferences. But the other remote offices asked for us to come out, and we did. This year, we just did one in Sunnyvale. And then we have another one coming up in Philadelphia.
And then the number of open source contribution requests that come in. So, we look at those and that number, we’re watching the growth. And it’s really good to watch those because that’s when we can actually justify that we need more people on our team.
And then the type of open source contribution requests that are coming in. Are they bug fixes? Are they full-blown projects? Are they documentation? We look at that.
And then here are some examples of the community metrics that we track. I’m just going to show you all exactly the slide deck that I pretty much put out twice per year. And I won’t include every single slide, because in the interest of time, but I’ll include a few so that you can get an idea.
So, the Slack community growth, here, you’ll see that our community, we have about 900 people in our Slack channel today. I’ve encouraged people on our team to just put it in their email signature. I have it in mine, I just say, “Hey, join our Slack channel if you have questions.” Whenever people DM me, I’ll reach back out to them, and I’ll tell them, “Hey, we do have this Slack channel where you’re more than welcome to ask that question.” And so, we’re continuing to grow that community, and it’s something that we’re watching.
Internal Presentations. These are the ones that I mentioned that we do. We have a showcase series. We do an open source educational series. I did a recent webinar on innersource. We had a very good turnout. Innersource is a hot topic. So, back by popular demand, I did another presentation on innersource at a internal event that we did.
So, we want to make sure that we’re continuing to raise awareness about Open Source Program Office, what’s happening. And so, these are the internal presentations.
And then the events that we help organize internal and external, shameless plug, June 20th in Denver, we have an open source conference that’s open to the public, and it’s free, so you’re more than welcome to come if you’re around. But that’s a great example of an event that we’re helping organize. I’m also the meetup organizer for the Prometheus group in Philadelphia. And also, we do a number of other events, like I had mentioned, the internal ones.
The number of open source contribution requests that are filed. Here, you can see that there’s a tremendous growth since 2017, since we opened our doors. And we’re projecting to go even higher this year.
And this is exactly what we used to show that we need more help on our team. So, we now, just a month ago, brought another open source program manager to help us with these contribution requests, because we only had one person working on these.
So, the external community metrics that we track. Pull requests to the open source projects that we maintain. This is super important to us. And we’re not micromanaging or policing anyone, but we want to make sure that the open source projects that we’re maintaining, if there’s a pull request that comes in, it’s not sitting there for a year.
And also, we help coach our engineers on, you know, how to respond to things in the open source community. We do a lot of blogging. We let folks know. Sometimes people, it’s their first time working in open source, and so they don’t know. You know, there could be a pull request that sits there for six months, and that other person on the other end has no idea what’s going on. So, we want to make sure that, that kind of thing doesn’t happen.
Also, outside contributors to the open source projects we maintain. It’s really important to have diverse committers, diverse contributors from different companies, different areas, etc. And they’re not just Comcast-heavy. So, that’s something we definitely look at, especially for the projects that we think are going to be really useful to the community.
Also, most starred, most forked, and the newest open source projects we maintain. We understand that this doesn’t necessarily measure the popularity of a certain project. But there are some communities that require a certain amount of stars in order to be incubated into, for example, a foundation or something. And so, it’s something that we look at, and we use the GitHub API to do that. And that information’s actually on comcast.github.io as well. It’s in the statistics at the bottom of the page.
Blogging. We have an engineering blog that is mainly funneled through our corporate communications team. We work closely with them to get stuff out there. We don’t blog all the time, but we blog when we have huge announcements, like, let’s say, “Trickster is now open source.” We’ll talk about something like that.
If we have a whirlwind tour of number of open source conferences, and we have a bunch of updates, we’ll write a blog post about that. Recently, we’ve been doing blog posts on opensource.com. They’re super helpful. Great, great team to work with. We were able to push out a couple of posts through them. So, we’re also watching to make sure that we’re continuing to put content out.
And then also outside press pickups. I have an alert setup in Google. Anytime Comcast, Open Source, anybody blogs, or writes anything, a news press, whatever, I get a notification. So, that’s something we look at.
We love to see positive press come out. And we think that giving back to the open source community, and sharing our use cases, and putting projects out is definitely helping us and giving us a lot of positive, positive feedback in the open source community.
Sponsorships with open source foundations, that’s another thing that we look at. We want to make sure that we’re on top of what’s happening, where we’re actually sponsoring, where the money’s going, and then where it should go next. So, we talk to our engineers.
We have a survey that we put out, and we say, “Hey, here are the open source foundations and projects that we’re sponsoring.Is there something we’re missing? Is there a community that you work with or you’re a huge consumer of that we should be looking at?” And so, that’s really helpful. So, we track all that.
Our external presentations, today counts as an external presentation. Our team goes out to a number of different places like FOSDEM, All Things Open, CHAOSCon, Open Source Summit, etc. We definitely track those.
And Twitter engagements. We look at the impressions, the clicks to see, especially if we have like a big event going on, and we want to make sure that we’re projecting that message out to the community. We look at that kind of thing.
And then lastly, traffic to comcast.github.io, which is our public facing and open source website.
So, a few examples of the external community metrics that we track. Again, I won’t go through every single one of those, but I’ll show you just a few examples of what we report out on.
So, Trickster, that was the open source project that we recently put out to the community, and about 70% of the contributors are non-Comcast employees. Kuberhealthy, similar. This was actually the more recent open source project that we published. About 50% of the contributors are outside of Comcast. And then about 80% of the contributors for the Apache Traffic Control project, which is now a top-level project, are now outside employees.
And I must say that once you donate your project to a foundation, you do have an advantage of getting the community from that foundation or from that project. So, that’s what it looks like for these three.
Published blog posts. We have eight so far. We have an editorial calendar. And so, we make sure to track it. And then OSPO public speaking engagements. It looks really low right now. And that’s because conference season hasn’t hit us too hard yet. Around September through December is when it will be crazy. And so, that number will likely spike up.
And I’ll also include that we do put, and I don’t have it here, but we also put out a highlight deck that lists out all of these things, so that people can see. It’s not just, “Oh, they went to nine public speaking engagements.” We actually list out what those speaking engagements are.
And then, like I mentioned, the most starred, most forked, and our overall stats, the total members of our overall stats look really low. We’re actually have hundreds of members in our Comcast GitHub organization. However, by default, it’s set to private once you get added to an organization, so we’re working on talking to folks and see if they want to switch it to public. That’ll really help us with our metrics.
And then this is the comcast.github.io website traffic, and this is using Google Analytics. So, we’re really looking at the total number of clicks, and then the difference between last year and this year, and then figure out how we can grow that.
So, as you can see, our community metrics are a big mix of internal and external community.
So, what’s next for us? Automation, automation, automation, times a million.
So, right now, what I am doing to get these metrics are, I use Twitter, I use Jira. I go in, and I search out through the filters. I also use GitHub API. Because the GitHub API has a time limit on how much data you can get, we’re using another tool called GitHub Archives. And that’s what we’re using to get the top open source contributors from Comcast in the last one year. And also, that’s what we’re using to get a number of other data points for the open source projects that we put out.
Luckily, our summer intern is here to help us with our metrics. We do have a dashboard, in-flight. And right now, it includes our top contributors, as well as some community health metrics. But ideally, we’d love to plug the Jira API, Twitter API, and get all that information into a one-stop shop, and then have everybody just be able to access it at any time, transparent. We don’t want to have to put out some report, you know, twice a year. We want anyone from our community to just be able to click in and see what’s happening.
And with that, thank you.