Here at Audiosear.ch, we’re always thinking about new and innovative ways to surface the most relevant and interesting podcasts our users might be looking for. With our ever-expanding database of hundreds of thousands of episodes, that becomes a more and more difficult task by the day. We ingest thousands of shows of varying quality and popularity, but there are few metrics for quantifying that. We wanted to figure out how we could use publicly available data and social media to come up with a comprehensive, sophisticated, and intuitive way to quantify the “buzziness” of a given show or episode, thereby allowing us to generate better recommendations. What we came up with we call: the Buzz Score.
To understand why evaluating “buzziness” is such a difficult task, it’s helpful to understand a little of the history of podcasts up to now. In 2005, Apple launched podcasts as a new way to listen to radio-style programs by downloading them to your iPod (thus the name) and playing them later. This was before the iPhone and other smartphones were introduced, and cell networks wouldn’t be able to support streaming media for another several years, so the process was a bit clunky and podcasts didn’t quite explode onto the scene. Apple broke out Podcasts into a separate iOS app in 2012, thereby setting them apart to users as a distinct form of media.
Since then, Apple has made occasional but marginal improvements. Although many competitors have tried to overtake them, and Android users can only use rival apps to listen, Apple still dominates the market with over 55% of users listening on the Podcasts app. Apple Podcasts is great for finding a show if you already know what you’re looking for, but it’s not great for serendipitous discovery or surfacing less well-known shows that nonetheless deserve exposure. Developers who have built alternative podcatchers by and large haven’t been able to do much better with discovery, since Apple provides only a minimal public API for searching podcasts and user reviews and ratings have been the only metric for determining quality.
Despite the lack a significant push from Apple or other major distributors, though, podcasts have become more and more popular over the years, with 67 million Americans now listening to at least one podcast on a monthly basis. Given that Apple is still the main way people access podcasts, we decided to use the data they make publicly available as a basis for evaluating audience engagement — and by extension some sense of the quality of the show.
The most obvious metric seemed to be the star rating, which comes from user reviews submitted through iTunes, on a scale of 1 to 5. However, as we started comparing ratings, we realized that the vast majority were between 4 and 5, and especially above 4.5. This suggests that listeners don’t generally bother leaving reviews if they don’t like a show, and they are most likely trying to help the shows they love by giving them a high rating, especially if the host has asked listeners to do so. We decided to run this number through a mathematical function which expands the difference at the upper end of the rating scale. This allows us to make more meaningful distinctions between ratings that would otherwise appear very similar, such as 4.6 and 4.9, and weight them accordingly.
Number of ratings
We also wanted to factor in the number of ratings, which gives a sense of the audience size and enthusiasm, and, to some extent, how long the show has been popular. Since there are many shows with only a few ratings and a few with tens of thousands, we decided to boost shows that have at least a few reviews in order to not let them get completely overshadowed by the most popular ones.
We have also been keeping track of the positions of shows on the iTunes charts, and we wanted to incorporate that somehow as well. Since it is quite an accomplishment to make it onto the charts, we decided to count a show’s appearance as more significant than its highest position achieved.
Cultural commentary (in this case, on Twitter)
All of this is great for getting a sense of the overall quality of a show, but what about individual episodes? The iTunes API doesn’t provide any episode-level data, and there isn’t another obvious source for such metrics. We turned to social media to gather this information, scraping Twitter for episode recommendations and associating them to entries in our database. This allows us to bump up the Buzz Score for any given episode that’s getting a lot of attention. The more people are talking about it, the higher the score goes. This levels the playing field a bit for new or relatively unknown podcasts which manage to get noticed and become part of the social media conversation.
Of course, not every episode that gets a conversation going will be relevant to the average user. There are countless niche communities on Twitter and many podcasts that cater to niche audiences. However, we use many different data points to serve search results and recommendations, and the Buzz score is just one way to help users find a podcast they may have heard about or that they’re more likely to enjoy.
As the podcast industry has grown in audience size and value, producers and publishers have been eager to see better and more detailed metrics in order to better understand their listeners and give them more engaging content. Apple recently announced that they are doing just that, releasing a new producer portal with fine-grain analytics and user behavior data. With this new push by Apple and with other providers such as Spotify entering the field, we hope that the breadth and depth of publicly available data will expand so that we can continue to develop our Buzz Score algorithm.
Want to know your episode or show’s Buzz Score? You can check it out here.