Article recommendations and reader engagement: A case study in data science-driven local news collaboration
How The Local News Lab worked with The Philadelphia Inquirer and The Texas Tribune in our first Knight Foundation-funded cohort project
The Local News Lab at the Brown Institute is exploring different ways that AI can help boost reader revenue. Some AI tools provide assistance with time consuming tasks like selecting the stories to go behind a paywall and others are completely automated, like our recommendation engine that suggests a list of articles a reader might be interested in.
To be effective, our experiments require an understanding of a news outlet’s editorial values, their business needs and reader interests. We rely on collaborations with actual newsroom partners.
In October of last year, we started working with two local outlets — The Philadelphia Inquirer and The Texas Tribune. We supported our partners from basic data collection to the deployment and testing of an automated recommendation system that compiles a list of ‘what to read next’ given the article you are currently perusing. Our work with these newsrooms involved scaling our system to handle their traffic — our goal was to support up to 10M monthly unique visits — . In the process, we led them through various workshops, each an opportunity for creative exploration around data and automation. While our goal was page-specific recommendations for readers, our development methods relied on tools that the news outlets could use long after our project finished — from “product thinking” to workshops that surfaced newsroom values around recommendation, automation and AI.
From development, to deployment and testing, these newsrooms were part of the whole process. They were tremendous partners to the lab in terms of our learning and technical improvements. As a practical matter, the recommendations were also a success! They boosted reader engagement (as measured by clickthrough rate, or CTR) by 10% on the Tribune and 16% on The Inquirer.
This project was about more than one product, though. With our cohort design, we sought to bring two different news organizations together and to help fill some gaps in expertise. To that end, we are proud of the work we’ve done. Here is a note we received from an engineer and partner at The Texas Tribune, whose permission we have to share:
“We want to thank you again for all the work you and your colleagues have put into this project and all the assistance you provided through the A/B tests. We’re very excited that Trib editors and reporters will no longer have to spend so much time manually picking story recommendations. Should be a huge productivity gain for our entire editorial operation”
- Dan Simmons-Ritchie (Frontend Engineer, The Texas Tribune)
Prior to this cohort-based project, we created and deployed an initial version of our recommendation system with our first experiment partner. Washington City Paper, a 40-year old publication covering Washington, D.C. that previously used a recommendation module featuring the most recently published articles and wanted to improve recirculation. Our system was trained using a machine learning approach known as collaborative filtering and based on an engagement metric of time spent on page (or “dwell time”). We ran our recommendations for five weeks on the Washington City Paper site, and with a sample size of roughly 500,000 sessions, we evaluated our recommendation module based on CTR. We found a 50% increase in CTR compared to our partner’s previous recency-based feed. The substantial gains demonstrate a desire readers have to more deeply engage with the paper’s content, but had previously lacked the ability to do so. The Washington City Paper needed new tools, tools that are typically only available for large publishers. Readers are hungry to engage with more local news content, and industry-standard data interventions like our recommendation module can deliver real value to local newsrooms and readers alike.
This site has significantly less traffic than our cohort partners’ but this first collaboration helped inform the type of machine learning models we chose, our framework for considering editorial values in recommendations, and to test our product in a production environment. We were excited to scale it to a whole new level with The Philadelphia Inquirer and The Texas Tribune.
One of the overarching goals of the lab is to communicate how we do our work, our successes, what we learn and how we can improve. What follows is a recap of our work with our first cohort in three main parts: the product process; scaling our system; and collaboration and communication.
Making a Recommendation “News Product”
What we did well
Scaled the technology. Having our recommendation system already in production gave us a head start with this project. This meant that the lion’s share of the work over the six months would be scaling our infrastructure (as mentioned above). It also meant that we could explore recommendations conceptually with our partners.
Workshopped the concept of “recommendations”. We held multiple workshops; we considered blue sky ideas inspired by platforms we’ve encountered in our daily lives as well as features that could be possible given the constraints in our current system. Our team’s data scientist and machine learning expert, Sam Petulla, also created and led a session focused on equity in recommendations, and prompted our partners to examine how their respective organizations’ values could inform all of their products, including recommendations
Kept a solid roadmap. A solid roadmap kept us on track as experimental product innovation can be ambiguous work. We are especially proud of creating software development processes that built expertise and capacity within our cohort. Our “pair programming” sessions on data collection and API integration, for example, not only helped with our implementation, but also were novel and successful learning experiences for our partners.
What we learned
On the flip side, having a product already in production meant that we were not starting fresh, and that our partners had less control over the features our recommendation system were built upon; it was trained on a collaborative filtering model that, if we’d had more time, we might have compared it to a different type, perhaps a semantic — or content — similarity model). For our next cohort, we will attempt to tease apart our existing work, brief our partners on the component parts, and together decide on what new direction we will take. Our hope is this will provide a more creative and collaborative product planning process, be more engaging for our partners, and take us in a new and exciting direction.
Scaling a System
What we did well
Our technical team members, Sam Petulla, Ting Zhang, and Sylvan Zheng, did an extraordinary job in scaling our infrastructure from where it was (by all measures stable and successful running our system for a small local publication with modest traffic) to the point where it is now: supporting two distinct recommendation systems deployed on each of our cohort partners’ sites via APIs, both of which have significantly more traffic. To achieve this, Sylvan scaled our codebase so that we can more easily onboard additional newsrooms into our data collection and processing systems. This work has resulted in a 98% time savings in fetching data. Also our model can now accommodate 4x as much data as our initial prototype, with a training speedup of 30–50x faster, and our API can handle 50 more requests per second, which represents a 10x speed improvement. In short: we accomplished a huge amount of scaling work in a short period of time.
What we learned
Completing the amount of technical work that we did in this timeframe means we also accumulated a not-insignificant amount of “technical debt.” With more time, we would be able to ensure our test coverage was high, and put an automated deployment system in place rather than relying on manual processes as we currently do. For these reasons (and more), we have extended our second cohort project to 8 months to make room for such improvements.
Collaboration + Communication
What we did well
As a distributed and fully remote team with remote partners in different parts of the country, it was a challenge to create a collaborative and creative environment. We established a roadmap along a timeline so the team was aligned and working toward the same goals. We set up regular meetings and Slack channels for both real-time and asynchronous communication. At the lab, we fiercely respect our and our partners’ time. We worked hard to ensure everyone was prepared for our meetings by circulating agendas and notes in advance and our efforts paid off — synchronous sessions were punctual and efficient. Most importantly, we listened to our partners: when the team at The Texas Tribune told us they wanted to know more about A/B testing, we created a presentation on best practices and worked directly with them to ensure our tests were set up well and that they could institutionalize this learning.
What we learned
Given the nature of the project and that we were starting off with a product to scale, rather than a blue sky, we had some built-in constraints (which are not always a bad thing). While this meant the core product was less a group collaboration, we did make a point of running workshops around recommendation features and priorities, and customized the system to each newsroom’s specific needs. For our next cohort, we will have a keen eye towards intra-cohort collaboration and facilitate even more cross-partner sharing of learning and expertise. One big change we’re making is that our second cohort will consist of three partners instead of just two.
Our next post will announce the start of our second cohort that we are currently selecting. New people, new partners, and new product all lie ahead — stay tuned to read about the next chapter of the lab.
