What is a Quality Conversation? Simply put, it’s a discussion that:
- is fact-based,
- stays on-topic,
- is informative, and
- doesn’t stray into personal, often offensive, attacks.
This definition is derived not only from our extensive experience building safe online communities, but is also informed by direct feedback from millions of users across our network.
Building quality isn’t just about removing toxicity, it’s about surfacing the most productive comments that move the conversation forward.
And with great scale comes great responsibility! We have to ensure our moderation technology is constantly evolving to keep up with the unprecedented scale and complexity of the online conversations that OpenWeb hosts.
We achieve this by taking a multi-pronged approach:
1. Moderate: Proven, AI and Machine Learning-powered moderation technology that identifies and removes toxic content.
2. Educate: Tools that support a transparent user engagement journey and encourage them to create civil, engaging content.
3. Curate: Algorithms—like “Best” Sorting—that identify and expose high-quality content while minimizing risk exposure.
Keeping Quality at the Top, Minimizing Risk Exposure
We are excited to bring you our newest methodology: A smart “Best” Sorting Algorithm that automatically curates your comments section, bringing quality to the top. This is reinforced with a nuanced Exposure Metering that ensures the right checks and balances are in place to avoid exposure to risky content.
Across the OpenWeb network, our “Best” Sorting Algorithm identifies and exposes the civil and thoughtful comments, bringing them to the top — as 86% of the users only see the discussion at the top.
Research suggests that when people are exposed to civil content, they are much more likely to create civil content themselves.1 This new sorting feature has successfully improved civil engagement across the network, including key safety and engagement metrics2.
We ran an A/B test, pitting our new “Best” versus “Newest” sorting algorithm, and here are the results:
- +12% lift in average reading time
- +5% lift in safety rate
- +2% lift in approval rate
How does it work?
The new algorithm takes into account three main parameters:
- The toxicity level of the comment
- The thoughtfulness (depth) of the comment
- The community signals such as likes/dislikes, reports, etc.
Based on these parameters we set an exposure score that determines the position of the comment/reply within the Conversation stream. By measuring the percentage of good comments exposed out of the total comments exposed, we understand how effectively our exposure metering hides the risky content from view.
Curating Replies, Boosting the Depth of the Conversation
‘Best’ sorting is not limited to the top level comments—it is also implemented within the reply threads. We have noticed that when replies get an equal amount of exposure as top level comments, this invites trolls to jump in and take advantage of otherwise successful, positive contributions.
The “Best” sorting algorithm ensures Replies are sorted based on quality corresponding to the top level comments as well, when the ‘Best’ sorting is selected in the main sorting option. It then automatically reveals quality Replies, and adapts the UX, to show more interactions. This influences the user to read from top to bottom.
The Reply sorting enhancements led to a 47% reduction in the number of views on non-safe messages during the A/B test. We also recorded a 48% decrease in users who were exposed to non-safe messages during the same period. Consequently, we saw a drop in users reporting toxic comments.
Note: Users can always select their own sorting in the Conversation itself.
Simple Tools, Significant Impact
Host a conversation that brings out the best in your community. In our tests, we saw publishers successfully achieve a lift in their community engagement and safety metrics with our “Best” Sorting Algorithm. This enhancement will be rolled out as the default sorting mechanism across the OpenWeb network in the coming weeks.
You can easily change this default sorting for OpenWeb Conversations in the Admin Dashboard. As always, reach out to your Partner Success Manager if you have any questions. And if you’re not an OpenWeb partner yet, let’s have a conversation.
1 Chen & Lu (2017); Sukumaran, Abhay & Vezich, Stephanie & McHugh, Melanie & Nass, Clifford. (2011);
2Approval rate is the percentage of approved comments out of total comments. Safety is the percentage of good comments exposed out of the total comments exposed.
