0:00
/
Transcript

"Engagement" is a dumb metric

An algorithm doesn’t always tell you what content people truly find valuable. Plus more from my chat with Mike Solana.

Yesterday, I spoke with Mike Solana of the excellent newsletter Pirate Wires, one of the best places anywhere to find some truly heterodox takes about the relationship between tech and politics. And this was timely, because last week Mike had a response/critique to “Social media has become a freak show”, my post that criticized the low-quality posts that are being prioritized by the Twitter algorithm. (Which, ironically, triggered a truly viral firestorm on X.)

It was a good, free-flowing conversation, but we wound up spending the first half of the chat on AI, including the recent violent attacks at the home of Sam Altman, the OpenAI CEO. So we didn’t get into quite as much detail about the mechanics of the Twitter algorithm itself; today’s newsletter is intended to remedy that.

Twitter was always a better news hub than “social media” platform

I’ve written a lot about Twitter and its competitors at Silver Bulletin (see here, here, here and here) and I’d like to think I have a pretty nuanced critique of the platform. Even so, it can be easy to get one’s timelines scrambled about exactly which changes were made when.

The point of last week’s post was that relatively subtle changes to the algorithm can make huge differences to the end result. On a case-by-case basis, I think many of the changes that Elon Musk and X’s head of product, Nikita Bier, have made to the algorithm are defensible or even good.1

The cumulative effects, however, have been both to promote slop and to greatly reduce the amount of traffic to publishers like the New York Times who mostly use the platform to link to offsite articles. Some of this is because the Twitter algorithm has, at times, punished tweets that contain offsite links through a variety of implicit and explicit methods. (The extent to which it still does this is disputed and is constantly changing.)

Probably the more important factor, though, is that X now very strongly defaults people to the algorithmic “For You” feed instead of a chronological feed of the accounts users signed up to follow. That’s why the NYT feed often only gets around 70,000 views on a tweet even though it has 53.2 million followers. It’s not (just) that fewer people are clicking on tweets with links. Rather, it’s that users are mostly seeing the algorithmic feed, and the @nytimes account doesn’t tend to have a lot of “voice” or to trigger the other “engagement” signals that the algorithm rewards — particularly not replies, which are valued much more heavily than “likes”. It’s also not clear how much passively following an account is taken as a positive signal, if at all, even though this reflects an explicit preference on the part of the user.

This greatly reduces Twitter’s value as a news aggregator. It can be almost useless for following, for example, developments in Iran, where you might mostly be interested in reporting and reliably sourced hard news (and not necessarily interested in commentary, especially from non-experts). And this is a shame, because Twitter used to be an absolutely unparalleled way to “monitor the situation”, essentially the social media equivalent of London Heathrow, with departure points to everywhere else that you might want to visit.

What happened within the platform — an anonymous account would yell at a reporter like Maggie Haberman, and she might even respond back — might be a value-add to some users and annoyance to others. Of course, once you're on Twitter posting links and promoting your outside work, you might get swept up in it. (I’m a prime example.) But it wasn’t necessarily the reason you were there or the most differentiating feature, any more than you go to Heathrow to visit the BA Lounge.

Did it all start going downhill in March 2016?

However, the switch to the algorithmic feed didn’t happen under Elon. Instead, the algorithmic feed became the default in March 2016, part of a series of efforts to increase “time on site” and make it more attractive to advertisers. (Twitter has never been a particularly good business despite its once-very-high “mindshare” and capacity to serve as a hub to “drive the conversation”.)

The timing is interesting in that early 2016 was about the time when Twitter became a much less pleasant place for anyone like me who tends to have sharp elbows and abhor groupthink. This was also an inflection point for certain progressive-coded pieties, a.k.a. wokeness. Twitter was probably not the main reason that progressive and even neutral spaces suddenly became much more woke2, though I suspect it contributed to it — and that changes to the platform, including the addition of the quote tweet in 2015, further catalyzed some of this behavior.

Here’s the thing, though. The content that people engage with in the form of things like comments and quote tweets isn’t necessarily the content they find the most valuable — especially not in the long run where it’s exhausting if the volume is always turned up to 11.

Opening up the Silver (Bulletin) curtain

I recognize that Silver Bulletin is an unusual newsletter in many ways. We cover a few topics intensively — elections, sports, and sometimes tech/media/AI stuff — when most other newsletters are either broader or narrower. We have a mix of very data-driven stuff and other work that’s more take-y. We have a slower publishing cadence but with longer, more detailed articles than most of our competitors. And in contrast to most other Substacks, only about half of our overall views come from people reading in email; we’re really using Substack as a hybrid of a newsletter-delivery system and a web publishing platform. Fortunately, enough of you seem to like it — or at least like enough of it — that it’s a pretty good business.

Share

Nonetheless, we do have a lot of fairly reliable data on Silver Bulletin’s engagement metrics — something hard to come by for anybody else’s stats — so let me share some of it with you.

Substack basically gives you seven different metrics to play with:

  • Likes

  • Comments

  • Shares (within the Substack ecosystem)

  • Total views (web pageviews + views in email + views on the Substack App)

  • Email open rate

  • Paid subscriptions

  • Free subscriptions

Let’s take everything Silver Bulletin has published in the past six months and see how they compare. I originally ran these numbers back to the start of 2026, but on second thought, I went ahead and included everything since Oct. 14, 2025 to introduce a little more variety in the form of things like the Nov. 2025 elections, the endgame of the government shutdown, some posts that touched a nerve with libs rather than conservatives3, the launch of our NFL model ELWAY4, plus a few things that were just plain weird.

The first five metrics are all fairly strongly correlated with one another, especially likes and shares, which almost exactly match. (The correlation coefficients shown here run on a scale from -1 to +1 from perfectly inversely correlated to perfectly correlated.) The one partial exception is email open rates, which may reflect the fact that people who read us in their email browsers tend not to engage in these other behaviors that are easier to do on the web or in the app.5

But how do these measures correlate with the number of paid subscriptions generated? Not very well at all.

If you look at every Silver Bulletin post over the past six months, the correlation between “engagement” metrics and paid signups is literally almost zero or, in some cases, even slightly negative. Now, I certainly wouldn’t say that paid subscriptions are the only thing we care about: if that were true, we’d probably paywall a (slightly6) higher share of posts than we do. I’d even like to think some of the things we publish are in the public interest in various ways, and those are almost always free. But paid subscriptions are unique in that they very literally require a higher commitment than any of these other things; it’s never been particularly easy to get people to pay for content or to go through the mechanics of punching in their credit card information.

To be fair, there are several confounders here. Our models7 often do disproportionately well as a driver of paid subscriptions and can also produce a disproportionate number of pageviews since they’re updated repeatedly. There’s inherently some trade-off between satisfying your existing customers (as measured by things like open rates) and expanding your reach to new ones. Most importantly, the correlations do turn positive — although they remain low — if you disaggregate between paid and free posts. Even though comments are always limited to paid subscribers at Silver Bulletin, free subscribers are also less likely to like or share posts that are paywalled.8

But the impulse to like a story and to pay for a story don’t really come from the same place. Liking and sharing posts, especially on Substack where likes are public, are almost definitionally a social behavior. Posts that get the most “engagement” on Silver Bulletin are often posts like this one or this one that contain a relatively simple hypothesis that is revealed by the headline/subject line and are broadly agreeable to a portion of the Silver Bulletin audience.9 But even though free posts can induce more paid subscriptions than you might assume, these high-engagement posts often do just OK relative to others.

In contrast, highly detailed deep-dives like our NBA draft model often do quite well for producing paid signups, even if it’s pretty lonely in the comments section. Many Silver Bulletin readers aren’t interested in things like our women’s March Madness predictions at all, but those who are might be quite intensively interested, enough that they’ll pay for distinctive work.

Now, to complete the story, free subscriptions do look much like the other metrics:

Still, it’s interesting that comments — equivalent to the replies that the X algorithm values most — are quite inferior to likes, shares and views even as predictors of free subscriptions. As you’re probably aware, here are some common templates you might encounter in Silver Bulletin comments:

  • There’s a typo in the first paragraph

  • Nate is an idiot

  • The other commenter is an idiot

  • WHY DOES ELWAY HATE THE CHICAGO BEARS?!?!

Don’t get me wrong: the first category is genuinely constructive and helpful. Still, you sometimes get more comments because you 1) screwed something up; 2) touched a nerve that you didn’t necessarily intend to touch; or 3) “disappointed” (i.e., pissed off) a portion of your user base because your take was half-baked. All of those are negative signifiers that a dumb algorithm might mistake for a positive one.

I was going to say that there’s a simple lesson here — don’t chase engagement. But it’s a little more subtle than that: don’t chase engagement for its own sake. Even if it sends some positive signal on average, it doesn’t always, and it deteriorates when you over-optimize for it. (See also Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”)

Especially for web content, many of your best customers don’t “engage” with it in any way other than in the most important way: by reading it. (And hopefully subscribing before long.) At Silver Bulletin, for every 1000 people who read a post, only about one likes, shares or comments on it. A platform that doesn’t value this “silent majority” of customers can become an algorithmic hellscape before long.

Silver Bulletin is a reader-supported publication. To receive new posts and impact these correlations, please consider becoming a subscriber.

1

For instance, I like that the Trending Topics module is now customized to the individual user, which reduces the amount of dogpiling to some extent. (Many people seem not to realize this; if you’re seeing a controversy about a niche subject in Trending Topics — or a controversy involving yourself — that doesn’t mean that the rest of the platform is.)

2

Note that Bluesky replicates much of the same behavior without an algorithmic feed.

3

Looking at our posts so far in 2026, I’ll grant you that most of our hot takes have been of a varietal that our more liberal readers will tend to like. But that’s what’s going to happen in a year that’s been bookended by Minneapolis and Iran, with Trump’s approval rating falling the whole time. There’s no deliberate attempt at balance, but we are responsive to the news.

4

Yes, ELWAY launched in the middle of the NFL season. Not ideal. But that’s when it was ready. We’ll have it in working order for the start of the season this year.

5

Also, we have extremely little variation in open rates; they stay within a narrow range. I’d like to think that’s because we have a pretty high threshold for sending out an email and for especially writing about subjects that are “off-topic”, even though the “off-topic” posts we do publish often wind up being popular.

6

There are diminishing returns to paid posts even if you strictly want to maximize revenues; it’s also important to draw in new customers since what’s clinically called the “conversion funnel” usually goes from lurker → free subscriber → paid subscriber.

7

This category includes predictive models like ELWAY and non-predictive ones like our Trump approval tracker.

8

Although we usually try to put some of the valuable content above the paywall so it’s not a waste of your time to open the email.

9

For us at least, perhaps because our audience is quite heterodox, I wouldn’t say there’s a strong correlation between whether the implicit sentiment in the headline is liberal or conservative and the number of likes, but having a clear thesis in either direction expressed in the headline probably helps.

Discussion about this video

User's avatar

Ready for more?