Dear YouTube: Video A/B Testing Is Going Too Far

I think YouTube spends too much time chasing other platforms when it should be doubling down on what makes YouTube, YouTube.

There have been a lot of changes over the past couple of years. Some affect the people making videos, some affect the people playing them, and some are genuinely great. But the announcement of video A/B testing might be one of YouTube’s riskiest ideas yet.

Not thumbnail testing. Not title testing. Testing different versions of the actual video, under the same upload, to figure out which cut holds attention longest.

I understand the pitch. I also think it creates some pretty fundamental problems, both for the community around a video and for the creative process behind it.

What YouTube Video A/B Testing Is Supposed to Do

YouTube introduced the feature at its Made on YouTube event, alongside custom homepages, new livestream features, and AI tools for editing Shorts. I like that these events exist. Proactively explaining changes to the platform is a good thing, even when some of those changes are features I might never use.

The pitch for video A/B testing is straightforward: say you recorded two different opening hooks and cannot decide which one works better. Instead of making that decision before publishing, you upload multiple cuts. YouTube distributes those versions to different portions of the audience, compares their performance, and helps you choose a winner.

The announced concept applies to both Shorts and long-form videos. The idea is to compare audience retention across the cuts and let the creator select the version that holds attention best.

On the surface, this looks like a logical extension of thumbnail and title testing.

I took a little while to warm up to thumbnail A/B testing, but I have used it a handful of times and learned something about which packaging people respond to. I have tried title tests once or twice, too.

But there is an important distinction: changing the packaging is not the same as changing the thing inside it. Different thumbnails can lead people to the same video. Different video cuts mean they might not have the same experience once they get there.

The First Problem: How Different Can the Cuts Be?

Testing two slightly different openings is probably the most innocent possible use of this feature. You have two hooks, both introduce the same subject, and you want help choosing one. Fine. That is the generous version of the pitch.

But my experience with thumbnail testing makes me question how useful tiny video differences will actually be.

When I test thumbnails with very small changes, such as slightly larger text, a different arrow position, or an arrow versus no arrow, the results often are not far enough apart to teach me much. Sometimes the test ends without a conclusive result.

When I test two completely different thumbnail concepts, that is usually where the performance differences become large enough to learn something.

Now apply that logic to the video itself. I am skeptical that swapping two or three shots in an opening sequence will consistently produce a meaningful statistical difference. If the useful results require bigger changes, how far will creators be allowed to go?

Even YouTube’s own demonstration included cuts with substantially different runtimes, roughly 17 minutes versus 19 minutes. That is not just moving an arrow.

And different runtimes immediately raise another question: what happens to timestamps?

What YouTube’s Engineers Told Me

To YouTube’s credit, I got on a call with its engineers and asked about these concerns. There are some planned guardrails, and they matter.

A similarity model would limit the differences

The engineers explained that the planned broader rollout would use a semantic analysis model to examine the uploaded video files and determine how similar they are. Only cuts that meet a sufficiently high similarity threshold would qualify for a test.

In theory, that means someone could not upload completely unrelated videos and call them variations of the same thing.

I hope that works. Really.

But I am not automatically confident in a model’s ability to understand the important differences between two videos. My experience asking Gemini to find things inside YouTube videos has not been particularly reassuring. It often seems to rely on the spoken words rather than fully understanding everything happening on screen.

Still, a similarity check is something. It is an attempt to limit the feature instead of making it an unrestricted replacement tool.

Tests would be limited to a new video’s launch

The other important restriction is timing. The engineers said testing would only be allowed for a limited period at the launch of a new upload.

So this would not be a tool for going through a channel’s back catalog and replacing old videos with different cuts. That answers one of my biggest initial questions.

Replacing a video has historically been a special exception

YouTube has already had a tightly restricted ability to replace a video file while keeping the same URL. Almost nobody gets to use it. It generally requires a YouTube contact and a specific justification, such as correcting one factual error while leaving everything else alone.

An Apple commercial with a visible glitch is an example: the problem attracted attention, then the glitch disappeared from the existing upload.

As I understand it, those replacement files have had to preserve the original length, with only the specific correction changed. There is a reason that capability has been treated carefully. Changing what exists at an established video URL is a big deal.

The Questions Those Guardrails Do Not Solve

A similarity threshold and a launch window help define the boundaries. They do not resolve the experience inside those boundaries.

Based on those conversations, people would not be told that the video they received was part of an A/B test. Creators also would not know which cut a particular commenter had been served.

That leaves several practical problems:

• Comments lose context. Someone can react to a moment that is different or absent in another cut, without anyone knowing which version they mean.
• Timestamped comments can become confusing. If scenes move or runtimes change, a timestamp may no longer point to the moment being discussed.
• Sharing becomes less predictable. Sending someone a link may not guarantee they receive the same cut you did.
• Returning to a video can mean finding a different version. A person could receive a losing cut during the test, then return after a different cut becomes the permanent selection.

I do not think all of those issues had clear answers yet. The timestamp problem, in particular, seems hard to avoid when the cuts have different lengths.

Then there is the question of what the similarity model actually considers meaningful.

Suppose two cuts differ by only one sentence, but that sentence changes the conclusion. Does the model catch that? What if both files contain essentially the same material, but the scenes appear in a different order? Does that qualify as the same story?

A numerical similarity score does not automatically tell us whether two versions mean the same thing. We will have to see how that works in practice.

YouTube Works Because There Is a Shared Experience

Underneath the technical questions is something less measurable.

There is an unspoken agreement around a YouTube upload: we are all discussing the same thing. Maybe a different title or thumbnail brought each of us there, but the video itself is the common reference point.

That is part of how a video becomes a cultural moment. People quote it, reference it, discuss its conclusion, and point one another toward particular scenes. The conversation works because there is a shared object at its center.

Serving two or three different versions simultaneously weakens that agreement.

It does not have to turn into some dramatic abuse case to feel wrong. Even the ordinary, intended version of the feature introduces uncertainty into something that used to be simple: this link leads to this video.

A Bigger Pattern: Chasing What Other Platforms Do

This concern is not happening in isolation. To me, it fits a broader pattern of YouTube giving up useful distinctions while responding to its competitors.

Take public dislike counts. YouTube framed their removal around protecting creators from harassment and targeted dislike campaigns. That was its stated reasoning, as explained in its announcement about dislikes.

But Facebook, Instagram, and TikTok do not display an equivalent public dislike count either. Whatever the motivation, YouTube ended up looking more like those platforms while losing something that was uniquely useful: the visible like-to-dislike ratio.

We have also discussed more liberal play counting on the Waveform podcast, including counting a play as soon as the first frame loads. My concern is what gets lost when the definition of a view moves in that direction.

YouTube’s more selective approach to measuring a real view gave those numbers a different meaning. A million plays on TikTok and a million YouTube views were not interchangeable, and we understood that.

Then there is the attention paid to television and getting late-night hosts onto the platform. None of these things is exactly the same as video A/B testing. But together, they feed the same frustration: why spend so much energy resembling other media when YouTube already has something special?

Dear Creators: Better Retention Is Not Automatically a Better Video

I expect YouTube will move forward with this feature. So the more useful question for creators might be whether we should use it.

For most of us, I do not think we need to. I would go further: most of us probably should not.

The feature is presented as a way to improve videos. More precisely, it is a way to identify which version has the strongest audience retention. Those are not necessarily the same thing.

Hot take, apparently.

I have made plenty of creative decisions that were almost certainly not the most retention-optimized choices available. So have lots of other creators. Sometimes a choice stays in because it is fun, unusual, entertaining, or full of personality.

There will always be channels built around reaching the largest possible audience and optimizing every detail toward that goal. That is fine. But mass appeal is not the whole point of YouTube.

A lot of its appeal comes from the opposite: the small hobby, the niche interest, the unusual discovery, the subject only a handful of people are talking about. Those things are interesting partly because they are specific and personal.

If every channel starts optimizing toward the same retention objective, there is a real risk of sanding away the qualities that made those channels worth finding.

Keep trimming whatever is not the most efficient way to tell the story, and eventually some of the creativity goes with it.

Make the Video, Learn From It, Make the Next One

The skill of being a creator is not simply finding the most efficient possible way to hold attention. It is finding creative ways to share, teach, explore, explain, or review something.

For most creators, I think making, finishing, and publishing more videos is a better use of time than producing endless variations of one upload. Multiple titles, multiple thumbnails, multiple cuts, all to squeeze out the last few percent of performance.

There is another way to improve:

1. Make deliberate creative choices.
2. Finish the video and publish it.
3. Learn from what worked and what did not.
4. Bring those lessons into the next thing you make.

That is not an argument against feedback. It is an argument for learning through a body of work instead of treating every individual piece as an experiment that must be optimized until the numbers approve.

Stand behind the choices you made. Some videos will go well. Some will not. Both can teach you something.

Maybe that is just the old-school YouTuber in me. Maybe this is my old-man-yells-at-cloud moment. But I still think there is value in making something with conviction, putting it into the world, and then making the next thing.

YouTube should protect the things that make it different. And creators should not have to optimize all the personality out of their work to feel like they are doing it right.