A/B testing with AI content generation

AI content generation is finding its place throughout the marketing stack. Suggesting engaging headlines, writing email subject lines, assisting customer support teams, and writing full-blown articles are just a few examples.

Demand generation activity like capturing leads with gated content is also ripe for getting an efficiency boost from AI content generation.

There are many different strategies for lead generation, but one common approach especially for enterprise companies, is to host landing pages that lead to informative educational assets like white papers, case studies, industry overviews, or specialized data-driven dashboards.

The goal is to provide real value to your visitors while trying to learn something about them and how they might fit into your marketing funnel. If a number of people demonstrate interest in a particular asset, you can assume that some percent of those people are qualified candidates to purchase your product or service.

However, how do you know you’ve designed an effective landing page that will lead a visitor to your asset and into your sales funnel?

You will learn over time what works for your company, customer, and product but it’s often quicker to perform segmented A/B tests on landing page design and copy to run small-scale experiments on the traffic that lands on your page.

There are a few of components of a good landing page that matter. Elements like the headline, the description of the asset, the form you present to the user, and the image you display will all affect conversion.

Generating multiple versions of a single landing page can be difficult and time consuming. It’s hard to create copy with enough variation to see any real differences. Everything ends up reading and looking pretty much the same. After all, you are the expert and you pretty much know what is going to work and what isn’t. If you want to really test the edge cases and try some out-of-the-box messaging, using AI-generated content is an effective approach to get you thinking along some perpendicular lines of creativity.

Variation in A/B Testing

For A/B testing (or split testing), sufficient variation is critical. If you generate 5 versions of a headline that are all basically the same, you might see some differences in engagement, but it will be hard to know the reason behind it.

Busting out of your comfort zone and publishing something wildly different might be what’s needed to converge on the magic formula for your product and your market.

A blessing and a curse of AI-generated content is that it can spit out some really weird stuff. This isn’t totally surprising because the AI models are trained on text from all across the internet including places like Reddit. Generally we try to avoid the weird stuff, but sometimes it’s good to turn up the weirdness temperature and see what the AI sends our way. By doing this, we increase the variation in our A/B testing and increase our chances of uncovering a magical combination of text blocks that resonates with our audience.

There are many ways of evaluating how similar two pieces of text are. This can be useful when evaluating the variance of scenarios in your A/B test. One approach is to essentially count the number of additions and subtractions that are required to get from one piece of text to another. This metric is called the Levenshtein Distance. Another option is the cosine similarity metric where you convert each piece of text to a numeric vector, and calculate how much overlap those two vectors have. Two pieces of AI-generated content likely have a weaker overlap than what you would think of yourself.

Examples of AI-Generated Headlines

One way to generate a few landing page options for your A/B test is to start with the title of an asset and use Automata’s Asset Expansion platform to review a few options.

Here are five AI-generated landing page headlines you could use for the following asset: Architecting the Zero Trust Enterprise by Palo Alto Networks:

  • Do not compromise security with technology you cannot trust
  • You’re an attacker now, what are you going to do?
  • Eliminate Shadow IT by delivering an intelligent Zero Trust IT infrastructure
  • Meet the critical systems, processes, and environments that protect your business from malicious insiders
  • More than your physical walls keep your company safe from hackers

Although some may not fit the bill for your company’s branding and tone, there is certainly a large degree of variability in these options. Even if you’re not able to use these outputs verbatim, these examples might offer a point of view outside of your own.

When it comes to designing an A/B test for digital assets like landing pages, it’s critical to optimize the amount of variability in the tests to have a statistically different outcome for each case. AI language generation platforms can help produce text with enough variability to identify patterns of success for your unique product and audience.