Every AI Marketing Test is a Risk to the Unprepared

I think this gets to the heart of a serious problem in the technical approach in marketing right now: you can A/B test two bad options and one will still "win". We use a lot of scientific terminology when we talk about marketing optimization, but it really isn't usually performed with scientific rigor. There are a lot of vendors out there that will give you everything you ask for when testing creative and not give you what you need.

The truth of the matter is these systems will drive marketers into even more detailed and small cadres to test against, as size of the sample goes down your capacity to be sure of any sort of testing goes down with it, so even these A/B tests are increasingly likely to give you bad signals. An inability to tell where failures are occurring or why serves the vendor. It's easy to see why Facebook might push marketers into using this tool without them understanding it: it hides the flaws of their platform and system.

Even if you still believe you have accuracy in these small personalized groups, the big challenge here is deciding on what your tests are and what your controls are.

Two bad ideas can still have a better idea, but that doesn't make it a good idea. As we've seen with the impact of personalized sorting via algorithm on political discourse, it's very easy for automation--if you put your entire business in its hands--to lead you down a bad path for your brand and your performance with no clear warning signals. I encourage people not to put their entire trust in tools like these without engaging their critical facilities and maintain the supervision of experts. A/B testing of AI marketing materials can lie to you just as smoothly and undetectably as any generative AI system. The danger here of AI slopping your marketing campaign down a hole of decreasing performance, and then losing track of why, is high.

As AI slop goes, this is relatively tame fare, but it is notable that a company as big as Ticketmaster is using generative AI for its Facebook ads. It is also an instructive example that shows a big reason why Facebook and Google are bringing in so much revenue right now, and highlights the fact that social media is not so slowly being completely drowned in low-effort AI content and ads.