iOS A/B Testing: Strategies & Fundamentals

‘You can have the best technology, you can have the best business model, but if the storytelling isn't amazing, it won't matter. Nobody will watch.’ These words of Jeff Bezos ring true on the mobile marketing scene. App store optimization (ASO) and iOS A/B testing play a vital role in crafting compelling narratives for mobile apps.

We've already delved into the basics of creating the best ASO strategy and discussed the process of ASO A/B testing in detail. Today’s article will cover the fundamentals of iOS app A/B testing. We will be shedding light on how it can transform your mobile marketing approach. Keep reading to learn more about:

• the basics of iOS A/B testing;
• Apple Product Page Optimization (PPO) and its role in A/B testing;
• iOS elements you can test with Product Page Optimization;
• the difference between PPO & CPP;
• why A/B testing is good for App Store users;
• how to define what you aim to achieve through split testing;
• how to create meaningful variations for iOS split testing;
• the main steps that help you prepare for your A/B testing routine;
• how to implement A/B tests effectively;
• how to interpret data and draw insights;
• tips to effectively help you work within the App Store A/B testing limitations;
• the best practices and tools for successful iOS A/B testing.

‘Though they’ve been warned for decades about the dangers of overrelying on gut instinct and personal experience, managers keep failing to critically examine — much less challenge — the ideas their decisions are based on. To correct this problem they need to think and act like scientists.’ This quote from Stefan Thomke’s and Gary W. Loveman’s article called ‘Act Like a Scientist: Great Leaders Challenge Assumptions, Run Experiments, and Follow the Evidence' comes with one loud and clear idea: running tests produces hard evidence and thus, helps you make informed decisions. And that’s exactly what you need on your way to mobile marketing success.

So, you've created an awesome mobile app. Now, how do you make it stand out among fierce rivals? That's where iOS A/B testing comes to the rescue. This practice is about comparing two versions (A and B) of your app to see which one rings louder with your potential consumers. Below, we've provided a quick rundown of the basics:

• Why A/B Test? To find out what works and what doesn't. As Edison put it, 'I have not failed. I've just found 10,000 ways that won't work.'
• Clear Goals. Define what you want to achieve. Your app's success is like a treasure hunt. You need a map.
• Key Metrics. Identify what matters most. A wise marketer once said, 'You can't manage what you don't measure.'
• Craft Variations. Change elements like icons, screenshots, or preview vids. It's like trying different ingredients for the perfect dish.
• Test, Don't Guess. The data doesn't lie. A recent survey found that apps with A/B testing saw a 30% increase in user engagement.
• Analyze & Iterate. Learn from the data, tweak, and repeat. As Darwin might say, 'Adapt or become extinct.'

Back in 2021, Apple introduced iOS 15 at the World Wide Development Conference. Along with various new features, one that stood out was Product Page Optimization (PPO), a tool designed for app marketing.

With PPO, you no longer need to be a marketing genius to make your app succeed. It lets you optimize your creative assets through native A/B testing. It’s a major step forward in switching from external A/B testing tools and working closely with the tools offered on Google Play Store Listing Experiments.

App marketers can run tests in specific locations for up to 90 days. You can also decide what portion of your audience sees the test. It helps you focus your efforts better.

The whole practice aids you in comparing how well each version of your app page or store listing performs against the original one. You get valuable data on views, downloads, and how many people actually install your app. These insights help you figure out which images and content work best when you're advertising your app, increasing the chances of getting more people to install it. You can combine it with using the Asolytics competitor analysis tool for a maximized outcome or use the Asolytics App Dashboard solely to see how well the app assets are optimized.

Via PPO, you get to test three creative assets:

• App Icon. Experiment with different app icons to find the one that catches users' most focused attention.
• App Screenshots. Test an array of screenshots to determine which visual elements and captions your potential users fancy.
• App Video Previews. Create various video previews to see which one demonstrated your app's functionality in the most eye-catching light.

The option is up for grabs in App Store Connect to anyone with an iOS app. PPO grants you the possibility to work up 3 different variations of your default product page. Similar to A/B testing in Google Play, it helps you figure out what makes people more interested in your app. It's a great way to make your app page more appealing and get more people to check out and download your app from the App Store.

You’re probably aware that iOS 15 also launched a feature called Custom Product Pages (CPPs). CPPs allow you to replicate your main product page with variations in screenshots, preview videos, and promotional text. Generally speaking, CPPs grant you the flexibility to craft separate landing pages for each of your paid campaigns.

In contrast to the Product Page Optimization feature, Custom Product Pages come with the flexibility to generate a maximum of 35 unique pages for a single app. This opens up a wide array of customization possibilities. Numerous app marketers make the most of CPPs to improve their A/B testing strategies in the App Store. However, it's crucial to understand the distinction between CPPs and PPOs, based on the following parameters:

• Goals. Custom Product Pages is focused on boosting the number of users who sign up or make a purchase via customized user experience. This usually suggests Apple Search Ads and Facebook Ads. Product Page Optimization is aimed at experimentation with different product page designs — for the purpose of determining the variation of the main product page that works best.
• Traffic Sources. Custom Product Pages are accessible only through special web links and certain paid sources of traffic. You can't find them organically. They are designed for use with paid ad campaigns. With regard to Product Page Optimization, it uses a single source of traffic, and all visitors to the App Store product page are evenly divided between the main page and the different test versions.
• Testing & Statistical Significance. Yes, CPPs can be a nice addition to your App Store split testing endeavors. But we don’t recommend that you make them your main tool for testing. First, they are used with paid ads, which means you’ll need to pay for the traffic to your custom landing page. Second, to be more certain that your results are reliable, you might have to spend more money on your budget and keyword bids.

This, in its turn, can make it hard to test on a larger scale. And even after spending more, you might still end up with poorer results and a wasted budget. Product Page Optimization uses all the visitors to your page, doesn't cost anything to use or set up, and gives you trustworthy results. Yes, it does come with some restrictions. However, it's a better choice for A/B testing compared to Custom Product Pages.

With iOS A/B testing via Apple’s PPO, app publishers finally get a free and native tool that makes the most of App Store data and has dedicated analytics. Below, you’ll find reasons why A/B testing is good for App Store users:

• Better User Experience. A/B testing helps polish the app's interface.
• Optimized Performance. Through A/B testing, app developers can pinpoint and get rid of performance issues.
• Relevance. A/B testing allows developers to personalize content and, therefore, show users what's most relevant to their needs.
• Fewer Annoying Ads. Testing can help determine the right frequency and placement of ads. As a result, it can reduce the chances of annoying ad experiences.
• Speedier Load Times. Optimized apps load faster. This reduces user frustration and saves heaps of their time.

Bug Reduction. A/B testing can help you deal with app bugs. It will provide your users with a more stable experience.

Know your goals when iOS A/B testing. Below, you’ll find useful tips to follow in this context:

• Ask Yourself Questions & Analyze Answers. What exactly am I aimed at? App downloads? User engagement? Higher in-app purchases?
• Know Your Audience. Define the audience segments you're targeting. Are you trying to attract new users? Is it all about retaining existing ones? Trying to re-engage lapsed customers, perhaps?
• Focus on KPIs. Determine the key performance indicators representing your goals. Maintain laser-like focus on click-through rates, app conversion rates, and revenue per user.
• Evaluate Progress. Establish standards backed by historical data or industry standards to assess the success of your A/B tests.
• Prioritize Changes. Identify the elements of the product page that your goals are dependent on the most.
• Understand Potential Limitations. Take into consideration the available time, budget, and team resources you have on hand for conducting split tests.
• Create a Hypothesis. Formulate hypotheses about how changes will influence your application’s user behavior.
• Plan Testing Duration. Set a realistic deadline for running tests and make sure you make it to collect enough data.

Below, you'll find some tips on creating variations like a pro:

• Test one element at a time. It allows you to pinpoint what evokes changes in user behavior patterns.
• Make sure that every alteration serves a certain purpose. For example, if you change the app icon, think about how it can better convey your app's core value.
• Reap the perks from user feedback when working up variations. Analyze typical complaints/suggestions to take the overall user experience to the next level.
• Analyze user behavior patterns. This will help you determine pain points or areas that can be enhanced with variations.
• Be consistent with your app's branding and communication. Your variations should represent your app's unique ‘character.’
• Make changes in accordance with the conclusions you arrive at after each test. Yes, creativity is crucial here. But remember that your variations should majorly rely on data and what your users fancy.

Before you start with Product Page Optimization, follow the same steps you'd take for any A/B test. Below, you’ll find a handy checklist to help you prepare:

• Plan Your A/B Test. Think about what, why, and how you want to test before actually setting it up. This planning is crucial when you're forming your testing strategy and hypotheses.
• Focus on Your Target Market. PPO gives you limited reporting data. Therefore, it's a good idea to concentrate on one specific location at a time.
• Match User Intent. Figure out which keywords usually drive traffic to your app by researching the App Store. The App Store doesn't provide this data, it’s true. But you can always look at the most popular keywords in your category or use the brilliant Asolytics keyword tool for maximized effectiveness.

Apple’s Product Page Optimization functions like a typical A/B testing tool. Follow these steps to make the most of split testing with PPO:

Just like Google Play's Store Listing Experiments, the App Store lets you pick how your test will work. That includes the different versions, how many people will see them, and where you want to target. Apple then calculates how long it will take to see the results you want based on what improvement you're aiming for and how many people have visited your product page recently. You can only have one test going at a time, and it can run for up to 90 days.

But you have the flexibility to stop it whenever you like. Upon the beginning of the test, you can't make any changes to it. If you release a new app version, the test will automatically end.

During this step, Apple will ask you to provide the following info:

• your tests' reference name;
• number of variations you will use;
• traffic proportion between the main product page and testing variations. Like Google Play's A/B testing, each variation's traffic is divided equally;
• localizations selection;
• how long the test will last.

In the words of Ron Kohavi, who is an authority in experimentation and has worked on controlled experiments, machine learning, search, personalization, and AI for nearly 3 decades, ‘However beautiful the strategy, you should occasionally look at the results.’ During this stage, you will head to the App Analytics section in App Store Connect and take a look at your test's progress. You’ll get the full picture on the following data:

• how many people have seen your test pages;
• how many of them took action;
• as well as the percentage of improvement.

Also, you will analyze how reliable the results are in contrast to your main product page. If you have several test versions, you can easily compare their performance specifics. The App Store will also give you useful tips, for example, whether the test needs more data to be reliable and which version is performing the best.

It’s time to make the right decision. At this stage, you need to choose whether to keep your existing product page as is or switch to one of the test versions. If you decide to apply a variation, the test will come to an end. To make the right decision, be certain that the new version is noticeably better than your original product page. This conclusion should come with a 90% confidence level.

Bonus Tip. The Asolytics app analysis platform offers valuable reports on keyword volumes, competition, ad keywords, and essential App Store insights. Product Page Optimization doesn't use keywords. But understanding why users visit your product page enhances your testing. You can also link your Google Play Console to Asolytics to identify keywords driving Android installs. Even with differences between Android and iOS users, grasping Google Play data and the keywords responsible for organic installs remains beneficial for your app's success.

If you want to regularly test App Store elements, new concepts, and metadata while targeting an array of locations at once, Product Page Optimization will feel limited. Below, we’ve listed these limitations:

At the end of the day, Apple’s Product Page Optimization alone cannot provide you with the best A/B testing iOS app platform. However, it does come with certain perks and can be used to take your app performance to the next level — in combination with top-notch external A/B testing tools.

For example, you can give Firebase A/B testing iOS a try. It’s a mobile app testing and optimization framework provided by Google's Firebase platform. Firebase allows app developers to conduct experiments to test different variations of app features, content, or user experiences on iOS devices. It comes with a treasure trove of tools for setting up experiments, defining goals, monitoring results, and ultimately optimizing the app.

If you’ve practiced A/B testing for some time, chances are you know exactly how it works in your specific case. If you’re a beginner, below, we’ve rounded up the best App Store Split testing tips and beneficial examples of iOS A/B testing strategies for you:

At the end of the day, with the right approach, iOS A/B testing is the playground where tweaks become triumphs. Split testing-wise, consistency, and data are your most loyal allies. Remember, it's not just about what you test but how you test it, too. Now that you’re aware of the basics and specifics of App Store split testing, feel free to switch from theory to practice.