iOS A/B Testing: Strategies & Fundamentals

‘You can have the best technology, you can have the best business model, but if the storytelling isn’t amazing, it won’t matter. Nobody will watch.’ These words of Jeff Bezos ring true on the mobile marketing scene. App store optimization (ASO) and iOS A/B testing play a vital role in crafting compelling narratives for mobile apps.

We’ve already delved into the basics of creating the best ASO strategy and discussed the process of ASO A/B testing in detail. Today’s article will cover the fundamentals of iOS app A/B testing. We will be shedding light on how it can transform your mobile marketing approach. Keep reading to learn more about:

• the basics of iOS A/B testing;
• Apple Product Page Optimization (PPO) and its role in A/B testing;
• iOS elements you can test with Product Page Optimization;
• the difference between PPO & CPP;
• why A/B testing is good for App Store users;
• how to define what you aim to achieve through split testing;
• how to create meaningful variations for iOS split testing;
• the main steps that help you prepare for your A/B testing routine;
• how to implement A/B tests effectively;
• how to interpret data and draw insights;
• tips to effectively help you work within the App Store A/B testing limitations;
• the best practices and tools for successful iOS A/B testing.

‘Though they’ve been warned for decades about the dangers of overrelying on gut instinct and personal experience, managers keep failing to critically examine — much less challenge — the ideas their decisions are based on. To correct this problem they need to think and act like scientists.’ This quote from Stefan Thomke’s and Gary W. Loveman’s article called ‘Act Like a Scientist: Great Leaders Challenge Assumptions, Run Experiments, and Follow the Evidence’ comes with one loud and clear idea: running tests produces hard evidence and thus, helps you make informed decisions. And that’s exactly what you need on your way to mobile marketing success.

The Basics of iOS A/B Testing

So, you’ve created an awesome mobile app. Now, how do you make it stand out among fierce rivals? That’s where iOS A/B testing comes to the rescue. This practice is about comparing two versions (A and B) of your app to see which one rings louder with your potential consumers. Below, we’ve provided a quick rundown of the basics:

• Why A/B Test? To find out what works and what doesn’t. As Edison put it, ‘I have not failed. I’ve just found 10,000 ways that won’t work.’
• Clear Goals. Define what you want to achieve. Your app’s success is like a treasure hunt. You need a map.
• Key Metrics. Identify what matters most. A wise marketer once said, ‘You can’t manage what you don’t measure.’
• Craft Variations. Change elements like icons, screenshots, or preview vids. It’s like trying different ingredients for the perfect dish.
• Test, Don’t Guess. The data doesn’t lie. A recent survey found that apps with A/B testing saw a 30% increase in user engagement.
• Analyze & Iterate. Learn from the data, tweak, and repeat. As Darwin might say, ‘Adapt or become extinct.’

Apple Product Page Optimization & A/B Testing on iOS

Back in 2021, Apple introduced iOS 15 at the World Wide Development Conference. Along with various new features, one that stood out was Product Page Optimization (PPO), a tool designed for app marketing.

With PPO, you no longer need to be a marketing genius to make your app succeed. It lets you optimize your creative assets through native A/B testing. It’s a major step forward in switching from external A/B testing tools and working closely with the tools offered on Google Play Store Listing Experiments.

App marketers can run tests in specific locations for up to 90 days. You can also decide what portion of your audience sees the test. It helps you focus your efforts better.

The whole practice aids you in comparing how well each version of your app page or store listing performs against the original one. You get valuable data on views, downloads, and how many people actually install your app. These insights help you figure out which images and content work best when you’re advertising your app, increasing the chances of getting more people to install it. You can combine it with using the Asolytics competitor analysis tool for a maximized outcome or use the Asolytics App Dashboard solely to see how well the app assets are optimized.

iOS Elements You Can Test With Product Page Optimization

Via PPO, you get to test three creative assets:

• App Icon. Experiment with different app icons to find the one that catches users’ most focused attention.
• App Screenshots. Test an array of screenshots to determine which visual elements and captions your potential users fancy.
• App Video Previews. Create various video previews to see which one demonstrated your app’s functionality in the most eye-catching light.

The option is up for grabs in App Store Connect to anyone with an iOS app. PPO grants you the possibility to work up 3 different variations of your default product page. Similar to A/B testing in Google Play, it helps you figure out what makes people more interested in your app. It’s a great way to make your app page more appealing and get more people to check out and download your app from the App Store.

PPO & CPP: What’s the Difference?

You’re probably aware that iOS 15 also launched a feature called Custom Product Pages (CPPs). CPPs allow you to replicate your main product page with variations in screenshots, preview videos, and promotional text. Generally speaking, CPPs grant you the flexibility to craft separate landing pages for each of your paid campaigns.

In contrast to the Product Page Optimization feature, Custom Product Pages come with the flexibility to generate a maximum of 35 unique pages for a single app. This opens up a wide array of customization possibilities. Numerous app marketers make the most of CPPs to improve their A/B testing strategies in the App Store. However, it’s crucial to understand the distinction between CPPs and PPOs, based on the following parameters:

• Goals. Custom Product Pages is focused on boosting the number of users who sign up or make a purchase via customized user experience. This usually suggests Apple Search Ads and Facebook Ads. Product Page Optimization is aimed at experimentation with different product page designs — for the purpose of determining the variation of the main product page that works best.
• Traffic Sources. Custom Product Pages are accessible only through special web links and certain paid sources of traffic. You can’t find them organically. They are designed for use with paid ad campaigns. With regard to Product Page Optimization, it uses a single source of traffic, and all visitors to the App Store product page are evenly divided between the main page and the different test versions.
• Testing & Statistical Significance. Yes, CPPs can be a nice addition to your App Store split testing endeavors. But we don’t recommend that you make them your main tool for testing. First, they are used with paid ads, which means you’ll need to pay for the traffic to your custom landing page. Second, to be more certain that your results are reliable, you might have to spend more money on your budget and keyword bids.

This, in its turn, can make it hard to test on a larger scale. And even after spending more, you might still end up with poorer results and a wasted budget. Product Page Optimization uses all the visitors to your page, doesn’t cost anything to use or set up, and gives you trustworthy results. Yes, it does come with some restrictions. However, it’s a better choice for A/B testing compared to Custom Product Pages.

Why A/B Testing Is Good for App Store Users

With iOS A/B testing via Apple’s PPO, app publishers finally get a free and native tool that makes the most of App Store data and has dedicated analytics. Below, you’ll find reasons why A/B testing is good for App Store users:

• Better User Experience. A/B testing helps polish the app’s interface.
• Optimized Performance. Through A/B testing, app developers can pinpoint and get rid of performance issues.
• Relevance. A/B testing allows developers to personalize content and, therefore, show users what’s most relevant to their needs.
• Fewer Annoying Ads. Testing can help determine the right frequency and placement of ads. As a result, it can reduce the chances of annoying ad experiences.
• Speedier Load Times. Optimized apps load faster. This reduces user frustration and saves heaps of their time.

Bug Reduction. A/B testing can help you deal with app bugs. It will provide your users with a more stable experience.

How to Define What Your Aims Through iOS A/B Testing

Know your goals when iOS A/B testing. Below, you’ll find useful tips to follow in this context:

• Ask Yourself Questions & Analyze Answers. What exactly am I aimed at? App downloads? User engagement? Higher in-app purchases?
• Know Your Audience. Define the audience segments you’re targeting. Are you trying to attract new users? Is it all about retaining existing ones? Trying to re-engage lapsed customers, perhaps?
• Focus on KPIs. Determine the key performance indicators representing your goals. Maintain laser-like focus on click-through rates, app conversion rates, and revenue per user.
• Evaluate Progress. Establish standards backed by historical data or industry standards to assess the success of your A/B tests.
• Prioritize Changes. Identify the elements of the product page that your goals are dependent on the most.
• Understand Potential Limitations. Take into consideration the available time, budget, and team resources you have on hand for conducting split tests.
• Create a Hypothesis. Formulate hypotheses about how changes will influence your application’s user behavior.
• Plan Testing Duration. Set a realistic deadline for running tests and make sure you make it to collect enough data.

How to Create Meaningful Variations for iOS Split Testing

Below, you’ll find some tips on creating variations like a pro:

• Test one element at a time. It allows you to pinpoint what evokes changes in user behavior patterns.
• Make sure that every alteration serves a certain purpose. For example, if you change the app icon, think about how it can better convey your app’s core value.
• Reap the perks from user feedback when working up variations. Analyze typical complaints/suggestions to take the overall user experience to the next level.
• Analyze user behavior patterns. This will help you determine pain points or areas that can be enhanced with variations.
• Be consistent with your app’s branding and communication. Your variations should represent your app’s unique ‘character.’
• Make changes in accordance with the conclusions you arrive at after each test. Yes, creativity is crucial here. But remember that your variations should majorly rely on data and what your users fancy.

Getting Ready for A/B Testing iOS Apps

Before you start with Product Page Optimization, follow the same steps you’d take for any A/B test. Below, you’ll find a handy checklist to help you prepare:

• Plan Your A/B Test. Think about what, why, and how you want to test before actually setting it up. This planning is crucial when you’re forming your testing strategy and hypotheses.
• Focus on Your Target Market. PPO gives you limited reporting data. Therefore, it’s a good idea to concentrate on one specific location at a time.
• Match User Intent. Figure out which keywords usually drive traffic to your app by researching the App Store. The App Store doesn’t provide this data, it’s true. But you can always look at the most popular keywords in your category or use the brilliant Asolytics keyword tool for maximized effectiveness.

• Set the Right Test Duration. Be patient and allow at least 7 days to gather data and discover a better-performing version.
• Limit External Influences. Avoid starting a new advertising campaign at the same time as your Product Page Test. Keep in mind that your test uses all traffic sources, so a significant increase in paid ads could affect the results.
• Consider Seasonal Factors. Run your campaign for a duration that smooths out seasonal effects (holidays or weekends), which could distort your results.
• Test Both Small and Big Changes. Small changes are great for your ASO strategy. But if you’re looking for a major conversion boost, don’t be afraid to try significant alterations in your testing (redesigned app icons, new fonts in screenshots, or more focused videos).

A/B Testing iOS Apps With PPO: How-to Guide

Apple’s Product Page Optimization functions like a typical A/B testing tool. Follow these steps to make the most of split testing with PPO:

1. Set up your test

Just like Google Play’s Store Listing Experiments, the App Store lets you pick how your test will work. That includes the different versions, how many people will see them, and where you want to target. Apple then calculates how long it will take to see the results you want based on what improvement you’re aiming for and how many people have visited your product page recently. You can only have one test going at a time, and it can run for up to 90 days.

But you have the flexibility to stop it whenever you like. Upon the beginning of the test, you can’t make any changes to it. If you release a new app version, the test will automatically end.

During this step, Apple will ask you to provide the following info:

• your tests’ reference name;
• number of variations you will use;
• traffic proportion between the main product page and testing variations. Like Google Play’s A/B testing, each variation’s traffic is divided equally;
• localizations selection;
• how long the test will last.

2. Monitor your test’s performance

In the words of Ron Kohavi, who is an authority in experimentation and has worked on controlled experiments, machine learning, search, personalization, and AI for nearly 3 decades, ‘However beautiful the strategy, you should occasionally look at the results.’ During this stage, you will head to the App Analytics section in App Store Connect and take a look at your test’s progress. You’ll get the full picture on the following data:

• how many people have seen your test pages;
• how many of them took action;
• as well as the percentage of improvement.

Also, you will analyze how reliable the results are in contrast to your main product page. If you have several test versions, you can easily compare their performance specifics. The App Store will also give you useful tips, for example, whether the test needs more data to be reliable and which version is performing the best.

3. Choose the best version

It’s time to make the right decision. At this stage, you need to choose whether to keep your existing product page as is or switch to one of the test versions. If you decide to apply a variation, the test will come to an end. To make the right decision, be certain that the new version is noticeably better than your original product page. This conclusion should come with a 90% confidence level.

Bonus Tip. The Asolytics app analysis platform offers valuable reports on keyword volumes, competition, ad keywords, and essential App Store insights. Product Page Optimization doesn’t use keywords. But understanding why users visit your product page enhances your testing. You can also link your Google Play Console to Asolytics to identify keywords driving Android installs. Even with differences between Android and iOS users, grasping Google Play data and the keywords responsible for organic installs remains beneficial for your app’s success.

Analyze your app performance in Asolytics

10 App Store A/B Testing Limitations to Be Aware Of

If you want to regularly test App Store elements, new concepts, and metadata while targeting an array of locations at once, Product Page Optimization will feel limited. Below, we’ve listed these limitations:

1. PPO allows for just one test simultaneously. You can run one test, but you can aim it at different locations. If you realize that a single test needs at least a week to get meaningful results, and you have many things to try out, it will take quite a bit of time to complete all your A/B tests.
2. You can’t test metadata. Although trying out icons, screenshots, and preview videos is good, it’s a shame you can’t test other elements. We’d really want to test iOS subtitles, for instance, to see how they affect how often people can find our app.
3. The App Store should let us see how our tests do in each location. Although we can aim tests at different places, we can’t easily compare their performance. If, for example, we want to test app screenshots in German, French, and Spanish, we have to do three tests in a row, taking at least 4 weeks (assuming one week per test plus extra days for submissions). We need a way to analyze results for each location separately.
4. In the App Store, you can’t use just one traffic source for testing. PPOs will take all your traffic and split it between the main product page and the test versions.
5. Every PPO test you want to run has to be approved by the Apple team. At a first glance, it might not sound like a big deal. However, some app marketers have mentioned that this approval process can sometimes take even longer than when you submit a new app version regularly.
6. Testing app icons can be quite complicated. First, you have to add a different icon to your app files for the next release. Then, you must wait for the test before launching the new app version. After that, you need to assess how well this new icon does. Even if it’s better, you can’t use it right away. You have to include it in the next app submission and make it the default icon. Long story short, the whole icon testing process requires careful planning and can take at least a couple of weeks to get it done correctly.
7. What you’re testing is only shown to iOS 15 users. This problem becomes less noticeable over time, especially taking into consideration the arrival of iOS 16. But as of today, it means that some users may not participate in your A/B test.
8. App Store the highest confidence score you can get is 90%. We’d like the freedom to pick our own level, similar to what Store Listing Experiments offers. This is important because the industry standard is to use a 95% confidence level.
9. PPOs require a hefty amount of data to be trustworthy. The best testing time is around 1-2 weeks. If it’s shorter, you won’t get good results. If it’s longer, other things can mess with the test. So, you have to watch the tests closely to get it right.
10. If you discover a bug and have to unexpectedly release a new app version, your App Store PPO test will stop automatically. If your product team schedules new app releases every week, it becomes really tough to conduct PPOs the right way. The app updates will always come first and make it difficult to test properly.

At the end of the day, Apple’s Product Page Optimization alone cannot provide you with the best A/B testing iOS app platform. However, it does come with certain perks and can be used to take your app performance to the next level — in combination with top-notch external A/B testing tools.

For example, you can give Firebase A/B testing iOS a try. It’s a mobile app testing and optimization framework provided by Google’s Firebase platform. Firebase allows app developers to conduct experiments to test different variations of app features, content, or user experiences on iOS devices. It comes with a treasure trove of tools for setting up experiments, defining goals, monitoring results, and ultimately optimizing the app.

App Store Split Testing: 5 Best Tips

If you’ve practiced A/B testing for some time, chances are you know exactly how it works in your specific case. If you’re a beginner, below, we’ve rounded up the best App Store Split testing tips and beneficial examples of iOS A/B testing strategies for you:

1. Avoid making too many changes during A/B testing Keep the ‘control group,’ the part of your test that doesn’t change, as stable as possible. Even though you can’t update things like keywords and descriptions without stopping your PPO test, try to keep your paid advertising campaigns consistent so that you don’t disrupt the sources of traffic too much.
2. Know exactly what you’re testing and why This is especially important in the App Store, given the limitations we’ve talked about earlier.
3. If you want to include organic traffic in your tests, additionally use Custom Product Pages We’ve mentioned before that CPPs can work together with your App Store testing. If you’re focusing on testing organic traffic exclusively, you can configure your CPPs to receive all your paid traffic. This way, your tests will primarily have organic search and browsing traffic.
4. Plan well in advance To reap all the perks from Product Page Optimization, plan your app releases and decide on the tests you will run in between them. Get everything ready for testing before your next app update, so you can make the most of the time in between submissions.
5. Adhere to these basic rules when using PPO:
   • analyze your competitors nonstop and adapt your test results to match market trends;
   • get the gist of where your traffic comes from to see which source currently suits you best;
   • concentrate on one location or a few similar ones, this will help you work within the aforementioned PPO limitations.

To Wrap Up

At the end of the day, with the right approach, iOS A/B testing is the playground where tweaks become triumphs. Split testing-wise, consistency, and data are your most loyal allies. Remember, it’s not just about what you test but how you test it, too. Now that you’re aware of the basics and specifics of App Store split testing, feel free to switch from theory to practice.

FAQ