Justin Allen
An indie iOS developer
Where to hear and read about Justin Allen1 episode or test
Justin Allen is an indie iOS developer who tests his app icons with Apple's own tools before launch. The lesson most worth taking is to put every icon variant in a reviewed build, then run a Product Page Optimization test in pre-order so real data picks the launch icon.
for app icon, you had to submit it with the build.
How to A/B Test App Icons on iOS
Justin Allen: the story and the lessons
1How Justin Allen sources and tests app icons
Allen tests icons before launch, while his app is in pre-order. His unreleased pre-order app was the test subject, so the result decides which icon he launches with.
Product page optimization works before an app is live, which makes pre-order a cheap time to compare icons.
Allen buys one icon from a Fiverr designer, then edits variants himself. The designer delivered a single icon as a Figma file. Allen made several versions from it to test.
It is a low-cost way to get multiple options without paying for each one.
Allen treats icon design as a download lever. He names color, visibility and how quickly the icon feels relatable to the target market as the things that shape downloads.
2Allen's Xcode setup for icon A/B tests
Allen puts the default and every alternate icon in one assets file. He adds each as a new iOS app icon in Assets.xcassets, via right-click or the plus button, and drags the images in from Finder.
If the icons are missing from that file, the app icon tab never appears in App Store Connect and only the iPhone and iPad tabs show.
Allen uses the single size option so each icon needs one image. In the right sidebar he selects single size for iOS. Apple added it recently, and it removes the need to export every size.
The icon names are free. The only rule he gives is that every icon sits in the same file.
Allen ticks 'include all icon assets' in the main app target. The box is under General, in App Icons and Launch Screen. The app icon field right above it sets which icon is the default.
Allen ships the icons in an approved build before any test. Screenshots can be deleted and re-uploaded freely, but icons must go through an archived build and normal App Review.
Only once the app shows ready for distribution can he create the test under Product Page Optimization.
3How Allen runs the test in App Store Connect
Allen changes only the icon and keeps screenshots identical. Treatment screenshots stayed the same as the original because the icon was the only thing under test.
The treatment's screenshot field works like the normal product page, and he considered tinting screenshots to match each icon but saw no need yet.
Allen runs one treatment at a time on low traffic. One treatment is a plain A/B test with a 50/50 split by default. Two make it A/B/C, and Apple would not allow 100% traffic with two.
Estimated duration is calculated from traffic, so more traffic means a shorter test. Allen says most viewers lack the traffic for more than one treatment.
Allen expects every new icon test to go through review. After he configures a treatment and presses start test, it is submitted to Apple for review before it goes live.
He swaps the icon in the app icon tab rather than uploading it, using the 'change' button on the pre-filled original.
Allen is open about what he does not know. He tried adding a second test and believes Apple allows only one at a time. He does not know how Apple picks which visitors see which icon.
4What Allen's first icon test showed, and his next step
Allen's first test used a tiny icon change, which he calls weak. The simple icon showed a 40% improvement, but on very low numbers and with a confidence level of zero.
He made the change small only so he had something to show. He says dramatic differences are the better approach.
Allen waits for confidence before calling a winner. He thinks the simpler icon is better, but the tool could not confirm it with so little traffic.
The analytics view lets him change the baseline, and a large lift means little when confidence is zero.
Allen followed up with a bolder purple icon test. He stopped the old test and started "Icon 2 test", pitting a purple icon against the simple icon that won the first round.
He submitted it for review after pressing start test, and plans more bold color changes.
How to A/B Test App Icons on iOS
A step-by-step walkthrough of setting up app icon A/B tests with Xcode and App Store Connect product page optimization, including what to test and how to run it. The useful result is that icons can be tested even while an app is in pre-order, so you can pick the launch icon with real data.
Allen tests icons before launch, while his app is in pre-order. His unreleased pre-order app was the test subject, so the result decides which icon he launches with.
Product page optimization works before an app is live, which makes pre-order a cheap time to compare icons.
Allen buys one icon from a Fiverr designer, then edits variants himself. The designer delivered a single icon as a Figma file. Allen made several versions from it to test.
It is a low-cost way to get multiple options without paying for each one.
Allen treats icon design as a download lever. He names color, visibility and how quickly the icon feels relatable to the target market as the things that shape downloads.
Allen puts the default and every alternate icon in one assets file. He adds each as a new iOS app icon in Assets.xcassets, via right-click or the plus button, and drags the images in from Finder.
If the icons are missing from that file, the app icon tab never appears in App Store Connect and only the iPhone and iPad tabs show.
Allen uses the single size option so each icon needs one image. In the right sidebar he selects single size for iOS. Apple added it recently, and it removes the need to export every size.
The icon names are free. The only rule he gives is that every icon sits in the same file.
Allen ticks 'include all icon assets' in the main app target. The box is under General, in App Icons and Launch Screen. The app icon field right above it sets which icon is the default.
Allen ships the icons in an approved build before any test. Screenshots can be deleted and re-uploaded freely, but icons must go through an archived build and normal App Review.
Only once the app shows ready for distribution can he create the test under Product Page Optimization.
Allen changes only the icon and keeps screenshots identical. Treatment screenshots stayed the same as the original because the icon was the only thing under test.
The treatment's screenshot field works like the normal product page, and he considered tinting screenshots to match each icon but saw no need yet.
Allen runs one treatment at a time on low traffic. One treatment is a plain A/B test with a 50/50 split by default. Two make it A/B/C, and Apple would not allow 100% traffic with two.
Estimated duration is calculated from traffic, so more traffic means a shorter test. Allen says most viewers lack the traffic for more than one treatment.
Allen expects every new icon test to go through review. After he configures a treatment and presses start test, it is submitted to Apple for review before it goes live.
He swaps the icon in the app icon tab rather than uploading it, using the 'change' button on the pre-filled original.
Allen is open about what he does not know. He tried adding a second test and believes Apple allows only one at a time. He does not know how Apple picks which visitors see which icon.
Allen's first test used a tiny icon change, which he calls weak. The simple icon showed a 40% improvement, but on very low numbers and with a confidence level of zero.
He made the change small only so he had something to show. He says dramatic differences are the better approach.
Allen waits for confidence before calling a winner. He thinks the simpler icon is better, but the tool could not confirm it with so little traffic.
The analytics view lets him change the baseline, and a large lift means little when confidence is zero.
Allen followed up with a bolder purple icon test. He stopped the old test and started "Icon 2 test", pitting a purple icon against the simple icon that won the first round.
He submitted it for review after pressing start test, and plans more bold color changes.