CaddieOpen in CaddieDownload for Mac

How to A/B Test App Icons on iOS

Shared by Caddie Library · Updated
App Masters12:31Open on YouTube
Chapters show their main points

The bottom line

Apple's native tools let you A/B test app icons: add every alternate icon in Xcode, ship them in an approved build, then run the test in App Store Connect. This shows which icon wins before you launch, even during pre-order.

Key takeaways

  • Every icon you want to test must first sit in the app's asset catalog. 1:21
    • Keep the default icon and each alternate in Assets.xcassets, added as a new iOS app icon through the context menu or the plus button.
    • Choose "single size" for iOS in the right sidebar, so each icon needs only one image.
  • The icons must ship in a build that Apple approves before any test can run. 3:46
    • In the main app target's General tab, tick the box to include all icon assets.
    • Archive the build and send it through the normal review cycle. Screenshots can be swapped freely, but icons cannot.
  • The test itself runs under Product Page Optimization in App Store Connect. 4:36
    • Once the app is ready for distribution, press the plus button, name the test and pick the treatment.
    • The app icon tab appears only if the icons were set up correctly in the asset catalog.
    • With one treatment, traffic splits 50/50 by default. With two treatments, Apple will not let you pick 100%.
  • Change only the icon, and make the versions look clearly different. 7:21
    • Keep screenshots identical so the icon is the only variable. Use dramatic changes, such as a purple icon against the original.
    • Run one treatment at a time unless you have a lot of traffic.
  • Results need enough traffic, and the test can start before the app is live. 8:49
    • The sample test showed a 40% improvement but a confidence level of zero, because the numbers were too low.
    • The demo app was still in pre-order, so the test helps pick the launch icon.

Keep the screenshots the same, change the app icon. You know, for me, in any AB test, I just change one parameter at a time.

Steve P. Young

Chapter summaries

0:00 Setting up the icons in Xcode

A better icon lifts downloads, and testing it starts in the Xcode project, not in the store.

  • Icon names are free, but every icon must share one file. 1:46
    • Xcode names the default icon app icon on its own.
    • The alternate names were made up on the spot.
    • No required naming scheme is known. The only rule is one shared file.
    • A Fiverr designer made one icon and sent a Figma file. Justin edited it into several versions.
  • The default icon is chosen separately in the target settings. 4:10
    • The app icon field sits right above the include-all-icons checkbox.
    • Changing that field changes which icon is the default.
    • Single size is a recent Apple feature, so one image per icon is enough.
    • Color, visibility and how fast the icon feels relatable to the target market all shape downloads.

4:30 Creating the test in App Store Connect

The test setup screen estimates how long the test will take and what share of visitors will see it.

  • Traffic decides how long the test runs and how many people see it. 5:12
    • The estimated duration is calculated from your traffic. More traffic means less time.
    • The traffic proportion is the share of visitors who see the test.
    • The Product Page Optimization list is empty the first time. Justin already had one test running.
    • Justin tried adding a second test and believes Apple allows only one at a time.
    • Justin doesn't know how Apple picks which visitors see which icon.

6:16 Reading a test that is already running

A running icon test uses the same screens as a screenshot test, so the same habits apply.

  • The demo test used a tiny icon change, which is a weak example. 7:00
    • The original icon is Treatment A, and only one treatment was chosen.
    • Justin made the change small only so he had something to show.
    • He plans to run bolder color changes soon.
    • Screenshots stayed identical because only the icon was being tested.
  • The screenshot field works like the normal product page. 7:47
    • Treatment screenshots use the same field as the normal distribution page.
    • Tinting screenshots to match each icon was considered, but it is not needed yet.
  • The analytics view is the same as for any other test. 8:40
    • The baseline can be changed.
    • The simpler icon was ahead, and Justin thinks it is better.
    • The tool could not confirm it with so little traffic.

9:09 Creating a new purple icon test

A new test takes a few clicks, and it only goes live after Apple reviews it.

  • A treatment is one extra version beyond the original. 9:34
    • One treatment is a plain A/B test. Two treatments make it A/B/C.
    • Justin stopped his old test and named the new one "Icon 2 test", with a purple icon.
    • Most viewers lack the traffic for more than one treatment.
  • The icon is swapped in the app icon tab, not uploaded. 10:04
    • The tab pre-fills the original icon, and "change" picks the alternate.
    • Screenshots are different. Delete all and upload new ones.
    • An icon has to ship with the build.
  • Starting the test sends it through a review cycle. 11:04
    • Justin picked purple against the simple icon, since simple won the earlier test.
    • The treatment name can be edited before starting.
    • After "start test", he submitted it for review.

In the growth library

Justin Allen and Steve P. Young, Justin Allen is an indie iOS developer; Steve P. Young is founder of App Masters. A step-by-step walkthrough of setting up app icon A/B tests with Xcode and App Store Connect product page optimization, including what to test and how to run it. The useful result is that icons can be tested even while an app is in pre-order, so you can pick the launch icon with real data.

Lessons that draw on this:

More from App Masters · The whole library, and ask it anything

Transcript288 lines
0:00

Coming up, you're going to discover how to properly We're going to get a little nerdy here. How to properly AB test your app icon. Stay tuned. What is up app nation? It is Steve P. Young, founder of appmasters.com, the place you go when you want actionpacked content related to helping you grow your app downloads and more importantly those revenues. And a part of that app downloads equation is how colorful, how visible, how instantly relatable your app icon is to your target market. And joining me today to break down the nerdy, the techy part of AB testing app icons because it is a little techy is Justin Allen. He's part of our community and a great indie developer as well. Justin, welcome to the show. Hello everyone. All right, Justin. So, I know there's some Stack Overflow articles on this, but we really wanted it to be more of a step-by-step guide into helping the developer out there who knows a little bit about

1:01

Xcode. Is that the phrase? Okay, good. Who knows a little bit about Xcode? Because I didn't know if Apple shift it from Objective C to anyways Xcode to show you how to AB test. And for those who are marketers like me, well, this is how you send this video to your developer as well. All right, Justin, take it away, my friend. All right, so in Xcode, you'll have the assets.xcass assets file. In there, you will want to have your regular icon, but then every one of the icons that you are going to want to AB test at any point in time.

1:39

Um, so you'll want to have that. And then also, hey Justin, why did you name it the way you named it? Is there a particular way to name these app icons? Um, so the default one it will just create this. Mhm. It'll create just app icon. The other ones I just came up with them off the top of my head. There really isn't that I know of a specific naming, but you do want to have them all in the same file. And as a non-coder, is that pretty easy to do? Like how do you create that new app icon within Xcode?

2:16

Is that pretty easy? Like that's a no-brainer. When you're in this file, you will go to this area here, the gray, right underneath the list, and then you pull up the context menu and you will see the image set and then iOS new app icon. Got it. And then you just drop your icon. After you're doing that, um you can also do it from this plus button. Got it. It'll bring up the same menu. And then you will drag in your image from Finder into this. Did you create those icons yourself or did you just go through Canva, Fiverr? I've done some things on Fiverr. I have a person that does them from Fiverr. So, he actually only did this one and he sent me a Figma file and I went through and made some changes so I could have multiple. something else to show on this specifically, you probably want to go over and expand the right hand sidebar.

3:21

And for iOS, select single size instead of all sizes. Um, that's a feature that Apple added pretty recently and that makes it a lot simpler because you only need one. Okay. Image for it. So after you have these all in and configured, you'll have to go to the regular project file. Go to your main app target general and then scroll down to where it's got the app icons and launch screen. Make sure this here is check marked. Got it.

3:59

Also, if you want just listening, it just says include all icon assets. Yeah, I recommend watching the video. Yeah. And then if you want to change the default app icon, right above there, it'll have a field for app icon. That is the default one. So after you do all of that, you have to archive and submit a build to Apple. Do a regular review cycle through Apple so that they can go through and approve it. All right. So, in App Store Connect, once your app says ready for distribution, you will go on this regular sidebar here, go to product page optimization. When you first do this, you won't have any in here, but I have one that's been running. You'll hit the plus button to create a new one. Set your name, treat, set the name, treatment, any other settings you want.

4:55

Create test. Can we try it? I know you're running it. Can we try it right now? Let's see what happens. I believe they only allow you to have one. Let's try it. One out of 10. So, two treatments traffic percentage. You can see the estimated duration. If you want to see that um that just calculates based off of Yeah. If you got more traffic, obviously less time. Less time. But the traffic trends the proportion is just how many people of your traffic will see this AB test. So Justin put 66, you can put 100. That means everybody that comes to your product page, your app icon will see 100. Is this what this is or? So yes, but if you have two treatments, it won't let you select 100. Oh, so you have to do 50/50 so that you give everything a fair shot. Is that how it works? I don't know how they determine which ones to show necessarily, but if you have Yeah, if you have one treatment, it'll do 50/50 by default.

6:01

Okay. So, cool. So, you'll create that there. That happens. Okay. So, yeah, it is it only allows you to have one. So, do your app icons show up pretty normally like you would do a like a product a screenshot test? Let let me actually open this and show you what it does. When you open this, if you did not properly put all your image, your app icons into the XC assets file, you will not see this tab. You will only see the iPhone and iPad once.

6:37

Um, so I just have it running with the same screenshots just cuz I didn't feel like changing them for this one because I just wanted to see how it works for the app icon. But you will go into the app icon and it'll show you the two different show you what ones are running. Um, so this is the original one. Treatment A, cuz I only selected to have one treatment one with this is the it's very little minute difference. Um, which is not how you normally want to run these, but I was just doing it so I would have an example cuz I like it.

7:21

Um, you should probably try to do dramatic differences like the the other two or three that Justin showed off right here. Yeah, like purple, totally different colors. I I'll be running those ones here soon. Um, so you will go through this and then you drag and drop. Well, for these for the treatments of the screenshots, you can, as far as I know, the field here is the same field that you would have on the normal distribution page. Yeah. And this is what I would recommend too, Justin. Keep the screenshots the same, change the app icon. You know, for me, in any AB test, I just change one parameter at a time.

8:05

You just keep it simple. Keep it simple and keep going. And so yeah, I was thinking maybe to have them be different colors. Like if the icon is different, have them match the icon, but that's I like that too. Not really necessary at this point. Um, I hit the view analytics or anything else you want to show off here. One little thing before that. When you actually go to configure this, it'll give you a drop down, I believe, of the statements. I could actually cancel this one out and start a new one to show that. Um, but yeah, let's go to the view analytics. Okay, the simple one's a little bit. What's that? The simple one's winning a little bit.

8:49

Yeah, by It's a 40% improvement. I mean, it's very low numbers, but confidence level of zero. Like, it doesn't it doesn't know. I actually think the simpler one is better. Anyway, you can change the baseline on this. Basically, this is the same as you would see if you were running an AB test without the icons. All right, Justin, let's create a new icon test. And we're going to do from the very beginning. So, we stopped the previous one. We're going to start a new one just for you guys. Justin decided to stop his previous test and run a new one just for this video. So, we're going to show you exactly how to do it. All right. Icon 2 test. Maybe purple. Icon two. Yeah, I will be putting in the purple. I'm going to set one treatment. So, create test. That makes sense. Now, treatments are like the number of AB. So, instead of AB, that's treatment one. And then ABC would be treatment two. So, I I like AB. Like,

9:42

always pick one treatment. I like to do it one one at a time, unless you have a lot of traffic. Most people watching this video probably don't. If you don't have a lot of traffic, just do one treatment at a time. All right. Yeah. You got the icon tab open. So when you create it, it'll pull whatever the original is and it'll fill them in. So yep, that did it. So then you go to app icon tab. So it's showing the original one selected.

10:11

Hey Justin, if I could just pause here, if you can go back to the screenshots really quick, the 6.9. So if you guys are testing screenshots here, you can just upload all the images. But for an app icon, it's a little bit more complicated. So if you wanted to do screenshots, you could hit delete all and then just upload your new screenshots and it's very very simple. But for app icon, you had to submit it with the build. And so that's why we're creating this dedicated video just for the app icon. All right, Justin, go ahead and hit change for me. Yeah, thank you, sir. Ah, look at that. Beautiful.

10:44

So what are you thinking? Purple simple because simple won the the previous one. Yeah, let's do the purple simple. Yeah. So there it is. You can edit the name if you want to change the treatment A to something else. Other than that, I think I'm ready to start it. Let's do it, dude. So, start test. So, they will want you to run it through a review cycle also. Mhm. So, submit it. There it is, my friend. Cool. Justin, thank you so much for this tutorial.

11:24

Anything we missed, you want to make sure we cover? I don't think so. I think we got it all. And Justin, the cool thing, if you guys have been following along, this app actually isn't live yet. It's in pre-order. So, Justin's testing this while the app's in pre-order, just to get a sense of which icon he should launch with, right, Justin? Yep. All right. If you want to learn more about Justin, go check out logicalmagic.te.

11:46

logicalmagic.te. You can see all the great apps that he has up. This is the one he's AB testing right now. that's in pre-order and he's got this music guesser app that we're going to run on Indie App Santa and hopefully, you know, grow some downloads on there. And so he's got a suite of apps that you guys can check out as well. And if you want to follow Justin along through his indie app development journey, his Twitter is linked up in your favorite podcast and YouTube video, it is JJ Allen Tech. JJ Allen Tech. All that is linked up in your favorite podcast app and YouTube description as well. Justin, thank you so much for coming on and doing this.

12:23

You're welcome. Thank you all for watching. I'll see you on the next video.