A Beginner’s Guide to A/B Testing
Crafting campaigns that convert, a brief introduction to A/B & Multivariate testing.
In a previous blog post, we discussed the importance of adding actionable data to your campaign; now, we are looking at how to get the best performing message to the user. A campaign that converts best in any way that you want it to. Either with relevant, appealing content or with the most user-friendly button for the right call to action.
3-step test
A/B Testing, at its most basic, is a way to compare two versions of something to determine which performs better. A/B testing is widely used online to turn off your feelings and to base your decisions on hard data. This form of testing is often used to test different content or design variations on websites, within apps, but especially within your Email, SMS and Push campaigns. You can use A/B testing for different purposes, and that's why the goal has to be clear beforehand. You have a particular urge to improve, testing just for testing has no use.
The first step is to ask yourself what it is you want to examine. And so on, decide which medium you want to use. In the Notifcare Dashboard, you can choose between Push, Email, or SMS. Then, determine which part of the content you want to test. Take small steps by adding "special promotion" or "tap to open" to the message and see if it works for your audience. But also whether or not the use of a different Title or Subject line results in more opens. Test which image works best, whether making your message personal increases the conversion or whether adding a sound to a push notification has any influence. But beware, with an A/B test, you only try to test one part, so keep it simple.
Next, you draw up your hypothesis so that you can validate your results after running the test. A hypothesis is a prediction that you make before running a test. It is a statement in the form of a problem, solution, and result.
Finally, you perform the analysis and use the results to challenge your hypothesis. Then you are ready to use your test results for your future campaigns. Conducting a thorough A/B test, although we've made it super simple within the Notificare Dashboard, will take time and needs your full attention. Doing it right will achieve continuous improvement.
A/B vs. Multivariate testing
Testing more than one thing at a time, such as subject line and a call to action, is a multivariate test, which is more complex to run. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations. Once you get the hang of A/B testing, you can make the step up to multivariate testing, but this also requires dedication to do it right.
The Split vs. the Winner
Notificare offers two types of A/B & Multivariate tests that you can execute:
You can either Split your audience by the number of variations you create. With a minimum of 2, the traditional A/B and maximum of 10 variations, A/B/C/D/E/F/G/H/I/J. The Split feature will simply divide your selected audience into equal parts according to the number of variations you create. Then our system randomly selects the recipients of your test within your selection. It is up to you to decide whether you send your test to your entire user base or only to a specific target audience. Using our advanced criteria, a familiar mechanism within our dashboard, this selection can be done intuitively. Please note that this choice also influences your perception and research result.
With the Winner option, you can decide how many variations you want to create, define a smaller sample group first, and wait for results. Then, Notificare will automatically send the best performing content, the Winner, to the remainder of your audience.
Only a small part of your selected audience will get to see the variations of your content if you use the Winner feature. The sample size is important and shouldn't be too small. Having a small sample size might return a winner, but the result might not be accurate enough. So keep in mind that when using this type, your sample group shouldn't be too small. But remember these are real people, providing you results for your tests. Running a live experiment with content that is extremely different, might not engage your audience at all. Again, success is the sum of small efforts.
Strategy first?
If you want to know more about how to set up a good A/B & Multivariate testing strategy, feel free to contact our Professional Services Team. And as always, our engineers can also assist you via our Technical Support.