Creating and running an A/B-test

Create a new A/B test

1. Click on "Modules" in the left navigation.

2. Click on "A/B-Test" in the left navigation:

3. To create one or more A/B-tests, select a channel in the top bar of the page and make it editable. Please note that you cannot create A/B-tests for a parent channel.

4. If the channel is editable, a button with the text "CREATE NEW TEST +" is visible on the page. Click on this button:

The following window, in which you will configure your A/B-test, appears:

Configure the A/B test

5. a. Give the test a name

5. b. Click on "UNTIL TARGET EFFECT REACHED" and set your target effect. (Do not set an end date. With an end date the test would run for exactly the specified time, but you would have no control over the statistical validity of the results. The end-date option should only be used, if the statistical validity of the results is of no importance or if you calculate the validity of the results yourself)

5. c. Set a start date and start time by deselecting "Start in 10 minutes" and selecting a start date in the field "Starts at". Keep in mind that after the moment, when the start time has passed, you cannot make changes to the search-configuration of the B-variant of this A/B-test anymore. We recommend to let your A/B-test begin 30 minutes or 24 hours in the future, or something in between.

5. d. Leave the other values as set by default, as these values should be ideal for most cases.

5. e. Click "CREATE TEST"

6. Save your changes: click the "check"-icon right to the channel-select-box on top of the window. See example screenshot below.

Then update your shop data by clicking on the black symbol on the yello background:

Make your config changes

7. Select the search-configuration of the B-variant of your A/B-test by selecting your A/B-test, in the example below it has the name "a-b-test-changed-thesaurus", and making it editable (See example screenshot below). Now you can change many of the parameters which are available in the UI. For a complete list of available features follow this link.  These changes will only be applied to the B-variant and will not influence your normal search-configuration. Don't change too many parameters at once. You will get clearer results in the analytics of your B-variant this way.

To leave the scope of the A/B-Test B-variant and return to the base-variant, select "Base configuration" in the A/B-Test selection.

Run the A/B Test

8. Proceed here when all necessary changes have been made to your B-variant's configuration. Go to "Modules", then "A/B-Tests", select your channel and the A/B-test and make it editable. Begin your A/B-test (It is always safe to press this button. Your A/B-test will not run before the configured start time has arrived) by clicking on your A/B-test in the upper right corner on the symbol , and selecting "begin test". Safe the configuration by clicking on the "check"-symbol near to the A/B-test's name on top of the page (see example screenshot below)

Then update your data by clicking on the black symbol on the yello background:

9. The A/B test starts executing at the configured start time. Your test has run until the end when you can read "completed" and "Ended ... ago" on your test on the A/B-tests page (see example screenshot below). Hint: while it is possible to let 2 or more A/B-test run at the same time, we recommend to let only one A/B-test run at a time.

Analyze A/B Test results

10. Select the channel, which contains the A/B-test that is completed, on top of the page. Make this channel editable. Now your A/B-test is displayed like this and you can see the results of your A/B-test. (see example screenshot below) . You see the values for "Search Count", "Click Through Rate %", "Conversion %" and "Sales per Search ∅". For more detailed values click on the header "Click Through Rate %", "Conversion %" or "Sales per Search ∅".

11. Compare the values of the base- and the B-variant. The relevant values are:

• The click through rate in %: the higher this value is, the better is the performance of your configuration.
• The conversion in %: the higher this value is, the better is the performance of your configuration.
• The sales per search mean value: the higher this value is, the better is the performance of your configuration.

In the example screenshot above, the advantages of the B-variant are a better click-through rate and a higher conversion compared to the baseline configuration. The advantage of the variant A (the baseline variant) is that there are more sales per search in this configuration.

You can get a more detailed result by clicking on "Start analysis" (see screenshot below) in the upper right corner of the A/B-test:

Finalize the A/B Test

12. Now select the channel that contains the completed A/B test at the top of the page and make that channel editable if the channel is not editable already. What follows now, is the last step: click "apply variant B" if you want to apply the B-configuration to all searches. Click "retain variant A" if you want to stay with variant A, which is the base variant of your A/B-test.  Choose the variant with the better performance. If you are in doubt about the reliability of your test, choose the baseline configuration and repeat your test with the "UNTIL TARGET EFFECT REACHED"-feature instead of a fixed end date for your test. Save your changes to the channel.