QA Guide

QA Tools in Convert

Convert offers several tools to help your QA experiences while you are developing, and before launching them.

Tip

Chrome Debugger Extension

This tool outputs the logs of the Convert script to the Chrome developer tools console.

The output includes experiments triggered, specific variations triggered and allows you to determine the sequence of events in regards to testing. This provides a lot of useful information when debugging a test.

This tool should be used throughout the whole QA process to determine if you are bucketed into the experiment and determine the variation that you are seeing.

You can find more information on how to install it in this article.

save image


It is very common to be bucketed on the original variation and assume mistakenly that the experiment is not working. This is why we recommend that you enable the extension in incognito sessions and perform the QA there.

All variation and experiment references in the debugger output are made with the corresponding IDs. You can find out how to determine IDs in this article.

Preview Variations URL

These URLs are meant to be an aid while developing an experience using the Visual Editor. While working, the experiment QA tester should go back and forth between these URLs and the Visual Editor view. The Visual Editor view may not be completely accurate since the window in which the editing occurs is done in a frame.  Also, the code that makes changes in the editor runs repeatedly for ease of use. However, using the Preview link, this code is only run at the beginning of the page load, which is closer to how an experiment works in real life.

save image


save image

There are some considerations to take into account:

  • The experiment conditions from the Site Area and the Audience are not taken into account when doing a preview with the Live Preview URL. The changes are made to the page URL on which you append the parameters. This is for making it easy to test different views.
  • The Previews are served from our app server, instead of the CDN servers. Normal visitors to an experiment are served the experiment from the CDN servers. Because of this, there might be a difference in download speeds and other conditions when using the Preview URL but not during the normal visitor experience.

    You can find these URLs automatically generated for you in the Visual Editor > Variation Menu > Live Preview, and in the report alongside the Original and each Variation (look for the eyeball icon as shown in the screenshot above).

Force Variation URLs

The purpose of Force Variation URLs is for you to test your experiment in the closest environment to what a visitor will experience. These URLs should be used to do the final testing of your experiment.

Variations served with this URL are served from the same CDN server we use to serve the experiments.

save image

When using a Force Variation URL, make sure that your Site Area and Audience conditions are also met.

Also, when using a Force Variation URL and testing goals, take into account that goal configurations often include URL conditions.  If your goals do not trigger when using a Force Variation URL this might be why.

You will find more information on how to format them in the this article.

Use of query parameters in "QA Test" Audiences

The purpose of a special audience is to allow you to Activate a test and verify it without allowing any visitors to be bucketed into it.

There is a defined process for this, as described in this article.

Also, be aware that adding a special audience might affect your ability to match the Site Area and goal URL conditions.   You would have to take this into account.  More information can be found in this article.

QA Stages in Convert

1st Stage - Visual Editor and Preview Variation URLs

Tools used:

  1. Preview Variations URLs

Goal:

You will be able to validate (to a certain degree of accuracy) that the changes created by the Visual Editor look good in a normal browser window.

Process:

  1. Develop your test using the Visual Editor.
  2. Preview each variation change with the Preview URLs provided in either the Visual Editor or on the Reports page. Previews open in a new tab.
  3. Make changes in the Visual Editor, then preview; repeat until your variation changes look the way you want them.

2nd Stage - Preview with Force Variation URLs

Tools Used:

  1. Force Variation URLs
  2. Incognito Window
  3. Chrome Debugger Extension (the extension needs to be enabled in Incognito)
  4. Query parameter Audience  (eg, special qa_test audience defined by you)

Goal: Validate how a specific variation looks when served from Convert CDN servers and test your experiment Site Area and Audience conditions, but restrict it to your QA audience.

Process:

  1. Complete your experiment configuration, including the Site Area and Audience conditions.
  2. Set a QA audience in your experiment.
  3. Open a fresh Incognito browser window (only one instance should be open at a time as they share cookies between them -- this will affect your results if you have more than one incognito window or tab open or forget to close the incognito window after a test).
  4. Trigger each of the variations with Force Variation URLs (original is not necessary). Close and open another incognito session when testing each variation.  Make sure you append the query parameter that you defined in your qa_test Audience to the end of your Force Variation URL (ie, something like ”&utm_medium=qa_test”).
  5. If you see that the forced variation is not showing what you expect, open the Chrome Debugger Extension, and go through the logs to see if the experiment conditions are met. You can search the log with the specific experiment ID to aid in your search.
  6. Fix any experiment issues if encountered. Repeat testing process until experiment and changes show correctly. If problems are found, please contact Convert Support.

3rd Stage - QA the whole setup including the Reports

Tools Used:

  1. Force Variation URLs
  2. Incognito Window
  3. Chrome Debugger Extension (enabled on the incognito window)
  4. Query parameter Audience
  5. Experiment Report

Goal: Validate that the entire experiment works as expected.

Process:

  1. Open your experiment report.
  2. Open an incognito browser window and the page of the experiment including the QA audience parameter.
    -- Use this if you don't want visitors to see the experiment yet -- additionally, if you have visitors increasing the numbers on your report, it's impossible to tell if the numbers that appear on the report are from your own visit and conversion, or your visitors' visits and conversions.
    -- If your experiment was Activated in the past without a qa audience, previously bucketed visitors might trigger conversions even if you set a qa audience later.  The reason for this is that once a visitor is bucketed into an experiment (and has not yet triggered all goals), when they return we do not check to see if they match the audience conditions).
  3. Note the variation you see. Check if your visit to that variation is shown on the report (visitor count will be incremented by one, usually within a couple of seconds). Trigger your goals and check that your conversion are counted on the report. Verify that the variation is functioning and visually showing what you expect.  Also, remember that visitors and goals are unique - you cannot trigger them again unless you close and open a brand-new incognito window.
  4. Close and reopen the incognito window. Repeat process until all variations and goals are validated.
  5. Use Chrome debugger to aid you in identifying and fixing any issues encountered.
  6. If problems are found, please contact Convert Support.

4th and Final Stage - You're Ready!

Remove QA Audience and Reset Experiment

Once you have finished all of the above, remove your special QA Test Audience to allow your visitors to trigger the experiment.  You can also use "Reset Experiment" on the Reports page under Advanced Settings.  This will reset the start date of the experiment, as well as clear the previous visitor numbers and previous conversions to 0 on the report.

Once again, keep in mind that the Site Area and Goals might refer to specific URLs that might be matched with your special audience query parameter, but not without it, so check carefully.

Clone Experiment, Double-Check Configuration and Activate

Important
Have more questions? Submit a request

Comments