QA Guide

QA Tools in Convert

Convert offers several tools to help your QA experiences while you are developing, and before launching them.

Chrome Debugger Extension

This tool outputs the logs of the Convert script to the Chrome developer tools console.

The output includes experiments triggered, specific variations triggered and allows you to determine the sequence of events in regards to the testing. This provides a lot of useful information when debugging a test.

This tool should be used throughout the whole QA process to determine if you are bucketed into the experiment and determine the variation that one is seeing.

You can find more information on how to install it in the following article.

save image


Is very common to be bucketed on the original variation and determine mistakenly that the experiment is not working. This is why we recommend to enable the extension of incognito sessions and perform the QA on them.

All variation and experiment references on the debugger are done with their corresponding id. You can find how to determine their id, in the following article.

Preview Variations URL

These URLs are meant to be an aid while developing the experiences along the Visual Editor. The experiment QA tester should go back and forth between these URLs and Visual Editor view. Since the Visual Editor view is not completely accurate since the window in which the editing occurs is done in a frame, and also because code that does the changes runs repeatedly on the editor for easiness of use. On the preview, this code is only run at the beginning of the page load, which is closer to how an experiment works in real life.

save image


save image

However, there are some considerations to take into account:

  • The experiment conditions from the Site Area and the Audience are not tested when doing a preview with this URLs. Only the changes are deployed in the URL with which you append the parameters. This is for making it easy to test views.
  • The previews are downloaded from the app server, instead of the CDN servers. Normal visitors download their experiment and variations from the CDN servers. So, there might be a difference in download speeds and other conditions from the normal visitor experience.

    You can find these URLs automatically generated for you on the Visual Editor > Variation Menu > Preview URL and in the report along each variation reported.

Force Variation URLs

Its purpose is for you to test your variations in the closest environment to what a visitor will experience. These URLs should be used to do the final testing of your experiment.

Variations served with this URLs are served from the CDN server we use to deploy the experiments.

save image

When using make sure that your URL Site Area conditions are met with the force variation URL you are utilizing. So, you can include it in your Site Area settings or change them to also include this URLs with its conditions.

Also when testing goals, take into account that a lot of the goal configurations included URL conditions. So be aware if your goals do not trigger when using this URLs.

You will find more information on how to format them in the following article.

Use of query parameters in Audiences to QA

The purpose of this tool is to allow you to launch the test, and verify it before any visitors are bucketed into it.

There is a defined process for this described in the following article.

Also, be aware that the Site Area and goal URL conditions would have to take into account these parameters for launching the experiment with the parameter described in the article.

QA Stages in Convert

1st Stage - Visual Editor and Preview Variation URLs

Tools used:

  1. Preview Variations URLs

Goal:

In this, you will be able to validate that the changes created by the Visual Editor look good in a normal window.

Process:

  1. Develop your test using the visual editor.
  2. Preview each variation changes with the Preview URLs provided via the variation menus for reports. These previews open on another tab.
  3. Make changes on visual editor until your variations changes look the way you want them.

2nd Stage - Preview with Force Variation URLs

Tools Used:

  1. Force Variation URLs
  2. Incognito Window
  3. Chrome Debugger Extension (enabled on the incognito window)
  4. Query parameter Audience
  5. Experiment Report

Goal: Validate how a specific variation looks while deployed from ConverCDN servers and test your experiment Site Area and Audience conditions, but restricted to your QA audience.

Process:

  1. Complete your experiment configuration, setting Site Area and Audience settings.
  2. Set a QA audience in your experiment.
  3. Open an incognito browser window.
  4. Trigger each of the variations with “force variation URLs”. (original not needed). Close and open another incognito session when moving between variation checks. Make sure you have the QA query parameter added to your URL while forcing the variation (ie.”&utm_medium=qa”).
  5. If you encounter that the forced variation is not showing, open the Chrome Debugger Extension, and go through the logs to see if the experiment conditions are met. You can search the log with the specific experiment id to aid in your search.
  6. Fix any experiment issues if encountered. Repeat testing process until experiment and changes show correctly. If problem not found, contact support.

3rd Stage - QA the whole setup

Tools Used:

  1. Force Variation URLs
  2. Incognito Window
  3. Chrome Debugger Extension (enabled on the incognito window)
  4. Query parameter Audience
  5. Experiment Report

Goal: Validate the whole experiment functioning.

Process:

  1. Open your experiment report.
  2. Open an incognito browser window and the page of the experiment with the QA audience (optional if you don't want visitors to see the visitors to see the experiment at this point). Note the variation you get. Check if visits are shown in the report. Trigger your conversions and check them if they are shown on the report as goals. Validate if the variation is showing correctly and reported on the experiment report.
  3. Close and reopen the incognito window. Repeat process until all variations and goals are validated.
  4. Use Chrome debugger to aid you fixing any issues encountered.
  5. Repeat testing process until experiment and changes show correctly. If problem not found, contact support.
Have more questions? Submit a request

Comments