Even Distribution between Variations

Introduction

Convert A/B experiments has even traffic distribution between the various variations of an experiment. However, sometimes you may notice that this is not the case. Find below some of the most common reasons.

Low number of visitors

When Convert buckets a visitor into an experiment that has a 50/50 traffic distribution, it "flips a coin" for each visitor. That is, each visitor has a 50% chance of being bucketed into one of the variations. Because of this, sometimes distributions will look a little lopsided when there's not a lot of traffic. Statistically, it should even out as you get more visitors. 

Check Site Area and Audiences conditions

The next thing that you should check is your Site Area and Audiences conditions. You might have defined wrongly these conditions and need to be checked.

Bots

The vast majority of bots that can be identified by the user agent (Google, Bing, Yahoo) will be excluded from the experiment results. However, there are some bots that are difficult to be detected and these can cause the uneven distribution.

Paused Variations

Check if either the variations were involuntarily paused or something happens on the page, which interferes with the Convert tracking script.

Conclusions

The good news is that it does not matter if you have a minor uneven distribution. In fact, the allocation is done with a random function and the statistical model we use can accommodate uneven splits. So even, if the experiment is uneven your experiment results will still be valid. 

This is because, Convert calculates results based on the conversion rates between variations (or original and variation), and not based on the number of visitors each variation receives. Be assured that your Convert experiment results are valid and reliable even with the minor differences in the traffic split ratio.

Have more questions? Submit a request

Comments