Understand Stopped Experiments
Convert Experiments helps you setup tests and tries to bring testing best practices inside the software. For example we consider an experiment successful (enough) at 97% and not before and we suggest on every new test to let the test run for a minimum of 7 days. You can change these minimum days and add maximum days as well. But some automations you have to be aware of.
- An experiment is marked as "completed" after at least one variation performs better with a significance greater than 97% OR all the variations were stopped because they performed worse. This automation can only be stopped by setting a minimum running time. For example, setting a minimum running time for 6 month will remove the automation for the whole duration of six months.
- An experiment can keep winning variation running - after test is marked as completed, the winning variation is kept active (this setting defaults to active)
- An experiment can stop losing variations - when a variation is decided as being losing (performs worse with a significance of more than 97%) it is stopped and all traffic is redistributed across the other active ones. This setting is active by default
You can do this by selecting the options menu in the experiments summary. If you have other default settings we should be considering or changing let us know in the comments.