ResultsAiX Optimize uses Bayesian statistics. From the data so far it simulates 3,000 possible outcomes and counts how often each version comes out best. That share is the "P(best)" figure. 95% means that in 95 of 100 simulated outcomes that version was the best one. It is a probability, not a certainty, and it describes the visitors in the test, not future results.
The three checks:
Under "Is it ready to call?" the report shows three bars.
- "Visitors per version". The smallest visitor count of any version.
- "Days running". Set against your "Minimum days", so weekdays and weekends are both included.
- "Leader's probability". Set against your "Confidence".
A test is marked Ready when every version has reached the visitors it needs (your "Minimum visitors per version" or the number needed to detect a 10% difference, whichever is larger, see below), the test has run your "Minimum days", and the leader's probability has reached your "Confidence". All three must hold at once.
The defaults, set in section 4 of the builder:
Minimum visitors per version 100 (from 20 to 100,000), Minimum days 7 (from 0 to 90), Confidence 95%, with 90%, 95%, 98% and 99% to choose from.
The visitors bar is the stricter of two numbers:
The plugin also estimates how many visitors per version are needed to detect a 10% difference from the Original's current rate. The bar, the Ready check and the automatic winner all use that estimate or your minimum, whichever is larger, and the "Too early to call" message tells you about how many more visitors per version that would take. On a page with a low conversion rate the estimate can be much larger than 100.
What you will see:
- "Too early to call." with the number of visitors still needed and, if relevant, the days still to run.
- "Ready." with the leader's name, its probability and a button to declare it the winner.
- On the Dashboard and the Tests tab, "too early to call" or "ready to call" beside each running test.
Automatic winners:
With "Declare the winner automatically when the test is ready" ticked, which is the default, the plugin checks running tests once an hour. When a test is Ready it ends the test, records the leader as the winner with a note of its probability and visitors per version, and emails the site's admin address once with the subject "[ResultsAiX Optimize] Winner declared:" and the test name. An ended test is never declared again, whether or not its winner is kept showing.
You can declare a winner by hand at any time. If the report says too early, the careful choice is to wait.