> For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete.
But they want to do something still to try and improve sales. Seems reasonable even if not scientific.
This is not the right approach to take then. There are lots of other approaches to decision making outside of hypothesis testing, use them! This is not an appropriate use of hypothesis testing and can very much lead you toward making the WRONG decision.
For example, with such small numbers, there isn't much value in aggregate statistics. It would take a day or two to go through each one individually and see what happened, and you'd probably learn way more about your customers.
> For example if you want to test a tweak that results in 5% more conversions, you need about 3000 sales to detect it! For Candy Japan this would mean waiting for about 10 years for the test to complete.
But they want to do something still to try and improve sales. Seems reasonable even if not scientific.