In this article
Significance Testing evaluates differences in measurement. Choose confidence levels (80-99.9%) and ensure minimum sample size for accuracy.
1: What is Significance Testing?
Significance Testing evaluates the difference between two measurements by taking into account the number of observations and the standard deviation or spread of the data. For instance, when conducting a customer satisfaction survey, you might have sent it to 1000 people last year and 5000 people this year. Last year, 500 people responded with "Likely to recommend" while this year, 2000 people responded with the same answer. Significance testing helps determine whether this change is real and not just due to random noise. It allows us to assess how confident we are that the improvement in customer satisfaction is not just a result of normal variations in data.
2: What Significance Testing Doesn't Do
When conducting significance testing, it's crucial to keep in mind that it does not provide a definite conclusion on whether you or anyone in your organization is doing anything right or wrong. Instead, it simply indicates that the differences observed are likely caused by something. However, that something could be something as innocuous as cultural differences in customer satisfaction between different countries.
While significance testing is a valuable tool that can guide you to where you need to focus, it's essential to understand what the numbers are indicating and what they are not.
3: Confidence level
When conducting Significance Testing, you have the option to choose from six confidence levels. The higher the confidence level, the greater the certainty that the difference between the two groups being tested is real. For example, at a 95% confidence level, there is a 5% chance that the difference observed is just due to random noise in the numbers. If you opt for a lower confidence level of 80%, the likelihood of obtaining significant results increases, but so does the chance that the observed increase is just due to noise, with a one-in-five probability. The confidence level you choose affects the level of certainty that the differences are worth paying attention to.
In Studio, you can select the following confidence levels:
80
90
95
98
99
99.8
4: Minimum Sample Size
The Minimum Sample Size refers to the minimum number of responses required before conducting significance testing. This ensures that the results of the test are more accurate.