Significance Tests (normal distribution assumed)

Groups	Requirements	Test	Tails	Statistic	Distribution	Degrees of freedom	Example
1	normal dist, n > 30 and known variance	z-test	1 and 2	z-score	Normal	NA	A sample of the population is tested for height. Do they match the known population? Population mean and standard deviation are known, sample > 30
1	n < 30 and unknown variance	One-sample t-test	1 and 2	t-score	Students t	n-1	A company wants to compare the average weekly sales of a new product to the historical average weekly sales of similar products, which is $5000.
2	Same 2 groups (before and after). 2 Dependent samples.	Paired t-test	1 and 2	t-score	Students t	n-1	A company implements a new training program and wants to determine its effectiveness by comparing employee performance before and after the training.
2*	2 Independent groups, equal variance	Independent t-test (equal variance – pooled)	1 or 2	t-score	Students t	n1​+n2​−2	A retailer wants to compare the average sales of two different stores located in different regions.
2*	2 Independent groups, unequal variance	Independent t-test (Unequal/Welch’s)	1 and 2	t-score	Students t	n1​+n2​−2
3+	3+ groups – look at one variable	One-Way ANOVA	1 Only	F-score	F	between groups k-1 (where k is num of groups).	Three groups are given different drugs in a test to see the improvement in blood sugar.
3+	3+ groups – look at multiple variables	Two-way ANOVA	1 Only	F-score	F		Effect of water and sun on height of sunflowers, where 3+ combinations of water and sun (3 + groups)
2 Categories	2 Categories – comparison is to see if they are related	Pearsons Chi-Square	neither, just different	Chi-Squared	Chi-Squared	(no. of rows – 1) * (no. of columns – 1).	Individuals that received social media vs. those that received email – 2 Groups purchased and not purchased.
Correlation
2	Correlation	Pearsons Correlation Coefficient (R)	1 or 2	t-statistic	t-distribution table	n−2	Relationship between daily hours of sunlight and temperature. Best for continuous data with linear relationships. Provides both the direction (positive or negative) and the strength of a linear relationship between two variables. Note for simple linear regression. R squared is the Pearson r squared. It ranges between -1 and 1
2	Correlation	Spearmans Rank	1 or 2	t-statistic?	Spearmans Rho?	n-2	Relationship between the ranks of employees’ performance and their years of experience. Best for ordinal data or when data does not meet Pearson’s assumptions.
	r Squared (used in regression analysis) One (simple linear regression) or more independent variables (Multiple linear regression) + 1 dependent variable	Not used in significance testing directly			NA		Explaining the variance in academic performance based on study hours, attendance, and other factors. Used in the context of regression analysis to explain the proportion of variance in the dependent variable. Indicates the proportion of the variance in the dependent variable that can be explained by the independent variable(s). It does not provide information about the direction of the relationship. Ranges from 0 to 1
		Regression Analysis		F-Statistic	F-Distribution		Fit a regression model and computer squared. Perform an F-test to determine overall significance.
Non-Parametric Tests	Not Normal distribution
2	Compares distributions of the 2 groups	Mann Whitney U		U Statistic	Mann-Whitney U Distribution	NA	2 Independent groups (like the t-test)
2	Compares Medians of the 2 groups	Wilcoxen signed rank		T Statistic	Wilcoxen signed rank table	NA	When comparing two related samples, matched samples, or repeated measurements on a single sample.

Significant Testing Steps

1. Null hypothesis: There is no significant difference between variables between the two groups.
   
   Examples:
   1. In a drug efficacy study, the null hypothesis might state that the drug does not affect patients compared to a placebo.
   2. The mean cholesterol levels of patients taking the new drug are equal to the mean cholesterol levels of patients taking a placebo.
   3. The mean sales after the advertising campaign are equal to the mean sales before the campaign.
   4. The mean sales of the product with the new packaging design are equal to the mean sales of the product with the old packaging design.
   
   Alternative hypothesis: There is a significant difference between the means of the groups.
   
2. The significance level (𝛼α) is the probability threshold for rejecting the null hypothesis. Commonly used significance levels include 0.05, 0.01, and 0.10. e.g. 5%: The result of the test only has a 5% chance of occurring.
   
3. Calculate test statistic (t-value, z-value, etc). The test statistic is used to look up the p-value from the relevant distribution tables.

4. Calculate the p-value using the test statistic. The p-value is used to determine if the test is significant. e.g. a p-value of less than 5% (0.05) is significant and we can reject the null hypothesis. If the p-value is higher than the significance level, then the results are reported as statistically significant.

1 or 2 Tailed Tests
One-tailed test if measuring greater or less than Left or Right Sided
Two-tailed if measuring the difference

Type 1 and Type 2 Errors
Type 1 Error: We reject the null hypothesis even though it was true (Alpha)
Type 2 Error: We accept the null hypothesis even though it was false (Beta)

Two Methods of accepting or rejecting Null Hypothesis
1. Critical Values (taken from normal or t-distribution)
2. P-values (if p < or > than 0.05)