Category: Statistics

Creating a simple Linear regression model and preparing for multi-linear regression.

In this example, we use a sample of marketing spend data vs. sales and inspect the correlation between radio spend and total sales. The regression line is fitted using the ols function from statsmodels.formula.api

You can download the original file from Kaggle here then just replace the location you save it to in the (df = pd.read_csv() line.

import pandas as pd
import seaborn as sns
from statsmodels.formula.api import ols
import statsmodels.api as sm
import matplotlib.pyplot as plt
pd.set_option('display.max_columns', None)
df = pd.read_csv("marketing_sales_data.csv")

# Drop rows with any null values
df.dropna(inplace=True)

# Check and handle duplicates if needed
if df.duplicated().sum() > 0:
    df.drop_duplicates(inplace=True)

#rename columns to snake 
df.rename(columns = {'Social Media': 'Social_Media'}, inplace = True)

# Simple order encoding
tv_dict = {'Low': 1, 'Medium': 2, 'High': 3}
df['TV'] = df['TV'].replace(tv_dict)

# One-hot encoding for non-ordinal variable
df = pd.get_dummies(df, columns=['Influencer'], dtype=int)


# Define and fit the model
ols_formula = "Sales ~ Radio"
OLS = ols(formula=ols_formula, data=df)
model = OLS.fit()
summary = model.summary()
print(summary)  #Prints off the statistical summary including R squared and the Beta coefficients.

# Calculate residuals and predicted values
residuals = model.resid
y_pred = model.predict(df['Radio'])

Results return from the model.summary() method from the OLS (ordinary least squares) function from the statsmodels module. R squared is calculated as 0.757 meaning 76% of the variability in y (sales) is accounted for by radio. However, if we look at other media, we will see that other variables (TV) also have a strong correlation.

The Beta coefficient for radio spend is 8.17, which means that for every $1 million in Radio spend, we get $8.17 million in sales.

Significance Tests (normal distribution assumed)

Significant Testing Steps

1. Null hypothesis: There is no significant difference between variables between the two groups.
   
   Examples:
   1. In a drug efficacy study, the null hypothesis might state that the drug does not affect patients compared to a placebo.
   2. The mean cholesterol levels of patients taking the new drug are equal to the mean cholesterol levels of patients taking a placebo.
   3. The mean sales after the advertising campaign are equal to the mean sales before the campaign.
   4. The mean sales of the product with the new packaging design are equal to the mean sales of the product with the old packaging design.
   
   Alternative hypothesis: There is a significant difference between the means of the groups.
   
2. The significance level (𝛼α) is the probability threshold for rejecting the null hypothesis. Commonly used significance levels include 0.05, 0.01, and 0.10. e.g. 5%: The result of the test only has a 5% chance of occurring.
   
3. Calculate test statistic (t-value, z-value, etc). The test statistic is used to look up the p-value from the relevant distribution tables.

4. Calculate the p-value using the test statistic. The p-value is used to determine if the test is significant. e.g. a p-value of less than 5% (0.05) is significant and we can reject the null hypothesis. If the p-value is higher than the significance level, then the results are reported as statistically significant.

1 or 2 Tailed Tests
One-tailed test if measuring greater or less than Left or Right Sided
Two-tailed if measuring the difference

Type 1 and Type 2 Errors
Type 1 Error: We reject the null hypothesis even though it was true (Alpha)
Type 2 Error: We accept the null hypothesis even though it was false (Beta)

Two Methods of accepting or rejecting Null Hypothesis
1. Critical Values (taken from normal or t-distribution)
2. P-values (if p < or > than 0.05)

In business, the most commonly used t-test is often the Independent Two-Sample t-test. This test is widely utilized to compare the means of two independent groups or samples to determine if there is a significant difference between them.

The Independent Two-Sample t-test is particularly useful in various business scenarios, such as:

1. Comparing the performance metrics of two different products or services.
2. Analyzing the effectiveness of different marketing strategies or campaigns.
3. Assessing the impact of changes in processes or procedures on business outcomes.
4. Evaluating the differences in customer satisfaction levels between different segments or demographics.
5. Comparing the financial performance of different investment portfolios or asset classes.

1. One-Sample t-Test
   

Purpose:

• To determine if the mean of a single sample differs significantly from a known or hypothesized population mean.

Example:

• Scenario: A company wants to compare the average weekly sales of a new product to the historical average weekly sales of similar products, which is $5000.
• Hypothesis: The mean weekly sales of the new product is different from $5000.
• Application: The company collects weekly sales data for the new product over several weeks and uses the one-sample t-test to determine if the average sales differ from $5000.
  
  Calculation
  x-X/sd/SQRT(n)
  Where x = sample mean, X = est., population mean, n = sample size
  
  Nominator: Mean – hypothesized population mean
  Denominator: sample standard deviation/SQRT(sample size n)
  

2. Independent Two-sample t-test

Purpose:

• To compare the means of two independent groups to see if they differ significantly.
• Null Hypothesis: The means of 2 groups are equal. If rejected, there is a significant difference between the 2 means.
  

Types:

• Equal Variance (Pooled) t-Test: Assumes that both groups have the same variance.
• Unequal Variance (Welch’s) t-Test: Does not assume equal variances between the groups.
  
  How do we determine if they have the same variance?
  Use the Levenes Test first to determine if the variances are equal between groups

Example (Equal Variance) or Pooled t-Test:

• Scenario: A retailer wants to compare the average sales of two different stores located in different regions.
• Hypothesis: The mean sales of Store A are equal to the mean sales of Store B.
• Application: Sales data from Store A and Store B are collected over the same time period. An independent two-sample t-test assuming equal variances is used to compare the average sales.
  
  t-Value = Complex Calculation
  
  Degrees of Freedom
  df=n1​+n2​−2
  

Example (Unequal Variance, Welch’s t-test):

• Scenario: A company wants to compare the average salaries of employees in two different departments.
• Hypothesis: The mean salary in Department X is equal to the mean salary in Department Y.
• Application: Salary data from both departments are collected. An independent two-sample t-test with unequal variances (Welch’s t-test) is used to compare the average salaries.
  
  Calculation
  Usage Independent t-test is used when the number of samples and the variance of the two data sets is different.
  
  T-Value: t = (X1-X2)/SQRT((var1/n1)+(var2/n2))
  
  Nominator: mean1 – mean2
  Denominator: SQRT((var1/n1)+(var2/n2))
  
  Degrees of Freedom Calculation more complex.
  
  

3. Paired Sample t-Test

Purpose:

• To compare the means of two related groups (e.g., the same subjects measured at two different times).
• To test if the sample mean is significantly greater than or less than the hypothesized population mean (used for directional hypotheses).
  

Business Example:

• Scenario: A company implements a new training program and wants to determine its effectiveness by comparing employee performance before and after the training.
• Hypothesis: The mean performance score after training is different from the mean performance score before training.
• Application: Performance scores of employees are recorded before and after the training. A paired sample t-test is used to compare the average performance scores before and after the training.
  
  More Examples:
  
  example: Comparing the mean blood pressure of patients before and after treatment
  example: Testing if a new drug increases recovery rates (greater than) compared to the known recovery rate without the drug.
  
  Calculation
  T-Value =
  nominator: mean1 – mean2 (difference between the means)
  denominator: s(Diff)/SQRT(n)
  
  s(Diff) = (standard deviation of the differences of the paired data values)
  n = sample size (number of paired differences)
  
• Degrees of Freedom:
• n-1 = degrees of freedom
  

One Side or Two-Sided:

a) One-Sided t-Test

Purpose:

• To test if the sample mean is significantly greater than or less than the hypothesized population mean (used for directional hypotheses).

Example:

• Scenario: A company believes that a new marketing campaign will increase the average number of customers visiting their store per day compared to the previous average of 100 customers.
• Hypothesis: The mean number of customers after the campaign is greater than 100.
• Application: The company collects daily customer visit data after the campaign starts. A one-sided t-test is used to determine if the average number of customers is significantly greater than 100.
  
  Example 2: Comparing the average test score of a class to a national average
  

b) Two-Sided t-Test

Purpose:

• To test if the sample mean is significantly different from the hypothesized population mean (used for non-directional hypotheses).

Example:

• Scenario: A company wants to compare the customer satisfaction ratings of two different products to see if there is any difference.
• Hypothesis: The mean satisfaction rating of Product A is different from the mean satisfaction rating of Product B.
• Application: Customer satisfaction ratings for both products are collected. A two-sided t-test is used to compare the average satisfaction ratings of the two products.
  
  
  Levenes Test – Python Code:
  Test if the variances of two independent groups are equal – use for determining whether to use the Equal or Unequal Independent Samples test
  
  If the p-value for the Levene test is greater than .05, then the variances are not significantly different from each other (i.e., the homogeneity assumption of the variance is met).
  In this case, we can use the Independent Samples Equal Variance T-Test
  If they are significantly different (p<0.05), then we use the Independent Samples Equal Variance T-Test

import scipy.stats as stats

# Data for Set 1 and Set 2
set1 = [19.7, 20.4, 19.6, 17.8, 18.5, 18.9, 18.3, 18.9, 19.5, 21.95]
set2 = [28.3, 26.7, 20.1, 23.3, 25.2, 22.1, 17.7, 27.6, 20.6, 13.7, 23.2, 17.5, 20.6, 18, 23.9, 21.6, 24.3, 20.4, 23.9, 13.3]

# Check if the variances are equal
# Levene's test for equal variances
levene_stat, p_levene = stats.levene(set1, set2)

# Print the results of the Levene's test
print("Levene's test for Equal Variances:")
print(f"F statistic: {levene_stat:.3f}")
print(f"p-value: {p_levene:.3f}")

Analysis: UK House Price Index
Data Source: Office Of National Statistics
Period: Monthly
Graph: Showing December Yearly Prices

Linear Trendline

Polynomial Trendline (order 3)

Observations (from the raw data):
Over the 19 years between YE 2005 and YE 2023, average UK house prices have increased from:
£160,209 in 2006 to £284,681 in 2023 or 78% with an average yearly increase of £6,551

The maximum Average House Price was: £288,784 in December 2022.

The drop from 2007 to 2008 of £28,239 is evident and goes against the positive trend relating to the 2007 financial crash. The drop from 2022 to 2023 of £4053 relates to the higher interest rates in the UK and the cost of living crisis.

Trendlines

If we add a linear regression line (left graph), to the data points, we get an R-squared score of 0.85%, meaning 85% of the variability of average house prices is accounted for by year, which is a pretty strong correlation as we would expect.

If we add the forecast based on the linear regression line equation (not included here), we get £303,618 in 2028.
(I’ve used the FORECAST.LINEAR() Excel function to generate the forecasts).

An alternative is to use the polynomial regression line which is used when there are peaks and valleys in the data. In this example, we are using an order 3 polynomial trendline which accounts for 2 peaks or valleys. This gives an R-squared value of 0.96, which appears to be a very good fit, but I’m somewhat hesitant as we’re right at the point of a 2nd drop. One could look back at the years following 2007 and apply the growth to the years following 2022. This would be similar to a seasonal adjustment, but I’m not sure what tool will pick this up at present (and of course, it’s not the same situation).

If we apply the forecasting from the Polynomial regression trendline, we get £389,243 for 2028.

So the linear regression forecast gives £303K for 2018 and the Polynomial regression method gives £389K, that’s a pretty big difference.
As you can see the polynomial regression is less impacted by the peaks and troughs than the regression line.

If we use the Excel forecast tool which uses the ‘AAA Exponential Triple Smoothing (ETS) algorithm’, for the next 5 years, we can see the forecast of £319,647 for 2028, however, the upper and lower (non-bold) lines indicate the confidence levels, which indicate we can the estimate will lie between £204,369 and £434,925 with 95% certainty, which is very uncertain.

Some of the reasons for this are that we only have 18 years (18 data points) to work with, we have 2 dips in the data, and a somewhat flat period, this doesn’t help the model and hence the confidence limit is less certain.

The algorithm is best used for linear seasonal fluctuating data, which isn’t quite what we have here.

To create a better model, we could create a multi-variate model, including additional features such as interest rate, population, housing supply, and region. We can also see if there is more data available pre-2005.