Damian Wong

Marketing Campaign A/B Test Analysis

Project Overview

This project analyzes the effectiveness of a marketing campaign by comparing conversion rates between paid advertisements and public service announcements (PSAs). Using statistical methods including chi-square testing and confidence interval analysis, I determined whether the advertising campaign was successful.

Ad Campaign Effectiveness AnalysisDownload

Dataset

The dataset, sourced from Kaggle, contains:

  • 588,101 total impressions
  • Control group (PSA): 23,524 impressions
  • Treatment group (Ads): 564,577 impressions

Methodology

A/B Testing Approach

I implemented an A/B test to measure the causal impact of advertisements versus PSAs:

  1. Control Group: Users shown Public Service Announcements
  2. Treatment Group: Users shown paid advertisements
  3. Measured conversion rates between groups

Statistical Analysis

  1. Chi-Square Test
    • Created contingency table of conversions
    • Calculated chi-square statistic: 54.01
    • Degrees of freedom: 1
    • P-value: < 0.0001
  2. Conversion Rate Analysis
    • Control (PSA): 1.79%
    • Treatment (Ad): 2.55%
    • Absolute difference: 0.76 percentage points
    • Relative lift: 43.09%

Results Visualization

1. Sampling Distribution

Show Image The sampling distribution under null hypothesis shows our observed difference (0.769%) significantly outside the expected range, supporting the statistical significance of our results.

2. Conversion Rate Comparison

Show Image The visualization demonstrates clear separation in conversion rates between control and treatment groups, with 95% confidence intervals showing minimal overlap.

Key Findings

  1. Campaign Success
    • The advertising campaign showed a significant positive effect
    • 43.09% lift in conversion rate
    • Results statistically significant (p < 0.0001)
  2. Attribution Analysis
    • Clear causal relationship established through A/B testing
    • Conversion rate improved from 1.79% to 2.55%
    • Chi-square test confirms non-random effect

Conclusions

The analysis definitively answers both key questions:

  1. The campaign was successful, showing a 43.09% lift in conversion rate
  2. There is a statistically significant correlation between seeing ads and higher conversion rates
    • The treatment group showed higher conversion rates than the control group
    • The difference is unlikely to be due to random chance (p < 0.0001)

Check out the repository on GitHub!