New Dimensions in Analytics for Product Promotions in Retail

Promotions, like other investments, fundamentally work by earning revenue in exchange for a smaller investment. It is important to evaluate the impact of sales promotions on profit because it is possible for promotions to lead to higher sales but lower profits.

Only 30 out of 45 promotions reported as successes are actually successful, while 19 out of 46 promotions reported as failures actually increased profits (Hardy 1986).

The Boston Consulting Group’s research indicates that 20 to 50 percent of promotions generate no noticeable lift in sales—or, worse, have a negative impact. Another 20 to 30 percent dilute margins in that they don’t generate an increase in sales sufficient to offset promotion costs.

Types of Promotions

These are types of promotions:

  1. Promotions That Increase Sales and Margin. These are the most successful promotions, showing a positive impact across both dimensions.
  2. Promotions That Dilute Margin but Increase Sales. These promotions have a minor negative impact on margin but have a high positive impact on incremental sales. To determine if these promotions are worth continuing, calculate their targeted ROI—in other words, the amount in incremental sales that is required for every dollar of margin invested.
  3. Expensive Promotions. These have a negative effect on margin and don’t boost sales sufficiently to be worth the investment.
  4. Promotions That Dilute Sales. These promotions are the least favorable, since they reduce sales.
  5. Low-Impact Promotions. These promotions don’t increase sales or margin enough to warrant running

Effects of Promotion

Fig 1: Effects of Promotions

Analysis of Promotion Success

In order to accurately measure the increase in sales volume, you need to know your baseline (what sales were prior to the promotion) using both data you’ve already collected and data you collect throughout the duration of the promotion. This equation can be used to find your overall increase in sales:

Overall Sales Lift = Lift in Promoted Item Sales + Halo – Cannibalization – Pull-forward

Promotional Breakdown

Promotional breakdown are as follows:

  1. Lift in Promoted Item Sales: The difference between the sales baseline and the new sales volume.
  2. Halo: The effect that occurs when promoting one item leads to an increase in sales of another item. Measure using basket analysis – look at historical transactional data across multiple promotional periods and analyze all baskets that contained the item promoted and determine among all those baskets which were the common items that were not on promotion
  3. Cannibalization: When the promoted item’s sales increase but “eats up” the sales of another item. This can lead to a lower category lift.
  4. Pull-forward: When the sale item is something that consumers regularly purchase and has a longer shelf-life, causing consumers to stock up on the item while it’s on sale, causing a spike in sales during the promotion and a dip in sales after the promotion.

Breakdown of Promotion

Below we can look into overall breakdown of promotional events:

Fig 2: Breakdown of promotion

Benefits of utilizing cannibalization

Below are the benefits:

  1. The opportunity to pinpoint exactly why a consumer purchases a new product over their typical product
  2. Better product performance analytics – When new products are released, or promotions are run, there is clear visibility into whether they’re bringing in extra revenue, or just eating away from existing products
  3. Accurate promotion effectiveness – Grocers know the true performance of a promotion

Cannibalization w.r.t to promotion & price change

Below example we can see demand for Coca Cola increased after promotion. Here, retailer will be in high risk of overstocking of Pepsi as it was in Demand before.

Fig 3 : Cannibalization w.r.t to promotion & price change

Cannibalization Effect after Promotion

When the promoted item’s sales increase but “eats up” the sales of another item. This can lead to a lower category lift.

Fig 4: Cannibalization Effect after Promotion

Benefits of utilizing Halo Effects

Below are the benefits:

  1. We can know why one product influenced a consumer to buy another product.
  2. Effective inventory management – Grocers are able to accurately project promotional impact across products and project inventory needs accordingly
  3. Accurate promotion effectiveness – Grocers know the true performance of a promotion

Market basket analysis for Halo Effect

Market Basket Analysis to know which products are being sold together

Fig 5: Market basket analysis for Halo Effect

Sales Uplift forecasting

Example for one of the product, an endcap display with no price change results in a notable sales uplift, but the uplift is modest compared to the effect of a 50% price reduction.

Fig 6: Sales Uplift forecasting

Pull Forward Effect

When the sale item is something that consumers regularly purchase and has a longer shelf-life, causing consumers to stock up on the item while it’s on sale, causing a spike in sales during the promotion and a dip in sales after the promotion.

Fig 7: Pull Forward Effect

Difficulty in detecting cannibalization & Halo Effect

Benefits sounds great but it is difficult to accurately detect cannibalization & halo Effect:

  1. It is difficult to accurately identify pairs/groups of products that share this relationship
  2. If you know that wine and cheese, share a halo relationship, you need to identify the strength of that relationship. For every extra unit of wine sold on promotion, how many units of cheese is being sold? You need to know this if you want to manage your inventory in preparation for a promotion.

Case Study : Market Basket Analysis to plan Halo Effect

During shopping online or watching movies, we often come around of some instances where we get recommendations like – ‘You might want to buy this item with that’ or ‘Movie Recommedation as per your liking’ and we sometimes think how come the site knows about my liking. As funny as it seems, this is a prime example of Association Rule where algorithm finds relation between multiple item based on data inputs and helps to drive insights for big online shopping companies or Movie Channels which is also popularly known in real world as Market – Basket Analysis

In this case study, I have taken a approach to implement the same on a grocery dataset to find out relationships between items sold.

Data Dictionary

  • Member_number: Customer Id
  • Date: Date of purchase
  • itemDescription: Description of product purchased

Steps of execution:

  • Loading Necessary Packages
  • Loading dataset
  • Data Pre-Processing
  • Performing EDA
  • Apriori Implementation
  • Result Customization

Step – 1 : Loading Necessary Packages

import numpy as np # linear algebra
import pandas as pd # Data pre-processing
import seaborn as sns # Required for plotting
import matplotlib.pyplot as plt # Required for plotting

Step – 2 : Loading dataset

df = pd.read_csv("Groceries_dataset.csv") ## Loading dataset
df.head()

df.isnull().sum().sort_values(ascending=False)

Interpretation: – No Null values should be present

Step – 3 : Data Pre-Processing

df['Date'] = pd.to_datetime(df['Date']) ## Type-Conversion from Object to Dateime
df.head()

Step 4: Performing EDA

Step – 4.1 : Top 10 Sold Items

# Creating distribution of Item Sold

Item_distr = df.groupby(by = "itemDescription").size().reset_index(name='Frequency').sort_values(by = 'Frequency',ascending=False).head(10)

## Declaring variables

bars = Item_distr["itemDescription"]
height = Item_distr["Frequency"]
x_pos = np.arange(len(bars))

## Defining Figure Size

plt.figure(figsize=(16,9))

# Create bars
plt.bar(x_pos, height, color=(0.3, 0.4, 0.6, 0.6))

# Add title and axis names
plt.title("Top 10 Sold Items")
plt.xlabel("Item Name")
plt.ylabel("Number of Quantity Sold")

# Create names on the x-axis
plt.xticks(x_pos, bars)

# Show graph
plt.show()

Step – 4.2 : Month-Year Sales

df_date=df.set_index(['Date']) ## Setting date as index for plotting purpose
df_date

df_date.resample("M")['itemDescription'].count().plot(figsize = (20,8), grid = True, title = "Number by Items Sold by Month").set(xlabel = "Date", ylabel = "Number of Items Sold")

Step – 5 : Apriori Implementation


Apriori is an algorithm for frequent itemset mining and association rule learning over relational databases. It proceeds by identifying the frequent individual items in the database and extending them to larger and larger item sets as long as those item sets appear sufficiently often in the database. The frequent itemsets determined by Apriori can be used to determine association rules which highlight general trends in the database: this has applications in domains such as market basket analysis.

Step – 5.1 : Data Preparation

cust_level = df[["Member_number", "itemDescription"]].sort_values(by = "Member_number", ascending = False) ## Selecting only required variables for modelling
cust_level['itemDescription'] = cust_level['itemDescription'].str.strip() # Removing white spaces if any
cust_level

Step – 5.2 : Create Transaction list

transactions = [a[1]['itemDescription'].tolist() for a in list(cust_level.groupby(['Member_number']))] ## Combing all the items in list format for each cutomer

Step – 5.3 : Train Model

from apyori import apriori ## Importing apriori package
rules = apriori(transactions = transactions, min_support = 0.002, min_confidence = 0.05, min_lift = 3, min_length = 2, max_length = 2) ## Model Creation
results = list(rules) ## Storing results in list format for better visualisation
print(results)

Step – 6 : Result Customization

## Creating user-defined function for arranging the results obtained from model into readable format

def inspect(results):
    lhs         = [tuple(result[2][0][0])[0] for result in results]
    rhs         = [tuple(result[2][0][1])[0] for result in results]
    supports    = [result[1] for result in results]
    confidences = [result[2][0][2] for result in results]
    lifts       = [result[2][0][3] for result in results]
    return list(zip(lhs, rhs, supports, confidences, lifts))
resultsinDataFrame = pd.DataFrame(inspect(results), columns = ['Left Hand Side', 'Right Hand Side', 'Support', 'Confidence', 'Lift'])
resultsinDataFrame.nlargest(n=10, columns="Lift") ## Showing best possible scenarios

Above we can see relationship between different products, we can also look into Support, confidence and lift of the products, which can help us to take important decisions on halo effect. You look into this link to learn more about support, confidence and lift : https://www.thedataschool.co.uk/liu-zhang/understanding-lift-for-market-basket-analysis

In this blog, we have learned about importance of promotional event for a company. We also studied about importance of cannibalizations and Halo Effects. And, we looked into one example of market basket analysis.