Customer choice is endless and acquisition costs are at an all-time high, so it makes sense that marketers are turning to customer loyalty programs to stand out.

But these programs aren’t a silver bullet. Take Bed Bath & Beyond‘s well-known yet generic 20% off coupons, for instance. They’re a classic example of a “loyalty” scheme that ultimately wasn’t enough to avert the brand’s bankruptcy. 

While it’s easy to find advice on how to structure and build loyalty programs, it’s not always easy to understand if they’re working—or more importantly, how to measure their success. If a program isn’t driving business value, it’s time to adjust your strategy. 

The first step for any brand is to set distinct goals for their loyalty program. One of the most effective goals comes down to incremental revenue—revenue gained as a result of the program through increased customer retention, higher average order value or more frequent purchases. The focus here is on “incremental,” implying revenue that wouldn’t have been realized without the program.

The cost side of a loyalty program includes both variable costs, like rewards, and fixed costs, such as software or administrative expenses. These inputs need to be meticulously tracked to understand the program’s financial impact, and the method used to calculate incremental revenue is key.

Traditional methods involve comparing the average revenue of loyalty program members with nonmembers. However, this approach is flawed due to self-selection bias, where members of a loyalty program are inherently more engaged or valuable customers. To address this, more sophisticated methods can be employed. 

From our experience, difference-in-difference analysis is the most effective. This method measures incremental revenue by comparing the change in revenue from loyalty program members before and after joining the program against the change in revenue from a control group of nonmembers over the same period. This helps to control for external factors affecting both groups, providing a clearer picture of the program’s true impact.

Other approaches include randomized controlled trials, where customers are randomly assigned to either the loyalty program or a control group. However, this approach might overlook unobserved factors that influence both the likelihood of joining the loyalty program and spending behavior, leading to biased results.

Propensity score matching (PSM) is also sometimes used, which involves matching each loyalty program member with a nonmember who has a similar profile, or “propensity” to join, based on characteristics like past purchase behavior or demographics. A key issue with PSM is that it can be logistically challenging and expensive to implement.