Monitoring your KPIs performance with Adobe Workspace Functions

Today I am going to show you how to use Adobe’s Calculated Metric functions in order to set up a dashboard and monitor when your critical metrics & KPI’s perform outside normal/predicted behaviour.

Currently Adobe Sitecatalyst gives a feature called “anomaly detection”, know to most Google analytic users as “Intelligent alerts”.

Anomaly Detection in Adobe Sitecatalyst and why you don’t really need it

Anomaly detection will spot unpredicted behavior – whether this is positive or negative – based on previous data you hold (you can read more about this here). They way it does this is by calculating the daily total for the selected metric and compare it with the training period using each of the following algorithms:

• Holt Winters Multiplicative (Triple Exponential Smoothing)
• Holt Winters Additive (Triple Exponential Smoothing)
• Holts Trend Corrected (Double Exponential Smoothing)

Each algorithm is applied to determine the algorithm with the smallest Sum of Squared Errors (SSE). The Mean Absolute Percent Error (MAPE) and the current Standard Error are then calculated to make sure that the model is statistically valid.

Theoretically speaking, Triple Exponential Smoothing is ideal for including “seasonality” trends (unlike Double exponential), however Anomaly Detection biggest weakness is it’s 90 day window.

90 days is pretty much insignificant for retail web sites where seasonality and trends plays a big role in terms of performance, hopefully they will add a YoY feature in future releases but in case they don;t that is ok because as long as you have the data available you can do it yourself as shown below.

How to set a benchmark for your KPI

For the purpose of this exercise we are going to use Conversion Rate as our KPI (which most likely pretty much every web site monitors). Before we kick off with the metris, a quick explanation of the numbers you will need in order to build those metrics.

• • Conversion rate: Self explanatory
  • Mean Value: The average
  • Median Value: The middle value
  • Standard Deviation: a quantity expressing by how much the members of a group differ from the mean value for the group.
  • Control Limits (High & Low): Control limits are used to detect signals in your KPIs that indicate that performance is not in control and, therefore, not operating predictably.

This metric should be built in order to measure positive (Higher Control Limit – The month that your web site performed better than predicted, based on historical data) & negative unpredicted behaviour (Lower control limit – The month that your web site performed worse than predicted – based on historical data).

Use all metrics with at least 12+ months of data or even more if possible, especially if you are on retail as a smaller time frame can give the wrong impression (for example Black Friday & Christmas shopping will most likely always been shown as an Outlier – outside your control limits which if you think about it is, you can’t replicate a black friday / Christmas event every day).

Calculated Metrics you will need for Adobe Workspace KPI Benchmark Report

Mean, Median and Standard Deviation are available by default within the Adobe Calculated Metric Functions. All you have to do is select them from the drop down menu and add the metric you want to measure with.

Lower Control Limit Calculation

Lower control limit is calculated by selecting the Mean value of your KPI minus the Standard Deviation as shown below:

Lower Control Limit = Mean – STD 

Higher Control Limit Calculation

You make exactly the same calculation as above but you replace – with + since Higher

Higher Control Limit = Mean + STD 

Conversion Rate % Performance VS Median Value

This will show your stakeholders how your KPI is performing against the media value.

Performance VS Median Value = Conversion Rate – Median(Conversion Rate) 

Conversion Rate % performance when Higher Control Limit Exceeded

The calculation looks like the image below but it is way simpler than it looks:

 Just use the logic below by using your already created calculated metrics and the “IF” & “Greater than or Equal” functions from Adobe’s Function menu:

IF conversion rate is greater than or equal to your high control limit then return Conversion Rate value minus the higher control limit value, else return zero (0).

We use else return zero (0) as we can then exclude that value from Adobe Workspace by ticking interpret zero as no value form the column settings as shown below. This will help your conditional formating make more sense in workspace as normal performance will appear blank.

And here is the calculation in adobe;s calculated metrics interface:

Conversion Rate % performance when Lower Control Limit Exceeded

you will follow exactly the same logic in order to create your lower control limit metric with the difference that here you include the “Mean – STD” value instead “Mean + STD” value.

Calculation:

IF conversion rate is lower than or equal to your lower control limit then return Conversion Rate value minus the lower control limit value, else return zero (0).

I can not give example how that looks in Adobe Workspace but ideally you want to be using horizontal and vertical bars. You will have a value ONLY when your website performs outside the predicted behaviour. If you see no values that is absolutely fine, there will be cases where you have a pretty flat performance YoY (even though in retail if this quite hard with Black Friday, Christmas shopping and other similar events).

Try to break this down by month and if you sport an anomaly, break that down by day. You will then be able to find the exact days you over/under performed so you can dive deeper in order to understand what went right or wrong.