Performance metrics

Several metrics are provided for evaluating how well the predictions match the target time series. The metrics are correlation, MAE, WAPE and R2, as described below.

All the metrics can be evaluated on either of:

  • the time series as they are

  • the change in the time series versus the previous target value

  • the YoY change of the time series versus the target value of the same quarter in the previous year

Note that for the latter two transformations, also for predictions we are calculating the relative change from the previous target (actual) value to the current prediction, not the change from the previous prediction to the current prediction.

correlation(target, prediction, growth_period=None, delta=False, start_date='2000-01-01', min_points=5)

Calculate the correlation between the two time series.

The output is a time series, where the value for a given date is the correlation for all the observations from the start_date up to and including the given date.

Parameters:

  • target – The target signal that was predicted, such as reported sales numbers.

  • prediction – The prediction time series.

  • growth_period – If specified, transforms both the target and the prediction time series into percentage change versus a previous target value before calculating the correlation. If growth_period=1, the change versus the previous target value is calculated. For quarterly data, if growth_period=4, the change versus the target value for the same quarter in the previous year is calculated.

  • delta – If set to True, subtracts the previous target value from both the target and the prediction before calculating the correlation. This option cannot be combined with growth_period.

  • start_date – Predictions on or after this date are used to calculate the correlation.

  • min_points – The minimum number of data points needed to calculate the correlation.

Example:

Calculate the correlation between the actual year-over-year growth in sales versus the predicted year-over-year growth in sales:

correlation(Sales_Actual, Sales_Prediction_Card_Spending, growth_period=4)
mae(target, prediction, growth_period=None, start_date='2000-01-01', min_points=1)

Calculate the Mean Absolute Error (MAE) of the predictions.

The output is a time series, where the value for a given date is the MAE for all the observations from the start_date up to and including the given date.

Parameters:

  • target – The target signal that was predicted, such as reported sales numbers.

  • prediction – The prediction time series.

  • growth_period – If specified, transforms both the target and the prediction time series into percentage change versus a previous target value before calculating the MAE. If growth_period=1, the change versus the previous target value is calculated. For quarterly data, if growth_period=4, the change versus the target value for the same quarter in the previous year is calculated.

  • start_date – Predictions on or after this date are used to calculate the correlation.

  • min_points – The minimum number of data points needed to calculate the correlation.

Example:

Calculate the MAE of the sales predictions:

mae(Sales_Actual, Sales_Prediction_Card_Spending)
wape(target, prediction, growth_period=None, start_date='2000-01-01', min_points=1)

Calculate the Weighted Absolute Percentage Error (WAPE) of the predictions. This is the average error divided by the average target value.

The output is a time series, where the value for a given date is the WAPE for all the observations from the start_date up to and including the given date.

Parameters:

  • target – The target signal that was predicted, such as reported sales numbers.

  • prediction – The prediction time series.

  • growth_period – If specified, transforms both the target and the prediction time series into percentage change versus a previous target value before calculating the WAPE. If the value is 1, the change versus the previous target value is calculated. For quarterly data, if the value is 4, the change versus the target value for the same quarter in the previous year is calculated.

  • start_date – Predictions on or after this date are used to calculate the correlation.

  • min_points – The minimum number of data points needed to calculate the correlation.

Examples:

Calculate the WAPE of the sales predictions:

wape(Sales_Actual, Sales_Prediction_Card_Spending)

Calculate the WAPE of the sales estimates from 2018 onwards:

wape(Sales_Actual, Sales_Estimate_fiscal, start_date='2018-01-01')
r2_score(target, prediction, growth_period=None, delta=False, start_date='2000-01-01', min_points=3)

Calculate the R-squared (R2) metric of the predictions. This represents the proportion of the variance of the target that’s explained by the predictions.

The output is a time series, where the value for a given date is the (R2) score for all the observations from the start_date up to and including the given date.

Parameters:

  • target – The target signal that was predicted, such as reported sales numbers.

  • prediction – The prediction time series.

  • growth_period – If specified, transforms both the target and the prediction time series into percentage change versus a previous target value before calculating the R2. If growth_period=1, the change versus the previous target value is calculated. For quarterly data, if growth_period=4, the change versus the target value for the same quarter in the previous year is calculated.

  • delta – If set to True, subtracts the previous target value from both the target and the prediction before calculating the R2. This is the calculation shown in the Prediction Model UI. This option cannot be combined with growth_period.

  • start_date – Predictions on or after this date are used to calculate the correlation.

  • min_points – The minimum number of data points needed to calculate the correlation.

Examples:

Calculate the R2 of the sales predictions, in terms of their ability to predict the delta versus the previous quarter:

r2_score(Sales_Actual, Sales_Prediction_Card_Spending, delta=True)

Calculate the R2 of the sales predictions, in terms of their ability to predict the growth rate of the sales numbers versus a year ago:

r2_score(Sales_Actual, Sales_Prediction_Card_Spending, growth_period=4)