My boss is the CFO, so he always wants to know revenue impact, and often, my model only outputs predictions of some intervening variable.
Should the model increase revenue? Absolutely. How much? Well...
There are a few factors that feed into this difficulty:
- Dependent variable: By the nature of the business, many times my initial predicted (dependent) variables are not "revenue", but early operational outcomes (payment default, returned checks, fraud).
- Exogenous costs: Back-end cost of capital and other varying and fixed overhead costs make calculations more complex when considering where break-evens should be set.
- Statistical optimization: Classic data science techniques for evaluating model effectiveness (AIC, Mean Squared Error, Area Under the Curve) are not necessarily revenue optimizing, especially considering reason (1) above.
Over time, I've developed strategies to deal with this issue, such as a variation on the model building process and a set of R functions that are specific to our business.
The R functions are very specific, and largely because each business converts model outcomes revenue in a way unique to the industry and business model. As a result, releasing those functions back to the R community wouldn't be valuable.
My model building process, on the other hand, seems somewhat valuable (although really, it just adds an additional step):
- Collect data, try new variables, choose dependent variables that have a known or conceptual relationship with net revenue.
- Build and refine exploratory models. Just as you would in any other model building project.
- Optimize the models to classic statistical algorithm measures (AIC, AUC, etc)
- Validate model performance using validation set.
- Calculate revenue breakpoints and net revenue changes by implementing the models. If not satisfied, try to determine potential sources of bias and return to step 1.
This process isn't novel, but the important part is to realized that optimized prediction is not the final step, and that the initial predictions are only the first step in increasing revenue for the business.