Summary

• Data Scientists have focused so much on getting models into production, we often confuse a pipeline that runs for a model that is correct.
• We have forgotten a fundamental aspect of Data Science: a pipeline built around a model that is not fit for purpose is not worth having, even if it runs.
• Creating a model that is fit for purpose requires having the right blend of rigour and pragmatism.
• Simply continuing to add simplifications and hacks to your model and hoping things will be ok because the pipeline still runs will ultimately lead to a pipeline that outputs nonsense.
• You can prevent this by ensuring your model has a robust appropriate mathematical structure that encodes the domain knowledge. You do this by investing some time upfront analyising the mathematical behaviour of your model.

Introduction – The pipeline is king

Putting a data science model into production requires two things; 1) a model, 2) a production pipeline that uses that model. In the old days of data science, before it was even called “data science”, we tended to just have models. How to analyse data, build predictive models and do prescriptive calculations was largely the preserve of academia. So the work stayed in academic publications. It never made it into production. There was also another reason why models never really got into production, and that was lack of data. There’s no point trying to make automated predictions unless there’s value to doing so and that typically requires solving a problem of a significant scale. And that requires data and compute at scale.

In the 21st century that changed in a number of domains, both scientific and commercial. Data and compute became available at scale. That meant there was now a premium on engineering skills that got those models into production and kept them in production. Every Data Scientist now, quite rightly, has to have some deployment and CI/CD skills in their toolkit. But given the potential returns to a pipeline that produces high quality accurate outputs, getting the pipeline running has often become the dominant focus of Data Scientists. The model inside the pipeline has become a secondary focus, as the schematic figure below illustrates.

A downside to this is that by focusing on how to do the plumbing we have forgotten how to build models of sufficient quality. One of the problems that I frequently now encounter is that many Data Scientists mistake a pipeline that runs and produces numbers for a model that’s correct. The pipeline has been built. The plumbing was put in place and debugged. We have numbers flowing, surely they must be correct!

A common refrain I’ve heard is, “You said it wouldn’t work! Well, we’ve got it to work. It’s producing numbers!” No, I said it wouldn’t be sufficiently accurate. Yes we have numbers, just not the right numbers!

You still need a good model

We have focused so much on getting a pipeline running that we have confused a pipeline that runs with a model that’s fit for purpose.

Ok, so the model needs to be good. But how good? It doesn’t need to be perfect for sure. After all, we don’t want perfection to be the enemy of good. Any model will have mathematical approximations and simplifications that make the calculations easier, and by the nature of software it will contain implementation errors. Surely this needn’t be an issue?

The problem with this viewpoint is that many Data Scientists believe we can add one more approximation or hack into the model with minimal consequences, just repeating each time the mantra about “not letting perfection be the enemy of good”. Do this many times and you’ll end up a long long way from perfection. Worrying about perfection won’t be the problem you have. And more importantly, the impact of mathematical approximations, issues and hacks don’t build up linearly. A small extra mathematical simplification doesn’t add a correspondingly small decrease in prediction accuracy. Instead we often see a tipping point – the accuracy of the model predictions can rapidly decrease, sometimes failing catastrophically.

Ok, but sometimes the model approximations and simplifications don’t cause problems. How do we know which situation we’ll end up in? How do we deal with issues if we do reach that tipping point? How do we prevent issues happening in the first place. What can we do? In other words, how do we make sure our models have just the right amount of rigour to them and are fit for purpose? Use mathematics of course! It is the laws of mathematics that determine whether your model, and thus pipeline, outputs nonsense, so use mathematics to stop any nonsense output.

What can you do?

So maybe I have convinced you that you do need to pay attention to what’s inside your model mathematically, and whether it is mathematically fit for purpose, but what can you do practically? Without the specific details of a problem it is difficult to give detailed advice, but there is some general high-level advice I can give, distilled from over 30yrs of building predictive models. It also depends whether we are trying to cure a problem that has arisen or whether we are trying to prevent problems from arising in the first place. I’ll talk about cure first.

Cure

For the purposes of discussion I’m going to focus on a discontinuity of some sort arising in the pipeline/model output. Something has suddenly changed within the model and it is our job to figure out why. Less sudden changes, such as data drift are typically issues with the inputs into the pipeline, not the model. Note that a sudden change could also be in the gradient of the output, not necessarily just a level change. So we’re looking at changes that affect and characteristics of our model with respect to some control parameter or other variable.

1. The first question to ask is the obvious one. Have we actually made any changes recently to the model form, e.g. added extra terms to a linear predictor, changed the form of the loss function used to train the model, etc.? If so, can they potentially explain the changes we see? For example, if we have changed the form of our loss function but the change in the resulting trained model parameter values is small, then the question shifts from “is it the change in loss function” to “could a small change in this parameter cause a large change in output”. So then we start to ask if this might be because we are near a bifurcation point or other singularity in the model parameter space?
   

2. If we haven’t changed our model form or our model training algorithm but we are seeing some sort of discontinuous behaviour, then we might need to start thinking in terms of overall system behaviour, rather than just the model which is just one part of the system. Most systems are defined in terms of smooth microscopic rules, e.g. simple formulae or evolution equations, so discontinuous behaviour is more often an emergent phenomenon and stems from a constraint on the global system, e.g. optimizing some criterion. Think of how phase changes in physical systems result from the requirement to minimize the Helmholtz free energy. So we need to ask if we have changed (inadvertently or otherwise) any control parameters that we apply to our overall system, e.g. settings in a business process optimization calculation, or if the characteristics of the system-level input data have changed.

3. And finally, we shouldn’t forget the possibility of a good old-fashioned bug in the code. In fact, this is the most likely explanation for sudden changes in output. If we are seeing discontinuous changes in output this can be the result of a change in the code branch that is being executed code and will stem from an if statement being executed when it shouldn’t have been.

Summary: If your model monitoring has detected large discontinuous changes in the output from your deployed computational pipeline, then consider the following order for investigation,

a. Look for recent changes in the mathematical specification of the model form and the model training process, and ask if these can explain the size of the changes you are seeing in the model output. This will require understanding where the non-linear sensitivities are in your model.

b. If you haven’t made changes to your model or training process, then look for code bugs, particularly in flow control statements.

c. If the output is the result of some system optimization process, then look for changes in system control parameters or system-level data input characteristics. Again, understanding if these can explain the size of output changes you are seeing will require understanding where the non-linear sensitivities are, but at a system level, not just model level.

Prevention

Without specific details about the domain or particular problem being modelled, it is even harder to give advice on what can be done proactively, so my advice is by necessity going to be very high-level. But there is some advice we can give,

1. Firstly, just being aware that a small change in the mathematics doesn’t mean a small change in the output provides you with a lot of protection. Being forewarned is forearmed. This will stop you from making big changes to the mathematical form of a model or estimation algorithm without first of all thinking through some of the potential consequences and trying to anticipate where things might go wrong. This doesn’t mean you shouldn’t make those mathematical changes, but it does mean you have an idea what the downstream consequences might be and you can put some mitigation in place in advance, e.g. error checks, guardrails, monitoring for specific anticipated failure modes. And when things do go wrong you have a priority list of where to look for the culprit.

2. But the best preventative measure we have is ensuring our model has a sensible structural mathematical form. For example, want to model a probability or a proportion? Then build a model of a linear predictor and apply a link function that maps to [0, 1]. I’m always amazed when I see even these simple steps ignored by new Data Scientists. Yes, I have seen many times probabilities modelled using linear regression, and then the Data Scientist uses the model significantly out of sample and gets predicted probabilities either less than zero or greater than 1. This is easily avoided with just a bit of prior thinking, pen & paper work, and some domain knowledge. A model that has a sensible mathematical structure for the problem being modelled will always give sensible output. This is what you want. And the thing is, particularly in commercial Data Science, senior stakeholder don’t always notice when a model is not as accurate as you might hope but they do always notice when the output of a model is nonsense.

So how do you get the right structural mathematical form? As we’ve already hinted at, this is built on domain knowledge. Does that mean we can’t use complex general purpose models such as neural networks to model our problem. Yes, we can use them, but they do not automatically encode in a structural form the domain knowledge we have. So what I will often do if I want to use a neural network, is to include it as a sub-component of a larger model that does have a structure based upon domain considerations. The ‘sensible’ structure of the larger model ensures that the overall output will be sensible (may be not accurate) even if the output from the neural network sub-component is not sensible.

3. Use mathematics to understand the behaviour of your model. Analyse the model behaviour under various asymptotic limits to understand what are the main classes of behaviour and on what timescales, lengthscales, etc do the different behaviours emerge. Whilst doing the asymptotic analysis you may also get a hint of what, if any, singularities in the model behaviour exist at finite times, lengths, or parameter values. For any model I have, no matter how simple (and I mean no matter how simple) I will always sketch out its behaviour at a high-level, playing around with the mathematics of the model for an hour just using pen and paper.

Again, once your have this asymptotic analysis and singularity analysis complete you are forewarned about where your model is sensitive to small changes in form or small changes in the inputs. You can also use the analysis to guide you on how to construct appropriate dimensionless parameters and/or how to parameterize your model to emphasize the more stable regions of its behaviour.

Conclusion

Don’t forget when building a Data Science pipeline that it is the model part of the pipeline that is doing the critical part of the calculation. Sure, getting the plumbing, connections, and services connected with the rest of the pipeline working may take the majority of the time, and may take the majority of ongoing support resources, but that doesn’t mean you should spend disproportinately little of your time developing the model. The model encodes the solution to the domain problem, and so the quality of the model dictates the quality of the pipeline overall.

© 2026 David Hoyle. All Rights Reserved