The trouble with models

Note: This was cross-posted to my own blog.


As a research epidemiologist, I love all things data. I will totally nerd out with a book on best practices for designing and maintaining disease surveillance systems all over the world (which I may have done this summer by reading this book by my apartment complex pool). My husband joked once that “you are one of probably five people who read MMWR in the entire country…ten, tops.” I spent a significant portion of my time at APHA in Chicago scoping out doctoral programs, so research is kind of my thing. Nonetheless, I found myself agreeing with an editorial in Lancet Global Health two issues ago which discussed the value and limitations of theoretical/mathematical epidemiological models. I had intended to write about it, but things have an unfortunate tendency to slip off my radar during busy days…and then, unexpectedly, another editorial in the current Lancet Global Health issue, this time on a malaria vaccine trial, jogged my memory.

The October editorial lauded a research paper in the same issue which plugged ten years worth of HIV surveillance data in South Africa into ten different models used to predict HIV prevalence in 2012, and then compared those predictions to actual 2012 data collected in that year’s household survey. (Note to self: read this paper.) The editorialists praised the paper authors for their courage (which, although actually testing the validity of models should not be a terribly scary thing, I suppose researchers do not enjoy proving themselves wrong any more than anybody else) and raised some very good points about the utility of models.

Overall, the models got many of the details correct, such as a shift in HIV burden from younger to older age groups, but got the big picture wrong—predicting stable or declining overall prevalence, whereas prevalence actually increased…Only one model predicted a noticeable increase in HIV prevalence towards the level measured in the survey, and the best estimates of only two of the ten models were within the 95% CIs of the 2012 household survey data. This finding raises the sobering question: if we can get model predictions so wrong in the data-rich setting of South Africa, where there are ten leading HIV epidemiological modelling groups focusing their attention, where can we get it right with confidence?

One possible answer is to redefine what is meant by getting it right. Three of the ten models included in this study incorporated uncertainties…For most indicators for these three models, the empirical data did fall within the uncertainty bounds of the models. If all models provide wide limits of possible epidemic projections based on all plausible trajectories, which ultimately include what does occur in future findings, then they could be regarded as right, but they would not be very helpful.

It is also possible that models can correctly predict what would have been expected to occur, had unforeseen [political, financial, programmatic, or behavioural] changes in underlying conditions not affected the epidemic…In such circumstances, the models project a counterfactual that can be compared with the actual outcome to assess the effect of the changes in conditions, but in themselves might not be able to be validated.

These raise some very interesting questions about the real-world value of mathematical models. While they can provide a framework for understanding epidemiological patterns, or using them for resource planning, their predictive power strikes me as not terribly reliable. What good are predictive models with massive error margins? There is also, of course, the inability of models to account for political chaos change, social unrest, or natural disasters (which underscores the importance of disaster planning). Of course, no model is perfect or able to account for all contingencies, but it is important to acknowledge that their inability to do so inherently limits their everyday usefulness in public health planning.

I was a bit surprised, then, to see an editorial in this month calling for swift action on widespread malaria vaccination citing model predictions as the evidence base for its recommendation.

The [WHO Strategic Advisory Group of Experts on Immunization (SAGE) and the Malaria Policy Advisory Committee (MPAC)] advised that, despite the vaccine having shown partial efficacy in a large phase 3 trial published in The Lancet in April, further real-world demonstration studies should be done before wider roll-out. This small bombshell was doubtless on many minds at ASTMH as four sets of malaria modelling groups presented the results of a major collaborative project on the potential public health impact and cost-effectiveness of the vaccine.

The models used empirical data on vaccine efficacy from the phase 3 trial and historical data relating clinical and severe incidence to mortality. Over a 15-year follow-up period, with 72% coverage of four doses, and at a parasite prevalence in 2–10 year olds (ie, transmission intensity) of 10–65%, the models predicted that the vaccine could prevent a median of 116 480 clinical cases (range across models 31 450–160 410) and 484 deaths (189–859) per 100 000 fully vaccinated children.

When the inevitable question from the floor about the SAGE/MPAC advice came, WHO’s Vasee Moorthy was quick to stress that the organisation had not yet stated its formal position on the matter. Peter Smith, chair of one of the technical expert groups reporting into MPAC, added that the modelling study had shaped the group’s thinking, but that uncertainties remained regarding implementation practicality and safety (the phase 3 trial showed a higher number of cases of meningitis and cerebral malaria in the vaccine group).

I am by no means a modeler, and there may be distinct differences in the reliability of models for HIV versus malaria transmission – particularly considering that HIV transmission is heavily influenced by behavior, while the vector-borne nature of malaria may make predictions more accurate. But I think that mathematical models would have similar limitations with predicting the efficacy of vaccination campaigns, since those are also affected by a whole host of political and economic factors that are difficult to account for. The authors of the malaria editorial also cited research modeling the efficacy of antimalarial drugs at controlling the disease spread, and I would imagine that access to, and distribution of, pharmaceuticals are impacted by factors that models simply are not able to capture.

In the end, though, I suppose models are one of the few robust tools that scientists have to guide policymakers in resource and programmatic planning for public health. As the authors in the first editorial point out, “[Models] are instruments for assessment of the available data, often attempting to reconcile several sources of data together, to provide implications, inferences, and further insights with more rigorous predictions from the knowledge base than could be achieved otherwise through simple extrapolation of past trends or speculation.” Nonetheless, I think it is important to use caution when arguing for “bold action” on the basis of theoretical models. WHO may be right in considering additional studies before a mass vaccination campaign, particularly, when injury rates are high.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s