Axel Leijonhufvud's piece on economists, "Life Among the Econ" did a brilliant takedown of what he termed "modls" as in "recognized status has come increasingly to be tied to cleverness in modl-making." Raising model-making to such high status has created considerable scepticism towards the discipline. There is now a new brilliant paper by 22 researchers - a manifesto with five principles to ensure that mathematical models best serve society (June 2020 Nature 582(7813):482-484).
1. Mind the assumptions: The results of models are only as good as their assumptions, which are often not adequately assessed. Solutions from one model therefore do not fit alternative situations. The recommendation is to undertake global uncertainty and sensitivity analyses - that is, allowing all that is uncertain — variables, mathematical relationships and boundary conditions — to vary simultaneously as runs of the model produce its range of predictions. Many a times the uncertainty in predictions is substantially larger than originally asserted, there should be transparency about the analysis to provide real value and applicability.
2. Mind the hubris: Researchers must resist the attraction of adding complexity to a model without reason. The authors note the lack of accountability leads to such hubris: "Whereas an engineer is called to task if a bridge falls, other models tend to be developed with large teams and use such complex feedback loops that no one can be held accountable if the predictions are catastrophically wrong."
3. Mind the framing: This is a critical principle that many forget to factor into their work. The example given of cost-benefit analysis is a common one we run up against when projects are being evaluated.
"Modellers know that the choice of tools will influence, and could even determine, the outcome of the analysis, so the technique is never neutral. For example, the GENESIS model of shoreline erosion was used by the US Army Corps of Engineers to support cost–benefit assessments for beach preservation projects. The cost–benefit model could not predict realistically the mechanisms of beach erosion by waves or the effectiveness of beach replenishment by human intervention. It could be easily manipulated to boost evidence that certain coastal-engineering projects would be beneficial7 . A fairer assessment would have considered how extreme storm events dominate in erosion processes."
"Modellers should not hide the normative values of their choices."
The authors therefore recommend that there should be international guidelines for a set of social norms that would include how to produce a model, assess its uncertainty and communicate the results.
4. Mind the consequences: The authors take examples from the ongoing Covid-19 crisis to show how quantification can backfire towards giving precisely wrong results and spurious predictions cause significant harm. The recommendation therefore is to avoid opacity, and give full disclosure on the uncertainty - this alone will build trust and utility of modelling.
5. Mind the unknowns: Acknowledging ignorance is no crime, in fact it builds accountability and trust. "Experts should have the courage to respond that “there is no number-answer to your question”, as US government epidemiologist Anthony Fauci did when probed by a politician."
These principles have been set for responsible modeling in all disciplines, but will mean the most towards setting appropriate policies in developing economies.
1. Mind the assumptions: The results of models are only as good as their assumptions, which are often not adequately assessed. Solutions from one model therefore do not fit alternative situations. The recommendation is to undertake global uncertainty and sensitivity analyses - that is, allowing all that is uncertain — variables, mathematical relationships and boundary conditions — to vary simultaneously as runs of the model produce its range of predictions. Many a times the uncertainty in predictions is substantially larger than originally asserted, there should be transparency about the analysis to provide real value and applicability.
2. Mind the hubris: Researchers must resist the attraction of adding complexity to a model without reason. The authors note the lack of accountability leads to such hubris: "Whereas an engineer is called to task if a bridge falls, other models tend to be developed with large teams and use such complex feedback loops that no one can be held accountable if the predictions are catastrophically wrong."
3. Mind the framing: This is a critical principle that many forget to factor into their work. The example given of cost-benefit analysis is a common one we run up against when projects are being evaluated.
"Modellers know that the choice of tools will influence, and could even determine, the outcome of the analysis, so the technique is never neutral. For example, the GENESIS model of shoreline erosion was used by the US Army Corps of Engineers to support cost–benefit assessments for beach preservation projects. The cost–benefit model could not predict realistically the mechanisms of beach erosion by waves or the effectiveness of beach replenishment by human intervention. It could be easily manipulated to boost evidence that certain coastal-engineering projects would be beneficial7 . A fairer assessment would have considered how extreme storm events dominate in erosion processes."
"Modellers should not hide the normative values of their choices."
The authors therefore recommend that there should be international guidelines for a set of social norms that would include how to produce a model, assess its uncertainty and communicate the results.
4. Mind the consequences: The authors take examples from the ongoing Covid-19 crisis to show how quantification can backfire towards giving precisely wrong results and spurious predictions cause significant harm. The recommendation therefore is to avoid opacity, and give full disclosure on the uncertainty - this alone will build trust and utility of modelling.
5. Mind the unknowns: Acknowledging ignorance is no crime, in fact it builds accountability and trust. "Experts should have the courage to respond that “there is no number-answer to your question”, as US government epidemiologist Anthony Fauci did when probed by a politician."
These principles have been set for responsible modeling in all disciplines, but will mean the most towards setting appropriate policies in developing economies.
No comments:
Post a Comment