Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review
Raw Text
Search in PubMed
Search in NLM Catalog
Add to Search
.
2010 May;14(25):iii-iv, ix-xii, 1-107.
doi: 10.3310/hta14250.
J Chilcott  1 ,
P Tappenden ,
A Rawdin ,
M Johnson ,
E Kaltenthaler ,
S Paisley ,
D Papaioannou ,
A Shippam
Expand
Affiliation
1 School of Health and Related Research (ScHARR), Regent Court, Sheffield, UK.
PMID: 20501062
DOI: 10.3310/hta14250
Free article
J Chilcott  et al.
Health Technol Assess .
2010 May .
Free article
Show details
Display options
Format
Search in PubMed
Search in NLM Catalog
Add to Search
.
2010 May;14(25):iii-iv, ix-xii, 1-107.
doi: 10.3310/hta14250.
Authors
J Chilcott  1 ,
P Tappenden ,
A Rawdin ,
M Johnson ,
E Kaltenthaler ,
S Paisley ,
D Papaioannou ,
A Shippam
Affiliation
1 School of Health and Related Research (ScHARR), Regent Court, Sheffield, UK.
PMID: 20501062
DOI: 10.3310/hta14250
Cite
Display options
Format
Abstract
Background: Health policy decisions must be relevant, evidence-based and transparent. Decision-analytic modelling supports this process but its role is reliant on its credibility. Errors in mathematical decision models or simulation exercises are unavoidable but little attention has been paid to processes in model development. Numerous error avoidance/identification strategies could be adopted but it is difficult to evaluate the merits of strategies for improving the credibility of models without first developing an understanding of error types and causes.
Objectives: The study aims to describe the current comprehension of errors in the HTA modelling community and generate a taxonomy of model errors. Four primary objectives are to: (1) describe the current understanding of errors in HTA modelling; (2) understand current processes applied by the technology assessment community for avoiding errors in development, debugging and critically appraising models for errors; (3) use HTA modellers' perceptions of model errors with the wider non-HTA literature to develop a taxonomy of model errors; and (4) explore potential methods and procedures to reduce the occurrence of errors in models. It also describes the model development process as perceived by practitioners working within the HTA community.
Data sources: A methodological review was undertaken using an iterative search methodology. Exploratory searches informed the scope of interviews; later searches focused on issues arising from the interviews. Searches were undertaken in February 2008 and January 2009. In-depth qualitative interviews were performed with 12 HTA modellers from academic and commercial modelling sectors.
Review methods: All qualitative data were analysed using the Framework approach. Descriptive and explanatory accounts were used to interrogate the data within and across themes and subthemes: organisation, roles and communication; the model development process; definition of error; types of model error; strategies for avoiding errors; strategies for identifying errors; and barriers and facilitators.
Results: There was no common language in the discussion of modelling errors and there was inconsistency in the perceived boundaries of what constitutes an error. Asked about the definition of model error, there was a tendency for interviewees to exclude matters of judgement from being errors and focus on 'slips' and 'lapses', but discussion of slips and lapses comprised less than 20% of the discussion on types of errors. Interviewees devoted 70% of the discussion to softer elements of the process of defining the decision question and conceptual modelling, mostly the realms of judgement, skills, experience and training. The original focus concerned model errors, but it may be more useful to refer to modelling risks. Several interviewees discussed concepts of validation and verification, with notable consistency in interpretation: verification meaning the process of ensuring that the computer model correctly implemented the intended model, whereas validation means the process of ensuring that a model is fit for purpose. Methodological literature on verification and validation of models makes reference to the Hermeneutic philosophical position, highlighting that the concept of model validation should not be externalized from the decision-makers and the decision-making process. Interviewees demonstrated examples of all major error types identified in the literature: errors in the description of the decision problem, in model structure, in use of evidence, in implementation of the model, in operation of the model, and in presentation and understanding of results. The HTA error classifications were compared against existing classifications of model errors in the literature. A range of techniques and processes are currently used to avoid errors in HTA models: engaging with clinical experts, clients and decision-makers to ensure mutual understanding, producing written documentation of the proposed model, explicit conceptual modelling, stepping through skeleton models with experts, ensuring transparency in reporting, adopting standard housekeeping techniques, and ensuring that those parties involved in the model development process have sufficient and relevant training. Clarity and mutual understanding were identified as key issues. However, their current implementation is not framed within an overall strategy for structuring complex problems.
Limitations: Some of the questioning may have biased interviewees responses but as all interviewees were represented in the analysis no rebalancing of the report was deemed necessary. A potential weakness of the literature review was its focus on spreadsheet and program development rather than specifically on model development. It should also be noted that the identified literature concerning programming errors was very narrow despite broad searches being undertaken.
Conclusions: Published definitions of overall model validity comprising conceptual model validation, verification of the computer model, and operational validity of the use of the model in addressing the real-world problem are consistent with the views expressed by the HTA community and are therefore recommended as the basis for further discussions of model credibility. Such discussions should focus on risks, including errors of implementation, errors in matters of judgement and violations. Discussions of modelling risks should reflect the potentially complex network of cognitive breakdowns that lead to errors in models and existing research on the cognitive basis of human error should be included in an examination of modelling errors. There is a need to develop a better understanding of the skills requirements for the development, operation and use of HTA models. Interaction between modeller and client in developing mutual understanding of a model establishes that model's significance and its warranty. This highlights that model credibility is the central concern of decision-makers using models so it is crucial that the concept of model validation should not be externalized from the decision-makers and the decision-making process. Recommendations for future research would be studies of verification and validation; the model development process; and identification of modifications to the modelling process with the aim of preventing the occurrence of errors and improving the identification of errors in models.
Similar articles
How has the impact of 'care pathway technologies' on service integration in stroke care been measured and what is the strength of the evidence to support their effectiveness in this respect? Allen D, Rixson L. Allen D, et al. Int J Evid Based Healthc. 2008 Mar;6(1):78-110. doi: 10.1111/j.1744-1609.2007.00098.x. Int J Evid Based Healthc. 2008. PMID: 21631815
Procedures and methods of benefit assessments for medicines in Germany. Bekkering GE, Kleijnen J. Bekkering GE, et al. Eur J Health Econ. 2008 Nov;9 Suppl 1:5-29. doi: 10.1007/s10198-008-0122-5. Eur J Health Econ. 2008. PMID: 18987905
Risk management frameworks for human health and environmental risks. Jardine C, Hrudey S, Shortreed J, Craig L, Krewski D, Furgal C, McColl S. Jardine C, et al. J Toxicol Environ Health B Crit Rev. 2003 Nov-Dec;6(6):569-720. doi: 10.1080/10937400390208608. J Toxicol Environ Health B Crit Rev. 2003. PMID: 14698953 Review.
[Procedures and methods of benefit assessments for medicines in Germany]. Bekkering GE, Kleijnen J. Bekkering GE, et al. Dtsch Med Wochenschr. 2008 Dec;133 Suppl 7:S225-46. doi: 10.1055/s-0028-1100954. Epub 2008 Nov 25. Dtsch Med Wochenschr. 2008. PMID: 19034813 German.
Reducing obesity and related chronic disease risk in children and youth: a synthesis of evidence with 'best practice' recommendations. Flynn MA, McNeil DA, Maloff B, Mutasingwa D, Wu M, Ford C, Tough SC. Flynn MA, et al. Obes Rev. 2006 Feb;7 Suppl 1:7-66. doi: 10.1111/j.1467-789X.2006.00242.x. Obes Rev. 2006. PMID: 16371076 Review.
See all similar articles
Cited by
A Simple Cost-Effectiveness Model of Screening: An Open-Source Teaching and Research Tool Coded in R. Lin YS, O'Mahony JF, van Rosmalen J. Lin YS, et al. Pharmacoecon Open. 2023 Jul;7(4):507-523. doi: 10.1007/s41669-023-00414-1. Epub 2023 Jun 1. Pharmacoecon Open. 2023. PMID: 37261616 Free PMC article.
Stakeholder Engagement in the Development of Public Health Economic Models: An Application to Modelling of Minimum Unit Pricing of Alcohol in South Africa. Gibbs NK, Angus C, Dixon S, Parry CDH, Meier PS. Gibbs NK, et al. Appl Health Econ Health Policy. 2023 May;21(3):395-403. doi: 10.1007/s40258-023-00789-6. Epub 2023 Mar 9. Appl Health Econ Health Policy. 2023. PMID: 36894828 Free PMC article.
A Smart Shoe Insole to Monitor Frail Older Adults' Walking Speed: Results of Two Evaluation Phases Completed in a Living Lab and Through a 12-Week Pilot Study. Piau A, Steinmeyer Z, Charlon Y, Courbet L, Rialle V, Lepage B, Campo E, Nourhashemi F. Piau A, et al. JMIR Mhealth Uhealth. 2021 Jul 5;9(7):e15641. doi: 10.2196/15641. JMIR Mhealth Uhealth. 2021. PMID: 36260404 Free PMC article. Clinical Trial.
A parsimonious model to validate cost-effectiveness analyses on preventive health care. Gandjour A. Gandjour A. BMC Health Serv Res. 2021 Nov 9;21(1):1213. doi: 10.1186/s12913-021-07217-2. BMC Health Serv Res. 2021. PMID: 34753466 Free PMC article.
Use of RWE to Inform Regulatory, Public Health Policy, and Intervention Priorities for the Developing World. McNair D, Lumpkin M, Kern S, Hartman D. McNair D, et al. Clin Pharmacol Ther. 2022 Jan;111(1):44-51. doi: 10.1002/cpt.2449. Epub 2021 Oct 31. Clin Pharmacol Ther. 2022. PMID: 34655224 Free PMC article. Review.
See all "Cited by" articles
Publication types
Review Actions Search in PubMed Search in MeSH Add to Search
MeSH terms
Data Interpretation, Statistical Actions Search in PubMed Search in MeSH Add to Search
Decision Support Techniques* Actions Search in PubMed Search in MeSH Add to Search
Evidence-Based Medicine / methods Actions Search in PubMed Search in MeSH Add to Search
Health Policy* Actions Search in PubMed Search in MeSH Add to Search
Humans Actions Search in PubMed Search in MeSH Add to Search
Policy Making Actions Search in PubMed Search in MeSH Add to Search
Qualitative Research Actions Search in PubMed Search in MeSH Add to Search
Reproducibility of Results Actions Search in PubMed Search in MeSH Add to Search
Research Design / standards* Actions Search in PubMed Search in MeSH Add to Search
Technology Assessment, Biomedical / methods* Actions Search in PubMed Search in MeSH Add to Search
Related information
Books
Cited in Books
LinkOut - more resources
Full Text Sources National Institute for Health and Care Research Journals Library
Single Line Text
Search in PubMed. Search in NLM Catalog. Add to Search. . 2010 May;14(25):iii-iv, ix-xii, 1-107. doi: 10.3310/hta14250. J Chilcott  1 , P Tappenden , A Rawdin , M Johnson , E Kaltenthaler , S Paisley , D Papaioannou , A Shippam. Expand. Affiliation. 1 School of Health and Related Research (ScHARR), Regent Court, Sheffield, UK. PMID: 20501062. DOI: 10.3310/hta14250. Free article. J Chilcott  et al. Health Technol Assess . 2010 May . Free article. Show details. Display options. Format. Search in PubMed. Search in NLM Catalog. Add to Search. . 2010 May;14(25):iii-iv, ix-xii, 1-107. doi: 10.3310/hta14250. Authors. J Chilcott  1 , P Tappenden , A Rawdin , M Johnson , E Kaltenthaler , S Paisley , D Papaioannou , A Shippam. Affiliation. 1 School of Health and Related Research (ScHARR), Regent Court, Sheffield, UK. PMID: 20501062. DOI: 10.3310/hta14250. Cite. Display options. Format. Abstract. Background: Health policy decisions must be relevant, evidence-based and transparent. Decision-analytic modelling supports this process but its role is reliant on its credibility. Errors in mathematical decision models or simulation exercises are unavoidable but little attention has been paid to processes in model development. Numerous error avoidance/identification strategies could be adopted but it is difficult to evaluate the merits of strategies for improving the credibility of models without first developing an understanding of error types and causes. Objectives: The study aims to describe the current comprehension of errors in the HTA modelling community and generate a taxonomy of model errors. Four primary objectives are to: (1) describe the current understanding of errors in HTA modelling; (2) understand current processes applied by the technology assessment community for avoiding errors in development, debugging and critically appraising models for errors; (3) use HTA modellers' perceptions of model errors with the wider non-HTA literature to develop a taxonomy of model errors; and (4) explore potential methods and procedures to reduce the occurrence of errors in models. It also describes the model development process as perceived by practitioners working within the HTA community. Data sources: A methodological review was undertaken using an iterative search methodology. Exploratory searches informed the scope of interviews; later searches focused on issues arising from the interviews. Searches were undertaken in February 2008 and January 2009. In-depth qualitative interviews were performed with 12 HTA modellers from academic and commercial modelling sectors. Review methods: All qualitative data were analysed using the Framework approach. Descriptive and explanatory accounts were used to interrogate the data within and across themes and subthemes: organisation, roles and communication; the model development process; definition of error; types of model error; strategies for avoiding errors; strategies for identifying errors; and barriers and facilitators. Results: There was no common language in the discussion of modelling errors and there was inconsistency in the perceived boundaries of what constitutes an error. Asked about the definition of model error, there was a tendency for interviewees to exclude matters of judgement from being errors and focus on 'slips' and 'lapses', but discussion of slips and lapses comprised less than 20% of the discussion on types of errors. Interviewees devoted 70% of the discussion to softer elements of the process of defining the decision question and conceptual modelling, mostly the realms of judgement, skills, experience and training. The original focus concerned model errors, but it may be more useful to refer to modelling risks. Several interviewees discussed concepts of validation and verification, with notable consistency in interpretation: verification meaning the process of ensuring that the computer model correctly implemented the intended model, whereas validation means the process of ensuring that a model is fit for purpose. Methodological literature on verification and validation of models makes reference to the Hermeneutic philosophical position, highlighting that the concept of model validation should not be externalized from the decision-makers and the decision-making process. Interviewees demonstrated examples of all major error types identified in the literature: errors in the description of the decision problem, in model structure, in use of evidence, in implementation of the model, in operation of the model, and in presentation and understanding of results. The HTA error classifications were compared against existing classifications of model errors in the literature. A range of techniques and processes are currently used to avoid errors in HTA models: engaging with clinical experts, clients and decision-makers to ensure mutual understanding, producing written documentation of the proposed model, explicit conceptual modelling, stepping through skeleton models with experts, ensuring transparency in reporting, adopting standard housekeeping techniques, and ensuring that those parties involved in the model development process have sufficient and relevant training. Clarity and mutual understanding were identified as key issues. However, their current implementation is not framed within an overall strategy for structuring complex problems. Limitations: Some of the questioning may have biased interviewees responses but as all interviewees were represented in the analysis no rebalancing of the report was deemed necessary. A potential weakness of the literature review was its focus on spreadsheet and program development rather than specifically on model development. It should also be noted that the identified literature concerning programming errors was very narrow despite broad searches being undertaken. Conclusions: Published definitions of overall model validity comprising conceptual model validation, verification of the computer model, and operational validity of the use of the model in addressing the real-world problem are consistent with the views expressed by the HTA community and are therefore recommended as the basis for further discussions of model credibility. Such discussions should focus on risks, including errors of implementation, errors in matters of judgement and violations. Discussions of modelling risks should reflect the potentially complex network of cognitive breakdowns that lead to errors in models and existing research on the cognitive basis of human error should be included in an examination of modelling errors. There is a need to develop a better understanding of the skills requirements for the development, operation and use of HTA models. Interaction between modeller and client in developing mutual understanding of a model establishes that model's significance and its warranty. This highlights that model credibility is the central concern of decision-makers using models so it is crucial that the concept of model validation should not be externalized from the decision-makers and the decision-making process. Recommendations for future research would be studies of verification and validation; the model development process; and identification of modifications to the modelling process with the aim of preventing the occurrence of errors and improving the identification of errors in models. Similar articles. How has the impact of 'care pathway technologies' on service integration in stroke care been measured and what is the strength of the evidence to support their effectiveness in this respect? Allen D, Rixson L. Allen D, et al. Int J Evid Based Healthc. 2008 Mar;6(1):78-110. doi: 10.1111/j.1744-1609.2007.00098.x. Int J Evid Based Healthc. 2008. PMID: 21631815. Procedures and methods of benefit assessments for medicines in Germany. Bekkering GE, Kleijnen J. Bekkering GE, et al. Eur J Health Econ. 2008 Nov;9 Suppl 1:5-29. doi: 10.1007/s10198-008-0122-5. Eur J Health Econ. 2008. PMID: 18987905. Risk management frameworks for human health and environmental risks. Jardine C, Hrudey S, Shortreed J, Craig L, Krewski D, Furgal C, McColl S. Jardine C, et al. J Toxicol Environ Health B Crit Rev. 2003 Nov-Dec;6(6):569-720. doi: 10.1080/10937400390208608. J Toxicol Environ Health B Crit Rev. 2003. PMID: 14698953 Review. [Procedures and methods of benefit assessments for medicines in Germany]. Bekkering GE, Kleijnen J. Bekkering GE, et al. Dtsch Med Wochenschr. 2008 Dec;133 Suppl 7:S225-46. doi: 10.1055/s-0028-1100954. Epub 2008 Nov 25. Dtsch Med Wochenschr. 2008. PMID: 19034813 German. Reducing obesity and related chronic disease risk in children and youth: a synthesis of evidence with 'best practice' recommendations. Flynn MA, McNeil DA, Maloff B, Mutasingwa D, Wu M, Ford C, Tough SC. Flynn MA, et al. Obes Rev. 2006 Feb;7 Suppl 1:7-66. doi: 10.1111/j.1467-789X.2006.00242.x. Obes Rev. 2006. PMID: 16371076 Review. See all similar articles. Cited by. A Simple Cost-Effectiveness Model of Screening: An Open-Source Teaching and Research Tool Coded in R. Lin YS, O'Mahony JF, van Rosmalen J. Lin YS, et al. Pharmacoecon Open. 2023 Jul;7(4):507-523. doi: 10.1007/s41669-023-00414-1. Epub 2023 Jun 1. Pharmacoecon Open. 2023. PMID: 37261616 Free PMC article. Stakeholder Engagement in the Development of Public Health Economic Models: An Application to Modelling of Minimum Unit Pricing of Alcohol in South Africa. Gibbs NK, Angus C, Dixon S, Parry CDH, Meier PS. Gibbs NK, et al. Appl Health Econ Health Policy. 2023 May;21(3):395-403. doi: 10.1007/s40258-023-00789-6. Epub 2023 Mar 9. Appl Health Econ Health Policy. 2023. PMID: 36894828 Free PMC article. A Smart Shoe Insole to Monitor Frail Older Adults' Walking Speed: Results of Two Evaluation Phases Completed in a Living Lab and Through a 12-Week Pilot Study. Piau A, Steinmeyer Z, Charlon Y, Courbet L, Rialle V, Lepage B, Campo E, Nourhashemi F. Piau A, et al. JMIR Mhealth Uhealth. 2021 Jul 5;9(7):e15641. doi: 10.2196/15641. JMIR Mhealth Uhealth. 2021. PMID: 36260404 Free PMC article. Clinical Trial. A parsimonious model to validate cost-effectiveness analyses on preventive health care. Gandjour A. Gandjour A. BMC Health Serv Res. 2021 Nov 9;21(1):1213. doi: 10.1186/s12913-021-07217-2. BMC Health Serv Res. 2021. PMID: 34753466 Free PMC article. Use of RWE to Inform Regulatory, Public Health Policy, and Intervention Priorities for the Developing World. McNair D, Lumpkin M, Kern S, Hartman D. McNair D, et al. Clin Pharmacol Ther. 2022 Jan;111(1):44-51. doi: 10.1002/cpt.2449. Epub 2021 Oct 31. Clin Pharmacol Ther. 2022. PMID: 34655224 Free PMC article. Review. See all "Cited by" articles. Publication types. Review Actions Search in PubMed Search in MeSH Add to Search. MeSH terms. Data Interpretation, Statistical Actions Search in PubMed Search in MeSH Add to Search. Decision Support Techniques* Actions Search in PubMed Search in MeSH Add to Search. Evidence-Based Medicine / methods Actions Search in PubMed Search in MeSH Add to Search. Health Policy* Actions Search in PubMed Search in MeSH Add to Search. Humans Actions Search in PubMed Search in MeSH Add to Search. Policy Making Actions Search in PubMed Search in MeSH Add to Search. Qualitative Research Actions Search in PubMed Search in MeSH Add to Search. Reproducibility of Results Actions Search in PubMed Search in MeSH Add to Search. Research Design / standards* Actions Search in PubMed Search in MeSH Add to Search. Technology Assessment, Biomedical / methods* Actions Search in PubMed Search in MeSH Add to Search. Related information. Books. Cited in Books. LinkOut - more resources. Full Text Sources National Institute for Health and Care Research Journals Library.