Appendix A. Measure Evaluation Framework

Appendix A presents a matrix of measure evaluation criteria* used by five major national organizations. The matrix is borrowed from AHRQ's Guidance for Using the AHRQ Quality Indicators for Public Reporting or Payment - Appendix B: Public Reporting Evaluation Framework. Please note that terminology may vary by framework.

Public Reporting Evaluation Framework
Evaluation criteria and requirements	National Quality Forum	AHRQ Quality Indicators	National Healthcare Quality Report	The Joint Commission	National Committee for Quality Assurance
1. Importance	Important	—Face validity —Foster real quality improvement	Importance		Relevance
Assesses an important leverage point for improving quality; significant to target audiences; impact on health Opportunity for improvement, considerable variation in quality of care exists Aspect of quality is under provider or health system control Should not create incentives or rewards to improve without truly improving quality of care	Leverage point for improving quality Considerable variation in quality of care exists Performance in the area is suboptimal Aspect of quality is under provider or health system control¹	Measure an important aspect of quality that is subject to provider or health system control Should not create incentives or rewards to improve without truly improving quality of care	Impact on health Meaningfulness Susceptibility to being influenced by health care	Targets improvement in the health of populations Under provider control	Strategic importance Health importance Meaningfulness to decision makers Variance among systems Potential for improvement Controllability Financial importance

* Criteria are taken verbatim from the various sources and have not been edited.

¹ This criterion is in the NQF framework at the scope/priority level and not at the individual measure evaluation level.

Evaluation criteria and requirements	National Quality Forum	AHRQ Quality Indicators	National Healthcare Quality Report	The Joint Commission	National Committee for Quality Assurance
2. Scientific acceptability	Scientifically acceptable	—Precision —Minimum bias —Construct validity	Scientific soundness		Scientific soundness
Relationship to quality is based on scientific evidence Well defined and precisely specified Valid, measures the intended aspect of quality; accurately represents the concept being evaluated; data sources are comparable Adequate proportion of total variation is explained by provider performance and amount of variation in measurement is small after provider performance and patient characteristics are taken into account Reliable, producing the same results a high proportion of time in the same population Precise, adequately discriminating between real differences in provider performance and reasonable sample size exists to detect actual differences; captures all possible cases and bias related to case exclusion or limited data are minimal. Risk adjustment is adequate to address confounding bias	Well defined and precisely specified Reliable, producing the same results a high proportion of time in the same population Valid, accurately representing the concept being evaluated Precise, adequately discriminating between real differences in provider performance Adaptable to patient preferences and variety of settings Adequate and specified risk adjustment strategy exists Evidence is available linking process measures to outcomes	Have relatively large variation among providers that is not due to random variation or patient characteristics Should not be affected by systematic differences in patient case-mix When systematic differences exist, an adequate risk adjustment system is available based on HCUP discharge data Supported by evidence of a relationship to quality Related to other indicators intended to measure the same or related aspects of quality	Explicitness of the evidence base Reliability Validity	Precisely defined and specified Reliable Valid Risk-adjusted or stratified	Clinical evidence linking processes, outcomes, interventions Reproducibility Validity (face, construct, content) Accuracy Case-mix risk adjustment methods Comparability of data sources

Evaluation criteria and requirements	National Quality Forum	AHRQ Quality Indicators	National Healthcare Quality Report	The Joint Commission	National Committee for Quality Assurance
3. Usability	Usable	Application
Effective (understandable and clear) presentation and dissemination strategies exist Statistical testing can be applied to communicate when differences in performance levels are greater than would be expected by chance Has been used effectively in the past and/or has high potential for working well with other indicators currently in use Compelling content for stakeholder decision making	Measure can be used by stakeholders for decision making Performance differences are statistically meaningful Performance differences are practically and clinically meaningful Risk stratification, risk adjustment, and other forms of recommended analyses can be applied appropriately Effective presentation and dissemination strategies exist Information produced can be used by at least one health care stakeholder audience to make a decision or take action Information about specific conditions under which the measure is appropriate to use has been given Methods to aggregate the measure with related measures are defined if determined to be more under-standable and useful	Have been used effectively in the past Have high potential for working well with other indicators currently in use		Can be interpreted and useful in the accreditation process

4. Feasibility	Feasibility		Feasibility		Feasibility
Consistent construction and assessment of the measure Feasible to calculate; benefits exceed financial and administrative burden of implementation Confidentiality concerns are addressed Audit strategy can be implemented, quality of data are known	Data collection tied to care delivery when feasible Timing and frequency of measure collection are specified Benefit evaluated against financial and administrative burden of implementation Confidentiality concerns are addressed Audit strategy is designed and can be implemented		Availability of required data across the system Cost or burden of measurement Existence of prototypes Capacity of data and measure to support subgroup analyses	Data collection effort is assessed	Precise definition (under scientific soundness in other frameworks) Reasonable cost Logistical feasibility Confidentiality Auditability

Source: Remus D, Fraser I. Guidance for using the AHRQ Quality Indicators for hospital-level public reporting or payment. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ Pub. No. 04-0086-EF. Available at: http://qualityindicators.ahrq.gov/downloads/technical/qi_guidance_appendix_B.pdf.

Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives

Appendix A. Measure Evaluation Framework

Table of Contents