"Measuring the Impact of Nurse Staffing on Patient Outcomes: The Effect of Data Aggregation and Estimation Methods"

Please note: All research in progress seminars are off-the-record. Any information about methodology and/or results is embargoed until publication.

Research Objectives: A growing body of evidence shows that nurse staffing levels and composition affect patient outcomes.  This evidence has come from difference data sources, with different levels of data aggregation, and used different estimation methods.  The problem of unobserved heterogeneity (unobserved characteristics that affect outcomes) is large for this area of research and estimates that don’t address this are almost certainly biased.  We used a large, longitudinal, patient-level dataset with monthly, unit-level nurse staffing data to examine how different levels of data aggregation and different statistical methods affected the estimates of the effect of nurse staffing on patient outcomes.

Study Design:  Monthly staffing for each unit, for each type of nurse (registered nurse, Licensed Practical Nurses, nursing aides, contract nurses), were obtained from VA accounting data.  Payroll data provided education levels and how ng each nurse had worked on the unit (unit tenure).  Patient characteristics and length of stay (LOS) were obtained from VA hospital discharge records.  Log(LOS) was used as the dependent variable as it captures the effect of many nursing-sensitive patient outcomes.  The model controlled for patient age, expected LOS, and patient co-morbidities; the variables of interest were nurse staffing, nurse skill-mix, and unit tenure.  The models were estimated using both ordinary least squares (OLS) and fixed-effects (FE) regressions; the latter was used to address unobserved heterogeneity.  All regressions were patient-level, with different levels of aggregation for the nurse staffing variables (unit-month, unit-year, hospital-month, and hospital-year) and the unit level models were estimated for all units together, and separately for acute care units and intensive care units. 

Population Studied:  All VA acute medical care units (including ICUs) for 2003-2006.  1,923,048 patients from 427 units across 138 VA Medical Centers.

Principal Findings:  The results were quite sensitive to both estimation method and unit of aggregation.  The change in the point estimates of the effects of nurse staffing on LOS of switching from monthly to annual staffing data ranged from 14-1177% for the FE models and 13-276% (plus two reversals, -0.20 to 0.27 and -0.09 to 0.40) for the OLS models.  These ranges were even larger across all levels of aggregation.  For the same level of aggregation, the difference between the OLS and FE estimates ranged from 0-304% and there were two cases of sign reversal (-0.21 to 0.27 and -0.19 to 0.30).

Conclusions:  The magnitude and even the direction of the effects of different elements of nurse staffing on patient outcomes are quite sensitive to the level of aggregation and estimation method. 

Implications for Policy or Practice:  Interpretation of the results of studies of nurse staffing on patient outcomes needs to account for the level of data aggregation and the statistical methods used.  Higher levels of aggregation, both across time and across units, probably masks effects.  Thus, studies that measure nurse staffing at the unit-level data with shorter time intervals yield more reliable estimates.  Studies that fail to account for unobserved heterogeneity are probably biased.  But, FE models also have limits, as they only estimate marginal effects and can’t directly compare the effects of high vs. low staffing levels.

"Equalizing child sex ratios in India: Understanding the trends, distribution, composition, potential drivers, and impact on fertility"

Please note: All research in progress seminars are off-the-record. Any information about methodology and/or results is embargoed until publication.

Abstract

I will present on the findings of my research on the changing patterns of child sex ratios in India, this includes an exploration of whether child sex ratios are improving in districts with the most uneven child sex ratios in recent years, and what factors are associated with this improvement. I also decompose the improvements in child sex ratios into improvements due to less sex selective abortion vs improved girl child mortality compared to boy child mortality. I then discuss initial findings on potential drivers of the improvement, with the hope to spark a discussion on other ways to think about these findings, specifically related to measuring the impact of policies. Finally, I will present some very preliminary findings from a work in progress on how sex preferences are impacting overall fertility in India.

"Re-tooling cost-effectiveness analysis for global health relevance"

Please note: All research in progress seminars are off-the-record. Any information about methodology and/or results is embargoed until publication.

To identify priorities for action in global health, decision makers need information on the potential impact, costs and cost effectiveness of different possible choices regarding health technologies and interventions. A large volume of cost-effectiveness analysis has been produced to try to meet this need, but its impact on policies and programs in low- and middle-income countries has evidently been limited. In this seminar we will explore some possible reasons for the relatively modest policy impact of cost-effectiveness analysis in global health and propose directions for re-thinking the approaches and methods that are commonly used in the field. Drawing examples from our recent and ongoing research in areas such as HIV/AIDS, tuberculosis and maternal and child health, we will describe an agenda to pivot the practice of decision science in global health toward a more systematic approach to comparative strategy evaluation.

"Wisdom of the Crowd or Tyranny of the Mob? OrderRex: Data-Mining Electronic Health Records for Clinical Decision Support"

Please note: All research in progress seminars are off-the-record. Any information about methodology and/or results is embargoed until publication.

Background

Uncertainty and undesirable variability is pervasive in medical decision making.  Clinical decision support like order sets help distribute expertise, but are constrained by resource intensive manual development. 

Objective

To overcome scalability limitations by automatically generating decision support content from existing practice patterns, analogous to Amazon.com’s product recommender.  To perform the first structured validation of such a system against external standards-of-care and outcome predictions.

Methods

We extracted deidentified electronic health record data from all hospitalizations at Stanford Hospital in 2011 (>5.4M structured data items from >19K patients) to build a system with association statistics for 811 clinical orders (e.g., labs, imaging, medications) and clinical outcomes.  We manually reviewed the National Guideline Clearinghouse for diagnoses of chest pain, gastrointestinal hemorrhage, and pneumonia.  We compared system generated clinical orders against guideline referenced orders by receiver operating characteristic (ROC) analysis.  Human authored order sets provided real-world benchmarks.  We compared predicted vs. actual outcomes by ROC analysis for separate validation patients.

Results

System generated orders were overall consistent with guidelines (ROC AUC c-statistics 0.89, 0.95, 0.83) and improve upon statistical prevalence (0.76, 0.74, 0.73) and pre-existing order sets (0.81, 0.77, 0.73) (P<10^-30 in all cases).  Clinical outcome prediction ROC AUC c-statistics were 0.84 for 30 day mortality , 0.84 for 1 week ICU life support, 0.80 for 1 week discharge / length of stay, and 0.68 for 30 day readmission.

Conclusions

Automatically generated order suggestions can reproduce and even optimize manual constructs like order sets while remaining largely concordant with guidelines and avoiding inappropriate recommendations.  This has even more important implications for prevalent cases where well-defined guidelines and order sets do not exist.  The same methodology is predictive of clinical outcomes comparable to state-of-the-art prognosis models (e.g., APACHE II), pointing to opportunities to link suggestions against favorable outcomes.

Abstract: A number of senior Intelligence Community (IC) officials describe compliance as one of the IC’s biggest problems, perhaps the biggest. The underlying legal and informational issues are bound to become more acute and complex.  How can AI help? The IC protects our nation by analyzing the relationships between people, places, and things - essentially "connecting the dots.”  Doing so while remaining compliant with policies such as Executive Order 12333[1] and Presidential Policy Directive 28[2] is a balancing act. The interpretation, implementation, and enforcement of policy vary across organizations and administrations.  This frequently leaves analysts struggling to determine what data they can and cannot look at. The Internet, mobile, and “Big Data” generally further complicate the problem. The sheer volume, velocity, and variety of data that is constantly being generated necessitate automation, and even AI, to manage.  However, the benefits of analytic automation over the data deluge will remain limited, until the IC finds a way to scale the processing of legal judgments at a comparable rate.

Abstract:

This research explores the conditions under which government actors and citizens recognize and incorporate the interests of others in their policy decisions and actions. To do so requires a change in beliefs about whose interests are intertwined and what it is possible and appropriate to do as a polity. This in turn demands changes in governance arrangements and, ultimately, norms of behavior.

Speaker Bio:

Image

Co-sponsored by the Health Economics Seminar

We present the first direct evidence on the relative quality of public and private healthcare in a low-income setting, using a unique set of audit studies. We sent standardized (fake) patients to rural primary care providers in the Indian state of Madhya Pradesh, and recorded the quality of care provided and prices charged in each interaction. We report three main findings. First, most private providers lacked formal medical training, but they spent more time with patients and completed more essential checklist items than public providers and were equally likely to provide a correct treatment. Second, we compare the performance of qualified public doctors across their public and private practices and find that the same doctors exerted higher effort and were more likely to provide a correct treatment in their private practices. Third, in the private sector, we find that prices charged are positively correlated with provider effort and correct treatment, but also with unnecessary treatments. In the public sector, we find no correlation between provider salaries and any measure of quality. We develop a simple theoretical framework to interpret our results and show that in settings with low levels of effort in the public sector, the benefits of higher diagnostic effort in the private sector may outweigh the costs of market incentives to over treat. These differences in provider effort may partly explain the dominant market share of fee-charging private providers even in the presence of a system of free public healthcare.

Image

Born and raised in India, he earned an A.B. in economics (summa cum laude) from Harvard, an M.Phil. in economics from Cambridge (UK), and a Ph.D. in economics from Harvard. He is a Research Associate of the National Bureau of Economic Research (NBER), an Affiliate at the Bureau for Research and Economic Analysis of Development (BREAD), a Member of the Jameel Poverty Action Lab (J-PAL) network, an Affiliate at the Center for Effective Global Action (CEGA), and a Research Affiliate with Innovations for Poverty Action (IPA).

Prof. Muralidharan's primary research interests include development, public, and labor economics. Specific topics of interest include education, health, and social protection; measuring quality of public service delivery; program evaluation; and improving the effectiveness of public spending (with a focus on developing countries). Courses taught include undergraduate and graduate classes in development economics, program evaluation, and the economics of education.