empirical research health care utilization

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Empirical‐Based Typology of Health Care Utilization by Medicare Eligible Veterans

Mary vaughan sarrazin , ph.d., gary e rosenthal , m.d., f.a.c.p., carolyn l turvey , ph.d..

• Author information
• Article notes
• Copyright and License information

Address correspondence to Mary Vaughan Sarrazin, Ph.D., Iowa City VA Health Care System (152), 601 Highway 6 West, Iowa City, IA 52246, and Department of Internal Medicine, University of Iowa, Iowa City, IA; e‐mail: [email protected] .

Corresponding author.

Issue date 2018 Dec.

Up to 70 percent of patients who receive care through Veterans Health Administration (VHA) facilities also receive care from non‐VA providers. Using applied classification techniques, this study sought to improve understanding of how elderly VA patients use VA services and complementary use of non‐VA care.

The study included 1,721,900 veterans age 65 and older who were enrolled in VA and Medicare during 2013 with at least one VA encounter during 2013. Outpatient and inpatient encounters and medications received in VA were classified, and mutually exclusive patient subsets distinguished by patterns of VA service use were derived empirically using latent class analysis (LCA). Patient characteristics and complementary use of non‐VA care were compared by patient subset.

Five patterns of VA service use were identified that were distinguished by quantity of VA medical and specialty services, medication complexity, and mental health services. Low VA Medical users tend to be healthier and rely on non‐VA services, while High VA users have multiple high cost illnesses and concentrate their care in the VA.

Conclusions

VA patients distinguished by patterns of VA service use differ in illness burden and the use of non‐VA services. This information may be useful for framing efforts to optimize access to care and care coordination for elderly VA patients.

Keywords: Administrative data uses, access/demand/utilization of services, chronic disease, Medicare, VA Health Care System

An estimated 40–70 percent of patients who receive care through Veterans Health Administration (VHA) facilities also receive care from non‐VA community providers (Hynes et al. 2007 ; Liu et al. 2010 , 2011a ; U.S. Department of Veterans Affairs 2011 ; Humensky et al. 2012 ). This proportion is likely to grow in light of the 2014 Veterans Choice Act, which stipulates specific conditions under which VA will pay for care received in the community. Moreover, the proportion of Veterans who qualify for Medicare, a major contributor to dual use, will grow as the Veteran population ages.

Although the VA provides a comprehensive array of services, including acute medical and surgical care, primary care, mental health services, prescription medications, and other specialty care, dual use of VA and non‐VA services may increase access to important services for many veterans. However, dual use also complicates efforts to coordinate health services and optimize outcomes, potentially leading to duplication of services (Trivedi et al. 2012 ), care fragmentation (Hynes et al. 2007 ; Shen et al. 2008 ), and worse health outcomes (Wolinsky et al. 2006 , 2007 ; Jia et al. 2007 ).

In the VA, coordination of care occurs by nurse care managers functioning in teams in VA primary care clinics. However, efforts to coordinate care for VA patients are complicated by the fact that substantial portions of VA patients receive care outside the VA, and dual use is growing. These issues are particularly important for elderly veterans, given the availability of Medicare to most patients age 65 and older and the substantial morbidity in elderly populations.

Understanding how veterans access different services within the VA and private sector is important for policy planning and allocation of resources on a national level as well as providing a framework around which efforts to coordinate care can be implemented on the patient level. To address this gap, the current study used VA administrative data for patients age 65 and older to empirically derive a typology of VA patient service utilization using latent class analysis (LCA; Collins and Lanza 2010 ). LCA is a probabilistic method of cluster analysis that reduces a complex array of multivariate data to a format that is parsimonious and easy to comprehend. Using LCA, we grouped VA patients into similar classes based on their VA utilization patterns. We subsequently examined patient characteristics and patterns of utilization of non‐VA services within each class defined to reveal insights into how Veterans complement VA care with care in the community.

Sources included administrative records pertaining to services received during 2013 through the VHA, and outside the VHA through Medicare. Information pertaining to VA encounters were obtained from VA's Medical SAS (MedSAS) datasets which contain national administrative data on VA patient inpatient and outpatient encounters (VIREC 2015a , b ). Medications received through VA pharmacies were identified from Managerial Cost Accounting National extracts (Phibbs, Barnett, and Fan 2015 ). Medicare enrollment details, Part A (hospital insurance) and Part B (medical insurance) fee for service claims submitted by non‐VA providers for reimbursement for hospital inpatient, hospital outpatient, and clinician office services provided to VA patients were obtained from the VA Information Resource Center (VIREC). Medicare records are available through an Information Exchange Agreement between the VA and the Centers for Medicare and Medicaid Services (CMS), which allows VIReC to maintain a repository of Medicare claims for VA patients who receive care through the VA and are also eligible for Medicare (U.S. Department of Veterans Affairs 2015 , 2016 ).

This initiative began in 1998 to establish a VA system of records and to acquire, merge, and maintain VA and Medicare beneficiary utilization data. The repository represents all individuals who have used health care services from the VHA since 1992, enrolled in the VHA, or are eligible for VHA health care services through compensation or pension benefits. The repository is updated annually through a process by which VHA submits to CMS an updated list of social security numbers (SSN) for individuals newly eligible for VHA care. Prior to release, VIREC assigns unique scrambled social security numbers to the Medicare claims that are consistent with scrambled social security numbers provided in other VA national patient‐level data, to allow merging CMS claims with VA health care records.

We identified 2,304,251 patients age 65 or older who used VA in 2013 who were also enrolled in Medicare Part A for at least 1 month in 2013. Patients were excluded if they were enrolled in a Medicare managed care plan at any time during 2013 (due to incomplete capture of claims for Medicare managed care enrollees), leaving 1,721,900 unique patients (Figure  1 ).

Summary of Cohort Selection Criteria

Service Use

Outpatient services received in VA clinics were categorized using a multi‐step process, building on an established algorithm developed by Burgess et al. ( 2011 ). First, visits to physicians, physician assistants, nurse practitioners, and nonphysician mental health providers were identified; visits not in one of these provider categories were excluded. We then categorized provider types and CPT codes associated with each visit. Provider types included primary care (internal medicine, family practice, and geriatric medicine), mental health (psychiatrist, psychologist, mental health social workers), diagnostic radiology, other specialty care (medical subspecialists and surgical specialties), and rehabilitative therapists (physical, occupational, respiratory therapists). Physician assistants and nurse practitioners were considered primary care providers unless another specialty was noted. CPT categories included psychiatry services, evaluation and management (E and M) visits, diagnostic radiology, and rehabilitative therapy. Finally, each visit was categorized based on a combination of provider type and CPT category as follows: Mental health visits included all visits with a psychiatry CPT code or visits with a mental health provider and a concurrent E and M CPT code; primary care included visits to a primary care physician with an E and M CPT code; specialty care included visits to all other physician specialties with a E and M CPT code. Visits with a diagnostic radiology CPT code or visits to a diagnostic radiologist were considered diagnostic radiology; visits with a physical, rehabilitative, or occupational therapist were considered rehabilitative therapy. We applied the same multi‐step algorithm of Burgess et al. ( 2011 ) to Medicare Part A and Part B claims to identify the use of hospital outpatient or clinician office services outside the VA by patients age 65 and older.

Acute inpatient admissions in VA and non‐VA hospitals during FY2013 were identified in the VA MedSAS Files and Medicare (Part A) Provider Analysis and Review (MedPAR) files, respectively. Finally, medications received through the VA were categorized using a modification of the VA Drug Class variable to identify 25 different classes of drugs. In addition, patients who received any psychiatric medication were identified. Psychiatric medications were defined primarily as tricyclic anti‐depressants, selective serotonin update inhibitors, serotonin norepinephrine reuptake inhibitors, anti‐psychotics, central nervous system stimulants, benzodiazepines, and miscellaneous sedatives and hypnotics used to treat anxiety and depression. We did not confirm that these drugs were taken for a psychiatric diagnosis; thus, this list may include occasional use of some medications for nonpsychiatric purposes.

Latent Class Models

Latent class analysis models were developed to reflect VA service use only and included 1,721,900 patients age 65 and older who used any VA services during FY2013. Prior to generating latent class models, we inspected the distribution of each VA service variable and categorized service use into two to four levels. The number of primary care visits and number of unique drug classes were both categorized into four levels (categories were 0, 1, 2–3, and 4 or more for primary care visits, 0–4, 5–9, 10–14, and 15 or more for drug classes). Specialty physician visits and mental health visits were categorized into three levels (0, 1, and 2 or more specialty visits; 0, 1–2, and 3 or more mental health visits). Receipt of rehabilitative therapy, diagnostic radiology, inpatient admission, and psychiatric medication was dichotomized to reflect none or any. In sensitivity analyses, we modified the category cutoffs.

Latent class models were generated using PROC LCA with SAS and were estimated using maximum‐likelihood estimation (Lanza et al. 2007 ). Because the optimal number of classes is not known in advance, it is often necessary in LCA to estimate the optimal number of classes underlying the data. This was performed by comparing the fit of models with increasing numbers of classes on the basis of interpretability and meaning of classes, latent class separation, model entropy, and the Bayesian Information Criteria (BIC). Latent Class separation refers to the certainty with which patients can be assigned to a single class using the mean posterior probability of class membership (Collins and Lanza 2010 ). Thus, in a five‐class LCA model, each patient has five posterior probabilities (indicating the probability of belonging to each of the five classes, given the observed service use). When there is good class separation, patients have a large probability of membership in a single class, with low probabilities for the remaining latent classes. Entropy is a single score that summarizes the certainty in patient classification (Collins et al. 1993 ). Entropy ranges from 0 to 1, with 1 indicating perfect separation between classes. The BIC reflects the degree to which the model fits the underlying data (Dziak et al. 2012 ).

After model development, we compared the use of specific types of health services inside versus outside the VA across LCA classes as identified in VA and Medicare claims. We also compared patient characteristics across LCA classes, including demographics (age, sex, race, ethnicity, rural residence, enrollment in Medicare Part D or Medicaid per Medicare enrollment files, VA Means Test category) and illness burden. Race and ethnicity were defined using the race code developed by the Research Triangle Institute (RTI) that is available on the CMS Beneficiary Summary File and has excellent agreement with self‐reported race (Bonito et al. 2008 ). Rural residence was defined based on the VA Office of Rural Health designation which incorporates Rural–Urban Commuting Areas (RUCA; U.S. Department of Agriculture 2015 ). Distance to nearest VAMC and nearest primary care were calculated using straight line distance between patient residence (based on longitude and latitude coordinates) and the longitude and latitude coordinates of the nearest VAMC or primary care clinic as identified on the VA Station Tracking (VAST) database. Illness burden was measured by the Medicare Hierarchical Condition Category (HCC) software, which identifies 79 mutually exclusive medical conditions by ICD‐9‐CM code. Conditions are additive, so that more HCC conditions means a greater disease burden. In addition, we identified and tallied for each patient the presence of 15 specific HCC conditions identified by Medicare as requiring chronic or special needs (Research Triangle Institute 2008 ). In order to obtain a full assessment of patient illness burden regardless of where services were obtained, we used diagnoses present on either VA or Medicare claims during the year to identify HCC conditions.

We chose the five‐class model of VA utilization as the optimal solution. While the six‐ and seven‐class models had modestly lower (i.e., better) BIC values, the five‐class model had the best entropy (0.77) and was the most parsimonious model with good fit. Moreover, most patients could easily be assigned to a single class based on posterior probabilities, and class provided meaningful interpretations.

Overall, the five classes representing different patterns of VA service use were distinguished by level of medical services, number of prescription medication classes filled, and mental health service use (Table  1 ). The LCA classes were given the following descriptive labels based on observed VA utilization: (1) Low VA medical use with minimal VA medication and mental health use (representing an expected 43 percent of patients based on posterior probabilities from the LCA model; 45.4 percent based on assigning patients to the most probable class); (2) Low VA medical users with significant VA medication and mental health use (12.7 percent mean posterior probability); (3) Moderate VA medical use with minimal VA medication and mental health use (23.1 percent); (4) Moderate VA medical use with significant VA medication and mental health use (10.5 percent); and (5) High use of all VA services including inpatient hospitalizations (10.8 percent). The five‐class LCA solution provided good agreement between latent class membership based on the posterior probabilities of class membership, and based on assignment of patients to most likely class (Table  1 ). For example, the LCA model estimated that 10.8 percent of patients are high users of all services; 10.5 percent of patients were assigned to the “High VA Medical User” class based on their conditional response probability. The largest group of patients, “Low VA Medical Users with Minimal VA Medication and Mental Health Use,” represents 43.0 percent of the entire patient sample based on prior probabilities and 45.4 percent based on assignment of patients to the most likely class.

Latent Class Model Results: Agreement of Class Assignment Based on Estimated Posterior Probabilities and Based on Assignment of Patients to the Most Likely Class

Bolded numbers represent the probability that the LCA model assigns patients to the correct class (e.g., 89% of “Low users with minimal medication and mental health services” are assigned to the correct class based on the LCA model).

Table  2 shows VA services used by patients in each of the five classes. Use of VA primary care, specialty care, inpatient acute care, and diagnostic radiology services was very small for the two classes labeled “Low Medical Users.” However, Low Medical Users were further distinguished by the level of mental health and pharmacy services. Twenty percent (20 percent) of Low VA Medical Users with Significant VA Medication and Mental Health Use had 3 or more VA mental health visits, 96 percent received a psychiatric medication through the VA, and 28 percent received 10 or more categories of medications through the VA. In contrast, Low VA Medical Users with Minimal VA Medication and Mental Health Use received very few of these services. Overall, the two groups of “Moderate VA Medical Users” incurred more VA medical services compared to Low VA Medical Users , including more VA primary care visits, specialty physician visits, rehabilitative services, and diagnostic radiology compared to either group of lower VA users. However, they are also distinguished by the use of the VA for mental health and pharmacy services. Finally, 65 percent of patients in the High VA User group had 4 or more VA primary care visits during the year, and 71 percent had 2 or more specialty physician visits. High VA users also had significant use of VA diagnostic radiology, rehabilitative therapy, and prescription drugs.

Use of Services by Type of Service for Five Categories of Patients Defined by Latent Class Analysis

In additional sensitivity analyses, we evaluated the robustness of the final LCA model to alternative categorizations of VA health services. For example, we collapsed the number of primary care visits and unique drug classes into three categories rather than the four categories used in our primary model (i.e., 0–1, 2–3, and 4 or more primary care visits; and 0–4, 5–9, and 10 or more drug classes). The resulting model had slightly lower entropy than our primary model (entropy = 0.75 vs. 0.77) but resulted in identical interpretations of the resulting categories, with generally similar proportions of patients categorized as Low VA Users with Minimal Medications and Mental Health, Low VA Users with Significant Medications and Mental Health, Moderate Users with Minimal Medications and Mental Health, Moderate Users with Significant Medications and Mental Health, and High Users all services (49, 15, 20, 8, and 9 percent of patients, respectively). Further collapsing primary care and drug classes into two categories or expanding to five categories also resulted in categories with identical interpretation, but slightly modified proportions of patients in each category and slightly lower entropy.

High VA users were more likely to live in urban centers and live closer to VA medical centers, on average, than patients who used fewer services. They were also less likely to be non‐Hispanic White compared to other groups and more likely to be enrolled in Medicaid (Table  3 ). Low VA Medical Users lived, on average, farther from the nearest VA medical center. The mean number of medical conditions identified by Medicare HCC model, and number of conditions representing chronic or special needs were similar for Low VA users with minimal VA medications and mental health (column a) and Moderate VA users with minimal VA medications and mental health (column c), suggesting roughly similar illness burden for the two groups despite the difference in the use of VA care. Low VA users with significant VA medications and mental health (column b) have a slightly greater number of HCC conditions—possibly reflecting the increased burden associated with mental illness. Not surprisingly , High VA users had the most medical conditions, including conditions representing chronic or special needs.

Demographic Characteristics of Patients in Five Service Use Categories Defined by Latent Class Analysis a

All patient characteristics are significantly related to LCA class membership using statistical criterion, p  < .01.

Interesting patterns of complementary dual use between VA and Medicare services are revealed corresponding to the first four patient LCA classes of VA Utilization (Table  4 ). The mean of total number of primary care visits (i.e., VA and non‐VA combined) is generally similar across the four LCA classes of low and moderate VA medical users, with the mean total primary care visits during FY2013 ranging from 3.51 to 3.73 across the four classes. While low and moderate VA users experienced roughly similar total numbers of primary and specialty service visits, moderate VA users have a greater reliance on the VA for these visits. For example, among L ow VA users with minimal VA medications and mental health services , 32 percent of total primary care visits occurred through the VA (i.e., on average, 1.11 of 3.51 total primary care visits occur in the VA)—compared to 73 percent among moderate VA users. Patients in the high VA user class rely substantially on the VA for all services, with the exception of emergency room visits. For all classes, patients received the majority of mental health services through the VA.

VA and Private Sector Service Use by VA Patients Age 65 and Older Enrolled in Medicare

This study provides a comprehensive typology of VA health system use by applying latent class analysis to administrative utilization data from the VA health system. We successfully identified five unique classes of VA patients distinguished by low, moderate, and high VA service use, with further distinctions based on their use of VA mental health and pharmacy services. These parameters suggest that the use of VA care by veterans is determined by medical acuity as well as the need for mental health or pharmacy services. Specifically, we classified more than half of all elderly VA patients as low users of VA medical and specialty services, with the majority of Low users also using few VA mental health and pharmacy services. Moderate users of VA medical services comprised about one‐third of all elderly VA patients, and nearly two‐thirds of moderate users also used relatively few VA mental health and pharmacy services. Approximately 10 percent of elderly VA patients were categorized as high users, meaning that they incurred substantial use of all VA medical, mental health, and pharmacy services, as well as acute inpatient and emergency care.

We also demonstrated that the reliance on VA services relative to Medicare services differed substantially within each of the five patient classes. Surprisingly, Low VA users and Moderate VA users had relatively similar illness burden and total health service utilization, but differed substantially in the degree to which they relied on the VA for care. Like previous studies (Weeks, Mahar, and Wright 2005 ; Liu et al. 2009 ), our study found that veterans who needed mental health service typically seek those services within the VA. Reliance on the VA was highest for patients with the greatest disease burden.

Multiple prior studies have investigated dual use, particularly among the elderly with Medicare coverage. Liu et al. ( 2010 b) found that over 30 percent of Medicare‐eligible VA primary care patients also received primary care through Medicare, and over 60 percent received specialty care through Medicare. Moreover, the proportion of VA patients who rely on the VA for care has decreased over time, especially for specialty care (Liu et al. 2011b ).

A few prior studies have used LCA to classify individuals on the basis of service use, although few that focused specifically on VA patients. Hastings et al. ( 2010 ) identified five categories of ED patients based on the use of primary care, specialty care, ED visits, and hospitalization during the 12 months prior to the index ED encounter. Other studies used LCA models to investigate services used by caregivers (Beeber, Thorpe, and Clipp 2008 ; Hong 2010 ) and to describe the demand for health services among elderly patients (Deb and Triveki 2002 ). Prior applications of LCA to veteran patients have focused on risk profiling (Funderburk et al. 2008 ; Cho et al. 2017 ), psychiatric comorbidity patterns (Cadigan, Klanecky, and Martens 2017 ; Richardson et al. 2017 ), and service utilization among veterans with diabetes (Radomski et al. 2016 ).

Overall, these findings have multiple important implications for the management of elderly VA patients with dual use of community services. Importantly, this study demonstrates that VA patients are largely delineated by use of medical services, pharmacy, and mental health services, suggesting that activities of clinical primary care teams can be targeted to specific patient subsets. For example, mental health services are a common reason to seek care from the VA, even among patients with few medical needs or patients whose medical needs are addressed through providers outside the VA. Thus, one could argue that facilitating the provision of mental health services within the VA for these patients is important, yet coordinating other care that is largely delivered outside the VA may be more challenging and possibly even duplicate coordination efforts of primary care teams outside the VA. Similarly, many patients opt to receive medications through the VA yet may not receive the majority of their care through the VA. Having access to pharmacy management and medication reconciliation strategies may be important for managing these patients, particularly for patients whose VA medical records are incomplete due to not including documentation of services obtained outside the VA. In addition, assessing VA's performance on patient outcomes is incomplete and potentially misleading if care received outside the VA is not also considered. For example, within the VA, adoption of medical home features has been shown to be associated with lower rates of avoidable hospitalizations (Yoon et al. 2013 ). However, these results do not account for hospitalizations occurring outside the VA, which account for the majority of inpatient admissions among VA patients. Thus, changes in the use of non‐VA hospitals for inpatient care may distort perceived benefits of PACT within the VA and obscure overall costs to the federal government.

Second, the finding that high users of VA services rely on the VA for the vast majority of their care highlights the important role of the VA as a safety net for veterans, particularly veterans who are most burdened by illness and require substantial health services. Coordination of care for these patients occurs primarily within the VA system, but is nevertheless complex due to the diverse services and medications used by these patients. In additional analysis, we determined that, although high VA users represent only 10 percent of all VA patients, they account for 90 percent of all inpatient admissions to VA hospitals and nearly half of all specialty care visits. Thus, careful management of this relative small portion of VA patients may have a tremendous impact on overall VA health services utilization.

Finally, this study also identifies categories of patients for whom information in VA medical records may be missing important information about medical services utilization that is received outside the VA. In our study, more than 40 percent of elderly VA patients use relatively few VA services, with the majority of care received outside the VA for those patients. While VA providers may have a rough understanding that a large proportion of elderly VA patients use relatively few services, they may be unaware of the degree to which such patients use services outside the VA. Failure to recognize total illness burden and the use of other health services has important implications for the provision of care and allocating resources to meet the needs of aging veterans. For example, Ried et al. ( 2007 ) identified 25 percent more VA patients with a history of stroke using Medicare and VA records combined, compared to using VA records only. This study provides information regarding the potential that VA data do not capture all services for specific classes of patients, types of services that are most likely missing, and categories of patients for whom VA medical records likely represents all services.

This study has limitations. First, evaluating service use using administrative data sources from different health care providers or payers is challenging. Burgess et al. ( 2011 ) describe in detail the nuances and complexities of evaluating service use in administrative data from different administrative data sources. With Medicare data, diagnosis and procedure codes are often recorded to maximize reimbursement, while in the VA reimbursement is not relevant. Nevertheless, VA providers are increasingly pressured to capture complete data for their patients as such data are used for performance and workload measures. Our study built on the system developed by Burgess et al. ( 2011 ) for classifying individual services in VA data sources, which was demonstrated to have good negative and predictive values for classifying primary and specialty services. Second, our typology of VA service use included patients age 65 and older only and therefore may not be generalizable to younger veterans. Third, the cutoffs we used to define levels of VA service utilization were admittedly somewhat arbitrary. We conducted sensitivity analysis to determine whether the cutoffs used to define levels of primary care, specialty care, and other services impacted results substantially. We found that the interpretation of LCA classes identified was remarkably similar regardless of the specific cutoffs, but the proportion of patients identified as low, moderate, or high users of VA services varied modestly depending on the cutoffs. We did not perceive substantial impact of these analyses on our conclusions. Fourth, we conducted the LCA analysis on VA data only. Future studies will use this analysis for combined VA and Medicare data. Fifth, our analysis did not reflect medications received through Medicare. Nevertheless, Medicare enrollment data indicated that the majority of VA patients (nearly 80 percent) in our cohort were never enrolled in Part D during the observation period, with reasonably similar Part D enrollment across the five classes of VA patients (ranging from 76 to 83 percent). Finally, our study did not include fee‐based VA care, which is care provided in non‐VA facilities that is paid by the VA. Inclusion of outpatient fee‐based care is problematic because VA data files for outpatient fee‐based care generally contain fewer details regarding specific types of services provided. We did, however, conduct additional analysis in which fee‐based inpatient admissions were included in our analysis, and results were very similar.

VA serves an important role in the care of elderly veterans. It is a central provider of mental health care, its pharmacy benefits are highly valued, and veterans with high need for health services and accompanying acute care seek most of their services at VA relative to non‐VA care. Nevertheless, most elderly veterans also receive care outside the VA. While the use of non‐VA care will likely increase with the Veterans Choice Act, combined VA and community care, in one form or another, has been common in the VA for decades. Such care requires effective coordination of care, user‐friendly health information exchange platforms, and protocols for transitions between VA and the community. Since 1998, the VA Information Resource Center and Centers for Medicare and Medicaid Services (CMS) have made linked VA‐Medicare claims and enrollment information available to VA researchers, clinicians, and policymakers to better understand care received outside the VA for veterans dually eligible for VA and Medicare services. This linking of data provides an opportunity to gain clinical insight into the needs of elderly veterans and potential missed opportunities for care coordination. Moreover, by linking data over multiple years, future research may evaluate the impact that policy changes, such as changes in pharmacy benefits or the availability of inpatient and specialty services has on access to care and the use of services outside the VA.

Supporting information

Appendix SA1: Author Matrix.

Acknowledgments

Joint Acknowledgment/Disclosure Statement : Drs. Vaughan Sarrazin and Turvey are Core Investigators in the Comprehensive Access & Delivery Research and Evaluation (CADRE) Center Iowa City VA Health Care System, which is funded through the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service.

This work was supported by the VA VISN 23 Patient Aligned Care Teams (PACT) Demonstration Laboratory, VA Office of Patient Care Services. Support for VA/CMS Data provided by the Department of Veterans Affairs, VA Health Services Research and Development Service, VA Information Resource Center (Project Numbers SDR 02‐237 and 98‐004), Hines, IL. This work was supported using resources and facilities at the VA Informatics and Computing Infrastructure (VINCI), VA HSR RES 13‐457, Salt Lake City, UT.

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Disclosure : None.

Disclaimer : None.

• Beeber, A. S. , Thorpe J. M., and Clipp E. C.. 2008. “Community‐Based Service Use by Elders with Dementia and Their Caregivers: A Latent Class Analysis.” Nursing Research 57 (5): 312–21. [ DOI ] [ PubMed ] [ Google Scholar ]
• Bonito, A. J. , Bann C., Eicheldinger C., and Carpenter L.. 2008. “Creation of New Race‐Ethnicity Codes and Socioeconomic Status (SES) Indicators for Medicare Beneficiaries.” RTI International report for the Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare and Medicaid Services (CMS) under CMS contract No 500‐00‐0024. Available at http://www.ahrq.gov/research/findings/final-reports/medicareindicators/index.html
• Burgess, J. F. , Maciejewski M. L., Bryson C. L., Chapko M., Fortney J., Perkins M., Sharp N. D., and Liu C. F.. 2011. “Importance of Health Sector Context for Evaluation Utilization Patterns across Sectors.” Journal of Health Economics 20 (2): 239–51. [ DOI ] [ PubMed ] [ Google Scholar ]
• Cadigan, J. M. , Klanecky A. K., and Martens M. P.. 2017. “An Examination of Alcohol Risk Profiles and Co‐Occurring Mental Health Symptoms among OEF/OIF Veterans.” Addictive Behaviors 70: 54–60. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cho, J. , Stock E. M., Liao I. C., Zeber J. E., Ahmedani B. K., Basu R., Quinn C. C., and Copeland L. A.. 2017. “Multiple Chronic Condition Profiles and Survival among Oldest‐Old Male Patients with Hip Fracture.” Archives of Gerontology and Geriatrics 74: 184–90. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins, L. M. , and Lanza S. T.. 2010. Latent Class and Latent Transition Analysis: With Applications in the Social, Behavioral, and Health Sciences. Hoboken, NJ: John Wiley & Sons Inc. [ Google Scholar ]
• Collins, L. M. , Fidler P. L., Wugalter S. E., and Long J. L.. 1993. “Goodness‐of‐Fit Testing for Latent Class Models.” Multivariate Behavioral Research 28: 375–89. [ DOI ] [ PubMed ] [ Google Scholar ]
• Deb, P. , and Triveki P. K.. 2002. “The Structure of Demand for Health Care: Latent Class Versus Two‐Part Models.” Journal of Health Economics 21: 601–25. [ DOI ] [ PubMed ] [ Google Scholar ]
• Dziak, J. J. , Coffman D. L., Lanza S. T., and Li R.. 2012. “Sensitivity and Specificity of Information Criteria: Technical Report Series #12‐119.” Methodology Center and College of Health and Human Development, Penn State, State College, PA [accessed on January 2, 2018]. Available at https://methodology.psu.edu/media/techreports/12-119.pdf
• Funderburk, J. S. , Maisto S. A., Sugarman D. E., and Wade M.. 2008. “The Covariation of Multiple Risk Factors in Primary Care: A Latent Class Analysis.” Journal of Behavioral Medicine 31: 525–35. [ DOI ] [ PubMed ] [ Google Scholar ]
• Hastings, S. N. , Horney C., Landerman L. R., Sanders L. L., Hocker M. B., and Schmader K. E.. 2010. “Exploring Patterns of Health Service Use in Older Emergency Department Patients.” Academic Emergency Medicine 17 (10): 1086–92. [ DOI ] [ PubMed ] [ Google Scholar ]
• Hong, S. I. 2010. “Understanding Patterns of Service Utilization among Informal Caregivers of Community Older Adults.” Gerontologist 50 (1): 87–99. [ DOI ] [ PubMed ] [ Google Scholar ]
• Humensky, J. , Caretta H., deGroot K., Brown M. M., Tarlov E., and Hynes D. M.. 2012. “Service Utilization of Veterans Dually Eligible for VA and Medicare Fee‐for‐Service: 1999–2004.” Medicare & Medicaid Research Review 2 (3): E1–E22. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hynes, D. M. , Koelling K., Stroupe K., Arnold N., Mallin K., Sohn M. W., Weaver F. M., Manheim L., and Kok L.. 2007. “Veterans' Access to and Use of Medicare and Veterans Affairs Health Care.” Medical Care 45: 214–23. [ DOI ] [ PubMed ] [ Google Scholar ]
• Jia, H. , Zheng Y., Reker D. M., Cowper D. C., Wu S. S., Vogel W. B., Young G. C., and Duncan P. W.. 2007. “Multiple System Utilization and Mortality for Veterans with Stroke.” Stroke: A Journal of Cerebral Circulation 38: 355–60. [ DOI ] [ PubMed ] [ Google Scholar ]
• Lanza, S. T. , Collins L. M., Lemmon D. R., and Schafer J. L.. 2007. “PROC LCA: A SAS Procedure for Latent Class Analysis.” Structural Equation Modeling 14 (4): 671–94. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Liu, C. F. , Bolkan C., Chan D., Yano E. M., Rubenstein L. V., and Chaney E. F.. 2009. “Dual Use of VA and Non‐VA Services among Primary Care Patients with Depression.” Journal of General Internal Medicine 24 (3): 305–11. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Liu, C. F. , Chapko M., Bryson C. L., Burgess J. E., Fortnery J. C., Perkins M., Sharp S. M., and Maciejewski M. L.. 2010. “Use of Outpatient Care in Veterans Health Administration and Medicare among Veterans Receiving Primary Care in Community Based and Hospital Outpatient Clinics.” Health Services Research 45 (5): 2368–85. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Liu, J. H. , Manning W. G., Burgess J. F., Hebert P. L., Bryson C. L., Fortney J., Perkins M., Sharp N. D., and Maciejewski M. L.. 2011a. “Reliance on Veterans Affairs Outpatient Care by Medicare‐Eligile Veterans.” Medical Care 49: 911–7. [ DOI ] [ PubMed ] [ Google Scholar ]
• Liu, J. H. , Manning W. G., Burgess J. F., Hebert P. L., Bryson C. L., Fortney J., Perkins M., Sharp N. D., and Maciejewski M. L.. 2011b. “Reliance on Veterans Affairs Outpatient Care by Medicare‐Eligile Veterans.” Medical Care 49 (10): 911–7. [ DOI ] [ PubMed ] [ Google Scholar ]
• Phibbs, C. S. , Barnett P. B., and Fan A.. 2015. Research Guide to the Managerial Cost Accounting National Cost Extracts. Guidebook. Menlo Park, CA: VA Palo Alto, Health Economics Resource Center. [ Google Scholar ]
• Radomski, T. R. , Zhao X., Thorpe C. T., Thorpe J. M., Good C. B., Mor M. K., Fine M. J., and Gellad W. F.. 2016. “VA and Medicare Utilization among Dually Enrolled Veterans with Type 2 Diabetes: A Latent Class Analysis.” Journal of General Internal Medicine 31 (5): 524–31. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Research Triangle Institute . 2008. “RTI Analysis of Special Needs Plan Chronic Condition Panel: Final Report.” North Carolina: Research Triangle Park.
• Richardson, J. D. , Ketcheson F., King L., Schnaider P., Marlborough M., Thompson A., and Elhai J. D.. 2017. “Psychiatric Comorbidity Pattern in Treatment‐Seeking Veterans.” Psychiatry Research 258: 488–93. [ DOI ] [ PubMed ] [ Google Scholar ]
• Ried, L. D. , Cameon R., Jia H., Findley K., Hinojosa M. S., Wang X., and Tueth M. J.. 2007. “Identifying Veterans with Acute Strokes with High‐Specificity ICD‐9 Algorithm with VA Automated Records and Medicare Claims Data: A More Complete Picture.” Journal of Rehabilitation Research and Development 44 (5): 665–73. [ DOI ] [ PubMed ] [ Google Scholar ]
• Shen, Y. , Findley P. A., Maney M., Pogach L., Crystal S., Rajan M., and Findley T. W.. 2008. “Department of Veterans Affairs‐Medicare Dual Beneficiaries with Stroke: Where Do They Get Care?” Journal of Rehabilitation Research and Development 45 (1): 43–51. [ DOI ] [ PubMed ] [ Google Scholar ]
• Trivedi, A. N. , Grebla R. C., Jiang L., Yoon J., Mor V., and Kizer K. W.. 2012. “Duplicate Federal Payments for Dual Enrollees in Medicare Advantage Plans and the Veterans Affairs Health Care System.” Journal of the American Medical Association 308 (1): 67–72. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• U.S. Department of Agriculture . 2015. Rural‐Urban Commuting Area Codes. Washington DC: Economic Research Service; [accessed on August 20, 2017]. Available at http://www.ers.usda.gov/data-products/rural-urban-commuting-area-codes.aspx [ Google Scholar ]
• U.S. Department of Veterans Affairs . 2011. 2011 Survey of Veteran Enrollees' Health and Reliance upon VA. Washington, DC: Veterans Health Administration. [ Google Scholar ]
• U.S. Department of Veterans Affairs . 2015. “System of Records Notice 97VA10P1: Consolidated Data Information System‐VA.” 76 FR 25409. May 4, 2011.
• U.S. Department of Veterans Affairs . 2016. “VHA Directive 1153: Access to Centers for Medicare and Medicaid Services (CMS) and United States Renal Data System (USRDS) Data for Veterans Health Administration (VHA) Users within the Department of Veterans Affairs (VA) Information Technology (IT) Systems.”
• VIReC . 2015a. VIReC Research User Guide: Fiscal Year 2014 VHA Medical SAS Inpatient Datasets, 2d Edition Hines, IL: U.S. Department of Veterans Affairs Health Services Research & Development Service, VA Information Resource Center. [ Google Scholar ]
• VIReC . 2015b. VIReC Research User Guide: Fiscal Year 2014 VHA Medical SAS Outpatient Datasets and Inpatient Encounters Dataset. Hines, IL: U.S. Department of Veterans Affairs Health Services Research and Development Service, VA Information Resource Center. [ Google Scholar ]
• Weeks, W. B. , Mahar P. J., and Wright S. M.. 2005. “Utilization of VA and Medicare Services by Medicare‐Eligible Veterans: The Impact of Additional Access Points in a Rural Setting.” Journal of Healthcare Management and American College of Healthcare Executives 50 (2): 95–106. [ PubMed ] [ Google Scholar ]
• Wolinsky, F. D. , An H., Liu L., Miller T. R., and Rosenthal G. E.. 2006. “Dual Use of Medicare and the Veterans Health Administration: Are There Adverse Health Oucomes?” BMC Health Services Research 6: 131. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wolinsky, F. D. An H., Liu L., Miller T. R., and Rosenthal G. E.. 2007. “Exploring the Association of Dual Use of the VHA and Medicare with Mortality: Separating the Contributions of Inpatient and Outpatient Services.” BMC Health Services Research 7: 70. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Yoon, J. , Rose D. E., Canelo I., Upadhyay A. S., Schectman G., Stark R., Rubenstein L., and Yano E. M.. 2013. “Medical Home Features of VHA Primary Care Clinics and Avoidable Hospitalizations.” Journal of General Internal Medicine 28 (9): 1188–94. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

• View on publisher site
• PDF (214.5 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

“What’s the evidence?”—Towards more empirical evaluations of the impact of OR interventions in healthcare

Guillaume lamé, sonya crowe, matthew barclay.

• Author information
• Article notes
• Copyright and License information

CONTACT Guillaume Lamé [email protected] Université Paris-Saclay, CentraleSupélec, Laboratoire Génie Industriel, 3 rue Joliot-Curie, F-91192 Gif-sur-Yvette Cedex, France.

Received 2019 Oct 8; Accepted 2020 Nov 24; Collection date 2022.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Despite an increasing number of papers reporting applications of operational research (OR) to problems in healthcare, there remains little empirical evidence of OR improving healthcare delivery in practice. Without such evidence it is harder both to justify the usefulness of OR to a healthcare audience and to learn and continuously improve our approaches. To progress, we need to build the evidence-base on whether and how OR improves healthcare delivery through careful empirical evaluation. This position paper reviews evaluation standards in healthcare improvement research and dispels some common myths about evaluation. It highlights the current lack of robust evaluation of healthcare OR and makes the case for addressing this. It then proposes possible ways for building better empirical evaluations of OR interventions in healthcare.

KEYWORDS: Health Quality and Evaluation, healthcare Improvement Research, impact

1. Introduction

Many operational research (OR) practitioners and academics involved in healthcare will have been confronted with the comment “I like the idea of OR, but what’s the evidence for it?” when talking to clinicians, funders, policy-makers or editors and reviewers in medical journals. With this question, they are asking for evidence of OR improving healthcare organisations and often expect this evidence to have a certain form and be generated through specific evaluation processes.

Research applying OR techniques (including Soft OR and Problem Structuring Methods) to improve healthcare delivery is developing rapidly (e.g., Brailsford et al., 2009 ; Zhang et al., 2018 ), but falls short in demonstrating impact through sound empirical evaluations. Most healthcare OR papers do not discuss implementation (Brailsford et al., 2009 ; Brailsford & Vissers, 2011 ), let alone assess the impact of the OR intervention. Therefore, we have little to support the potential of OR in bringing about desirable change to the quality, safety, and efficiency of healthcare delivery, and lack key information to learn and iteratively refine our approaches.

Evaluation helps decision-makers understand what works in a given context, as well as how and why it works, before choosing a course of action (The Health Foundation, 2015 ). In this viewpoint, we argue that we need more, and better, evaluations of the impact of OR interventions in healthcare. We take a broad view of OR, including but not limited to Problem Structuring Methods, mathematical modelling and simulation (see (Pitt et al., 2016 ) for examples). First, we discuss the type of evidence currently generated to support claims of OR’s potential impact. We then highlight the gap between the objectives of evidence-based healthcare improvement and the reality of current practice in healthcare OR. Finally, we identify possible ways to address this gap.

2. Evaluating healthcare improvement interventions

Evaluations aim “to determine merit, worth, value or significance” (Patton et al., 2014 ). In the context of interventions for improving healthcare, evaluation means answering the questions (Walshe, 2009 ):

Does the intervention work, and how well? How much does it cost? The efficiency, effectiveness and efficacy of the intervention needs to be assessed to know if the resources invested are well spent. This is referred to as outcome evaluation .

Why and how does the intervention generate these outcomes in this context? Underlying mechanisms linking an intervention to a given set of outcomes need to be analysed to understand in which other circumstances the intervention (or parts of it) may be useful. This is labelled process evaluation .

What is it like to use the intervention? Learning from practical implementation by exploring the experience of those implementing and participating in the intervention, to improve the implementability and feasibility of the intervention.

The emerging consensus among healthcare improvement evaluators is to use a “programme theory” to plan the evaluation and synthesise its results. Programme theories synthesise different types of knowledge to explain how the components of an intervention will generate a certain effect through intermediate processes and subject to moderating factors (Davidoff et al., 2015 ; Funnell & Rogers, 2011 ).

The key advantage of a theory-driven approach is that it goes beyond asking whether the intervention works, to exploring where it works, for whom, and why, through unpacking the mechanisms that generate these impacts. This makes it easier to understand to what extent success is transferrable, not least by assessing what aspects of the intervention are particularly contingent on context (Davidoff et al., 2015 ). Programme theories also help explain unsuccessful interventions (Funnell & Rogers, 2011 ).

Programme theories can be used in quantitative, qualitative or mixed-methods evaluations, and in experimental, quasi-experimental or observational designs. In general, a degree of pragmatism is considered necessary in choosing an appropriate evaluation design due to the complexity of some interventions, their dynamic and evolving character, and the cost of running multisite experimental studies (Barry et al., 2018 ). Nonetheless, this flexibility does not mean that all evaluation designs and methods are equivalent or provide the same level of confidence in the impact of the intervention. For example, simple uncontrolled before-and-after studies are a very weak form of evaluation since any changes observed may be caused by secular trends, changes in the environment or the phenomenon of “regression to the mean”, rather than the intervention (Eccles et al., 2003 ). Interrupted time-series designs, where the outcomes of interest are measured at several points before, during, and after the intervention, are a stronger design because they allow the effect of the intervention to be distinguished from secular trends, while remaining simpler to organise than controlled experiments (Fretheim & Tomic, 2015 ). Although the literature tends to discuss quantitative designs more extensively, qualitative studies are also part of an evaluator’s toolkit in healthcare improvement (Portela et al., 2015 ).

In the Appendix to this article, we dispel some common myths about the evaluation of healthcare improvement interventions along with illustrative examples.

3. The current state of evaluation in healthcare OR

Much of the healthcare improvement research literature focuses on publishing evaluations but the healthcare OR literature has largely evolved separately. Evaluations of OR interventions are rarely reported. OR case studies typically describe the process of building models and improving their performance with few papers mentioning implementation. Literature reviews have highlighted this in healthcare OR in general (Brailsford & Vissers, 2011 ; Mahdavi et al., 2013 ; Van Sambeek et al., 2010 ), in modelling and simulation studies (Brailsford et al., 2009 ; Fone et al., 2003 ; Jahangirian et al., 2012 ; Long & Meadows, 2018 ; Mohiuddin et al., 2017 ; Van Lent et al., 2012 ; Wilson, 1981 ), multi-criteria decision analysis (Marsh et al., 2014 ), the application of Soft Systems Methodology (Augustsson et al., 2019 ), optimisation (Ahmadi-Javid et al., 2017 ), scheduling (Marynissen & Demeulemeester, 2019 ; Samudra et al., 2016 ) or in specific areas of healthcare, such as outpatient chemotherapy (Lamé et al., 2016 ), global health (Bradley et al., 2017 ) or community healthcare (Palmer et al., 2018 ). The results of Brailsford et al. ( 2009 ) that only 5 to 8% of modelling and simulation papers in healthcare mention the implementation of results in practice seem to hold for healthcare OR more broadly.

OR researchers and practitioners sometimes argue that the learning generated through an OR project is more important than the “answers”, with stakeholders gaining understanding through the process about how their organisations work and what affects their performance (Sterman, 1994 ). However, we still need to assess who learns what, and how (Lamé & Simmons, 2020 ). Some researchers have started to do so through lab experiments (Monks et al., 2014 ), or retrospective interviews with experts (Thompson et al., 2016 ). Empirical evaluations in real improvement projects have also been published in a corporate context (Cavaleri & Sterman, 1997 ; Read et al., 2012 ), but remain rare. In many cases, there is no measurement of changes to participants’ behaviour that could be linked to modelling projects, nor of participants’ reactions to and opinions of the modelling effort (Kunc et al., 2018 ). There also does not appear to be a consensus on what constitutes learning in OR interventions or on how to assess it.

4. The gap between standards of evaluation in healthcare improvement and current practice in healthcare OR

The lack of robust evaluation of the impact of OR interventions is particularly problematic in healthcare, where the evidence-based paradigm is spreading from clinical practice to management and policy, leading to increasing pressure for better evidence on what works to improve healthcare delivery (Auerbach et al., 2007 ; Bevan et al., 2005 ; Grady et al., 2018 ; The Health Foundation, 2015 ). In Section 2 , we described the standards for evaluation of healthcare improvement interventions. The gap between these standards and current practice in healthcare OR raises pragmatic and scientific arguments for empirical evaluations of the impact of OR interventions in healthcare organisations:

Research on OR interventions in healthcare must produce evidence that is acceptable to those who will use the results, providing empirical evidence to meet the needs and expectations of healthcare improvement funders, practitioners and policy-makers.

Expectations for sound evidence that OR brings about improvement are healthy and should drive us to better evaluate interventions not only as an end in itself, but as a means to enhance the effectiveness of the work we do in bringing about positive change in healthcare.

Empirical data on observed changes (or absence of such changes) linked to OR interventions is needed so that we can understand their impact. Recent examples show this is not out-of-reach. For instance, Monks et al. ( 2015 ) present a case study of the impact of a simulation study in emergency stroke care combining different types of quantitative evidence (before-and-after analysis on different process duration metrics, time series analysis on the implementation of certain good practices) with an analysis of the modeller’s field notes during the intervention. Outcomes of interest improved after the intervention and the time-series analysis gives confidence that this was attributable to the intervention, whilst the qualitative evidence helped identify aspects of the intervention that supported stakeholder engagement and the credibility of the results. In another example, Crowe et al. ( 2017 ) use ethnography to provide insights into the role and specific contribution of OR in multidisciplinary projects in healthcare. Other studies have also started to look at the cost-effectiveness of building a model to tackle issues in healthcare (Soorapanth & Young, 2019 ; Young et al., 2018 ). Impact evaluations of Soft OR approaches are also appearing (Emes et al., 2018 ).

Another salient aspect of current healthcare OR publications is that the evaluator is often embedded in the intervention team (Barry et al., 2018 ). Many papers are self-reports of interventions carried out by the authors, with very few external evaluations of OR projects. The stroke project mentioned above (Heaton et al., 2016 ) is an exception, albeit aimed at evaluating the funding initiative that supported the project rather than the OR intervention. In another case, the design of the UK NHS Direct national 24-hour telephone helpline service, an external evaluation was reported independently from the OR intervention itself (Munro et al., 2000 ). However, the external evaluation focused on the solution proposed by the OR project (the telephone helpline) and does not mention the OR intervention. Therefore, it is not possible to understand from this evaluation how the OR intervention affected decisions that led to the implementation of the chosen solution. Elements on the role played by operational researchers in this project have been reported by the OR team (Royston et al., 2003 ), but fall into the category of self-reports rather than external evaluations. Existing systematic reviews of the literature, detailed in Section 3 , show most evaluations of healthcare OR are reported by the operational researchers who led the intervention, and focus on the modelling stages rather than the implementation of the findings. Current OR evaluations published in the academic literature primarily cover only one of the four quadrants of possible evaluation modalities ( Figure 1 ):

We rarely look at the outcomes of our interventions,

We rarely design external evaluations.

Positioning current evaluations of OR approaches in a landscape of evaluation approaches.

There is nothing wrong with internal process evaluations of OR interventions, which can address many interesting questions. The problem comes when we avoid other types of evaluation. By not measuring what happens after our interventions, or relying on self-reported data and internal evaluations, we risk introducing biases that make interventions appear useful when they might do little to solve issues in practice. In reality, many interventions do not work as well as planned or equally well in every context (Øvretveit, 2011 ), yet few unsuccessful studies are reported in OR journals (for examples of OR interventions in healthcare reported as partially unsuccessful by their authors, see Bennett & Worthington, 1998 ; Connell, 2001 ).This suggests either some form of publication bias (where only positive studies are reported), or outcome reporting bias (where reporting is biased towards the more positive aspects of interventions, overlooking less successful dimensions) (Fanelli, 2012 ).

External evaluation and outcome evaluations are not panaceas. They are not always needed by healthcare stakeholders, nor are they always appropriate. Yet including more of them in our research portfolio could help us to understand how OR interventions work, specify which work best in different settings, and anticipate their likely impact. Systematic evaluation and reporting would also allow us to critique and improve the interventions we develop and our practice as operational researchers. Ultimately, these types of evaluation would provide stronger arguments for using OR methods to bring about improvement in healthcare organisations, and enable continuous learning and improvement.

5. Towards empirical evaluations of OR in healthcare

5.1. what do we need to evaluate.

How should we explore the other quadrants of Figure 1 and what exactly do we need to evaluate in OR interventions? When using OR methods to structure, model and better understand a problem situation in order to take informed action, the intervention and potential outcomes are harder to define upfront than for, say, an annual training module on patient safety. For instance, an OR intervention may use simulation to understand patient flows in an emergency department and evaluate the impact of different ways of organising resources and processes. The proposed changes from this OR intervention (e.g., shifting resources across different parts of the pathway) will then be debated and either implemented or not, and if implemented may or may not bring improvement.

Determining whether improvement has occurred is not always straightforward and notions of improvement may be contested by different stakeholders. Therefore specifying appropriate evaluation outcomes for OR interventions can be challenging, particularly when the situations being tackled are multi-faceted and messy (Williams, 2008 ). However, some OR interventions in healthcare aim to address relatively bounded issues, such as patient flow (Mohiuddin et al., 2017 ; Palmer et al., 2018 ), for which indicators can be defined and used to measure impact. In more complex situations, qualitative methods may be better suited to assessing whether and in which ways things improve or worsen.

A simple model of the possible outcomes from an OR intervention can be helpful when planning evaluations. For example, Figure 2 shows a generic model appropriate for relatively simple, linear, OR interventions. It raises generic questions for each possible outcome:

Generic model of the outcomes of OR interventions (similarities can be noted with the four stages of success in simulation projects suggested by Robinson and Pidd, 1998).

1. The changes recommended by the OR intervention are implemented, and the situation improves. Can the improvement be attributed to the implementation of the recommended changes? Can the choice to implement these changes be attributed to the OR intervention (or would it have been chosen anyway)? Did the recommended changes generate side effects?

2. The changes recommended by the OR intervention are implemented, but the situation does not improve. Was the decision to implement these changes attributable to the OR intervention? Were the recommended changes implemented correctly? Was the OR model appropriate (complexity, type of model, hypotheses, factors included and excluded)? Did the recommended changes generate side effects?

3. The changes recommended by the OR intervention are not implemented, and the situation improves. Why were the recommended changes not chosen? Might the recommended changes have led to greater improvement?

4. The changes recommended by the OR intervention are not implemented, and the situation does not improve. Why were the recommended changes not chosen?

Addressing these questions requires a range of evaluation methods. For example, matters of attribution (“did the improvement come from the intervention?”) can be approached using experimental or quasi-experimental quantitative methods (e.g., using Statistical Process Contro over the course of the modelling process (Perla et al., 2011 )), while qualitative methods (e.g., interviews and observations) are well suited to identifying why people did or did not accept the OR recommendations. Side effects can be investigated both qualitatively and quantitatively, either by assessing against pre-identified potential side effects or in a more exploratory manner.

Existing frameworks for evaluating “complex interventions” in healthcare (like that proposed by the UK Medical Research Council, c.f. Campbell et al., 2007 ) may provide a useful basis on which to develop evaluation frameworks for OR interventions. Useful insights might also be drawn from the growing field of Behavioural OR, which seeks to examine the role and impact of behaviour on the use of OR to support decision-making (Franco & Hämäläinen, 2016 ), including through qualitative approaches and experimental designs that could help us to understand stakeholders’ acceptance of, and fidelity to, recommendations from OR interventions.

5.2. Programme theories for OR interventions

A programme theory offers a theoretical model of how an intervention is expected to generate certain outcomes in a given context. Programme theories can help in designing appropriate evaluations. Figure 2 stems from a simple generic programme theory for OR, with three elements:

Modelling supports the establishment and sharing of a common, simplified, representation of a complex situation.

Manipulation of this simplified representation allows assessment of the likely effect of changes, and an exploration of the importance of different factors in the overall behaviour of the system.

This experimentation allows people to learn about the behaviour and dynamics of a system and gives a shared foundation for debate, leading to better decisions about how to make changes in their organisation.

This simple programme theory will not always be appropriate. Many OR projects are iterative or involve continuous negotiation (Williams, 2008 ), so the programme theory would need to reflect this. There may need to be additional steps addressing the construction and presentation of the model: for example, some would argue that building models collaboratively, in a facilitated environment, improves their acceptance by stakeholders and that effective visualisation of model outputs can affect how users perceive the model. Fine-tuning an evaluation requires all these elements be considered and integrated into a programme theory that reflects the specifics of the OR intervention and the context.

We will often be able to develop programme theories based on our understanding of OR methods and the context in which they will be applied, but theoretical approaches from other disciplines also offer a rich repertoire for building programme theories for OR interventions. Activity theory (Leroy White et al., 2016 ), the concept of boundary objects (Franco, 2013 ), the mangle of practice framework (Ormerod, 2014 ) and single and double-loop learning (Monks et al., 2014 ) have all been used for this purpose, albeit not in healthcare OR. Drawing on theory from the social sciences forms a key aspect of the Behavioural OR research agenda (Becker, 2016 ; Brocklesby, 2016 ).

5.3. Practical challenges to evaluating OR

Clarifying the scientific challenge and providing elements of a programme theory are merely first steps towards more empirical evaluations of OR interventions in healthcare, and challenges in practical attempts to evaluate healthcare OR may well occur.

For example, evaluation of OR may not fall within the remit of many traditional funding sources for OR projects. However, evaluations have started to appear in the UK, funded by organisations such as the National Institute for Health Research’s Applied Research Collaborations (NIHR ARCs, previously CLAHRCs) and the Health Foundation (e.g., Crowe et al., 2017 ; Monks et al., 2015 ).

Most OR researchers are not trained in the standards, methods and practices of evaluating healthcare improvement interventions. In this case, collaborating with evaluation-driven disciplines such as health services research, implementation science or improvement research can help (Brailsford & Klein, 2015 ), and there may be scope to incorporate evaluation techniques within OR degrees and professional development. This resonates with one stream of the Behavioural OR movement, seeking to use theories and methods from the social and behavioural sciences to better understand what happens in and around OR interventions (Becker, 2016 ; Brocklesby, 2016 ).

Even so, methodological challenges arise when evaluating complex, evolving interventions (Burke & Shojania, 2018 ). Specific frameworks may need to be developed for OR interventions, depending on the type of problem, the modelling methods used and the scope of the project.

Finally, robust evaluation of OR methods and interventions may need multisite projects to compare the effect of an intervention in different places and contexts, requiring different research project management skills and strategies from the one-off projects that constitute the majority of applied OR papers at present.

Although there is no “one best way” that would apply to every OR method and every situation in healthcare, some practical recommendations can be identified ( Box 1 ).

Recommendations

6. Conclusion

Promoting the value of healthcare OR is challenging, despite steady developments since the 1950’s and areas of notable success (for example, embedded OR in the UK NHS, c.f. Royston et al., 1999 , 2003 ), because we rarely evaluate the impact of our projects. A key limitation in this analysis is our reliance on the peer-reviewed academic literature. Indeed, there could be OR practitioners generating very good evidence of the impact their OR interventions have on the organisations they are working with, enabling them to illustrate the value of the techniques within their organisation or to their next potential client. However, we are not aware of such evidence in the grey literature, and, if kept confidential, such evidence is of little use to the wider OR community.

Failing to engage in evaluation limits our impact on practices and performance in healthcare, not least because we are less able to have meaningful conversations with healthcare professionals who increasingly seek an evidence-base for change. Systematic approaches to assess the impact of our efforts are required, and there are already a range of approaches for evaluating healthcare improvement interventions that could be adapted for use with OR interventions. Evidence is not everything, and having it will not suddenly change how OR is perceived and used in healthcare, but not having it makes OR easy to dismiss. Importantly, beyond promoting OR methods, evidence from evaluations would allow us to learn about their use in practice so that we can improve their effectiveness.

In this article, we have reviewed recommended practice in healthcare improvement research, which combines approaches prevalent in health services research, social sciences and public policy. We do not mean to place these approaches on a pedestal, nor suggest that healthcare improvement research is only of the highest methodological quality (Auerbach et al., 2007 ). Rather, we wish to trigger a debate on what constitutes evidence that OR is effective in bringing about desirable changes in healthcare. Many questions remain open. What outcomes should we measure? What are the key mechanisms that make OR interventions effective? We often talk about learning as a key process in OR projects, but how can we operationalise this concept for evaluation? On a more mundane level, how can this enterprise be funded? Who should we partner with? All these questions open exciting avenues for experimentation and progress for the healthcare OR community.

Supplementary Material

Acknowledgments.

The authors thank Graham Martin for valuable and constructive suggestions on earlier drafts of this paper. Any mistake remains our own.

Supplemental data

Supplemental data for this article can be accessed here .

Disclosure statement

No potential conflict of interest was reported by the authors.

• Ahmadi-Javid, A., Jalali, Z., & Klassen, K. J. (2017). Outpatient appointment systems in healthcare: A review of optimization studies. European Journal of Operational Research , 258(1), 3–34. http://www.sciencedirect.com/science/article/pii/S0377221716305239 [ Google Scholar ]
• Auerbach, A. D., Landefeld, C. S., & Shojania, K. G. (2007). The tension between needing to improve care and knowing how to do it. New England Journal of Medicine , 357(6), 608–613. https://www.nejm.org/doi/full/10.1056/NEJMsb070738 [ DOI ] [ PubMed ] [ Google Scholar ]
• Augustsson, H., Churruca, K., & Braithwaite, J. (2019). Re-energising the way we manage change in healthcare: The case for soft systems methodology and its application to evidence-based practice. BMC Health Services Research , 19(1), 666. 10.1186/s12913-019-4508-0 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Barry, D., Kimble, L. E., Nambiar, B., Parry, G., Jha, A., Chattu, V. K., … Goldmann, D. (2018). A framework for learning about improvement: Embedded implementation and evaluation design to optimize learning. International Journal for Quality in Health Care , 30(suppl_1), 10–14. 10.1093/intqhc/mzy008 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Becker, K. H. (2016). An outlook on behavioural OR – Three tasks, three pitfalls, one definition. European Journal of Operational Research , 249(3), 806–815. http://linkinghub.elsevier.com/retrieve/pii/S0377221715008978 [ Google Scholar ]
• Bennett, J. C., & Worthington, D. J. (1998). An example of a good but partially successful or engagement: Improving outpatient clinic operations. Interfaces , 28(5), 56–69. https://pubsonline.informs.org/doi/abs/10.1287/inte.28.5.56 [ Google Scholar ]
• Bevan, H., Eglin, S., Gollop, R., Inglis, S., Laughton, J., Lee, J., … Wilkins, J. (2005). Evaluating improvement . NHS Institute for Innovation and Improvement. [ Google Scholar ]
• Bradley, B. D., Jung, T., Tandon-Verma, A., Khoury, B., Chan, T. C. Y., & Cheng, Y.-L. (2017). Operations research in global health: A scoping review with a focus on the themes of health equity and impact. Health Research Policy and Systems , 15(1), p 32. 10.1186/s12961-017-0187-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Brailsford, S. C., Harper, P. R., Patel, B., & Pitt, M. (2009). An analysis of the academic literature on simulation and modelling in health care. Journal of Simulation , 3(3), 130–140. 10.1057/jos.2009.10 [ DOI ] [ Google Scholar ]
• Brailsford, S. C., & Klein, J. H. (2015). The value of modelling and simulation in healthcare. A review of the evidence and some possible ways forward . B. University. http://cumberland-initiative.org/wp-content/uploads/2015/07/the_value_report_web.pdf [ Google Scholar ]
• Brailsford, S. C., & Vissers, J. (2011). OR in healthcare: A European perspective. European Journal of Operational Research , 212(2), 223–234. 10.1016/j.ejor.2010.10.026 [ DOI ] [ Google Scholar ]
• Brewster, L., Aveling, E.-L., Martin, G., Tarrant, C., & Dixon-Woods, M. (2015). What to expect when you’re evaluating healthcare improvement: A concordat approach to managing collaboration and uncomfortable realities. BMJ Quality & Safety , 24(5), 318–324. https://qualitysafety.bmj.com/content/qhc/24/5/318.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Brocklesby, J. (2016). The what, the why and the how of behavioural operational research—An invitation to potential sceptics. European Journal of Operational Research , 249(3), 796–805. http://linkinghub.elsevier.com/retrieve/pii/S0377221715008760 [ Google Scholar ]
• Burke, R. E., & Shojania, K. G. (2018). Rigorous evaluations of evolving interventions: Can we have our cake and eat it too? BMJ Quality & Safety , 27 (4), 254–257. https://qualitysafety.bmj.com/content/qhc/early/2018/02/09/bmjqs-2017-007554.full.pdf [ DOI ] [ PubMed ] [ Google Scholar ]
• Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., … Kinmonth, A. L. (2007). Designing and evaluating complex interventions to improve health care. BMJ , 334(7591), 455–459. http://www.bmj.com/cgi/doi/10.1136/bmj.39108.379965.BE [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cavaleri, S., & Sterman, J. D. (1997). Towards evaluation of systems-thinking interventions: A case study. System Dynamics Review , 13(2), 171–186. [ DOI ] [ Google Scholar ]
• Connell, N. A. D. (2001). Evaluating soft OR: Some reflections on an apparently ‘unsuccessful’ implementation using a Soft Systems Methodology (SSM) based approach. Journal of the Operational Research Society , 52(2), 150–160. 10.1057/palgrave.jors.2601054 [ DOI ] [ Google Scholar ]
• Crowe, S., Turner, S., Utley, M., & Fulop, N. J. (2017). Improving the production of applied health research findings: Insights from a qualitative study of operational research. Implementation Science , 12(p), 112. 10.1186/s13012-017-0643-3 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Davidoff, F., Dixon-Woods, M., Leviton, L., & Michie, S. (2015). Demystifying theory and its use in improvement. BMJ Quality & Safety , 24(3), 228–238. http://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2014-003627 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Eccles, M., Grimshaw, J., Campbell, M., & Ramsay, C. (2003). Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality & Safety in Health Care , 12(1), 47–52. http://qualitysafety.bmj.com/content/qhc/12/1/47.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Emes, M., Smith, S., Ward, S., & Smith, A. (2018). Improving the patient discharge process: Implementing actions derived from a soft systems methodology study. Health Systems , 8 (2), 117–133. 10.1080/20476965.2018.1524405 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics , 90(3), 891–904. http://link.springer.com/10.1007/s11192-011-0494-7 [ Google Scholar ]
• Fone, D., Hollinghurst, S., Temple, M., Round, A., Lester, N., Weightman, A., … Palmer, S. (2003). Systematic review of the use and value of computer simulation modelling in population health and health care delivery. Journal of Public Health Medicine , 25(4), 325–335. 10.1093/pubmed/fdg075 [ DOI ] [ PubMed ] [ Google Scholar ]
• Franco, L. A. (2013). Rethinking Soft OR interventions: Models as boundary objects. European Journal of Operational Research , 231(3), 720–733. 10.1016/j.ejor.2013.06.033 [ DOI ] [ Google Scholar ]
• Franco, L. A., & Hämäläinen, R. P. (2016). Behavioural operational research: Returning to the roots of the OR profession. European Journal of Operational Research , 249(3), 791–795. http://www.sciencedirect.com/science/article/pii/S0377221715009601 [ Google Scholar ]
• Fretheim, A., & Tomic, O. (2015). Statistical process control and interrupted time series: A golden opportunity for impact evaluation in quality improvement. BMJ Quality & Safety , 24(12), 748–752. https://qualitysafety.bmj.com/content/qhc/24/12/748.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models (1st ed.). Jossey-Bass. [ Google Scholar ]
• Grady, D., Redberg, R. F., & O’Malley, P. G. (2018). Quality improvement for quality improvement studies. JAMA Internal Medicine , 178(2), 187. 10.1001/jamainternmed.2017.6875 [ DOI ] [ PubMed ] [ Google Scholar ]
• Heaton, J., Day, J., & Britten, N. (2016). Collaborative research and the co-production of knowledge for practice: An illustrative case study. Implementation Science , 11(1), p 20. 10.1186/s13012-016-0383-9 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Jahangirian, M., Naseer, A., Stergioulas, L., Young, T., Eldabi, T., Brailsford, S. C., … Harper, P. (2012). Simulation in health-care: Lessons from other sectors. Operational Research , 12(1), 45–55. 10.1007/s12351-010-0089-8 [ DOI ] [ Google Scholar ]
• Kunc, M., Harper, P., & Katsikopoulos, K. (2018). A review of implementation of behavioural aspects in the application of OR in healthcare. Journal of the Operational Research Society , 71 (7), 1055–1072. 10.1080/01605682.2018.1489355 [ DOI ] [ Google Scholar ]
• Lamé, G., Jouini, O., & Stal-Le Cardinal, J. (2016). Outpatient chemotherapy planning: A literature review with insights from a case study. IIE Transactions on Healthcare Systems Engineering , 6(3), 127–139. 10.1080/19488300.2016.1189469 [ DOI ] [ Google Scholar ]
• Lamé, G., & Simmons, R. K. (2020). From behavioural simulation to computer models: How simulation can be used to improve healthcare management and policy. BMJ Simulation and Technology Enhanced Learning , 6(2), 95–102. https://stel.bmj.com/content/bmjstel/early/2018/10/20/bmjstel-2018-000377.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Long, K. M., & Meadows, G. N. (2018). Simulation modelling in mental health: A systematic review. Journal of Simulation , 12(1), 76–85. 10.1057/s41273-017-0062-0 [ DOI ] [ Google Scholar ]
• Mahdavi, M., Malmström, T., van de Klundert, J., Elkhuizen, S., & Vissers, J. (2013). Generic operational models in health service operations management: A systematic review. Socio-economic Planning Sciences , 47(4), 271–280. http://www.sciencedirect.com/science/article/pii/S0038012113000360 [ Google Scholar ]
• Marsh, K., Lanitis, T., Neasham, D., Orfanos, P., & Caro, J. (2014). Assessing the value of healthcare interventions using multi-criteria decision analysis: A review of the literature. PharmacoEconomics , 32(4), 345–365. 10.1007/s40273-014-0135-0 [ DOI ] [ PubMed ] [ Google Scholar ]
• Marynissen, J., & Demeulemeester, E. (2019). Literature review on multi-appointment scheduling problems in hospitals. European Journal of Operational Research , 272(2), 407–419. http://www.sciencedirect.com/science/article/pii/S0377221718302108 [ Google Scholar ]
• Midgley, G. (1998). Theory and practice in operational research. Journal of the Operational Research Society , 49(11), 1219–1220. 10.1057/palgrave.jors.2600653 [ DOI ] [ Google Scholar ]
• Mohiuddin, S., Busby, J., Savović, J., Richards, A., Northstone, K., Hollingworth, W., … Vasilakis, C. (2017). Patient flow within UK emergency departments: A systematic review of the use of computer simulation modelling methods. BMJ Open , 7(5), e015007. http://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2016-015007 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Monks, T., Pearson, M., Pitt, M., Stein, K., & James, M. A. (2015). Evaluating the impact of a simulation study in emergency stroke care. Operations Research for Health Care , 6, 40–49. http://www.sciencedirect.com/science/article/pii/S2211692314200348 [ Google Scholar ]
• Monks, T., Robinson, S., & Kotiadis, K. (2014). Learning from discrete-event simulation: Exploring the high involvement hypothesis. European Journal of Operational Research , 235(1), 195–205. http://www.sciencedirect.com/science/article/pii/S0377221713008047 [ Google Scholar ]
• Munro, J., Nicholl, J., O’Cathain, A., & Knowles, E. (2000). Impact of NHS Direct on demand for immediate care: Observational study. BMJ , 321(7254), 150–153. https://www.bmj.com/content/bmj/321/7254/150.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ormerod, R. J. (2014). The mangle of OR practice: Towards more informative case studies of ‘technical’ projects. Journal of the Operational Research Society , 65(8), 1245–1260. http://link.springer.com/10.1057/jors.2013.78 [ Google Scholar ]
• Øvretveit, J. (2011). Understanding the conditions for improvement: Research to discover which context influences affect improvement success. BMJ Quality & Safety , 20(Suppl1), i18–i23. https://qualitysafety.bmj.com/content/qhc/20/Suppl_1/i18.full.pdf [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Palmer, R., Fulop, N. J., & Utley, M. (2018). A systematic literature review of operational research methods for modelling patient flow and outcomes within community healthcare and other settings. Health Systems , 7(1), 29–50. 10.1057/s41306-017-0024-9 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• American Evaluation Association, Patton, M. Q., Asibey, E., Dean-Coffey, J., Kelley, R., Miranda, R., & Newman, G. F. (2014). What is Evaluation? American Evaluation Association, Wahsington, DC, USA. https://www.eval.org/p/bl/et/blogid=2&blogaid=4
• Perla, R. J., Provost, L. P., & Murray, S. K. (2011). The run chart: A simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety , 20(1), 46–51. https://qualitysafety.bmj.com/content/qhc/20/1/46.full.pdf [ DOI ] [ PubMed ] [ Google Scholar ]
• Pitt, M., Monks, T., Crowe, S., & Vasilakis, C. (2016). Systems modelling and simulation in health service design, delivery and decision making. BMJ Quality & Safety , 25(1), 38–45. 10.1136/bmjqs-2015-004430 [ DOI ] [ PubMed ] [ Google Scholar ]
• Portela, M. C., Pronovost, P. J., Woodcock, T., Carter, P., & Dixon-Woods, M. (2015). How to study improvement interventions: A brief overview of possible study types. BMJ Quality & Safety , 24(5), 325–336. http://qualitysafety.bmj.com/lookup/doi/10.1136/bmjqs-2014-003620 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Read, M., Gear, T., & Vince, R. (2012). Group inquiry to aid organisational learning in enterprises. Journal of the Operational Research Society , 63(6), 736–747. 10.1057/jors.2011.80 [ DOI ] [ Google Scholar ]
• Royston, G., Dost, A., Townshend, J., & Turner, H. (1999). Using system dynamics to help develop and implement policies and programmes in health care in England. System Dynamics Review , 15(3), 293–313. https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291099-1727%28199923%2915%3A3%3C293%3A%3AAID-SDR169%3E3.0.CO%3B2-1 [ Google Scholar ]
• Royston, G., Halsall, J., Halsall, D., & Braithwaite, C. (2003). Operational Research for informed innovation: NHS Direct as a case study in the design, implementation and evaluation of a new public service. Journal of the Operational Research Society , 54(10), 1022–1028. 10.1057/palgrave.jors.2601617 [ DOI ] [ Google Scholar ]
• Samudra, M., Van Riet, C., Demeulemeester, E., Cardoen, B., Vansteenkiste, N., & Rademakers, F. E. (2016). Scheduling operating rooms: Achievements, challenges and pitfalls. Journal of Scheduling , 19(5), 493–525. 10.1007/s10951-016-0489-6 [ DOI ] [ Google Scholar ]
• Soorapanth, S., & Young, T. (2019). Assessing the value of modelling and simulation in health care: An example based on increasing access to stroke treatment. Journal of the Operational Research Society , 70(2), 226–236. 10.1080/01605682.2017.1421857 [ DOI ] [ Google Scholar ]
• Sterman, J. D. (1994). Learning in and about complex systems. System Dynamics Review , 10(2–3), 291–330. 10.1002/sdr.4260100214 [ DOI ] [ Google Scholar ]
• The Health Foundation (2015). Evaluation: What to consider. Commonly asked questions about how to approach evaluation of quality improvement in health care . The Health Foundation, London, UK https://www.health.org.uk/sites/default/files/EvaluationWhatToConsider.pdf [ Google Scholar ]
• Thompson, J. P., Howick, S., & Belton, V. (2016). Critical Learning Incidents in system dynamics modelling engagements. European Journal of Operational Research , 249(3), 945–958. http://www.sciencedirect.com/science/article/pii/S0377221715008905 [ Google Scholar ]
• van Lent, W. A. M., VanBerkel, P., & van Harten, W. H. (2012). A review on the relation between simulation and improvement in hospitals. BMC Medical Informatics and Decision Making , 12(1), p 18. http://www.biomedcentral.com/1472-6947/12/18 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• van Sambeek, J. R. C., Cornelissen, F. A., Bakker, P. J. M., & Krabbendam, J. J. (2010). Models as instruments for optimizing hospital processes: A systematic review. International Journal of Health Care Quality Assurance , 23(4), 356–377. 10.1108/09526861011037434 [ DOI ] [ PubMed ] [ Google Scholar ]
• Walshe, K. (2009). Pseudoinnovation: The development and spread of healthcare quality improvement methodologies. International Journal for Quality in Health Care , 21(3), 153–159. http://intqhc.oxfordjournals.org/cgi/doi/10.1093/intqhc/mzp012 [ DOI ] [ PubMed ] [ Google Scholar ]
• White, L. (2006). Evaluating problem-structuring methods: Developing an approach to show the value and effectiveness of PSMs. Journal of the Operational Research Society , 57(7), 842–855. 10.1057/palgrave.jors.2602149 [ DOI ] [ Google Scholar ]
• White, L., Burger, K., & Yearworth, M. (2016). Understanding behaviour in problem structuring methods interventions with activity theory. European Journal of Operational Research , 249(3), 983–1004. http://www.sciencedirect.com/science/article/pii/S0377221715006785 [ Google Scholar ]
• Williams, T. (2008). Management science in practice . Wiley. [ Google Scholar ]
• Wilson, J. C. T. (1981). Implementation of Computer Simulation Projects in Health Care. Journal of the Operational Research Society , 32(9), 825–832. 10.1057/jors.1981.161 [ DOI ] [ PubMed ] [ Google Scholar ]
• Young, T., Soorapanth, S., Wilkerson, J., Millburg, L., Roberts, T., & Morgareidge, D. (2018). The costs and value of modelling-based design in healthcare delivery: Five case studies from the US. Health Systems , 9 (3), 253–262. 10.1080/20476965.2018.1548255 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Zhang, C., Grandits, T., Härenstam, K. P., Hauge, J. B., & Meijer, S. (2018). A systematic literature review of simulation models for non-technical skill training in healthcare logistics. Advances in Simulation , 3(1), 15. 10.1186/s41077-018-0072-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

• View on publisher site
• PDF (457.7 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Empirical-Based Typology of Health Care Utilization by Medicare Eligible Veterans

Affiliations.

• 1 Iowa City VA Health Care System, Iowa City, IA.
• 2 Department of Internal Medicine, University of Iowa, Iowa City, IA.
• 3 Department of Internal Medicine, Wake Forest School of Medicine, Winston-Salem, NC.
• 4 Comprehensive Access and Delivery Research and Evaluation (CADRE), Iowa City VA Health Care System, Iowa City, IA.
• 5 Department of Psychiatry, University of Iowa, Iowa City, IA.
• PMID: 29896771
• PMCID: PMC6235816
• DOI: 10.1111/1475-6773.12995

Objective: Up to 70 percent of patients who receive care through Veterans Health Administration (VHA) facilities also receive care from non-VA providers. Using applied classification techniques, this study sought to improve understanding of how elderly VA patients use VA services and complementary use of non-VA care.

Methods: The study included 1,721,900 veterans age 65 and older who were enrolled in VA and Medicare during 2013 with at least one VA encounter during 2013. Outpatient and inpatient encounters and medications received in VA were classified, and mutually exclusive patient subsets distinguished by patterns of VA service use were derived empirically using latent class analysis (LCA). Patient characteristics and complementary use of non-VA care were compared by patient subset.

Results: Five patterns of VA service use were identified that were distinguished by quantity of VA medical and specialty services, medication complexity, and mental health services. Low VA Medical users tend to be healthier and rely on non-VA services, while High VA users have multiple high cost illnesses and concentrate their care in the VA.

Conclusions: VA patients distinguished by patterns of VA service use differ in illness burden and the use of non-VA services. This information may be useful for framing efforts to optimize access to care and care coordination for elderly VA patients.

Keywords: Administrative data uses; Medicare; VA Health Care System; access/demand/utilization of services; chronic disease.

© Health Research and Educational Trust.

Publication types

• Research Support, U.S. Gov't, Non-P.H.S.
• Aged, 80 and over
• Ambulatory Care / statistics & numerical data
• Health Status
• Hospitalization / statistics & numerical data
• Insurance Claim Review
• Medicare Part A / statistics & numerical data*
• Medicine / statistics & numerical data
• Mental Health Services / statistics & numerical data
• Polypharmacy
• Socioeconomic Factors
• Transportation
• United States
• United States Department of Veterans Affairs / statistics & numerical data*
• Veterans Health

• Open access
• Published: 23 November 2023

Trajectories of primary health care utilization: a 10-year follow-up after the Swedish Patient Choice Reform of primary health care

• Hannes Kohnke 1 , 2 ,
• Andrzej Zielinski 3 ,
• Anders Beckman 4 &
• Henrik Ohlsson 4  

BMC Health Services Research volume  23 , Article number:  1294 ( 2023 ) Cite this article

1255 Accesses

3 Citations

Metrics details

A Correction to this article was published on 18 March 2024

This article has been updated

In January 2010, the choice reform was instituted in Swedish primary health care establishing free entry for private primary health care providers and enabling patients to choose freely among primary health care centers. The motivation behind the reform was to improve access to primary care and responsiveness to patient expectations. Reform effects on health care utilization have previously been investigated by using subgroup analyses assuming a pattern of homogeneous subgroups of the population. By using a different methodological approach, the aim of this study was to, from an equity perspective, investigate long term trends of primary health care utilization following the choice reform.

A closed cohort was created based on register data from Region Skåne, the third most populated region in Sweden, describing individuals’ health care utilization between 2007–2017. Using a novel approach, utilization data, measured as primary health care visits, was matched with socioeconomic and geographic determinants, and analyzed using logistic regression models.

A total of 659,298 individuals were included in the cohort. Sex differences in utilization were recorded to decrease in the older age group and to increase in the younger age group. Multivariable logistic regression showed increasing utilization in older men to be associated with higher socioeconomic position, while in women it was associated with lower socioeconomic position. Furthermore, groups of becoming high utilizers were all associated with lower socioeconomic position and with residence in urban areas.

The impact of demographic, socioeconomic and geographic determinants on primary health care utilization varies in magnitude and direction between groups of the population. As a result, the increase in utilization as observed in the general population following the choice reform is unevenly distributed between different population groups.

Peer Review reports

In Sweden, as in many countries, primary health care (PHC) forms the basis of the health care system and is regarded as an efficient way to address main causes of, and risk factors for, poor health [ 1 , 2 , 3 ]. In most societies, PHC is distinguished as the part of the health care system that benefits those of lower socioeconomic status (SES) to a higher degree [ 4 , 5 ]. Fundamental for providing an equitable PHC is that physicians in PHC (general practitioners or GPs) appropriate their services according to differences in patient needs. The appropriation of GP utilization is in turn influenced by several other factors such as geographical inequity in the distribution of primary health care centers (PCCs), user fees, health literacy and incentives related to funding and competition for patients [ 6 ].

In Sweden, the health care system is mainly tax-funded and citizens are ensured a universal coverage with no or minimal patient fees. PHC accounts for about 20% of total health care expenditures and is provided by multi-professional PCCs staffed with nurses, general practitioners, and to varying extent, other health care professionals [ 7 ]. Half of all doctor visits are made in PHC [ 7 ]. Twenty one independent regions are responsible for the financing and organization of health care. Although PHC in a majority of the regions has no formal gatekeeping function, typically, PCCs serve as patients’ entry point into the health care system and as the continuing focal point for most needed health care services [ 3 ]. This study is set in Region Skåne, the third largest Swedish region with 1.3 million residents.

Dissatisfaction with the performance of PHC services has led to political interest in policies that promote competition in many countries [ 8 ]. Quasi-markets and public competition have been introduced and assessed in health care systems around the world since the early 1990s, with varying degrees of success [ 9 ]. Between 2007 and 2010 a wave of PHC reforms was initiated throughout Sweden. These reforms, commonly referred to as the Patient Choice Reform , established free entry for private PHC providers and enabled patients to choose freely among PCCs. The motivation behind the reforms was to improve access to PHC and responsiveness to patient expectations [ 10 ]. Following the reform date in Region Skåne (1 May 2009), any provider that fulfilled the specified conditions for accreditation would be allowed to open a PCC, it would no longer be allowed for providers to reject patients who wished to enroll, and patients could choose freely among all PCCs in the region. Before the reform, patient choice was more limited, and providers were allowed to reject patients living outside their geographical catchment area. Also, with the reform the reimbursement system for PCCs was gradually redesigned from being based 80% on capitation to enrolled patients and 20% on fee-for-service, to being solely capitation-based [ 11 ]. To accommodate for variations in expected costs, capitation is higher for patients with many diagnoses or low SES. In 2010, it became mandatory for all regions to provide freedom of establishment for private primary care providers, and patients with choice of provider, through a change in the national Health Care Act [ 12 ]. Following years, the patient choice system in Region Skåne gradually became more comprehensive, specifying standards for coordination of elderly care and various diagnose specific nurse led clinics, and also came to include certain specialized health care services [ 11 ].

The effects of the Patient Choice Reform have been analyzed in scientific reports and by government and regional agencies. Overall, the number of PCCs has increased and new PCCs have mainly been established in urban areas or in areas where healthcare needs are lower [ 13 , 14 , 15 ]. The closing of PCCs has mainly affected areas with higher expected health care needs [ 13 ]. Consequently, concerns have been raised about increasing geographical inequity in PCC allocation along with declining access and continuity of care for part of the population [ 13 , 16 , 17 ]. With an increasing number and broader range of providers, the health care system has become more complex and challenging to navigate. Following this, demands on patient participation and health literacy are expected to increase – health literacy referring to individuals’ abilities to access, understand and communicate health-related information needed to make informed health decisions [ 18 ]. Investigations of the impact of the reform show an overall increase in PHC utilization but individuals with higher income or minor symptoms have increased their utilization to a higher degree than those with low income or more severely ill [ 6 , 13 , 19 ]. However, these investigations are in part limited by cross-sectional designs, not being able to detect changes of utilization on an individual level, or short follow-up periods after the reform.

A common approach when assessing the effects of the Patient Choice Reform on health care utilization has been by subgroup analyses assuming a pattern of homogeneous subgroups of the population, each subgroup having similar outcomes [ 6 , 13 , 19 , 20 , 21 ]. However, such characteristics may not be true in gross subgroup divisions where outcomes may not work the same way in all patients. The use of a person-oriented technique for subgrouping, that does not assume the relationship between variables to be the same for all patients, can provide an understanding of how differences in independent variables between patients affect outcome [ 22 ]. To our knowledge such an approach has never been applied when investigating effects of the Patient Choice Reform. By using a method allowing for person-oriented subgroup analyses, the purpose of this study was to investigate long term trends of PHC utilization following the Patient Choice Reform, and to identify groups of the population with changed utilization.

This study is based on administrative register data from Region Skåne, describing individual health care utilization of all publicly funded PHC services. Privately funded health care services, which comprise less than 1% of total health care spendings in Sweden, were not included [ 23 ]. On the individual level, health care utilization data from Region Skåne was linked to demographic (age and sex), socioeconomic, (income, education and civil status) and geographic (municipality of residence) determinants obtained from Statistics Sweden.

Retrospectively, a closed cohort was created of all inhabitants with a registered address in Region Skåne between 2007 and 2017. Individuals aged under 20 or over 69 in 2007 or have had a registered address outside of Region Skåne at some point during follow-up were excluded from the study. The lower age limit was chosen to allow the use of income and educational level as a proxy for SES, and the upper limit to allow inclusion of retirees but to exclude older age groups with a tapering population.

Outcome measure

The outcome measure was defined as the total number of GP visits per individual and calendar year. GP visits due to preventative health care (i.e. maternity care, child health care) were excluded. Based on frequency analyses of the number of annual visits, four distinct utilization-groups were created; 0 visits, 1 visit, 2–3 visits, and more than 3 visits. These groups will be referred to as low- (0–1 visit), intermediate- (2–3 visits), and high-utilizers (> 3 visits).

Independent variables

All independent variables (age, sex, income, educational level, civil status and municipality of residence) were defined at baseline (1 January 2007).

Due to the influence of age on income, analyses were made on three different age groups based on age at baseline; 20–34 (young), 35–54 (middle-aged) and 55–69 (older) years of age (corresponding birth years: 1987-73, 1972-53, and 1952-38).

Income was defined as pre-tax household income equalized to the number of family members. Pre-tax income included earnings from employment, business, income transfers (e.g., pension payments, unemployment benefits, or paid sick leave), and capital gain, but not return of capital. The income variable was categorized into three equal-sized groups.

Educational level was categorized to elementary school, high school or higher education, and civil status to either married/cohabiting or single/divorced/widow/widower.

The level of urbanicity of municipality of residence was grouped into three categories. The Swedish Association of Local Authorities and Regions classifies municipalities in nine categories based on structural parameters such as population, degree of urbanization and commuting patterns [ 24 ]. This classification was modified to fit into three groups: 1) large cities (population over 200,000) and medium-sized towns (population over 50,000), 2) small towns (population over 15,000) and commuting municipalities near large cities, 3) rural municipalities and commuting municipalities near medium- and small-sized towns. These three groups are referred to as urban, semi-urban and rural.

Statistical method

Datasets were constructed for each sex and age group. Separately for each dataset, group-based trajectory modeling (GBTM) was used to identify individuals with similar health care utilization patterns. GBTM is a semi-parametric model, designed to analyze longitudinal data [ 25 , 26 ]. Based on the assumption of a discrete distribution of the population, GBTM makes it possible to distinguish subgroups of individuals in a population with a similar trajectory [ 22 , 25 , 26 ]. For each individual, the model determines the probability of belonging to one subgroup or another (posterior group probability). Individuals are assigned to a subgroup based on their highest posterior group probability [ 25 , 26 ].

In each dataset the number of trajectory-groups, indicated by the observed variables, was determined by comparing model fit statistics between nested models. Improvement in model fit was indicated by smaller values of the Bayesian information criterion (BIC) and Akaike’s Information Criterion (AIC). However, as the number of trajectory-groups is influenced by the number of observed variables, both empirical (improved model fit) and theoretical (model interpretability) aspects were considered. Given the large sample size, statistical power allowed for identification of trajectory-groups too small to be useful in clinical- or research work. Hence, an arbitrary limit was set and only datasets where all identified trajectory-groups had a prevalence of ≥ 2% of the dataset population were considered for further analyses.

In the next step, trajectory-groups with similar health care utilization at baseline and with diverging patterns over time, were selected pairwise. Logistic regression was used to describe differences between the pairwise selected trajectory-groups (see Tables 2 and 3 ). Both bivariate- and multivariable regression models were constructed, the latter with age, income, education, civil status, and municipality of residence as independent variables. In the regression models, age was treated as a continuous variable and all other as categorical. The results were presented as odds ratios (ORs) with 95% confidence intervals (CIs).

GBTMs were performed with SAS 9.4 (SAS Institute, Inc., Cary, NC) and logistic regressions with IBM SPSS version 27 (IBM Corp., Armonk, NY).

Descriptive

In 2007, Region Skåne had 766,029 residents aged 20 to 69 where of 659,298 (86%) were included in the study. Study population demographics are provided in Table 1 .

Visits to GPs

In total, 9,577,498 GP visits were registered during the study period. Of these visits, 60% of were made by women and 3% of the study population made no visits at all. The average number of annual visits per individual rose from 1.21 in 2007 to 1.42 in 2017, corresponding to an absolute increase by 131,370 visits or relative increase by 17% (Table 1 ). The increase in utilization varied between groups but was for all groups more distinct between 2007–2012 and gender specific differences were most pronounced in the youngest and oldest age groups. Compared to the corresponding age group of the opposite sex, the relative increase in utilization was 9 percentage points higher for younger women and 13 percentage points higher for older men. Older men had the highest relative increase (25%), but nonetheless fewer annual GP visits on average than the corresponding female age group in 2017 (1.66 and 1.94 respectively).

Trajectories of primary health care utilization

Based on model fit statistics (see Additional Table 1 ) six trajectory datasets were chosen for further analysis, one set for each sex- and age group (see Additional Fig. 1 ). Independent variables varied between trajectory-groups in each dataset (see Additional Table 2 a for males and b for females). In general, each dataset followed a generic pattern of trajectories as shown in Fig.  1 . Common for each dataset was the presence of trajectory-groups with relatively unchanged continuously low, intermediate, and high utilization as well as trajectories with changing utilization in close to linear fashions. For simplicity, trajectories will henceforth be referred to by their roman numeral as denoted in Fig.  1 .

Generic trajectory analysis output. Generic trajectory analysis output. On the Y-axis, the number of annual GP visits per individual are categorized by 4 utilization-groups corresponding to; 0 visits, 1 visit, 2–3 visits, and more than 3 visits

Common for all trajectory datasets, trajectory-groups between low utilization and intermediate utilization (I, III, V and VI) contained larger proportions of individuals. Lowest proportion of individuals were found in trajectory-groups indicating continuously high utilization (VII) or changing utilization between intermediate- and high levels (II and IV). Trajectory-groups with changed utilization between low and intermediate levels were recorded in all sex- and age groups except for in older women. Trajectory-groups with changed utilization between intermediate- and high levels (from intermediate- to high- or from high- to intermediate levels) were exclusively recorded in female groups.

In the trajectory dataset for younger women, trajectories varied from the generic pattern in Fig.  1 by being more dispersed on the y-axis at the starting point, making analysis more complex. Depending on the position of the reference trajectory, above or below the trajectory being analyzed, the results might be subjected to under- or overestimation. Two trajectories showed increased utilization – increase from low utilization (I) and increase from intermediate utilization (II). Due to their positioning relative to reference trajectories, each increasing trajectory was analyzed with two different reference trajectories separately.

Logistic regression

Results from the bivariate regression model (see Additional Table 3 a for males and b for females) differed inconsiderably from the multivariable model (Tables 2 for males and 3 for females).

Multivariable regression analyses showed consistent effects of age on utilization. In most trajectory-groups, increased utilization was associated with increasing age, and decreased utilization was associated with decreasing age. These associations were stronger among older men and women as well as among middle-aged men.

Analyses of the effect of income or education on changed utilization between low- and intermediate levels (I, III) showed distinct results for older men compared to other sex- and age groups where similar change in utilization was recorded. Among older men, ORs for increased utilization from low levels (I) was 1.33 (CI 1.24–1.43) and 1.27 (CI 1.18–1.37) respectively for middle and high income vs. low income. Among middle-aged men and younger women, increased utilization from low levels (I) was associated with lower income and lower education. The strongest association was observed among younger women with recorded ORs of 0.86 (CI 0.82–0.90) and 0.95 (CI 0.90–1.00) for high income vs. low income and 0.79 (CI 0.71–0.88) and 0.72 (CI 0.64-0.082) for higher education vs. primary school). Decreased utilization from intermediate levels (III) was among older men associated with lower income and with a corresponding OR of 0.87 (CI 0.82–0.92) for middle or and high income vs. low income. Among middle-aged and young men and women, the same decrease in utilization was associated with higher income and higher education.

The effect of income or education on changed utilization between intermediate- and high levels (II, IV) showed a similar pattern in all female groups. In all female age groups, increased utilization from intermediate levels (II) was associated with lower education. Among middle-aged and younger women, increased utilization from intermediate levels (II) was also associated with lower income. The strongest associations with low education and low income were recorded among younger women with ORs of 0.65 (CI 0.60–0.71) and 0.53 (0.48–0.57) for high income vs. low income and 0.65 (CI 0.57–0.74) and 0.36 (CI 0.31–0.41) for higher education vs. primary school. Decreased utilization from high levels (IV) was recorded among older and middle-aged women and was here associated with both higher income and education.

Residence in urban areas showed to be indicative of increased utilization when compared to residence in semi-urban or rural areas. When compared to residence in urban areas, differences in the effect on change in utilization was small between residence in semi-urban and rural areas. Without exceptions, all trajectories with increasing utilization from intermediate levels (II) were associated with residence in urban areas. For younger women, increased utilization from both low- and intermediate levels (I and II) were associated with residence in urban areas.

From an equity perspective, this 10-year follow-up examined trends of PHC utilization following the Swedish Patient Choice Reform. Prior investigations of the effects of the reform have in part been limited by design or short follow-up time and by using a novel person-oriented approach to subgrouping, this study adds knowledge to previous findings. First, our findings showed a gradual increase in utilization in the population, unevenly distributed dependent on sex and age. Secondly, socioeconomic determinants affected utilization in different directions in different subgroups. Increasing utilization among older men was associated with higher SES, while among women a similar change was associated with lower SES. Third, increase to high levels of utilization was primarily observed among women, and was irrespective of age associated with lower SES and residence in urban areas, the associations being stronger in younger age groups.

On a national level, PHC utilization in terms of GP visits, has steadily increased until 2011 whereafter utilization has been more constant [ 13 ]. In Region Skåne, previous reports have shown the number of GP visits to be relatively constant the years before the reform, to increase the year of the reform, and then to slightly decline between 2011 and 2014 [ 6 , 11 ]. Similarly, our results showed a more pronounced increase in utilization between 2007–2012 and relatively constant utilization between 2012–2017. Given the timely correlation, the observed increase between 2007–2012 is believed to be a reform effect. PHC utilization patterns following the reform have been described to correlate well with expected health care needs based on sex and age [ 13 ]. Here, increased utilization was shown to be unevenly distributed between defined gross sex- and age groups. Sex differences in utilization were observed to decrease in the oldest age group while they increased in the youngest age group. Taking sex differences in relative utilization at baseline into account, these changes could be interpreted as shifts in gender equity.

By choosing a different approach to investigate socioeconomic and geographical equity in PHC utilization, this study gives nuance to previous findings on the subject. Reports have previously shown how the Patient Choice Reform has facilitated an uneven distribution of PCCs within and between regions, affecting geographical equity in a negative direction [ 13 , 14 , 15 ]. In Region Skåne most new PCCs following the reform have been established in the more densely populated southwestern part of the region [ 14 ]. Despite the uneven distribution of PCCs, GP visits remained pro-poor 2 years following the reform in the three largest regions (Stockholm, Västra Götaland and Skåne) [ 27 ]. In Region Skåne the number of consultations increased relatively more for individuals with high-income than with low income [ 13 ]. A previous study using a cohort from Region Skåne, identified men and women over 64 years and with an income above median to have increased their PHC utilization the most at follow-up 2 years post-reform [ 6 ]. Here, using a longer follow-up, the only trajectory-group with both increasing utilization and association with higher SES, was that of older men. Furthermore, decreasing utilization in older men was associated with lower SES. To the contrary, increasing levels of utilization in the majority of female groups was associated with lower SES, the association getting stronger with falling age and higher levels of utilization. Also, female groups with decreasing utilization were all associated with higher SES. These shifts in utilization can be interpreted as pro-rich for older men and pro-poor for women. To our knowledge, relationships between PHC utilization in women and SES have not previously been reported.

Increased geographical inequity in the allocation of PCCs could be a contributing factor to observed changes in utilization. With few exceptions, trajectory-groups with decreasing utilization were associated with residence in non-urban areas. Furthermore, all trajectory-groups where utilization increased to high levels were associated with residence in urban areas. These groups of becoming high utilizers were all female and associated with lower SES. A theoretically possible explanation for this could be increased health care needs in defined female population groups, and that improved accessibility following the Patient Choice Reform has led to increased equity in terms of utilization. This explanation seems unlikely given the relatively younger urban population (median 41 years) compared to the non-urban population (median 45 years). More likely, increased PHC utilization in urban areas is facilitated by an improved PHC accessibility with a lesser regard to actual health care needs. In support of this notion, there have been reports of change in health care seeking behavior among young adults since the reform, indicating a lower threshold for consulting health care professionals for minor health care issues [ 21 , 28 ]. A contributing factor could be increased health care seeking behavior for mental health related issues. Such an increase would also provide insights to the marked sex difference in utilization between young women and men (utilization in younger women increased three-fold compared to men of the same age group). Increasing numbers of young adults, predominantly female, have been reported to experience mental health issues such as depression and anxiety [ 29 , 30 , 31 ]. Furthermore, PHC utilization for mental health issues has been shown to increase and diagnoses of depression and anxiety in PHC are twice as common in women as in men [ 31 , 32 , 33 ].

An explanation to the associations between the becoming high utilizers and low SES might also reside in differences in health literacy. Health literacy, referring to an individual’s capacity to meet complex demands of health in modern society [ 18 ], has likely become more important with the Patient Choice Reform. In urban areas, where the majority of new PCCs are located, patient choice is made more complex both due to the multitude of PCCs, but also due to the diversification of offered services. In addition to more traditional PCCs, the range of available services include drop-in units, out-of-hours clinics, nurse-led clinics, and diagnosis-specific health care services [ 34 ]. As inadequate health literacy is strongly associated with low SES [ 35 ], it likely serves as a contributing factor to the observed pattern of increasing utilization in urban areas. Health literacy might also be an explanation for the observed increase in utilization in older men associated with higher SES and residence in non-urban areas. Adequate health literacy for this group might serve as an enabling factor to navigate the choice-system and to gain access to PCCs. PCCs located in urban areas will likely require more means to access for individuals living in non-urban areas.

As proxies for high SES, one might expect higher income, higher educational level and being married or cohabitant to affect utilization in a similar fashion. Our results suggest income and civil status rather than education to be indicative for utilization in older individuals. Furthermore, income and education rather than civil status seemed indicative for utilization among younger and middle-aged individuals. These observations are likely due to changing availability and relevance of higher education along with changing social implications of marriage over time.

Strengths of this study are the cohort design, large study population, and reliable data sources. Compared to an ecological design, a closed cohort allows for analyses on changes in utilization over time on an individual level. By constructing a cohort, a constant population could be studied throughout a 10-year period and potential effects by fluctuations in the population due to new entries or exits could be disregarded. Consequently, other factors implicative for PHC utilization are likely to have changed during the observation period as well – both on the individual and population level – and inferences about the potential effect of the reform on utilization should be made with caution.

On the individual level, both income and education can vary over time, typically in early adulthood and following retirement. Sensitivity analyses defining income- and educational level at endpoint instead of at baseline showed little or no effect on outcomes. The data in this study are based on register information gathered for administrative purposes. In Sweden, there is a long tradition of recording, storing, and managing information in registers, both by individual regions and government authorities. Data validity in these registers is known to be high. In addition, the use of family income per consumption unit instead of individual income gives a more realistic estimation of an individual’s SES.

On the population level, health care utilization can be affected by a range of different factors such as the external environment (physical, political and economic), the organization of the health care system and population characteristics (predisposing characteristics, enabling resources and need) [ 36 ]. Throughout the study period, external factors have remained stable and are not believed to have influenced the results. However, during a 10-year follow-up, minor changes in specialized health care provision are likely to have taken place. Although minor, such changes might have impacted subgroups of the population and thus have affected PHC utilization in those subgroups. Furthermore, the increasing age of the population has probably impacted subgroups of the population to different degrees during the 10 year follow-up. Identified trajectory-groups are likely a representation of heterogeneity in utilization affected by the reform as well as heterogeneity in underlying health trajectories of individuals and, in turn, their individual care needs. Hence, a limitation of this study is the absence of control for morbidity and changes in diagnostic patterns over time. Considering the basis for an equitable health care – the appropriation of health care services according to differences in patient needs – analyses of inequities in health care utilization should be controlled for morbidity (as a proxy for ‘need’). The Johns Hopkins ACG (Adjusted Clinical Groups) Case-Mix System was developed to evaluate the relationship between individual morbidity and utilization and is a widely used tool when controlling for morbidity [ 37 ]. To evaluate the relationship between individual morbidity and utilization, the initial regression model in this study applied ACG as an independent variable. At the beginning and end of follow-up, all individuals were subscribed one of six Resource Utilization Bands (RUBs) – RUB 0 corresponding to no need for health care and RUB 5 a high need, each RUB consisting of individuals with the same type and degree of comorbidity. In this initial model, regardless of being defined at baseline or endpoint, ORs for RUB showed an absolute correlation to health care utilization. Adding no useful information, ACG was excluded in the final regression model. Since knowledge of potential changes in diagnostic patterns might have given useful insights to underlying causes of observed changes in utilization, absence of diagnostic information limits the study.

Another limitation to this study concerns loss of information due to inherent characteristics of the study method. A premise for analyzing differences in utilization trajectories, according to the method used here, is the categorization of utilization-groups based on the annual number of GP visits. This categorization implies a loss of information and, hence, could have affected the sensitivity of the results in a negative fashion. Due to the large study population, and the large amount of PHC utilization data, we do not believe this information loss sufficient to affect the final results.

In this study, only real-life visits were registered in the outcome variable. However, excluding virtual doctor visits is not believed to have a significant effect on reported outcomes. Online care platforms, making virtual doctor visits possible, just entered the Swedish PHC market in 2016. In 2017, virtual visits constituted only 2% of the total number of GP visits nationwide [ 28 ].

An essential limitation to this study is the used outcome measure. The measure of the number of GP visits over time showed nothing about the content or the quality of individual visits – if the patient has received the care needed or not. A fewer number of longer visits might be more beneficial to a patient than several shorter visits. A possible explanation for the observed increase in utilization in younger women might be a loss in quality in the health care given. For a better understanding, further studies using different measures and methodology are warranted. In this respect, potential changes in continuity of care, as a quality measure, would be relevant to investigate for groups of the population.

The results of this study can be generalized to other Swedish regions with some considerations – foremost regarding population demographics and regional variation in health care provision. In addition, regional reimbursement models are known to differ in terms of to which degree they are based on capitation or fee-for-service, and whether capitation is risk-adjusted or not. Regional variations in PHC reimbursement likely affect PHC utilization and comparisons between regions need to take such differences into consideration.

Conclusions

This study showed changes in PHC utilization over a 10-year period overlapping the patient choice reform in the Swedish Region of Skåne. The observed increase in utilization in the general population was unevenly distributed between defined sex- and age groups. Sex differences in utilization decreased in the oldest age group while they increased in the youngest age group, the latter explained by a marked increase in utilization among younger women. Furthermore, the impact of socioeconomic and geographic determinants on PHC utilization varied in magnitude and direction between groups of the population. Increasing utilization among older men was associated with higher SES and in female groups associated with lower SES. Defined groups of becoming high utilizers were all female and were all associated with low SES and residence in urban areas, these associations getting stronger with falling age. Likely there are several plausible explanations behind the unequal distribution of PHC utilization. Possible contributing factors are changing health care seeking behavior among young adults and the increasing importance of health literacy in the patient choice system. To get a more complete understanding of how trends in PHC utilization affect equity in health care, further studies should also include quality aspects of PHC utilization, such as continuity of care, and equity aspects of secondary health care utilization.

Availability of data and materials

The trajectory datasets generated and analyzed during this study are available from the corresponding author upon reasonable request.

Change history

18 march 2024.

A Correction to this paper has been published: https://doi.org/10.1186/s12913-024-10858-8

Abbreviations

General practitioner

Primary health care center

• Primary health care

Socioeconomic status

WHO & UNICEF. A vision for primary health care in the 21st century: towards universal health coverage and the Sustainable Development Goals. Geneva: World Health Organization; 2018.

Starfield B. Primary care and equity in health: the importance to effectiveness and equity of responsiveness to peoples’ needs. Humanit Soc. 2009;33:56–73.

Article   Google Scholar  

Anell A, Glenngård AH, Merkur S. Sweden health system review. Health Syst Transit. 2012;14(5):1–159.

PubMed   Google Scholar  

van Doorslaer E, Masseria C, Koolman X, Group OHER. Inequalities in access to medical care by income in developed countries. CMAJ. 2006;174(2):177–83.

Article   PubMed   PubMed Central   Google Scholar  

Devaux M. Income-related inequalities and inequities in health care services utilisation in 18 selected OECD countries. Eur J Health Econ. 2015;16(1):21–33.

Article   PubMed   Google Scholar  

Beckman A, Anell A. Changes in health care utilisation following a reform involving choice and privatisation in Swedish primary care: a five-year follow-up of GP-visits. BMC Health Serv Res. 2013;13:452.

Sveriges Kommuner och Regioner [Swedish Association of Local Authorities and Regions]. Ekonomi och verksamhetsstatistik [Statistics on economy and organization]. Available from: https://skr.se/skr/halsasjukvard/ekonomiavgifter/ekonomiochverksamhetsstatistik.46542.html . Accessed 17 Feb 2023.

Gaynor M, Ho K, Town RJ, National Bureau of Economic R. The industrial organization of health care markets. Cambridge: National Bureau of Economic Research; 2014. 62 pages: illustrations (black and white).

Book   Google Scholar  

Fotaki M. Is patient choice the future of health care systems? Int J Health Policy Manag. 2013;1(2):121–3.

Anell A. The public-private pendulum–patient choice and equity in Sweden. N Engl J Med. 2015;372(1):1–4.

Article   CAS   PubMed   Google Scholar  

Anell A. Översyn av primärvårdens utveckling efter införandet av hälsoval Skåne [Review of primary health care development following the choice reform in Skåne]. Lund: Rådet för KommunalEkonomisk Forskning och Utbildning (KEFU); 2016.

Google Scholar  

Riksdagen [Government of Sweden]. Lag om valfrihetssystem [Act on Freedom of Choice in the Public Sector]. SFS. 2008;2008:962.

Riksrevisionen [The Swedish National Audit Office]. Primärvårdens styrning – efter behov eller efterfrågan? [Primary health care governance – based on need or demand?]. 2014.

Ellegård L, Dackehag M. Competition, capitation, and coding: do public primary care providers respond to increased competition? CESifo Econ Stud. 2019;65(4):402–23.

Isaksson D, Blomqvist P, Winblad U. Free establishment of primary health care providers: effects on geographical equity. BMC Health Serv Res. 2016;16:28.

Regeringen [Government of Sweden]. Effektiv vård [Efficient care]. In: Utredningar SO, editor. Stockholm: Elanders Sverige AB; 2016. p. 793.

Maun A, Nilsson K, Furåker C, Thorn J. Primary healthcare in transition–a qualitative study of how managers perceived a system change. BMC Health Serv Res. 2013;13:382.

Sorensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12:80.

Agerholm J, Bruce D, Ponce de Leon A, Burström B. Equity impact of a choice reform and change in reimbursement system in primary care in Stockholm County Council. BMC Health Serv Res. 2015;15:420.

Myndigheten för vård- och omsorgsanalys [The Swedish Agency for Health and Care Services]. Vem har vårdvalet gynnat? [Who benefited from the choice reform?]. Available from: https://www.vardanalys.se/rapporter/vem-har-vardvalet-gynnat/ . Accessed 17 Feb 2023.

Myndigheten för vård- och omsorgsanalys [The Swedish Agency for Health and Care Services]. Vårdval och jämlik vård inom primärvården - En jämförande studie mellan tre landsting före och efter vårdvalets införande [Chioce and equality in primary health care]. Available from: https://www.vardanalys.se/rapporter/vardval-och-jamlik-vard-inom-primarvarden/ . Accessed 17 Feb 2023.

Nagin DS, Odgers CL. Group-based trajectory modeling in clinical research. Annu Rev Clin Psychol. 2010;6:109–38.

Statistiska centralbyrån [Statistics Sweden]. Häsloräkenskaper 2020 [Health accounts 2020]. Available from: https://www.scb.se/hitta-statistik/statistik-efter-amne/nationalrakenskaper/nationalrakenskaper/halsorakenskaper/pong/statistiknyhet/halsorakenskaper-2020/ . Accessed 17 Feb 2023.

Sveriges Kommuner och Regioner [Swedish Association of Local Authorities and Regions]. Kommungruppsindelning [Classification of municipalities]. Available from: https://skr.se/tjanster/kommunerlandsting/faktakommunerochlandsting/kommungruppsindelning.2051.html . Accessed 17 Feb 2023.

Nagin DS. Analyzing developmental trajectories: a semiparametric, group-based approach. Psychol Methods. 1999;4(2):139–57.

Nguena Nguefack H, Pagé M, Katz J, Choinière M, Vanasse A, Dorais M, et al. Trajectory modelling techniques useful to epidemiological research: a comparative narrative review of approaches. Clin Epidemiol. 2020;12:1205–22.

Svereus S, Kjellsson G, Rehnberg C. Socioeconomic distribution of GP visits following patient choice reform and differences in reimbursement models: evidence from Sweden. Health Policy. 2018;122(9):949–56.

Myndigheten för vård- och omsorgsanalys [The Swedish Agency for Health and Care Services]. Besök via nätet [Virtual visits]. Available from: https://www.vardanalys.se/rapporter/besok-via-natet/ . Accessed 17 Feb 2023.

Socialstyrelsen [The Swedish National Board of Health and Welfare]. Psykisk ohälsa bland unga [Mental health issues in young people]. Available from: https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/artikelkatalog/ovrigt/2013-5-43.pdf . Accessed 17 Feb 2023.

Carlsson AC, Wandell P, Osby U, Zarrinkoub R, Wettermark B, Ljunggren G. High prevalence of diagnosis of diabetes, depression, anxiety, hypertension, asthma and COPD in the total population of Stockholm, Sweden - a challenge for public health. BMC Public Health. 2013;13:670.

Lundin A, Forsell Y, Dalman C. Mental health service use, depression, panic disorder and life events among Swedish young adults in 2000 and 2010: a repeated cross-sectional population study in Stockholm County, Sweden. Epidemiol Psychiatr Sci. 2018;27(5):510–8.

Wandell P, Carlsson AC, Wettermark B, Lord G, Cars T, Ljunggren G. Most common diseases diagnosed in primary care in Stockholm, Sweden, in 2011. Fam Pract. 2013;30(5):506–13.

Forslund T, Kosidou K, Wicks S, Dalman C. Trends in psychiatric diagnoses, medications and psychological therapies in a large Swedish region: a population-based study. BMC Psychiatry. 2020;20(1):328.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Avby G, Kjellström S, Andersson BM. Tending to innovate in Swedish primary health care: a qualitative study. BMC Health Serv Res. 2019;19(1):42.

Svendsen MT, Bak CK, Sorensen K, Pelikan J, Riddersholm SJ, Skals RK, et al. Associations of health literacy with socioeconomic position, health risk behavior, and health status: a large national population-based survey among Danish adults. BMC Public Health. 2020;20(1):565.

Andersen RM. Revisiting the behavioral model and access to medical care: does it matter? J Health Soc Behav. 1995;36(1):1–10.

Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res. 1991;26(1):53–74.

CAS   PubMed   PubMed Central   Google Scholar  

Download references

Acknowledgements

The authors would like to thank Patrick Reilly for language revision.

Open access funding provided by Lund University. This work was funded by the Research council of Region Blekinge, Sweden. The funding body was not involved in the study design, collection, management, analysis, and interpretation of data or in writing of the report.

Author information

Authors and affiliations.

Department of Clinical Sciences, Faculty of Medicine, Lund University, Lund, Sweden

Hannes Kohnke

Bräkne-Hoby Vårdcentral, Parkvägen 4, Bräkne-Hoby, 370 10, Sweden

Blekinge Centre of Competence, Region Blekinge, Vårdskolevägen 5, Karlskrona, 371 41, Sweden

Andrzej Zielinski

Department of Clinical Sciences, Center for Primary Health Care Research, Lund University Clinical Research Centre (CRC), Box 50332, Malmö, 202 13, Sweden

Anders Beckman & Henrik Ohlsson

You can also search for this author in PubMed   Google Scholar

Contributions

HK, AB and HO contributed to the concept, design, data analysis and interpretation. HK wrote the main manuscript with contributions from AZ, AB, and HO. All authors participated in the review and approved the final version of the manuscript.

Corresponding author

Correspondence to Hannes Kohnke .

Ethics declarations

Ethics approval and consent to participate.

The study was approved and the Informed consent was waived by The Regional Ethical Review Board in Lund university. (Dnr 2019-00168). All methods were performed in accordance with relevant guidelines and regulations.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: errors in Table 1 were corrected.

Supplementary Information

Additional file 1: additional table 1..

Trajectory fit statistics. Additional Table 2a. Trajectory-group characteristics at baseline for males. Numbers indicate percent (%) of subgroup total if not otherwise specified. Additional Table 2b. Trajectory-group characteristics at baseline for females. Numbers indicate percent (%) of subgroup total if not otherwise specified. Additional Table 3a. Bivariate logistic regression on male trajectory-groups showing odds ratios (ORs) of change in utilization due to differences in predisposing and enabling factors. Additional Table 3b. Bivariate logistic regression on female trajectory-groups showing odds ratios (ORs) of change in utilization due to differences in predisposing and enabling factors.

Additional file 2: Additional Figure 1.

Trajectory analysis datasets. Trajectory analysis output showing trajectories of primary health care utilization between 2007–2017. On the Y-axis, the number of annual GP visits per individual are categorized by 4 utilization-groups corresponding to; 0 visits, 1 visit, 2–3 visits, and more than 3 visits.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kohnke, H., Zielinski, A., Beckman, A. et al. Trajectories of primary health care utilization: a 10-year follow-up after the Swedish Patient Choice Reform of primary health care. BMC Health Serv Res 23 , 1294 (2023). https://doi.org/10.1186/s12913-023-10326-9

Download citation

Received : 27 February 2023

Accepted : 14 November 2023

Published : 23 November 2023

DOI : https://doi.org/10.1186/s12913-023-10326-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Delivery of health care
• Health care utilization
• Health equity
• Health policy
• Patient choice
• Privatization

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]