Comments to the Equitable Data Working Group from Leaders of State Data Collaboratives

Denice Ross

U.S. Chief Data Scientist

Dear Ms. Ross:

Thank you for this opportunity for state leaders to respond to the request for information (87 FR 54269) on equitable data engagement and accountability from the White House Office of Science and Technology Policy (OSTP) and the Subcommittee on Equitable Data of the National Science and Technology Council. We are excited to share the following information on the vision, value, projects, and products of an emerging network of multistate data collaboratives.

3. What resources, programs, training, or other tools can facilitate increased data sharing between different levels of government (Tribal, territorial, local, State, or Federal) related to equitable data?

Background

Multi-state data collaboratives are developing as professional affinity networks above and beyond the basic technology and tools necessary for state agencies to share relevant data within and across state boundaries. With support from foundations and other funding, an administrative entity (the National Association of State Workforce Agencies) and a platform and training organization (the Coleridge Initiative), the collaboratives are empowering member state agencies to:

• Identify shared research questions that are critical to a thriving society;
• Build cross-sector (e.g., workforce, education, human services, corrections) innovation sandboxes to explore how research might address those questions; and
• Accelerate development and scaling of local and timely insights and data products that, if they choose:
  • Rely on both in-state and out-of-state administrative data sources; and
  • Can be shared and scaled.

Multi-state data collaboratives provide state agencies:

• Access to ideas, practices, and relationships with more states, through a structured network; 
• Streamlined data access through common agreements to enable both regional and national project partnerships;
• Access to project revenue streams;
• Better, more effective engagement with federal agencies and policymakers; and
• Most importantly, more frequent and rapid data product development leading to better policymaking and practice.

Multi-State Data Collaboratives

As the administrative organization supporting the network of multistate data collaboratives, the National Association of State Workforce Agencies (NASWA) works (currently under a foundation grant) as an honest broker across sectors (e.g., workforce, higher education, human services, corrections) to support:

• Collaborative governance, including facilitating the development and implementation of a final governance model, convening the regular meetings of the governing bodies, and identifying and convening advisory experts;
• Development of final bylaws and execution of an agreement (MOU) among states and the administrative organization (NASWA) that sets out the conditions for common data projects;
• Outreach to and engagement of additional states; • Enhanced visibility to external research institutions and policymakers;
• Critical partnerships with higher education, human services, and other state associations to support state agency engagement; and
• Implementation of grant-funded opportunities for collaborative training and product development.

NASWA also provides expert support to the Coleridge Initiative, the entity providing the data platform and supporting advanced data analytics training. The Coleridge Initiative, which is also a non-profit entity, provides a secure remote access environment, the Administrative Data Research Facility (ADRF), which has been authorized to host both state and federal workforce and education data. The ADRF’s security protocols (user identity, password complexity and expiration, etc.) follow strict FedRAMP guidelines and the ‘five-safes’ best practice. All user access activity is logged and monitored. The ADRF has provided secure access to over 300 confidential government datasets from 50 different agencies at all levels of government.

The Coleridge Initiative also built a complementary Applied Data Analytics (ADA) program to train government employees to perform applied data analysis with confidential individual-level data generated through administrative processes and extensive project-focused work. The aim of the ADA program is to help agencies tackle important policy problems by using modern computational and data analysis methods and tools. It has demonstrated the potential value of working with individual-level data across agency and jurisdictional lines for hundreds of government agency staff from federal, state and local agencies.

Since 2017, the Coleridge Initiative has partnered with over a dozen top universities and organizations to provide this professional development training to participants across more than 100 government agencies and 40 states, facilitating cross-agency and cross-state collaboration to address shared policy challenges. The guiding research questions that have framed the training and class projects have focused, and will continue to focus, on issues of equity. Courses have concentrated on exploring employment outcomes for Temporary Assistance for Needy Families (TANF) recipients, examining the unemployment to reemployment trajectories of Unemployment Insurance (UI) claimants, and investigating employment outcomes of foster care youth. Projects completed within those courses have investigated the interaction of race, ethnicity, and gender with program outcomes such as and employment stability and earnings.

Research Questions

Through the emerging multi-state data collaboratives, state agencies are working together to identify shared agency priority questions focused on specific populations, such as historically underserved, marginalized, and adversely affected by persistent poverty and inequality. An example of work that is already underway is the following:

• How can the UI program improve service delivery for underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography) based on the following measures?
  • Recipiency rate;
  • Percent of applicants who receive first UI payments;
  • Certification time lapse;
  • Percent of filers in low-filing industries;
  • Claimants who exhaust UI benefits.

Data from these measures will create more robust profiles of various claimants and challenges they may have within the UI program. One process outcome from the likelihood to file (i.e., recipiency rate) will be knowing more about claimants who do not make it to or through the front door. The profile of those claimants and their challenges may pinpoint certain processes within service delivery system that could be changed to make the filing experience easier and less complicated. Knowing the profiles of claimants who fail outside of the normal range for timely payments will also point to user or system challenges that could be changed or improved. Having robust profiles of claimants who exhaust or nearly exhaust UI benefits before reemployment also will yield valuable information about interventions for specific populations earlier in their UI journey/experience and if the profiles of claimants who have challenges are similar to claimants who exhaust benefits, it become predictive information that can be used for early Service Delivery interventions to improve user experience. It will also be highly instructive to know the sectors and occupations that have low-filing profiles. This data might point to interventions in specific industries.

For instance, national research has shown that unionized workers have higher likelihoods to file for UI than non-unionized workers. We also know that workers in home healthcare, childcare, and retail and hospitality are lower wage workers who have higher financial distress profiles and could benefit tremendously if eligible and are not currently filing for UI. Further examples of questions that the collaboratives have expressed a strong interest in exploring, or where they have already completed substantial work, include the following:

• How can we use data to support and improve connections and access to post-secondary education and training for UI claimants, and what are the associated employment and earnings outcomes, particularly for underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography)?
• What are the employment and earnings outcomes for post-secondary completers, by institution and major, focused on underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography)?
• What are the employment and earnings outcomes for public school students who enroll in alternative post-secondary programs?
• What is the value of post-secondary credential attainment for specific credentials aligned with occupations or industry sectors, and how does that look for underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography)?
• Who are post-secondary non-completers and what are the completion, employment, and earnings outcomes associated with alternative strategies to engage and support them in post-secondary programs, particularly for underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography)?
• What are the employment and earnings outcomes of registered apprenticeship and work-based learning strategies, particularly for underserved and marginalized populations (e.g., race and ethnicity, age, gender, ability, geography)?
• What are the nature and dynamics of the essential workforce (e.g., childcare workers, teachers) at the regional labor market level, including employment trajectories, and how do labor market, public policy and economic factors influence the workforce?
• What new measures of leaver wage progression for TANF recipients would be more effective for policy and practice decisions?
• How can the collaborative data-sharing ecosystem leverage cross-program and interstate social determinants of health (SDOH) data to support better service and outcomes for Medicaid customers?

4. What resources, programs, training, or other tools can expand opportunities for historically underrepresented scholars and research institutions to access and use equitable data across levels of government?

To broaden the reach of analytical capacity and data product relevance, the multi-state data collaboratives will be issuing Requests for Information (RFIs) to solicit interest from external researchers in developing data tools and products using state administrative records housed in the ADRF. This will create more streamlined and cost-effective opportunities for research institutions working with a sponsor state or states, including historically underrepresented scholars and research institutions, to access de-identified state administrative data and collaborate on data tools and products that address equity through workforce development, education and training, and other aspects of civil society. Especially for research institutions that have smaller capacity and budgets, this could be a game changer and create new and accessible entry points to participate in evidence-based policymaking activities.

As MDRC, a nonprofit, nonpartisan education and social policy research organization, has aptly described, information regarding individual workers’ earnings—a critical outcome measure—is maintained in state labor or employment security agencies for the UI program. “Currently, an evaluator must obtain this data from each state agency where it is housed. Because evaluations of governmental programs take place in multiple jurisdictions, the evaluator must spend considerable resources to ascertain each state’s requirements for data acquisition and then apply for the data. State statutes and administrative procedures govern access, and these procedures differ among the states. The significant costs of these data acquisition efforts are passed on to the federal agency and ultimately to taxpayers. And not all evaluators can successfully overcome the hurdles imposed by state agencies for access to the data—thereby limiting what policymakers know about program performance. The costs for the states to make the data available are also significant. While states typically charge for transmitting the data to the evaluation firm, costs associated with the negotiation for the data in the first place are likely not covered and thus are also passed on to state taxpayers.”¹

The Coleridge Institute’s ADA training courses detailed in the response above can build equity among scholars, analysts, and researchers by specifically recruiting participants who identify with under-represented demographic populations. An example of this approach was the Coleridge Initiative’s ADA course conducted in partnership with the United Negro College Fund and Excelencia, “Leveraging Big Data to Achieve Equity.” This 2021 workshop was an effort to increase data-based research capacity among institutions of higher education that serve a large share of Black and Latinx students.

6. What resources, programs, training, or tools can make equitable data more accessible and useable for members of the public?

Working through the emerging multi-state data collaborative structure, states have developed exploratory projects into collaborative data products that are informing practice among state, local, and regional leaders. Two core collaborative data products are the Unemployment to Reemployment Portal and the Multi-State Postsecondary Report, which have been replicated and expanded upon through collaborative efforts.

Unemployment to Reemployment Portal

Created by the State of Illinois, this interactive tool draws in weekly de-identified UI claims data, processed through the Coleridge Initiative’s secure ADRF environment, and allowed participating states to identify which local labor markets and equity groups were hit the hardest by COVID-19-induced layoffs, analyze the overall economic impact on those local economies, and document the recovery paths for these markets and groups. The Unemployment to Reemployment Portal has been replicated by the states of Indiana and Tennessee. The states of Texas and Ohio created UI data tools inspired by the Illinois tool. Planned expansions include the states of Wisconsin and Arkansas. Illinois is currently enhancing the portal to generate timely measures of reemployment activity.

Multi-State Postsecondary Report

Developed by the Kentucky Center for Statistics, the Multi-state Postsecondary Report is a dynamic tool that connects postsecondary graduates with in-state and out-of-state employment outcomes. Users can explore the connection between employment outcomes associated with credential level, major earned, and state origin. The Multi-state Postsecondary Report empowers states to better understand the employment outcomes of individuals who participated in their state’s education programs. Currently, the states of Kentucky and Ohio contribute data to the Multi-state Postsecondary Report.

Plans are underway to add data from Indiana and Tennessee. The states of New Jersey, Rhode Island, and Virginia recently received a Democratizing our Data Challenge grant through the Coleridge Initiative to bring the Multi-state Postsecondary Report to the Eastern States Longitudinal Data Collaborative. The state of Ohio also received a Democratizing our Data Challenge grant to incorporate workforce credentials and data on postsecondary “non-completers” into the dashboard.

On the Horizon – New and Emerging Projects, Products, and Priorities

• Illinois and Missouri – Study of turnover in the early childcare and education workforce
• New Jersey – Developing a K-12 data model for the ADRF
• Indiana – “Return on Credential Investment” app
• Southern Regional Data Collaborative – Recently established four working groups to focus their efforts on developing collaborative projects and products. The four priority topics, which they are viewing through an equity lens enhanced by equity data, are:
  • What is the value of post-secondary credential attainment for specific credentials aligned with occupations or industry sectors?
  • Who are post-secondary non-completers and what are the completion, employment, and earnings outcomes associated with alternative strategies to engage and support them in post-secondary programs?
  • How can we use data to support and improve connections and access to post-secondary education and training for UI claimants and what are the associated employment and earnings outcomes?
  • How can we refine the Unemployment to Reemployment Portal to create additional timely measures of UI program participation and claimant and employer labor market activity?

Sincerely,

Midwest Data Collaborative (MWC):

George Putnam, Chair, MWC Interim Executive Committee, and LMI Director, Illinois Department of Employment Security
Anna Hui, Member, Interim Executive Committee, MWC, and Director of the Missouri Department of Labor and Industrial Relations
Coretta Pettway, Interim Executive Committee, MWC, and Assistant Deputy Director, Ohio Department of Job and Family Services

Southern Region Data Collaborative (SRDC):

Jessica Cunningham, Chair, SRDC Interim Executive Committee, and Executive Director, Kentucky Center for Statistics (KYSTATS)

Eastern States Longitudinal Data Collaborative (ESLDC):

Dana Brandt, Chair, ESLDC Interim Executive Committee, and Director, DataSpark, University of Rhode Island

¹ “Improving Access to Federal Data for More Efficient Evaluations,” Testimony of Gordon L. Berlin, President, MDRC, Before the Human Resources Subcommittee of the House Ways and Means Committee, March 20, 2012 (https://www.mdrc.org/publication/improving-access-federal-data-more-efficient-evaluations).