2007 IDHS Summary of Program Evaluation Findings - Updated

Helping Families. Supporting Communities. Empowering Individuals.

February 2007

Overview

As of October, 2006, 16 programs funded by the Illinois Department of Human Services (IDHS) had independent evaluations that were ongoing or completed during 2006. Through grant or GRF dollars, these evaluations were being funded annually at $3,134,717.

This report represents the second effort, following a report issued in February, 2006, to combine summaries of all of the IDHS evaluations in a single report. IDHS receives findings from evaluations in several ways. In most cases, the evaluations are part of federally-funded grant programs. In these cases, evaluation findings are reported in annual continuation applications to the funding agency, and the evaluator may issue a final evaluation report that is delivered to the funding agency. In larger studies, such as a study on license-exempt child care, the evaluation or research team may disseminate reports at conferences or make them available on their web sites.

The report presents, alphabetically by division, each evaluation in place or completed by the end of September, 2006. Each evaluation summary provides basic information about the grant program and the most recent evaluation findings at that point. Therefore, some information is preliminary, and researchers/evaluators have cautioned against widespread use of the information prior to final reporting. Sources for this information ranged from continuation applications to full evaluation reports. An appendix to this report is a table presenting basic information about the evaluations' administration and design. Please note also that the table in the appendix includes information about new grant programs with evaluations begun in October, for which there are not yet any findings. A second appendix is a table presenting information about evaluations organized by evaluator.

The Office of Grants Administration envisions the following uses for the information in this report:

  • Assisting in Departmental integration by providing information to program staff across the divisions about what programs are being evaluated and what those evaluations are finding.
  • A demonstration of the level of effectiveness of IDHS programs across the divisions-showing "what works."
  • A set of examples for program staff writing new grants to see how other evaluations have been structured and to refer to in demonstrating to funding entities the extent of IDHS experience with program evaluations large and small.
  • A source of information about evaluations that may direct the interested reader to the full continuation application or evaluation report.

Eventually, all continuation applications and evaluation reports will be available in the IDHS Electronic Grants Library, housed on the IDHS OneNet. Access to the Electronic Grants Library can be obtained by contacting the Office of Grants Administration.

Division of Alcoholism and Substance Abuse

Access to Recovery (ATR)

Data Value
DHS Contract Mgr Rex Alexander
Annual Grant $ $7,575,685

Evaluator

Name & Contact Info

IOTA
Annual Eval $ $240,000
Funding Source SAMHSA/CSAT
Eval Period 08/03/04 through 08/02/07

Evaluation activities were described in the Interim Summary of Project Data Collection and Reporting Activities, August 2006 - Revised.

Project Summary

The Illinois Department of Human Services, Division of Alcoholism and Substance (IDHS/DASA) is administering Illinois ATR - Pathways to Reentry and Recovery. Key partners in this initiative are Treatment Alternatives for Safe Communities (TASC) and the Great Lakes Addiction Technology Transfer Center (GLATTC). The program will see 8,000 probationers over three years in two targeted geographic areas, Cook County and Illinois' Fifth and Sixth Judicial Circuits and the 10 counties immediately adjacent to them. Probationers are targeted because as a group, they generate exceptionally high social and economic costs and have recovery and reentry needs that are particularly well-served by a network of recovery support services. The Illinois System for Tracking Access and Recovery Services (I-STARS) supports a real-time utilization management and service authorization system, which moves enrolled programs toward more cost-effective service provision and greater reliance on recovery support services. A group of treatment and recovery support providers, including faith-based organizations, are enrolled to provide vouchered services to Illinois ATR clients. Client choice among alternative service providers is the primary determining factor in the referral of these clients. TASC, Enhanced Clinical Solutions, Inc. (ECS), Family Guidance Centers, Inc. (FGC), and IOTA, Inc. conduct screening and referral, as well as client GPRA data collection. GLATTC leads a multi-tiered training, technical assistance and technology transfer process targeting the courts and clinical treatment and recovery support programs. Data in the seven required domains is collected at time of entry to ATR-support services, at specified points during service participation, and at time of discharge from ATR-supported services. I-STARS monitors points at which clients come due for data collection and permits IDHS/DASA to track reporting compliance and timeliness.

Description of Project Evaluation Activities

TASC, ECS, FGC, and IOTA, Inc. collect data at time of entry into ATR-supported services for clients screened as non-dependent and clients screened as being likely dependent (cohorts 1 and 2). Treatment provider staff collect baseline and status data for adult probationers who are in treatment who are eligible for referral to ATR-supported recovery supports services (cohort 3). TASC collects status and discharge data for all Cohort 2 and 3 clients following their discharge from treatment and while they remain actively enrolled in ATR-supported recovery supported services.

I-STARS has embedded within it tools that are administered to clients to obtain their perceptions of the process through which they are able to choose among alternative clinical treatment and recovery support service providers, the treatment and recovery support services that they received, the extent to which these services met their needs, the staff who provided these services, and the facilities at which these services were received. The versions of this client satisfaction tool vary according to point in time, and are administered at time of initial assessment and referral and at the time of each status GPRA while the client remains active in ATR supported services.

Relatively slow client enrollment during Year 1, due to delayed start-up, resulted in the project being substantially behind its client intake target at the end of the first year. A corrective action plan was implemented in response to this deficiency, beginning in the first quarter of Year 2. The steps taken as part of this plan have been extremely effective. The project is now at over 110% of the interim client enrollment target. This is an increase from the 76.9% intake coverage rate that was reported in May 2006 and the 48.6% intake coverage rate that was reported in January 2006. This steady increasing rate of coverage of the interim client enrollment target is a direct result of the changes that were made through the corrective action plan.

Staff of enrolling agencies attempt to interview clients within 30 days of enrollment in the project. An interim analysis was conducted of data from the 2,452 clients whose completed surveys were entered into ISTARS as of August 25, 2006. These ATR clients are 77.7% (1,905) male, 7.3% (180) of Hispanic/Latino ethnicity, 72.3% (1,774) African American, 20.1% (492) white, and have an average age of 37.8 years. There are no significant differences between these characteristics and those of the entire group of Illinois ATR positive screen clients.

Early Evaluation Findings

The majority of comparisons between client responses at enrollment and 30 days post-enrollment were statistically significant, each in a direction that would indicate client improvements.

  • There were statistically significant increases in the percentages of clients who reported being enrolled in school or training and being employed. These increases were evident for both part-time and full-time training and employment.
  • There were statistically significant increases in the percentages of clients who reported attendance at non-FBO (faith-based organization) self-help groups, FBO self-help groups, and other activities that support activity. There was also a statistically significant increase in the percentage of clients who reported interaction with family and/or friends who are supportive of recovery.
  • There was a statistically significant change in the pattern of client responses regarding who they sought out in times of trouble. There was a decrease in the percentage of clients who said they tended to turn to "no one," and increases in the percentages of clients who stated they turned to clergy members, family members or friends.
  • There were statistically significant reductions in reported average numbers of arrests, arrests for substance abuse-related offenses, and nights spent in jail.
  • There were statistically significant decreases among clients in average number of days for use of any alcohol, alcohol to intoxication (both 5 or fewer drinks), any illegal drugs, cocaine/crack, marijuana, and heroin. Over 80% of clients reported abstinence from alcohol at 30 days post-enrollment and over 85% reported abstinence from illegal drugs.

Adolescent Substance Abuse Treatment Coordination Initiative

Data Value
DHS Contract Mgr Danielle Kirby
Annual Grant $ $400,000

Evaluator

Name & Contact Info

Maria Bruni, Ph.D. and Beth-Anne Jacob, Ph.D.

Independent

Annual Eval $ $39,820
Funding Source SAMHSA/CSAT
Eval Period 07/01/05 through 06/30/08

Evaluation activities were described in a report prepared for a CSAT site visit on September 27, 2006, Summary of Baseline Open-Ended GPRA Responses.

Program Summary

The primary goal of the Illinois Adolescent Substance Abuse Treatment Coordination Initiative is to improve the capacity in the State of Illinois to provide effective, accessible, and culturally competent substance abuse treatment for youths and their families, through the statewide coordination and enhancement of youth treatment services. IDHS/DASA has hired a full-time Adolescent Coordinator and a full-time Workforce Development Specialist (Work Force Specialist). Furthermore, IDHS/DASA is in the process of entering into agreements with IDHS/DASA-licensed organizations to expand the use of evidence-based treatment (EBT) with Illinois adolescents. The Adolescent Substance Abuse Treatment Coordinator (Adolescent Coordinator) has begun the transition into overseeing the adolescent substance abuse treatment system and has begun to coordinate efforts with key staff in the mental health, child welfare, criminal justice and education systems. The Workforce Specialist will work with provider organizations, higher education organizations and training institutes to develop strategies to ensure that an adequate supply of competent workers enters and stays in the youth substance abuse treatment workforce. In addition, IDHS/DASA has selected organizations through a competitive RFA process to be trained and certified in the GAIN assessment by Chestnut Health Systems' Global Appraisal of Individual Needs (GAIN) Coordinating Center. Staff from these selected organizations will also receive training and implementation guidance through Chestnut's Lighthouse Institute in either Motivational Enhancement Therapy/Cognitive Behavior Therapy (MET/CBT5) or the Adolescent Community Reinforcement Approach, both developed and tested as part of the CSAT-funded Cannabis Youth Treatment study.

Evaluation Activities

The evaluation and data collection plan includes the administration of the following CSAT GPRA forms: CSAT Baseline and Follow-Up Training Satisfaction Survey, CSAT Baseline and Follow-Up Technical Assistance Satisfaction Survey and CSAT Baseline and Follow-up Meeting Satisfaction Survey.

Training on the GAIN, Adolescent Community Reinforcement Approach (ACRA), and Motivational Enhancement Therapy/Cognitive Behavior Therapy (MET/CBT) took place in April, 2006, with evaluation follow-up in May and June. The results of the training satisfaction surveys are presented in the Summary with divided responses and numerous recommendations for training improvements. A number of comments were critical of the GAIN tool itself.

Chicago Collaborative Initiative to End Chronic Homelessness

Data Value
DHS Contract Mgr Kelli Gage
Annual Grant $ $628,312

Evaluator

Name & Contact Info

Elizabeth Durkin, Ph.D. and John Lyons, Ph.D.

Northwestern University

Annual Eval $ $57,616
Funding Source SAMHSA/CSAT
Eval Period 9/30/03 through 9/29/06

Evaluation activities were described in a brief update provided by the evaluators.

Project Summary

The Chicago Collaborative Initiative to End Chronic Homelessness is proposing to establish a specialized Assertive Community Treatment (ACT) team linked with 59 tenant based Shelter Plus Care units to provide permanent housing and cost-effective and comprehensive services to single individuals who have a serious mental illness or substance use disorder or both, and who have been living on the streets or in shelters consistently for at least one year, or who have experienced four or more episodes of homelessness in the past three years. Approximately twenty-percent of the tenants (11 individuals) will be veterans. The project will target 22 of the most impoverished of Chicago's 77 communities on the South Side.

Evaluation Findings

Nine months of data have been collected on 33 clients. Results indicate clients have made significant improvement in the first nine months within the program in the functioning and strengths dimensions. More specifically, clients have significantly improved from intake to nine months on the substance abuse, employment/education functioning, residential stability, family strengths, vocational/educational strengths, spiritual/religious strengths, and talents/interests items. This suggests the ARCH program has helped clients to reduce their substance use and find and maintain housing. The results also suggest the program assisted clients in finding employment or educational opportunities. Finally, the results suggest that the ARCH program helped to identify and support client strengths such as vocational/educational abilities, spiritual/religious affiliation and talents or interests of the clients.

Screening, Brief Intervention and Referral to Treatment (SBIRT)

Data Value
DHS Contract Mgr Deborah Levi, Ph.D.
Annual Grant $ $3,500,000

Evaluator

Name & Contact Info

Richard E. Sherman, Ph.D.

IOTA, Inc.

&

Christy Scott, Ph.D.

Lighthouse Institute

Annual Eval $

$35,000 (IOTA)

$315,000 (Lighthouse)

Funding Source SAMHSA/CSAT
Eval Period 9/30/03 through 9/29/08

Evaluation findings are described in Interim Summary of Project Data Collection and Reporting Activities, August 2006, by IOTA, Inc., and Illinois Screening, Brief Intervention, Referral and Treatment Initiative Year 03 Semi-Annual Evaluation Report, April 2006 - October 2006, by Mark Foss, Ph.D., Belinda Willis, and Lisa Nicholson, Lighthouse Institute, Chestnut Health System.

Project Summary

The project provides substance use disorder screening, brief intervention, and referral to treatment (SBIRT) services within hospitals and community health clinics operated by the Cook County Bureau of Health Services (CCBHS). Three CCBHS general hospitals (Stroger, Provident and Oak Forest Hospitals) and three community health centers (Logan Square, Woodlawn and Near South) are participating in the project. Screening and Brief Intervention services are provided within CCBHS settings by SBIRT health counselors. TASC community care coordinators assist in linking hospitalized substance dependent patients to substance abuse treatment at 11 IDHS/DASA-licensed treatment providers and to their first ambulatory health care appointment. Through this project, Brief Treatment is being implemented as a new treatment modality for patients determined to be appropriate for this level of care. Key project partners also include the Great Lakes Addiction Technology Transfer Center (ATTC) and Treatment Alternatives for Safe Communities (TASC). Lighthouse Institute is conducting six-month post-screening follow-up interviews with at least 10% of the patients classified in the last three of these service categories.

Process Data

The Interim Summary provides data for the project as of August, 2006 for screening only, brief intervention, brief treatment and referral to other treatment service. A total of 41,204/42,720 (actual/target; 96.5% of target) participants received screening and feedback, 6,074/4,875 (actual/target; 124.6%) received brief intervention services, 837/1,310 (actual/target; 63.9% of target) participants received a referral to brief treatment, and 1,776/1,345 (actual/target; 124.6%) received a referral to more traditional levels of substance abuse treatment. Through August, 49,891 patients had received SBIRT services, which is 99.3% of the cumulative target (50,250) to date. The only data element below the 80% threshold is the intake coverage rate for patient referrals to Brief Treatment. Some difficulties were experienced in generating referrals to Brief Treatment at a rate that was originally proposed. Strategies to increase Brief Treatment referrals were implemented and are showing the desired effect. Referrals are currently at nearly 70% of the interim target, up from 30% a year ago. It is anticipated that within the next six months the project should be at a minimum of 80% of the interim Brief Treatment referral target.

Outcome Data

The Year 03 Semi-Annual Evaluation Report provides information on the outcome evaluation. The outcome evaluation focuses on the changes in a sample of participants' functioning between intake and six months post-intake. Six-month follow-up interviews are required for 10% of patients receiving services. As of September 30, 2006, 95.6% of the 432 participants who were due for their six-month follow-up had completed their follow-up interview. The evaluation report summarizes the following data about project participants: demographics, education and economic, health and psychological status, health risks, criminal justice involvement, substance use.

Satisfaction with Services. Participants were asked eight questions to assess attitudes about the services provided. Among the 402 participants with follow-up data, 35 did not recall receiving any services. As a result, only 367 participants rated the services. The three groups (Brief Intervention, Brief Treatment, and Referral to Treatment) did not differ significantly on any individual item nor did the three groups differ significantly on their overall Satisfaction Scale scores. The participants were mostly satisfied to very satisfied with the services they received.

Outcomes: Abstinence, Days of Use and Psychological Problems. Participants in all three service groups showed a significant reduction in substance use. Substances that showed no change were typically substances rarely used by that specific group at intake. There was no change in days of depression; however, days of anxiety increased among participants in the Brief Treatment group. Interpretation of this result should wait for confirmation until further data are collected.

Strengthening Treatment Access and Retention - State Implementation (STAR-SI)

Data Value
DHS Contract Mgr Peggy Alexander
Annual Grant $ $325,000

Evaluator

Name & Contact Info

Richard E. Sherman, Ph.D.
Annual eval. $ $28,000
Funding Source SAMHSA/CSAT
Eval Period 9/30/06 through 9/29/09

Evaluation information is available in the new competitive application submitted March, 2006.

Project Summary

The Illinois Department of Human Services, Division of Alcoholism and Substance Abuse (IDHS/DASA) will administer a Strengthening Treatment Access and Retention - State Implementation (STAR-SI) project designed to improve rates of client access to and retention in publicly funded substance abuse outpatient treatment programs in Illinois. The Illinois STAR-SI Project will involve two provider networks: Year One rural based clients and Year 2 Chicago based clients. The urban STAR-SI network will consist of five providers in Cook County who receive the greatest number of referrals from the Screening, Brief Intervention, and Referral to Treatment (SBIRT) project (described above). Five providers serving clients in rural southeastern Illinois comprise the second network and are members of the Small and Rural Providers Association. Additional provider organizations will be recruited to each network during Year 3 of the project.

Currently over 55% of clients admitted to publicly funded outpatient services in Illinois receive no more than five hours of clinical services. The project will identify and implement strategies to 1) reduce waiting times to access treatment; 2) reduce the rate of appointment no-shows; 3) increase outpatient treatment admissions; and 4) improve rates of retention in treatment. IDHS/DASA will involve consultants from Network for the Improvement of Addiction Treatment (NIATx) in planning and training and will implement the STAR performance improvement program and the Plan-Do-Study-Act approach to the improvement of organizational processes. IDHS/DASA will focus on administrative systems issues and potential modifications that can be made to support the positive changes identified in each of the peer-to-peer networks. These changes may include licensure waivers, staff training, and policy or procedure changes.

Evaluation Summary

As a project with the intent of improving organizational processes, this evaluation will focus on the following domains: 1) the degree to which the project adheres to its plan to increase and improve elements of the Illinois outpatient treatment system; 2) barriers to the planned strategies and improvements; 3) the identification and implementation of organizational and infrastructure changes; 4) perceptions of key stakeholders; and, 5) system and client level outcomes of implemented changes. Evaluation activities will be both qualitative and quantitative in nature. A key component of the evaluation will be embedded in the project design and will take place at the provider and SSA levels. This component, to be implemented with assistance from NIATx consultants, is the "Rapid Cycle Testing" which is one of NIATx's five key principles to overcoming barriers to process improvement. In a "nutshell," this approach consists of collecting baseline data, implementing process changes for a short period of time, evaluate or assess outcomes, and keep the change in place or try something else.

Another component of the process evaluation will involve qualitative data collection and analysis. Interviews with DASA staff, treatment provider staff and NIATx and other training staff will be employed along with analysis of project documentation and materials to answer questions related to domains 1-4. The majority of the outcome evaluation-namely, assessment of the results of the implementation of process changes-will involve quantitative data collection and analysis. Data from the abbreviated GPRA tool required of STAR-SI projects as well as provider-specific documentation, client satisfaction surveys and DASA administrative data (hereafter called DARTS) will be used to answer questions related to domain 5.

At the date of this report, the STAR-SI project is in start-up phase and there are not yet any evaluation findings.

Targeted Capacity Expansion - HIV East St. Louis, IL

Data Value
DHS Contract Mgr Kelli Gage
Annual Grant $ $500,000

Evaluator

Name & Contact Info

Cheryl Mejta, Ph.D. Governors State University
Annual Eval $ $27,316
Funding Source SAMHSA/CSAT
Eval Period 10/01/01 through 9/30/06

Evaluation activities were described in Bi-Annual Report submitted to CSAT covering the period April 1 to September 29, 2006. The project's final report is due in February, 2007.

Project Summary

The project expands availability of Methadone Maintenance treatment to persons at high risk for HIV and/or hepatitis C (HCV) at the Cornell Interventions East St. Louis Clinic in East St. Louis, Illinois. The project also expands availability of treatment for mentally ill substance abusers (MISA) through increased psychiatric care.

Process Evaluation

During this evaluation period, 34 patients were admitted into the TCE HIV/HCV project. Six month follow-ups were completed with 3 patients. No 12 months follow-ups were due during this period.

Intake: 34 Patients

A demographic description of patients includes gender, race/ethnicity, age, education, and employment status. Data was collected and reported regarding recent drug use frequency and type. Information was reported regarding mental health issues. This information includes the following. Patients (38%, n=13) reported that their use of alcohol or other drugs caused some or considerable emotional problems for them. The emotional difficulties encountered included: depression by 24% (n=8) of the patients, anxiety or tension by 35% (n=12) of the patients, and difficulty understanding or concentrating by 15% (n=5) of the patients. Legal problems were less frequent among patients with only 6% (n=2) of the patients reporting having been arrested within the past 30 days.

The primary risk behaviors reported by patients were injection drug use and unprotected sexual contacts. Injection drug use within the past 30 days was reported by 35% (n=12) of the patients. O f these patients, 33% (n=3) reported that they shared their "works" with other people. Sexual activity within the past 30 days was reported by 59% (n=20) of the patients. Unprotected sexual contacts were reported by 70% (n=14) of these patients.

6 and 12 Month Follow-ups:

During this quarter, a total of 3 patients completed the 6 month follow-up. There were too few patient follow-ups completed to analyze behavioral changes between intake and 6 months.

No 12-month patient follow-ups were scheduled during this reporting period.

Collection of 6 month and 12 month patient follow-up data has been problematic during this reporting period. To remedy this situation, a consultant was sent by CSAT to review our follow-up approach and to recommend strategies to improve our follow-up completion rates. Additionally, we are attempting to locate patients using public records including information from the criminal justice system's database. During the remaining 6 months of the project, patient follow-up has become the focus of all staff efforts.

Current Activities

We are concluding the five-year TCE HIV/HCV project. We are completing our analyses of the data and writing the final report.

Division of Community Health and Prevention

Healthy Families Illinois

Data Value
DHS Contract Mgr Denise Simon
Annual Grant $ $8,450,000

Evaluator

Name & Contact Info

Julie Crouch

Office of Sponsored Projects Northern Illinois University

Annual Eval $ $333,000
Funding Source GRF
Eval Period 7/1/00 - 6/30/06

Evaluation information was available from the substantial report Healthy Families Illinois: Evaluation Findings, September, 2006.

Program Summary

Healthy Families Illinois (HFI) is a part of a national initiative, Healthy Families America, that seeks to improve parent-child interactions, including preventing child abuse and neglect and promoting optimal child development. HFI provides intensive home visitation services that are culturally relevant, promote connections with community resources, and provide support and education related to positive parenting practices. HFI provides extensive training to its home visiting staff through a training institute provided by the Ounce of Prevention Fund.

Summary of Key Findings

HFI services are associated with promotion of optimal parent-child outcomes during the first two years of life.

For example, caregivers receiving HFI services exhibit significant improvement in their ability to promote their children's social, emotional, and cognitive growth during the first six months of life. Parents receiving all other usual services who were not able to receive HFI services because of limited availability did not exhibit these early improvements in their caregiving skills.

At one year, HFI parents displayed greater acceptance and tolerance of their growing children's challenging behaviors.

At two years, families receiving HFI services offered their children a wider array of materials to stimulate intellectual development.

Findings from this evaluation demonstrate that HFI programs are successful at engaging and retaining parents at high risk for problems in parenting. Further, families with the highest risk for problems in parenting demonstrated the greatest benefit across time from HFI's prevention services.

The evaluators point to limitations to be considered when interpreting their study's findings. First, although the comparison groups were equivalent "in terms of risk and motivation to receive services," assignment to the two groups was not random. Second, participation in the evaluation itself was voluntary. "Thus, the extent to which the present findings generalize to families who did not volunteer remains unknown."

With these limitations in mind, findings from this large-scale, longitudinal evaluation may be interpreted as evidence that HFI's services are associated with improved parent-child outcomes, including reduced risk for child abuse and associated problems in parenting.

Illinois Steps AHEAD (GEAR UP)

Data Value
DHS Contract Mgr Ivonne Sambolin
Annual Grant $ $3,500,000

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Center for Prevention Research and Development

University of Illinois at Urbana-Champaign

Annual Eval $ $300,000
Funding Source US Department of Education
Eval Period 10/1/05 - 9/30/11

Program Summary

Illinois Steps for Attaining Higher Education through Academic Development (AHEAD), provides early intervention educational services and post-secondary educational scholarships for priority middle and high school students. Program components include increased parent involvement, creative and engaging academic support for students, career exploration programs, college preparation and increased collaboration with local schools. The program is implemented through leveraging resources within a successful infrastructure: the Illinois Department of Human Services Teen REACH Initiative and State Board of Education 21st Century Community Learning Centers (CCLCs).

Illinois Steps AHEAD is overseen by a State-level advisory board that includes leaders from state agencies and organizations that are integral to academic and school reform, higher education, human services, professional development, and evaluation.

Evaluation

As of December 15, 2006, no evaluation activities or results were reported.

Substance Abuse Prevention

Data Value
DHS Contract Mgr Kim Fornero
Annual Grant $ $20,277,712

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Center for Prevention Research and Development (CPRD)

University of Illinois at Urbana-Champaign

Annual Eval $ $1,062,888
Funding Source SAMHSA
Eval Period 7/1/90 - 6/30/06

Brief evaluation information was available through the Strategic Prevention Framework continuation application submitted in March, 2006. No report was provided specific to the Block Grant portion of the evaluation.

Program Summary

The substance abuse prevention evaluation covers programs funded by the Illinois Substance Abuse Prevention and Treatment Block Grant and the Strategic Prevention Framework - State Incentive Grant.

The Block Grant specifies that 20% of funds will be spent on prevention. The Bureau of Substance Abuse Prevention has a three tier system consisting of Comprehensive programs, the InTouch network and Statewide programs.

  • Comprehensive Programs: The Comprehensive program grants are awarded to not-for-profit organizations/governmental entities. These agencies deliver prevention services at the community level.
  • InTouch Network: The Illinois Network to Organize the Understanding of Community Health (InTouch) are also awarded to the types of organizations listed above. The InTouch network covers regions of the state called Prevention Service Areas (PSAs). All 102 counties in the State are served by an InTouch office.
  • Statewide Programs: The Statewide programs receive funding to serve large geographic areas or defined target populations.

The Illinois Strategic Prevention Framework State Incentive Grant (SPF SIG) provides a platform for leadership and coordination at the state level while supporting partnership and ownership at the community level. At the state level, policy makers and others interested in better coordinated systems of substance abuse prevention come together to collect and analyze relevant data from many sources to create a comprehensive picture of who is affected by what substance abuse problems and where. A strategic plan for the state will be based upon the priorities revealed through the needs and resources, and strategies will be proposed to achieve outcomes that produce reductions in rates of underage drinking, substance abuse onset and substance abuse progression. Evaluation data will continuously be gathered to inform how to implement strategies that are grounded in science but shaped and molded by experience.

Over $1.9 million annually will be directed to communities throughout Illinois. With sufficient support and technical assistance, these communities will thoroughly assess their own community to identify the most relevant and pressing needs, engage a broad spectrum of community representatives and implement programs, policies and practices that are both proven and appropriate for the cultures, history and context of that community. Finally, efforts will be made at the state and local level to identify creative ways to sustain those that work and terminate those that do not.

Process Information

State level evaluation activities mirrored project progress. The Statewide Advisory Council did not convene and a strategic plan was not completed, so neither of these components was part of the evaluation this reporting period.

State Epidemiological Outcomes Workgroup (SEOW) activities were tracked and monitored and the Epidemiological Profile development process was documented. Project Staff and SEOW meetings and the attendance by agency and position were continually monitored and rosters maintained. In addition, all meeting agendas, minutes, handouts and other documents distributed at meetings were recorded and filed. Project accomplishments and limitations were also informally captured in a journal comprised of evaluator observations.

Two versions of Key Informant Survey were conducted with the SEOW members and project staff. However, the survey was not implemented with the Statewide Advisory Council because the group did not convene during this reporting period. The SEOW member survey was conducted in October and November of 2005 and the project staff survey was executed in December of 2005. No description of the results was provided.

Center for Prevention Research and Development (CPRD) continues to monitor the progress of the project relative to project goals through the Goals, Objectives, Activities, Milestones (GOAMS) analysis relying on the documents analysis and interviews. However, while we continue to apply the GOAMS framework to track project progress, due to funding constraints which arose after the proposal was drafted, we have found it necessary to forego the eGOAMS initiative.

The Strategic Plan Quality Index (SPQI) was originally going to be a part of the year one evaluation but has not been implemented. The SPQI will be used when the Strategic Plan is completed.

Outcome Evaluations

No outcome information was provided.

Targeted Intensive Prenatal Case Management

Data Value
DHS Contract Mgr Beverly English
Annual Grant $ $4,693,400

Evaluator

Name & Contact Info

James W. Masterson

Hamilton Bell Associates

Annual Eval $ $168,600
Funding Source GRF, MCH Block Grant & Healthy Start
Eval Period 7/1/01 - 6/30/06

Evaluation findings are available in the substantial Hamilton Bell Associates report, HBA Annual Report (FY 06): "Taking the TIPCM Project to the Next Level," July 2006.

Project Summary

Ten providers are contracted to provide Targeted Intensive Prenatal Case Management (TIPCM) to high risk pregnant women residing in geographic areas of Illinois identified by DHS as having higher costs of infant care. During FY 2006, DHS provided start-up monies for three additional sites in Cook and Will Counties, although six additional sites had been forecast in the previous year. Patient enrollment for the new Agencies was scheduled to begin on 7/1/2006.

The program uses outreach and incentives to engage eligible clients and keep them engaged in case management services. Case managers have caseloads of no more than 40 and have at least two face-to-face contacts with each client each month. At least half of the contacts take place in the pregnant woman's home. Services include a risk assessment, care plan, referrals made and referrals kept, including for mental health, substance abuse treatment, smoking cessation services and domestic violence intervention services, prenatal visits. Some project funds may be used to purchase services if no other third party payment source can be identified. Data is collected in the Cornerstone management information system. Provider agencies compete for outreach performance bonuses based on growth in the number of high-risk women who enroll in program services and the number of live births that exceed five pounds, eight ounces. There are now six providers in Cook County and seven in downstate Illinois.

Process Evaluation

Evaluators examined performance of the project's core services: outreach and intensive case management.

Regarding outreach, the TIPCM project documented an 18.3% increase in the enrollment of hard to reach, high risk pregnant women in 2005. Three of the 10 agencies were not meeting targets (the other seven over-produced, resulting in the project's meeting its overall targets) and underwent Focused Review and CQI activity in FY06. DHS and the agencies agreed to discontinue the performance measure related to the percentage of high risk women identified who were successfully enrolled in the program. Under consideration for new performance measures are Percent of Women Enrolled in First Trimester of Pregnancy and/or Average Month of Prenatal Care at Enrollment.

Regarding intensive case management, there were mixed results in meeting performance objectives. Agencies showed "significant improvement" in performance, but overall the project still fell short of targets. Given the significant challenges of the target population, as a result of CQI activities the project decided to adjust the performance targets to be more realistic. Three under-performing agencies are targeted for further Focused Reviews and CQI activity.

Preliminary Findings

HBA finds that:

  1. Based on 2000-2002 data, the TIPCM Program is effective. However, "HBA was not provided access to 2003 and 2004 Vital Records Data required" to provide a more current finding.
  2. The TIPCM Program is "efficient."
  3. The TIPCM Program finds and enrolls the "hard to reach, high risk" pregnant population in the Targeted Communities.
  4. The TIPCM Program has a CQI Program that has demonstrated effectiveness in identifying and resolving barriers to improving Agency performance in implementing the Core Services and meeting other DHS expectations (i.e., data entry into Cornerstone).
  5. The TIPCM Program is making steady progress toward achieving its stated Program Objectives.
  6. DHS and the participating TIPCM Agencies have successfully collaborated in the identification of areas needing improvement in FY 2007 and has successfully "institutionalized" a CQI Program that seeks to identify improvement wherever and whenever possible.

HBA provides five recommendations to improve services and address identified program weaknesses.

Teen-Parent Family Services

Data Value
DHS Contract Mgr Denise Simon
Annual Grant $ $350,000

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Center for Prevention Research and Development

University of Illinois at Urbana-Champaign

Annual Eval $ $75,000
Funding Source DHHS/Office of Adolescent Pregnancy Programs
Eval Period 10/01/05 - 9/30/10

Evaluation information was available from the Adolescent Family Life Care Demonstration Project End of Year Report.

Program Summary

The Teen Parent Family Services Project is a five year federally funded demonstration project (Adolescent Family Life Care Demonstration) designed to serve the families of the TANF teen mothers who are participants of Teen Parent Services - Central located at 2014 South Michigan Avenue in Chicago. This project enhances and expands upon the existing services of DHS Teen Parent Services - Central. The project offers services to the siblings (age 7-18) of the teen mother, the parents of the teen mother or mother's long term boyfriend, the father of the baby, and the paternal grandparents of the baby. The project is designed to help enrolled families with the necessary support to develop and strengthen family unity. We accomplish this by connecting each family member to community resources and involving them in a variety of programs, with crafted instruments for various uses, such as familial interests, individual interests, familial needs, individual needs and individual and familial problem areas.

Evaluation Activities

Dr. Peter Mulhall, Mr. Ernie Hill, and the staff at the Center for Prevention Research and Development at the University of Illinois have served as the project evaluator for the Illinois Department of Human Services Teen Parent Family Services demonstration project. Over the past year, the evaluators and project staff have met monthly to discuss the evaluation at both sites. The evaluators have worked with project staff in the development of an electronic database system that monitors participants and their family members as they enter the program and receive services. The database has a screening tool to assess the needs of the teen mother and child and has a series of process indicators capturing the programs and services they have participated in and completed. The evaluators and project staff have begun collecting the required surveys from teen mothers, and other surveys were developed or adapted by the evaluators for boyfriends, grandparents, and siblings. The evaluation database and data collection procedures are in place at both the intervention and the comparison sites, and once the sample sizes are large enough, evaluators will analyze both process and outcome data.

The evaluation approach used by CPRD will be the Context, Inputs, Process and Product Evaluation (CIPP) model that is widely used in the evaluation literature (Stufflebeam, 2003). The CIPP model begins by evaluating context of the setting, program, needs and priorities for the evaluation. Input evaluation assesses the basic structures, organization, resources, supports, staffing needed to implement the targeted programs and attain project goals and objectives. Process evaluation describes and captures planning, implementation and performance of services delivery and provides feedback for program improvement. Finally, products are the outcomes and impacts that we typically classify as short term, intermediate and long term, depending upon the target, magnitude and temporal direction of the outcomes as described in the program logic model which the evaluation team and program staff refined during the year.

Evaluation Findings

The evaluation is in too early a stage to provide any findings.

Division of Human Capital Development

No current evaluations are in place.

Division of Mental Health

Community Reintegration Collaborative

Data Value
DHS Contract Mgr Anderson Freeman
Annual Grant $ $399,976

Evaluator

Name & Contact Info

Lisa A. Razzano, Ph.D.

University of Illinois (Chicago)

Annual Eval $ $57,400
Funding Source SAMHSA - H79 SM57316-01
Eval Period 9/30/2005 through 9/29/2006

The status of the evaluation was described in the FY07 continuation application, submitted March 31, 2006, with supplemental information provided upon request from DMH in November, 2006.

Project Summary

The principal goal of the Community Reintegration Collaborative (CRC) is to increase jail diversion for criminal justice-involved individuals with mental illness in Cook County. The project is being implemented in two phases. The Planning Phase includes the participation of key stakeholders and will culminate with the development of a comprehensive strategic plan. This plan will inform the Implementation Phase, during which lessons learned from the planning process will be integrated into a coordinated, trauma-informed, evidence-based service delivery system for individuals with mental illness involved in the Cook County criminal justice system, especially those with the most intensive service needs. There are two goals for each phase of the program, with specific objectives related to each goal.

With collaborative partners, especially TASC, Thresholds, the Chicago Police Department and the Cook County Mental Health Court, DMH has the following goals:

Evaluation Activities

The project evaluator monitors the process and outcomes of both the planning and implementation phases of the two-part project. The evaluator for the CRC program remains University of Illinois at Chicago. However, the principal investigator was changed in the beginning of the year 1 grant period from Judith Cook, Ph.D. to Lisa Razzano, Ph.D. as a result of over commitment for Dr. Cook. Dr. Razzano works with Dr. Cook at the Center on Mental Health Research Services and Policy at the UIC.

Dr. Razzano has overseen evaluation planning and activities in the Planning Phase of the CRC program. Pre- and post-training inventories were administered to Strategic Planning Work Group members to measure the impact of training on their knowledge of ACT and IDDT models. Additionally, Dr. Razzano oversaw the evaluation planning process, which took place with input from CRC partners, Advisory Committee members, and Work Group members. As a result of this work, additional site-specific measures were identified and added to the evaluation measures for the program.

Dr. Razzano has been working with the University's Institutional Review Board (IRB) to obtain full-board approval for the evaluation component of the program. All key personnel (Drs. Freeman and Razzano, and Ms. Hamilton, the project manager) have completed IRB training at UIC. In addition, the project's interviewer, Ms. Suzanne Caulfield, has been hired and completed training.

The evaluation program manager (Marie Hamilton) and the evaluation interviewer (Suzanne Caulfield) have participated with Dr. Razzano in required training teleconferences convened by SAMHSA's Technical Assistance and Policy Analysis (TAPA) Center. The evaluators have worked closely with the TAPA Center to prepare for the full implementation of the project's evaluation activities. This has included obtaining databases, installation of software, and training relevant evaluation staff with use of these programs.

Evaluation Plan

DMH anticipates that efforts put forth in the implementation phase will provide encouragement for ongoing collaboration and systems integration to benefit justice-involved individuals with mental illness in Cook County, and produce the following results for CRC participants.

  • Increased access to trauma-informed case management and/or ACT and IDDT. Training was conducted by TAPA Center experts, on May 12, 2006.
  • Increased housing stability and standard of living. Living situation and relevant residential support services will be assessed at each of the three evaluation interviews and compared across time as well as within the context of other services received by program participants.
  • Decreased symptoms of mental health and substance use. It is expected that participants will demonstrate improvements in psychiatric functioning, measured for the GPRA interview using the Colorado Symptom Index and the Brief Psychiatric Rating Scale (BPRS). With regard to substance use, improvements in this outcome will be tracked using information collected in GPRA interviews and program interviews using a measure developed by TASC.
  • Decreased use of emergency psychiatric and substance abuse services. This outcome will be demonstrated using services data from TASC, Thresholds, and the Division of Mental Health, as well as information collected in the required GPRA evaluation interviews.
  • Decreased arrests and incarcerations. This outcome will be demonstrated using services data from the Chicago Police Department's Crisis Intervention Team (CIT) program, TASC, Thresholds, and the Cook County Data Link Project, as well as information collected in the required GPRA evaluation interviews.

System of Care - Chicago

Data Value
DHS Contract Mgr Tanya R. Anderson, MD
Annual Grant $

$2,000,000 in FY06

(varies each grant year)

Evaluator

Name & Contact Info

Annual Eval $ $185,000 in FY06
Funding Source SAMHSA/CMHS
Eval Period 11/01/04 through 9/29/08

John Lyons, Ph.D.

Mental Health Services and Policy Program

Northwestern University

Evaluation findings summarized below were available in the FY07 continuation application (March, 2006) and a project year-end summary provided in October, 2006.

Project Summary

The goal of System of Care Chicago (SOCC) is to implement a system of care based on the primary partnership established between DHS/DMH Metropolitan Child and Adolescent Network and the Chicago Public Schools. This effort is aimed at bringing together parents, youth, community networks and resources, along with the responsible child-serving agencies, which include local mental health centers, schools, healthcare providers, child welfare, substance abuse and juvenile justice, in order to meet the needs of children with a serious emotional disorder and their families. Family involvement and cultural competence are key issues that are addressed throughout the initiative. Evaluation tools are being designed to include child and family outcome data as well as findings related to the processes of system change itself. Service redesign is intended to blend evidence-based practices and non-traditional approaches and resources to address the full array of children's needs while identifying and enhancing the strengths of those families. Social marketing plays a key role in the dissemination of information regarding system of care development through quarterly newsletters, staff training, special events, and the development of communication pathways that decrease confusion and enhance collaboration.

Evaluation Activities

John S. Lyons, Ph.D. will continue to serve as the chief evaluator for the project. Crystal Jackson will remain as the evaluation coordinator. David Goodrich, Kertrell Deal, Steve Garcia and Jason Brennen have joined the NU evaluation staff for the project.

Overview

Evaluation staff continue to work closely with the SOCC Program Director, Clinical Director, mental health agencies' clinical supervisors, school principals and the SOCC school-based staff to further implement the local and national evaluation processes. Northwestern will also continue to communicate with Macro, the national evaluator, to assure that the site is maintaining integrity to the national research and evaluation goals.

Two hundred and thirty-two families have received services from SOCC since its inception. One hundred and thirty-five families have been enrolled since obtaining IRB approval of the evaluation process in November 2004. Fifty-seven of the families enrolled in services since November 2004 were closed as of December 31, 2005. Seventy-four families (47 boys and 27 girls) were successfully enrolled in the national evaluation as of December 31, 2005.

As of September 30, 2006 two hundred and ninety-six families have received services from the SOCC program since its inception in September 2002. Of those, 200 families have been enrolled and served since obtaining IRB approval of the evaluation process in November 2004. One hundred fourteen of the families enrolled since November 2004 have been closed to services as of September 30, 2006. One hundred and thirty-four families (i.e., 81 boys, 53 girls) have been successfully enrolled in the national evaluation as of September 30, 2006. Total enrollment for the FY5 (October 1, 2005 - September 30, 2006) is 91 (i.e., 56 boys, 35 girls). As of September 30, 2006 the overall number (FY2-FY5) of children presently receiving services is 94 (i.e., 59 boys, 35 girls). Total number of families enrolled in the national evaluation for during FY5 (ending on September 30, 2006) is 72 (i.e., 41 boys and 31 girls). Total enrollment in the national evaluation reflects an increase from the previous fiscal year but still fails to meet the required number of participants mandated by SAMHSA as only 78% of the enrollment goal was achieved. The failure to meet the enrollment goal is largely attributable to multiple staff changes and unfilled positions at each of the school sites over the course of the fiscal year.

Meetings

Members of the evaluation team continue to regularly participate in each the following meetings: Operations team, Clinical Services Team, Cultural Inclusion Workgroup and the combined meetings of the Comprehensive Care Coordinators (CCC) and Family Resource Developers (FRD). The evaluation team will also continue to facilitate the meeting of the Community Evaluation Team (CET). The Evaluation Coordinator has maintained monthly communication with the Macro liaison assigned to the System of Care Chicago site.

Outcomes

The data indicate that when wraparound interventions are implemented, children experience fewer clinical symptoms related to serious emotional disturbances. Improvements in family and school relationships have also been demonstrated. Reduced rates of behavioral sanctions (i.e., suspensions and expulsions), and increases in school attendance and academic achievement have also been positively associated with enrollment into the project. Patterns of service utilization also support that families experience improved access to services and supports.

System of Care - McHenry

Data Value
DHS Contract Mgr Patricia Roy, MD
Annual Grant $

$1,499,444 in Federal FY 07

(varies each grant year)

Evaluator

Name & Contact Info

Joel B. Cowen, MA

University of Illinois - Rockford

College of Medicine

Health Systems Research

Annual Eval $ $172,577 in Federal FY 07
Funding Source SAMHSA/CMHS
Eval Period 10/01/06 through 9/30/11

Evaluation information was made available in the continuation application in March 2006 and in an October 31, 2006 progress report.

Project Summary

McHenry County Family CARE is a Substance Abuse Mental Health Services Administration (SAMHSA) Systems of Care Grant funded opportunity. The mission is to transform McHenry County's system of care for serious emotionally disturbed youth and their families through system-wide strategic collaboration and implementation. The effective system of care will be Family-Driven, Youth-Guided, Culturally Competent, Evidence-Based and provide individualized care. DHS/DMH has partnered with the McHenry County Mental Health Board to implement system of care transformation, on a local level, within McHenry County.

McHenry County Family CARE emphasizes the use of the Wraparound philosophy to empower youth and families as well as implement system-wide collaboration. The use of Child and Family Teams engages traditional and non-traditional supports in care planning while family members are taught how to manage their care, resources and desired outcomes. Family Resource Developers, parents of SED youth, are employed throughout the community to facilitate Child and Family Teams as well provide support and resources to family members in need. School Sector Coordinators, who are also parents of SED youth, are located within several local school districts. They have access to and knowledge of key local community and system partners. Within the school, they facilitate Resource Review Panels, engaging key local community partners in the process of allocating resources, including available flex funding. McHenry County Family CARE has developed an active Family Council and Youth Council. Both are already engaged in social marketing, evaluation, governance and service planning activities. The Governance Council meets on a quarterly basis with workgroups focused on budget, evaluation and sustainability planning.

Evaluation Activities

The first Federal grant year is considered a planning year. During this year, the Lead Evaluator, Project Director, Clinical Director, Family Involvement Coordinator and some members of the administrative team spent a considerable amount of time learning about National Evaluation activities and expectations. The Lead Evaluator worked with the Family CARE administrative team, Family Council and Youth Council to gather necessary data and perspectives to develop the IRB proposal. Health Systems Research facilitated several planning and educational sessions for families, youth and community stakeholders. Dr. Kline, MD, IRB Chair, presented to the Family and Youth Councils at a community meeting in July. All staff have completed the necessary IRB certification course. ORC Macro, technical assistance for National Evaluation activities conducted a site visit in September. The IRB proposal was submitted for review by the IRB at their October meeting. All system of care newly funded sites are currently awaiting Federal OMB approval of the National Evaluation instruments. The tentative date Family CARE will enroll families and youth into the National Evaluation is November 1, 2006. With the development of the Governance Council and a recent ORC community presentation, Family CARE is beginning to develop a community discussion with families, youth and community partners on the design of the Local Evaluation.

Health Systems Research has hired the parent of an SED youth as the full-time Evaluation Administrative Assistant. Two Research Assistants will be hired upon IRB approval of the evaluation proposal.

No implementation or outcome evaluation information is reported at this time as no families or youth are currently enrolled in evaluation activities.

Addendum

Separately Funded National Evaluation That Includes an HCD Grant Program:

Employment Retention and Advancement

Data Value
DHS Contract Mgr Debbie Gibson / Nancy Crossman
Annual Grant $

$1,500,000

(total funding Feb 2002-June 2004 was $3,589,232. Initial funding was secured from the Governor's Discretionary portion of the state's Welfare-to-Work block grant. The U.S. Department of Labor provided supplemental funding in 2004 after Congress rescinded unspent Welfare-to-Work funds.)

Evaluator

Name & Contact Info

Dan Bloom

MDRC

Annual Eval $ Unknown (not funded by DHS)
Funding Source

DHHS, Admin for Children and Families, Office of Planning, Research and Evaluation

(for evaluation funding, not program funding)

Eval Period November 1, 2000 through October 31, 2005

Results from: Dan Bloom, Richard Hendra, Karin Martinson, Susan Scrivener, The Employment Retention and Advancement Project: Early Results from Four Sites. June 2005. MDRC: New York. Accessed at http://www.mdrc.org. Per e-mail communication with MDRC on January 5, 2007, a full evaluation report on the Illinois project will be available "in the next couple of months." Note: The citation above has already been reported on in the Summary of Program Evaluation Findings of February 2006; no new material has been added to the sections below.

Project Summary

Operating in Chicago and St. Clair County, the Illinois Employment Retention and Advancement (ERA) program targets a group that appeared to be "stuck" in low-wage jobs: TANF recipients who reported full-time employment to the welfare agency for at least six consecutive months and yet continued to qualify for cash assistance. Operated by contracted service providers, the program provides a range of services designed to help participants increase their earnings in their current job or, more typically, to find a higher-paying job. The control group is not referred to an ERA service provider but may receive services from welfare office staff.

Year One Impacts

The one year impacts measure outcomes during the first year after people entered the study, from February 2002 through September 2002. The Illinois ERA program generated statistically significant increases in both unemployment insurance (UI)-covered employment and earnings during the first year of follow-up. Just over 71% of the ERA group worked in a UI-covered job at some point during Year 1, compared with 65% of the control group. The ERA group earned, on average, $539 more than the control group during year 1, an increase of about 9%. The Illinois ERA program also produced a fairly large decrease in TANF cash assistance payments. On average, the ERA group received $193 less (about 12% less) in TANF benefits than the control group during year 1.

Impacts Over Time and for an Early Cohort

Data for one additional quarter of follow-up are available for the cohort of people who entered the study from February through June 2002. Results for this early cohort show that the Illinois ERA program's impacts on both employment and earnings generally grew larger over time. The impacts on employment and earnings appear to have increased dramatically just after the end of year 1. It is too early to say whether this pattern will persist when longer-term follow-up data are available for a larger sample of participants.

Since everyone in the sample had reported earnings to DHS for at least six consecutive months prior to their random assignment date, it is likely that the individuals with no UI-covered earnings in the quarter prior to random assignment were working in jobs not covered by the UI system. The results show that the Illinois ERA program generated very large increases in UI-covered employment and earnings for sample members who had no UI earnings in the quarter prior to random assignment. For example, average earnings for the ERA group are about twice as high as for the control group. This provides further evidence that ERA may have helped individuals move from non-UI-covered jobs to UI-covered jobs. The program also produced a large decrease in TANF payments for this group, but this loss in income was more than offset by the very large earnings gain. Thus, total measured income from UI earnings, TANF, and food stamps was substantially higher for the ERA group.

The early results for the Illinois ERA program are promising. The program has increased both UI-covered employment and earnings, possibly by helping some individuals move from non-UI-covered jobs to UI-covered jobs. It will be important to see whether these effects continue with longer follow-up and also whether survey data - which should include some information on non-UI-covered employment - bear out the results from UI records.

Appendix A

Chart: Evaluations by Division

Appendix B

Chart: Evaluations by Evaluator