DHS Contract Mgr Danielle Kirby
Annual Grant $ $400,000
Evaluator Name & Contact Info

Maria Bruni, Ph.D.

Independent

Maria.Bruni.@illinois.gov

Annual Eval $ Closed
Funding Source SAMHSA/CSAT
Evaluation Period 07/01/05 through 06/30/08

Program Summary

The Illinois Adolescent Substance Abuse Treatment Coordination Initiative (IL-SAC) was implemented to improve the capacity in the State of Illinois to provide effective, accessible, and culturally competent substance abuse treatment for youths and their families, through the statewide coordination and enhancement of youth treatment services. In order to understand the accomplishments and barriers related to developing the infrastructure for adolescent treatment, an evaluation of project activities was carried out over the three years of the grant award. Since most (over 90%) of the evaluation was completed prior to FY '09, much of the description below is re-printed from the FY '08 Summary of Program Evaluation Findings. The last section of this article provides an update on the final evaluation activities of the IL-SAC project, which focused on qualitative interviews with families of adolescents involved in the treatment system, in order to better understand the barriers that may prevent full access to services among youth in need of substance abuse treatment.

As the IL-SAC project began, DASA hired a full-time Adolescent Coordinator and a full-time Workforce Force Specialist (as required by the Center for Substance Abuse Treatment). DASA also entered into agreements with DASA licensed organizations to expand the use of evidence-based treatment (EBT) with Illinois adolescents. The Adolescent Coordinator worked closely with the treatment providers that make up the adolescent substance abuse treatment system. The Adolescent Coordinator coordinated efforts with key staff in the mental health, child welfare, criminal justice and education systems. The Workforce Specialist worked with provider organizations, higher education organizations, and training institutes to develop strategies to ensure an adequate supply of competent workers enter and are retained in the adolescent substance abuse treatment workforce.

DASA selected organizations through a competitive procurement process to be trained and certified in the GAIN assessment instrument by Chestnut Health Systems' Global Appraisal of Individual Needs (GAIN) Coordinating Center. Staff from these organizations received training and implementation guidance through Chestnut's Lighthouse Institute in either Motivational Enhancement Therapy/Cognitive Behavior Therapy (MET/CBT5) or the Adolescent Community Reinforcement Approach, both developed and tested as part of the CSAT-funded Cannabis Youth Treatment study.

Evaluation Activities

The evaluation and data collection plan included the administration of the following Center for Substance Abuse Treatment (CSAT) Government Performance Results Act (GPRA) forms: CSAT Baseline and Follow-Up Training Satisfaction Survey, CSAT Baseline and Follow-Up Technical Assistance Satisfaction Survey and CSAT Baseline and Follow-up Meeting Satisfaction Survey. Also, focus groups, stakeholder interviews, and process observation of project meetings were used to evaluate program activities.

The process evaluation involved qualitative data collection and analysis. Interviews with DASA staff, treatment provider staff and various council/committee members, as well as focus groups and organizational assessments were employed to answer process evaluation questions, such as "Is this project facilitating linkages, coordination and sharing of information across state agencies that serve adolescents?" Focus groups were conducted with staff from the treatment agencies participating in the evidence based practice (EBP) and GAIN pilot projects. In addition, initial stakeholder interviews were completed with policy makers and staff from substance abuse prevention organizations throughout Illinois.

To answer the project's evaluation questions, the project's evaluator, (Maria Bruni, Ph.D.), developed three separate data collection instruments. The Adolescent Provider Survey instrument was administered to licensed adolescent treatment providers to collect data on the scope of services provided to adolescents, including but not limited to: the use of evidence-based practices, services for co-occurring disorders, screening and assessment processes and instruments, and training needs. Both the Regional Provider Service Satisfaction Survey and the Parent/Consumer Satisfaction Survey assessed the stakeholders' perceptions of access to and coordination of adolescent treatment services. These instruments explored stakeholder perceptions of key changes related to the delivery of services for adolescents.

The outcome evaluation used GPRA data, existing DASA administrative data, and primary data collected from the Adolescent Provider Survey, the Regional Provider Service Satisfaction Survey and the Parent/Consumer Satisfaction Survey. Focus group and semi-structured interview data along with the narrative responses from the Adolescent Provider Survey, the Regional Provider Service Satisfaction Survey, and the Parent/Consumer Satisfaction Survey were the sources of qualitative data. Analysis of these data produced answers to outcome evaluation questions.

Specifically, the measurement of outcomes focused on:

  • Reductions in barriers to the coordination between state agencies, providers and consumers;
  • Increased involvement of parents and consumers in treatment, practice, and policy;
  • Increased interest in training related to evidence-based practices (EBPs);
  • Reduction in barriers to the implementation of EBPs in adolescent treatment programs;
  • Increased knowledge of financing/allocation related to adolescent treatment; and
  • Improvements in clinician-supportive features to state credentialing guidelines.

Final Evaluation Findings

The most important findings are related to:

  1. The implementation of a standardized substance abuse assessment instrument for adolescents. The implementation of any standardized assessment instrument into the treatment system is complicated. The training and use of the standardized instrument (GAIN) was less successful for programs that employ clinically advanced front line staff than for those who employ less experienced clinical staff. Programs with seasoned clinical staff found the standardized instrument (GAIN) to severely limit probing and their capacity to uncover details of their clients' circumstances essential to the assessment phase and to rapport building during the starting point of treatment. Programs with less experienced staff tended to appreciate the increased structure of the standardized instrument and the assessment document generated from the instrument.
  2. Challenges involved in training and implementing the use of evidenced-based practices (EBP) among adolescent providers: Eight adolescent substance abuse provider agencies were selected to participate in the pilot project designed to learn about EBP training and implementation. Providers involved in this pilot chose to be trained and certified in either the Adolescent Community Reinforcement Approach (A-CRA) or Motivational Enhancement Treatment/Cognitive Behavioral Treatment (MET/CBT).

    Immediately after the training, one of the agencies dropped out of the EBP pilot due, in part, to major structural constraints in the workplace that would not allow for successful EBP training (The agency serves youth in correctional facilities and staff would not have been able to tape record any of their session and complete planned training and certification activities.) This early event in the EPB pilot highlighted for the evaluators the fact that structural impediments to implementation exist despite much forethought.

    From the focus groups conducted with the seven remaining agencies in the EBP pilot and from initial stakeholder interviews, it was determined that the success of implementing evidenced-based practices is limited by severe workforce issues such as 1) high rates of staff turnover and 2) the cost of training and materials of current evidence based practices. Providers interested in using EBPs need to consider prior to investing in a practice, the cost of it, the staffing patterns at their agency including typical employment lengths, and the structure of their practice that may impede adequate training (for example, being housed in correctional facilities).

  3. Critical barriers to providing integrated care for adolescents with co-occurring disorders. During the first 22 months of the project, goal-specific workgroups were created in an effort to impact structural change in specific areas. The MISA workgroup (comprised of representatives from the Division of Alcoholism and Substance Abuse, the Division of Mental Health, The Department of Child and Family Services, and The Illinois MISA Institute) produced information about critical barriers to providing integrated care for Illinois adolescents with co-occurring disorders. For example, in Illinois, state mental health and substance abuse treatment systems operate in different and at times, conflicting ways. Discussions focused on the possibility of creating a single point of entry for adolescents in need of mental health and substance abuse services.

The most surprising finding from the project's evaluation had to do with spending related to adolescent treatment. From the process observation of the project's Funding Workgroup, it was found that despite the fact that six (6) residential facilities for adolescents exist in Illinois, 80% of the spending for adolescents in Illinois is for residential treatment. The Funding Workgroup spent a significant amount of time discussing this issue and is continuing to track adolescent spending across service categories in order to examine the effect of policy changes made through the SAC project.

FY '09 Evaluation Update

The final phase of the Illinois SAC evaluation centered on an examination of family/consumer involvement in the state's system of adolescent treatment. Central to effective adolescent treatment is family/consumer involvement. Historically, participation of family members in the treatment process and policy creation process has been minimal in Illinois. During this phase of the evaluation, the evaluator used data collected from over 20 family listening sessions held across the state as well as surveys to better understand the critical barriers to family/consumer involvement.

These sessions allowed the evaluator to gather information about parents' experiences with their children in treatment and recovery and to assess parents' interest in creating ongoing groups throughout the state to give input to infrastructure change. During the family listening sessions, one of the criticisms heard most often was parents' disappointment with the lack of information in schools about adolescent treatment. In response, IDHS/DASA partnered with the Illinois Federation of Families (IFF) to write and produce a Parent Handbook, which guides parents through the adolescent treatment process. The Parent Handbook, which was published at the end of 2008 (in English and Spanish), has been distributed to parents and schools and is posted on the Illinois Treatment Works website.

Adolescent Substance Abuse Treatment Coordination Initiative Evaluation Design

Maria Bruni, Ph.D. and Beth-Anne Jacob, Ph.D.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process:

  • The degree to which the project adheres to its plan to increase and improve elements of its system.
  • Existing barriers to the planned increases and improvements.
  • The occurrence of infrastructure change.
  • Perceptions of key stakeholders.

Outcomes:

  • Changes in access to treatment.
  • Changes in clinically appropriate assessment skills and interventions for treating adolescents.
  • Changes in client outcomes as a result of all other changes.

GAIN

DARTS

Interviews

Focus groups

GPRA satisfaction survey

An evaluation team of DASA contractors conducted the evaluation.