2010 IDHS Summary of Program Evaluation Finding - Update

Grants Evaluation Report

IDHS Summary of Program Evaluation Finding - Update, November 2010

Dear Colleagues:

We are pleased to distribute the Illinois Department of Human Services (IDHS), Summary of Program Evaluation Findings 2010. As of November 2010, sixteen (16) programs representing $35,594,640 in federal grants funded through the Illinois Department of Human Services (IDHS) for FY2010 had independent evaluations that were ongoing or recently completed. A total of $2,973,830 of the $35,594,640 was used to fund these evaluations. This year marks the fifth year of this annual report that is recognized as a one-of-a-kind publication in Illinois designed to increase awareness of best practices.

Developing systems of care that transform the delivery of health and human services within states is a major national trend. This year's report showcases two major system of care initiatives underway in Illinois, and one system of care which is in its final year of funding. In September 2009, the Illinois Department of Human Services, Division of Mental Health was awarded two (2) grants of $9,000,000 each for a total of $18,000,000 to be disbursed over 6 years by the U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (SAMHSA). Illinois was one of two states to receive two new awards in a single year.

This report also contains evaluation findings from federally-funded projects currently underway in the Illinois Department of Human Services. The fundamental purpose of evaluation is to examine services and/or systems to determine what works most efficiently and effectively. When the findings are fully integrated into the operations of a system or the delivery of services, it results in an overall improvement of services and outcomes for Illinois residents.

In addition to the IDHS/Division of Mental Health, this report also includes findings from federally funded projects currently underway in the IDHS Divisions of Community Health and Prevention and Alcoholism and Substance Abuse. A federally funded project in the Division of Community Health and Prevention is reported by the Illinois Department of Transportation of aiding in the significant decrease of alcohol related fatal crashes from 472 in 2004 to 259 in 2009 and alcohol related fatalities from 531 in 2004 to 293 in 2009. Another federally funded project in the Division of Community Health and Prevention is aimed at integrating behavioral health and primary care to promote child health and wellness. This project will serve four communities on Chicago's west side where the infant mortality rate and proportion of births to teen mothers exceed city and state rates. The IDHS/Division of Alcoholism and Substance Abuse is concluding its Access to Recovery-II grant which was awarded in 2007. The data collected by this project demonstrated significant client improvement in functioning and in October 2010, DASA was awarded an ATR III grant to expand the current ATR program for another four years. This new grant of $13.1 million will allow DASA to expand options for over 7,600 people seeking drug and alcohol treatment and recovery services over the next four years so that DASA can continue to increase access to recovery for Illinois residents.

The following individuals are to be acknowledged for their diligence and hardwork in developing, editing, and providing the information provided in this report.

  • IDHS staff: Tanya R. Anderson, MD; Amy Starin, Ph.D.; JoAnne Durkee; Karrie Rueter; Kim Fornero; Denise Simon; Myrtis Sullivan, MD; Xochitl Martirosayn; Rex Alexander; Richard Sherman, Ph.D.; Carolyn Hartfield; Peggy Alexander; Lisa Cohen; Patricia Kates-Collins; Phillip Matute; Denise Eligan, MA.; Kathryn Bass; and Sharon Zahorodnyj, MSW and MBA.
  • Evaluators: Nicole E. Allen Ph.D.; Mark Aber, Ph.D.; Mary Spooner, Ph.D.; Jennifer Cartland, Ph.D.; Peter Mulhall, Ph.D.; Nancy Flowers, B.S.; Beth Welbes, M.S.P.H.; Crystal Reinhart, Ph.D.; Julie Spielberger, Ph.D.; Richard Sherman Ph.D.; IOTA, Inc.; and Sherman Consulting Group, LLC.

Sincerely,

Grace Hong-Duffin, J.D., M.P.A.

Acting Secretary and Chief and Staff

Illinois Department of Human Services

IDHS Division of Mental Health

System of Care - Champaign County ACCESS Initiative

DHS Contract Mgr Tanya R. Anderson, MD
Annual Grant $

$1,000,000 in Federal FY 10

(varies each grant year)

Evaluator

Name & Contact Info

Nicole E. Allen, Ph.D. & Mark Aber, Ph.D.

University of Illinois at Urbana-Champaign

allenne@illinois.edu; maber@illinois.edu

Annual Eval $ $200,000 in Federal FY 10
Funding Source SAMHSA/CMHS
Eval Period 10/1/2009 - 6/30/2010

The ACCESS Initiative of Champaign County initiated a cooperative agreement with the Substance Abuse Mental Health Services Administration (SAMHSA) to develop a System of Care in Champaign County in October 2009.

Project Summary for the System of Care in Champaign County 

The ACCESS Initiative aims to create a System of Care (SOC) in which youth and families have access to a full range of programs, services, and formal and informal supports across a continuum of needs. ACCESS will target youth aged 10 to 18 who have been involved in or are at risk for involvement in the juvenile justice system, have had multiple systems involvements (e.g., child welfare, juvenile justice, school disciplinary), and/or have social, emotional, and behavioral challenges. ACCESS will focus on the subpopulation , in particular, of African American youth given they comprise almost 80% of youth being detained in the local juvenile detention center. Funding is granted over six-year period to encourage the full implementation of a System of Care that is family-driven, youth-guided, culturally and linguistically competent, coordinated and sustainable beyond the funding period. The first year of funding is devoted to planning and creating the necessary infrastructure to begin System of Care implementation in Year 2. As strongly encouraged by SAMHSA, local efforts aim to be data-driven with a commitment to continuous quality improvement (CQI) and the implementation of evidenced-based practices and practice-based evidence throughout our systems. Co-lead evaluators have been involved in the effort from the beginning and are working as collaborative partners to build a data-informed System of Care. The following report highlights three aspects of Year 1 effort through June 30, 2010 and details the activity of the Evaluation Team. However, formal evaluation activities are just beginning and process and outcome data will be available beginning in Year 2.

Year 1 activities have focused heavily on building the necessary groundwork to support the implementation of the System of Care. Based on a review of documents archived by ACCESS staff and co-lead evaluators, this summary report highlights efforts in three critical areas including activities related to: a) engaging and preparing key stakeholders, b) building ACCESS Initiative infrastructure and c) developing evaluation plans and infrastructure. Indeed, this report does not reflect all of the activity of the ACCESS Initiative, but demonstrates the depth of effort to lay the foundation on which the ACCESS Initiative System of Care will be built.

Engaging and Preparing Key Stakeholders

A critical step in the development of a System of Care (SOC) is the engagement of a broad range of key stakeholders, including youth and families, and the development of a shared understanding of core System of Care principles and values. While this is an ongoing process, ACCESS Initiative Year 1 activities reflect a commitment to such broad engagement by offering a) orientation trainings to interested stakeholders (including community members, and agency staff and leaders across systems); b) training for youth and parents to be full participants in System Of Care (SOC) development; c) a technical assistance retreat; d) participation in SAMHSA-sponsored webinars; e) travel to SAMHSA sponsored training events; f) public presentations to promote and inform community groups about ACCESS; and g) cultural and linguistic competence consultations to agencies funded by the local Community Mental Health Board.

Orientations to Systems of Care

From the onset of the cooperative agreement and throughout the year, an orientation was developed and offered to introduce key stakeholders to the ACCESS Initiative and Systems of Care. At least 12 orientation training events were co-led by the Technical Assistance Coordinator, Cultural and Linguistic Competence Coordinator, Lead Family Liaison, the Parents Promoting Presence (P3) President and parent leaders. Over 70 participants attended these trainings including the entire juvenile probation staff, members of the Interim Governing Board, stakeholders from schools and the juvenile justice system and service provides from various community-based agencies. Tailored presentations were made to the Local Area Network (LAN) and to the local Human Services Council. These orientations were designed to bring participants "up to speed" regarding the goals of System of Care and the aims of the ACCESS Initiative, in particular. The training was hands-on and gave participants an opportunity to begin to think about how they and/or their agency (when applicable) might be engaged in System of Care effort. The trainings were offered at various times of day to make them maximally accessible. Feedback from these sessions suggests that participants found them valuable and informative. In open-ended feedback on evaluation forms, many participants reported learning about the depth and scope of the ACCESS initiative; commented that they were excited about the focus on families and youth; and valued the emphasis on collaboration among community members and agencies. In general, attendees seemed to report a better understanding of the SAMHSA grant and the broad, systemic approach of the initiative.

The orientation training has become a venue by which interested stakeholders can become involved in the System of Care and will be ongoing. In addition, advanced trainings are being explored to build on the basic knowledge shared in the initial training and to prepare stakeholders who are involved in the Full Partnership, Coordinating Council, Working Groups, and Committees of the ACCESS Initiative.

Targeted Training and Recruitment of Parents and Youth

From the beginning of the ACCESS Initiative there has been an emphasis on the genuine involvement and voice of youth and parents in the developing SOC. Existing parent groups (in particular, Parents Promoting Presence [P3]) and youth groups (in particular, the Peer Ambassadors [PA]) provided an invaluable start for the meaningful engagement of families in SOC development.

P3 has maintained a leadership role in convening parents to form the Family Organization. Throughout the fiscal year they have met at least three to four times a month (and more when activities were being planned).

Building on the leadership provided by P3 and the PA groups, throughout Year 1, at least nine training events were held for youth and parents focusing on a wide variety of topics including, for example, evaluation, Medicaid, cultural competency, the role of a parent and youth leader, and hiring practices. The parents and youth leaders also participated in regular weekly meetings to prepare for Governing Board and other ACCESS related activities. While orientation trainings were also open to youth and parents, these targeted trainings were specifically created for youth and families, aimed to prepare them for full participation and leadership in ACCESS and reflected their identified interests and priorities. Approximately eight youth and six parents consistently attended these events. In addition, P3 held three open house events to invite parents to learn more about their organization and about the ACCESS Initiative and networked with other parent groups throughout the year to expand their membership and reach. Approximately 12 to 15 parents attended each open house event.

In June, following the hiring of a permanent Administrative Team (more on this below) an intensive 40-hour training was offered to engage additional youth and parents in ACCESS. A total of 36 parents and 36 youth completed the full training (67 youth attended at least parts of the training; some youth were involved in multiple efforts and could not attend all sessions, but had a significant engagement in the ACCESS training). Many of these youth and parents have become members of the ACCESS Coordinating Council and will be on various workgroups and committees as they are developed. Consistent with earlier training efforts, topics covered in the trainings were designed to help participants better understand the service delivery system, their own leadership styles, family-driven and youth-guided care, cultural competency, evaluation, and social marketing in a Systems of Care context. The trainings were not only beneficial to the participants but also served as a social marketing/engagement tool by engaging members of the Governing Board (e.g., on law enforcement response, mental health, substance use), Administrative Team (e.g., on social marketing) and stakeholders as presenters and helpers.

Technical Assistance Retreat

Following the formation of the permanent Administrative Team, a retreat was held to provide intensive training to individuals identified as possible members of the permanent coordinating council. The federal technical assistance (TA) coordinator worked closely with ACCESS staff to develop a local event that would provide information regarding critical tasks, including for example, the development of broad-based governance and working committees. The TA Coordinator attended the local event and other TA providers from the TA Partnership called in to present information regarding collaboration and intervention with key systems including juvenile justice and education. There were 37 participants including representatives from key child serving agencies, child welfare, parents, and youth.

Webinars

SAMHSAs TA Partnership offers a wide variety of training webinars for new communities and on topics related to System of Care development and implementation. Virtually all webinars offered were attended by representatives from the local community. To use these as community building opportunities, space was often reserved in the local community so that participants could attend the virtual webinar together. This often engendered discussion among local participants.

There were nine New Communities webinars, six Wraparound webinars, two on Trauma Informed practices, four Evaluation webinars, and a series (four calls) sponsored by the Federation of Families designed to help families better understand their role in the evaluation process and how to use data to advocate for improved family driven policies, procedures, etc. Community members, parents, and youth also regularly participated in SAMSHA sponsored affinity calls: on Family Driven and Youth Guided Care, Cultural Competency, Social Marketing, and the Fatherhood Initiative.

Federal Training Events

There are multiple training events that ACCESS Initiative partners have attended including the 2010 Child Mental Health Initiative New Community Training, and the 2010 System of Care Primer held in Washington, DC. A wide variety of representatives (including representatives from the Illinois Department of Human Services, Division of Mental Health) were sent with the goal of bringing information back to the community. Ten stakeholders attended the New Community Training and four attended the System of Care Primer.

Public Presentations

Multiple presentations were made by ACCESS participants to inform the public and service providers about the developing SOC. For example, presentations were made to the City of Champaign Human Relations Commission, the Champaign County Human Services Council, the Local Area network and on two local radio shows.

Cultural and Linguistic Competence Consultations

As part of their ongoing commitment to the development of the ACCESS Initiative System of Care, the local Community Mental Health Board had funded a cultural competence liaison who served as the Interim and now as the permanent Cultural and Linguistic Competence Coordinator on the ACCESS staff. To continue to develop local capacity to delivery culturally and linguistically competent services (an ongoing process), the Coordinator offered organizational consultation and training to key child-serving agencies.

Building Infrastructure

A key goal of Year 1 activity was getting the necessary infrastructure in place to develop the System of Care. A critical strength of the local effort is the involvement of youth and parents as key stakeholders from the beginning of the process. The ACCESS Initiative's commitment to having a truly parent-driven, youth-guided process was reflected in the process employed to build the ACCESS Initiative infrastructure.

From the beginning of the fiscal year, ACCESS has focused on building the infrastructure to ensure broad-based governance. This began with the formation of a group that was open to any and all individuals, groups and organizations interested in being part of the SOC - ACCESS-ALL. During the first month of funding (November, 2010), a smaller group was created (from ACCESS-ALL) and charged with identifying an interim governing team; the interim governing team was charged with hiring the ACCESS administrative staff. Throughout this process, transparency and "bottom-up" approaches aimed to encourage a broad sense of ownership and engagement. Youth and families were engaged in all phases of this development. In addition, the Champaign County Mental Health Board (CCMHB) and the Illinois Department of Human Services, Division of Mental Health (co-PIs) have provided ongoing technical assistance and participation to support the development ACCESS Initiative.

ACCESS-ALL

ACCESS-ALL reflects the full partnership and is open to any and all individuals, groups and organizations that wish to participate in the System of Care effort. ACCESS-ALL has met almost monthly since October, 2009 and has representation from parents, youth, multiple systems (e.g., child welfare, mental health, juvenile justice), key child serving agencies (e.g., Champaign County Mental Health Board, Prairie Center, Family Advocacy in Champaign County, Parenting with Love and Limits, Talks Mentoring, Dan Moyer Boys and Girls Club), and community groups (e.g., Champaign Urbana Area Project). To begin to facilitate communication, a Google group was created (an official website will be developed for ACCESS); this group includes 145 members, many representing groups and organizations. It is not yet the case that all group members attend ACCESS-ALL meetings, but the goal is to steadily increase opportunities for ACCESS-ALL members to actively engage in ACCESS (e.g., via workgroup membership).

ACCESS Interim Working Group

The first goal was to form an interim governing team which could focus on hiring key ACCESS Initiative staff. As a first step, a working group was formed to devise a process to identify interim governing team members. This group met from October to December at least monthly (and sometimes bi-weekly) and included representatives from a variety of groups and agencies, including, for example, parents, youth, child welfare, and juvenile justice. This group was viewed as a temporary venue for the development of a process by which community members could nominate interim governing team members across a variety of domains including, family, youth, education, juvenile justice, child welfare, community members, and faith-based organizations.

The working group emphasized that the development of the interim governing team was a transparent process and developed a list of values and qualifications that were required for participation on the interim governing team. Community members were invited to nominate individuals through ACCESS-ALL. All individuals nominated were invited to attend a meeting to introduce themselves and be interviewed in a group format. Members of the working group then voted on nominees to fill slots within each of the domains. In addition, families and youth had a simultaneous process to identify the family and youth representatives that would comprise 51% of the team. The resulting group of 19 individuals was then convened as the ACCESS Interim Governing Team. In addition to those nominated, the local Principal Investigator (from the Champaign County Mental Health Board), the Co-Lead Evaluators, a Facilitator and the interim Cultural and Linguistic Competence and Technical Assistance Coordinators were invited as non-voting ex officio members.

Interim Governing Team (ACCESS-GOV)

The primary charge of the Interim Governing Team was to identify an inclusive hiring process that would include youth and family voice and to transition from an interim governance structure to a permanent structure following the hiring of staff. To this end, multiple hiring committees were formed and chaired by Interim Governing Team members. All committees included youth and family representatives, and County and State partners. Ultimately, the respective searches resulted in the formation of the permanent Administrative Team including the Project Director, Technical Assistance Coordinator, Social Marking Coordinator, Cultural and Linguistic Competence Coordinator, Lead Family Contact and Youth Engagement Coordinator. The Interim Governing Team met two times/month (and as needed in subcommittee) beginning in January 2010 and ending in June 2010 (with the formation of a permanent board).

While the primary charge of the Interim Governing Team was hiring the Administrative Team, they also developed a draft of bylaws to direct their work (and that could be built upon by the permanent governance team); approved the budget and fiscal agent for the cooperative agreement; approved the selection of space for ACCESS offices; and identified participants to attend required SAMHSA training in Washington DC. At the same time, the Interim Governing Team began to think about what form broad-based governance would take in the ACCESS Initiative. These are both tasks (among others) that will continue to be pursued by the permanent council.

Interim Administrative Team

From the beginning of the cooperative agreement, the co-Principal Investigators, the Co-Lead Evaluators, a Facilitator and the interim Cultural and Linguistic Competence and Technical Assistance Coordinators met as part of an Interim Administrative Team to plan for Interim Governing Team meetings, to identify and provide support regarding critical tasks and time-sensitive decisions.

Transition to a Permanent Coordinating Council and Full Partnership

Following the formation of the Administrative Team in June, 2010 a permanent Interim Governing Team, renamed the Coordinating Council was formed and included those members of the interim body who wished to continue (some moved into ACCESS administrative roles). The Project Director in consultation with ACCESS staff and community members identified additional key stakeholders to invite to the permanent coordinating council. This body includes 29 individuals covering the core domains detailed above and including 51% youth and family members. Similar to the Interim Governing Team, the co- Principal Investigators, Project Director, Administrative Staff, and Co-Lead Evaluators attend Coordinating Council meetings as non-voting ex officio members.

Structures for Ongoing Youth and Parent/Family Involvement

As reflected in the targeted training events for youth and parents, the ACCESS Initiative adopted a comprehensive family and youth leadership engagement strategy. Following the June training of additional parents, P3 (Parents Promoting Presence) membership has expanded and became the "umbrella" family organization. In addition, the Family Advisory Board (FAB) was created to specifically advise the Lead Family Contact and to coordinate the efforts of the family organization. The youth leadership structure was previously coordinated through the Peer Ambassadors program and now the Youth Advisory Board who works in close contact with the Youth Engagement Coordinator. Both groups have met regularly since before the inception of the Initiative and have been active and critical members of the Initiative from its official beginning. P3 has organized a parent support group and has a strong cross-system collaboration presence with a broad array of partners. The youth PAs are equally active working on statewide collaborations with other youth leaders (e.g., on the statewide YOUTH M.O.V.E. application and youth media and advocacy) and local efforts to build a System of Care (e.g., the formation of MMM - Motivating More Males; ongoing focus group at the juvenile detention center; community events).

County and State Support and Involvement

The Illinois Department of Human Services (IDHS), Division of Mental Health and the local Community Mental Health Board (co-PIs) have provided ongoing technical assistance to support the ACCESS Initiative. For example, the co- Principal Investigators have provided support in the identification of priorities and strategies for the initiative; have informed the development of local infrastructure development; and have identified potential State partners to assist in implementation efforts. The IDHS Division of Mental Health Chief, Clinical Services Systems, Dr. Tanya R. Anderson (co-PI) and her staff, have worked to connect ACCESS parent leadership to a regional Family Consumer specialist and have kept local ACCESS partners abreast of potential funding, program and legislative changes. The County Mental Health Board, directed by Peter Tracy (co-PI), acts as the fiscal agent for the cooperative agreement and is actively involved in all facets of local implementation efforts (e.g., budgetary management). Both County and State partners were directly involved in the hiring of the ACCESS Administrative Team and attended federal training in Washington DC as part of the ACCESS constituency. This support provides an essential foundation for developing a sustainable System of Care in Champaign County and beyond.

Evaluation Planning and Infrastructure

Evaluation plays an important role in ensuring a data-driven and -informed process in the development and implementation of a System of Care. The ACCESS co-lead evaluators have been integrally involved in the Interim and Permanent Administrative Team and are non-voting ex officio members of the Coordinating Council. Given the emphasis on a collaborative and participatory evaluation in the cooperative agreement, in January, 2010, the Evaluation Collaboration Team (ECT) was formed. The ECT has over 20 members including youth, families, systems and community stakeholders and has met twice per month since formation (usually with 12 to 15 members in attendance). The Evaluation Collaboration Team (ECT) has drafted a working theory of change logic model (and continues to seek input on the model and improve its graphic presentations) and provides input into the evaluation research process.

The central focus of Year 1 is planning and infrastructure development to ensure a high quality implementation of the national evaluation, local evaluation activities and mechanisms to ensure continuous quality improvement (CQI). In collaboration with the ECT, the co-lead evaluators have been building the necessary supports to pursue the local and national evaluation (e.g., hiring staff as funds are available; pursuing IRB approval; conducting literature and instrument searches). All evaluation activities have been and will be pursued with ECT input and guidance. Ongoing activities focus on the development of specific local evaluation activities that support the development of the System of Care and preparation for the National Evaluation. Early data collection efforts will include interviews and surveys of key stakeholders and youth and caregiver recipients of services.

The following documents accomplishments to date of the Evaluation Team and plans for ongoing activities.

Development of the ACCESS Initiative Infrastructure

Co-Lead Evaluators attended meetings of the ACCESS Initiative Administrative Team and Coordinating Council to a) serve as non-voting ex officio members to support local implementation of the ACCESS Initiative and b) to inform local evaluation planning.

Co-Lead Evaluators worked collaboratively with the interim Technical Assistance Coordinator to structure the initial Self-Assessment required by the National Technical Assistance Partnership.

Formation of the Evaluation Collaboration Team (ECT)

Developed an Evaluation Collaboration Team (ECT) with over 20 volunteers (i.e., an open invitation to those who expressed interest in participating). To date, bi-weekly meetings have included approximately 12 - 15 stakeholders including youth and parents. Parents are regular participants on the team.

Will continue to hold bi-weekly meetings of the evaluation team and also work to actively expand representation and to work with possible shifting membership as more working groups of the ACCESS Initiative are formed.

Established an online list-serve to encourage communication among evaluation collaboration team members, to make meeting announcements and to share evaluation-related documents.

Will continue to work with the Social Marketing Coordinator to explore more sophisticated web-based technology to share information and documents for review.

Worked with the ECT to develop a working theory of change logic model. Specifically, provided all team members with the University of South Florida primer on logic model development, reviewed recommendations and primer content, began the process of "fleshing out" program context, strategies for change and desired outcomes at multiple levels of analysis (e.g., youth/family, organizational, systems, and community). Co-lead evaluators have presented the model to get input from the Interim Governing Team (and now Coordinating Council), the ACCESS Administrative Team (many of whom were involved in model development on the ECT), and to providers at a care coordination meeting regularly held at the Juvenile Detention Center. Seeking input and refining the model will be an ongoing process. In addition, the working model has been used to guide the development of evaluation activities.

Will continue to develop and expand the model with broader input from ACCESS Initiative partners by presenting the draft model to the Interim Governing Team and at ACCESS-all meetings.

Will support emerging working groups and organizations (e.g., the Family Organization) with logic model development and strategic planning.

Created a training on evaluation for youth and families that has been presented on two parent/youth training events.

Will continue to provide training on evaluation processes, terminology, logic modeling, etc. to youth, families, providers, and community members so that a wide range of stakeholders can be full partners in the process.

Beginning Stages of Local Evaluation Activities

Worked with the Evaluation Collaboration Team (ECT) to develop a set of interview questions to pursue with a broad set of current and potential stakeholders of the ACCESS Initiative including youth, parents and system stakeholders. These interviews will inform our understanding of the current service array, current strengths and weaknesses in our community (formal and informal) response to youth with multiple systems involvements, change strategies and the specific roles that various stakeholders can play in the process, and current assets/challenges regarding System of Care principles and practices.

Technical Assistance/Training

Attended all new community training/technical assistance webinars and National Evaluation Webinars. In addition, the co-lead evaluators attended two national conferences through June 30th including the 2010 Child Mental Health Initiative New Community Training, February 8-11, 2010, Washington, DC and the 23rd Annual Children's Mental Health Research & Policy Conference, March 7-10, 2010, Tampa, Florida. In July, they attended the 2010 Georgetown Training Institutes: New Horizons for Systems of Care, July 13-18, 2010, Washington, D.C. These events were invaluable as they introduced the evaluators to the efforts of other communities, available tools to support evaluation activities and networking opportunities with evaluators from other communities.

Will continue to gather necessary information regarding the National Evaluation and prepare the necessary local infrastructure to successfully implement the evaluation.

Will continue to attend required training events and webinars and seek technical assistance as needed.

Building Local Evaluation Support Staff and Infrastructure

Pursued Institutional Review Board (IRB) approval for three protocols to date.

The first received approval for systematic meeting observations, archival review of documents (e.g., meeting minutes, training materials), and interviews with key system stakeholders including youth and parents involved in ACCESS leadership roles. This protocol seeks information from a wide variety of stakeholders as described above (e.g., youth, families, service providers, agency leaders, criminal justice officials, educators) to inform the theory of change logic model and local development of the ACCESS Initiative. This phase of the evaluation aims to get a "lay of the land" and to actively engage current and potential partners in the ACCESS Initiative via an open-ended interview process.

The second IRB protocol received approval to interview youth and families currently navigating various formal systems and agencies as well as informal supports relevant to the developing System of Care. Again, the goal is to build on youth and family experiences to identify critical areas for change as well as current assets in the local response.

The third IRB protocol (under review) seeks approval for local implementation of the National Evaluation.

Hired a full-time evaluation coordinator, graduate research assistants and community members to assist with all evaluation activities.

Compiled a set of resources drawing on System of Care research archives and social science data bases to inform evaluation activities and assist with the identification of relevant instrument and evaluation tools. A database has been constructed that can be queried to find relevant research and instruments to support local evaluation efforts.

Currently developing processes (e.g., data collection, data entry, regular reporting) to successfully implement the National Evaluation and engage in local evaluation activities (e.g., training interviewers, refining recruitment processes).

As data becomes available, the ECT will explore multiple venues (meetings, luncheons) to share emergent evaluation findings with all partners with an orientation to Continuous Quality Improvement.

Prepared the first draft of an IRB protocol to seek approval to gather baseline data from service providers and agency leaders regarding various facets of the current service delivery infrastructure, including, for example, attitudes and behaviors related to System of Care principles (e.g., cultural and linguistic competence) and inter-agency coordination.

In addition, we will seek approval to assess and track youth/family and systems-level outcomes that reflect the priorities of the ACCESS Initiative:

Track relevant youth/family outcomes over time including, for example:

  1. Engagement with service provision.
  2. Access to needed resources that meet family priorities.
  3. Experiences of System of Care principles in service providers' practice (e.g., cultural/linguistic competency; strengths-based; individualized; youth-guided; family-driven; coordinated services).
  4. Decreased distress, increased well-being (e.g., emotional/psychological; social; academic) and improved family functioning.
  5. Improved outcomes in varied life domains: social, academic, interpersonal and psychological.
  6. Reduced negative outcomes (e.g., juvenile detention, recidivism, out of home placement).

2. Track relevant systems-level outcomes, including, for example:

  1. Reduced use of restrictive placements and punitive approaches (e.g., downward trends in Juvenile Detention Center (JDC) and Department of Juvenile Justice (DJJ) placement, recidivism rates, out of home placements, disciplinary actions in schools, and increased "return rates" from foster care.
  2. Increased use of restorative options ("in school, at home and in the community")

In summary, Year 1 ACCESS Initiative engagement strategies and infrastructure development reflects an intensive processes that is laying a solid foundation for System of Care development in Champaign County.

System of Care - McHenry

DHS Contract Mgr Tanya R. Anderson, MD
Annual Grant $

$1,500,000 in Federal  FY 10

(varies each grant year)

Evaluator

Name & Contact Info

Mary Spooner Ph.D.

Mental Health Services and Policy Program

Northwestern University

Annual Eval $ $275,000 in Federal FY 10
Funding Source SAMHSA/CMHS
Eval Period 12/1/05 through 9/29/11

The IDHS/Division of Mental Health is currently operating a System of Care cooperative agreement funded by the United States Department of Health and Human Services, Substance Abuse and Mental Health Services Administration/Center for Mental Health Services. The McHenry County Family CARE cooperative agreement, a Children's Mental Health Initiative, was awarded to McHenry County in December 2005.

Project Summary for the System of Care McHenry County:

The Illinois Department of Human Services, Division of Mental Health, in collaboration with service providers, youth and families in McHenry County, have developed Family CARE, the Child/Adolescent Recovery Experience (CARE), a system of care for the county's youth with serious emotional disturbances (SED). The overarching goal of this project is to transform McHenry County's system of care for serious emotionally disturbed youth and their families through system-wide strategic collaboration and implementation of family-driven and youth-guided services. The Illinois Department of Human Services, Division of Mental Health has partnered with the McHenry County Mental Health Board to implement system of care transformation, on a local level. The mission of this project "is to meet the social and/or emotional needs of families, children, and youth by providing leadership to develop and sustain a system of care that provides continuous support and easy access at every level of care."

Family CARE is led by a parent-led Governance Council charged with representing community voice, promoting systems transformation and guiding programming and financial decisions. The Governance Council meets monthly and comprises workgroups that focus on collaboration, budget, evaluation and sustainability planning. Over the last year, the Governance Council facilitated a community-wide strategic planning process, restructured its leadership, informed budget development, prioritized and established funding levels for community partnerships, increased engagement in evaluation activities and established a steering committee that is developing by-laws and orientation materials. Within the last year, parent leadership within the Governance Council has become stronger, with three parents acting as co-conveners and driving major decisions such as finance and strategic planning. The Governance Council is exploring a partnership with the Child and Adolescent Local Area Network to sustain parent participation in systems planning beyond the cooperative agreement.

McHenry County Family CARE empowers parents and caretakers through the Targeted Parenting Assistance (TPA) model. This model recognizes that each parent is at a different place on a continuum in managing the challenges of raising a youth with a serious emotional disturbance. Nonetheless, each parent possesses different strengths that can be leveraged to provide support to others and to develop the system of care. Within McHenry County, families are active partners in care through the incorporation of the Child and Family Team process in most system of care services. As families navigate the system, they begin to inform the system through support groups, trainings, educational events, workgroups, family resource development, wraparound facilitation, governance and evaluation. Over the past year, parent participation has remained consistent in our Transitional Age Youth Group, Early Childhood Workgroup and Local Evaluation meetings. The National Alliance on Mental Illness - McHenry County (NAMI-MC) chapter and Families Empowered to Change (Families ETC) have become strong advocates and providers of much needed support, training and education opportunities. Many parents remain employed throughout the community as Family Resource Developers, Parent Wraparound Facilitators, data collectors and peer supports.

Family CARE promotes youth voice and development within McHenry County. The Youth Council is active and promotes peer leadership models of care. Youth with a serious emotional disturbance provide support to youth across the community through an active Youth Council and AmeriCorps after-school program. Training and support is provided to help youth in finding their voice and clearly articulating their message through new forms of media. Family CARE is working with leadership from the System of Care Chicago, Champaign County Mental Health Board and Egyptian Health Department to establish state-wide youth engagement opportunities, including the development of a state-wide Youth MOVE (Youth Motivating Others through Voices of Empowerment) chapter.

Cultural and Linguistic Competency (CLC) remains a bedrock of any healthy system of care. Family CARE Cultural and Linguistic Competency (CLC) efforts focus on increasing awareness and access to care for multiple populations, with a primary focus on the Latino population. Six local partners completed a second CLC organizational self-assessment this past spring. Through a CLC Training Partnership, these organizations are currently developing organizational CLC plans. CLC plans will drive collaborative initiatives to increase access to care and strengthen the diversity and skill set of our workforce. Next spring, we will begin our third assessment process, with an increased focus on community and targeting specific consumer populations.

Challenged in a tough fiscal environment, parents and service providers are tasked to ensure that the most efficient and effective models of support and services are employed. Family CARE continues to support family organizations to ensure access to family-driven support, education and training for parents and youth in a time of limited access to services. Family CARE partners incorporate family-driven strategies into the treatment process, utilizing education and training opportunities as orientations to the system as well as extending them as part of the continuum of support upon discharge. Partners are committed to the sustaining the Family Resource Developer and Parent Wraparound Facilitator roles by linking these roles directly to youth outcomes and moving to a fee-for-service model by December 2010. One partner initiated Family Night, a national model that is used to increase access to care and engage consumers in a more culturally responsive manner. The Integrated Co-Occurring Treatment (ICT) model, a partnership with Juvenile Court Services, is receiving national recognition for its work with youth with an serious emotional disorder and substance abuse disorder. The ICT program was most recently awarded the 2010 iAward for Innovation in Behavioral Healthcare Services. The Incredible Years curriculum is used in several community settings to provide parent skills training. Theraplay is utilized to strengthen the bonding between parents and youth who have experienced significant trauma. Community partnerships engaged with Family CARE also include initiatives with local school districts to promote life skill development and cultural responsiveness. Within the next two years, additional initiatives will focus on the integration of mental health and education services within the community setting.

A significant focus of Family CARE over the last two years, and into the future, is moving our system to a data-driven system of care. An overall Evaluation Plan was developed this year and is used as the foundation for consultation, training and development. Significant work is done to engage partners in routine data collection, analysis and reporting processes. Family CARE invests in the required consultation to develop the required processes, tools and skills required for an effective continuous quality improvement process. The Mental Health Board has incorporated multiple facets of the continuous quality improvement framework into its overall funding and evaluation process, reinforcing the sustainability of local evaluation efforts beyond the cooperative agreement.

Evaluation Activities for McHenry County Family CARE

The evaluation team presently comprises the lead evaluator, a local evaluation research coordinator, a data coordinator, and five part-time parent interviewers. The evaluation team successfully collected enrollment demographic data for 764 youth and assessment data for 169 youth enrolled into the national longitudinal study, as of June 30, 2010. Significant cooperative agreement resources are being utilized to promote study recruitment and retention for the three-year longitudinal study timeframe.

The evaluation team led the development of an Evaluation Plan to guide local evaluation efforts. The local evaluation focuses on developing community processes to identify, collect, analyze and report local data related to on-going Family CARE activities. The evaluator works with program managers and community partners to utilize their logic models and develop outcomes management processes. This is done through specific one-on-one consultations, quarterly continuous quality improvement meetings, a monthly clinical implementation team meeting and submission of periodic program outcome reports. Data are now being used routinely as an integral part of the decision making process. Next steps include more formal analysis of local Child/Adolescent Needs and Strengths (CANS) data.

Additional effort is now being placed on engaging youth and families in the evaluation of the system of care. Youth are being engaged to participate in data collection and together with caregivers are being engaged in quality assurance processes. In particular, parents participate in monthly Local Evaluation Team meetings where program managers share data from various program serving youth and families. National Evaluation data are shared through the Local Evaluation Team, during Governance Council meetings, as well as through a monthly Evaluation Update. The evaluation team worked with local parents to develop local research questions and added several local research questions to the National Evaluation protocol. These questions will be used as a framework for developing focus groups to further explore findings.

Additionally, the McHenry County Wraparound program has initiated a sub-study utilizing the Wraparound Fidelity Index, WFI, to assess fidelity with the National Wraparound Initiative model. The findings of the study will help to assess the extent to which youth and families who receive Wraparound services are effectively engaged in and guide the process and are able to achieve the outcomes established as part of Wraparound planning process.

Preliminary Evaluation Findings for the System of Care McHenry

The National Evaluation provides CQI data on a quarterly basis. Data are extracted from interviews collected at baseline, 6 months, 12 months, 18 months, and 24 months after the initial baseline interviews. These data are used to gain an understanding of the experiences of youth and families.

In a study to ascertain the impact of the services that families received and caregivers' personal advancement, the following correlations were found. The data showed a positive association between youth participation in group therapy and caregivers' increased ability to do their jobs while assessment/evaluative services, family therapy, and flex funds were positively associated with increased caregiver income. None of the other services received made a real difference to caregivers' ability to gain additional education or vocational skills, do their jobs, or to develop more job-related skills. We found that, of the 46 caregivers for whom data were available at intake and at 6 months after intake, 77% received group therapy, 70% received assessment/evaluative services, 39% received family therapy, and 22% received flex funds. These are initial findings and we will be exploring these issues in greater depth as the sample size increases.

A study was also conducted to establish caregivers' perceptions of system of care effectiveness. The study included 35 caregivers' participating in the national longitudinal study and completing surveys at intake and at 6 months. The results showed that caregivers' satisfaction with family life was positively associated with satisfaction with services and getting the help that they needed in caring for the child. Caregivers also perceived increased functioning by youth where service providers persevered in helping them, where there were helped by informal supports, and satisfied with services and family life. No significant relationships were found between caregivers' satisfaction with family life, youth functioning, and cultural sensitivity, accessibility to services and appropriateness of services. This study was presented at the 23rd Annual Children's Mental Health Research & Policy Conference in Tampa, Florida, March 4-7, 2010.

The evaluation team is presently collecting and analyzing data gathered from research questions developed by youth and families. The questions seek to understand how caring for a youth with serious emotional challenges truly impacts the lives of caregivers. To date, data have been collected from 35 caregivers. In response to the questions asked, 50% of caregivers said that they were stressed because of caring for the child quite frequently or nearly always. About a third of the respondents (34%) said that quite frequently or nearly always they do not have time to for themselves because of the time spent with the child with a serious emotional disorder. Also, about 21% of caregivers said that their health suffered quite frequently and nearly always because they have to care for a child with serious emotional challenges.

The Integrated Co-occurring Treatment (ICT) model has been fully developed and tailored to respond to the needs of the local population. The framework which supports the implementation and evaluation of the program was presented at the Georgetown Training Institute in Washington D.C. in July, 2010. As providers look to this model for guidance this framework will inform how other sites develop and implement their ICT programs.

Six local organizations participated in a second organizational CLC self-assessment in May 2010. There were 435 staff, management and board survey responses and 51 consumer responses. Each organization received an individualized report and set of recommendations. Overall recommendations included the following:

Agencies should proceed with the development of a system wide and/or individualized professional development program based on their identified needs.

Agency timetables should be established for the implementation of strategies to remedy the identified concerns.

The revised assessment instruments and collection procedure should be reviewed and modified as needed in order to increase the effectiveness of this research process in the future.

Agencies' overall climate must be open to diversity and remain open to a long-term professional development commitment to cultural competence education.

Agency leadership, including the governance members, must be committed to model and apply cultural competence skills and knowledge.

Each organization is developing an organizational CLC plan based on their assessment results. Primary areas of focus will be on recruitment and retention of bilingual/bicultural staff, awareness and staff training.

System of Care Outcomes

The following outcomes were reported for 35 youth and 50 families enrolled in the longitudinal study as of August 2010.

Improved functioning of youth in academic, social, family, and economic domains (n=35)

School enrollment increased from 71% of youth at intake to 91% at 6 months after intake

Arrest decreased from 31.4% of youth at intake to 20% at 6 months after intake

Secure confinement decreased from 5.7% of youth at intake to 0% at 6 months after intake

Alcohol use decreased from 51.4% of youth at intake to 40% at 6 months after intake

Reduction in anxiety from 50% of youth at intake to 31.3% at 6 months after intake (Revised Children's Manifest Anxiety Scale)

Reduction in depression from 41.2% at intake to 17.5% at 6 months after intake (Reynolds Adolescent Depression Scale-Second Edition)

Academic Performance (n=35)

* Increase in youth academic achievement with 55% of youth achieving Grades A or B at 6 months after enrollment into services, compared to 42% at intake

Improved family status (n=50)

Increase in the number of families able to deal with crises and major problems from 36% at intake to 58% at 6 months after intake

Increase in the number of families engaging in social activities outside the home from 28% at intake to 38% at 6 months after intake

System of Care - McHenry County Evaluation Design

Mary Spooner Ph.D.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process:

Level of incorporation of System of Care principles in service planning, delivery, and governance

System of care compliance with required grant components, such as reports, audits and match funding

Collaboration between system partners in service delivery, planning, finance, and governance

Utilization of evaluation findings to refine and improve the quality, timeliness, accessibility, and efficiency of services

outh and family engagement in all aspects of system of care planning, service delivery, outcomes evaluation

Cultural and linguistic competency of services delivered at all levels of the system of care

Outcome:

Improved functioning of youth in academic, social, family, and economic domains

Improved family status

Decrease in numbers institutionalized

Decrease in service costs

Decrease in child welfare and juvenile justice involvement of system of care youth

Increase in transitional youth employment

Inreased access to mental health care

Increased access to parent and professional training and supports

Increase in cross-system collaboration to facilitate planning, service delivery, financing and evaluation

  • National evaluation (i.e., ORC Macro) protocol
  • Child and Adolescent Needs and Strengths (CANS) assessment tool
  • Local evaluation team meetings
  • Youth and family satisfaction with services
  • Local evaluation surveys to collaborating partners
  • Service data from partner agencies
  • Focus groups
  • CLC needs assessment
  • Wraparound fidelity assessment
  • System of care strategic plan

Participation in national system of care conferences and trainings

Publication of research findings and sharing of system of care outcomes at the local and national levels

A Rural System of Care - Project CONNECT

DHS Contract Mgr Amy Starin, Ph.D.
Annual Grant $

$1,000,000 in Federal FY 10

(varies each grant year)

Evaluator

Name & Contact Info

Jenifer Cartland, Ph.D.

Children's Memorial Hospital, Chicago, IL

JCartland@childrensmemorial.org

Annual Eval $ $200,000 in Federal FY 10
Funding Source SAMHSA/CMHS
Eval Period 10/1/2009 - 6/30/2010

In August, 2009, the Illinois Department of Human Services/Division of Mental Health was awarded $9 million federal grant to be disbursed over 6 years to work in collaboration Egyptian Public and Mental Health Department, Children's Memorial Hospital, the youth, family and child serving agencies (i.e. mental health services, schools, juvenile justice system, child protective services, health care, etc.) to establish a system of care for youth with serious emotional disturbances and their families. The Child Health Data Lab at Children's Memorial Hospital is facilitating the evaluation.

Project Summary for a Rural System of Care, Project Connect -

Project Connect's, mission is to provide a seamless System of Care for the three rural, southeastern Illinois counties that is family-driven, youth-guided, strengths-based, sustainable, culturally and linguistically competent. Our target population lives in a service area that includes three rural counties that are adjacent to each other: Saline, White, and Gallatin counties. These counties are located in the far southeastern portion of the State of Illinois, along the Ohio and Wabash rivers. The counties are rural, with the least populated being Gallatin County. Gallatin has a population density similar to frontier counties.

About 80% of the families of children with serious emotional disturbances currently receiving services are on some form of public assistance and over 60% are from female-headed households. The median income is less than two-thirds of the median income for Illinois.

The three counties have high poverty rates, low levels of adult education, high levels of disability, and Medicaid enrollment. The area is substantially underserved for mental health, with only 10% of the children and youth with serious emotional disturbances receiving special education services; outpatient services are limited, resulting in 10% of youth with serious emotional disturbances being hospitalized in a year. Project Connect will serve all children, birth to 21, in these counties, but will also target three groups that are particularly in need of additional support: (1) youth transitioning to adulthood (16-21). (2) youth receiving special education services, and, (3) youth undergoing major developmental transitions (into grade school, into middle school, and into high school).

Project Connect will serve at least 200 children and youth each year and at least 700 over the course of federal funding. Project Connect will expand access to and speed entry into services implementing universal screening of youth through schools at three points in their K-12 education; Project Connect will expand services by hiring Family Resource Developers and Care Managers to work in concert with school-based social workers and mental health services providers in the community. Project Connect will deepen services by offering evidence-based practices to support youth and family development (such as Wraparound services, parent skills training, and services focused on transitioning to adulthood); Project Connect will increase the quality of services by instituting individual care plans, training all mental health providers in best practices and providing centralized psychiatric assessments; Project Connect will decrease the powerful stigma of mental health conditions by implementing a social marketing campaign which will be culturally appropriate in three rural counties.

The first year of the grant was a planning year. During this year, staff members were hired and trained, buy-in from local youth-serving organizations was obtained, and a governing structure for Project Connect was developed and has become active. The Project Connect Coordinating Council has monthly meetings. A parent was elected as Co-Chair of the Coordinating Council. The following Workgroups have been established: Evaluation Workgroup, Social Marketing Workgroup, Cultural Linguistics Competency Workgroup; and Family-Center Services Workgroup.

The staff has also been working to bring in more partners and to make the community aware of the grant and the need for improved services. For example, the Project Connect Parent Involvement Workgroup is working on ways to raise awareness about mental health and to become connected with other family organizations outside of the region. The Project Connect staff members hosted open houses, were interviewed by the media, and conducted an email campaign to address stigma for Children's Mental Health Awareness Day.

Evaluation of Project Connect

The goal of the evaluation is to determine the extent to which Project Connect is able to achieve its goals of improving access to mental health services and improving the quality of those services for children and youth in the three county service areas. The evaluation plan, which is not yet final, involves participating in the national evaluation for SAMHSA's Children's Mental Health Initiative and developing a local evaluation that will be focused on the needs of Illinois and the region served by the Egyptian Health Department.

The national evaluation involves collecting longitudinal outcomes data for children and youth who receive services through Project Connect. The National Evaluation will be piloted, revised and fully implemented in the fall of 2010.

In keeping with the spirit of family-driven care, the local evaluation is being developed through the Evaluation Workgroup, a team of stakeholders for Project Connect. The Workgroup includes parents of children with mental health conditions, youth with mental health conditions, service providers, and the leadership of Project Connect. Over the last six months, the evaluators have worked with this team to identify key local evaluation questions and develop a logic model to frame the local evaluation plan. The local evaluation plan will be completed in the fall of 2010.

This year, the evaluation team from Children's Memorial Hospital developed a plan to conduct the National Evaluation. The materials and procedures to conduct the National Evaluation were submitted and approved by the CMH Institutional Review Board. In addition, the evaluation team developed a plan for an Evaluation Regional Workgroup that will draw membership from participating youth service agencies. The goal of the Regional Workgroup is to build evaluation capacity within the local youth serving organizations.

Jail Diversion - Trauma Recovery (priority to veterans)

DHS Contract Mgr Debra Ferguson, Ph.D.
Annual Grant $

$411,694 in FY 10

(varies each grant year)

Evaluator

Name and Contact Info

Sue Pickett, Ph.D.

Associate Professor

UIC Department of Psychiatry

Center for Mental Health Services Reseach & Policy

Annual Eval $ $78,362 in FY10
Funding Source SAMHSA/CMHS
Eval Period 9/30/08 - 9/29/2013

The State of Illinois was one of six states awarded the Substance Abuse and Mental Health Services Administration Jail Diversion - Trauma Recovery (Priority to veterans) Grant. This grant, for approximately $2 million over 5 years has enabled the Illinois Department of Human Services, Division of Mental Health (IDHS/DMH) to establish the Illinois Veterans Reintegration Initiative (VRI) to increase diversion for criminal justice-involved veterans with trauma histories in Cook and Rock Island counties. VRI will strengthen partnerships among justice agencies and service providers, expand diversion opportunities, and establish an infrastructure for intervention and service delivery that can be replicated across the State.

Project Summary for the Jail Diversion - Trauma Recovery (priority to veterans) Grant

Veterans in the criminal justice system with mental illnesses and combat-related trauma disorders represent a growing population with unique service needs. Critical barriers to successful reintegration for this population include lack of interface between veteran, justice, and treatment systems and lack of access to dedicated services such as mental health and substance abuse treatment, housing, and trauma-informed treatment. In Illinois, the paucity of military base communities amplifies the need for community and systems-level responses to support this population. The significant number of returning veterans to Illinois also underscores the importance of adapting current training and treatment strategies to meet the needs of returning soldiers and their families. Without these services, veterans with mental health disorders or co-morbid substance abuse may lack the supports necessary to achieve successful reintegration, and find themselves caught in a cycle of homelessness, hospitalization, and incarceration.

IDHS/DMH has taken a leadership role in developing significant statewide initiatives for justice-involved individuals with mental illness at every stage, including street-level intervention, jail diversion, correctional programming, and offender reentry. IDHS/DMH has been instrumental in developing integrated processes of identification, reentry linkage, and service delivery between the criminal justice, mental health and substance abuse networks, and recovery support services, such as housing and employment. These efforts have laid the groundwork for a more comprehensive and effective diversion approach based on leveraging existing successful intervention models, enhancement of capacity, and increased availability of clinically appropriate services.

The VRI evaluation has two primary goals: (1) to assess the development and implementation of the statewide infrastructure of services for veterans with trauma histories who are involved in the criminal justice system; and (2) assess how the pilot programs in Cook County and Rock Island County meet the needs of this target population. The first goal involves identifying the stakeholders who participate in the development and implementation process; what goals and action steps are addressed and achieved in project meetings; and stakeholder satisfaction. The second goal focuses on client outcomes, and involves administering in-depth interviews to VRI clients at program enrollment, 6 months post-enrollment, and 12 months post-enrollment. Interviews assess trauma, military and criminal histories, mental health symptoms, housing, social support, and service use and satisfaction.

This year's efforts have been greatly enhanced by the passage of new legislation is support of the development of veterans' courts statewide. This is a significant support to our jail diversion efforts as counties, in additional to Cook and Winnebago, develop these specialty courts. Three cycles of veteran-specific Crisis Intervention Teams (CIT) training have been offered to officers within the Chicago Police Department and approximately 100 officers trained to date. The CITs are specially trained police officers that respond to mental health issues on the street. One training has been held in Rock Island with local and Quad City Law Enforcement personnel with 12 officers trained to date. The program Linkage Specialist screens veterans from the Cook County veterans' court for program eligibility and refers them to one of the area treatment providers based on geography. Rock Island now screens individuals from their mental health court and from the local homeless shelter (which services a large veteran population).

Evaluation Activities for the Jail Diversion - Trauma Recovery (priority to veterans) Grant

A team from the University of Illinois at Chicago's Center conducts the evaluation component of the VRI grant for Mental Health Research & Policy. This team, headed by Dr. Susan Pickett, is tasked with gathering and analyzing the project data and reporting findings to both the local partners and to SAMHSA.

In FY2010, the VRI evaluation achieved the following goals:

UIC Institutional Review Board (IRB) approval was obtained in October 2009. No data collection is possible prior to UIC IRB approval.

A federal Certificate of Confidentiality was obtained from SAMHSA in December 2009. The Certificate of Confidentiality protects the privacy and confidentiality of veterans who participate in the evaluation.

Dr. Pickett and her evaluation team trained all VRI service staff on the evaluation component, including eligibility criteria, and screening and enrollment procedures.

All evaluation team members completed mandatory training on the National Outcomes Measures (NOMs) data collection tools required by SAMHSA.

VRI service sites began enrolling clients in May 2010. As of August 10, 2010, 11 individuals have been screened for eligibility for VRI services.

Four individuals have enrolled in VRI services. Three of these clients have agreed to participate in the evaluation.

Initial (baseline) interviews have been completed with each of these 3 VRI clients who are enrolled in the evaluation.

An additional 4 individuals are being screened for eligibility by VRI service staff.

IDHS Division of Community Health and Prevention

Illinois Steps AHEAD (GEAR UP)

Program Summary

DHS Contract Mgr Karrie Rueter
Annual Grant $ $3,500,000

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Nancy Flowers, B.S.

Center for Prevention Research and

Development

University of Illinois at Urbana-Champaign

Annual Eval $ $300,000
Funding Source US Department of Education
Eval Period 10/1/05 - 9/30/11

Program Summary

The Illinois Steps for Attaining Higher Education through Academic Development (AHEAD) provides early intervention educational services for middle and high school students and post-secondary educational scholarships. This initiative has been implemented across the state of Illinois in 21 communities since 2006. The fundamental purpose of Illinois Steps AHEAD is to increase the number of low-income students that attend and succeed in college. Program components include; early intervention educational services, creative and engaging academic support for students, career exploration programs, college preparation services, scholarships, increased parent involvement, and increased collaboration with local schools. The Illinois Steps AHEAD program works with students and their families in middle and high school to improve achievement, build study skills, assist in course selection, and increase their knowledge of admission requirements for college education.

Evaluation Activities

The primary goals of the evaluation are to assess the extent to which the Illinois Steps AHEAD program is attaining the expected outcomes and to guide ongoing program improvement efforts through a performance feedback system. The evaluation incorporates a mixed-method approach. The quantitative data elements include surveys collected from program staff, youth, and parents. The qualitative elements include site visits and focus groups with youth and parents. Student academic records are gathered on an annual basis and linked with other data elements to assess program impact. Attendance records at providers are gathered on an ongoing basis from the IDHS eCornerstone database. Evaluation highlights for FY09 include:

  • Research Briefs. Based on analyses of data collected from 2008 focus groups, interviews, and program observations, The Center for Prevention Research and Development (CPRD) completed and distributed a series of research briefs. Topics for the briefs included recruitment and retention practices, family involvement, and program implementation. Each brief includes best practices, resources for providers, and reflects on the impact of the program on participants.
  • Evaluation Trainings. All providers attended one of two evaluation trainings in the fall of 2008. CPRD convened the trainings to discuss the evaluation requirements and timeline for the year, data collection protocols, and data findings.
  • Survey Data Collection. All providers participated in the time 2 survey data collection, including youth surveys, parent surveys, and program staff surveys. CPRD assisted providers with implementing strategies for improving response rates and data quality.

Process Evaluation Findings

Illinois Steps AHEAD has served an average of 1,587 youth per year for the past four years. In FY09, the program provided services to more than 1,900 youth throughout the year; 82% of which were in high school (grades 9 to 12). Fifty percent (50%) of youth report that they attend the program three or more days per week. The core components of the program (tutoring, mentoring, and college advisement) were delivered by all 21 providers. Students received more than 30 hours of tutoring, 10 hours of mentoring, and 13 hours of counseling/advising about college throughout the year. Providers delivered Illinois Steps AHEAD services after school as well as during the school day when feasible.

In addition to youth services, Illinois Steps AHEAD also provides services to parents such as academic planning and college preparation/financial aid counseling in order to provide them with the tools and knowledge to assist their child. Engaging parents in the Illinois Steps AHEAD program, however, remains challenging for providers. Although there are more than 1,900 youth participating in the program, only 900 parents are enrolled. Among these parents, attendance data for only an average of 330 parents per year has been collected. In FY09, these data showed that the most frequently attended services by parents (an average of once per month) were academic counseling and advising as well as family events.

Outcome Evaluation Findings

Academic Services. Academic support services are fundamental to the Illinois Steps AHEAD program. They include activities such as one-to-one or group tutoring, assistance with homework, and academic enrichment. Data collected from focus groups, interviews, and surveys identified the following best practices for implementing the academic component of the program for their ability to address challenges identified by providers and for their potential to increase youth academic outcomes:

? Deliver academic support that goes beyond homework assistance.

The most effective way to improve youth academic outcomes is with a combination of academic assistance services that includes:

  1. Homework help (assistance with assignments and preparing for tests).
  2. Content-based tutoring in mathematics, reading, and science (guided curriculum).
  3. Skill-based tutoring (planned lessons to address areas that need improvement).

It is important for providers to deliver academic assistance that goes beyond homework help. This is particularly critical for students that are performing below grade level. While homework help is very important and can lead to better grades and increased understanding of schoolwork, struggling students need to gain skills and knowledge in areas where they are deficient in order for them to continue to advance their studies. Content-based tutoring and skill-based tutoring use planned and deliberate lessons to address gaps in knowledge and build necessary academic skills for future success.

? Provide one-to-one tutoring to youth.

Increasing the amount of time that youth work individually with a tutor will result in lessons that are more focused and allow greater interaction between the tutor and youth. One-to-one tutoring sessions are more productive because distractions are minimal and the tutor can concentrate on the individual needs of the student.

? Use tutors that have strong teaching skills and deep subject knowledge.

The most effective tutors are those that have strong teaching skills and deep subject content knowledge. Since many providers employ a variety of tutors including teachers, college students, volunteers, and peer tutors, training and supervision by a certified teacher becomes an important issue so that the quality of tutoring remains high. All tutors should receive professional development on effective instructional strategies and be provided with regular information about youth progress.

? Tutors should establish supportive relationships with youth.

Effective tutoring occurs when tutors work consistently with the same youth. Tutors who spend time building a supportive relationship with youth tend to have more success engaging them in productive and interactive tutoring sessions. Youth are more responsive to tutors who show that they care about them and are invested in their success.

? Deliver services at schools.

Providers that deliver services to youth at their school building report more success establishing a meaningful presence for the program. The benefits of the program being at the school include: easier to build relationships with school personnel, gain teacher support and have shared goals for students, easier to get academic data, school personnel have a presence at the program and make referrals to the program.

? Involve tutors and parents in the (Individual Learning Plan) ILP process.

Involving tutors in the ILP (Individual Learning Plan) development process results in tutors that have the information they need to tailor instruction to each individual youth. When tutors know and understand each youth's academic challenges and goals, they are better equipped to address the areas that need improvement. Tutors that regularly review youth weekly assignments, grades on tests, missing assignments, and report card grades can use this information to adjust youth goals and intervene when needed.

College Preparation Services. The college preparation component of Illinois Steps AHEAD provides youth with numerous opportunities to learn about college admission requirements, financial aid, how to apply for college, and what the college experience will entail. Data collected from focus groups, interviews, and surveys identified the following best practices for implementing the college component for their ability to address challenges identified by providers and for their potential to increase youth college readiness:

? Deliver the college preparation component in a multi-faceted way.

The most effective college preparation programs are ones that use a variety of activities and experiences to prepare youth for a successful transition to college. It is important for providers to take a multi-faced approach that addresses academic planning, social and life skill building, knowledge about college and financial aid, and career exploration. College readiness activities can encompass helping youth understand the value of a college education, helping youth make and sustain good grades in high school, assisting with course selection, building skills needed to succeed in college, setting academic expectations and goals, teaching youth and parents about college, visiting colleges, using online enrichment tools, doing practice college entrance exams, assisting with financial aid and scholarship applications, and assisting with college entrance applications.

? Combine college visits with pre-discussion.

When college visits are combined with some pre-discussion of what it takes to get into the college (e.g., test scores, GPA), degree programs, tuition costs, and scholarships available for the college, it prepares both youth and parents for the visit. With prior knowledge of the college, visits are more meaningful and useful because youth and parents are more prepared and can truly assess whether that particular college is right for them. Additionally, every college visit should include a meeting or presentation with college staff about admission criteria in addition to a tour of campus.

? Provide support to parents.

Parents who engage in and receive support from the program are more prepared to assist their child throughout the process of enrolling and succeeding in college. For many parents who either have not attended college themselves or who are not familiar with enrollment procedures and financial aid stipulations, benefit from learning about these critical aspects themselves. They become more knowledgeable and are then able to assist their child. Successful parent activities at providers include seminars on financial aid planning, workshops on the college application process, creating a timeline and task list for college enrollment, and presentations on scholarship opportunities.

Impact on Student Outcomes. When youth were asked in the 2009 survey about their academic performance and study habits since participating in the program, 50% said their grades in school this year are better than last year and 49% described their study habits as being better than last year. In addition, since attending the program, youth strongly agree or agree that they complete their homework more often (87%) and feel more prepared to attend college (86%). Overall, 91% of youth in 2009 report that it is very important to them to graduate from high school (up from 88% in 2007). Although an examination of report card grades showed no overall change among youth from 2008 to 2009, twice as many youth are enrolled in honors/advanced courses. Additionally, report card analyses showed that as youth attitudes about the education (e.g., academic expectations, self-aspirations) improved, their report card grades were more likely to improve as well.

Youth survey data show promising results about the increase in youth knowledge about college. Youth were more knowledgeable about 4-year colleges in 2009 (96%) than they were in 2007 (89%). Youth were also more familiar with the tuition costs in Illinois as well (58% in 2009; 29% in 2007). Youth who attend the program more often were more likely to increase their knowledge about college (e.g., 4-year colleges, tuition costs, scholarships). Among parents, knowledge about college has also increased. While 36% of parents were 'familiar' with how much a college education in Illinois costs in 2007, it increased to 48% in 2009.

Illinois Steps AHEAD (GEAR UP)

Evaluation Design

Peter Mulhall, Ph.D. and Nancy Flowers

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments
  • Increased number of students who enroll in and complete college prep courses (e.g., pre algebra, Algebra 1, etc.)
  • Improved students academic performance (grades, achievement test scores)
  • Improved school attendance
  • Increased rate of grade level promotion
  • Increased number of students who take college entrance exams
  • Increased number of students who enroll in post-secondary education
  • Increased perceptions of access to higher education
  • Improved parent expectations and aspirations related to college
  • Increased knowledge/understanding of college financial options
  • Increased number of completed applications for college financial aid
  • Increased number of college visits
  • Increased number of students who report having an adult to talk to about establishing educational goals, planning for college and/or career
  • Increased educational expectations and aspirations related to college attendance, success, and graduation
  • Improved students' expectations for a productive future
  • Increased parent involvement in their child's education
  • Improved parent educational expectations and aspirations related to college attendance, success, and graduation
  • Improved teacher expectations for academic performance of low-income students
  • Increased contact between schools and families related to college preparation and career planning.
  • Increased type and number of strategies that teachers use to promote educational success
with "at-risk" students
  • Student surveys
  • Parent surveys
  • Program staff surveys
  • Academic records
  • Achievement test scores
  • Program attendance records
  • Site visits and focus group discussions with program staff, collaborative partners, and the students and families served by Illinois Steps Ahead
Training and support is being provided to statewide project staff and local program staff as they engage in the ongoing evaluation process.

Substance Abuse Prevention Block Grant

DHS Contract Mgr Kim Fornero
Annual Grant $ $14,100,315

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Beth Welbes, M.S.P.H.

Center for Prevention Research and

Development (CPRD)

University of Illinois at Urbana-Champaign

Annual Eval $ $821,343
Funding Source SAMHSA
Eval Period 7/1/08 - 6/30/09

Program Summary

The substance abuse prevention evaluation covers programs funded by the Illinois Substance Abuse Prevention and Treatment (SAPT) Block Grant. The SAPT block grant requires that 20% of the block grant funds be spent on prevention activities. In Illinois, the funds are used to support a prevention infrastructure with types of service providers:

  • Comprehensive Grant Program: 103 Comprehensive grants are awarded to not-for-profit organizations/governmental entities. These agencies deliver prevention services and build prevention capacity of coalitions and other organizations at the community level.
  • Statewide Programs: 14 Statewide Programs receive funding to serve large geographic areas or defined target populations.
  • Specialized Services Providers: 8 Specialized Services Providers are funded to target special populations including people with disabilities, youth in alternative educational settings, etc.

The goal of this prevention infrastructure is to prevent the onset of substance use and delay the progression of substance abuse. The primary emphasis is to prevent substance abuse among Illinois youth ages 10-17 through impacting the environments where they live, learn, and grow.

Process Evaluation

During FY09, one hundred three (103) Comprehensive Grant Providers delivered 1312 substance abuse prevention programs that served 230,561 people across Illinois. Fifty-seven percent of the population served was between the ages of 12 and 17 with about three-fourths (72%) under the age of 21. Based on the race and ethnicity categories required by SAMHSA, 20% of the population served was African-Americans and 14% were identified as Hispanic. Geographically, prevention programs throughout the state directly served 43 Chicago community areas, 331 school districts, and 303 municipalities. In addition, prevention strategies were implemented countywide within 90 of the 102 counties in Illinois.

Illinois uses a variety of measures to monitor the performance of Comprehensive Grant programs individually and across the prevention system as a whole. Comprehensive Grant Providers are expected to devote 75% of their contract hours to implement evidence based programming. In FY09, 54% of the Comprehensive Grant Providers met that contract expectation. In addition, the majority (85%) of all programs delivered were considered evidence-based. Evidence-based programming includes model programs, strategies, and practices (for example, mentoring) that are recognized by prevention experts as best practices. It is important to note that these practices, programs and policies include locally-developed efforts, consistent with locally-defined cultures, and meet stated standards of practice that research has found to be "essential ingredients" of effective and successful prevention programs.

To support accountability and ensure that programs are implemented in line with best practice, grant funded programs are evaluated annually. Comprehensive Grant Providers submit annual performance reports that present evaluation data to Illinois Department of Human Services. For direct service programs such as youth prevention education, these reports are reviewed against a set of implementation criteria which include completion rates, fidelity to their program plan, and sufficient levels of staff training. For environmental strategies such as public policy, the reports are reviewed against standards derived from the research and best practice. University of Illinois' Center for Prevention Research and Development conducts these program reviews, known as Program Performance Analysis. The findings from these reviews are communicated in writing to funded agencies within 2 months. These results provide Illinois Department of Human Service staff and prevention managers with critical information on the functioning of funded programs and whether they meet state prevention standards and goals. The attention on "adherence to standards" has focused agency efforts on issues within their control, namely ensuring high quality implementation of programs. In FY09, 82% of the funded agencies had at least one program rated satisfactory or excellent. Agencies that have not demonstrated sufficient adherence to standards are required to submit to Illinois Department of Human Services a Program Improvement Worksheet that details the plans for addressing unmet standards within the first 6 months of the fiscal year.

Outcome Evaluation

From FY03 - FY08, a subset of agencies was selected to participate in multi-year program level outcome evaluation using a quasi-experimental design that employed a matched comparison group. The agencies were selected into this intensive outcome evaluation service based on the results from their annual program reviews. After 6 years of conducting intensive outcome evaluation with 35 Comprehensive Grant Providers, a decision was made to discontinue this protocol due to concerns about the quality and utility of the outcome data. These agencies faced a number of challenges beyond their control which negatively impacted their ability to complete the evaluation and produce reliable results. Virtually all evaluated programs were located in school settings. As a result, both the intervention school data collection (by the program implementer) and comparison school data collection (by the paid comparison school personnel) were heavily influenced by local school personnel and turnover. Barriers to data collection included comparison schools leaving the study early, intervention schools prohibiting data collection at prescribed time points required for the evaluation, implementation staff turnover and reduced sample sizes due to difficulties with tracking subjects over time using anonymous subject tracking methods. Results of outcome evaluations were subject to so many qualifications that decisions could not confidently be made regarding a prevention program's effect.

While intensive program level outcome evaluation was largely phased out by FY09 (one agency initiated an intensive outcome evaluation in FY09 to be complete in FY11), outcome evaluation remains a focus for the SAPT Block Grant prevention system. First, Comprehensive Grant Providers implementing Social Norms Marketing campaigns are required to conduct biannual survey data collection to monitor progress toward campaign outcome objectives. Second, a protocol was established in FY09 to conduct retrospective monitoring of student outcomes using archival data sources. The Illinois Youth Survey (IYS), offered to all schools in Illinois every two years, serves as the primary outcome data source. The Program Performance Analysis database (previously Readiness for Outcome Evaluation database) serves as the primary program implementation data source. Through merging these two existing data sets, trends in substance use outcomes are compared between youth exposed to prevention programming and those not exposed to prevention programming. The results of both the Social Norms Marketing trend analysis and the Illinois Youth Outcomes Surveillance System are reported below.

Social Norms Marketing (SNM) is a prevention strategy based on the theory that youth choose to use alcohol, tobacco and other drugs because they believe use is the norm among their peers. In fact, research has demonstrated that adolescents consistently overestimate both the prevalence of use and the frequency of use. SNM requires that data are collected locally about the actual use rates and that messages are disseminated about the factual levels of use to highlight that use is not normative.

Two Comprehensive Grant Providers collect annual data to monitor the progress of their SNM efforts in three suburban Chicago high schools. Two of the three high schools have a high proportion of Hispanic youth. There are no comparison groups against which to judge the changes observed over time but because census data collection procedures are utilized at the participating schools, changes in alcohol, tobacco and marijuana use are viewed as true population level change which does not require comparison.

During a SNM campaign conducted from 2006-2009, there was a 3% decrease in smoking and a 1% decrease in alcohol use observed among students in one high school.

During a SNM campaign conducted from 2004-2009, there was a 3% decrease in smoking and a 1% increase in alcohol use observed among students in another high school. Notably, the campaign targeted these two substances but did not target marijuana use norms. During the same period, there was a 3% increase in marijuana use.

During a SNM campaign conducted from 2004-2009, there was a 7% decrease in alcohol use and a 3% decrease in smoking among students in the third high school.

The Illinois Youth Outcomes Surveillance System has been developed to evaluate the impact of school-based prevention services delivered throughout the state. The most common age range for implementation of youth prevention education curricula is 6th through 8th grade. Illinois Youth Survey data from 2006 and 2008 (the two most recent data points) were merged with annual Readiness for Outcome Evaluation (i.e. measures of program implementation quality) ratings from 2006, 2007, and 2008. In order for a school to be included in the Illinois Youth Outcomes Surveillance analysis, at least 50% of students at 6th and 8th grade levels must be surveyed. This requirement excluded a large number of schools but was intended to improve confidence in reducing the impact of sampling bias on the analysis.

The first analysis was based on a design in which a cohort of youth within a school were compared with youth in the same grade level but from a data point prior to when the prevention program was introduced into the school. Specifically, 8th grade youth who were exposed to prevention programming from 2006-2008 were compared with a previous cohort of 8th grade youth from their school who were not exposed to prevention programming. The prevalence of alcohol use in the past month among 8th grade youth in 2006(unexposed) was compared with the prevalence of alcohol use in the past month among 8th grade youth in 2008 (those who had been exposed to prevention programming for a minimum of 2 years). A positive outcome would be a lower prevalence of past 30 day alcohol use among the 8th graders exposed to SAPT Block Grant prevention services.

Eight schools met the criteria for having sufficient grade level sample sizes and prevention programming that targeted 6th, 7th and/or 8th grade youth from 2006 to 2008.

The prevalence of 30 day alcohol use was lower among youth that received prevention programming compared to youth that had not received prevention programming in six (75%) of the schools. Among these schools, past month alcohol use DECREASED by an average of 8% among 8th graders from 2006 to 2008,

To determine if this trend was a function of declining 8th grade alcohol use rates throughout the state, four schools were identified from the database that received no prevention programming from 2006 to 2008 (e.g. "control" schools). In three (75%) of these "control" schools, past month alcohol used INCREASED by an average of 6% among 8th graders from 2006-2008.

Because the majority of schools with prevention programming demonstrated improvements and the majority of schools without prevention programming demonstrated negative trends, it seems reasonable that the prevention programming played a role in the positive findings.

The second analysis compared trends in past-month alcohol use between youth exposed to prevention programs in 6th through 8th grade and those not exposed to prevention programs from 6th to 8th grade. It is normative for alcohol use to increase as youth mature. A positive outcome would be a lower escalation of use from 6th to 8th grade in the schools receiving SAPT Block Grant prevention services. In addition, a differentiation was made between prevention services rated highly and those rated poorly (per annual Readiness for Outcome Evaluation implementation quality reviews). If alcohol use trends in schools with poorly rated prevention programs more closely mirrored trends in schools with no prevention programming, then the importance of program quality review would be emphasized. In addition, this would validate the use of annual Program Performance Analysis (formerly Readiness for Outcome Evaluation review) as the primary accountability measure for SAPT Block Grant prevention services.

Twelve schools met the inclusion criteria for sufficient IYS data in 6th and 8th grade over the 2006 and 2008 data points. Eight of these schools received high quality prevention programming, starting with 6th graders in 2006 and continuing for a minimum of two years (high quality). Two schools were unexposed to prevention services (unexposed) and two schools were exposed to prevention programming that failed to meet best-practice standards (poor).

The rate of increase from 6th to 8th grade in past 30 day alcohol use among students exposed to high quality prevention programming was 8% whereas the rate of increase among students exposed to lower quality programming was 16%. The highest rate of escalation in past 30 day alcohol use (an 18% increase from 6th to 8th grade) was found among students NOT exposed to any prevention programming.

While these comparisons do not demonstrate the impact of any individual prevention program, the pattern of change suggests that the escalation of alcohol use between 6th to 8th grade is lower among those who receive high quality prevention programming than among the norm (unexposed youth). The best way to monitor delivery of high quality prevention programming is through the annual Program Performance Analysis. The Illinois Youth Outcomes Surveillance System illustrates biannual population-level outcome trends as a supplemental "pulse-check" of system performance.

Substance Abuse Prevention Block Grant

Evaluation Design

Peter Mulhall, Ph.D. and Beth Welbes, M.S.P.H.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

- Increased number of students who enroll in and complete college prep courses (e.g., pre algebra, Algebra 1, etc.)

??Improved students academic performance (grades, achievement test scores)

- Improved school attendance

??Increased rate of grade level promotion

??Increased number of students who take college entrance exams

??Increased number of students who enroll in post-secondary education

- Increased perceptions of access to higher education

??Improved parent expectations and aspirations related to college

??Increased knowledge/understanding of college financial options

??Increased number of completed applications for college financial aid

??Increased number of college visits

- Increased number of students who report having an adult to talk to about establishing educational goals, planning for college and/or career

??Increased educational expectations and aspirations related to college attendance, success, and graduation

  • Improved students' expectations for a productive future
  • Increased parent involvement in their child's education
  • Improved parent educational expectations and aspirations related to college attendance, success, and graduation
  • Improved teacher expectations for academic performance of low-income students
  • Increased contact between schools and families related to college preparation and career planning.
  • Increased type and number of strategies that teachers use to promote educational success

with "at-risk" students

  • Student surveys
  • Parent surveys
  • Program staff surveys
  • Academic records
  • Achievement test scores
  • Program attendance records
  • Site visits and focus group discussions with program staff, collaborative partners, and the students and families served by Illinois Steps Ahead
Training and support is being provided to statewide project staff and local program staff as they engage in the ongoing evaluation process.

Teen Parent Family Services (TPFS)

DHS Contract Mgr Denise Simon
Annual Grant $ $350,000

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Crystal Reinhart, Ph.D.

Center for Prevention Research and

Development

University of Illinois at Urbana-

Champaign

Annual Eval $ $70,000
Funding Source

DHHS/Office of Adolescent Pregnancy

Programs

Eval Period 10/1/08-9/30/09

Program Summary

Teen Parent Family Services is a youth development program that cultivates capacity building for families impacted by adolescent pregnancy. TPFS engages youth in their mid-teens to their mid-twenties. Program participants include young fathers or siblings of adolescent mothers or fathers. Siblings are targeted for service because having a sibling who bears a child in their teen year's places siblings at a higher risk of becoming adolescent parents as well. Each family unit is connected to an adolescent mother who receives state benefits called Temporary Assistance for Needy Families (TANF). The program provides services through four service delivery methods: Individual case coordination (support, advocacy, counseling and mentoring), Individual referral to community service providers, Group workshops, and Special events (community and family activities). Capacity building focuses on the following three key competencies:

I. Educational Advancement

GED preparation courses

High school re-enrollment

Academic support: coordination with schools, parents, tutoring, etc

II. Employment

Job readiness: workforce skills, job search, resume and cover letter writing and mock interviews

Job Placement: part-time employment positions provided through partnership

Volunteer Program (monthly community service project)

III. Personal Development (Direct and referral services)

Mental Health: family counseling and referrals for assessment and ongoing care

Physical Health: nutrition education, sports and fitness activities (yoga, basketball, dance)

Social/Emotional Health: violence prevention education, gender specific youth empowerment groups, parenting education, cultural identity and art education, goal setting/future mapping, self esteem and healthy relationships.

TPFS project is designed to help enrolled families with the necessary support to develop and strengthen family unity. These supports include connecting each family member to community resources and involving them in a variety of programs, customized to their own interests and needs typically focusing on education, employment and personal development.

Evaluation of Teen Parent Family Services - 2009

Dr. Peter Mulhall, Dr. Crystal Reinhart, and the staff at the University of Illinois's Center for Prevention Research and Development (CPRD) work closely with project staff to execute the evaluation plan, discuss data results, and resolve new challenges. CPRD sends a staff member 2-3 times per week to collect data at the comparison site for the project.

The outcome results at this time did find some mixed but interesting differences specified by the evaluation goals and SMART objectives. The overall interpretation of the results must be viewed cautiously due to concerns with sample selection and attrition. The results are presented by the project SMART objectives.

Evaluation Goal: Does the TPFS program enhance self-sufficiency through school completion?

* SMART Objective: Increase the percentage of teen parents, siblings and fathers/partners who have completed high school or received their GED from 55% to 65% by October 2008.

The majority of teen mothers in both locations are currently enrolled in school or a GED program; however, most of these participants have not graduated yet, so the percentages are lower than the SMART objective at this time. Teen moms have high aspirations with both the intervention and comparison group reporting that graduating from high school is very/extremely important at Time 3 (80% vs. 88%, respectively). We also asked extended family members-fathers and siblings-about their goals for education attainment and expectations. There were no differences between Time 1 and Tim 2, but when dividing by participation using the dosage variable for TPFS high and low participation, fathers/partners in the high participation group reported a statistically significant increase in wanting to improve their educational status. Also, siblings who participated in more hours of TPFS activities also placed a greater value on getting education compared to low attendees.

* SMART Objective: Increase the percentage of family members and father/partners who are employed from 20-30% by October 2009.

Although we believe that improving one's education is essential for improving a person's position in life, it may also be appropriate for the fathers/partners and extended family members (including the mothers) to prepare for and eventually find a good job. A comparison of employment status of teen mothers or their participation in job preparation activities was very low, and neither group met the SMART objective. A comparison of father/partners between Time 1 and 2 did show a statistically significant increase that met SMART objective (29%).

We also examined sources of support for teen moms and found that the most frequent source of financial support comes from Temporary Assistance for Needy Families (TANF), a spouse or partner, and other. This was a change from last year when the TPFS group reported significantly less public assistance than the comparison group. Lastly, we found a "dosage" effect whereby fathers/partners who participated in TPFS activities report higher contribution of monetary support than individuals in the lower groups. This may reflect a closer relationship or stronger sense of responsibility to the teen moms by those who were active in the TPFS program.

Evaluation Goal: What impact does the TPFS have on the interaction between the teen parent and her child, the involvement of the father and his child, the grandparents and grandchild?

* SMART Objective: Increase the percentage of parenting/caregiver knowledge and skills of the father/partners and grandparents from 70% to 80% and for the teen parents and siblings from 30% to 40% by October 2009.

The results of the teen mothers' reports of her and the child's father's interactions with their child showed limited results. The overall analysis shows that teen mothers interact with their babies significantly more often than teen fathers (5.5 times vs. 2.0), but no overall differences between the intervention and comparison groups. Based on specific types of parent child interactions, moms in the TPFS group report more telling and reading stories, but the comparison group fathers reported more storytelling than the intervention group. Most types of parent-child interactions showed no differences.

Evaluation Goal: Does involvement in the TPFS project improve goal setting/sense of future of teen mothers and extended family members?

* SMART Objective: Increase the percentage of family members (teen parents, father/partners, siblings) who aspire to get more education or training from 75% to 80% by October 2008.

Although having a strong sense of future does not guarantee completion of school and getting a high quality job, it does provide participants with a sense of resolve and direction that they can build upon. A comparison of the teen mothers self report of personal goals (working hard, making decisions) for the future are generally high (3.73 on 4 point scale), and appear to be on an upward trend. However, there were not differences between the intervention and comparison group. An analysis of siblings' report of sense of future showed increases in both groups, with the higher dosage group starting higher and increasing more than the lower dosage group.

Evaluation Goal: Does the TPFS program reduce the subsequent pregnancies among teen parents?

* SMART Objective: Maintain the statewide TPS target to keep subsequent pregnancy rates for teen parents at or below the 1.3% level.

A major goal and SMART objective for the TPFS program was to keep the subsequent pregnancy rate below the Teen Parent Service rate of 1.3%. Based on Question 43 from the survey, the findings show that TPFS mothers report went from 0 of 12 births (0%) to 2 of 13 births or 13%. The comparison group went from 6 of 25 (19%) to 4 of 35 or 10%. Again, these numbers are very small and unstable, but substantially above the SMART objective. Further investigation into these numbers and differences are needed.

Evaluation Goal: What impact does the TPFS project have on the improvement of immunization rates of child of teen parents?

* SMART Objective: Increase the percentage of child of teen parents with up-to-date immunizations from 80% to 90% by October 2008.

The results of the immunization question showed a wide range of immunization levels reported by teen mothers ranging from 69% to 100%. The only immunization that meets the SMART objective at Time 3 is the Hepatitis B shot in the TPFS and comparison groups (100% and 85% respectively), and the group comparisons show no statistically significant difference. However, when comparing the aggregate number of shots, it does appear that the TPFS group started at a higher immunization rate and remained higher (3.5), while the comparison group started slightly and significantly lower. Overall, the level of immunization is 20-40% lower than would be expected, and may reflect a lack of understanding regarding various shots. We will attempt to understand these numbers and the validity of these questions by examining immunization rates in the Illinois Department of Human Services health care billing system.

At the completion of Year 4, TPFS had a number of challenges related to program operations, participant recruitment and retention, and follow-up data collection. The eight months that the program was on hold due to IRB issues seriously affected all dimensions of project activities. Moreover, it also created significant problems in finding participants in the intervention and comparison groups, which seriously affected the quality of the evaluation.

Despite the limitations of the TPFS evaluation, the implications of these findings show both promising and ambiguous results. The challenges of a project that serves a small group of individual an intensive way, and the challenge of matching teen moms over time, limit the statistical power to detect group differences. However, the addition of a "dosage" variable this year provides an additional way to examine both group and participation level differences. Using the dosage variable, we find that TPFS participants with higher "dosage" report higher educational aspirations, current employment, greater commitment to future goals, and provision of financial support from fathers/partners to teen mothers.

Recommendations

Based on the findings presented in this report, the following recommendations emerged:

Identify a way to better link TPFS participants and teen mothers to determine if extended family members' participation has an impact on teen mom outcomes, and whether these outcomes are influenced by levels of participation.

Examine TPFS participant's views regarding the quality and effectiveness of TPFS services.

Determine factors that results in the retention and long term commitment of TPFS family members.

Teen Parent Family Services (Adolescent Family Life)

Evaluation Design

Peter Mulhall, Ph.D.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Evaluation Objectives for FY08-FY09:

  • Meet or exceed the Teen Parent Services statewide program minimum performance target of 84% for immunizations for Adolescent Family Life Demonstration (AFLD) participants by August 31, 2010. The AFLD immunization percent is 85%.
  • Meet or exceed the AFLD program minimum performance target of 40% for High School/GED completion for sibling and young father/partner participants by August 31, 2010.
  • Meet or exceed the minimum performance target of 75% of family members (TPS teen mothers, AFLD father/partners and siblings) who aspire to get more education or training by August 31, 2010.
  • Decrease the number of subsequent births to teen mothers who have family members in the AFLD project from the statewide Teen Parent Services minimum target of no greater than 4.0% to below 1.0 % by August 31, 2010.
  • Meet or exceed the minimum performance target of 20% employment rate for AFLD participants (grandparents and partners of teen mother/child's father) by August 31, 2010.
  • Meet or exceed the minimum targeted percentage of 70% for family members' survey response indicating that they positively interact with the teen mother's child(ren) by August 31, 2010; meet or exceed the minimum targeted percentage of 30% for teen mother's survey response indicating that, according to her perception, her family members positively interact with her child(ren) by August 31, 2010.

Process indicators:

  • type of services
  • number of services provided
  • participation in core service and supplemental service areas
  • teen and family member participants are surveyed to assess social-demographic background, levels of education, educational aspirations, number and types of positive social interactions with family members, contraceptive behavior, and parental competency/attachment/and commitment
  • increase the percent of babies meeting developmental milestones.
  • Intake assessments
  • Participant service/logs
  • Federally required core surveys for parenting and pregnant teens
  • Family surveys assessing parent-child communication, parenting skills, family support, family-child interaction, future career and educational aspirations
The evaluation team developed a Program Management Information System (PMIS) data base for Teen Parent Family Services which tracks participation, case status, activities, and referrals for external services.

Strategic Prevention Framework- State Incentive Grant

DHS Contract Mgr Kim Fornero
Annual Grant $ $2,350,965

Evaluator

Name & Contact Info

Peter Mulhall, Ph.D.

Beth Welbes, M.S.P.H.

Center for Prevention Research and

Development (CPRD)

University of Illinois at Urbana-Champaign

Annual Eval $ $265,545
Funding Source SAMHSA
Eval Period 10/1/04 - 9/30/09


Program Summary

The Substance Abuse Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) awarded the Strategic Prevention Framework State Incentive Grant (SPF-SIG) to Illinois in October 2004. The goals of the SPF-SIG are to:

Prevent onset and reduce progression of substance abuse, including childhood and underage drinking;

Reduce substance abuse-related problems in communities; and

Build prevention capacity and infrastructure at the state and community levels.

CSAP requires the creation and maintenance of two statewide groups to provide leadership and oversight for the SPF-SIG grant. The Statewide Advisory Council (SAC) consists of agency directors and policy makers who use the data in order to develop, review, and monitor the progress of the state's SPF strategic plan. The Director of the Illinois Department of Alcoholism and Substance Abuse, Theodora Binion-Taylor, chairs this advisory council. Dr. Myrtis Sullivan (Associate Director, Office of Family Health Services, Division of Community Health and Prevention, Illinois Department of Human Services) chairs the State Epidemiological Outcomes Workgroup (SEOW). These two groups, the SAC and the SEOW, provide a platform for leadership and coordination at the state level. The SEOW (epidemiological workgroup) is specifically charged with analyzing the substance abuse indicators data to determine statewide needs and recommend strategies to improve the framework.

In June 2007, the Illinois Department of Human Services (IDHS) awarded 85% of the grant funds to 18 community-based agencies. IDHS intentionally funded some communities with pre-existing coalitions (72%) and some without pre-existing coalitions (28%) that would presumably require more mobilization efforts to implement that SPF. Just over one-half of the funded communities were in urban settings, 28% were located in rural settings, and the remainder were in suburban settings (17%). After a one to two year planning period, grantees began implementing evidence based prevention strategies at the community level.

State Level Evaluation Activities

The state level evaluation has been monitoring progress of the project since its inception in 2004. At least one member from the CPRD evaluation team attends all SEOW and SAC meetings. All SEOW and SAC activities are tracked and monitored with a SPF/SIG Event Tracking Database, developed by CPRD for monitoring all aspects of the SPF/SIG project. This database allows for tracking: attendance at each meeting; decisions reached; level of participation of all stakeholders and leadership involvement; and the decisions reached on controversies/obstacles identified. This database also tracks all key project events, including activities such as the development of the State Epidemiological Profile, completion of the Comprehensive State Plan for Prevention, and process for funding the 18 SPF/SIG communities.

A key focus of SPF-SIG is to change the way state agencies work together to achieve prevention goals. The project expects that the state's prevention agencies will utilize the data and outcomes learned to better strengthen and focus their impact in the SPF-SIG communities. The SEOW has already demonstrated collaborative action by identifying other epidemiological focused organizations and groups throughout the state, compiling a database of prevention resource dollars per youth capita by county, and has begun to develop a process to review data sources for indicators of quality. CPRD provides technical support to the SEOW through the evaluation contract for the SPF.

As of the end of FY09, the goal to build the prevention infrastructure to support a long-term focus and capacity to address substance abuse has been met in several ways. Collecting, synthesizing, and reviewing data to guide policy and programming decisions represents a major shift for Illinois. In FY09, the SEOW produced a document, "Illinois Epidemiology of Alcohol Use," that further explored the trends and patterns associated with three indicators that have been used to track the alcohol misuse priority in Illinois:

  1. Alcohol use in the past 30 days among 12-20 year olds,
  2. Binge drinking, and 3) alcohol-related motor vehicle fatalities. These findings permit state, sub-state, and community-based prevention planners to focus prevention efforts, especially when local youth survey data are not available, and
  3. Alcohol-related motor vehicle fatalities.

Alcohol Related Outcomes in Illinois

Illinois has observed some positive trends in the state-level indicators associated with our alcohol misuse priority since the SPF grant was awarded in 2004. These trends represent improvements in the health and well-being of our youth and communities. We will continue to monitor these trends in 2010 as we report on changes in Illinois resulting from SPF SIG funding and throughout the time period over which Illinois will be implementing the Partnerships for Success State Incentive Grant. The key findings are presented below.

Trends in Key State Alcohol Indicators Based on a 30 day Illinois Youth Survey Results of 6th, 8th, 10th, and 12th Graders from 2006-2008

  • The prevalence of ALCOHOL USE in the past 30 days has DECREASED for all age groups
  • BINGE DRINKING in the past two weeks has DECREASED for all age groups
  • FIRST ALCOHOL USE PRIOR TO AGE 14 has DECREASED among 12th graders
  • SOCIAL ACCESS to alcohol has decreased among 8th, 10th and 12th graders
  • RETAIL ACCESS to alcohol has decreased among 10th and 12th graders

Trends in Alcohol-Related Motor Vehicle Fatalities

Illinois' Alcohol Related Fatal Crashes and Alcohol Related Fatalities have decreased significantly from 2004 to 2009 according to the Illinois Department of Transportation:

Indicator 2004 2005 2006 2007 2008 2009

Alcohol

Related

Fatal

Crashes

472 444 456 459 380 259

Alcohol

Related

Fatalities

531 497 506 508 425 293

Community Level Evaluation: Progress To Date

All communities funded in July 2007 to implement SPF were required to collected youth survey data on an annual basis starting in Spring 2008. For those sites with a baseline in 2008 within the first year of SPF funding (N=4), trends in use, consequences, and contributing factors are reported from 2008-2009. For those sites with a baseline in 2006, prior to SPF funding (N=15), trends in use, consequences, and contributing factors are reported from 2006-2009. One site was not funded until late 2008 and was expected to establish a baseline in 2009. That site has been excluded from all findings reported in this preliminary report. This site will be included in the final evaluation analysis as the final outcome evaluation will be based on a follow-up collected during spring 2010.

Key indicators were selected from the youth survey data to reflect both logical outcomes to track (given state and local priorities) and include relevant SAMHSA National Outcome Measures. In the final report (due Sept 2010), youth survey data from 2010 will be linked with strategy implementation data to provide a comprehensive evaluation of SPF impacts within funded communities.

Community Level Key Indicator Outcome Analysis Results- 2008 to 2009

Alcohol Consumption Trends (Desired outcome is to DECREASE)

Alcohol Use in the Past 30 Days (% who report use in the past 30 days)

* 63% of sites demonstrated a reduction within the 6th-12th grade population as a whole or within a targeted subgroup (e.g., middle school students, high school students). Reductions ranged from 1% to 29%.

Binge Drinking (% who report 5 or more drinks at one sitting over the past 2 weeks)

* 79% of sites demonstrated a reduction within the 6th-12th grade population as a whole or within a targeted subgroup (e.g., middle school students, high school students). Reductions ranged from 1% to 14%.

Alcohol Use in the Past Year (% who report use in the past year)

* 94% of sites demonstrated a reduction within the 6th-12th grade population as a whole or within a targeted subgroup (e.g., middle school students, high school students). Reductions ranged from 1% to 38%.

Alcohol Consumption Overall

The majority of sites (58%) demonstrated a reduction in all three indicators of alcohol consumption (alcohol use in past 30 days, binge drinking, and alcohol use in past year). Reductions in two indicators were found in an additional one-fifth of sites (21%), and 11% of sites reduced one alcohol consumption indicator. Overall, 90% of communities demonstrated reduced alcohol consumption among the total population or a subgroup of targeted youth in at least one alcohol indicator.

Four of the sites with the most sizable reductions in alcohol use and binge drinking are focused on Chicago community areas, representing high concentrations of unique racial/ethnic groups including Latino/Hispanic youth, Asian youth, and African-American youth.

Alcohol Use and Driving Consequence Trends (Desired outcome is to DECREASE)

Self-Reported Driving Under the Influence of Alcohol (% who report they have driven a car or other vehicle in the past 12 months when they had been drinking alcohol)

* 73% of sites demonstrated a reduction within the 10th-12th grade population. Reductions ranged from 1% to 15%.

Self-Reported Riding with a Teen Driving Under the Influence of Alcohol (% who report they have ridden in a car or other vehicle in the past 12 months when a teenager had been drinking alcohol or using drugs)

* 100% of sites demonstrated a reduction within the 6th-12th grade population as a whole or within their high school population. Reductions ranged from 1% to 20%.

Strategic Prevention Framework State Incentive Grant

Evaluation Design

Peter Mulhall, Ph.D. and Beth Welbes, M.S.P.H.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process:

  • Develop a reliable database of substance abuse prevention-related services funded by all state agencies
  • Develop and implement a methodology for data analysis that integrates need with services available across the State of Illinois.
  • Develop a data-driven plan substance abuse prevention in the State of Illinois.
  • Develop a data-driven plan for reducing risk factors, enhancing protective factors, and preventing the initiation, progression and negative consequences associated with substance abuse in each community selected for sub-recipient funding.

Outcome:

  • Reduce use of alcohol in the PAST YEAR among youth age 10-20 by at least 10% within each sub-recipient community.
  • Reduce use of alcohol in the PAST MONTH among youth ages 10-20 by at least 10% within each sub-recipient community.
  • Demonstrate a reduction in at least one alcohol-related problem targeted by each sub-recipient community
  • Enhance collaboration between prevention, intervention, treatment and extended care providers at the state and sub-recipient level
  • Improve the coordination and integration of state data systems to guide decisions about where and how resources should be directed.
  • Demonstrate adoption of or endorsement of the Strategic Prevention Framework among Illinois agencies and organizations committed to substance abuse prevention.
  • Leverage existing state resources to enhance coordinated planning, service delivery, professional development, and evaluation infrastructure for substance abuse prevention.

State Level Evaluation:

  • Meeting Tracking and Key Milestones database
  • State Advisory Council Member Survey
  • State Epidemiogical Outcomes Workgroup Member Survey
  • Key Informant Interviews

Community Level Evaluation

  • Quarterly Grantee Reports
  • Grantee Assessment Reports
  • Grantee Strategic Plans
  • Strategic Plan Ratings
  • Implementation Fidelity Ratings
  • Grantee Implementation and Evaluation Plans
  • Semi Annual Community Level Instrument (CLI) Evaluation data obtained from the national evaluation contractor
  • Coalition Member Surveys
  • OnSite and Phone Key Informant Interviews

The community level grantees are expected to participate in three levels of evaluation:

Local evaluation is a step in the SPF. Funds for local data coaches are allowable expenses. These data coaches are intended to build local capacity to use data to guide decisions.

National cross-site evaluation is implemented through a federal contractor (Westat). Grantees are expected to submit National Outcome Measures annually and complete an online semi-annual progress report called the Community Level Instrument.

Cross-state evaluation is executed by CPRD. the goal of the cross-state evaluation is to provide an overview of what changed in local communities (based on objectives outlined in strategic plans), key accomplishments of community grantees, barriers experienced by community grantees and lessons learned about community level implementation of the SPF. CPRD relies heavily on the data already gathered for the national cross-site evaluation and documents submitted by grantees to DHS per contractual requirements to minimize data collection burden on community grantees.

Strong Foundations

DHS Contract Mgr Denise Simon
Annual Grant $ $500,000

Evaluator

Name & Contact Info

Julie Spielberger, Ph.D., Chaplin Hall at the

University of Chicago

Annual Eval $ $267,000
Funding Source Administration of Children and Families
Eval Period SFY2008-SFY2012

Program Summary

The Illinois Department of Human Services, in collaboration with the Illinois State Board of Education and the Department of Children and Family Services, is implementing Strong Foundations, an integrated state-wide infrastructure to support three evidence-based models of home visitation: Parents as Teachers (PAT), Healthy Families America (HFA), and the Nurse-Family Partnership (NPF).

Early in state fiscal year 2010, the Administration of Children and Families cut funding to Strong Foundations in anticipation of the enactment of the Affordable Care Act in which there is substantial funding for early childhood home visiting. As a result of the reduction in funding, the scope of the evaluation was modified.

Research questions for the evaluation include the following:

To what extent do Strong Foundations partners collaborate and implement an effective state infrastructure to support evidence-based home visiting programs? What are its strengths and weaknesses? What factors affect implementation of the state infrastructure?

Are communities supported and assisted by the state infrastructure in selecting evidence-based programs to meet the needs of families?

Are the home visiting programs being implemented and delivered in a way that is faithful to their program models? What factors affect the fidelity of program implementation?

Are families satisfied with the services they receive?

To address these questions, the evaluation will include (1) a process evaluation to assess the implementation of the state infrastructure and improvements in the operation and impact of local programs and (2) an outcome study of a sample of families participating in these programs.

Illinois Project Launch (IPL)

Project Director Myrtis Sullivan, MD, M.P.H.
DHS Contract Mgr Xochitl Martirosayn
Annual Grant $ $850,000 per year

Evaluator

Name & Contact Info

Chaplin Hall, University of Chicago

Julie Spielberger

In FY2010, The Illinois Department of Human Services, Division of Community Health and Prevention was awarded a PROJECT LAUNCH grant of $850,000 per year for five years for a total of $4,250,000 to promote the wellness of young children from birth to 8 years of age by addressing the physical, emotional, social, cognitive and behavioral aspects of their development.

The purpose of the Illinois Project LAUNCH (IPL) is to integrate behavioral health and primary care to promote child health and wellness. This project will build upon Illinois' Children's Mental Health Partnership and our innovative All Our Kids Early Childhood Networks. The Illinois Project LAUNCH will provide more preventive services for young children, improve the service delivery system and develop the early childhood workforce. At the state level, IPL will enhance the collaboration between the Illinois Department of Human Services (which is responsible for the state's maternal and child health and mental health programs) and the Illinois Children's Mental Health Partnership. At the local level, Illinois Project LAUNCH's goals are to ensure that children maintain physical and emotional health, that families are linked to the services that they need and that children enter school ready to learn. To achieve these goals, IPL will enhance existing developmental screening and services; train health care providers on developmentally-oriented primary care; support an early childhood mental health consultant; coordinate intake and activities among home visiting programs; and significantly expand parenting education. The project will serve 14,000 children each year and an estimated 39,000 children over the life of the project.

The network serves four communities on Chicago's west side: East and West Garfield Park and North and South Lawndale. The population of these communities is nearly equally divided among African Americans and persons of Hispanic descent. There are approximately 28,000 children under nine in the target area. The infant mortality rate and proportion of births to teen mothers exceed city and state rates.

Illinois Project Launch is being evaluated by a local research team and participates in a national cross-site evaluation led by Abt Associates. The local evaluation, which is being conducted by Chapin Hall at the University of Chicago, plans to examine the process, outcomes, and costs associated with the implementation of Illinois Project Launch (IPL) activities and their sustainability.

The process evaluation will assess the implementation of the project plans at the state and local levels. At the state level, the evaluation will examine the extent to which the Illinois Department of Human Services (IDHS) and its partners have implemented an infrastructure that is consistent with the proposed approach and assessed the extent to which the project has effectively supported local programs. This process evaluation will include interviews with members of the IPL State Council and other selected stakeholders working at the state level.

At the local level, the evaluation will examine the extent to which activities to expand the existing network of services, improve service coordination and track service referrals, and develop the capacity of the local workforce were implemented as planned. It will also include interviews with members of the Local Council and interviews with program administrators and front-line staff in selected local programs participating in IPL activities, including primary care providers, parent educators, home visiting programs, and a sample of parents participating in the programs. The evaluation will also use data from Cornerstone and other administrative records to assess program performance and service quality.

Chapin Hall staff will participate in and observe the State and Local Councils formed to lead IPL. Chapin Hall will periodically present data and findings to this group, and members of the group will also be invited to read and respond to drafts of evaluation reports produced by Chapin Hall. This ongoing participation in the research process will provide a mechanism for bi-directional information exchange, where findings are discussed and incorporated into successive activities of both the program partners and the evaluation.

The table on the following page summarizes the main questions for the local evaluation, the sources of information, and a tentative data collection schedule. In addition, Chapin Hall will be collecting and tracking data for federal reporting requirements and the cross-site evaluation. Specifically, we will be tracking:

Number of collaborative partners participating in IPL at the state and local levels

Number of partners that provided screenings and/or assessments

Number of children receiving developmental screenings

Number of providers and caregivers trained in evidence-based or other informed practices that promote healthy child wellness

Number/percentage of council members who are consumers/family members

Number of grantees using evidenced-based mental health practices

Number of children receiving evidenced-based mental health-related services

Number of children referred to mental health or related services

Number of families participating in parenting and leadership trainings

IPL Local Evaluation Summary Table

Evaluation Component and Questions Data Source(s) and Timeline
State and Local Systems and System Changes
  • To what extent do Project Launch partners collaborate and implement an effective local infrastructure to support high quality accessible services?
  • What are the components of the infrastructure?
  • Who are the partners?
  • How closely does implementation of state and local infrastructures match the plan?
  • What changes in the plan occur and what factors led to these changes?
  • How might deviations from the planned intervention and capacity have impacted network and program outcomes?
  • How well did the enhanced collaboration between agencies at the state level support and improve the availability and coordination of prevention and health promotion services in the communities?
  • Semi-structured interviews with state and local agency staff (Fall 2010, 2011, and 2012)
  • Surveys with program staff and providers (tentatively Fall 2010 and Fall 2012)
  • Observations of Initiative activities (meetings, trainings, work groups, etc) (Fall 2009 - Fall 2014)
  • Analysis of documents (Fall 2010 -Fall 2014)
Program Implementation in 5 Enhancement Areas
  • Are the IPL services and programs (home visiting/referral database; integration of mental health into primary care settings Enhanced Developmentally Oriented Primary Care - EDOPC; parent education/leadership and family strengthening; child development/screening, mental health consultation) being implemented according to their program model or plan?
  • What factors affect program implementation?
  • Are agencies and families engaged in IPL activities as expected?
  • For example, are partner agencies participating in the referral database system? Are the referral follow-up procedures being followed
  • What factors facilitate or hinder engagement of agencies or families?
  • What changes in knowledge, behaviors, and practices occur for IPL participants (agencies and families)? For example:
  • Does EDOPC training result in increases in provider knowledge? Increases in the numbers of children screened?
  • Semi-structured interviews with state and local agency staff (Fall 2010, 2011, and 2012)
  • Surveys with program staff and providers (tentatively Fall 2010 and Fall 2012)
  • Observations of Initiative activities (meetings, trainings, work groups, etc) (Fall 2009 - Fall 2014)
  • Analysis of documents (Fall 2010 - Fall 2014)
  • Analysis of records and administrative data, including data from Cornerstone and e-Cornerstone (Fall 2010 - Fall 2014)

IDHS Division of Alcoholism and Substance Abuse

Access to Recovery - II (ATR-II)

DHS Contract Mgr Theodora Binion-Taylor
Annual Grant $ $4,636,800

Project Manager Name and

Contact Information

Rex Alexander

Rex.Alexander@illinois.gov

Contact for Copy of the Evaluation

Data Analysis Report

Richard Sherman, Ph.D.

Richard.Sherman@illinois.gov

Annual Eval $ $42,580
Funding Source SAMHSA/CSAT -Tl16845
Evaluation Period 9/30/2007 - 9/29/2010

Interim data analyses are described in the Interim Summary of Project Data Collection and Reporting Activities, April 2010.

Background. In September 2007, the Illinois Office of the Governor received a Notice of Award (NoA) in response to the Access to Recovery II (ATR-II) cooperative agreement application that was submitted by Illinois to the Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment (CSAT). ATR-II funding awards were made to 18 states, 5 Native American tribal organizations, and the District of Columbia. The original ATR-II application submitted by Illinois proposed serving 9,000 clients over a 3-year period, at a requested CSAT funding level of $7 million for each of 3 project years. On August 29, 2007 a facsimile transmission was received from CSAT stating that the Illinois application was being considered for funding but that the Year 1 award would be reduced to $4,636,800. There was a stipulation that at least $556,633 of Year 1 funds be spent in support of treatment and recovery support services for clients with methamphetamine abuse issues. It was further indicated in this transmission that there would be an expectation to serve at least 6,210 clients over the three years of federal funding, with the following annual client intake targets: Year One - 1,164; Year Two - 2,523; and, Year Three - 2,523. As required in this transmission, a response was returned to CSAT indicating agreement with these conditions. Following receipt of the initial Notice of Award, a request was made to CSAT that the State of Illinois, Office of the Governor would transfer programmatic and fiscal management responsibilities for the Illinois ATR-II project to the Illinois Department of Human Services, Division of Alcoholism and Substance Abuse (IDHS/DASA). This request was approved by CSAT.

Project Summary. As proposed in the original application to CSAT, Illinois ATR-II supported services are being continued in Cook County and the 5th and 6th Illinois Judicial Districts, and expanded to the 10th District in west central Illinois. The total number of unduplicated clients that will be enrolled in the Illinois ATR-II supported services is projected to be at least 6,210 individuals. As originally proposed, the continued Illinois ATR Program includes a methamphetamine treatment component that is targeted to residents of the 11 counties in east central Illinois that comprise the 5th and 6th Judicial Districts. Early in Year 2, a change in scope request was made to CSAT to expand Illinois ATR-II activities to additional judicial districts in the southernmost areas of the state. This request was approved by CSAT and is designed to assist in the identification of methamphetamine-involved clients.

The Illinois ATR-II screening and enrollment processes are structured in a manner that ensures objective and genuine client choice in selection of the providers to whom they will be referred for ATR-II supported treatment and recovery support services. As was the case with the Illinois ATR-I project, a primary emphasis is placed on clients involved with the criminal justice system. Clients of Illinois TASC are a major source of participating individuals within Cook County, as are women involved with the Department of Women's Justice Services within the Cook County Jail. Furthermore, it is anticipated that the majority of clients served through the methamphetamine treatment expansion component will be involved with the criminal justice system. The Illinois System for Tracking Addiction and Recovery Services (I-STARS), a customization of the CSAT WITS data infrastructure, supports a screening, enrollment, referral, vouchering, billing, and services utilization management system. Key partners in this initiative are TASC, Far East Industries (FEI) Inc., the Lighthouse Institute (LI), Family Guidance Centers (FGC), Inc., and IOTA, Inc. TASC, FGC and IOTA function as client enrollment and referral entities. TASC and LI provide recovery management services, to include the conducting of six-month follow-up interviews. TASC and FGC provide client toxicology services. FEI assists in the further customization of I-STARS required for the continued program and on-going maintenance support of the system.

Summary of Key Findings

Status of Progress toward Client Intake Target. This project has an intake goal of enrolling 6,210 clients to the expanded treatment and recovery support services during the 3 years of CSAT funding support. ATR-II grantees were expected to start serving clients at the start of January 2008. Illinois ATR-II enrolled its first client on January 3, 2008. As can be seen from the below table, our project had a 113.8% interim client enrollment rate as of April 1, 2010.

Intake Coverage Report

GFA Program: Access to Recovery 2

Grantee Information

Client

Target

(To-Date)

Intakes

Received

(To-Date)

Intake

Coverage Rate

(To-Date)

Avg. Rate of

All Grantees

in this GFA

State of Illinois - TI019513

Performance Period:

09/30/07 - 09/29/10

4,948 5,631 113.8% 138.0%
Total 4,948 5,631 113.8% 138.0%

Summary of Illinois ATR-II Client Baseline Data. This interim summary of Illinois ATR data collection and reporting activities is based on GPRA data for our project that was downloaded from the CSAT SAIS web site on April, 01, 2010. As was the case in ATR-I, CSAT is allowing ATR-II grantees to count both negative and positive screen clients to their intake targets. Illinois took this approach in ATR-I. In ATR-II however, Illinois has decided to count only positive screen clients to the project's intake target. A positive screen client is an individual who screens positive for a substance use problem (either currently using or in recovery), is indicated to be in need of treatment and/or recovery support services that are funded through Illinois ATR-II, chooses among alternative providers of the indicated service(s), and accepts a referral to the selected service provider(s). Baseline GPRA item responses are entered into I-STARS, and subsequently uploaded to the SAIS web site only for clients who satisfy these criteria.

Provided below is a summary of the baseline GPRA item responses for the 3,258 positive screen clients enrolled by Illinois ATR-II who were represented in the April 1, 2010 download. This count includes two duplicated clients that are included in the below analyses:

  • Nearly 75% of persons thus far enrolled in Illinois ATR II are male, nearly 80% are African American, 16.7% are white, and about 5% report being of Hispanic/Latino ethnicity. Mexican is the most commonly indicated ethnic group among clients who report that they are Hispanic/Latino, followed by Puerto Rican. The clients thus far enrolled are on average 37.4 years of age.
  • About 6.0% of clients thus far enrolled met the CSAT criteria of being methamphetamine clients.
  • Nearly 30% of the clients reported that they had been living in their own home or apartment during the past 30 days prior to enrollment, about 35% reported living in someone else's home or apartment, 13.5% reported living in a shelter, and 18% reported being in a halfway house or residential treatment.
  •  Over 70% of the clients report that they have children and 4.0% of the clients reported that they have children living elsewhere due to a court order.
  • About 5% of the clients reported at time of project enrollment that they are currently enrolled in training or school, and over 90% state that they are not currently employed.
  • About one-third of the enrolled adults report that they do not have a high school diploma, and an additional 41.2% of the clients state that they have no further educational experience or training beyond the completion of high school.
  •  A little over 45% of the clients report having attended self-help groups through a non-Faith-Based Organization (FBO) during the 30 days prior to enrollment, nearly 20% report attending groups through an FBO organization during this time period, and 11.0% reported participation in other activities that support recovery during this time.
  • Nearly 90% of clients report having had contact with family and/or friends who support recovery during the 30 days prior to enrollment in Illinois ATR.
  •  Less than 7% of the clients reported that there is no one who they turn to when they are in trouble, and 62.8% of the clients report that they primarily turn to family members.
  •  Nearly 45% of the persons enrolled in Illinois ATR II report being currently on probation or parole, and about 7% stated that they were awaiting trial or sentencing.
  •  The clients enrolled in Illinois ATR II reported generally low average levels of alcohol and other drug use during the 30 days prior to enrollment, and as a group have relatively high rates of recent abstinence from alcohol and other drugs. This reflects a pattern that was observed in Illinois ATR I. This can be attributed to the primary focus of Illinois ATR II on persons who are in recovery.

Summary of Client Outcome Analyses. Illinois ATR II clients first came due for six-month follow-up on June 3, 2008. A total of 3,812 six-month follow-up interviews were entered on the CSAT SAIS web site as of April 1, 2010, for the 4,041 clients who had come due for follow-up. This converted to an interim six-month follow-up rate of 94.3%. The cumulative six-month follow-up rate of all ATR II grantees on April 1, 2010 was 64.7%. The results of an interim analysis of baseline (project enrollment) and six-month follow-up GPRA item comparisons for these 3,812 clients are summarized below.

  • There was a statistically significant increase from enrollment to six-month follow-up in the percentage of interviewed clients who reported being employed either part-time (Enrollment - 4.1%; Follow-up - 12.2%) or full-time (Enrollment - 3.3%; Follow-up - 13.5%). There was a statistically significant increase in reported average income from wages among the interviewed clients from enrollment ($106.03) to six-month follow-up ($271.15).
  •  There were statistically significant decreases from enrollment to six-month follow-up among the interviewed clients in average days of alcohol, cocaine, marijuana, heroin, methamphetamine, and any illegal drug use during the previous 30 days.
  •  There were statistically significant increases from enrollment to six-month follow-up among the interviewed clients, in abstinence rates from alcohol, any illegal drugs, both alcohol and illegal drugs, cocaine, marijuana, methamphetamine, and heroin.
  •  There were statistically significant decreases from enrollment to six-month follow-up among the interviewed clients in average days of reported depression, serious anxiety, hallucinations, trouble concentrating or remembering, and trouble controlling violent behavior.

Voucher Information and Transaction Data. ATR II grantees are also expected to enter voucher information and transaction data onto the CSAT SAIS web site within 7 business days of being collected. Most client vouchers are created at time of enrollment into Illinois ATR II. Each voucher is specific to a particular service provider that is chosen by the individual client. The voucher can include multiple service types if these services are to be provided by the same organization. The voucher is assigned a funding level based upon the service rates and time period caps. The organization can bill against the established voucher amount based upon services provided to the client. It is also indicated whether the voucher is currently active. For vouchers that have been closed, date of closure is indicated and the amount of the voucher reflects the total amount billed against it rather than the amount assigned at time of creation. Below are comments that summarize the voucher information data that was downloaded from the CSAT SAIS web site on April 1, 2010. These comments also include the results of analyses of this data that were conducted by DASA.

  • 18,267 vouchers had been uploaded as of this date to the SAIS web site for the clients enrolled in Illinois ATR-II. The total redeemable amount of these issued vouchers was $10,715,464.
  • Based on analyses conducted by IDHS/DASA, the average duration between voucher issuance and entry on the SAIS web site was 18.0 days. The median duration for this interval was 7.0 days, which can be interpreted to mean that about 50% of the created vouchers were entered on the web site within 7 days or less time. It should be noted that these measures are in calendar days rather than business days. It can also be assumed that these measures were also influenced by the project's start-up period. More importantly, it was determined that IDHS/DASA=s VMS maintenance contractor experienced periodic delays in voucher information uploads to the SAIS web site that were attributed to work that needed to be conducted to accommodate changes in upload specifications.
  • As of April 1, 2010, 99.0% (18,079) of the vouchers created had been redeemed in that at least one transaction was billed against the voucher. The total amount redeemed against the created vouchers was $9,915,479.
  •  As of April 1, 2010, 93.8% (17,323) of the vouchers created up to that point had been cancelled or closed. These decisions are made primarily as a result to clients reaching their service billing caps, the service being otherwise completed, or clients having ceased participation in service delivery at the organization that was issued the voucher. The total amount that had been billed against these closed vouchers was $9,178,332. On this date, there were 1,045 open vouchers on the SAIS web site. The total amount available on these open vouchers was $1,542,899.

Voucher transactions represent billings submitted by providers against created vouchers. The voucher transaction data includes the amounts billed against individual vouchers by service type with an indication of numbers of service sessions or units represented in the billing. There is indication of the provider organization that submitted the voucher transaction, and whether the organization is "flagged" as a faith-based organization. The below comments summarize the Illinois ATR II voucher transaction data that was downloaded from the CSAT SAIS Web site on April 1, 2010.

  •  A total of 135,131 voucher transactions were downloaded from the SAIS web site on April 1, 2010. Based on analyses conducted by IDHS/DASA, the average duration between the transaction date and its entry on the web site was 12.5 days. The median duration between the voucher transaction date and entry on the SAIS web site was 1.0 day. Over 75% of the voucher transactions were entered on the website within 7 calendar days of their submission.
  •  A little more than 48% (65,397) of these voucher transactions were accounted for by faith-based organizations. The total amount redeemed by faith-based organizations was 33.5% ($2,838,242) of the total amount thus far redeemed against vouchers issued by Illinois ATR-II.

Interim Summary of Consumer Satisfaction Survey Responses. Methods were implemented to obtain participating client perceptions of, and satisfaction with, the services they receive through the Illinois ATR II Project. Participants are asked to complete a satisfaction tool as part of their initial enrollment process. This tool asks about their perceptions of various aspects of the enrollment process. Participants are asked to complete a second satisfaction tool when they are contacted for six-month post-enrollment follow-up. This second tool asks about their satisfaction with aspects of the services that they later received through Illinois ATR II. Each satisfaction tool consists of forced choice items that ask for their degree of agreement with the service-related statements being asked, followed by an opportunity for them to provide written comments about their experiences.

As of the preparation of this interim report, 3,104 clients who completed enrollment interviews had completed this service satisfaction tool. This represents 55.2% of the 5,631 baseline GPRAs entered for this project on the CSAT SAIS web site as of April 10, 2010. About 66% of the clients were male. This item was left blank by 9.4% (293) of the clients. These clients were on average 36.7 years of age, and their race/ethnic breakdown was as follows: African American - 76.1% (2,135); White, Non-Hispanic - 15.7% (488); Hispanic/Puerto Rican - 1.8% - 57; Hispanic/Mexican - 2.0% (61); Hispanic/Other - 0.6% (20); Native American - 0.5% (17); and, Other - 0.8% (26). The race/ethnicity item was left blank by 9.6% (297) of the clients. Below are summary findings from the client satisfaction with enrollment tool responses.

  •  Over 97% of the responding clients strongly agreed that they were treated with respect by enrollment organization staff, and that these staff spoke to them in a way they could understand.
  •  Over 90% of the clients strongly agreed that enrollment organization staff was sensitive to gender, racial and ethnic issues, and nearly 90% responded in the same manner to the statement that they were satisfied with the time it took to complete the screening and enrollment process.
  •  Nearly 90% of the responding clients strongly agreed that the enrollment process was helpful in identifying their service needs, and nearly 90% strongly agreed with the statement that they were able to participate in identifying the kinds of services that they need.
  •  Over 85% of the clients strongly agreed that they were given the information they needed to choose among alternative treatment and/or recovery support providers, and about 80% strongly agreed that they were given a sufficient number of providers from which to choose.
  •  Nearly 90% of the responding clients strongly agreed that the statement that they were satisfied with their choice of treatment and/or recovery support service providers.
  •  Over 85% of the clients strongly agreed with the statement that they know how the issued service voucher works, and a similar percentage of clients strongly agreed with the statement that they were clear about what they needed to do next.

Satisfaction with Services. The first group of enrolled Illinois ATR II clients came due for follow-up in early June 2008. As of the preparation of this interim report, 3,259 clients who completed six-month follow-up interviews also completed the Illinois ATR II consumer satisfaction survey. This represents 84.7% of the clients who have completed six-month follow-up interviews. The Lighthouse Institute completed 92.9% (3,028) of these interviews. The following pages provide a summary of item ratings for the 3,259 completed tools. Responses to the 15 items on this tool are on a four point scale, with a value of 4 indicating the most positive response. It can be seen that the responses to Items 1-15 are generally positive in nature from the standpoint that the average item responses are each over 3.0. Average response values for the 15 items on the consumer satisfaction tool completed at six-month follow-up range from 3.6 (Item 7) to 3.1 (Items 5, 9, 14 and 15). Below are summary findings from the average responses to the 15 items on the consumer satisfaction survey completed at six-month post-enrollment follow-up.

  • Slightly less than 80% of the responding clients rated the services they received as excellent or good, and responded completely or mostly to statements that they were the kinds of services they wanted and were sufficient or effective in helping them deal with their problems and/or needs.
  • Approximately 85% of the clients responded definitely or probably to statements that they would recommend the services they received to others who needed similar help, and that they would seek out these services if they needed assistance again in the future.
  • Approximately 75% of the responding clients responded excellent or good to statements regarding the location of services, professionalism of service provider staff, perceived understanding of their problems, length and frequency of service sessions, and the extent which the service sessions were helpful in resolving their problems or concerns.

Access to Recovery II (ATR II)

Evaluation/Performance Assessment Design

Richard E. Sherman, Ph.D. and Maria Bruni, Ph.D.

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process:

  • NO of Clients Enrolled
  • No of Vouchers Issued
  • No of Service Billings
  • Follow-up GPRA Completion Rate
  • No of Treatment Providers Enrolled
  • No of Recovery Support Providers Enrolled
  • No of Faith-based Providers Enrolled

Outcome:

Abstinence Rates among Participating Clients

Various Changes in Client Functioning

CSAT ATR-II GPRA Tool

Service Voucher Forms

Service Encounter/Billing Forms

Treatment and Recovery Support Provider MOUs and Service Profiles

Client Satisfaction Surveys

Data collection and analysis are focused on ensuring compliance with federal reporting requirements. Data is required to be collected from clients at project enrollment,, discharge from services, and six-month follow-up . Voucher information and transaction data must also be reported to CSAT

Targeted Capacity Expansion and HIV Services

El Rincon Community Clinic

DHS Project Director Theodora Binion-Taylor
DHS Contract Mgr Carolyn Hartfield
Annual Grant $ $500,000

Evaluator

Name & Contact Info

IOTA, Inc.

Emma J. Flowers/773-271-2348

Annual Eval $ $90,000

Contact for a copy of evaluation

reports

Richard Sherman, Ph.D.

773/478-9265

Funding Source SAMHSA/CSAT
Eval Period 9/30/2007 - 9/29/2012

Targeted Capacity Expansion and HIV Services (TCE/HIV) Grant - El Rincon Community Clinic (Chicago). In September 2007, IDHS/DASA was awarded funding of this five-year TCE/HIV project through an application submitted to Center for Substance Abuse (CSAT).

The purpose of this project is to expand and enhance the outpatient methadone treatment (OMT) services that are available to Chicago community areas. The target population of this TCE/HIV project is adult male and female Hispanic/Latino and African American residents of the following three mid-north City of Chicago community areas who are assessed to be in need of outpatient methadone treatment (OMT) services: Logan Square, Humboldt Park, and West Town. To be eligible for project services, male residents of the targeted communities must also be injection drug users or have been released from incarceration in prison or jail within the past two years.

It is projected that 250 minority male and female clients will be served through these expanded and enhanced OMT services during the five years of CSAT funding. IDHS/DASA contracted with El Rincon Community Clinic to provide the expanded and enhanced services supported through this CSAT TCE/HIV grant. The Outpatient Methadone Treatment services provided by El Rincon will be expanded by 75 slots, specifically for Hispanic/Latino and African American adult residents of the targeted Chicago community areas. Included in this project are service enhancements in the form of the evidence-based practice of comprehensive case management services.

IOTA, Inc. was contracted to perform this project's evaluation, under the direction of Richard E. Sherman, Ph.D. IOTA, Inc. is an African American female-owned corporation specializing in a range of program development and support services to human service organizations and agencies. IOTA was incorporated in Illinois in 1992. Two additional IOTA evaluation team members, IOTA President, Emma J. Flowers and Yvonne B. Sherman collaborate in the provision of evaluation data collection, processing, and reporting activities.

Capacity. This project's application contained a statement of goals and objectives that included a proposed cumulative intake target of 250 unduplicated minority male and female clients over the five years of CSAT funding, 50 during each project year. As of April 1, 2010, 186 client intakes had been entered on the CSAT web site. This represents 132.9% of the cumulative target of 140 intakes up to this point. The following is a summary of baseline characteristics of the minority male and female clients who have thus far been admitted to treatment through this TCE/HIV project.

  • Forty percent (39.8%) of the clients thus far admitted to this TCE/HIV project are female and they are on average 40.0 years of age at time of admission. About 30% of the clients indicated African American as their race.
  • Over 70% of the admitted clients reported themselves to be of Hispanic ethnicity. All but 22 of these 134 clients indicated that they were Puerto Rican.
  • Nearly 80% of the clients indicated that they had previous substance abuse service episodes, with about two-thirds of these clients indicating that they had previous service episodes that were not solely for detoxification.
  • Over 80% of the men indicated Aself@ as their source of referral, and about 10% indicated that another IDHS/DASA-supported service provider as their source of referral.
  • Almost 25% of the clients indicated that they had lived in a controlled environment during some portion of the six months prior to admission.
  •  Less than 10% of the admitted clients indicated that they had been mandated to treatment, and about one-half of these clients indicated the source of this mandate was from some level of the criminal justice system.
  • Over 55% of the clients reported that they had not completed high school.
  • Only three of the admitted clients reported being employed full-time at time of admission and only nine of the clients reported part-time employment. A little over 50% of the clients reported that they were unemployed looking for work, and almost 25% reported that they were unemployed not looking for work.
  • The admitted clients had an average income during the past 30 days from wages of less than $115, and an average income during the past 30 days from illegal sources of over $520.
  • The clients reported an average of over $1,100 that they spent on illegal drugs during the 30 days prior to admission.
  •  A little less than 45% of the clients reported living in their own home or apartment and all but nine of the remaining clients reported living in someone else's home or apartment.
  •  A little less than 20% of the clients reported significant problems with their spouse or partner during the 30 days prior to admission. Lesser percentages of the admitted clients reported significant problems with other family members and friends during the 30 days prior to admission.
  •  Over 20% of the clients reported that they were currently living with someone who was a Asubstance abuser.@
  • The majority of admitted clients reported some level of feelings of stress, reduction in important activities, and emotional problems during the 30 days prior to admission that they attributed to their substance use.
  •  Nearly 50% of the clients rated their current health as "fair" or "poor."
  • Over 60% of the clients stated that they had injected drugs during the 30 days prior to admission, and about 15% stated that they had used "works" that had been used by others.
  •  Over 25% of the admitted clients reported being on probation or parole at time of admission and about 10% reported that they were awaiting trial or sentencing.
  •  Over one-third of the clients reported unprotected sexual contacts during the 30 days prior to admission. Seven of the clients reported unprotected sexual contacts with a drug user and 10 reported unprotected contacts with someone "high" on some substance.
  • Relatively few of the clients reported attendance at any type of recovery support group during the 30 days prior to admission. However, over 80% reported interactions with family and/or friends who are supportive of recovery during the 30 days prior to admission.
  • Only five of the clients indicated they had a twelve-step group sponsor at time of admission.
  • Nearly 40% of the admitted clients reported serious depression, nearly 35% reported serious anxiety or tension, and nearly 30% reported trouble concentrating or remembering during the 30 days prior to admission that were not attributable to their substance use. However, less than 8% of the admitted clients reported having taken prescribed psychiatric medications during that time period.
  • Over one-third of the clients reported to some degree being bothered by psychological or emotional problems.
  •  Over 80% of the clients reported having children. There was an average of 2.8 children among the admitted clients. Eight of the clients reported having children living elsewhere due to a court order, and nine reported having lost custody of their children.
  •  The clients thus far admitted to this project reported an average of 25.3 days of heroin use during the 30 days prior to admission. These clients also reported an average of 3.4 days of cocaine use, an average of 1.7 days of marijuana use, and an average of 2.8 days of alcohol use during the 30 days prior to admission.

Effectiveness. As part of this TCE/HIV project evaluation, attempts are being made to administer the expanded GPRA data collection tool at six-months post-admission. This is the follow-up interval required by CSAT for TCE/HIV projects. The first clients admitted to this project came due for six-month follow-up during May 2008. IOTA is responsible for conducting these six-month follow-up interviews. As of this interim report's preparation, a total of 144 completed follow-up tools were received for the 153 clients who had come due for six-month follow-up. This represents an interim follow-up rate of 94.1%. The following is a summary of findings from an interim analysis of comparative client responses on these baseline and six-month follow-up interviews.

Self-sufficiency

  • There was a statistically significant change from admission to six-month follow-up in the employment status pattern among the interviewed clients. About 7% of the clients were employed at admission, but at six-month follow-up nearly 10% reported being employed full-time and about 10% reported part-time employment.
  • There were significant increases from baseline to six-month follow-up among the interviewed clients in average income from wages, and days paid for working during the past 30 days.

Criminal Justice Status

  • $ There was a significant decrease from baseline to six-month follow-up among the interviewed clients in average income from illegal sources.
  •  There was a statistically significant decrease from admission to six-month follow-up in the average amount of money spent by the clients on illegal drugs during the previous 30 days.
  •  There were statistically significant decreases from admission to follow-up among the interviewed clients in average number of crimes committed and average number of days of illegal activity during the previous 30 days.

Social Support/Functioning

  •  There were statistically significant increases from baseline to six-month follow-up in the percentages of interviewed clients who reported attendance at non-Faith Based Organization (FBO) self-help groups, and other groups that support recovery.
  •  There was a statistically significant increase from admission to follow-up in the percentage of clients who reported not being bothered by family problems.

Alcohol and Drug Use

  • There were significant reductions at time of follow-up in the percentages of clients who reported feelings of stress, reduction in important activities, and emotional problems due to substance abuse during the past 30 days.
  •  At time of six-month follow-up clients reported significant reductions during the past 30 days in average days of use of cocaine, marijuana, and heroin as compared to the 30 days prior to admission. There were statistically significant increases in the percentages of clients who reported abstinence from these substances. About 55% of the interviewed clients reported abstinence from heroin during the 30 days prior to their follow-up interviews.
  •  There was statistically significant decreases from admission to six-month follow-up among the interviewed clients in the average number of days of experienced drug problems, and the average number of days of experienced alcohol problems, during the previous 30 days.
  •  There was also a significant reduction from baseline to six-month follow-up in the percentage of clients who reported injecting drugs during the past 30 days.
  •  Among the interviewed clients, there was a statistically significant increase from admission to follow-up in the percentage of clients who felt treatment for alcohol problems was extremely or considerably important.

Mental/Physical Health Status

  • There was a statistically significant decrease from admission to follow-up in the percentage of clients who reported not being bothered by medical problems.
  •  There was a statistically significant increase from admission to six-month follow-up in the percentage of interviewed clients who reported their health status as excellent or very good.
  • There was a statistically significant increase from baseline to follow-up in the percentage of clients who reported being bothered by psychological and/or emotional issues. Although not statistically significant, there were some decreases from baseline to follow-up in average days of various self-reported mental health issues during the previous 30 days. Participation in the treatment process may account for clients being more cognizant of their mental health issues. Nevertheless, the program may want to examine the extent to which mental health issues are addressed during treatment.

Summary of Client Satisfaction Data. Clients who complete their six-month follow-up interviews are also asked to complete a satisfaction with treatment data collection tool. This tool consists of 12 forced-choice items that ask clients to rate the extent to which they agree with various statements made about their treatment experience. An additional forced-choice item asks them to rate their overall perception of the treatment services they have received. Clients are also encouraged to write open-ended comments regarding the services they have received. Through April 1, 2010 responses on 144 of the satisfaction with treatment data collection tool had been entered into a computerized database being maintained by the project evaluator. A summary of client ratings is provided below. The narrative comments provided by clients support these positive ratings.

  •  Over 97% of the clients agreed with the statement that front office staff treated them with respect and were courteous to them. About 95% of clients agreed with statements that the treatment facility was clean and easy to get to.
  •  Over 85% of clients agreed that the initial treatment assessment process was sensitive to race, gender, and ethnic issues, and nearly 95% agreed that they were able to participate in the development of their treatment plan goals.
  •  Over 95% of the clients agreed with the statement that their counselor is helpful and speaks to them in a way they can understand, and over 95% of the clients agreed that staff take the time to explain to them what they need to do next.
  •  Over 90% of clients agreed with the statement that staff are sensitive to race, gender, and ethnic issues.
  •  Over 95% of clients agreed with the statement that they are satisfied with the amount it takes for them to complete visits to the clinic.
  •  Over 95% of the clients agreed with the statement that treatment is helping them, and over 95% of the clients agreed with the treatment that they would recommend these services to others who were in need of similar help.

Targeted Capacity Expansion and HIV Services Grants

(El Rincon and Family Guidance Centers)

Evaluation Design - Richard E. Sherman, Ph.D. , Evaluator

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process Evaluation Questions

  • Was the project implemented as intended?
  • What types of project plan deviations occurred, what led to the deviations, and what effects did the deviations have on the project?
  • What are the staffing, logistical and cost characteristics of the delivered services?

Outcome Evaluation Questions

  • What are the characteristics of the clients who participate in the implementation of the evidence based substance abuse treatment services?
  • What outcomes are evident among the clients who participate in the expanded and enhanced services?
  • What is the durability of observed effects and what individual factors are related to observed effects?
  • To what extent did the project achieve its quantitative objectives for client admission and service delivery? What are the stakeholder group perceptions of project services?

GPRA

Client Satisfaction Survey

Focus Groups

Not Applicable

Strengthening Treatment Access and Retention

State Implementation (STAR-SI)

FINAL REPORT

Project Director Theodora Binion-Taylor
DHS Contract Mgr Peggy Alexander
Annual Grant $ $325,000

Evaluator

Name & Contact Info

Richard E. Sherman, Ph.D.

Richard.Sherman@illinois.gov

Annual Eval $ $28,000
Funding Source SAMHSA/CSAT
Eval Period

9/30/06 through 6/30/10 (Nine-month no-cost

extension)

Project Summary. The Strengthening Treatment Access and Retention - State Implementation Grant (STAR-SI) cooperative agreement awarded to IDHS/DASA in 2006 by the Center for Substance Abuse Treatment was designed to improve the rates of client access to and retention in publicly funded substance abuse outpatient treatment programs in Illinois. The three years of funding to IDHS/DASA from CSAT was scheduled to end on 9/29/2009. A nine-month no-cost extension was approved by CSAT that supported project activities through 6/30/2010. This represents a summary of the final evaluation of this project. Specifically, STAR-SI focused on four goals:

Reduction in waiting time for treatment;

Reduction in client no-shows;

Increased admissions; and

Increased continuation in treatment.

The project model was quality driven, customer centered, and outcome focused. The project had three provider peer networks, one developed in each of the three years of CSAT funding.

Summary of Key Findings The purpose of the evaluation of the Illinois STAR-SI project funded was to assess the project's ability to improve access to and retention in outpatient substance abuse treatment. The evaluation focused on two of the project's major goals:

Identification of strategies and process improvements that will increase rates of access and retention in Illinois outpatient substance abuse treatment programs; and,

Increase substance abuse outpatient treatment program efficiency in Illinois.

Objectives associated with the first of these goals included the formation of three networks of outpatient treatment provider organizations, the first in southern Illinois (Cohort 1), a second in Cook County (Cohort 2), and a third in the final year of the project (Cohort 3) comprised of provider organizations in multiple areas of the state. Additional objectives under the first project goal pertained to the development of a collaboration with the Network for the Improvement of Addiction Treatment (NIATx), local strategic planning and training activities, and the identification of process improvement strategies to be implemented within each established network.

The second Illinois STAR-SI Project goal concerned the achievement of the following system outcomes:

  1. Reduction in no-shows or failure to keep outpatient treatment appointment rates;
  2. Increased client retention rates and length of stay; and,
  3. Increased outpatient treatment admissions.

The CSAT Cooperative Agreement required that all STAR-SI grantees collect and report the following data at baseline and periodic intervals throughout the course of the project. This data is derived from the Government Performance and Results Act (GPRA) data elements that IDHS/DASA submitted to CSAT on a routine basis.

Number of treatment providers participating in STAR-SI implementation;

Number of unique (unduplicated) client admissions;

Length of Client stay (defined as duration between dates of admission and last service); and,

Number of units of service provided between intake and discharge.

Project Year 1 Illinois STAR-SI project services and activities began in collaboration with the following five IDHS/DASA treatment provider organizations located in southern Illinois (Cohort 1).

Community Resource Center

Egyptian Public and Mental Health Department

Franklin-Williamson Human Services, Inc.

Heartland Human Services

Southeastern Illinois Counseling Centers

Midway through Year 1, Heartland Human Services elected to cease participation in Illinois STAR-SI. This provider organization was experiencing work stoppage problems from the beginning of Year 1. This organization's reduced staffing level made it difficult for them to actively participate in the project. Also, they were experiencing problems in meeting the minimum 150 client admissions/year criteria for participation. The other four Year 1 providers remained active in the project. At the start of Year 2, Illinois STAR-SI project services and activities were expanded to include collaboration with the following five IDHS/DASA Cook County treatment provider organizations (Cohort 2).

Breaking Free

Healthcare Alternative Systems, Inc.

Human Resource Development Institute

Pilsen-Little Village Community Mental Health Center

South Suburban Council on Alcoholism and Substance Abuse

The following five provider organizations accepted an invitation to participate in Year 3 of the Illinois STAR-SI project and represent this project's Cohort 3. The Cohort 1 and Cohort 2 provider organizations also remained active in the project during Year 3.

Gateway Foundation, Inc.

Human Resource Center of Edgar/Clark Counties

Renz Addiction Counseling Center

Tazwood Mental Health Center

The McDermott Center.

Updated Access and Retention (A&R) Measure Summary Tables for the Illinois Cohort 1, Cohort 2, and Cohort 3 providers were developed on a monthly basis and distributed to the participating organizations. These tables included monthly levels for each provider specific to the two required measures reported by each STAR-SI state, and the three additional measures on which Illinois elected to report. The final versions of these updated A&R tables were circulated in September 2009. The three most recent months on these spreadsheets were April through June 2009. As was the case with previous spreadsheets the client data for these three attest months were not included in cumulative A&R measure analyses. The reason for this was the lag between the delivery of services and billing entries in DARTS, which was the source of this data. An exception was made in the case of the Cohort 3 providers in order to offer a broader timeline.

The Illinois A&R Summary Table format included data not requested in the template distributed by NIATx and CSAT. One of the additional pieces of data provided was the number of clients represented in each month's calculations. The tables below are among the pages in the Cohort 1, Cohort 2, and Cohort 3 final A&R Summary Table Excel workbooks that were developed and distributed to providers. These tables summarize individual provider and overall Cohort progress toward achieving STAR-SI measure targets over the indicated time periods. Following these tables are summary dot points based on the data in these final tables.

Cohort 1 Providers

Illinois STAR-SI Project - Year 1 Cohort

Average Level I Admissions/Month

State Target - 10% Increase from Baseline

01/2007 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

CRC 28.0 31.0 28.4 0.4 1.4%
EPMHC 19.0 21.0 21.5 2.5 13.2%
FWHS 39.0 43.0 41.0 2.0 5.1%
SECC 62.0 68.0 76.0 14.0 22.6%
Year I Cohort 148.0 163.0 166.9 18.9 12.8%

CRC - Community Resource Center (N=766)

EPMHC - Egyptian Public and Mental Health Center (N=581)

FWHS - Franklin/Williamson Human Services (N=1107)

SECC - Southeastern Illinois Counseling Centers (N=2053)

Provider Baseline Target Monthly Avg Net Change

Percent

Change

CRC 2.3 2.5 2.6 0.3 13.0%
EPMHC 2.8 3.1 3.5 0.7 25.0%
FWHS 1.6 1.8 2.4 0.8 50.0%
SECC 1.8 2.0 1.7 -0.1 -5.6%
Year 1 Cohort 2.0 2.2 2.3 0.3 15.0%

CRC - Community Resource Center (N=766)

EPMHC - Egyptian Public and Mental Health Center (N-581)

FWHS - Franklin/Williamson Human Services (N=1107)

SECC- Southeastern Illinois Counseling Centers (N=2053)

Illinois STAR-SI Project - Year 1 Cohort

Avg. Duration (Days) from Admission to First Clinical Session

State Target - 10% Reduction from Baseline

01/2007 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

CRC (Unit 73) 29.5 26.6 14.1 -15.4 -52.2%
EPMHC (Unit 74) 19.1 17.2 13.5 -5.6 -29.3%
FWHS (Unit 71) 46.7 42.0 20.4 -26.3 -56.3%
SECC (Unit 71) 23.0 20.7 34.3 11.3 49.1%
Year 1 Cohort 34.5 31.1 22.0 -12.6 -34.6%

CRC - Community Resource Center (N=130)

EPMHC - Egyptian Public and Mental Health Center (N=82)

FWHS - Franklin/Williamson Human Services  (N=721)

SECC - Southeastern Illinois Counseling Centers (N=315)

Illinois STAR-SI Project - Year 1 Cohort

Avg. Duration (Days) from First to Second Clinical Session

State Target - 10% Reducation from Baseline

01/2007 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

CRC  (Unit 73) 25.7 23.1 17.6 -8.1 -31.5%
EPMHC (Unit 74) 20.5 18.5 14.7 -5.8 -23.3%
FWHS (Unit 71) 12.1 10.9 13.6 1.5 12.4%
SECC (Unit 71) 24.3 21.9 32.4 8.1 33.3%
Year 1 Cohort 17.8 16.0 18.1 0.3 1.7%

CRC - Community Resource Center (N=25)

EPMHC - Egyptian Public and Mental Health Center (N=186)

FWHS - Franklin/Williamson Human Services (N-619)

SECC - Southeastern Illinois Counseling Centers (N=241)

Illinois STAR-SI Project - Year 1 Cohort

% of Admitted Level 1 Clients/At Least 4 Sessions/First 30 Days

State Target - 10% Increase from Baseline

01/2007 - 03/2009

Provider Baseline Target Monthly Avg Net Change

Percent

Change

CRC 9.4% 10.3% 19.0% 9.6% 102.1%
EPMHC 29.8% 32.8% 47.7% 17.9% 60.1%
FWHS 5.2% 5.7% 23.3% 18.1% 348.1%
SECC 5.4% 5.9% 6.7% 1.3% 24.1%
Year 1 Cohort 9.2% 10.1% 18.2% 9.0% 97.8%

CRC - Community Resource Center (N=766)

EPMHC - Egyptian Public and Mental Health Center (N=581)

FWHS - Franklin/Williamson Human Services (N=1107)

SESCC - Southeastern Illinois Counseling Centers (N=2053)

  • Over the cumulative more than two-year period from January 2007 through March 2009, the four Cohort 1 providers had a combined average of 166.9 client admissions per month, which was a 12.8% increase over the baseline of 148 admissions per month. Two of the four providers had increases that exceeded the target 10% increase over baseline.
  • The four Cohort 1 providers had a combined monthly average of 2.3 service sessions provided to admitted clients during the first 30 days, which was a 15.0% increase from the baseline of an average of 2.0 sessions during the first 30 days. Three of the four providers had a monthly average that exceeded the target 10% increase over baseline.
  • The four Cohort 1 providers had a combined average duration from opening to first service of 22.0 days, which was a 34.6% decrease from the baseline of 34.6 days. Three of the four providers had decreases that exceeded the target 10% decrease from baseline.
  • The four Cohort 1 providers had a combined average duration from first to second service of 18.1 days, which was a 1.7% increase from the baseline of 17.8 days. Two of the four providers had decreases that exceeded the target 10% decrease from baseline.
  • The four Cohort 1 providers had a combined average percentage of 18.2% of admitted clients who had at least four sessions during the first 30 days following opening, which was a 97.8% increase from the baseline of 9.2% of admitted clients. Each of the four providers had increases in percentage of admitted clients with at least four sessions in the first 30 days that exceeded the target 10% increase from baseline.

Cohort 2 Providers

Provider Baseline Target Monthly Avg Net Change

Percent

Change

Breaking Free 48.0 53.0 36.9 -11.1 -23.1%
HAS 72.0 79.0 76.0 4.0 5.6%
HRDI 47.0 52.0 42.2 -4.8 -10.2$
Pilsen Little Village 22.0 24.0 43.8 21.8 99.1%
South Suburban 35.0 39.0 31.5 -3.5 -10.0%
Year 2 Cohort 224.0 247.0 230.4 6.4 2.9%

Breaking Free (N=443)

HAS - Healthcare Alternative Systems (N=912)

HRDI - Human Resources Development Institute (N=507)

Pilsen - Little Village (N=525)

South Suburban Council (N=378)

Illinois STAR-SI Project Year 2 Cohort

Average # of Service Sessions/First 30 Days

State Target - 10% Increase from Baseline

04/2008 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

Breaking Free 2.9 3.2 2.8 -0.1 -3.4%
HAS 1.9 2.1 3.5 1.6 84.2%
HRDI 3.6 4.0  3.4 -0.2 .5.6%
Pilsen-Little Village 5.2 5.7 5.3 0.1 1.9%
South Suburban 2.7 3.0 3.3 0.6 18.2%
Year 2 Cohort 2.9 3.2 3.7 0.8 27.6%

Breaking Free (N=443)

HAS - Healthcare Alternative Systems (N=912)

HRDI - Human Resources Development Institute (N=507)

Pilsen - Little Village (N=525)

South Suburban Council (N=378)

Illinois STAR-SI Project - Year 2 Cohort

Avg. Duration (Days) from Admission to First Clinical Session

State Target - 10% Reduction from Baseline

04/2008 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

Breaking Free (Unit 26) 18.2 16.4 15.0 -3.2 -17.6%
HAS (Unit 10) 26.9 24.2 19.7 -7.2 -26.8%
HRDI (Unit 10) 0.6 0.5 0.7 0.1 16.7%
Pilsen-Little Village 10.0 9.0 8.5 -1.5 -17.8%
South Suburban Council 16.9 15.2 13.9 -3.0 -17.8%
Year 2 Cohort 12.9 11.6 10.6 -2.3 -17.8%

Breaking Free (N=280)

HAS - Healthcare Alternative Systems (N=194)

HRDI - Human Resources Development Institue (N=291)

Pilsen-Little Village CMHC (N=454)

South Suburban Council (N=240)

Illinois STAR-SI Project - Year 2 Cohort

% of Admitted Level I Clients/At Least 4 Sessions/First 30 Days

State Target - 10% Increase from Baseline

04/2008 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

Breaking Free (Unit 62) 10.7 9.6 11.6 0.9 8.4%
HAS (Unit 10) 8.0 7.2 7.0 -1.0 -12.5%
HRDI (Unit 01) 2.4 2.2 6.1 3.7 154.2%
Pilsen Little Village 6.6 5.9 6.0 -0.6 -9.1%
South Suburban Council 8.2 7.4 7.3 -0.9 -11.0%
Year 2 Cohort 7.8 7.0 7.5 -0.3 -3.8%

Breaking Free (N=248)

HAS - Healthcare Alternative Systems (N=184)

HRDI - Human Resources Development Institute (N=166)

Pilsen - Little Village CMHC (N=429)

South Suburban Council (N=226)

Illinois STAR-SI Project - Year 2 Cohort

% of Admitted Level I Clients/At Least 4 Sessions/First 30 Days

State Target - 10% Increase from Baseline

04/2008 - 03/2009

Provider Baseline Target Monthly Avg. Net Change

Percent

Change

Breaking Free 38.3% 42.1% 33.4% -4.9% -12.8%
HAS 18.8% 20.7% 47.1% 28.3% 66.4%
HRDI 48.5% 53.4% 47.8% -0.7%5 -1.4%
Pilsen-Little Village 62.7% 63.3% 63.9% 1.2% 1.9%
South Suburban Council 35.1% 38.8% 47.8% 12.7% 36.2%
Year 2 Cohort 35.1% 38.6% 47.8% 12.7% 36.2%

Breaking Free (N=443)

HAS - Healthcare Alternative Systems (N=912)

HRDI - Human Resources Development Institute (N=507)

Pilsen-Little Village (N=525)

South Suburban Council (N=378)

  • For the cumulative one-year period from April 2008 through March 2009, the five Cohort 2 providers had a combined average of 230.4 client admissions per month, which was a 2.9% increase over the baseline of 224 admissions per month. One of the five providers had an increase that exceeded the target 10% increase over baseline. During this time period the numbers of monthly admissions among most DASA-funded providers were adversely impacted by budget cuts.
  • The five Cohort 2 providers had a combined monthly average of 3.7 service sessions provided to admitted clients during the first 30 days, which was a 27.6% increase from the baseline of an average of 2.9 sessions during the first 30 days. Two of the five providers had a monthly average that exceeded the target 10% increase over baseline.
  • The five Cohort 2 providers had a combined average duration from opening to first service of 10.6 days, which was a 17.8% decrease from the baseline of 12.9 days. Four of the five providers had decreases that exceeded the target 10% decrease from baseline.
  • The five Cohort 2 providers had a combined average duration from first to second service of 7.5 days, which was a 3.8% decrease from the baseline of 7.8 days. Two of the five providers had decreases that exceeded the target 10% decrease from baseline, and a third had a decrease of 9.1%.
  • The five Cohort 2 providers had a combined average percentage of 47.8% of admitted clients who had at least four sessions during the first 30 days following opening, which was a 36.2% increase from the baseline of 35.1% of admitted clients. Two of the five providers had increases in percentage of admitted clients with at least four sessions in the first 30 days that exceeded the target 10% increase from baseline.

Cohort 3 Providers

Illinois STAR-SI Project - Year 3 Cohort

Average Level I Admissions/Month

State Target - 10% Increase from Baseline

01/2009 - 06/2009

Provider Baseline Target

Monthly

Avg.

Net

Change

Percent

Change

Gateway Foundation 94.0 103.0 82.5 -11.5 -12.2%
HRC/Edgar and Clark Counties 18.0 20.0 21.7 3.7 20.4%
McDermott Center 77.0 85.0 148.5 71.5 92.9%
Renz Addiction Counseling Ctr. 23.0 25.0 40.8 17.8 77.5%
Tazwood Mental Health Center 38.0 42.0 37.8 -0.2 -0.4%
Year 3 Cohort 250.0 275.0 331.3 81.3 32.5%

Gateway Foundation (N=495)

Human Resource Center of Edgar and Clark Counties (N=130)

McDermott Center (N=891)

Renz Addiction Counseling Center (N=245)

Tazwood Mental Health Center (N=227)

Illinois STAR-SI Project - Year 3 Cohort

Average # of Service Sessions/First 30 Days

State Target - 10% Increase from Baseline

01/2009 - 06/2009

Provider Baseline Target

Monthly

Avg.

Net

Change

Percent

Change

Gateway Foundation 5.5 3.2 4.0 -1.5 -27.3%
HRC/Edgar and Clark Counties 5.6 2.1 3.7 -1.9 -33.9%
McDermott Center 2.8 4.0 1.6 -1.2 -.42.9%
Renz Addiction Counseling Ctr. 5.8 5.7 3.0 -2.8 -48.3%
Tazwood Mental Health Center 3.3 3.0 2.3 -1.0 -30.3%
Year 3 Cohort 4.4 4.8 2.7 -1.7 -38.6%

Gateway Foundation (N=495)

Human Resources Center of Edgar and Clark Counties (N=130)

McDermott Center (N=891)

Renz Addiction Counseling Center (N=245)

Tazwood Mental Health Center (N=227)

Illinois STAR-SI Project -Year 3 Cohort

Avg. Duration (Days) from Admission to First Clinical Session

State Target - 10% Reduction from Baseline

01/2009 - 06/2009

Provider Baseline Target

Monthly

Avg.

Net

Change

Percent

Change

Gateway Foundation 7.6 6.8 6.9 -0.7 -9.2%
HRC/Edgar and Clark Counties 11.3 10.2 5.0 -6.3 -55.8%
McDermott Center 26.8 24.1 16.8 -10.0 -37.3%
Renz Addiction Counseling Ctr. 10.8 9.7 9.3 -1.5 -13.9%
Tazwood Mental Health Center 19.5 17.6 10.9 -8.6 -44.1%
Year 3 Cohort 13.8 12.4 9.8 -4.0 -29.0%

Gateway Foundation (N=353)

Human Resource Center of Edgar and Clark Counties (N=114)

McDermott Center (N=205)

Renz Addiction Counseling Center (N=155)

Tazwood Mental Health Center (N=156)

Illinois STAR-SI Project - Year 3 Cohort

Avg. Duration (Days) from First to Second Clinical Sessio

Provider Baseline Target

Monthly

Avg.

Net

Change

Percent

Change

Gateway Foundation 6.7 6.0 5.7 -1.0 -14.9%
HRC/Edgar and Clark Counties 10.8 9.7 6.8 -4.0 -37.0%
McDermott Center 9.5 8.6 7.6 -1.9 -20.0%
Renz Addiction Counseling Ctr. 7.8 7.0 6.1 -1.7 -21.8%
Tazwood Mental Health Center 11.3 10.2 11.9 0.6 5.3%
Year 3 Cohort 8.4 7.6 7.2 -1.2 -14.3%

Gateway Foundation (N=309)

Human Resource Center of Edgar and Clark Counties (N=89)

McDermott Center (N=158)

Renz Addiction Counseling Center (N=87)

Tazwood Mental Health Center (N=120)

Illinois STAR-SI Project - Year 3 Cohort

% of Admitted Level I Clients/At Least 4 Sessions/First 30 Days

State Target - 10% Increase from Baseline

01/2009 - 06/2009

Provider Baseline Target

Monthly

Avg.

Net

Change

Percent

Change

Gateway Foundation 71.6% 78.8% 52.1% -19.5% -27.2%
HRC/Edgar and Clark Counties 59.2% 65.1% 50.0% -9.2% -15.5%
McDermott Center 20.3% 22.3% 10.1% -10.2% -50.2%
Renz Addiction Counseling Ctr. 65.3% 71.8% 33.1% -32.2% -49.3%
Tazwood Mental Health Center 41.8% 46.0% 31.3% -10.5% -25.1%
Year 3 Cohort 49.7% 54.7% 28.0% -21.7% -43.7%

Gateway Foundation (N=495)

Human Resource Center of Edgar and Clark Counties (N=130)

McDermott Center (N=891)

Renz Addiction Counseling Center (N=245)

Tazwood Mental Health Center (N=227)

  • For the cumulative six-month period from January through June 2009. the five Cohort 3 providers had a combined average of 331.3 client admissions per month, which was a 32.5% increase over the baseline of 250 admissions per month. Three of the five providers had an increase that exceeded the target 10% increase over baseline.
  • The five Cohort 3 providers had a combined monthly average of 2.7 service sessions provided to admitted clients during the first 30 days, which was a 38.6% decrease from the baseline of an average of 4.4 sessions during the first 30 days. None of the five providers had a monthly average that exceeded the target 10% increase over baseline. This failure to meet the target increase was attributed to the funding cuts experienced by these providers.
  • The five Cohort 3 providers had a combined average duration from opening to first service of 9.8 days, which was a 29.0% decrease from the baseline of 13.8 days. Four of the five providers had decreases that exceeded the target 10% decrease from baseline.
  • The five Cohort 3 providers had a combined average duration from first to second service of 7.2 days, which was a 14.3% decrease from the baseline of 8.4 days. Four of the five providers had decreases that exceeded the target 10% decrease from baseline.
  • The five Cohort 3 providers had a combined average percentage of 28.0% of admitted clients who had at least four sessions during the first 30 days following opening, which was a 43.7% decrease from the baseline of 49.7% of admitted clients. None of the providers had an increase in percentage of admitted clients with at least four sessions in the first 30 days that exceeded the target 10% increase from baseline. This failure to meet the target increase was also attributed to the funding cuts experienced by these providers.

No-Cost Extension Activities. The primary objective of the Illinois STAR-SI nine-month no-cost extension was the following.

Objective. Continue activities to sustain and diffuse STAR-SI principles throughout the State of Illinois as part of IDHS/DASA's transition to performance-based contracting.

Illinois previously requested the one-time use of Year 2 unobligated funds to support implementation of a statewide diffusion of STAR-SI principles and activities across all IDHS/DASA-funded treatment providers. This effort was part of IDHS/DASA's transition to performance-based contracting. This request, which was approved by CSAT, supported statewide diffusion activities that were completed by September 30, 2009. These activities involved the development of a baseline treatment delivery performance report for each IDHS/DASA-funded provider who opened clients in DARTS during calendar year 2007 for Level I (outpatient), Level II (intensive outpatient), or Level III.5 (residential rehabilitation) care. Obtaining this baseline performance measure data was the necessary first step in both diffusing STAR-SI principles and activities statewide, and implementing IDHS/DASA's performance-based contracting plan. This step was completed, reports were distributed to 118 providers, and the contents of these reports were discussed in meeting throughout the state prior to June 30, 2009.

A second round of provider reports was developed during the no-cost extension approved by CSAT. This second round of IDHS/DASA-funded treatment provider reports included calculated levels of performance measures by level of care for clients opened during calendar year 2008, and reported both organization-wide and by individual provider service locations. In addition to offering comparisons to the performance measure calculations from the first round of provider reports, the second round of reports involved expansions in terms of both the levels of care represented, and the scope of the types of measures included.

The following American Society of Addiction Medicine (ASAM) levels of care are represented in the second round of provider reports: Detoxification, Level I (outpatient-drug free), Level II (intensive outpatient), Level III.5 (residential rehabilitation), Level III.1 (Halfway House), and Recovery Home. In addition to this expansion in levels of care represented in the reports, client engagement measures were added to the measures of client access and retention. The client engagement and continuity of care measures were developed with input from provider organization representatives. These individuals were also asked to comment on the performance measures already in the report format. The expanded scope of measures in the second round of provider reports also include client outcome measures in the form of baseline to discharge analyses of National Outcome Measure (NOMS) data that is now required to be submitted by IDHS/DASA-funded providers through DARTS. The NOMS data tables in these reports mimic a format that is part of the annual SAPT Block Grant application submitted by IDHS/DASA to CSAT. The key client access/capacity, engagement, retention, and continuity of care measures in the below table by level of care are represented in the provider reports developed during the STAR-SI no-cost extension. These client service measures provide the basis for IDHS/DASA performance-based contracting decision-making. During this second round, a provider performance report was developed for each of 131 IDHS/DASA-funded treatment organizations. A separate report was developed that contains cumulative levels of the provider performance and outcome measures on a state-wide basis and specific to each of the five IDHS service regions.

Illinois Department of Human Services

Division of Alcoholism and Substance Abuse

Levels of Key Provider Performance and Outcomes Measures

Treatment Clients Opened in Calendar Year 2008: N = 69,560

Statewide 2008

Organization-wide - Access/Capacity

Avg. Duration from Initial Contact to Opening 9.6 days

% (#) of Admissions - No Reported Wait for Assessment 57.1% (38,833)

Detoxification - Engagement

% (#) of Clients/Length of Stay At Least Two Days 93.1% (12,615)

Detoxification - Retention

% of Discharged Detox Clients/Completed Treatment 53.3% (6,536)

Detoxification - Continuity of Care

% of Clients/Linked to Treatment/Same Organization 31.1% (4,219)

Level I - Engagement

% Admitted Clients at Least 4 Sessions/First 30 Days 37.6% (10,993)

Level I - Retention

of Discharged Level I Clients/Completed Treatment 40.6% (9,618)

of Clients/At Least 10 Sessions After First 30 Days 25.7% (7,533)

of Clients/At Least 10 Sessions After First 30 Days AND/OR Discharged as Treatment Completion

4% (13,904)

Level I - Continuity of Care

% of Discharged Clients/12-Step Group Participation 20.8% (3,599)

Level II - Engagement

% Admitted Clients/At Least 12 Sessions/First 30 Days 27.0% (2,797)

Level II - Retention

Admitted Clients/At Least 18 Sessions/First 60 Days 34.5% (3,572)

of Discharged Level II Clients/Completed Treatment 37.5% (3,103)

of Clients/At Least 18 Sessions in the First 60 Days

AND/OR Discharged as Treatment Completion

49.3% (5,141)

Level II - Continuity of Care

% of Discharged Clients/12-Step Group Participation 43.7% (2,303)

Level III.5 - Engagement

% of Clients/Length of Stay of At Least 7 Days 87.6% (12,371)

Level III.5 - Retention

% of Clients/Length of Stay of At Least 14 Days 75.0% (10,592)

Level III.5 - Continuity of Care

% of Clients/Linked to Lower Level of Care/Same Org. 21.7% (3,061)

Level III.1 - Engagement

% of Clients/Length of Stay of At Least 7 Days 90.5% (1,423)

Level III.1 - Retention

% of Clients/Length of Stay of At Least 14 Days 80.1% (1,259)

Level III.1 - Continuity of Care

% of Discharged Clients/Completed Tx/+ NOMS Status 35.6% (466)

Recovery Home - Engagement

% of Clients/Length of Stay of At Least 7 Days 87.5% (1,736)

Recovery Home - Retention

% of Clients/Length of Stay of At Least 30 Days 56.7% (1,125)

Recovery Home - Continuity of Care

% of Discharged Clients/Completed Tx/+ NOMS Status 48.4% (808)

Illinois Strengthening Treatment Access and Retention - State Implementation (STAR-SI)

Evaluation Design

Richard E. Sherman, Ph.D. , Evaluator

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

The Illinois STAR-SI project will collaborate with five treatment provider organizations in southern Illinois.

All STAR-SI states are required to report the baseline and periodic status of the following service and retention measures.

  • Average Units of Service Provided to Admitted Outpatients will measure the average level of treatment participation among admitted outpatients.
  • Average Length of Stay--average number of days that clients remain open in outpatient services.

Illinois has decided to additionally report on the following service access and retention measures.

  • Average Time to Assessment Appointment-from date client first contacts the agency.
  • Average Time from Assessment to First Outpatient Session.
  • Average Time from First to Second Outpatient Session.
  • Early Client Engagement Rate will measure the successful completion of four group or individual sessions within thirty days of admission.
DARTS Not Applicable at this time.

Targeted Capacity Expansion and HIV Services

Family Guidance Centers, Inc.

DHS Project Director Theodora Binion-Taylor
DHS Contract Mgr Carolyn Hartfield
Annual Grant $ $500,000
Evaluation Name & Contact Info

IOTA, Inc.

Emma J. Flowers/773-271-2348

Annual Eval $ $90,000

Contact for copy of evaluation

reports

Richard Sherman, Ph.D.

773-478-9265

Funding Source SAMHSA/CSAT
Eval Period 9/30/2007 - 9/29/2012

Targeted Capacity Expansion and HIV Services (TCE/HIV) Grant - Family Guidance Centers, Inc. (Chicago). In September 2007, IDHS/DASA was awarded funding of this five-year TCE/HIV project through an application submitted to Center for Substance Abuse (CSAT). The purpose of this project is to expand and enhance the outpatient methadone treatment (OMT) services that are available to Chicago community areas. The purpose of this TCE/HIV project is to expand and enhance the evidence-based outpatient methadone treatment (OMT) services that are available to African American and Hispanic/Latino injecting males, to include Men who have sex with other Men (MSM), who are residents of the near north and west side City of Chicago community areas that have high rates of HIV infection and AIDS.

It is projected that 250 unduplicated minority male clients will be served through these expanded and enhanced services during the five years of CSAT funding. IDHS/DASA will function as the grantee and will be responsible for all grant administration requirements. IDHS/DASA has contracted with Family Guidance Centers, Inc. (FGC) to provide the proposed expanded and enhanced services. IOTA, Inc. was contracted to implement an evaluation plan for this grant that includes administration of the CSAT Government Performance and Results Act (GPRA) items at admission, discharge from project services, and six-month post-admission follow-up. The evaluation is being conducted under the direction of Richard E. Sherman, Ph.D. Other IOTA staff assist in the data collection, processing, analysis, and reporting activities described below.

Capacity. This project's application contained a statement of goals and objectives that included a proposed cumulative intake target of 250 unduplicated minority male clients over the five years of CSAT funding. As of April 1, 2010, 152 client intakes had been entered on the CSAT SAIS web site. This represents 108.6% of the cumulative target of 140 intakes up to this point. The following is a summary of baseline characteristics of the 152 male clients who have thus far been admitted to treatment through this TCE/HIV project.

  •  The male clients thus far admitted to this TCE/HIV project are on average 42.8 years of age at time of admission. A little less than two-thirds of the men indicated African American as their race.
  •  About one-third (38.8% - 59) of the admitted clients reported themselves to be of Hispanic ethnicity, with about two-thirds of these clients indicating that they are Puerto Rican.
  •  Eighty percent (80.0%) of the clients indicated that they had previous substance abuse service episodes, with nearly 60.0% of these clients indicating that they had previous service episodes that were not solely for detoxification.
  • Over 60% of the men indicated Aself@ as their source of referral, and about one-fourth indicated a health care provider as their source of referral.
  •  A little more than 15% of the clients indicated that they had lived in a controlled environment during some portion of the six months prior to admission.
  •  Only one of the admitted clients indicated that they had been mandated to treatment, and only one client indicated the source of this mandate was from some level of the criminal justice system.
  • Nearly 40% of the clients reported that they had not completed high school.
  •  Only three of the admitted clients reported being employed full-time at time of admission and only four of the clients reported part-time employment. Over 50% of the clients reported that they were unemployed not looking for work.
  •  The admitted clients had an average income during the past 30 days from wages of less than $65, and an average income during the past 30 days from illegal sources of nearly $380.
  • The clients reported an average of over $900 that they spent on illegal drugs during the 30 days prior to admission.
  •  Nearly 40% of the clients reported living in their own home or apartment and over 50% reported living in someone else's home or apartment. Therefore, the majority of admitted clients reported unstable living arrangements.
  •  A little over 20% of the clients reported significant problems with their spouse or partner during the 30 days prior to admission. Lesser percentages of the admitted clients reported significant problems with other family members and friends during the 30 days prior to admission.
  •  Nearly 20% of the clients reported that they were currently living with someone who was a Asubstance abuser.@
  •  The majority of admitted clients reported some level of feelings of stress, reduction in important activities, and emotional problems during the 30 days prior to admission that they attributed to their substance use.
  •  Over one-third of the clients rated their current health as "fair" or "poor." Over 10% of the clients reported having visited an emergency room for physical health problems during the 30 days prior to admission.
  •  Over 40% of the clients stated that they had injected drugs during the 30 days prior to admission.
  •  Over one-fifth of the admitted clients reported being on probation or parole at time of admission and nearly 10% reported that they were awaiting trial or sentencing.
  •  Over 50% of the clients reported unprotected sexual contacts during the 30 days prior to admission. Three of the clients reported unprotected sexual contacts with a drug user and 20 reported unprotected contacts with someone who was "high" on some substance.
  •  Relatively few of the clients reported attendance at any type of recovery support group during the 30 days prior to admission. However, over 70% reported interactions with family and/or friends who are supportive of recovery during the 30 days prior to admission.
  •  Only nine of the clients indicated they had a twelve-step group sponsor at time of admission.
  •  Nearly 40% of the admitted clients reported serious depression and/or serious anxiety or tension, and nearly 20% reported trouble concentrating or remembering during the 30 days prior to admission that were not attributable to their substance use. However, only six of the admitted clients reported having taken prescribed psychiatric medications during that time period.
  •  About 40% of the clients reported to some degree being bothered by psychological or emotional problems.
  •  Over 70% of the clients reported having children. There was an average of nearly three children among the admitted clients. Six of the clients reported having children living elsewhere due to a court order, and four reported having lost custody of their children.
  •  The clients thus far admitted to this project reported an average of 27.0 days of heroin use during the 30 days prior to admission. These clients also reported an average of 4.5 days of cocaine use, and average of 2.3 days of marijuana use, and an average of 2.3 days of alcohol use during the 30 days prior to admission.

Effectiveness. As part of this TCE/HIV project evaluation, attempts are being made to administer the expanded GPRA data collection tool at six-months post-admission. This is the follow-up interval required by CSAT for TCE/HIV projects. The first clients admitted to this project came due for six-month follow-up during May 2008. IOTA is responsible for conducting these six-month follow-up interviews. As of this interim report's preparation, a total of 122 completed follow-up tools were received for the 136 clients who had come due for six-month follow-up. This represents an interim follow-up rate of 89.7%. Partial results from an interim analysis of Family Guidance client follow-up responses are provided in the below table. This analysis is based on the 122 paired baseline and six-month follow-up interviews that were downloaded from the CSAT SAIS web site on July 9, 2009. One of these interviews was conducted prior to opening of the client's follow-up window. The following is a summary of findings from these interim results. The summary of interim evaluation findings is arranged by GPRA domain.

Self-sufficiency

  •  There was a statistically significant change from admission to six-month follow-up in the employment status pattern among the interviewed clients. Although this change is statistically significant, a greater increase in reported employment at follow-up would be desirable. However, the fact that this is a relatively older client population with a relatively low rate of high school completion, present substantial challenges to attaining improved changes in employment.
  •  There was a significant increase from baseline to six-month follow-up among the interviewed clients in average income from wages during the past 30 days.

Criminal Justice Status

  •  There was a significant decrease from baseline to six-month follow-up among the interviewed clients in average income from illegal sources.
  •  There was a statistically significant decrease from admission to six-month follow-up in the average amount of money spent by the clients on illegal drugs during the previous 30 days.
  •  There were statistically significant decreases from admission to follow-up among the interviewed clients in average number of crimes committed and average number of days of illegal activity during the previous 30 days.

Social Support/Functioning

  •  There was a significant increase from baseline to follow-up among the interviewed clients in regards to the percentage who reported attendance during the past 30 days at self-help support groups and other groups that support recovery.

Alcohol and Drug Use

  •  There were significant reductions at time of follow-up in the percentages of clients who reported feelings of stress, reduction in important activities, and emotional problems due to substance abuse during the past 30 days.
  •  At time of six-month follow-up clients reported a significant reduction during the past 30 days in average days of cocaine, marijuana and heroin use as compared to the 30 days prior to admission. There were also statistically significant increases in the percentages of clients who reported abstinence from these substances. Nearly 40% of the interviewed clients reported abstinence from heroin during the 30 days prior to their follow-up interviews.
  •  There was also a significant reduction from baseline to six-month follow-up in the percentage of clients who reported injecting drugs during the past 30 days.
  •  There was a statistically significant decrease from admission to six-month follow-up among the interviewed clients in the average number of days of experienced drug problems during the previous 30 days.
  •  Among the interviewed clients, there were also statistically significant increases from admission to follow-up in the percentages of clients who felt treatment for alcohol problems, and treatment for drug problems, was extremely or considerably important.

Mental/Physical Health Status

  • There was a statistically significant decrease from admission to follow-up in the percentage of clients who reported not being bothered by medical problems. There was a statistically significant increase in the percentage of clients who felt treatment for their medical problems was extremely or considerably important.
  •  There was a statistically significant increase from baseline to follow-up in the percentage of clients who reported being bothered by psychological and/or emotional issues. Although not statistically significant, there were increases from baseline to follow-up in average days of various self-reported mental health issues during the previous 30 days. Participation in the treatment process may account for clients being more cognizant of these mental health issues. Nevertheless, the program may want to examine the extent to which mental health issues are addressed during treatment.

Summary of Client Satisfaction Data. Clients who complete their six-month follow-up interviews are also asked to complete a satisfaction with treatment data collection tool. This tool consists of 11 forced-choice items that ask clients to rate the extent to which they agree with various statements made about their treatment experience. An additional forced-choice item asks them to rate their overall perception of the treatment services they have received. Clients are also encouraged to write open-ended comments regarding the services they have received. Through April 1, 2010 responses on 112 of the satisfaction with treatment data collection tools had been entered into a computerized database being maintained by the project evaluator. Provided below is a summary of client ratings on this satisfaction with treatment tool. The narrative comments provided by clients support these positive ratings.

  • $ All of the clients agreed with the statement that front office staff treated them with respect and were courteous to them. Nearly 90% of clients agreed with the statement that the treatment facility was easy to get to, and nearly 95% agreed that the facility was clean.
  •  Over 80% of clients agreed that the initial treatment assessment process was sensitive to race, gender, and ethnic issues, and over 80% agreed that they were able to participate in the development of their treatment plan goals.
  • Over 95% of the clients agreed with the statement that their counselor is helpful and speaks to them in a way they can understand, and nearly 95% of the clients agreed that staff take the time to explain to them what they need to do next.
  •  Over 85% of clients agreed that staff are sensitive to race, gender, and ethnic issues, and nearly 95% of clients agreed with the statement that they are satisfied with the amount it takes for them to complete visits to the clinic.
  •  Over 95% of the clients agreed with the statement that treatment is helping them, and over 95% of the clients agreed with the treatment that they would recommend these services to others who were in need of similar help.

Targeted Capacity Expansion and HIV Services Grants

(El Rincon and Family Guidance Centers)

Evaluation Design - Richard E. Sherman, Ph.D. , Evaluator

Indicators/Measures Tools/Instruments/Data Sources Other Deliverables/Comments

Process Evaluation Questions

  • Was the project implemented as intended?
  • What types of project plan deviations occurred, what led to the deviations, and what effects did the deviations have on the project?
  • What are the staffing, logistical and cost characteristics of the delivered services?

Outcome Evaluation Questions

  • What are the characteristics of the clients who participate in the implementation of the evidence based substance abuse treatment services?
  • What outcomes are evident among the clients who participate in the expanded and enhanced services?
  • What is the durability of observed effects and what individual factors are related to observed effects?
  • To what extent did the project achieve its quantitative objectives for client admission and service delivery? What are the stakeholder group perceptions of project services?

GPRA

Client Satisfaction Survey

Focus Groups

Not Applicable

Offender Re-Entry Program (ORP)

Pathways to Re-Entry and Recovery

DHS Project Director Theodora Binion-Taylor
DHS Contract Mgr Lisa Cohen
Annual Grant $ $400,000

Evaluator

Name & Contact Info

Richard Sherman, Ph.D.

312-814-2290

Annaul Eval $ $14,000

Contact for a copy of evaluation

reports

Richard Sherman, Ph. D.

312-814-2290

Funding Source SAMHSA/CSAT
Eval Period 9/30/2009 - 9/29/2012

Summary of Project Performance Assessment Activities. The Illinois Department of Human Services, Division of Alcoholism and Substance Abuse (IDHS/DASA) was awarded an Offender Reentry Program (ORP) Grant from the Center for Substance Abuse Treatment (CSAT) (# TI021592) on 9/29/2009. The purpose of this Pathways to Re-entry and Recovery is to expand and enhance the substance abuse treatment and recovery support services that are available to adult female offenders who are returning to City of Chicago west and south side community areas following release from incarceration in the Illinois Department of Corrections (IDOC) Dwight and Lincoln Correctional Facilities. It is projected that 150 unduplicated returning female offenders will be served through these expanded and enhanced services during the three years of CSAT funding. The prior drug use patterns of returning female offenders, combined with their relatively poor educational and vocational backgrounds and the adverse environmental conditions common to the communities to which they are returning, serve to exacerbate the likelihood that they will commit future offenses and be re-incarcerated.

This performance assessment is being conducted under the direction of Richard E. Sherman, Ph.D., in his capacity as an IDHS/DASA contractual employee. Included among performance assessment data collection tools are the Government Performance and Results Act (GPRA) items mandated by CSAT, at admission, discharge from project services, and six-month post-admission follow-up. The process component of the performance assessment is guided by questions concerning the extent to which the project is implemented as intended, and will examine any deviations from the planned implementation. The outcome questions addressed by the performance assessment will determine achievement of the project's quantitative objectives and include focus on a range of outcomes that reflect treatment effectiveness among the returning female offenders who participate in the expansion of treatment and recovery support services. A key activity in conducting this ORP project's performance assessment is administration of a data collection tool to participating women at admission, discharge from services, and six-month post-admission follow-up. This tool consists of the GPRA items required by CSAT combined with selected items from the "Lite" version of the Addiction Severity Index (ASI).

Process Component. The process stage of the project's performance assessment is designed to address the following questions related to treatment and recovery support service implementation, delivery, and efficiency. The process component will also monitor project performance, provide immediate and ongoing feedback to project staff, and assure adherence to the project implementation plan.

Process Question. Is the project implemented as intended? (Data Sources: CSAT RFA, Grant Application, Project Time Lines, Project Reports, Manuals and Training Materials, Client Service Reports.)

Process Question. What are the staffing, logistical and cost characteristics of delivered services? (Data sources: I-STARS Service/Cost Reports, Client Service Vouchers, Client Service Summaries, Case Management Documentation Forms.)

Process Question. What types of project plan deviations occurred, what led to the deviations, and what effects did the deviations have on the project? (Data Sources: I-STARS Service Reports, Project Meeting Minutes, Corrective Action Plans, Other Project-related Documentation.)

Outcome Component. The outcome component in part examines changes in the participants' life functioning in a number of domains that are consistent with the GPRA and the CSAT NOMS. Evidence of change will be examined through comparisons of client responses to items on the expanded GPRA tool at admission, discharge, and six-month follow-up. This portion of the outcome component of the performance assessment represents a quasi-experimental one-group, pre/post design. A Client Follow-up Locator Form is completed for consenting returning female offenders as part of the project enrollment process, and prior to completion of the admission version of the expanded GPRA data collection tool. Clients who complete their six-month follow-up interviews are given a stipend in the value of $20 as reimbursement for their time and any incurred expenses.

Outcome Question. What outcomes are evident among the returning female offenders who participate in the expanded and enhanced services? (Data Source: Expanded GPRA Tool.)

The outcome portion of the performance assessment focuses on changes in participating adult female offender functioning between admission and six-month post-admission follow-up. The outcome objectives being addressed regarding treatment effectiveness include:

reduce alcohol and drug use (Expanded GPRA Section E);

reduce HIV/AIDS risk behaviors (health status) (Expanded GPRA Sections E&G);

reduce substance abuse related problems (Expanded GPRA Section E);

reduce the number of arrests, the number of illegal acts that participants commit, the number committed under the influence of drugs and/or alcohol, and the number committed to secure drugs and/or alcohol (social support/functioning) (Expanded GPRA Section F);

improve self sufficiency including vocational involvement in school or work, legal income, and public assistance (Expanded GPRA Section D);

improve the mental and physical status of participants (Expanded GPRA Sections G&H); and, improve social support and functioning including family and social relationships, and living arrangements (Expanded GPRA Section C).

Outcome Question. To what extent did the project achieve its quantitative objectives for client admission and service delivery? (Data Sources: Grant Application, Monthly Service Reports, CSAT GPRA Web Site Reports)

Outcome Question. What are stakeholder group perceptions of project services? (Data Sources: Staff Interviews, Client Focus Group Responses, Client Satisfaction Survey)

Capacity. This project's application contained a statement of goals and objectives that included a proposed cumulative intake target of 150 unduplicated released female offenders over the three years of CSAT funding. As of August 23, 2010, 37 unduplicated client intakes had been entered on the CSAT SAIS web site. This represents 97.4% of the cumulative target of 38 intakes up to this point.

Client Intake Demographic Data. A summary is provided below of responses on the baseline version of the 38 GPRA data collection tools downloaded from the CSAT SAIS web site on August 23, 2010. One of these tools is a duplicated baseline for a client who was admitted to services supported by this grant following discharge from a previous grant-supported treatment episode. The non-GPRA items in the table below are represented by an *.

Baseline Client Demographics and Other Characteristics (N = 38)

Gender

Female 100.0% (38)

Client Screened Positive for Co-Occurring Mental Health Disorder 65.8% (25)

Previous Treatment Experience* 84.2% (32)

Previous Treatment Experience - Non-Detox* 34.2% (13)

Source of Referral*

Illinois Department of Corrections

Court Services

IDHS DASA Licensed Organization

TASC 78.9% (30)

6% (1)

2% (5)

3% (2)

Lived in a Controlled Environment/Past 6 Months* 97.4% (37)

Lived in a Controlled Environment/Past 30 Days* 84.2% (32)

Mandated to Seek Treatment* 84.2% (32)

Source of Treatment Mandate/Criminal Justice System* 81.6% (31)

Average Age 39.3 years

Range: 21 - 56

Race

African American

Native American

Caucasian 78.9% (30)

6% (1)

4% (7)

Hispanic

No

Yes 92.1% (35)

7.9% (3)

Hispanic Ethnicity

Mexican

Puerto Rican 2.6% (1)

5.3% (2)

Marital Status*

Married

Divorced

Separated

Never Married

Widowed 13.2% (5)

2% (5)

9% (3)

5% (23)

3% (2)

In a Committed Relationship with Someone/Not Married* 44.7% (17)

Highest Level of Education Completed

8th grade

9th grade

10th grade

11th grade

12th grade/H.S. graduate

College/University - 1st Year

College/University - 2nd Year 7.9% (3)

7% (9)

5% (4)

4% (7)

9% (11)

3% (2)

3% (2)

Current Employment Status

Full time (35+ hours/week)

Part-time (< 35 hours/week)

Unemployed - looking for work

Unemployed - not looking for work

Unemployed - disabled 2.6% (1)

0% (0)

4% (18)

1% (16)

9% (3)

Average Income from Wages $30.11

Average Income from Public Assistance $55.61

Average Income from Family/Friends $11.71

Living Arrangements, Past 30 days

Shelter

Institution

Housed, Someone Else's House

Housed, Halfway House

Housed, Residential Treatment 10.5% (4)

2% (24)

4% (7)

3% (2)

6% (1)

Stress due to Substance Use/Past 30 Days

Not At All/Not Applicable

Somewhat

Considerably

Extremely 76.3% (29)

2% (5)

0% (0)

5% (4)

Reduction in Important Activities due to Substance Use/Past 30 Days

Not At All/Not Applicable

Somewhat

Considerably

Extremely 94.7% (36)

3% (2)

0% (0)

0% (0)

Emotional Problems due to Substance Use/Past 30 Days

Not At All/Not Applicable

Somewhat

Considerably

Extremely 89.5% (34)

3% (2)

6% (1)

6% (1)

Problems with Mother/Past 30 Days* 13.2% (5)

Problems with Father/Past 30 Days* 2.6% (1)

Problems with Siblings/Past 30 Days* 15.8% (6)

Problems with Spouse or Partner/Past 30 Days* 15.8% (6)

Problems with Children/Past 30 Days* 2.6% (1)

Problems with Other Family Members/Past 30 Days* 5.3% (2)

Abused Physically/Lifetime* 28.9% (11)

Abused Physically/Past Six Months* 7.9% (3)

Abused Physically/Past 30 Days* 5.3% (2)

Abused Sexually/Lifetime* 13.2% (5)

Abused Sexually/Past Six Months* 0.0% (0)

Abused Sexually/Past 30 Days* 0.0% (0)

Current Partner/Spouse/Source of Abuse* 13.2% (5)

Currently Living with Substance Abuser* 23.7% (9)

Rating of Overall Health

Excellent

Very Good

Good

Fair

Poor 15.8% (6)

2% (5)

3% (10)

6% (12)

2% (5)

Awaiting Trial or Sentencing - Yes 5.3% (2)

On Probation or Parole - Yes 100.0% (38)

Has a 12-Step Group Sponsor* 21.1% (8)

Attended Non-Faith-Based 12-Step Groups/Past 30 Days 63.2% (24)

Attended Faith-based Groups/Past 30 Days 44.7% (17)

Attended Other Recovery Support Groups/Past 30 Days 23.7% (9)

Interacted with Family/Friends who Support Recovery/Past 30 Days 76.3% (29)

Average Times Had Unprotected Sex/Past 30 Days

# of Non-Zero Respondents - 7 5.4

Range: 0-30

Tested for HIV Infection/Yes 100.0% (38)

Aware of HIV Status/Yes 97.4% (37)

Who Sought out When in Trouble

No One

Clergy

Family Member

Friends

Other 15.8% (6)

6% (1)

1% (16)

5% (4)

9% (11)

Inpatient Treatment/Past 30 Days

Physical Problems

Mental Health Problems

Substance Abuse Problems 2.6% (1)

6% (1)

8% (6)

Outpatient Treatment/Past 30 Days

Physical Problems

Mental Health Problems

Substance Abuse Problems 7.9% (3)

5% (4)

3% (2)

Emergency Department Visits/Past 30 Days

Physical Problems

Mental Health Problems

Substance Abuse Problems 18.4% (7)

0% (0)

0% (0)

Days of Serious Depression/Past 30 Days

Yes

Average

Range 50.0% (19)

7.3 Days

0 - 30 Days

Days of Serious Anxiety or Tension/Past 30 Days

Yes

Average

Range 52.6% (20)

7.6 Days

0 - 30 Days

Days of Trouble Concentrating or Remembering/Past 30 Days

Yes

Average

Range 44.7% (17)

6.7 Days

0 - 30 Days

Days of Trouble Controlling Violent Behavior/Past 30 Days

Yes

Average

Range 5.3% (2)

0.4 Days

0 - 14 Days

Days of Prescribed Psychiatric Medications/Past 30 Days

Yes

Average

Range 31.6% (12)

5.5 Days

0 - 30 Days

Bothered by Psychological/Emotional Problems/Past 30 Days

Not At All

Slightly

Moderately

Considerably

Extremely

Missing/Not Applicable 23.7% (9)

1% (8)

5% (4)

9% (3)

1% (8)

8% (6)

Pregnant - Yes 7.9% (3)

Has Children 76.3% (29)

Average Number of Children 4.0

Range: 0 - 10

Has Children Living Elsewhere/Court Order 21.1% (8)

N=20

Has Children/Lost Custody 31.6% (12)

N=42

Reported Substance Use. The following table presents information about the admitted women's use of substances during the 90 days prior to incarceration. Only the primary types of substances reported by released women are represented in the below table.

Days of Alcohol Use/90 Prior to Incarceration

Average Days of Use

Abstinent Rate - 52.6% (20) 24.7 Days

Range: 0 - 90 Days

Days of Illegal Drug Use/90 Prior to Incarceration

Average Days of Use

Abstinent Rate - 21.1% (8) 57.7 Days

Range: 0 - 90 Days

Days of Cocaine/Crack Use/90 Prior to Incarceration

Average Days of Use

Abstinent Rate - 42.1% (16) 36.3 Days

Range: 0 - 90 Days

Days of Heroin Use/90 Prior to Incarceration

Average Days of Use

Abstinent Rate - 63.2% (24) 29.6 Days

Range: 0 - 90 Days

Days of Marijuana Use/90 Prior to Incarceration

Average Days of Use

Abstinent Rate - 73.7% (28) 11.7 Days

Range: 0 - 90 Days

Injected Drugs/90 Prior to Incarceration 21.1% (8)

Injected Drugs/90 Prior to Incarceration/Used Clean Works

Less than Half the Time

Never

Not Applicable/Missing 5.3% (2)

5% (4)

2% (32)

Summary of Client Characteristics at Intake. The following is a summary of baseline characteristics of the released female offenders who have been admitted to treatment through this ORP project.

  • The women thus far admitted to this ORP project are on average 39.3 years of age at time of admission and 78.9% of the women selected African American as their race. A little less than 20% of the women reported themselves to be Caucasian.
  • Three (7.9%) of the women indicated they were of Hispanic ethnicity.
  • Nearly 80% of the women reported being referred to treatment through the Illinois Department of Corrections (IDOC) system.
  • Nearly 85% of the women reported previous substance abuse service episodes, but only about one-third of the women reported previous service episodes that were non-detox in nature.
  • Nearly 85% of the women indicated that they were being mandated to treatment, and the majority of these women indicated that some level of the criminal justice system was the source of this mandate.
  • Over 60% of the women have not completed high school.
  • Only one of the admitted women reported being employed full-time at time of admission and none reported part-time employment.
  • All but one of the women reported living in a controlled environment during some portion of the six months prior to admission, and nearly 85% of the women reported living in a controlled environment during some portion of the 30 days prior to admission.
  • Over 60% of the women reported "institution" as their primary living arrangement for the 30 days prior to admission.
  • Nearly 25% of the admitted women reported some level of feelings of stress during the 30 days prior to admission that they attributed to their substance use. About 5% reported reduction in important activities, and a little more than 10% reported emotional problems that they attributed to their substance use.
  • About 16% of the women reported experiencing problems with their spouse/partner or siblings during the 30 days prior to treatment. Somewhat lesser rates of experienced problems were reported with other family members during this time period.
  • Nearly 30% of the women reported being abused physically during their lifetime, and nearly 15% reported being victims of sexual abuse during their lifetime.
  • Nearly 25% of the women reported that they were currently living with someone who they believe to be a substance abuser.
  • About 45% of the women rated their current health as "fair" or "poor."
  • All of the admitted women reported being on probation or parole at time of admission and two reported that they were awaiting trial or sentencing.
  • About one-fifth of the women reported having a 12-step group sponsor.
  • Nearly two-thirds of the women reported attendance at non-faith-based support groups during the 30 days prior to admission, nearly 45% reported attendance at non-faith-based support groups during this time period, and nearly 45% reported participation in other types of group activities that support recovery during that time period.
  • Over 75% of the admitted women reported interactions with family and/or friends who are supportive of recovery during the 30 days prior to admission.
  • All of the women reported having been tested for HIV infection, and all but one of the women reported knowing her HIV status.
  • One-half of the admitted women reported serious depression, over 50% reported serious anxiety or tension, and nearly 45% reported trouble concentrating or remembering during the 30 days prior to admission that were not attributable to their substance use.
  • Over 75% of the women reported having children. There was an average of 4.0 children among the admitted women. Over 20% of the women reported having children living elsewhere due to a court order, and over 30% reported having lost custody of at least one of her children.
  • The average days of alcohol use during the 90 days prior to incarceration for the admitted women was 24.7 days. The abstinence from alcohol rate for the women during this time period was 52.6%.
  • The admitted women reported on average 23.8 days of illegal drug use during the 90 days prior to incarceration.
  • The average days of use during the 90 days prior to incarceration and the respective abstinence rates for the illegal drug types primarily reported by the admitted women were as follows: cocaine - 36.3 days and 42.1% abstinence rate; heroin - 29.6 days and 63.2% abstinence rate; and, marijuana - 11.7 days and 73.7% abstinence rate.
  • A little more than 20% of the women reported injection drug use during the 90 days prior to incarceration.

Effectiveness. As part of this ORP project evaluation, attempts are being made to administer the expanded GPRA data collection tool to each admitted woman at six months post-admission. This is the follow-up interval required by CSAT for ORP projects. The first women admitted to this project came due for six-month follow-up during August 2010. As of the preparation of this interim evaluation report, each of the four women who had come due for six-month follow-up was successfully interviewed during the eligibility window. A comparative analysis of participating client baseline and six-month follow-up responses will be conducted after a sufficient number of women have been contacted for follow-up.

Illinois RX: Resources for Ex-Offenders

Prisoner Re-Entry Initiative (PRI)

Final Evaluation Report

DHS Contract Mgr

Gloster Mahon (08/2008 - 11/2009)

Patricia Kates-Collins (11/2009 - 06/2010)

Annual Grant $ $539,732 (entire grant)

Evaluator

Name & Contact Info

Richard Sherman, Ph.D.

Sherman Consulting Group LLC

Res_scglic@att.net

Evaluation $ (See Narrative) $24,000 (Contract) $4,000 (Amount Invoiced)
Funding Source U.S. Department of Justice
Eval Period (See Narrative) 11/1/2009 - 6/30/2010

Background. The intended purpose of the evaluation of the Illinois Rx: Resources for Ex-Offenders Prisoner Re-Entry Initiative (PRI) grant awarded to the Illinois Department of Human Services (IDHS) in August 2008 was to assess the program=s ability to achieve its stated purpose to develop, coordinate and advance resources for ex-offenders returning to the Englewood and West Englewood communities within the City of Chicago. This grant initiative was awarded from the U.S. Department of Justice (DOJ), Office of Justice Programs, and was originally administered out of the IDHS Office of the Secretary. The primary goal of the national Prisoner Re-Entry Initiative (PRI) was to: Reduce recidivism rates for offenders in the program and increase public safety.

The Community Mental Health Council (CMHC), Inc. received a contract from IDHS to provide pre-release eligibility assessments, re-entry planning, and other pre-release services prior to participating ex-offender community re-entry. This CMHC contract included a subcontract with an individual who served as project manager. This individual functioned under the supervision of the IDHS contract manager. Liberated, Inc. received a contract to provide a range of post-release services to participating ex-offenders following community re-entry. These two service provider contracts were supported through the DOJ award to IDHS. Participating ex-offenders were to be men and women who were returning to the targeted Chicago communities following incarceration in the Illinois Department of Corrections (IDOC) system, and participation in pre-release services at one of two IDOC work release centers. Women who participated in Illinois RX were released to the community from the Fox Valley Work Release Center, located in Aurora, Illinois. The men who participated in Illinois RX were released from the Chicago Westside Work Release Center.

Administrative responsibility for the Illinois RX PRI was transferred to the IDHS Division of Alcoholism and Substance Abuse (DASA) in December 2009. This change occurred shortly after a change in the IDHS Secretary. The DOJ-funded services were discontinued on July 31, 2010. This was the end-date of the Federal budget period that was stated in the letter of award received by IDHS in August, 2008.

The PRI Initiative was structured such that a subsequent award would be made to successful DOJ grantees from the U.S Department of Labor (DOL). These DOL awards were intended to support a second community-based post-release services provider that would be under the administrative oversight of the DOJ-funded grantee. The DOL-funded community-based service provider was the result of a competitive grant application process that was limited to the geographic areas represented by those who received DOJ PRI grant awards. The successful DOL-funded grantee for the area represented by the IDHS Illinois RX PRI Initiative was the Chicago Christian Industrial League (CCIL), located in Chicago's near west side. Community re-entry services provided by CCIL were eligible to ex-offenders returning to areas of Cook County apart from the south side of Chicago following incarceration in the IDOC system.

The national DOJ PRI Initiative required that grantees comply with a performance assessment data collection and reporting plan that was designed to be responsive to mandates of the Government and Performance and Results Act (GPRA) of 1993. This plan required the reporting of participant data of relevance to the following project performance measures: 1.) Reduction in recidivism rate; 2.) Percent increase in number of ex-offenders in the target population (TP) who are assessed for risk and need while incarcerated; 3.) Percent increase in the number and type of pre- and post-release services being provided to ex-offenders in the TP; 4.) Percent increase in number of ex-offenders in TP for whom a transition plan is developed; 5.) Percent increase in number of ex-offenders in the TP who successfully complete the pre-release program; and, 6.) Percent of ex-offenders referred from TP to the DOL-funded community-based organization for post-release services. The DOJ award letter to IDHS included the expectation that at least 200 returning ex-offenders would complete post-release services. One hundred of these participants were to complete post-release services provided by the DOJ-funded community-based organization, and the remaining 100 returning ex-offenders would complete post-release services provided by the DOL-funded community-based organization. These total participant expectations were consistent with those stated in the DOJ announcement to which IDHS responded.

The funding application developed by IDHS and submitted to DOJ in January, 2008 stated that Sherman Consulting Group, LLC (SCG, LLC) would be contracted to conduct a local evaluation of the project, under the direction of Richard E. Sherman, Ph.D. An abbreviated local evaluation plan was provided in this application. After IDHS received the notice of DOJ award in August 2008, Dr. Sherman participated in a project start-up meeting and developed a more detailed local project evaluation plan, along with data collection tools that were designed to be responsive to the DOJ GPRA measures. The proposed evaluation plan included data collection tools and methods for assessment of the post-release service provider funded through the award from DOL. These materials were forwarded to the IDHS contract manager at that time.

However, there was a substantial delay in execution of a contract between IDHS and SCG, LLC for implementation of the local evaluation plan. During this delay, Dr. Sherman participated in a site visit to Fox Valley in February 2009, a meeting at Liberated, Inc. in late March 2009 regarding the proposed data collection tools, and a meeting at CCIL in June 2009 that focused on issues in providing this DOL-funded community-based organization with sufficient number of referrals to meet their service target. No other local evaluation services were provided by SCG, LLC during this period due to lack of a contract. A local evaluation contract was eventually provided to SCG, LLC in the late summer of 2009. This contract specified an effective time period of November 1, 2009 through June 30, 2010. The effective date of this contract was shortly before administrative responsibility for Illinois RX was transferred to DASA the following month. Dr. Sherman provided consultation to DASA and participated in meetings during the course of this transition. He also participated in communications with Illinois RX service provider staff and data collection activities during the course of project close-out. Dr. Sherman was asked to prepare a final evaluation report for the project based on the information and data that was available. This final evaluation report summarizes the data and information that was obtained during this close-out process, evaluation findings based on this data and information, and resulting conclusions and recommendations.

Format of Final Evaluation Report. The format of this Illinois RX final evaluation report summarizes information and data that was available specific to the process and outcome questions in the project evaluation plan that was originally submitted to IDHS. As discussed above, this proposed evaluation plan was not implemented due to the lack of execution of a services contract in a timely manner. While the proposed data collection tools that were related to these questions were largely not implemented, the content and focus of these questions are generally-accepted perspectives from which to assess a project's performance. This report also considers the project's performance relative to the PRI GPRA measures specified in the original DOJ request for proposals.

Process Evaluation. The proposed process stage of the evaluation was designed to address the following questions related to returning ex-offender service implementation, delivery, and efficiency.

Process Evaluation Question. Is the program implemented as intended? (Data Sources: Implementation Plan, Program Time Lines, Program Reports.)

Findings. The original DOJ PRI grant announcement included an expectation that a project's implementation phase be limited to one to three months following notice of an award. In the case of Illinois RX, the notice of award was dated August 8, 2008. Shortly after receipt of this letter, Dr. Sherman was invited to a meeting with the IDHS contract manager and representatives of CMHC and Liberated, Inc. that involved a preliminary discussion of project start-up. Dr. Sherman asked to be notified of future planning meetings and other activities. However, such communication generally did not occur, as evidenced by the three exceptions noted earlier in this report.

The site visit to Fox Valley on February 9, 2009 was primarily intended to describe the Illinois RX project to the work release center's staff, and to provide IDHS, CMHC, and Liberated, Inc. staff with a tour of the facility and a summary of the services provided. The date of this site visit was almost exactly five months after the date of the grant award letter. During the meeting Dr. Sherman inquired about any knowledge Fox Valley staff might have regarding the nature and extent of the post-release services received by women following their release from Fox Valley. Staff commented that they do not routinely follow-up on women after their release.

After the site visit to Fox Valley, Illinois RX staff learned that certain staff clearance procedures would need to be followed before CMHC would be permitted to implement on-site pre-release services. It was apparently the belief of Illinois RX staff that because CMHC was already engaged in the provision of mental health services within IDOC facilities, such clearance would not be required. However, this was not the case. Completion of this staff clearance process took several months. Once completed, problems were also experienced in regards to CMHC staff being on-site to provide the pre-release services. It was anecdotally reported that CMHC staff were not reporting due to salary compensation issues. As a result of these delays, returning ex-offenders did not begin to be enrolled in pre-release services until approximately one year after the grant award, and project participants were not referred to the community-based post-release providers until September 2009. As noted elsewhere in this report, during this delay in participant service implementation, CMHC and Liberated, Inc. were being reimbursed on a monthly basis one-twelfth of their annual contract award from IDHS.

There was also no evidence that an advisory group of the type described in IDHS's grant application and as required in the DOJ grant announcement, was ever implemented. An advisory group with a membership of the type described in these two documents might well have been useful in reducing the delay in project implementation and preventing the less than efficient use of grant funding.

Process Evaluation Question. What is the baseline number of ex-offenders who could be potentially referred to the program=s services, the prior availability of pre-release and post-release services for these ex-offenders, and what are their demographic characteristics? (Data Source: Grant Application, Other Project Documentation)

Findings. The GPRA performance measures stated in the FY 2008 DOJ PRI grant announcement to which IDHS applied, included quantitative determination of a Target Population (TP) of ex-offenders. The data provided in Illinois RX semi-annual reports to DOJ appear to reflect a belief that the TP was the number of released ex-offenders within a reporting period that were selected for participation. However, the GPRA grid in the FY 2008 DOJ PRI grant announcement would seem to indicate that a project's TP was a combination of those selected for participation combined with those released ex-offenders who were not selected. The narrative guidance provided in the FY 2008 announcement did not require applicants to define the target population that would be the focus of pre-release and post-release services. The FY 2009 DOJ PRI contained a more specific definition of the TP. The TP was to be a defined subset of the adults confined in a state or local prison or jail. While not referring to this group as the target population, the IDHS application to DOJ focused on adults released from the IDOC system who were returning to the City of Chicago Englewood and West Englewood community areas. This area of focus for the DOJ-funded community-based post-release services provider was later expanded to the entire south side of Chicago. Illinois RX seemed to focus solely with adults released from the Fox Valley and Chicago Westside work release centers. Fox Valley was a source of adult female released ex-offenders, while the Chicago Westside facility was a source of released adult males. Adults released to Chicago south side community areas from these two IDOC facilities seemed to define the project's TP. However, it is unclear how the total number of ex-offenders from which the TP was selected was determined. In any event, no matter what the project's TP, there was a responsibility to monitor various characteristics of all of the released ex-offenders in the TP whether or not they participated in Illinois RX. This would especially apply to numbers of persons who are convicted of a new crime, re-incarcerated, and/or violated terms of release, given the PRI national goal to reduce recidivism. There was no evidence that a system was put in place to collect this data for the entire TP. There was also no evidence of attempts to document the pre-release and post-release services that were available to members of the TP prior to Illinois RX.

Outcome Evaluation. The program's outcome evaluation questions were to focus on a range of outcomes among the returning ex-offenders who participated in Illinois RX. The outcome evaluation was to focus on the data collection needed to effectively respond to the six GPRA performance measures listed in the DOJ PRI grant announcement.

Outcome Evaluation Question. To what extent did the program achieve its quantitative objectives for client admission and service delivery? (Data Sources: Implementation Plan, Service Provider Documentation Forms, Monthly Service Reports)

Findings. The DOJ PRI grant announcement to which IDHS responded stated an expectation that at least 200 released ex-offenders would complete both pre-release and post-release services. The total target for completion of post-release services was divided equally between the DOJ-funded and the DOL-funded community-based providers. The DOJ GPRA measure grid also indicated a need to report, both for pre-release and post-release services, the number of participating released ex-offenders who received services in each of the following areas: cognitive behavioral therapy or other counseling, life skills, employment, education, substance abuse, mental health, overall health, family, anger management, mentoring, and other. Monthly reports in a format recommended by the proposed evaluator were completed by the IDHS project manager for the pre-release services provided by CMHC. This monthly report format was designed to collect the data needed to respond to the DOJ GPRA measures. Liberated, Inc. submitted monthly reports in a format different from that recommended by the proposed evaluator. The format used by Liberated, Inc. did not report the types of quantitative data needed to respond to all of the DOJ GPRA measures that pertained to post-release services. For example, while the CMHC reports indicated the number of participating ex-offenders who received pre-release services by the types listed above, the Liberated, Inc. reports did not provide such a breakdown for the post-release services. No monthly reports were made available that might have been submitted to IDHS by CCIL for the DOL-funded post-release services.

In an attempt to obtain information regarding the extent to which Illinois RX achieved the designated targets for participating released ex-offenders who completed pre-release and post-release services, Dr. Sherman developed spreadsheets that were designed to collect this information. These spreadsheets were submitted to the IDHS project manager and Liberated, Inc. following the end of Federal funding on June 30, 2010. The spreadsheet completed by the IDHS project manager indicated that 60 released ex-offenders completed pre-release services through CMHC. This represents 30% of the target of 200. As noted earlier in this report, referrals of those who completed pre-release services to the community-based post-release service providers began in September, 2009. The CMHC spreadsheet completed by the IDHS project manager indicated that 34 of the 60 participants who completed pre-release services were referred to Liberated, Inc., and 25 were referred to CCIL. There was one released offender who was not indicated to have been referred to either of these two post-release service providers.

Ten (10) of the names on the CMHC spreadsheet who were among the 34 released ex-offenders referred to Liberated, Inc., did not appear on the participant list returned by this DOJ-funded post-release services provider. This would indicate a 29.4% no-show rate for the released ex-offenders referred to Liberated, Inc. This would seem to infer that there would be 24 released ex-offenders on the Liberated, Inc. Illinois RX participant list. However, the participant list returned by Liberated, Inc. contained 36 names. There were a number of things that accounted for there being 12 more names on the Liberated, Inc. participant list than would be indicated by the information on the list submitted by the IDHS project manager for the pre-release services provided by CMHC.

The Liberated, Inc. participant list shows a person enrolled in January of 2009, but CMHC did not start referring released ex-offenders who had completed pre-release services until September of 2009. Additionally, this person's name was not on the CMHC participant list and was also not found in the IDOC on-line system. There were two more persons on the Liberated, Inc. participant list who were reportedly enrolled in post-release services in October of 2009, but were not on the CMHC list and could also not be found in the IDOC on-line system. There was a fourth person on the Liberated, Inc. participant list who was reportedly enrolled in post-release services in May of 2010 whose name was found in the on-line IDOC system, however this individual's name did not appear on the CMHC list of released ex-offenders who completed pre-release services. For the reasons summarized above, it was concluded that there was insufficient evidence to include these four individuals among those eligible to participate in Illinois RX post-release services. This left eight additional names of the 32 remaining on the Liberated, Inc. participant list that needed to be explained.

Upon comparing the names on the CMHC and Liberated, Inc. participant lists it was found that seven of the remaining 32 names on the Liberated, Inc. participant list are indicated on the CMHC list to have been referred to CCIL. The last CMHC monthly report completed by the IDHS project manager for June 2010 states that there were 13 of released ex-offenders who completed pre-release services who were referred to post-release services. This list includes the seven released ex-offenders mentioned above who appear on the Liberated, Inc. participant list. The last CMHC monthly report also states that all 13 of these referrals were made to CCIL. Dr. Sherman sent an e-mail to the IDHS project manager asking for clarification of this discrepancy. The IDHS project manager responded that these seven released ex-offenders were initially referred to Liberated, Inc. However, upon contacting Liberated, Inc. shortly thereafter, the IDHS project manager stated that he was informed by the Liberated Inc. chief executive officer that he had dismissed their Illinois RX case manager upon learning that the period of DOJ funding was coming to an end. The Liberated Inc. chief executive officer also reportedly informed the IDHS project manager that he would personally be handling the case management services for these seven released ex-offenders. The IDHS project manager stated that he felt uncomfortable with this arrangement and subsequently went to Liberated, Inc. to pick up the files for these individuals, and processed their referrals to CCIL. There was one additional name among the most recent post-release enrollments reported by Liberated, Inc. that was not on the CMHC participant list, but was found to be in the IDOC on-line system. Since this individual was not on the CMHC list of persons who completed pre-release services, it can not be verified that this individual was eligible for participation in Illinois RX.

While questions seem to remain about the project eligibility and level of participation in post-release services for the eight individuals discussed above, the most relevant issue is that all eight of these persons are reported to have not completed post-release services on the participant list submitted by Liberated, Inc. All of the remaining named participants on the Liberated, Inc. participant list were reported to have completed post-release services. With the exclusion of the four named individuals who did not seem to be eligible for the project's services, this left 24 released ex-offenders who reportedly completed post-release services provided by Liberated, Inc. This represents achievement of 24% of the service completion target for the DOJ-funded post-release service provider.

Attempts to obtain a participant summary list from the DOL-funded provider (CCIL) were unsuccessful. While the CMHC participant list indicates that 25 released ex-offenders who completed pre-release services were referred to the DOL-funded post-release services provider, it could not be determined how many of these referred released ex-offenders connected with the DOL-funded provider and how many completed post-release services. Therefore, it could not be determined to what extent Illinois RX achieved the expectation that at least 100 released ex-offenders complete post-release services provided by the DOL-funded community-based organization. It should be noted however, that the IDHS project manager communicated to DASA staff that CCIL did report participant service data to DOL. Nevertheless, the expectation that at least 100 released ex-offenders complete post-release services provided by the DOL-funded organization was among the conditions included in the DOJ grant award letter received by IDHS. The conditions in this award letter also included the expectation that the grantee put in place a "shared" data system that would support evaluation of both the DOJ-funded and DOL-funded post-release community-based service organizations.

Outcome Evaluation Question. What outcomes are evident among the returning ex-offenders who participate in PRI services? (Data Sources: Participant Data Collection Tools.)

Findings. As noted earlier in this report, there was no evidence of a process for routinely monitoring instances of re-arrest and/or re-incarceration among those returning ex-offenders who completed pre-release services and were referred to the community-based pre-release service providers. The monthly reports completed by CMHC beginning in January 2010 included mention of nine persons involved in pre-release services who were returned to the IDOC general population due to rule violations.

Outcome Evaluation Question. What are participating ex-offender perceptions of program services? (Data Sources: Staff Interviews, Client Focus Group Responses, Client Satisfaction Survey)

Findings. Consumer satisfaction surveys specific to CMHC, Liberated, Inc., and CCIL were among the data collection tools developed by SCG, LLC and submitted to the original IDHS contract manager. There is no evidence that they were ever implemented. Due to the delay in execution of services contract, SCG, LLC was unable to develop and provide the planned focus groups with participating ex-offenders. There was no indication of a concerted effort to obtain input from participating returning ex-offenders in regards to the pre-release and post-release services that they received.

Conclusions. This final evaluation of the Illinois RX project was severely limited due to an over one-year delay in execution of a contract. Nevertheless, a substantial number of issues could be readily observed that negatively impacted the project's implementation, performance, and outcomes. Consideration of the findings of this report should also take into account the change in project administration that occurred. As noted in this report, the DOJ-funded PRI grant awarded to IDHS was originally administered out of the Department's Office of the Secretary. At approximately the mid-point of the project's 24-month federal funding period there was a change in the IDHS Secretary position. Approximately five months after this change administrative responsibility for Illinois RX was transferred to DASA in December 2009. At about this time the individual who had served as the contract manager left IDHS employment.

After DASA staff had an opportunity to adequately assess Illinois RX status and progress, the amounts of time and resources that remained available severely constrained the steps that could be taken to improve the likelihood that the project would achieve its participant service targets. In regards to time availability, there was approximately a one-year delay in recruitment of released ex-offenders into the project. This left about 12 months of the federal funding period for 200 released ex-offenders to be provided and complete both pre-release and post-release services. The nature of the process through which incarcerated offenders became eligible for participation in Illinois RX made it unlikely that this target could be achieved in such a condensed time period. For example, it was extremely difficult to predict both when incarcerated offenders within the target population would become available to be recruited into pre-release services, and when those who completed pre-release services would be released to the community and be eligible for referral to post-release services. As indicated earlier in this report, only 60 incarcerated offenders completed pre-release services once services were implemented. One-fifth (14) of these individuals were referred with about one-month left in the federal funding period. This left little time for these individuals to be enrolled in and complete post-release services. Any consideration that DASA might have given to requesting a no-cost extension of the project to continue services after the delayed start-up was adversely affected by the limited financial resources that remained. This situation of limited resources resulted from CMHC and Liberated, Inc. being able to invoice one-twelfth of their prorated annual contract allocation for each of at least eight months before pre-release and post-release services were implemented. As mentioned previously, it appears these billings were allowable within the conditions of their contract awards.

Finally, it is the admittedly subjective opinion of this report's author that the timely implementation of the originally proposed evaluation would have prevented or limited many if not most of the issues identified in this report that adversely impacted project performance.