2006 IDHS Summary of Program Evaluation Findings Updated

Helping Families. Supporting Communities. Empowering Individuals.

February, 2006

Overview

As of July, 2005, 11 programs funded by the Illinois Department of Human Services (IDHS) had independent evaluations that were ongoing or completed during fiscal year 2005. Through grant or GRF dollars, these evaluations were being funded annually at $2,657,622.

Findings from these evaluations are conveyed in several ways. In most cases, the evaluations are part of federally-funded grant programs. In these cases, evaluation findings are reported in annual continuation applications to the funding agency, and the evaluator may issue a final evaluation report that is delivered to the funding agency. In larger studies, such as a study on license-exempt child care, the evaluation or research team may disseminate reports at conferences or make them available on their web sites. This report represents the first effort to combine summaries of all of the IDHS evaluations in a single report.

The report presents, alphabetically by division, each evaluation in place or completed by the end of state fiscal year 2005, with basic information about the grant program and the most recent evaluation findings at that point. Therefore, some information is preliminary, and researchers/evaluators have cautioned against widespread use of the information prior to final reporting. Sources for this information ranged from continuation applications to full evaluation reports. An appendix to this report is a table presenting basic information about the evaluations' administration and design.

The Office of Grants Administration envisions the following uses for the information in this report:

  • Assisting in Departmental integration by providing information to program staff across the divisions about what programs are being evaluated and what those evaluations are finding.
  • A demonstration of the level of effectiveness of IDHS programs across the divisions.
  • A set of examples for program staff writing new grants to see how other evaluations have been structured and to refer to in demonstrating to funding entities the extent of IDHS experience with program evaluations large and small.
  • A source of information about evaluations that may direct the interested reader to the full continuation application or evaluation report.

Eventually, all continuation applications and evaluation reports will be available in the IDHS Electronic Grants Library, housed on the IDHS OneNet. Access to the Electronic Grants Library can be obtained by contacting the Office of Grants Administration.

Division of Alcoholism and Substance Abuse

Access to Recovery (ATR)

Data Value
DHS Contract Mgr Rex Alexander
Annual Grant $ $7,575,685
EvaluatorName & Contact Info Richard E. Sherman, Ph.D.IOTA, Inc.
Annual Eval $ $240,000
Funding Source SAMHSA/CSAT
Eval Period 08/03/04 through 08/02/07

Evaluation activities were described in the year two continuation application submitted in March, 2005.

Project Summary

The Illinois Department of Human Services, Division of Alcoholism and Substance (IDHS/DASA) is administering Illinois ATR - Pathways to Reentry and Recovery. Key partners in this initiative are Treatment Alternatives for Safe Communities (TASC) and the Great Lakes Addiction Technology Transfer Center (GLATTC). It was originally proposed that 7,500 probationers would be served over three years in two targeted geographic areas, Cook County and Illinois' Fifth and Sixth Judicial Circuits and the 10 counties immediately adjacent to them. Probationers were targeted because as a group, they generate exceptionally high social and economic costs and have recovery and reentry needs that are particularly well-served by a network of recovery support services. The Illinois System for Tracking Access and Recovery Services (I-STARS) will support a real-time utilization management and service authorization system, which will move enrolled programs toward more cost-effective service provision and greater reliance on recovery support services. A group of treatment and recovery support providers, to include faith-based organizations, are being enrolled to provide vouchered services to Illinois ATR clients. Client choice among alternative service providers will be the primary determining factor in the referral of these clients. TASC has responsibility for screening and referral activities, as well as a major portion of client GPRA data collection. GLATTC will lead a multi-tiered training, technical assistance and technology transfer process targeting the courts and clinical treatment and recovery support programs. Data in the seven required domains will be collected at time of entry to ATR-support services, at specified points during service participation, and at time of discharge from ATR-supported services. I-STARS will monitor points at which clients come due for GPRA data collection and will permit IDHS/DASA to track reporting compliance and timeliness.

Description of Project Evaluation Activities

A formal evaluation design was not required to be included with the original ATR application to CSAT. However, a data plan was included as an appendix. It was necessary to revise this data plan due to the creation of the three client cohorts, and it was necessary to reconsider data collection responsibilities.

TASC will be responsible for GPRA data collection at time of entry into ATR-supported services for cohorts 1 and 2. Cohort 1 are those clients screened as non-dependent and will receive those GPRA items that are required by CSAT for "screening only" clients. The full ATR GPRA is embedded in the Illinois ATR Screen II tool that is administered to Cohort 2 clients who are screened as being likely dependent. The Screen II contains additional items that allow TASC staff to develop a diagnostic impression and level of care recommendation. A cross-walk was developed that will allow for generation of an initial version of the client's recovery support plan based on responses to items in the Screen II tool.

Cohort 3 are adult probationers who are in treatment at enrolled clinical treatment providers and for whom their treatment services are being supported by non-ATR funds. These clients will be eligible for referral to ATR-supported recovery supports services. Treatment provider staff will be responsible for completion of the baseline GPRA and all status GPRAs for these clients while they remain actively enrolled in treatment. TASC will have responsibility for completion of the status and discharge GPRAs for all Cohort 2 and 3 clients following their discharge from treatment and while they remain actively enrolled in ATR-supported recovery supported services.

The I-STARS system will have embedded within it tools that will be administered to clients to obtain their perceptions of the process through which they were able to choose among alternative clinical treatment and recovery support service providers, the treatment and recovery support services that they received, the extent to which these services met their needs, the staff who provided these services, and the facilities at which these services were received. The versions of this client satisfaction tool will vary according to point in time, and will be administered at time of initial assessment and referral and at the time of each status GPRA while the client remains active in ATR supported services.

No process or outcome evaluation information was reported at this time.

Chicago Collaborative Initiative to End Chronic Homelessness

Data Value
DHS Contract Mgr Kelli Gage
Annual Grant $ $628,312
EvaluatorName & Contact Info Elizabeth Durkin, Ph.D. and John Lyons, Ph.D.Northwestern University
Annual Eval $ $57,616
Funding Source SAMHSA/CSAT
Eval Period 9/30/03 through 9/29/06

Evaluation activities were described in the continuation application for year two, submitted in March, 2005.

Project Summary

The Chicago Collaborative Initiative to End Chronic Homelessness is proposing to establish a specialized Assertive Community Treatment (ACT) team linked with 59 tenant based Shelter Plus Care units to provide permanent housing and cost-effective and comprehensive services to single individuals who have a serious mental illness or substance use disorder or both, and who have been living on the streets or in shelters consistently for at least one year, or who have experienced four or more episodes of homelessness in the past three years. Approximately twenty-percent of the tenants (11 individuals) will be veterans. The project will target 22 of the most impoverished of Chicago's 77 communities on the South Side.

Evaluation Activities

The ARCH Continuous Quality Improvement (CQI) Plan was submitted to the Institutional Review Boards at both Northwestern University and IDHS/DASA. Preparation for this submission involved the design of several consent forms and a HIPAA release form, development of a semi-structured interview schedule for the qualitative interviews, and development of updated copies of ACT Fidelity, the MITI, and the Adult Needs and Strengths Assessment (ANSA). Both Institutional Review Boards have approved the plan.

The goals of the ARCH CQI plan are to train the ARCH team in the program model (ACT, motivational interviewing, and strengths-based assessment and treatment planning), to monitor the program's fidelity to those practices, to provide outcomes management, and to assess the level of systems integration resulting from the ARCH collaboration. The CQI team relies on several sources of data: client's periodic assessment data collected by the ARCH team workers, chart reviews, observation of the team itself, audio taping of client/worker interactions, and qualitative interviews with representatives of agencies that participate in the ARCH collaboration.

The evaluator, Dr. John Lyons, and staff of the Mental Health Services and Policy Program at Northwestern University trained ARCH workers on the use of the ANSA. To ensure reliability, the ARCH team workers were asked to assess a special vignette; the answers were then scored. All the ARCH workers obtained a reliability score of at least .70, the minimum necessary to be certified to use the ANSA. All members of the ARCH team were also required to complete a web-based training course on human subjects research offered through the University of Miami. Upon successful completion of the training modules, each team member received a certificate.

The CQI team trained ARCH workers on the procedures involved in recruiting clients and the specific use of the consent forms designed for the program.

The CQI team worked with the ARCH team leader to create a treatment plan based on the periodic ANSA assessments. The structure of this planning document allows for an easy correspondence between the needs and strengths identified at each assessment and the identification of services to address those needs or amplify those strengths. The treatment plan documents include a long version to be used initially and every 12 months thereafter, as well as a shorter version that is updated every 3 months. In conjunction with the ARCH team leader, a process for documenting progress notes was agreed upon. The format of the note is designed to allow for chart reviewers to quickly draw relationships between the services actually provided and the services proposed within the treatment plan.

Electronic versions of the ANSA and the treatment plan were created so that the workers can fill out the assessments on their computers and save electronic versions of their documentation. An SPSS database was created to store the ANSA data for subsequent outcomes management analysis. Procedures for the confidential and timely flow of data from ARCH to the CQI team were also developed, and training on these procedures was provided to the ARCH team members.

No outcome results were reported at this time.

Screening, Brief Intervention and Referral to Treatment (SBIRT)

Data Value
DHS Contract Mgr Deborah Levi
Annual Grant $ $3,500,000
EvaluatorName & Contact Info Richard E. Sherman, Ph.D.IOTA, Inc.&Christy Scott, Ph.D.Lighthouse Institute
Annual Eval $ $35,000 (IOTA)$315,000 (Lighthouse)
Funding Source SAMHSA/CSAT
Eval Period 9/30/03 through 9/29/08

Evaluation findings, as presented in the FY06 continuation application, pertain to the project's progress as of its second of five years.

Project Summary

The project provides substance use disorder screening, brief intervention, and referral to treatment (SBIRT) services within hospitals and community health clinics operated by the Cook County Bureau of Health Services (CCBHS). Three CCBHS general hospitals (Stroger, Provident and Oak Forest Hospitals) and two community health centers (Logan Square and Near South) are participating in the project. Screening and Brief Intervention services are provided within CCBHS settings by SBIRT health counselors. TASC community care coordinators assist in linking hospitalized substance dependent patients to substance abuse treatment at 11 IDHS/DASA-licensed treatment providers and to their first ambulatory health care appointment. Through this project, Brief Treatment is being implemented as a new treatment modality for patients determined to be appropriate for this level of care. Key project partners also include the Great Lakes Addiction Technology Transfer Center (ATTC) and Treatment Alternatives for Safe Communities (TASC). Lighthouse Institute will conduct six-month post-screening follow-up interviews are being conducted with at least 10% of the patients classified in the last three of these service categories.

Process Data

The evaluation report provides data for the project as of February 28, 2005 for screening only, brief intervention, brief treatment and referral to other treatment service. A total of 13,821/15,822 (actual/target; 87.4% of target) participants received screening and feedback, 1449/1806 (actual/target; 80.2%) received brief intervention services, 197/485 (actual/target; 40.6% of target) participants received a referral to brief treatment, and 886/498 (actual/target; 177.9%) received a referral to more traditional levels of substance abuse treatment. At this point in time, 16,353 patients had received SBIRT services, which was 87.9% of the cumulative target (18,611) to-date.

Outcome Data

Six-month follow-up interviews are required for 10% of patients receiving services. Of the required 10%, the outcomes evaluator had conducted six-month follow-up interviews with 105.1% of clients who had received referral and treatment; 108.8% of those receiving brief intervention; and 88.2% of those receiving brief treatment. In total, the outcomes evaluator followed up with 103.6% of clients due for six-month follow-up. The follow-up rates are greater than 100% because the outcomes evaluator begins attempts to conduct the interviews within the 30 days prior to the date patients come due for follow-up, which is permitted by CSAT. No follow-up interview results were reported at this time.

Targeted Capacity Expansion - HIV East St. Louis, IL

Data Value
DHS Contract Mgr Kelli Gage
Annual Grant $ $500,000
EvaluatorName & Contact Info Cheryl Mejta, Ph.D.Governors State University
Annual Eval $ $27,316
Funding Source SAMHSA/CSAT
Eval Period 10/01/01 through 9/30/06

Evaluation findings, as presented in the FY06 continuation application, pertain to project's progress in the fourth of five project years.

Project Summary

The project expands availability of Methadone Maintenance treatment to persons at high risk for HIV and/or hepatitis C (HCV) at the Cornell Interventions East St. Louis Clinic in East St. Louis, Illinois. The project also expands availability of treatment for mentally ill substance abusers (MISA) through increased psychiatric care.

Process Evaluation

The process evaluation showed that the project has kept its expanded Methadone "slots" full. The project is meeting its objectives to screen clients for HIV and HCV, conduct risk behavior reduction groups, conduct family risk behavior reduction education, conduct relapse prevention groups, to have a recovery coach support clients' recovery efforts, link clients to health care, conduct MISA counseling groups, screen for depression and anxiety, provide psychiatric monitoring, referral to mental health treatment as needed, and to provide treatment adherence groups to enhance patient adherence to psychiatric and medical treatment.

Outcome Evaluation

The evaluation reports outcomes from six and 12-month follow-up contacts in the areas of participation in productive activities, drug and alcohol use, HIV/HCV risk behaviors, physical health status, and psychological health status.

At six months, the evaluation found increases in productive activities (employment, training, or education) and housing; and decreases in illegal activities; use of drugs or alcohol; levels of stress and emotional problems; and experiences of health, behavioral, or social consequences related to their drug use. The evaluator found high percentages of clients still reported HIV/HCV risk behaviors, but did not report whether these percentages were higher or lower from intake to the point of six month follow-up.

At 12 months, the evaluation found a slight decrease in percentage of clients engaged in productive activities than at six months, but a still higher percentage than at intake. All clients were housed, and the decreases found at six months in illegal activities were maintained. An increase of 44.4% of clients had used no drugs or alcohol from time of intake. High percentages of clients were still reporting HIV/HCV risk behaviors. Clients continued to report lowered levels of stress and emotional problems related to drug or alcohol use, but reported high levels of overall anxiety and depression.

The evaluation report indicates that while the project has experienced difficulties in locating clients for 12-month follow-up interviews, the follow-up rate for 6 and 12-month client interviews showed a marked improvement from the period covered in the previous evaluation report. This general improvement in follow rates is due to clinic staff completion of additional CSAT sponsored GPRA training last year. From this training, they incorporated procedures to locate clients using a Locator Form that was enhanced to capture more finite client information for follow-up.

Division of Community Health and Prevention

Healthy Families Illinois

Data Value
DHS Contract Mgr Denise Simon
Annual Grant $ $8,450,000
EvaluatorName & Contact Info Julie CrouchOffice of Sponsored Projects Northern Illinois University
Annual Eval $ $333,000
Funding Source GRF
Eval Period 7/1/00 - 6/30/06

Evaluation information was available from J.S. Milner, T.R. McCanne, J.L. Crouch, and C. Thomsen, Healthy Families Illinois: Program Evaluation Interim Report, January, 2005, Center for the Study of Family Violence and Sexual Assault, Northern Illinois University. The researchers strongly cautioned that this is only an interim report and does not represent their final conclusions on the program.

Program Summary

Healthy Families Illinois (HFI) is a part of a national initiative, Healthy Families America, that seeks to improve outcomes for parents and their newborn children. The program is designed to strengthen family functioning and improve parent-child interaction through voluntary, intensive home visiting to families at risk for problems in parenting, including child abuse and neglect. All families eligible for HFI's intensive home visitation services possess risk factors for problems in parenting.

Preliminary Findings

The first series of preliminary analyses included 280 intervention families and 63 comparison families with complete data at birth, six months, and one year. This series of analyses examined changes across time in child physical abuse risk, parent-child interaction, and characteristics of the home environment. The second series of preliminary analyses involved examining rates of confirmed child maltreatment among 78 Intervention and 18 Comparison families who had been assessed at all five time points in the HFI evaluation project.

The Interim Report describes demographic information on project families in age of parents, number of years of education, racial/ethnic identity, number of children, and marital status.

Reductions in Child Physical Abuse Risk. Preliminary findings show "significant reductions in CAP [risk] scores across time for high- and medium-risk parents." Low-risk parents did not show significant changes in risk over time.

Improvements in Parent-Child Interaction. Researchers found that "high-risk Intervention parents appear to become more appropriately responsive to children's signals of distress," which they considered "particularly relevant to demonstrating HFI's effectiveness in reducing risk for child maltreatment." The researchers cautioned that there was not an adequate sample of comparison high-risk parents, which "limits interpretation of this finding."

Improvements in the Home Environment. Researchers noted that there was a general improvement in scores related to this indicator in both intervention and comparison groups, but that Intervention families "scored significantly higher than Comparison families at all three time periods." They further noted that high-risk families showed "significant improvement" and essentially "`caught up' with the low- and medium-risk families" at year one.

Child Maltreatment. Preliminary findings are that "there was substantially more risk for confirmed cases of child maltreatment among Comparison families compared to Intervention families."

The Interim Report concludes that "preliminary findings are congruent with observations that HFI services appear to improve parent-child interaction and reduce risk of maltreatment among families who are most at risk." But the report cautions that preliminary findings represent a "limited number of moderate- and high- risk families that have completed a minimum of one year of follow-up assessment. Additional data also are needed from a sufficient number of comparable (i.e., moderate- and high-risk) families who are not receiving HFI services in order to clearly demonstrate that changes observed across time are associated with receipt of HFI services."

Substance Abuse Prevention

Data Value
DHS Contract Mgr Kim Fornero
Annual Grant $ $20,277,712
EvaluatorName & Contact Info Peter Mulhall, Ph.D. Center for Prevention Research and Development (CPRD)University of Illinois at Urbana-Champaign
Annual Eval $ $1,062,888
Funding Source SAMHSA
Eval Period 7/1/90 - 6/30/06

Evaluation information was provided in a special report compiled by the evaluators, Illinois SAPT Block Grant Prevention Set Aside Evaluation Overview, February 14, 2006 (Evaluation Overview). There is no annual evaluation report.

Program Summary.

The Illinois Substance Abuse Prevention and Treatment Block Grant specifies that 20% of funds will be spent on prevention. The Bureau of Substance Abuse Prevention has a three tier system consisting of Comprehensive programs, the InTouch network and Statewide programs.

Comprehensive Programs: The Comprehensive program grants are awarded to not-for-profit organizations/governmental entities. The type of organizations receiving these grants are community mental health agencies, substance abuse treatment agencies, public health departments, regional offices of education, YMCAs and youth-serving organizations. These agencies deliver prevention services at the community level.

InTouch Network: The Illinois Network to Organize the Understanding of Community Health (InTouch) are also awarded to the types of organizations listed above. The InTouch network covers regions of the state called Prevention Service Areas (PSAs). The geographic boundaries of the PSAs were based upon the Illinois State Board of Education's educational regions. All 102 counties in the State are served by an InTouch office. The InTouch offices deliver technical assistance and consultative services that enhance a group, coalition, or community's capacity to deliver effective prevention services.

Statewide Programs: The Statewide programs receive funding to serve large geographic areas or defined target populations. Below is a list of the programs.

  1. Eastern Illinois University: provides technical assistance and training to public and private, two and four year colleges and universities
  2. Illinois State Chamber of Commerce and the Chicagoland Chamber of Commerce: provide drug free work place services, technical assistance and training, to small businesses (Chicagoland) and local chambers of commerce (Illinois)
  3. Operation Snowball (OS) and Illinois Teen Institute (ITI): the annual teen institute and Operation Snowball programs promote adult/youth partnership and leadership to address substance abuse prevention through a series of youth-led workshops and recreational activities
  4. Illinois Drug Education Alliance (IDEA): sponsor an annual youth conference and promote national campaigns, Red Ribbon Week and Alcohol Awareness Month
  5. Robert Crown Health Center: provides informational workshops for youth in grades K-12
  6. Northeastern Illinois University: trains education majors how to infuse substance abuse prevention into curricula, training is integrated into their educational course work
  7. Lighthouse Institute (LI): coordinates and administers the bi-annual youth survey
  8. Center for Prevention Research and Development (CPRD): manages the BSAP On Track (similar to MDS) web-based tracking system and provides evaluation support services (e.g. Process Evaluation Tool Kit)
  9. Prevention First, Inc (PFI).: provides training, information services (library, clearinghouse, web site) and conference planning
  10. University of Illinois: two programs are funded: 1) targets teen mothers and their mentors in one alternative high school and 2) targets the faith community in one Chicago community area.

Process Information

Evaluators from the Center for Prevention Research and Development (CPRD), University of Illinois at Urbana-Champaign worked with IDHS to develop OnTrack, a web-based management information system in July, 2000. Quarterly, CPRD generates system level reports that provide IDHS with both agency level and system-wide summaries of hours delivered and populations served. The example provided in the Evaluation Overview is that a contract requirement for community based contractors mandates that a minimum of 40% of contract hours are devoted to "evidence based" programs and strategies. In FY03, 44% of contract hours met this requirement; in FY04 again 44% met the requirement, and in FY05 46% met the requirement.

Community providers must submit to IDHS and CPRD a Semi Annual Evaluation Progress Report and an Annual Evaluation Progress Report which contain process evaluation information.

Outcome Evaluations

In FY03 the prevention system "moved beyond a system of evaluating all community-based ATOD prevention programs" to a system where outcome evaluations are conducted only for "well-implemented and documented prevention programs" because they would be "in a better position to demonstrate favorable outcomes because they would be less likely to face challenges with programmatic interruptions and implementation difficulties." To determine which programs would participate in outcome evaluations, CPRD implemented the Readiness for Outcome Evaluation Review Checklist, which measures programs in areas of sufficient outcome evaluation sample size, high levels of program completion, fidelity to model programs or DHS guidelines for evidence based practice, and staff experience and training.

Eighteen programs operated by 17 providers participated in outcome evaluations from FY03 through FY05. In an appendix, the Evaluation Overview provides brief details on the outcomes. Of the 18 program evaluations, 10 included a comparison group or some type of comparative data. "At least one positive knowledge-based, attitudinal or behavioral prevention outcome was demonstrated in all 10 of these evaluations." Programs evaluated over multiple years demonstrated the most positive outcomes in terms of behaviors and attitudes.

The remaining eight evaluations (without comparison groups or data) included four that served youth with specific risk conditions and four that served a general youth population. Five of these evaluations

demonstrated either one statistically significant change or retention of strong anti-ATOD perceptions and abstinence behaviors. Among the 3 that did not demonstrate any positive impact, the enhanced outcome evaluation revealed implementation practices that clearly violated evidence-based guidelines for two programs (e.g., youth participated less frequently in the program than expected) and the third was offered the opportunity to repeat outcome evaluation in the following fiscal year with a comparison group. The program demonstrated a positive outcome in the subsequent evaluation.

An additional 17 outcome evaluations were in progress as of September, 2005.

Targeted Intensive Prenatal Case Management

Data Value
DHS Contract Mgr Beverly English
Annual Grant $ $4,693,400
EvaluatorName & Contact Info James W. MastersonHamilton Bell Associates
Annual Eval $ $168,600
Funding Source GRF, MCH Block Grant & Healthy Start
Eval Period 7/1/01 - 6/30/06

Hamilton Bell Associates provided an evaluation report dated June, 2005.

Project Summary

Ten providers are contracted to provide Targeted Intensive Prenatal Case Management (TIPCM) to high risk pregnant women residing in geographic areas of Illinois identified by DHS as having higher costs of infant care. The program uses outreach and incentives to engage eligible clients and keep them engaged in case management services. Case managers have caseloads of no more than 40 and have at least two face-to-face contacts with each client each month. At least half of the contacts take place in the pregnant woman's home. Services include a risk assessment, care plan, referrals made and referrals kept, including for mental health, substance abuse treatment, smoking cessation services and domestic violence intervention services, prenatal visits. Some project funds may be used to purchase services if no other third party payment source can be identified. Data is collected in the Cornerstone management information system. Provider agencies compete for outreach performance bonuses based on growth in the number of high-risk women who enroll in program services and the number of live births that exceed five pounds, eight ounces. Four providers were in Cook County, and six were in downstate Illinois. The project anticipates adding as many as six providers in FY06.

Process Evaluation

The evaluator found that the project enrolled 97% of its target of 1,800 pregnant women. In other measures, the evaluator documents progress falling short of the objectives. Where agencies are identified by the continuous quality improvement (CQI) process as "poor performers" on a measure, they become subject to a Focused Review by the CQI committee, which has representation by all 10 provider agencies and DHS staff. The Focused Review looks closely at ways to improve performance.

Outcome Evaluation

To date, improvements were seen in the project area in reduced infant mortality rates, reduced very low birth weight rates, and increased prenatal care. For two other measures-reduce low birth weight rates and increase proportion of very low birth weight infants born in Level III or Level II+ facilities-there were insufficient data to make an analysis.

Preliminary Conclusions

The evaluator offered six preliminary conclusions about the targeted intensive case management project:

  1. The TIPCM Project is effective.
  2. The TIPCM Project is "efficient."
  3. The TIPCM Project finds and enrolls the "hard to reach, high risk" pregnant population in the Targeted Communities.
  4. The TIPCM Project has a CQI Program that has demonstrated effectiveness in identifying and resolving barriers to improving Agency performance in implementing the Core Services and meeting other DHS expectations (I.e., data entry into Cornerstone).
  5. The TIPCM Project is making steady progress toward achieving its stated Project Objectives.
  6. DHS and the participating TIPCM Agencies have successfully collaborated in the identification of areas needing improvement in 2005 and has successfully "institutionalized" a CQI Program that seeks to identify improvement wherever and whenever possible.

Division of Human Capital Development

Illinois Study of License-Exempt Child Care

Data Value
DHS Contract Mgr Linda Saterfield
Annual Grant $ $500,000 for total 3-yr period
EvaluatorName & Contact Info Steve AndersonSchool of Social WorkUniversity of Illinois at Urbana-Champaign
Annual Eval $ $143,600(in yr 3)
Funding Source DHHS, Admin for Children and Families,Child Care Bureau
Eval Period Fy03 through Fy05

Information on the evaluation study is available in the March 2005 report, Illinois Study of License-Exempt Child Care: Final Report. Prepared by: Steven G. Anderson, School of Social Work, University of Illinois at Urbana-Champaign; Dawn M. Ramsburg, Child Care Bureau, Administration on Children, Youth, and Families, U.S. Department of Health and Human Services; and Jeff Scott, School of Social Work, University of Illinois at Urbana-Champaign.

Project Summary

The Illinois Child Care Assistance Program (CCAP) is a large, statewide child care program that provides subsidies to families with incomes up to 50 percent of the state median income level. The program features parental choice of either licensed or license-exempt providers, and there are no waiting lists for services. The evaluation studied license-exempt care provided through CCAP. License-exempt care is child care provided in home settings that have been legally exempted from Illinois licensing requirements. License-exempt providers can care for no more than three children, including their own children, unless all children are from the same household. Four types of settings are included in this definition: non-relatives providing care in their own home; non-relatives providing care in the child's home; relatives providing care in the relative's home; and relatives providing care in the child's home.

Research Questions

Seven principal research questions guided project research activities:

  1. What are the patterns of care for families and children that utilize subsidized license-exempt child care, and how do these differ from families and children that rely on subsidized licensed child care?
  2. Do parents who use license-exempt child care differ in demographic characteristics and other important respects from parents who rely on licensed care?
  3. What factors influence families to choose license-exempt child care providers rather than licensed providers, or to choose a mix of these providers?
  4. What are the characteristics of license-exempt subsidized child care providers, and what levels of experience and training do they have in providing child care?
  5. How do parents and license-exempt child care providers describe the quality of license-exempt care, and what specific strengths and weaknesses do they identify with this type of care?
  6. What training and resources are needed to support the quality of care offered by license-exempt providers involved in the subsidy system, and how can these resources best be provided?
  7. Based on study findings and analysis of related research, what policy implications can be drawn for enhancing the quality of subsidized license-exempt child care?

Study Findings

The Final Report presented findings in response to each of the study questions identified above.

"[The] overarching study conclusion is that license-exempt caregiving in the CCAP generally represents a positive confluence of both parental choice factors and provider motivations. The broad inclusion of this form of care has facilitated parents' child care choices, and has allowed the program to serve large numbers of children whose parents work non-traditional schedules. [The] findings suggest that license-exempt care can play an important role in assuring that a continuum of child care options are offered in large subsidy programs."

The study presents two broad policy directions for consideration, as well as a number of more specific recommendations. The two broad areas are:

First, given that research increasingly has demonstrated the importance of early brain development and early childhood learning, it is important that parents facing child care decisions be informed about the benefits of early learning and related developmental activities. Continued experimentation on the most effective ways to educate parents about supportive caregiving practices, regardless of setting, therefore is needed. Second, making informed choices about caregivers in subsidized programs requires a clear understanding of program rules. In addition to providing written information, both program orientation sessions for parents and initial individual meetings with child care program staff are important opportunities to convey such information.

The full Final Report can be accessed at http://www.childcareresearch.org/discover/viewPdfFile.jsp?resourceId=7350&type=.pdf

Division of Mental Health

System of Care - Chicago

Data Value
DHS Contract Mgr Patricia Roy, MD
Annual Grant $  $2,000,000 in FY06(varies each grant year)
EvaluatorName & Contact Info John Lyons, Ph.D. Mental Health Services and Policy Program Northwestern University
Annual Eval $ $185,000 in FY06
Funding Source SAMHSA/CMHS
Eval Period 11/01/04 through 9/29/08

Evaluation findings, as presented in the FY06 continuation application (March, 2005), pertain to the project's progress as of its third of six years.

Project Summary

The goal of System of Care Chicago (SOCC) is to implement a system of care based on the primary partnership established between DHS/DMH Metropolitan Child and Adolescent Network and the Chicago Public Schools. This effort is aimed at bringing together parents, youth, community networks and resources, along with the responsible child-serving agencies, which include local mental health centers, schools, healthcare providers, child welfare, substance abuse and juvenile justice, in order to meet the needs of children with a serious emotional disorder and their families. Family involvement and cultural competence are key issues that are addressed throughout the initiative. Evaluation tools are being designed to include child and family outcome data as well as findings related to the processes of system change itself. Service redesign is intended to blend evidence-based practices and non-traditional approaches and resources to address the full array of children's needs while identifying and enhancing the strengths of those families. Social marketing plays a key role in the dissemination of information regarding system of care development through quarterly newsletters, staff training, special events, and the development of communication pathways that decrease confusion and enhance collaboration.

Evaluation Activities

The community evaluation team was established and meets regularly. IRB review and approval of the evaluation plan was concluded in October, 2004. The evaluator launched the SOCC participation in the national evaluation, and the evaluators have begun reporting data to ORC MACRO. The local evaluation plan was developed. As of March, 2005, approval of the local evaluation plan was pending. Also pending was determination of the best measure of fidelity to service provision. Evaluation staff worked with project partners to implement evaluation activities. Evaluators provided training on evaluation tools, obtained informed consent, and reviewed participant eligibility. Evaluators hosted evaluation update meetings at each of the schools sites. As of January 31, 2005, 141 families were enrolled in the program; 30 had consented to participate in the national evaluation. Evaluators provided training on the national evaluation.

No implementation or outcome evaluation information was reported at this time.

Office of Compliance Access and Workplace Safety

Real Choice Systems Change

Data Value
DHS Contract Mgr Martha Younger-White
Annual Grant $ $800,000
EvaluatorName & Contact Info Glenn Fujiara, Ph.D. University of Illinois at ChicagoDepartment of Disability and Human Development
Annual Eval $ $89,602
Funding Source Centers for Medicaid and Medicare Services
Eval Period 9/04 through 8/05

Evaluation findings were presented in Illinois Real Choice Systems Change: Project Review, September, 2005, Department of Disability and Human Development, University of Illinois at Chicago.

Project Summary

The intent of the program was to focus on partnering the disability and aging communities to design or implement lasting changes and to foster innovations in the support of individuals with a disability or long-term illness living in their communities. IDHS proposed to "enhance" the existing system of consumer-driven long-term supports by enhancing communication among organizations and agencies providing services and supports, developing strategies for affordable, accessible housing, developing tools for institutional transitions, and testing models of community supports. The general goals of the IDHS project were to foster communication; develop housing strategies; develop tools to assist transitions from institutions to communities; and establish a framework for successful long-term community supports via training resources and other tools.

IDHS developed contracts with collaborations in Chicago, Southern Illinois, Rockford, and Springfield. IDHS incorporated the work of the Threshold's Peer Success Center into the project in 2005 in order to expand system change efforts into the mental health community.

Implementation Results

The Project Review discusses the program in each of the four sites, including the participating non-profit and state agencies and program operations. The Rockford and Southern Illinois programs focused on transitioning consumers from institutional to community living settings, providing necessary supports to facilitate this goal. The Springfield and Chicago projects focused on facilitating home ownership for consumers, providing coordination and services to support this goal.

Outcomes

The review of projects involved a series of interviews, both on-site and telephone-based, and a review of relevant projects in other states. For each of the four projects, the Project Review provides outcomes in the areas of services provided, communication and policy change, tool development, and long-term supports framework or housing strategies.

Rockford Outcomes

Services Provided 1. Served 64 persons·
  • 25 moved from institution with CMS funds·
  • 25 deflected from institutional placement with CMS funds·
  • 14 other supports provided
Communication and Policy Change Established ongoing communication network among 7 area agencies and representatives of five state agencies..
Tool Development
  1. Developed manual for assessing assistive technology needs.
  2. Developed Team Consultation framework.
  3. Conducted 13 trainings for community service providers reaching over 400 persons (9 on AT, working with health care professionals, substance abuse awareness, wellness planning); 2 presentations at state conferences.
Long-Term Supports Framework
  1. Employed communication network to establish referral mechanism for at risk persons with disabilities under the age of 60.
  2. Developed an area Aging and Disability Resource Center based on project activities.

Southern Illinois Outcomes

Services Provided
  1. 29 persons moved from institutional care
  2. 87 persons provided community supports
  3. transition and independent living supports provided
Communication and Policy Change
  1. Developed System Change presentation
  2. Supported AT workshops and train-the-trainer series
  3. Established service delivery linkages between for profit sheltered care agency and the local MH Center and CIL; modified provision of day services.
Tool Development 1. Developed a needs assessment survey for the Southern IL area.
Long-Term Supports Framework
  1. Develop an area peer support group affiliated with the Depression and Bi-Polar Support Alliance
  2. Independent living orientation to all 116 consumers provided services.

Springfield Outcomes

Services Provided 1. 47 new homeowners
Communication and Policy Change

Established Emergency Fund for mortgage costs

2. Expanded Emergency Fund to include access modifications when other resources are not available

3. Hired full-time Fair Housing Specialist for education and outreach

4. Presentations on the Homeownership Coalition and Fannie Mae

5. Quarterly newsletters Homeownership Coalition

6. Home Modification Resource List

7. Brochures on the Homeownership Coalition, funding sources, HUD's Fair Housing laws, and SCIL's services.

Housing Strategies 1. Program replication in 6 communities
Tool Development
  1. Homebuyer Education manual in all alternative formats (audio, Braille).
  2. Developed a database to track demographics of homebuyers

Chicago Homeownership Outcomes

Services Provided 1. 17 persons with disabilities became homeowners
Communication and Policy Change
  1. 1,090 information and referral events
  2. 500 hits on the "Home Options" web site
  3. 18 educational seminars
  4. Development of the Chicago Housing Services and Resources Guide
Housing Strategies
  1. Modified (increased) maximum income range for the Chicago-Federal Home Loan Bank to increase access by persons with disabilities.
  2. Currently developing a consumer housing cooperative
Tool Development Development of the databases, "Resource Directory and Guide to Disability Services" and "Chicago Housing Services and Resources"

The Project Review further provides a section on lessons learned and key issues for each of the four projects and discussion on implications in the following areas:

  1. Why collaboration was so critical to project success: single point of entry.
  2. "Falling through the cracks" - un-served and under-served groups.
  3. Where are the alternatives? Increasing access to affordable and accessible housing.
  4. Little stuff counts: ongoing case management support.

IDHS's grant award was 76% less than originally requested. Activities were modified to take into account the reduced resources. Project innovations were largely designed to overcome fundamental barriers of the complex systems of care. Based on a review of other statewide systems transformations, Eiken (2004) identified common elements: 1) leadership, 2) consumer involvement, 3) shared vision, 4) precipitating events, 5) a political champion, and 6) plans for change. New models and demonstration projects are only the initial steps in a long process of transformation. Still, the evaluators conclude that the project obtained measurable outcomes in areas of persons served, inter-agency communication, staff trained, and collaborations established.

Addendum

Separately Funded National Evaluation That Includes an HCD Grant Program:

Employment Retention and Advancement

Data Value
DHS Contract Mgr Debbie Gibson / Nancy Crossman
Annual Grant $ $1,500,000(total funding Feb 2002-June 2004 was $3,589,232. Initial funding was secured from the Governor's Discretionary portion of the state's Welfare-to-Work block grant. The U.S. Department of Labor provided supplemental funding in 2004 after Congress rescinded unspent Welfare-to-Work funds.)
EvaluatorName & Contact Info Dan BloomMDRC
Annual Eval $ Unknown (not funded by DHS)
Funding Source DHHS, Admin for Children and Families, Office of Planning, Research and Evaluation(for evaluation funding, not program funding)
Eval Period Oct 2000 through Sept 2007

Results from: Dan Bloom, Richard Hendra, Karin Martinson, Susan Scrivener, The Employment Retention and Advancement Project: Early Results from Four Sites. June 2005. MDRC: New York. Accessed at http://www.mdrc.org.

Project Summary

Operating in Chicago and St. Clair County, the Illinois Employment Retention and Advancement (ERA) program targets a group that appeared to be "stuck" in low-wage jobs: TANF recipients who reported full-time employment to the welfare agency for at least six consecutive months and yet continued to qualify for cash assistance. Operated by contracted service providers, the program provides a range of services designed to help participants increase their earnings in their current job or, more typically, to find a higher-paying job. The control group is not referred to an ERA service provider but may receive services from welfare office staff.

Year One Impacts

The one year impacts measure outcomes during the first year after people entered the study, from February 2002 through September 2002. The Illinois ERA program generated statistically significant increases in both unemployment insurance (UI)-covered employment and earnings during the first year of follow-up. Just over 71% of the ERA group worked in a UI-covered job at some point during Year 1, compared with 65% of the control group. The ERA group earned, on average, $539 more than the control group during year 1, an increase of about 9%. The Illinois ERA program also produced a fairly large decrease in TANF cash assistance payments. On average, the ERA group received $193 less (about 12% less) in TANF benefits than the control group during year 1.

Impacts Over Time and for an Early Cohort

Data for one additional quarter of follow-up are available for the cohort of people who entered the study from February through June 2002. Results for this early cohort show that the Illinois ERA program's impacts on both employment and earnings generally grew larger over time. The impacts on employment and earnings appear to have increased dramatically just after the end of year 1. It is too early to say whether this pattern will persist when longer-term follow-up data are available for a larger sample of participants.

Since everyone in the sample had reported earnings to DHS for at least six consecutive months prior to their random assignment date, it is likely that the individuals with no UI-covered earnings in the quarter prior to random assignment were working in jobs not covered by the UI system. The results show that the Illinois ERA program generated very large increases in UI-covered employment and earnings for sample members who had no UI earnings in the quarter prior to random assignment. For example, average earnings for the ERA group are about twice as high as for the control group. This provides further evidence that ERA may have helped individuals move from non-UI-covered jobs to UI-covered jobs. The program also produced a large decrease in TANF payments for this group, but this loss in income was more than offset by the very large earnings gain. Thus, total measured income from UI earnings, TANF, and food stamps was substantially higher for the ERA group.

The early results for the Illinois ERA program are promising. The program has increased both UI-covered employment and earnings, possibly by helping some individuals move from non-UI-covered jobs to UI-covered jobs. It will be important to see whether these effects continue with longer follow-up and also whether survey data - which should include some information on non-UI-covered employment - bear out the results from UI records.

Appendix: Evaluations Chart