?DHS Contract Mgr Kim Fornero
Annual Grant $ $14,100,315
Evaluator Name & Contact Info

Peter Mulhall, Ph.D.

Beth Welbes, M.S.P.H.
Center for Prevention Research and Development (CPRD) University of Illinois at Urbana-Champaign

Annual Eval $ $821,343
Funding Source SAMHSA
Eval Period 7/1/08 - 6/30/09

Evaluation information was submitted via e-mail by Beth Welbes, MSPH, August 10, 2009.

Substance Abuse Prevention and Treatment Block Grant (SAPT) Brief Summary

The Substance Abuse and Mental Health Services Administration's (SAMHSA) Center for Substance Abuse (CSAT) and the Center for Substance Abuse Prevention (CSAP) make available allotments each year to States through the funding of the Substance Abuse Prevention and Treatment Block Grants for the purposes of planning, carrying out, and evaluating activities to prevent and treat substance abuse to include the abuse and/or illicit use of alcohol and other drugs. The Block Grant funds may be expended to provide for a wide range of activities to prevent and treat substance abuse and may be expended to deal with the abuse of alcohol, the use or abuse of illicit drugs, the abuse of licit drugs and the use or abuse of tobacco products.

The annual uniform application due October 1, 2008 serves as the application to SAMHSA for the SAPT Block Grant funds and includes: (1) funding agreements and certifications (2) an annual report (3) intended use plan (4) voluntary treatment and prevention performance measures.

The Substance Abuse Prevention and Treatment Block Grant FY 2009 allocation for Illinois is anticipated to be $69,617,036 based on the President's FY 2009 Budget Request. This figure is based upon a formula allocation determined by three elements: (1) the population of individuals needing services; (2) the cost of providing services; and (3) the state income level. There is not a line item budget submission requirement for the application. As part of the Federal funding requirements, Illinois must meet "Set Aside" requirements for expenditures for the following priority services and risk populations: 20% for Prevention, 5% for HIV and 5% for administration.

IDHS Summary of Program Evaluation Findings 2009

Program Summary

The substance abuse prevention evaluation covers programs funded by the Illinois Substance Abuse Prevention and Treatment (SAPT) Block Grant. The SAPT block grant requires that 20% of the block grant funds be spent on prevention activities. In Illinois, the funds are used to support a prevention infrastructure with different types of service providers:

  • Comprehensive Grant Program: 100 Comprehensive grants are awarded to not-forprofit organizations/governmental entities. These agencies deliver prevention services and build prevention capacity of coalitions and other organizations at the community level.
  • Statewide Programs: 14 Statewide Programs receive funding to serve large geographic areas or defined target populations.
  • Specialized Services Providers: 8 Specialized Services Providers are funded to target special populations including people with disabilities, youth in alternative educational settings, etc.

The goal of this prevention infrastructure is to prevent the onset of substance use and delay the progression of substance abuse. The primary emphasis is to prevent substance abuse among Illinois youth ages 10-17 through impacting the environments where they live, learn, and grow.

Process Information

During FY08, one hundred (100) Comprehensive Grant Providers delivered 1707 substance abuse prevention programs that served 258,752 people across Illinois. Fifty percent of the population served was between the ages of 12 and 17 with about three-fourths under the age of 21. Based on the race and ethnicity categories required by SAMHSA, 24% of the population served was African-Americans and 9% were identified as Hispanic. Geographically, prevention programs throughout the state directly served 45 Chicago community areas, 329 school districts, and 328 municipalities. In addition, prevention strategies were implemented countywide within 92 of the 102 counties in Illinois.

Illinois uses a variety of measures to monitor the performance of Comprehensive Grant programs individually and across the prevention system as a whole. Comprehensive Grant Providers are expected to devote 75% of their contract hours to implement evidence based programming. In FY08, 69% of the Comprehensive Grant Providers met that contract expectation. In addition, 88% of all programs delivered were considered evidence-based. Evidence-based programming includes model programs, strategies, and practices (for example, mentoring) that are recognized by prevention experts as best practices. It is important to note that these practices, programs, and policies include locally developed efforts, consistent with locally defined cultures that meet standards of practice that research has determined to be "essential ingredients" of effective and successful prevention programs.

To support accountability and ensure that programs are implementing best practice strategies, grant funded programs are evaluated annually. Comprehensive Grant Providers submit evaluation progress reports that present information on key indicators that define quality program implementation to the University of Illinois's Center for Prevention Research and Development. These reports are reviewed against a standard set of criteria based on the number of people served, completion rates, fidelity to their program plan, and sufficient levels of staff training. The results of these reviews are communicated in writing to funded agencies within 2 months. These results provide Illinois Department of Human Service staff and prevention managers with critical information on the functioning of funded programs and whether they meet state prevention standards and goals. In FY08, 55% of the funded agencies met the standard criteria for high quality program performance.

Outcome Evaluations

Each year, based on program performance reviews described above, a subset of agencies is selected to participate in a two to three-year outcome evaluation using a quasi-experimental design that employs a comparison group. In FY08, 3 Comprehensive Grant programs completed the outcome evaluation. An additional 2 agencies employ annual outcome data collection to track the progress toward addressing substance abuse issues via school-based social norms marketing campaigns. The results from these outcome evaluations are presented below based on the type of program evaluated.

Botvins Life Skills Training (LST) was evaluated in one rural and one urban/suburban setting. All three years of the program (core and two years of booster sessions) were delivered across all evaluation sites. The target population was 6th-8th graders and all sites were based in classroom settings during the school day. All evaluations included a comparison sample against which trends for substance use, attitudes and knowledge could be more confidently interpreted.

The following outcomes were demonstrated in program intervention groups relative to the comparison groups:

  • Increased intent to use ATOD resistance skills
  • Increased assertiveness skills
  • Delayed intent to use cigarettes in the future
  • Decreased perceptions that most adults smoke

Social Norms Marketing (SNM) is a prevention strategy based on the theory that youth choose to use alcohol, tobacco and other drugs because they believe use is the norm among their peers. In fact, research has demonstrated that adolescents consistently overestimate both the prevalence of use and the frequency of use. SNM requires that data are collected locally about the actual use rates and that messages are disseminated about the factual levels of use to highlight that use is not normative.

Two Comprehensive Grant Providers collect annual data to monitor the progress of their SNM efforts in three suburban high schools. Two of the three high schools are located in suburbs with a high proportion of Hispanic youth. There are no comparison groups against which to judge the changes observed over time.

  • During an SNM campaign conducted from 2006-2008, there was a 2% decrease in smoking and a 2% decrease in alcohol use observed among students in one high school.
  • During an SNM campaign conducted from 2004-2008, there was a 2% decrease in smoking and a 1% decrease in alcohol use observed among students in another high school. Notably, the campaign targeted these two substances but did not target marijuana use norms. During the same period, there was a 3% increase in marijuana use.
  • During an SNM campaign conducted from 2005-2008, there was a 10% decrease in alcohol use and a 4% decrease in smoking among students in the third high school.

One additional Comprehensive Grant Provider participated in a 5 year study of the impact of SNM on alcohol use among high school students from 2003-2008. This study employed a quasi-experimental design and used census level data collection within each school. Exposure to the SNM campaign:

  • had a significant effect on whether subjects believed most students at their school used alcohol at least once per month.
  • had a significant effect on whether subjects believed most students at their school get drunk at least once per month.
  • resulted in lower rates of 30 day alcohol use among subjects during years 2, 3 & 4 despite starting the study with significantly higher consumption rates than the comparison youth.

Substance Abuse Prevention Block Grant Evaluation

Evaluation Design Peter Mulhall, Ph.D. and Beth Welbes, M.S.P.H.

Indicators / Measures Tools / Instruments / Data Sources Other Deliverables / Comments

Substance Abuse Prevention Program

Process

  • % of agencies delivering at least one program deemed "ready for outcome evaluation"
  • % of programs deemed "ready for outcome evaluation"
  • 75% of contract hours are devoted to evidence-based programs, policies and practices
  • 100% of contract hours are delivered (with a10% variance given)

Outcome

  • Decrease in alcohol, tobacco and other drug use among youth
  • Decrease in risk factors and increase in protective factors contributing to youth alcohol, tobacco and other drug use

On Track MIS system developed and maintained by CPRD through this contract.

Readiness for Outcome Evaluation review process conducted annually

Comprehensive Grant Program annual work plans

Illinois Youth Survey

Social Norms Marketing Survey

Botvin's Life Skills Program Evaluation Survey

Biannual Provider Survey

This evaluation contract includes a wide array of evaluation services including development of the data collection systems and protocols described in this report and other deliverables such as:
  • Provide training and technical assistance to providers on evaluation issues;
  • Prepare research briefs that summarize outcomes and implementation standards for prevention approaches included in the CGP workplan;
  • Conduct a Provider Survey every two years;
  • Review all locally developed curricula for adherence to SAPP-mandated standards;
  • Create documents describing success stories of CGPs who have demonstrated high quality implementation; and,
  • Prepare and submit prevention data required by the annual SAPT Block Grant online application