DHS Contract Mgr Kim Fornero
Annual Grant $ $2,350,965
Evaluator Name & Contact Info

Peter Mulhall, Ph.D.

Beth Welbes, M.S.P.H.
Center for Prevention Research and Development (CPRD)
University of Illinois at Urbana-Champaign

Annual Eval $ $265,545
Funding Source SAMHSA
Eval Period 10/1/04 -9/30/09

Evaluation information was submitted via e-mail by Beth Welbes, MSPH, August 7, 2009.

Program Summary

The Substance Abuse Mental Health Services Administration (SAMHSA), Center for Substance Abuse Prevention (CSAP) awarded the Strategic Prevention Framework State Incentive Grant (SPF-SIG) to Illinois in October 2004. The goals of the SPF-SIG are to:

  • Prevent onset and reduce progression of substance abuse, including childhood and underage drinking;
  • Reduce substance abuse-related problems in communities; and
  • Build prevention capacity and infrastructure at the state and community levels.

CSAP requires the creation and maintenance of two statewide groups to provide leadership and oversight for the SPF-SIG grant. The Statewide Advisory Council (SAC) consists of agency directors and policy makers who use the data in order to develop, review, and monitor the progress of the state's SPF strategic plan. The Director of the Illinois Department of Alcoholism and Substance Abuse, Theodora Binion-Taylor, chairs this advisory council. Dr. Myrtis Sullivan (Associate Director, Office of Family Health Services, Division of Community Health and Prevention, Illinois Department of Human Services) chairs the State Epidemiological Outcomes Workgroup (SEOW). These two groups, the SAC and the SEOW, provide a platform for leadership and coordination at the state level. The SEOW (epidemiological workgroup) is specifically charged with analyzing the substance abuse indicators data to determine statewide needs and recommend strategies to improve the framework.

In June 2007, the Illinois Department of Human Services (IDHS) awarded 85% of the grant funds to 18 community-based agencies. IDHS intentionally funded some communities with pre-existing coalitions (72%) and some without pre-existing coalitions (28%) that would presumably require more mobilization efforts to implement that SPF. Just over one-half of the funded communities were in urban settings, 28% were located in rural settings, and the remainder were in suburban settings (17%). The grantees are now implementing the strategic prevention framework which allows them to make data-supported decisions about priorities and programming at the community level.

State Level Evaluation Activities

The state level evaluation has been monitoring progress of the project since its inception in 2004. At least one member from the CPRD evaluation team attends all SEOW and SAC meetings. All SEOW and SAC activities are tracked and monitored with a SPF/SIG Event Tracking Database, developed by CPRD for monitoring all aspects of the SPF/SIG project. This database allows for tracking: attendance at each meeting; decisions reached; level of participation of all stakeholders and leadership involvement; and the decisions reached on controversies/obstacles identified. This database also tracks all key project events, including activities such as the development of the State Epidemiological Profile, completion of the Comprehensive State Plan for Prevention, and process for funding the 18 SPF/SIG communities.

A key focus of SPF-SIG is to change the way state agencies work together to achieve prevention goals. The project expects that the state's prevention agencies will utilize the data and outcomes learned to better strengthen and focus their impact in the SPF-SIG communities. The SEOW has already demonstrated collaborative action by identifying other epidemiological focused organizations and groups throughout the state, compiling a database of prevention resource dollars per youth capita by county, and has begun to develop a process to review data sources for indicators of quality. In the spring of 2010, a follow-up survey will be conducted with SEOW and SAC members to determine to what extent agencies have adopted one or more components of the SPF. A final round of key informant interviews will be conducted to provide context for those changes. Through these data sources, the goal of state-level adoption of the SPF will be examined.

Community Level Evaluation Activities

The community-level outcome evaluation will be completed in June 2010. All but two grantees will be gathering community-wide youth survey data to determine if 30-day alcohol use rates were impacted by the implementation of prevention strategies. One of the remaining grantees will gather college youth survey data to ascertain if binge drinking has been reduced. The final grantee will compare baseline county-wide alcohol-related motor vehicle fatality rates with rates following implementation of prevention strategies. In addition, rates of relevant local contributing factors (e.g., easy alcohol access) will be compared from baseline to follow-up. Youth survey baseline data were gathered in Spring 2008 and follow-up data will be gathered in Spring 2010. The community-level outcome evaluation will reflect changes in alcohol use and consequences after 1 to 11/2 years of prevention strategy implementation. Local evaluations are supported through local "data coaches" hired by grantees. The role of the data coach is an important focus of the cross-community evaluation conducted by CPRD.

SPF Community Level Implementation Steps 1-3: Progress Report Findings

During FY08, all Illinois grantees implemented the first three steps of the SPF (Needs Assessment, Capacity Building, and Planning). CPRD created a progress report that chronicled their experiences. Key findings from the progress report are included below. The report explores the process grantees engaged in to assess "risky alcohol use" for their community; how they mobilized and built local capacity to address alcohol use (including developing their community coalition); and how they strategically planned and selected evidence-based, culturally competent programs, practices, and/or policies to address problems. IDHS provided training on each step of the process to grantees and provided templates for required assessment and strategic plan reporting forms.

In terms of Needs Assessment, all grantees conducted population level surveys, environmental scans of alcohol promotion/pricing/norms, community readiness assessments, and community resource assessments. For many, this was the first time the coalition used locally collected data to guide decision-making. Benefits of assessment included bringing more credibility to the coalition and having the ability to "prove" that risky alcohol use is an issue despite community denial. Some developed their own data collection tools and others used the IDHS-provided tools. Coalitions experienced challenges integrating the assessment activities when they were not accustomed to action-oriented collaboration (i.e., their historical focus had been networking and information sharing). The data collection expectations were quite intensive and proved very complex for some grantees.

The Capacity Building step focused largely on coalition member recruitment/retention and training members how to implement the SPF process. All grantees were successful in developing a coalition with at least 8 community sectors represented (a contractual expectation). Coalition members rated their level of structure and function very favorably. The chief challenge across grantees was engaging critical but hard-to-reach populations like youth or racial/ethnic minorities. Grantees voiced a need for training materials to help them bridge the gap between the training they received on the SPF and the capacity building they were expected to accomplish locally.

Strategic Planning was the final step all grantees had accomplished by the end of FY08. Many grantees engaged a subset of their coalition in the planning process. They were expected to develop a logic model that outlined what their local priorities were and how they would address them. Logic models proved to be challenging for grantees to complete, with some "disconnects" between their objectives, outcomes, and activities. Additional technical assistance was provided by IDHS to firm up the logic models. All plans were submitted and reviewed for completeness and quality by a review team comprised of IDHS staff, SEOW members, SAC members, and CPRD staff.

There were additional observations across all steps of the process:

  • Successful implementation of SPF Steps 1-3 is more likely when at least two of the key players (Project Coordinator, Community Coach, and Coalition Leader) have a high capacity to lead, negotiate, and facilitate. Role confusion was more prevalent when one or more key players did not have sufficient capacity or experience with expectations associated with their roles.
  • While the structure of required forms and data collection tools helped many of the coalitions that were new to assessment and planning, sometimes the purpose of the forms and tools was lost. It is important to carefully review the purpose of each tool (assessment, planning, and reporting) to assure that information is not unnecessarily repetitive and that all information will be used for a specific purpose.
  • Most grantees defined their "community" by geographic boundaries. Grantees that defined their "community" by population characteristics (i.e., race/ethnicity, membership in a group) found the SPF process most difficult to implement because the process is focused on community-level changes rather than individual-level change. Most data collection systems are organized around geographic jurisdictions (e.g., cities, counties).
  • Suburban coalitions experienced the fewest barriers to SPF implementation. The level of resources, identification with a geographic community, and reasonable size of the target population all seemed to influence assessment, capacity building, and planning. Rural coalitions consistently found distance an impediment to capacity building and often dealt with multiple jurisdictions in the assessment and planning phases. Urban areas seemed to have many competing issues often related to population density.

Strategic Prevention Framework State Incentive Grant

Evaluation Design Peter Mulhall, Ph.D. and Beth Welbes, M.S.P.H.

Indicators / Measures Tools / Instruments / Data Sources Other Deliverables / Comments

Process:

  • Develop a reliable database of substance abuse prevention-related services funded by all state agencies
  • Develop and implement a methodology for data analysis that integrates need with services available across the State of Illinois.
  • Develop a data-driven plan substance abuse prevention in the State of Illinois.
  • Develop a data-driven plan for reducing risk factors, enhancing protective factors, and preventing the initiation, progression and negative consequences associated with substance abuse in each community selected for sub-recipient funding.

Outcome:

  • Reduce use of alcohol in the PAST YEAR among youth age 10-20 by at least 10% within each sub-recipient community.
  • Reduce use of alcohol in the PAST MONTH among youth ages 10-20 by at least 10% within each sub-recipient community.
  • Demonstrate a reduction in at least one alcohol-related problem targeted by each sub-recipient community
  • Enhance collaboration between prevention, intervention, treatment and extended care providers at the state and sub-recipient level
  • Improve the coordination and integration of state data systems to guide decisions about where and how resources should be directed.
  • Demonstrate adoption of or endorsement of the Strategic Prevention Framework among Illinois agencies and organizations committed to substance abuse prevention.
  • Leverage existing state resources to enhance coordinated planning, service delivery, professional development, and evaluation infrastructure for substance abuse prevention.

State Level Evaluation:

  • Meeting Tracking and Key Milestones database
  • State Advisory Council Member Survey
  • State Epidemiogical Outcomes Workgroup Member Survey
  • Key Informant Interviews

Community Level Evaluation

  • Quarterly Grantee Reports
  • Grantee Assessment Reports
  • Grantee Strategic Plans
  • Strategic Plan Ratings
  • Implementation Fidelity Ratings
  • Grantee Implementation and Evaluation Plans
  • Semi Annual Community Level Instrument (CLI) Evaluation data obtained from the national evaluation contractor
  • Coalition Member Surveys
  • OnSite and Phone Key Informant Interviews

The community level grantees are expected to participate in three levels of evaluation:

Local evaluation is a step in the SPF. Funds for local data coaches are allowable expenses. These data coaches are intended to build local capacity to use data to guide decisions.

National cross-site evaluation is implemented through a federal contractor (Westat). Grantees are expected to submit National Outcome Measures annually and complete an online semi-annual progress report called the Community Level Instrument.

Cross-state evaluation is executed by CPRD. the goal of the cross-state evaluation is to provide an overview of what changed in local communities (based on objectives outlined in strategic plans), key accomplishments of community grantees, barriers experienced by community grantees and lessons learned about community level implementation of the SPF. CPRD relies heavily on the data already gathered for the national cross-site evaluation and documents submitted by grantees to DHS per contractual requirements to minimize data collection burden on community grantees.