DHS Program Evaluations - By Division

Ended FY06 and/or Ongoing

Grant ID DHS Contract Mgr

Annual

Grant $

Evaluator

Name & Contact Info

Evaluator Affiliation Annual Eval $ Funding Source Eval Type Eval Period Freq. of Reports

No. Reports on File

As of ___

Indicators / Measures Tools / Instruments / Data Sources Other Deliverables / Comments
DASA
Access to Recovery (ATR) Rex Alexander $7,575,685 Richard E. Sherman, Ph.D. IOTA, Inc. $240,000 SAMHSA/CSAT Data collection and analysis 08/03/04 - 08/02/07 Semi-annually; additional reports as needed 1 as of 6/30/2005

Process:

  • # of Clients Enrolled
  • # of Vouchers Issued
  • # of Service Billings
  • Status GPRA Completion Rate by Time Interval
  • # of Treatment Providers Enrolled
  • # of Recovery Support Providers Enrolled
  • # of Faith-based Providers Enrolled

Outcome:

Abstinence Rates among Participating Clients

CSAT ATR GPRA Tool

Service Voucher Forms

Client Recovery Support Plans

Service Encounter/Billing Forms

Client Toxicology Reports

Treatment and Recovery Support Provider MOUs and Service Profiles

Data collection and analysis activities are focused on ensuring compliance with federal reporting requirements. CSAT ATR GPRAs are required at project enrollment, 30 days after enrollment, and every 60 days thereafter as long as the client remains active in ATR-supported treatment or recovery support services (status GPRAs). Random drug screens must also be completed at least twice annuallly.
Adolescent Substance Abuse Treatment Coordination Initiative Danielle Kirby $400,000 Maria Bruni, Ph.D., and Beth-Anne Jacob, Ph.D. Independent $39,820 SAMHSA/CSAT Process and Outcome 7/1/05 - 6/30/08 Quarterly and annually New 05

Process:

  • The degree to which the project adheres to its plan to increase and improve elements of its system
  • Existing barriers to the planned increases and improvements
  • The occurrence of infrastructure change
  • Perceptions of key stakeholders

Outcomes:

  • Changes in access to treatment
  • Changes in clinically appropriate assessment skills and interventions for treating adolescents
  • Changes in client outcomes as a result of all other changes.

GAIN

DARTS

Interviews

Focus groups

GPRA satisfaction survey

An evaluation team of DASA contractors will conduct the evaluation.
Chicago Collaborative Initiative to end Chronic Homelessness Kellie Gage $628,312

Elizabeth Durkin, Ph.D.;

John Lyons, Ph.D.

Northwestern University $57,616 SAMHSA Process and Outcome 9/30/03 - 9/29/06 Monthly 17 as of 7/31/2005

Process:

  • Monitoring of treatment fidelity
  • Assessment of collaborations/systems intergration

Outcome:

Change in self identification of individual and family's needs and strengths over six months

  • The Dartmouth Assertive Community Treatment Scale (DACTS), Motivational Interviewing Integrity Code (MITI)
  • Adult Needs and Strengths Assessment (ANSA-TFI)
  • Qualitative Interviews
  • Participation in Project Management Team Meetings.

Evaluator to provide training in evidence-based practice (Motivational Interviewing, Stages of Change, Assertive Community Treatment)

Project is also part of national evaluation, not funded through DHS, overseen by the Veterans Administration.

Screening, Brief Intervention and Referral to Treatment (SBIRT) Deborah Levi, Ph.D. $3,500,000 Richard E. Sherman, Ph.D. IOTA, Inc. $35,000 SAMHSA/CSAT Process 09/30/03 - 09/29/08 Semi-annually; additional reports as needed 1 as of 04/2005
  • # of Screening Only Patients
  • # of BI Patients
  • # of BT Referral Patients
  • # of Other Treatment Referral Patients
  • Characteristics of Patients by Service Categories
  • Various Forms of Project Documentation

CSAT GPRA Tool

Project Screening and Assessment Tools

Service Referral Documentation Tools

The process evaluation includes monitoring project progress toward project implementation, achievement of patient enrollment targets, patterns of referral to service providers, assisting in fulfillment of project federal reporting requiremennts and analysis of patient demographics.
Screening, Brief Intervention and Referral to Treatment (SBIRT) Deborah Levi, Ph.D. $3,500,000 Christy K Scott, Ph.D. Chestnut Health Systems $315,000 SAMHSA/CSAT Outcome 09/30/03 - 09/29/08 Semi-annually; additional reports as needed 1 as of 04/2005

Process:

  • % of BI Patients Interviewed
  • % of BT Patients Interviewed
  • % of Patients Referred to Other Treatment interviewed

Outcome:

- Alcohol and Other Drug Abstinence Rates by Service Category and Substance of Use

CSAT GPRA Tool

Patient Satisfaction Tool

Patient interviews

The outcome evaluation consists of conducting six-month post-enrollment folllow-up interviews with a random sample of patients who are provided brief intervention or referral to treatment services. It is an expectation of the federal funder that at least 10% of patients in these servivce categories are actually interviewed for six-month follow-up.
Illinois Strengthening Treatment Access and Retention - State Implementation (STAR-SI) Project Peggy Alexander $325,000 Richard E. Sherman, Ph.D. IOTA, Inc. $28,000 SAMHSA/CSAT Process and Outcome 9/30/06 - 9/29/09 Semi-annually 0 as of 11/06

The Illinois STAR-SI project will collaborate with five treatment provider organizations in southern Illinois.

All STAR-SI states are required to report the baseline and periodic status of the following service and retention measures.

  • Average Units of Service Provided to Admitted Outpatients will measure the average level of treatment participation among admitted outpatients.
  • Average Length of Stay--average number of days that clients remain open in outpatient services.

Illinois has decided to additionally report on the following service access and retention measures.

  • Average Time to Assessment Appointment-from date client first contacts the agency.
  • Average Time from Assessment to First Outpatient Session.
  • Average Time from First to Second Outpatient Session.
  • Early Client Engagement Rate will measure the successful completion of four group or individual sessions within thirty days of admission.
TCE-HIV East St. Louis Kellie Gage $500,000 Cheryl Mejta, Ph.D. Governors State University $27,316 SAMHSA / CSAT Process and Outcome 10/01/01 - 9/30/06 Quarterly and annually 14 as of 7/31/05

Process:

  • Static capacity of 50 Methadone slots
  • Behavioral risk assessments for 100% of clients
  • HIV/HCV screens for 100% of clients
  • Behavioral risk reduction groups for 100% of clients
  • Family behavioral risk reduction education group for 100% of clients
  • Fidelity to treatment model (Methadone dose, counseling, relapse prevention, recovery coach activities)
  • Fidelity to mental health enhancements (Link to medical care, MISA counseling groups, screen for MH, link to MH care, Psych adherence groups)

Outcomes:

  • Increased participation in employment or education activities
  • Reduced drug/alcohol use
  • Reduced HIV/HCV behaviors
  • Improved physical health self report
  • Improved psychological health

Client intake forms

Behavioral Risk Screening Insturment

GPRA

Medical screens

Client records

Recovery Coach caseload size

Referral agreements and linkages

Psychological screens

Total $742,752
CHP
Adolescent Family Life Denise Simon $350,000 Peter Mulhall, Ph.D.

Center for Prevention Research and Development (CPRD)

University of Illinois at Urbana-Champaign

$75,000 DHHS/ Office of Adolescent Pregnancy Programs Quasi-experimental 10/1/05 - 9/30/10 Annual

0

Grant new as of 10/1/05

  • % of services provided, type of services, appropriateness
  • Use of culturally or developmentally appropriate methods
  • Changes in services /instruction or delivery
  • % of services, participants attend and complete
  • % of instructors who report program/services is excellent
  • % of students reporting program/services as good or excellent
  • Instructor satisfaction with training and TA
  • Support from state staff, site manager, and/or supervisor
  • Age, gender, race, ethnicity, immigration status
  • Income
  • High School degree
  • No. and types of immediate and extended family, where they reside
  • Living arrangement- self, family, spouse, etc.
  • Reduce problem behavior (alcohol, drug use, crime, violence, etc.
  • Increase number and types of positive social interactions with family members
  • Perceived social support for self and family
  • Reduction in depression and anxiety
  • Increase in responsible sexual behavior
  • Increase employment/career goals and aspirations; Complete high school education and beyond
  • Increase educational expectations and aspirations
  • Increased parenting knowledge and skills
  • Increase parental competency, attachment and commitment
  • Increase the percent the babies meeting developmental milestones
  • Individual care plans
  • Classroom direct observation
  • Satisfaction surveys TBD
  • Participant attendance and activity logs
  • Staff surveys TBD
  • Focus groups
  • Intake assessments
  • Service logs
  • Risk and Protective Survey (Hawkins and Catalano 1998)
  • Social Support Network Questionnaire (Barrera, 1981)
  • CED Depression Scale (Radloff)
  • Participant interviews
  • Staff interviews
  • Other participant, family, and staff surveys TBD
Evaluator must also coordinate with and participate in cross-site evaluation.
Healthy Families Illinois Denise Simon $8,450,000 Julie Krouch Office of Sponsored Projects Northern Illinois University $333,000 GRF Quasi-Experimental 7/1/00 - 6/30/06 Quarterly and semi-annually 13 as of 7/26/05

Process:

  • Well-baby checks
  • Immunizations
  • Screens for developmental delay

Outcome:

  • Improved parent-child relationships
  • Reduced child abuse and neglect
  • Referrals as appropriate to providers of care for developmental delay, etc.
  • Child Abuse Potential Inventory
  • Nursing Child Assessment Satellite Training (NCAST) Teaching Scale
  • HOMES (Home Observational Measurement of the Environment Scale) Scale
  • Cornerstone (MIS)
  • Interviews
Evaluator to provide training in administering the tools/instruments
Illinois Steps Ahead (GEAR UP) Ivonne Sambolin $3,500,000 Peter Mulhall, Ph.D.

Center for Prevention Research and Development (CPRD)

University of Illinois at Urbana-Champaign

$300,000 US Dept of Education Process and Outcomes 10/1/05 - 9/30/11 Annual
  • Increased number of students who enroll in and complete college prep courses (e.g., pre algebra, Algebra 1, etc.)
  • Improved students academic performance (grades, achievement test scores)
  • Improved school attendance
  • Increased rate of grade level promotion
  • Increased number of students who take college entrance exams
  • Increased number of students who enroll in post-secondary education
  • Increased perceptions of access to higher education
  • Improved parent expectations and aspirations related to college
  • Increased knowledge/understanding of college financial options
  • Increased number of completed applications for college financial aid
  • Increased number of college visits
  • Increased number of students who report having an adult to talk to about establishing educational goals, planning for college and/or career.
  • Increased educational e xpectations and aspirations related to college attendance, success, and graduation.

Improved students' expectations for a productive future.

Increased parent involvement in their child's education.

Improved parent educational expectations and aspirations related to college attendance, success, and graduation.

Improved teacher expectations for academic performance of low-income students.

Increased contact between schools and families related to college preparation and career planning.

Increased type and number of strategies that teachers use to promote educational success

with "at-risk" students.

  • Student surveys
  • Parent surveys
  • Teacher surveys
  • Academic records (already available through the Teen REACH evaluation)
  • Achievement test scores
  • Program attendance records
  • Illinois State Board of Education (ISBE) School Report Card Data
  • Site visits and focus group discussions with program staff, collaborative partners, and the students and families served by Steps AHEAD
Training and support will be provided to statewide project staff and local program staff as they engage in the ongoing evaluation process.
Substance Abuse Prevention Kim Fornero $20,277,712 Peter Mulhall, Ph.D.

Center for Prevention Research and Development (CPRD)

University of Illinois at Urbana-Champaign

$1,062,888 SAMHSA Process / Outcome 7/1/90 - 6/30/06 Quarterly and Annually 15 as of 7/26/05

Substance Abuse Prevention Program

  • Overall State progress related to State plan
  • State-level trends, needs assessment, training needs of prevention workers
  • Quality of 300 Community Based Provider (CBP) annual work plans
  • Compliance of CBP evaluation progress report submission
  • Full outcome study of selected CBPs
  • 30-day prevalence
  • age of 1st use
  • risk & protective factors
  • perceived risk
  • others
  • Collaboration/coalition effectiveness of In Touch providers
  • In Touch Providers' evaluation needs
  • Description of In Touch annual reports
  • High School Social Norms Marketing Effectiveness Study: Youth Survey and Parents Survey

Safe to Live

  • assessment of the quality and efficiency of the community based planning and coalition process
  • assessment of the quality and effectiveness of the programs and strategies that emerge from each site's plan

Strategic Prevention Framework

  • Baseline leadership survey with project staff
  • Effectiveness/appropriateness of state instruments
  • Community-level evaluation

On Track MIS system developed and maintained by CPRD through this contract.

InTouch annual reports

CBP annual evaluation progress reports

CBP annual work plans

Government Performance Results Act (GPRA)

National Outcome Measures (NOMs)

CPRD-developed surveys

Interviews

Evaluation Committee meeting minutes

State Epidemiological Workgroup meeting minutes

Advisory Council meeting minutes

This is a complex, multi-tier evaluation of State systems, In Touch programs, all community-based providers of prevention services, system support, and the Strategic Prevention Framework program. The project director and evaluator together determine evaluation elements annually related to providers funded by the Substance Abuse Prevention Block Grant.

Other deliverables:

  • Provide training and technical assistance to providers on evaluation issues
  • Draft and submit a publication on the analysis of project findings to date from the social norms study.
  • Prepare research briefs for a selected set of prevention approaches that were researched but not incorporated into the CBP FY06 work plan (youth engagement / leadership, coalition, youth development).
  • Submit a cross site summary of enhanced outcome evaluation findings for the first three years of the enhanced outcome evaluation model (FY03, FY04 & FY05).
Targeted Intensive Case Management Beverly English $4,693,400 James W. Masterson Hamilton-Bell Associates $168,600 GRF, MCH Block Grant & Healthy Start Process/Outcome 7/1/01 - 6/30/06 Quarterly and Annually 10 as of 7/26/05

Process:

Monthly caseload by project site

Staff retention by site

Continuity of case management during prenatal period

Prevalence of top five risk factors among participants (disease; previous pre-term birth; 4th pregnancy or 3rd child; low education level; homelessness/temporary housing)

Month of pregnancy at intake

Exposure to model

Length and type of prenatal care

Providers' correct input into Cornerstone

Outcome:

  • receiving adequate prenatal care
  • referred to WIC
  • linked to "medical home"

lowered infant mortality

lowered rate of NICU use

  • Data obtained from Cornerstone
  • Kessner Index
  • Interviews
  • Quarterly data review meetings
Deliverables:
  • A series of analyses that describe the demographic and health care behavior attributes of program participants
  • A series of analyses that detail the operations of TIPCM and Healthy Start
  • An analysis of TIPCM and Healthy Start to determine each program's effectiveness in improving birth outcomes and reducing health care costs
Total $1,939,488
DD
Telehealth Rod Curtis, M.D.

$233,959

(1st yr only)

Tess Ford, Ph.D.

618

453-1732

Southern Illinois University, Center for Rural Health and Social Service Development $37,500 HRSA / OAT Process and Outcome 10/01/06 - 09/30/09 Annually 0 as of 12/06

Technical aspects of each telehealth clinical case review and educational event.

Quality of service delivery for clinical case review

  • Cost effectiveness
  • Timeliness of clinical case review process
  • Level of professional involvement
  • Number and type of professionals involved
  • Efficacy of care
  • Collect data on health promotion issues, including Weight (cardiovascular and diabetes risk factor); BMI; Blood Pressure (heart disease, kidney disease and stroke prevention); recent fasting glucose level (for diabetes); recent fasting lipids (for cholesterol management - heart disease and stroke prevention); osteoporosis risk.
  • Medication sides effects
  • Clinician satisfaction with technology and support
  • Community agency satisfaction with technology and support
  • Education participant satisfaction with technology and quality of education sessions

A series of evaluation instruments and tools will be developed to collect data for the project evaluation.

Questionnaire regarding technical aspects of audio and video broadcast administered to technical support staff, education participants, and consumers

Questionnaire specific for each clinical venue that assesses professional involvement compared with clinical case reviews obtained by community agencies not involved with the telehealth project

MOSES (standardized side effect instrument for people with DD)

DISCUS (standardized instrument for TD assessment)

Total $37,500
HCD

No

Evaluations

$0
DMH

Community Reintegration Collaborative

(Jail Diversion)

Anderson Freeman, Ph.D. $400,000 Lisa Rozzano, Ph.D. University of Illinois-Chicago Mental Health Services Research and Policy Center $57,400 SAMHSA Implementation/outcomes 10/1/05 - 9/30/08 Quarterly and semi-annually

0

grant new 10/05

Process

  • Fidelity to ACT model
  • Fidelity to trauma services model
  • Service use
  • "Diversion events"
  • Ineligibility determinations

Outcome

  • Arrests
  • Jail days
  • Participants' quality of life
  • Participants' satisfaction w/ services
  • Participants' mastery/self efficacy
  • Meeting Minutes: Attendance
  • client charts
  • Interviews
  • IDDT Fidelity Scale
  • Dartmouth Assertive Community Treatment Scale (DACTS)
  • Illinois DataLink
  • Consumer Survey (IAPSRS)
  • Questionnaires: TBD
  • Semi-Structured Interviews:
Evaluation takes into account both planning and implementation phases of the program.
Family CARE (McHenry) Pat Roy, M.D.

Fed FY06

$999,980

Fed FY 07

$1,499,444

Fed FY 08

$1,998,223

Fed FY 09

$1,989,977

Fed FY 10

$1,500,000

Fed FY 11

$998,168

Joel B. Cowen MA

- Principal Investigator

815-395-5762

Andrea Doughty, Ph.D. -

Lead Evaluator

815-395-5756

University Of Illinois - Rockford College Of Medicine Health Systems Research Office

Fed FY06

$139,991

Fed FY 07

$172,577

Fed FY 08

$177,463

Fed FY 09

$182,496

Fed FY 10

$187,679

Fed FY 11

$193,018

SAMHSA Implementation / Process / Outcomes 10/1/05 - 9/30/12

Yearly Re-App-lication due in March

Year end Progress Report due in October

(1) - Re-Applica-

tion -

3/15/06

(1) Progress Report

as of 10/31/06

  • Increase number of children receiving services
  • Improve children's outcome and systems outcomes: (a) Increase percentage attending school 75% or more of the time after 12 months (b) Increase percentage with no law enforcement contacts at six months (c) Decrease utilization of inpatient facilities at six months
(d) Decrease inpatient costs
  • Sustain funding past federal funding 3.
  • Decrease inpatient costs by at least 10%
Local Evaluation
  • Child and Adolescent Needs Survey CANS
  • Client intakes
  • Interviews
  • National evaluation instruments
  • Other surveys TBD through on-going local evaluation community planning
Conduct local evaluation and coordinate participation in national evaulation
  • Ensure family participation in data collection, reporting, interpretation and planning
  • As of 10/1/06, awaiting approval of National Instruments by Federal OMB. Evaluation proposal has been submitted to IRB. Site awaiting approval.
  • Formal National Evaluation activities planned to commence, tentatively, November 1, 2006
  • Family CARE will enroll a minimum of 354 youth and their families within the National Evaluation between grant years 2-5 (avg. 89 per year)
  • Family CARE must retain 276 of 354 (80%) enrolled through 36 months of their respective enrollment date.
System of Care - Chicago Tanya R. Anderson, M.D.

$2,000,000 in FY06

varies each grant year

John Lyons, Ph.D. Mental Health Services and Policy Program Northwestern University $185,000 in FY06 SAMHSA / CMHS Process / Outcome 11/01/04 - 9/29/08 Monthly and as requested 1 annual and 24 monthly as of 9/05

Process:

  • Evolution of system of care infrastructure
  • Relationship of SOC infrastructure to educational and treatment outcomes
  • Integration of findings from evaluation into ongoing quality improvement methods and measures (including elimination of service duplication)

Outcome:

  • Effectiveness of service model
  • Improved functioning of children/youth receiving services
  • Capacity of families to advocate for youth
  • Effectiveness of training
  • Need for increased local and state resources
  • National evaluation (i.e., ORC Macro) protocol
  • Child and Adolescent Needs and Strengths (CANS) assessment tool
  • PBIS Educational Information Form
  • Local evaluation surveys to youth
  • Local evaluation surveys to families
  • Local evaluation surveys to school staff/collaterals and other collaborating partners
  • Service data received from participating sites
  • Focus groups
Conduct local evaluation and coordinate participation in national evaluation
Total $414,977
DRS
None
DHS Total $3,134,717

Addendum

Separately Funded National Evaluation That Includes an HCD Grant Program

Grant ID DHS Contract Mgr

Annual

Grant $

Evaluator

Name & Contact Info

Evaluator

Affiliation

Annual Eval $ Funding Source Eval Type Eval Period Freq. of Reports

No. Reports on File

As of ___

Indicators / Measures Tools / Instruments / Data Sources Other Deliverables / Comments
Employment Retention & Advancement Teresa Tudor

$1,500,000

(total funding Feb 2002-June 2004 was $3,589,232

Program was not funded through DHS)

Dan Bloom

212-532-3200

MDRC Unknown

DHHS, Admin for Children and Families, Office of Planning, Research and Evaluation

(refers to evaluation funding, not program funding)

Experimen-tal Oct 2000 - Sept 2007 Every 18 months

3

as of July, 2005

Ever employed (%)

Avg Quarterly employment (%)

Employed 4 consecutive quarters (%)

Earnings ($)

Earnings over $10,000 ($)

For those employed Year 1:

Avg quarterly employment (%)

Avg earnings per quarter employed (%)

Ever received TANF (%)

Amount of TANF received ($)

Ever received food stamps (%)

Amount of food stamps received ($)

Total measured income ($)

  • Unemployment-Insurance covered jobs attained
  • TANF participation records
  • Food stamp administrative records
  • MDRC-developed surveys
This is a national evaluation of ERA projects contracted by HHS directly with MDRC-we do not know the evaluation contract amount. The evaluation has been following up on clients of the ERA program, which ended in June, 2004. The annual grant amount was the award while the program was operational, from Feb 2002 - June 2004.