1. Overview of the Evaluation Plan
  2. Recruitment Procedures
  3. Data Collection Procedures
  4. Interviewers

Overview of the Evaluation Plan

Since its inception, Healthy Families Illinois has been, and remains, committed to rigorous evaluation of its outcomes.  The design of the statewide evaluation is noteworthy in the following ways.

Inclusion of a large, statewide sample of parents.

In total 4,506 Illinois parents supported the HFI evaluation effort by agreeing to share their time and open their homes to the data collection procedure.  Participants were referred from over 40 different HFI agencies across the state, allowing for an evaluation of the state-wide system as a whole.  Thus, the evaluation findings are not limited to outcomes associated with only one or two selected sites.  This large-scale recruitment effort has resulted in the creation of a sample that is racially, ethnically, and geographically diverse.

Tracking changes across time.

The HFI program evaluation involved tracking changes in families across a two-year time frame.  As has been suggested by others, in many families the beneficial effects of HFI services may not be immediately apparent.  Following families across time, although logistically challenging, provided an opportunity to more fully appreciate the impact of HFI on the life course of parents and their children.

Use of multiple methods for measuring changes in the parent-child interaction.

Reliably capturing changes in the parent-child interaction is a challenge faced by all child abuse prevention efforts.  The HFI evaluation involved multiple methods of assessing change including face-to-face interviews, in-home observations, and self reports.  Interviewers completed four days of intensive training and recertification in order to ensure reliable assessment of changes in families across time.

Recruitment of comparable comparison families.

Although a randomized trial was not deemed appropriate in this context, a comparison group design is integral to the meaningful demonstration of HFI's impacts.  Comparison parents were carefully selected and met the following criteria.

  1. exhibited the same level of risk for problems in parenting as HFI service recipients,
  2. were motivated to receive HFI services if such were available,
  3. were unable to receive HFI services because of limited resources in their service area.

Comparison parents were assessed in exactly the same manner as intervention families across time.  This procedure provided important data indicating how families change naturally over the course of time.  The comparison group design is a stringent test of the HFI program in that comparison families may receive all other usual services available within their community.  Thus, the HFI evaluation examined whether HFI services were associated with improved outcomes over and above all usual services.

Use of an independent evaluator to ensure unbiased data collection.

To increase the credibility of the program evalation findings, IDHS hired an independent evaluator, Northern Illinois University (NIU), to assess the effectiveness of the HFI programs in improving parent-child interactions and reducing risk for maltreatment among service recipients.  This standard of practice is yet another manner in which HFI has demonstrated its commitment to rigorous evaluation of its programs' outcomes.

"The HFI Evaluation examined whether HFI services were associated with improved outcomes over and above all usual services."

Recruitment Procedures

Participants in HFI's program evaluation included families that screened positive for risk for problems in parenting (as indicated by scores on the Family Stress Checklist) and/or were considered eligible for HFI's intensive home visitation services.  To recruit families into the program evaluation, HFI program staff described the evaluation project to eligible families, and parents who wished to volunteer were referred to NIU for enrollment in the data collection project.  Families for whom intervention services were available were assigned to the intervention group, whereas families for whom no services are available (because of the limited availability of the program) were assigned to the comparison group.  All families were informed that the project involved follow-up every six months for two years, although they also were informed that they could withdraw from the evaluation at any time without penalty.

Data Collection Procedures

As noted above, in-home interviews were conducted with participating families at five time points.  The initial visit (Time 1) took place within 3 months of the child's birth.  Follow-up visits occured 6, 12, 18, and 24 months after the birth date.

During each visit the following measures were completed (see Table 3 for a description of each measure):

  1. Nursing Child Assessment Teaching Scale (NCATS)
  2. Home Observation for Measurement of the Environment (HOME) Scale - Infant-Toddler version.
  3. Chid Abuse Potential (CAP) Inventory

For more detailed descriptions of the measure and their psychometric properties see the appendices.

Table 3: Description of measures

Measure Method Brief Description
Nursing Child Assessment Teaching Scale (NCATS) Direct observation Examines sensitivity, responsiveness, and other growth fostering behaviors during parent-child interactions.
Home Observation for Measurement of the Environment Scale (HOME) Direct observation and interview Measures the quality of stimulation, support and structure available to children in their home environments.
Child Abuse Potential (CAP) Inventory Self report questionnaire Designed to screen for child abuse risk and other problems in parenting.

Interviewers

In total, 148 interviewers were recruited and trained in the reliable administration of the study measures.

The demographic profile of the interviewers closely matched that of the HFI service population (see Figure 2).  Interviewers were racially and ethnically diverse and were obtained from all regions of the state that had IDHS - funded HFI programs.  Approximately half the interviewers were Caucasion (56.8%), 25% were African American, 14.2% were Hispanic and 4.2% were of another race/ethnicity.

Figure 2: Race/Ethnicity of Interviewers

Description of Figure 2: Race/Ethnicity of Interviewers

Interviewer's Race or Ethicity Percent
Caucasian 56.8%
African American 25.0%
Hispanic 14.2%
Other 4.2%

Legend: Interviewers ( n = 148 )

Nearly one out of five (18.9%) of the interviewers were bilingual (i.e., fluent in both English and Spanish).  Interviewers were 86.5% female.

All interviewers were required to complete four days of intensive training in order to ensure reliability in the data collection procedures.  Two days focused on the administration of the NCATS; one day on the administration of the HOME; one day covering the CAP Inventory and reviewing the procedure manual developed for this project.

All interviewers were required to achieve 85% interrater reliability on the observational measures (i.e., HOME and NCATS).  Interviewers who were not successful in achieving 85% agreement in their scorings of NCATS and HOME were given the opportunity for additional training and retesting.  Interviewers were invited to recertify every six months in order to correct for observer drift.

All interviewers were informed of their obligations as mandated reporters and were provided with written materials outlining procedures for reporting suspected child maltreatment (DCFS, 2000).  Issues of cultural sensitivity and personal safety were also reviewed in trainings.