Indicator #3 - Child Outcomes

Early Intervention
Illinois Annual Performance Report

Part C State Annual Performance Report for (FFY07)
(OMB NO: 1820-0578 / Expiration Date: 12/31/2009)

Child Outcomes


Part C State Performance Plan (SPP) for 2005-2010

Overview of the State Performance Plan Development:

The Illinois APR documents performance data on State targets for each Child and Family Connections (CFC) office and documents CFC and state progress or slippage toward measurable and rigorous targets. The Illinois Early Intervention (EI) Program makes the Illinois APR and State Performance Plan (SPP) available on its web site and through links from the other EI web sites (the Illinois Early Intervention Training Program; Provider Connections, the Early Intervention credentialing office; and the Early Childhood Intervention Clearinghouse). The APR and SPP documents are also available to the public at each of the 25 CFC offices.

The APR is part an ongoing process of performance measurement and strategic planning for the Illinois Early Intervention Program. For a number of years, Illinois has been reporting performance data to key stakeholders including the Illinois Interagency Council on Early Intervention (IICEI), the Child and Family Connections (CFC) offices, and the general public through various reporting mechanisms. Work groups of the IICEI have been formed to address specific indicators (Service Delay Work Group and Transition Work Group) and overall operation issues (Finance Work Group). These work groups have helped analyze data and suggest improvement strategies.

The EI Program will make the Illinois APR and SPP available on its web site and through links from the other EI web sites (the Illinois Early Intervention Training Program; Provider Connections, the Early Intervention credentialing office; and the Early Childhood Intervention Clearinghouse). The APR and SPP documents will also be available to the public at each of the 25 CFC offices.

On April 1, 2006, the Illinois Early Intervention program began utilizing the Early Childhood Outcomes (ECO) Center child outcomes summary form (COSF) statewide for all initial and renewal IFSPs as well as for exit conferences within 90 days of a child turning three. A few entries were made prior to that time in a pilot phase and are included in this analysis. The Illinois Child Outcomes Rating Scale and Summary form is attached. This form includes instructions for its use. Illinois also follows the ECO Center guidance that scores of six or seven on its seven-point scale be considered demonstrating development equivalent to same age peers.

Monitoring Priority: Early Intervention Services In Natural Environments

Indicator 3: Percent of infants and toddlers with IFSPs who demonstrate improved:

  1. Positive social-emotional skills (including social relationships);
  2. Acquisition and use of knowledge and skills (including early language/communication); and
  3. Use of appropriate behaviors to meet their needs.

(20 U.S.C. 1416(a)(3)(A) and 1442)

Measurement:

  1. Positive social-emotional skills (including social relationships):
    1. Percent of infants and toddlers who did not improve functioning = [(# of infants and toddlers who did not improve functioning) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    2. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers = [(# of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    3. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it = [(# of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    4. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers = [(# of infants and toddlers who improved functioning to reach a level comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    5. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers = [(# of infants and toddlers who maintained functioning at a level comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100. 
  2. If a + b + c + d + e does not sum to 100%, explain the difference.
  3. Acquisition and use of knowledge and skills (including early language/communication and early literacy):
    1. Percent of infants and toddlers who did not improve functioning = [(# of infants and toddlers who did not improve functioning) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    2. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers = [(# of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    3. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it = [(# of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    4. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers = [(# of infants and toddlers who improved functioning to reach a level comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    5. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers = [(# of infants and toddlers who maintained functioning at a level comparable to same-aged peers) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
  4. If a + b + c + d + e does not sum to 100%, explain the difference.
  5. Use of appropriate behaviors to meet their needs:
    1. Percent of infants and toddlers who did not improve functioning = [(# of infants and toddlers who did not improve functioning) divided by (# of infants and toddlers with IFSPs assessed)] times 100.
    2. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers = [(# of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers) divided by the (# of infants and toddlers with IFSPs assessed)] times 100.
    3. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it = [(# of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it) divided by the (# of infants and toddlers with IFSPs assessed)] times 100.
    4. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers = [(# of infants and toddlers who improved functioning to reach a level comparable to same-aged peers) divided by the (# of infants and toddlers with IFSPs assessed)] times 100.
    5. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers = [(# of infants and toddlers who maintained functioning at a level comparable to same-aged peers) divided by the (# of infants and toddlers with IFSPs assessed)] times 100.
  6. If a + b + c + d + e does not sum to 100%, explain the difference.

Overview of Issue/Description of System or Progress:

An infant or toddler's developmental status upon entry to the Part C Early Intervention Services System is determined via a comprehensive assessment using multiple sources of information, including one or more approved assessment instruments, a review of pertinent records, clinical observation, and parent interview. The assessment process for eligible children yields valuable information that is used in the development of child outcomes. Progress toward the outcomes identified on the Individualized Family Service Plan (IFSP) is measured annually, or more frequently if necessary, by the IFSP team through the use of approved assessment instruments, clinical observation and/or parent interview. Strategies, services and supports are created and/or modified as necessary to best meet the child and family's changing needs.

Measurement of progress toward the three child outcomes identified by OSEP has been incorporated into the current system of progress measurement. IFSP teams will continue to have available to them a variety of formal assessment instruments, including both curriculum based and norm-referenced instruments. In order to summarize the outcome data in a format consistent with OSEP's measurement criteria, IFSP teams will complete a Child Outcomes Summary Form (COSF) as outlined by the ECO Center. The Outcome Summary Form has been integrated into Illinois' current IFSP document, which is - for the most part - an electronic document and has been completed online.

In order to integrate the Outcome Summary Form into the current IFSP document, modifications to the Cornerstone data management system were completed. A workgroup made up of stakeholders representing service providers, service coordinators and families assisted in identifying needed system changes as well as policy and procedure changes related to assessment and IFSP development. The Illinois Interagency Council on Early Intervention (IICEI) discussed the process at several of its meetings and reviewed and approved final procedures prior to implementation. All data system changes were complete by March 2006. Testing was completed during March and statewide implementation went into affect on April 1, 2006.

The Early Intervention Training Program developed and implemented a training module that adequately addressed the new reporting requirements, new policies and procedures, methods for translating assessment information into progress measurement, achieving consensus and completing the new Outcome Summary Form. The training module was reviewed by the IICEI in January 2006 and was disseminated statewide in February 2006. Ongoing technical assistance is provided by Early Intervention Bureau staff as needed. Monitoring of appropriate assessment procedures and accurate data input will be conducted by the EI Monitoring Program.

Progress Data for FFY07/SFY 08:

A total of 3,397 records were found in Cornerstone with at least one matched entry-exit pair on at least one of the three child outcomes for children who were in care for at least six months. In the analysis that follows, the exact number for each measurement is slightly lower due to missing data or questions that could not be resolved. This is a significant increase from the 615 for the FFY06/SFY08 report and was mainly due to the passage of more time since the process first started on April 1, 2006. However, it also represents an improvement in administration of the Child Outcomes Summary Form (COSF) in the field.

For FFY06/SFY07, the proportion of children reported making progress (a score other than A) was: 94.3% for social relationships, 96.1% for acquisition of knowledge and skills and 95.4% for meeting needs. For FFY07/SFY08 more than 99.5% of children were reported to have made progress for all three measures. We believe this is a more appropriate representation of the caseload. There was a

great deal of misunderstanding about how to rate children initially, particularly as it relates to the

progress rating. Comparing the rating distribution between the two years, the differences are relatively small. Those children who rated A in the past appear to have just moved to B. There also was a slight reduction in children rating E. So, overall there was a small movement to the center. Considering there was a 5-1/2 fold increase in the number of useable matched pairs and the dramatic increase in the time in care of the sample, it is surprising there was not a more substantial change.

FFY 07/SFY 08 Total A B C D E
Positive Relationships
Count 3,376 15 642 501 840 1,378
Percent 100.00% 0.44% 19.02% 14.84% 24.88% 40.82%
Acquire Skills & Knowledge
Count 3,378 11 608 980 1,452 327
Percent 100.00% 0.33% 18.00% 29.01% 42.98% 9.68%
Meet Needs
Count 3,366 14 605 707 1,417 623
Percent 100.00% 0.42% 17.97% 21.00% 42.10% 18.51%

Social Relationships

Social Relationships FFY06 FFY07
Moved closer to same age peers 48.9% 48.5%
Stayed same relative to same age peers 36.0% 38.5%
Farther behind same age peers 15.1% 13.0%
Average Rating at Entry 5.02 5.13
Average Rating at Exit 5.81 5.79
Change 0.79 0.66

A significant difference in child outcomes scores was expected between the small number of cases that had been in care for a short period of time in the FFY06 group and the much larger number in care for a longer time in the FFY07 group. We had to assume the composition of the groups would be very different and the second group would have more time in care, which would mean more time to make a difference. However, the differences were small for the social relationships outcome. For this outcome the proportion of children who moved closer to same age peers was 0.4% lower. The average change in rating between entry and exit was down 0.13. This is the most unexpected difference. With a longer average time in care, a greater average improvement would have been expected. The difference may be a sign that as infants get older they exhibit more obvious behavioral problems. It also may be related to continued problem getting valid matched entry-exit pairs. We know the FFY07/SFY08 sample was more representative, but there still were problems. However, this may relate to the nature of the outcome itself. The average entry score is much higher than for the other two outcomes and care plans are less likely to directly work on social-emotional issues.


Outcome 1: Child has positive social relationships.

  Exit Ratings (Number of Children)
Exit Ratings (Number of Children) 1 2 3 4 5 6 7 Total % at Entry
1 5 6 14 8 1 2 4 40 1.2%
2 6 22 38 43 43 23 12 187 5.5%
3 1 12 72 78 153 88 68 472 13.9%
4 1 7 28 51 118 108 73 386 11.4%
5 3 5 26 42 190 261 203 730 21.5%
6 1 5 10 22 68 250 299 655 19.3%
7 0 2 10 12 63 117 716 920 27.1%
Total 17 59 198 256 636 849 1375 3390
% at Exit 0.5% 1.7% 5.8% 7.6% 18.8% 25.0% 40.6%

Moved closer to same age peers 1,643  48.5%

Stayed same relative to same age peers  1,306  38.5%

Farther behind same age peers 441  13.0%

Avg.
Scores
Entry 5.13
Exit 5.79
Change 0.66
Knowledge & Skills FFY06 FFY07
Moved closer to same age peers 66.0% 75.7%
Stayed same relative to same age peers 23.5% 17.6%
Farther behind same age peers 10.4% 6.7%
Average Rating at Entry 4.09 3.98
Average Rating at Exit 5.38 5.41
Change 1.29 1.43

Unlike the social relationships outcome, there was a notable distinction between what was seen in the small FFY06 sample that was comprised of just children in care for a short period and the larger FFY07 sample, which had been in care longer. The proportion of children who moved closer to same age peers was 75.7%, compared to 66.0% for the FFY06 group. The proportion of children who fell farther behind was just 6.7% for FFY07 matched pairs, compared to 10.4% for FFY06. The entry scores for those in the FFY07 group were lower than for the much smaller, less representative FFY06 group, 3.98 compared to 4.09. However, the average exit scores were almost the same. As a result, the amount of change noted, 1.43 points, was 0.14 points higher.


Outcome 2: Child acquires and uses knowledge and skills.

Exit Ratings (Number of Children)
Entry Ratings (Number of Children) 1 2 3 4 5 6 7 Total % at Entry
1 10 6 15 12 9 5 4 61 1.8%
2 7 35 64 75 69 39 31 320 9.4%
3 1 19 101 179 301 247 153 1,001 29.5%
4 1 3 30 78 254 214 170 750 22.1%
5 1 5 33 39 198 338 255 869 25.6%
6 2 4 7 42 101 128 284 8.4%
7 1 5 5 22 76 109 3.2%
Total 20 71 247 395 878 966 817 3,394
% at Exit 0.6% 2.1% 7.3% 11.7% 25.9% 28.5% 24.1%

Moved closer to same age peers 2,568  75.7%

Stayed same relative to same age peers  599  17.6%

Farther behind same age peers 227  6.7%

Avg.
Scores
Entry 3.98
Exit 5.41
Change 1.43
Meeting Needs FFY06 FFY07
Moved closer to same age peers 66.1% 70.6%
Stayed same relative to same age peers 22.8% 22.1%
Farther behind same age peers 11.1% 7.2%
Average Rating at Entry 4.43 4.34
Average Rating at Exit 5.62 5.60
Change 1.19 1.26

The differences noted between the FFY06 group and FFY07 were similar to those for the knowledge and skills outcome but the differences were smaller. The percentage of children who moved closer to same age peers was 70.6 for FFY07 versus 66.1%. The average entry scores were lower and the exit scores were almost identical. The net average change in rating was greater for FFY07 than for FFY06 but not by as much as was measured for the knowledge and skills outcome.


Outcome 3: Child takes appropriate action to meet needs.

Exit Ratings (Number of Children)
Entry Ratings (Number of Children) 1 2 3 4 5 6 7 Total % at Entry
1 7 8 13 17 13 2 5 65 1.9%
2 5 34 57 51 53 45 28 273 8.1%
3 4 17 84 118 187 189 125 724 21.4%
4 1 4 28 94 191 173 158 649 19.1%
5 0 0 21 40 197 379 322 959 28.3%
6 1 3 6 12 50 181 258 511 15.1%
7 0 2 2 3 12 34 153 206 6.1%
Total 18 68 211 335 703 1003 1049 3387
% at Exit 0.5% 2.0% 6.2% 9.9% 20.7% 29.6% 30.9%

Moved closer to same age peers 2,392  70.6%

Stayed same relative to same age peers  750  22.1%

Farther behind same age peers 245  7.2%

Avg.
Scores
Entry 4.34
Exit 5.60
Change 1.26

Discussion of Progress Data

There was more and better data available from FFY07/SFY08 than there was for FFY06/SFY07. In all, we found useable matched entry-exit pairs for 35.7% of cases in care more than six months, up from 18.9%. The percentage of instances where data had to be corrected was lower. A significant number of cases had to be ignored because of logic flaws that could not be corrected or indications that the child had reentered the system after exiting, and thus had not yet had a true exit for purposes of final outcomes assessment. Since the process did not start until April 1, 2006, FFY07/SFY08 data still does not reflect all children who left during the year because some entered before the process was initiated.

For a variety of reasons, it is not realistic to get a matched entry-exit outcome measurement pairing for 100% of children who exit with at least six months of service. The most obvious problems are cases that close when the program cannot contact the family. Two CFCs had matched pairs for over 76% of children and another CFC had pairs for almost 70%. Until we have more experience we will assume a 70% match rate as the goal.

Many areas had match rates between 50-58% but the state average was pulled down by five Chicago area agencies. New training activities on how to complete the COSF will focus on these agencies. However, we also need to focus on particular problems that prevent finding matched entry-exit pairs, including the following specific problems:

No Entry Record

This problem was greatly reduced in the past year but was still found and needs to be reinforced.

Writing Over Entry Records

A widespread misunderstanding in the field led to the assumption that a COSF record should be written over, instead of opening a new one for exit scores. This problem was identified and addressed during FFY07/SFY08. Therefore, it impacted FFY07/FFY08 performance. This should not be a problem going forward due to new edits in the system.

No Exit Record

The number of children with no exit records appears to have been reduced, but it cannot be eliminated. Retraining has emphasized the importance of getting both entry and exit data. Rules require an exit meeting within three months of a child's third birthday, but this does not help when the program loses contact with the family, the family chooses to leave the program, or they decline to participate in the exit/transition process. Therefore, this will be the major reason why we cannot achieve 100% "compliance" with child outcomes measurement.

Not completing Yes/No Responses

Although some missing yes/no responses can be completed, it is not possible to determine this status in other cases. Retraining emphasized the importance of putting those responses in. The Cornerstone data system now requires yes/no progress entries when a prior record exists, but that edit was not in place for the full year.

The program shares response rate data with service coordination agencies and utilizes its training program to help individual agencies improve compliance with child outcome reporting. In addition, the state's determination scorecard process includes a provision that assigns a mark against the two agencies with the lowest rate of useable child outcome reports at exit for the calendar year, if they also are less than 50% of the state average.

Since Illinois requires assessment of all children at both entry and exit, it should not have been a problem for the data to be representative of the state. However, due to the issues noted above, useable data was not representative. The proportion of cases closed for which we could identify useable matched entry-exit child COSF data varied from 5.1% to 76.6%, among CFCs.

In terms of the larger state regions, the greatest problems were in Chicago, where we could find matches for only 11.8% of cases, compared to 51.9% for downstate, 42.9% for the collar counties and 31.9% for suburban Cook County. Even within these groupings there were large differences. For instance, the match rate for south suburban Cook County was just 8.9% while it was 49.8% for the north suburban CFC. This result is that not only is the pool unrepresentative geographically, it is unrepresentative racially. All of the CFCs with low match rates have high African-American caseloads. The impact for Hispanics is less severe. They make up a large portion of the Chicago caseload, but they otherwise are concentrated in to the north and west of Chicago, where match rates were above the state average. Economic factors will be relatively representative. The low match rates for Chicago are countered by the high rates downstate. Downstate and Cook County have similar Medicaid eligibility rates.

Distribution of Child Outcome Matched Entry-Exit PairsIncluding Ratio of Compliance with Rules

CFC Eligible
Closures
State % UseableE/
E Pairs
State % Compliance
#1 ROCKFORD 290 3.05% 202 5.95% 69.7%
#2 LAKE COUNTY 447 4.70% 132 3.89% 29.5%
#3 FREEPORT 175 1.84% 42 1.24% 24.0%
#4 KANE-KENDALL 482 5.07% 137 4.03% 28.4%
#5 DUPAGE COUNTY 741 7.79% 390 11.48% 52.6%
#6 N. SUBURBS 729 7.66% 363 10.69% 49.8%
#7 W. SUBURBS 521 5.48% 154 4.53% 29.6%
#8 SW CHICAGO 391 4.11% 60 1.77% * 15.3%
#9 CENTRAL CHICAGO 558 5.87% 79 2.33% * 14.2%
#10 SE CHICAGO 434 4.56% 22 0.65% * 5.1%
#11 N. CHICAGO 983 10.34% 119 3.50% * 12.1%
#12 S. SUBURBS 517 5.44% 46 1.35% * 8.9%
#13 MACOMB 198 2.08% 151 4.45% 76.3%
#14 PEORIA 318 3.34% 113 3.33% 35.5%
#15 JOLIET 735 7.73% 407 11.98% 55.4%
#16 BLOOMINGTON 312 3.28% 239 7.04% 76.6%
#17 QUINCY 138 1.45% 31 0.91% 22.5%
#18 SPRINGFIELD 176 1.85% 46 1.35% 26.1%
#19 DECATUR 206 2.17% 105 3.09% 51.0%
#20 EFFINGHAM 231 2.43% 133 3.92% 57.6%
#21 METRO E. ST. LOUIS 323 3.40% 162 4.77% 50.2%
#22 CENTRALIA 191 2.01% 105 3.09% 55.0%
#23 NORRIS CITY 98 1.03% 57 1.68% 58.2%
#24 CARBONDALE 81 0.85% 34 1.00% 42.0%
#25 MCHENRY CO. 236 2.48% 68 2.00% 28.8%
State 9,511 3,397 35.7%
Chicago 2,366 24.88% 280 8.24% * 11.8%
Suburban Cook 1,767 18.58% 563 16.57% 31.9%
Collar Counties 2,641 27.77% 1,134 33.38% 42.9%
Downstate 2,737 28.78% 1,420 41.80% 51.9%

* Agencies in bold and shaded under 50% of state average compliance.

FFY 2005(2005-2006)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

FFY 2006(2006-2007)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

FFY 2007(2007-2008)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

FFY 2008(2008-2009)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

FFY 2009(2009-2010)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

FFY 2010(2010-2011)
Measurable and Rigorous Target: To be submitted with FFY2008 APR due February 2010

Improvement Activities/Timelines/Resources:

This is the initial reporting of paired entry and exit data. So, no progress or slippage can be reported. No targets will be established until 2/1/2010.

  • Continued training on the importance of completing the Child Outcomes Summary Form as a normal part of the IFSP and exit process.
  • Reassessment by the EI Bureau and the EI Monitoring agency of the uniformity of the administration of the summary form by the end of FFY 08/SFY 09.
  • Semi-annual evaluation to assure there are no patterns in the instances where assessments are not being completed at entry and at exit as required.
  • During FFY 07/SFY 08, the EI program will increase training that emphasizes the importance of completing child outcomes assessments at every IFSP meeting and the particular importance at the initial IFSP and at exit.
  • To improve uniformity of administration, having one of the two lowest percentage of compliant child outcomes entry-exit pairs will be made a negative factor in the CFC determination scorecard, if the percentage is below 50% of the state average effective with CY 2007 and each year thereafter.
  • During FFY 07/SFY 08, the EI program will emphasize the importance of correctly filling in the child outcome ratings at each IFSP, with particular emphasis on indicating progress was made.
  • During FFY 07/SFY 08 and FFY 08/SFY 09, the EI program will review the rates of compliance with rules regarding child outcomes measurement with CFCs, the IICEI and other interested parties and develop strategies to assure uniformity of administration.
  • During FFY 07/SFY 08 and FFY 08/SFY 09, the EI program will review the aggregate results from child outcome measurement and discuss what the results say about the program and make initial plans for activities to improve results.
  • During FFY 08/SFY 09 and FFY 09/SFY 10, the EI program will work with stakeholders, including the IICEI and CFCs to develop goals to improve child outcomes
  • During FFY 08/SFY 09 and FFY 09/SFY 10, the EI program will work with contractors and stakeholders to educate the public on the early results on child outcome measurement and why it is important.
  • During FFY 10/SFY 11, the EI program will implement specific goals to improve child outcomes.