SSIP - Phase III (Year Two) - April 1, 2018

Illinois continued our work with our pilot sites while submitting our Phase 3 Plan in April 2017. As stated in our prior Plans, these sites (Williamson County, Aurora, and East St. Louis) were selected to leverage resources and continue the work started during the Race to the Top Early Learning Challenge Grant. The majority of the work on the SSIP is now being completed by the evaluation team workgroups and the Leadership Team members in each of the pilot sites. The evaluation team continues to reference the Theory of Action and the activities listed in the Phase 2 plan (Plan). The evaluation team also continues to utilize the Theory of Change when dividing the work into meaningful pieces. The three evaluation team workgroups- Leadership Team (LT), Professional Development/Technical Assistance (PD/TA), and Performance Support (PS) continue to support the work. Membership representation for each of these teams is detailed on page 7 of this report in the section Stakeholder Involvement in SSIP Implementation. The LT group focuses on aspects of the Plan related to Leadership Team development and support and assisted with the development of a communication plan for our system stakeholders. The PD/TA group focuses on aspects of the Plan related to professional development. The PS group focuses on aspects of the Plan related to policy/procedure and implementation of practices.

The primary focus for the second implementation year has been supporting the leadership teams (LT) in our three pilot areas as they provide professional development on the child outcomes summary (COS) process for their local teams, refining the policy/procedure for our COS process, providing training and support for those in the field as they implement the COS process, reviewing ongoing professional development offerings utilizing our rubric to ensure that they are of high quality, and assessing fidelity to our desired COS process. Although we recognize that these changes to the COS process will not improve children's acquisition and use of knowledge and skills, our stakeholders felt that it was imperative to improve the quality of COS data. Better data will provide teams with better information for intervention planning and will allow us to better understand the impact of the family engagement practices we will be supporting in the coming year. Additionally, a Messaging workgroup revised the system brochure for families and created a brochure for primary referral sources. The evaluation plan has undergone modification and refinement as we collect data on the utility of the professional development, plan for assessing fidelity, and examine our progress on infrastructure improvements.

While much of the year was spent increasing the field's knowledge about the child outcomes summary process, we have begun thinking about other ways to demonstrate the important changes being made to our infrastructure as a result of the SSIP, how to assess fidelity to the desired child outcomes process (ensuring that outlined practices are being utilized), examining data to determine the need for adjustments/modifications to the COS process and professional development offerings, and how to address the professional development needs of the leadership teams as we begin to address our strategy on family engagement. In regard to our second improvement strategy, we have enhanced information about the DEC Recommended Practices in the Provider Handbook, have developed the curriculum for the initial Leadership Team training, have outlined the sections to be included in the family engagement resource package, and have crafted our initial evaluation questions including a draft plan for examining practice change. Although Illinois entered year two of Phase III without a balanced budget in the State, we continued to support the Early Intervention Services System with consistent, albeit delayed, payments. As outlined in previous submissions, Illinois continues to use multiple stakeholder groups to evaluate activities and refine the strategies intended to support the State-identified Measurable Result (SiMR).

SUMMARY OF PHASE 3, YEAR 2

The Theory of Action, Theory of Change, our Coherent Improvement Strategies, and our SiMR have not changed. They are all included below as a reference.

As we shared last time, the evaluation team felt that we needed a graphic that more explicitly showed how the pieces of the plan and the Theory of Action related to each other and how each activity was intended to support the vision of improving child outcomes. The Theory of Change shown below also created a way to break down the work listed in the Plan and assign tasks to new or existing workgroups. The Theory of Change also helps us think about what we need to be evaluating to determine the success of implemented activities.

Illinois Theory of Change for Part C Early Intervention

State-identified Measurable Result:

To increase the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e., Summary Statement 1) in the acquisition and use of knowledge and skills in our pilot areas (i.e., Aurora, East St. Louis, and Williamson) by .9 percentage points by 2018.

SiMR Targets by Federal Fiscal Year (FFY)

FFY 2014-FFY 2018 Targets

FFY 2014 2015 2016 2017 2018
Target 77.6% 77.6% 75.9% 77.3% 78.5%

SiMR Performance by FFY

Innovation Zone SS1 for 3B (FFY13/SFY14) # of Matched Entry-Exit Pairs SS1 for 3B (FFY14/SFY15) # of Matched Entry-Exit Pairs SS1 for 3B (FFY15/SFY16) # of Matched Entry-Exit Pairs SS1 for 3B (FFY16/SFY17) # of Matched Entry-Exit Pairs
1 2 3 4 5 6 7 8 9
East St. Louis 83.8% 13 73.7% 21 72.7% 13 83.3% 18
Aurora 75.0% 132 68.9% 201 71.4% 216 68.1% 148
Williamson Co. 84.2% 39 80.0% 6 100% 4 85.7% 16
Combined 77.6% 184 69.6% 228 72.0% 233 71.2% 182

The Coherent Improvement Strategies:

Our Coherent Improvement Strategies are:

  • Implement effective training for leadership teams and EI providers that focuses on infant/toddler development and the Child Outcomes Summary (COS) Process, and make related changes to state policy and guidance documents, so that early intervention teams implement the Child Outcomes Summary Process as desired.
  • Implement effective training for EI providers that focuses on evidence based, family capacity-building practices, and make related changes to the local support structure by creating leadership teams, providing technical assistance and revising state policy and guidance documents, so that early intervention teams utilize practices that encourage the active participation of families in the intervention process by embedding intervention strategies into family/caregiver routines.

Progress in Implementing the SSIP

The primary focus of practice change has been related to the child outcomes process. We have been working to help teams understand the desired process, to develop/compile resources to support this process, and to offer opportunities to reflect on ways to continually improve. We have conducted evaluations of professional development activities and have begun assessing teams' use of desired practices. This information will inform future PD activities being conducted by the local leadership teams. Early results indicate that participants are benefitting from the training and support being provided. Planning has also begun for how we will address our second improvement strategy. As with our first strategy, we will build leadership team capacity so that they feel capable of supporting the teams in their area. Also, our participation in the SSIP evaluation workshops has made it clear that we need to further develop our evaluation activities related to our infrastructure improvements.

Implementation progress is reflected in the timeline found in Appendix I. As part of our ongoing evaluation of implementation, the timeline continues to be updated to reflect progress and adjusted based on available resources and the needs expressed by local leadership team members. Each pilot site's activities continue to be driven by the Plan and the needs identified by the Local Leadership Teams.

As expected, the bulk of our ongoing activities focused on improving the child outcomes process through professional development and reflection opportunities with local early intervention teams. This work began at the end of last fiscal year and has continued throughout the majority of this

fiscal year as the local leadership team's work to reach their entire provider pool in the pilot areas. In addition, the delayed activity intended to build the local leadership teams' capacity to use data to inform professional development was completed by the end of SFY17. A detailed timeline with SSIP activities can be found in Appendix I.

The primary implementation activities for year 2 included: roll out of COS training to local early intervention teams, establishment of reflection opportunities for early interventionists, development of a communication plan, refinement of the PD rubric with continued quarterly reviews of select curriculum, training for local leadership teams on how to use data to inform professional development, development of a checklist to assess fidelity to our desired COS process, establishment of consents and procedures for examining COS process fidelity, and planning for local leadership team professional development on the selected DEC Recommended Practices that will be used to improve family engagement. In addition, the Messaging workgroup revised the brochure for primary referral sources and created a checklist that CFCs can use to ensure that locally-developed public awareness materials have consistent messages. Item A.7.1. (Develop a guidance document that can be used to evaluate existing materials about Child Outcomes and intervention) was eliminated due to the recognition that the system did not have existing materials available on Child Outcomes and intervention. The checklist that the Messaging workgroup has created will be used to guide the creation of any new materials. A list of activities that have been completed during the first two years of Phase 3 is provided below:

  • Training rubrics reviewed and new rubric with guidance document developed - COMPLETED
  • System developed to use rubric - COMPLETED
  • Professional development panel members selected (EITP staff, students, and faculty supports); reviewers trained prior to utilizing the rubric; prioritized curriculum deemed most relevant to SSIP implementation activities - COMPLETED
  • Development of COS resource guide - Tip Sheet for parents - COMPLETED
  • COS resource guide distributed and advertised widely including monitors- COMPLETED
  • Develop resource guides to support relevant training curriculum- COMPLETED
  • Advertise availability of resource guides linked to curriculum- COMPLTED
  • Include resource guides in supplements for appropriate trainings- COMPLETED
  • LTs in place with state supports (EI partners identified - EITP and Monitoring) - COMPLETED
  • LT guidelines created: Guidance document, Benchmarks of Quality and Action Plan - COMPLETED
  • Training calendar for LT members - COMPLETED AND ONGOING
  • General training about LT functioning and action planning - COMPLETED
  • COS train the trainer event for LT members - COMPLETED
  • Viable LT COS training calendar for local EI teams - COMPLETED AND ONGOING
  • Establish meetings for early interventionist to reflect on COS implementation- COMPLETED AND ONGOING
  • Support peer to peer observations, critique, and support- COMPLETED AND ONGOING
  • Messaging rubric (with list of critical messages) - COMPLETED
  • COS informational materials for parents for intake and IFSP - COMPLETED
  • COS materials for stakeholders and providers - COMPLETED
  • Creation and implementation of policy/procedure to define the "up to 20 hours" of EITP training - IN PROGRESS
  • Revision of payee/provider agreement - DELAYED
  • Revision of existing monitoring tools and development of observation tools/checklists for COS - COMPLETED
  • Team meets to identify topics for the focus of training on family engagement EBPs/DEC Recommended Practices - IN PROGRESS
  • Establish process for coordinating ongoing communication- COMPLETED AND ONGOING

The outputs for year 2 include: COS training for local early intervention teams, focused conversations (reflection opportunities) for early intervention teams that have received training, COS resource packages for use by local leadership teams, a checklist to examine COS fidelity developed in conjunction with our ECTA and DaSY TA partners (see checklist in Appendix II), consents and written process for COS fidelity observations, and an SSIP communication plan. Additional information about outputs and their relationship to implementation activities can be found in the section titled "Data on Implementation and Outcomes".

Stakeholder Involvement in SSIP Implementation

The large SSIP stakeholder group continues to receive quarterly written updates summarizing implementation. Each summary requests contact if stakeholders have questions and the summary contains a reminder that those receiving the summary should share widely with those whose interests they represent. Other stakeholders have been informed via information provided in the EI Partners' quarterly newsletters (EI Training, EI Clearinghouse, and Provider Connections) and postings on their websites. The SSIP work has also been discussed at each monthly CFC Managers Meeting, each quarterly Illinois Interagency Council on Early Intervention (IICEI) meeting, each monthly EI Partners' meeting, and through presentations at various professional conferences. In addition, the Early Intervention Training Program has a resource page dedicated to the State Systemic Improvement Plan (https://blogs.illinois.edu/view/6039/378910) and has created a specific resource page for materials that the local leadership teams are using.

In order to optimize stakeholder involvement and engagement, we continue to utilize the expertise of a variety of groups:

  1. SSIP Core Team - consists of Illinois Department of Human Services (IDHS) Early Intervention staff and the EI Ombudsman
  2. SSIP Stakeholder group - consists of the state interagency coordinating council members, direct service providers, contracted Child & Family Connections (CFCs which are regional entities that serve as points of entry for EI services) staff, parents, professional provider associations, the Part B/619 coordinator, IDHS planning/evaluation members, parent training and information center staff, and contracted EI partners for training, credentialing, monitoring and clearinghouse.
  3. SSIP Evaluation Team - consisting of Bureau staff, Training staff, Monitoring staff, CFC personnel, EI Provider, and IDC TA representative
  4. Leadership Team workgroup - consisting of Bureau staff, CFC managers, EI Training staff, EI Monitoring personnel, a parent liaison, and an external professional development representative
  5. Professional Development/Technical Assistance workgroup- consisting of EI Ombudsman, EI Training, EI Provider and IDEA Data Center representative
  6. Performance Support workgroup - consisting of EI Bureau, Local Leadership Team members (service coordinators and providers), ECTA representative, EI Training, EI Ombudsman, and EI Monitoring
  7. Local Leadership teams (in each of the three pilot sites) - consisting of CFC staff, EI Providers, EI Monitoring, EI Training and parents when possible.
  8. Messaging workgroup - ad hoc team consisting of EI Bureau staff, EI Ombudsman, Local Leadership Team members, EI Clearinghouse, and Illinois Race to The Top representatives

Cumulatively, these groups have been involved in all implementation activities and continue to inform the decision-making process and implementation timelines. The large stakeholder group primarily receives updates and they have requested an annual in person meeting to monitor progress. The evaluation team continues to meet, reviews evaluation data, determines need for adjustments, and plans for next steps/activities. The leadership team workgroup has developed a communication plan for system stakeholders and continues to report on, and support, the work of the local leadership teams. The PD/TA workgroup continues to review curricula and is preparing the plan for local leadership team training around family engagement. The performance support workgroup has been involved in the development of the fidelity checklist, in proposing revisions to the data system so that it supports the desired COS process, and in crafting the process for COS fidelity observations. The local leadership teams drive local efforts around training and support, monitor their performance in regard to the Benchmarks of Quality, and provide input around the evaluation of their activities. The messaging workgroup has been addressing the consistency of system messages and revising public awareness materials to ensure they reflect desired messages about early intervention. While we have been working to improve communication across the groups so that everyone can stay informed about, and provide input to, other groups' activities, this division of labor has allowed substantial stakeholder input without overburdening individuals. It has also ensured that information is constantly flowing between those responsible for implementation of the Plan and those working directly with children and families.

The evaluation team has continued to meet (July 16th and December 4th) to plan, complete, and monitor the activities included in the Plan. In addition to the work of the evaluation team, we have also solicited more intensive assistance from the IDEA Data Center (IDC) to further refine our evaluation plan and have had two teams participating in the ECTA/DaSy series on SSIP evaluation. The initial planning activities for the COS Improvement Strategy have been completed resulting in targeted evaluation questions, a plan for data collection, and data that will be useful for informing decision-making as we continue the implementation process. Our evaluation plan has been revised and refined based on an increased understanding of necessary plan components as a result of our participation in the TA centers' workshops on SSIP evaluation. In addition to the original outcomes and revised evaluation questions, we have included some new information about infrastructure evaluation in order to better align our evaluation with our theory of action and to demonstrate the progress being made while we wait for the changes we have made to impact our long-term outcome- a change in our SiMR.

We continue to evaluate the three general types of activities being conducted during the implementation phase- leadership team planning, leadership team capacity building, and leadership team provided professional development. As during year 1, leadership team planning has minimal evaluation as these events are used to keep the process moving forward. Leadership team capacity building evaluations continue to examine the extent to which the intended outcomes of the activity are achieved. Leadership team provided professional development opportunities are conducted by the leadership teams for local EI teams and the evaluations of these events include determining whether or not the intended outcomes were achieved by event participants. These evaluations have been used to understand our progress and answer our evaluation questions. In addition to evaluating our professional development activities, we have also begun examining implementation fidelity for the outlined COS process through the use of observations and checklists.

Additional, detailed descriptions of stakeholder involvement in reviewing SSIP evaluation data and making recommendations and decisions based on those data, are provided in the narrative under Data on Implementation and Outcomes.

Data on Implementation and Outcomes

The Plan below shows the outcomes we want to examine as well as evaluation questions, performance indicators, measurement plans, timelines, and related analyses.

Evaluation Plan for Coherent Improvement Strategy 1: 

Implement effective training for leadership teams and EI providers that focuses on infant/toddler development and the Child Outcomes Summary (COS) Process, and make related changes to state policy and guidance documents, so that early intervention teams implement the Child Outcomes Summary Process as desired.

Evaluation Plan for Infrastructure Improvements:

Outcome Type Outcome Description Evaluation Question(s) How Will We Know the Outcome Was Achieved? (performance indicator) Measurement/ Data Collection Method Timeline/ Measurement Intervals Analysis Description Data/Results
Short term Leadership teams will have the skills and knowledge to support local Early Intervention providers in implementing the desired Child Outcomes process using training and reflection activities Did the individuals on the leadership teams acquire the skills and knowledge necessary to effectively provide training and reflection activities to support local early intervention teams in their implementation of the Child Outcomes process? At least 75% of attendees report increase in the skills and knowledge acquired

Pre- survey of leadership team members about their knowledge of child outcomes process and ability to support others in implementation of the process.

Post-survey focused on what they now know about child outcomes

Pre-survey completed Oct-Nov 2016

Post-survey completed between 12/16 and 6/17

 Calculate the percentage of leadership team members who respond that they have the knowledge/skills on the pre-survey and the percentage who report acquiring knowledge/ understanding on post training surveys

85% of respondents reported that they had the knowledge and skills required on the pre-survey

Despite the high level of competency reported on the pre-survey, 88% of the LT members agreed or strongly agreed that they had increased their knowledge and understanding of critical COS components.
Short term Early Intervention providers have acquired the skills and knowledge necessary to understand how the Child Outcomes process is to be implemented Do individuals have the skills and knowledge necessary to understand how the Child Outcomes process is to be implemented? At least 75% of attendees report increase in the skills and knowledge acquired Post training survey of early intervention team members about the knowledge they have acquired February 2017- February 2018 Calculate the percentage of attendees who agree or strongly agree that they have increased knowledge and understanding 79-90% of participants agreed or strongly agreed that they had increased knowledge/
Intermediate Leadership teams will utilize reflection and ongoing PD activities to support local Early Intervention teams in implementing the Child Outcomes Process Are local EI teams benefitting from the support being provided by the leadership teams? At least 75% of the attendees report gaining additional information or knowledge about implementing the COS process Post reflection survey on information and knowledge acquired through participation in COS reflection activities July 2017- February 2018 Percent of attendees who report that they have acquired knowledge/ information 82-95% of the attendees agreed or strongly agreed that they received additional information or knowledge about COS process
Intermediate Early intervention teams utilize the recommended process for determining Child Outcomes ratings Are providers implementing the Child Outcomes process as outlined in training and procedures? 75 % of the items on the child outcomes fidelity checklist are being completed Fidelity assessed using recorded discussions & direct observations with a checklist developed based on IL policy/ procedure (25% of SCs in pilot areas to gain initial look at fidelity and to refine checklist and procedures) February- March 2018 Recordings and direct observations will be used to score the fidelity checklist to determine if process is being implemented as desired and to inform areas needing additional LT support; LTs can then use COS resource package to offer additional support Based on initial checklists, 83-94% of the items are being completed
Long term Children experience greater than expected growth in their acquisition and use of knowledge and skills in the pilot areas Has the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e. SS1) in the acquisition and use of knowledge and skills in our pilot areas increased? The percentage of children in summary statement 1 will increase by 0.9% over time in the three pilot areas Indicator 3.b collected in Cornerstone will show a .9 percentage point increase by FFY 2018 Data collected for FFY 2018 reporting Although change is not yet expected due to recent initiation of training and support, information about indicator 3B for summary statement 1 is already being collected for the three pilot areas Results included each year with SSIP report

Evaluation Plan for Infrastructure Improvements:

Outcome Type Outcome Evaluation Question(s) How will we know? (Performance Indicator) Measurement/Data Collection Method Timeline/Measurement Intervals Analysis Description/Data Management Data/Results
Short term CFCs will produce high quality child outcomes data Do CFCs have the tools and information necessary to collect high quality child outcomes data? At least 2 of the Quality Indicators in the Data Collection and Transmission from the State Child Outcomes Measurement System will have at least 50% of their elements fully implemented by February 28, 2018. State child outcomes measurement system data collection and transmission subcomponent

Baseline 2015

Post 3/2018

 Involve stakeholders in completion of the self-assessment. Compare self-assessment ratings from baseline and post and compute the percent of QI ratings for data collection and transmission that increased Quality Indicators 2 and 3 in the Data Collection and Transmission section now have at least 50% of their elements fully implemented.

The bulk of our evaluation measures examine the impact of training and technical assistance on practitioners' implementation of desired practices for the COS process and family engagement. We now also realize that the other two areas of our theory of action will require evaluation as well and that some additional intermediate outcomes will need more consideration by our evaluation team, e.g. determining the accuracy of COS ratings and examining how COS information impacts intervention planning and implementation. We have included a plan for looking at the pieces of the COS measurement system that we have been able to put in place. Evaluating the increased knowledge of family-centered practices by internal and external stakeholders still needs to be addressed more fully. While internal knowledge will be examined through the evaluation of PD activities targeting our second improvement strategy, the primary way we are currently addressing increased external knowledge is through Messaging workgroup activities. The Messaging workgroup was charged with creating items to address the needs of the Local Leadership Teams (and all CFCs in general) to support the knowledge of EI to the local communities, the referral sources and families. This included updates to existing materials and creation of a new Referral Brochure resource focused on how to speak to community partners on what to expect when referring to EI.

Data sources, available baseline data, data collection procedures and timelines are all outlined in the chart above. In addition, sampling procedures (when applicable) and data comparisons are included. In terms of data management, the data related to professional development activities is collected and stored by the Early Intervention Training Program. Results are reviewed quarterly and shared with Local Leadership Teams so that they can use this information to inform next steps in their professional development efforts. COS fidelity checklist information is collected by members of the local leadership teams and securely stored by the Early Intervention Training Program. COS data is collected by early intervention teams and entered into Cornerstone by service coordinators. Early analyses of the professional development information have allowed us to determine whether or not participants feel that the trainings and reflection opportunities are achieving the desired outcomes of increasing knowledge and understanding. So far, the feedback has been positive, so no adjustments have yet been made to the curriculum. Training participants report increased knowledge and understanding of the various COS components and indicate plans to use what they have learned in future COS discussions. The fidelity information has allowed us to determine whether or not the professional development and support provided has resulted in the process being conducted with fidelity. Initial results indicate a high level of fidelity with the outlined COS process. As with the other evaluation data, the information collected from the checklists will allow the local leadership teams to determine next steps in their professional development efforts. For example, many service coordinators are not using the "COS at IFSP" document created for families. We will need to determine why this is occurring, e.g. Is this a resource that they are unaware of? Does the document need modifying? Also, some of the items related to introducing the COS process to families were not as consistently implemented. We will need to determine what additional support can be provided to ensure better consistency. Although we do not yet expect our improvement strategies to be impacting our SiMR in the way we anticipated, we have been collecting and reporting this information. It has allowed us to gauge natural fluctuations in performance and monitor data collection practices.

Evaluation data has been used to understand the impact of the professional development being provided by the local leadership teams (see examples in Appendix III). This includes evaluation of training, reflection activities, and fidelity to the desired COS process. With our broadening understanding of SSIP implementation and evaluation, we have realized that we need to collect more information about the changes implemented related to our infrastructure. While those changes have largely proceeded as planned, we did not initially realize the importance of monitoring or measuring their implementation. We have used the Data Collection and Transmission section of the Child Outcomes Measurement System to examine some of the changes made to our child outcomes measurement system. In 2015 (Time 1/T1), none of the Quality Indicators (QI) in this section had half of their elements fully implemented. When reviewed again in 2018 (Time 2/T2), 2 of the 3 quality indicators have more than half of the elements fully implemented. Five of the 9 elements in the second quality indicator and 6 of the 11 elements in the third quality indicator are fully implemented. Only 3 of the 14 elements in the first quality indicator are fully implemented. We plan to utilize members of our Outcomes Workgroup to complete a review of all the sections in order to prioritize and direct additional infrastructure improvements.

Data Collection and Transmission (DATA COL)

DC 1 = Data collection procedures are carried out efficiently and effectively.

QI Rating (Time 1) = 4

# of Elements = 14

# of Elements in the QI with the rating (T1) # 1 = 1

# of Elements in the QI with the rating (T1) # 2 = 4

# of Elements in the QI with the rating (T1) # 3 = 6

# of Elements in the QI with the rating (T1) # 4 = 2

# of Elements in the QI with the rating (T1) Not Applicable = 1

QI Rating (Time 2)

# of Elements in the QI with the rating (T2) # 1 = 0

# of Elements in the QI with the rating (T2) # 2 = 2

# of Elements in the QI with the rating (T2) # 3 = 8

# of Elements in the QI with the rating (T2) # 4 = 3

# of Elements in the QI with the rating (T2) Not Applicable = 1

DC 2 = Providers, supervisors, and others involved in data collection have the required knowledge, skills, and commitment.

QI Rating (Time 1) = 4

# of Elements = 9

# of Elements in the QI with the rating (T1) # 1 = 0

# of Elements in the QI with the rating (T1) # 2 = 4

# of Elements in the QI with the rating (T1) # 3 = 5

# of Elements in the QI with the rating (T1) # 4 = 0

# of Elements in the QI with the rating (T1) Not Applicable = 0

QI Rating (Time 2) = 4

# of Elements in the QI with the rating (T2) # 1 = 0

# of Elements in the QI with the rating (T2) # 2 = 3

# of Elements in the QI with the rating (T2) # 3 = 1

# of Elements in the QI with the rating (T2) # 4 = 5

# of Elements in the QI with the rating (T2) Not Applicable = 0

DC 3 = State's method for entering, transmitting, and storing data is effective and efficient.

QI Rating (Time 1) = 5

# of Elements = 11

# of Elements in the QI with the rating (T1) # 1 = 0

# of Elements in the QI with the rating (T1) # 2 = 3

# of Elements in the QI with the rating (T1) # 3 = 3

# of Elements in the QI with the rating (T1) # 4 = 5

# of Elements in the QI with the rating (T1) Not Applicable = 0

QI Rating (Time 2) = 5

# of Elements in the QI with the rating (T2) # 1 = 0

# of Elements in the QI with the rating (T2) # 2 = 1

# of Elements in the QI with the rating (T2) # 3 = 4

# of Elements in the QI with the rating (T2) # 4 = 6

# of Elements in the QI with the rating (T2) Not Applicable = 0

Given the length of time that we have been measuring child outcomes and the timing of SSIP implementation, baseline data for our participants is not available for many of our measures. We do, however, have data from our annual needs assessment which surveyed service coordinators and providers about their use of desired COS practices. This data indicated that many practices were not being utilized. Therefore, we know that participants come to the offered PD opportunities with a variety of experiences. Our intent with our offerings has been to increase participants' knowledge of desired practices and then assess the impact of this knowledge change on practice. We will then determine if these changes in practice have impacted our interactions with children and families in a way that has improved children's outcomes.

Data, in its broadest sense, have informed all changes to implementation and improvement strategies. For example, the initial COS training for the local leadership teams indicated lingering questions about COS documentation. This lead to enhanced information about documentation being included in the procedures and additional information being added to an appendix in the CFC Procedure Manual. Feedback from local leadership teams after reviewing their teams' status on the Benchmarks of Quality (Appendix IV) indicated a need for additional time to complete training and come together as a team. This resulted in the delay of training to early intervention teams. Given the success of these trainings, it appears that the delay was justified. Local leadership teams also provided feedback that the rollout to early intervention teams needed to be tailored to their local areas (given their differences in provider pool size), so each team has taken a slightly different approach to obtaining provider participation.

If data on the PD opportunities had indicated that the activities were not achieving their desired outcomes, they would have been modified. In addition, the PD activity evaluation data and fidelity information will be used to guide future local leadership team offerings. These offerings will be supported by the activities outlined in our resource packages (the COS resource package is available on the LT resource page: https://blogs.illinois.edu/view/7582/592341. On another note, leadership teams noted decreased interest in, and attendance at, their focused conversations (opportunities to reflect on COS practice) and began planning for how they could make these offerings better to support continued growth around the child outcomes process.

One rather significant change to an outcome came about due to what we were learning about high quality professional development and feedback on the time commitment local leadership team members were being asked to make. As stakeholders (evaluation team and local leadership teams) reflected on the elements of high quality PD, the ongoing trainings currently offered by EITP, and our credentialing process, stakeholders determined that a separate training on child development was not necessarily going to improve child outcomes ratings. It was felt that a better solution would be to include necessary information within the context of the ongoing child outcomes trainings. The local leadership teams have embedded information about child development (particularly information about foundational, immediate foundational, and age-expected skills) into both their COS trainings and focused conversations. In addition, the teams have compiled resources for their service coordinators to use during discussions to make sure that the team has the information needed to age-anchor reported skills. This change also resulted in the child development piece of the COS resource packages being expanded. Though we still need to determine the accuracy of the ratings, we feel these modifications will enhance teams' understanding of child development without requiring additional training time for the leadership team members.

Beyond the examples already listed, most of our data have supported that SSIP implementation should proceed as planned. Data collected demonstrates improvements in the knowledge and skills of leadership team members, improvements in the knowledge of early intervention team members, and improved use of the desired components of the COS process. In addition to this evaluation data, we have also received anecdotal feedback from leadership team members that families are more engaged in the COS discussions, teams realize their earlier discussions were not appropriately age-anchored, and IFSP meetings are more streamlined with outcomes more closely tied to identified needs. As we develop additional intermediate outcomes related to the accuracy of COS data and the use of this data to plan intervention, we will gain a better understanding whether the use of desired practices results in improved data quality and improved intervention planning.

Each of the quarterly SSIP summaries sent to stakeholders included evaluation data with an accompanying description. The large stakeholder group was also convened last February to provide feedback on the evaluation plan. Their input on whether or not this plan met their informational needs and would be able to answer the questions they felt were most pertinent for decision making was solicited. We plan to have a similar convening of this group in the next few months to share what we have learned about evaluation, to share available data, and to determine the need for revisions to our process or measures. As mentioned previously, the evaluation team (consisting of stakeholders) continues to meet and engage in decision-making about the ongoing evaluation of activities.

Data Quality Issues

We have not yet identified any concerns with the quality or quantity of the data being collected for evaluation. We have established systematic processes for collection. We have also begun involving local leadership team members in the development of evaluation measures related to their activities. For example, local leadership team number four has begun some initial surveying of their providers around the use of various family engagement strategies so that they have a better understanding of where providers currently stand in terms of their understanding and use of these practices.

We also now realize that we have been missing some evaluation measures, particularly those that would more extensively examine our infrastructure improvement strategies such as the Child Outcomes Measurement System. Although we knew we would use our Benchmarks of Quality as our primary measure, that tool does not reflect the breadth of changes being made to the statewide infrastructure. We have begun to more fully develop these measures and hope to capitalize on some of the work we did during our original data analysis and review of system strengths and weaknesses. An example of what we can examine is found in Appendix V. Utilizing the ECTA/DaSY system framework, we are able to demonstrate changes made in regard to accountability within the EI system. Although consistent review of these components demonstrate that there is still room for improvement, it provides a way to demonstrate the progress that we have made. We will be working with our stakeholders in the coming year to determine which other areas need examination and targets for improvement. We also still have concerns about the overall quality of our child outcomes data and its ability to accurately evaluate our progress toward the SiMR. Prior to the FFY16/SFY 17 APR submission, additional outreach was conducted with all CFCs to show them the issues arising around data collection. Despite the data system edits implemented in FFY15/SFY16 to prevent incorrect COS entries, errors continue to be identified. These errors have identified the need for additional technical assistance within the CFCs and additional improvements to the data system. Additional guidance will be provided in the CFC Procedure Manual as well.

The lack of a well-developed plan for evaluating infrastructure improvements limits our ability to fully demonstrate the changes being made. The continued data entry issues are impacting our ability to utilize the full amount of COS data being entered and to understand our progress toward achieving the SiMR. Although more families are participating in the process and more data are being entered, the usable data do not reflect this increase. There are a variety of reasons that data had to be excluded but the two primary issues relate to an invalid entry (0) or data entered outside of the established timelines, e.g. exit entries that are completed more than 120 days prior to system exit. We have not yet determined if the children for whom we have matched entries differ in important ways from those who had to be excluded, so our understanding of child outcomes may not reflect the breadth of what is occurring in the field.

The state has plans, with the hiring of a qualified data manager, to work very closely with the CFCs, especially those in the pilot sites, to examine data more frequently and provide analyses to identify issues and trends in a more timely fashion. We feel that this improved use of the available data will lead to improved practice. More complete and accurate data will allow teams to better understand the improvements made by the child. Enhanced data use will also help the local leadership teams identify specific data or system needs to be addressed prior to statewide implementation.

Enhancing the evaluation plan for our infrastructure improvements will allow us to better understand what has been accomplished and what still needs to be done. It will also potentially allow us to determine whether or not the changes have impacted the system in the anticipated way or if modifications need to be made.

Progress Toward Achieving Intended Improvements

The primary infrastructure changes that support our SSIP, and in turn, our SiMR are related to enhancements to our child outcomes measurement system and the local leadership teams. Although the specific accomplishments have been highlighted earlier, it is important to note that we feel the changes to the Child Outcomes Measurement System will assist with sustainability and will be an important support when we begin scale-up. While these changes are a positive first step, we realize that more of the components of the child outcomes measurement system will need to be implemented in order to be assured that we are following a high-quality process that results in high quality data. In addition, we know we need to work with our stakeholders in the coming year to develop a plan for understanding how high-quality data may impact intervention planning in a way that will improve children's acquisition and use of knowledge and skills.

The local leadership teams are another vital part of our infrastructure change. They allow us to be responsive to local needs and provide local support for improved practices which is critical for a high quality professional development system. As noted earlier, these teams have been conducting professional development activities, participating in evaluation activities, and planning for future activities in their local area. The work of these teams has been guided by the Benchmarks of Quality established in our first year of implementation. All teams have demonstrated significant progress towards achieving the benchmarks necessary for a well-functioning leadership team.

As reported in our quarterly updates, each of the three local leadership teams showed substantial improvement in their Benchmarks of Quality ratings. Teams completed their initial rating of their status in November 2016 and a subsequent rating in November 2017. Each item on the Benchmark received a rating. The chart below shows the number of items on the Benchmarks that showed progress from 2016 to 2017.

The Benchmarks of Quality (see Appendix IV) have guided the local leadership teams' work (and were included in last year's appendix).

Implementing the items on the checklist has helped the teams serve as a support for practice change in their local area. The areas that teams were asked to consider were:

  • Leadership Team (creation, sustainability, objectives, membership, etc.)
  • Action Plan (development, implementation, etc)
  • Buy-In (communication, collaboration, feedback, etc.)
  • Family Engagement (representation, communication, etc.)
  • Communication (updating stakeholders, sharing data, routine schedule, etc.)
  • Professional Development and Ongoing Technical Assistance (consistent attendance at offerings, self-assessment, mentoring, etc.)
  • Data-Based Decision Making (data training, data entry, measuring improvement, etc.)

The charts below show the performance of the individual pilots for each of the identified areas. The blue information shows the teams' initial ratings when their teams were first convened. The orange bar shows where the teams were a year later. The progress over time is demonstrated side-by-side for each of the elements.

It is evident from these charts, that although the teams started in different places, they have all made progress in creating a valuable infrastructure element over the past year. This progress on the Benchmarks demonstrates that our local leadership teams are implementing the pieces necessary to be a valuable support for practice change. Further evidence of progress is seen in our other evaluation information which shows that the teams' support activities are increasing the COS knowledge of early intervention teams. Our initial use of the fidelity checklist has provided a first look at whether increased knowledge and understanding is leading to a COS process conducted with fidelity. The initial results indicate that the process is being implemented with a high level of fidelity. Although there will need to be some adjustments to the checklist, some discussion of missed items, and a plan for examining more teams' practices, we are quite pleased with the initial findings. These pieces of information are guiding our decisions about next steps for professional development and potential modifications.

Our results indicate that we have achieved our short-term outcomes related to our first improvement strategy. We are in the initial stages of evaluating our intermediate outcome for this strategy and will learn more about practice fidelity in the coming months. The first look at this outcome, though, has been positive and informative with many teams demonstrating a high level of fidelity to the proposed process. We feel that the training and support provided have been necessary steps for moving our COS process forward and our initial look is very promising. Ensuring that we are engaging in a high quality COS process gives us the best chance for understanding the impact of our family engagement practices' on the SiMR.

We are not yet seeing measurable improvement in the SiMR in relation to our targets (see charts on targets and performance at beginning of report), but we are not surprised. In fact, as implementation has proceeded, we realize that we may have been too ambitious in our prediction of how much impact we could have on the SiMR during the SSIP timeframe. Although we suspected that a focus on the COS would alter our performance data, the decrease in the summary statement has been even greater than we anticipated. We had also hoped that more focus on the COS would improve the number of usable matched entry-exit pairs in each of the pilot areas. While we expect small fluctuations in the number of children served in each area from year to year, the variability of matched pairs from year to year does not appear to be easily explained by this fluctuation. We will need to discuss these variations in more depth with the local stakeholders. We know that there are many moving pieces to achieving our SiMR and we have only begun to address the initial piece- producing high quality data. While many team members were trained during this fiscal year, the knowledge they acquired likely only impacted a portion of the COS discussions they conducted. With each step that gets implemented, however, we move closer to our goal. Although we needed to devote the time we have taken to improve the COS process and improve our data quality, we expect the real changes in the SiMR to occur after we have fully implemented our second improvement strategy.

PLANS FOR NEXT YEAR

As we have altered the timeline for some of the activities, at this time we do not plan on adding new activities, but rather plan on moving further with the implementation of already proposed activities. The primary activities in the coming year will be related to our family engagement strategy. Specifically, these activities are:

  • Training for leadership team members regarding family capacity building through the use of selected DEC Recommended Practices (RPs)- 4th quarter SFY18 and 1st quarter SFY19
  • Resource package of activities that can be used by local leadership teams to support family engagement/capacity building professional development activities- 4th quarter SFY18
  • Training and support for early intervention teams on family capacity building RPs- beginning 1st quarter SFY19
  • Continued professional development on the COS- 3rd quarter SFY18 throughout SFY19
  • Additional training on using data to inform professional development-beginning 4th quarter SFY 18
  • Continue creation and implementation of policy/procedure to define requirements for "up to 20 hours" of EITP training- ongoing during SFY19
  • Revision of payee/provider agreement and supporting documents to include language about new training requirements-beginning 1st quarter SFY19
  • Develop/modify materials related to evidence-based intervention practices (family capacity-building, family engagement, family decision-making, & family centered practices) utilizing recommendations for tailoring information to specific audiences (e.g. families/caregivers, providers, CFC staff)- ongoing throughout SFY19
  • Continued refinement of our evaluation plan- beginning 4th quarter SFY18

EVALUATION PLAN FOR SECOND COHERENT IMPROVEMENT STRATEGY

Implement effective training for EI providers that focuses on evidence based, family capacity-building practices, and make related changes to the local support structure by creating leadership teams, providing technical assistance and revising state policy and guidance documents, so that early intervention teams utilize practices that encourage the active participation of families in the intervention process by embedding intervention strategies into family/caregiver routines and activities.

Outcome Type Outcome Description Evaluation Question(s) How Will We Know the Outcome Was Achieved? (performance indicator) Measurement/Data Collection Method Timeline/ Measurement Intervals Analysis Description Data/Results
Short term Leadership teams will have the skills and knowledge to support local early intervention providers in implementing the selected DEC Recommended Practices (RPs) Did leadership team members acquire skills and knowledge to support local early intervention teams in their implementation of selected RPs? At least 75% of attendees report increase in the skills and knowledge acquired

Pre-survey of leadership team members about their knowledge of and ability to support use of RPs

Post-survey focused on what they now know about RPs

Pre-survey completed April 2018

Post-survey completed May 2018

 Calculate percentage of LLT members who report they have the knowledge/skills on the pre-survey and the percentage who report acquiring knowledge/ understanding on post training surveys
Short term Early intervention providers have acquired the knowledge necessary to implement selected RPs Do individuals have the knowledge necessary to implement selected RPs? At least 75% of attendees report increase in the skills and knowledge acquired

Administer pre-survey (RP checklists) about RP use RPs prior to training

Post training survey of early intervention team members about the knowledge they have acquired

 July 2018 - March 2019 Calculate the percentage of attendees who agree or strongly agree that they have increased knowledge and understanding
Intermediate Leadership teams will utilize reflection and ongoing PD activities to support local Early Intervention teams in implementing RPs Are local EI teams benefitting from the support being provided by the leadership teams? At least 75% of the attendees report gaining additional information or knowledge about the RPs Post reflection survey on information and knowledge acquired through participation in leadership team RP activities April 2019 -
July 2019		 Percent of attendees who report that they have acquired knowledge/information
Intermediate Early Intervention teams utilize the selected RPs in their work with families Are providers implementing the RPs when working with families? 75 % of surveyed providers will report greater use of family practices on the RP checklists At least half the providers who received training will complete a fidelity checklist about their use of the RPs April 2019-
July 2019		 Compare pre (acquired prior to Institute attendance) and post (need to determine timeline) checklists to determine change in practice
Long Families are able to help their children develop and learn Are families able to help their children develop and learn? 75% of the families surveyed in the IZs will report that they can help their children develop and learn on the FOS-R Families whose providers participated in training will be asked to complete a few questions from the FOS-R Initial measurements April- July 2019 Percent of families that report a mean of 4 or higher on this section of the FOS-R
Long term Children experience greater than expected growth in their acquisition and use of knowledge and skills in the pilot areas Has the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e. SS1) in the acquisition and use of knowledge and skills in our pilot areas increased? The percentage of children in summary statement 1 will increase by 0.9% over time in the three pilot areas Indicator 3.b collected in Cornerstone will show a .9 percentage point increase by FFY 2018 Data collected for FFY 2018 reporting Percent of children in SS1 for Indicator 3b

The largest anticipated barriers are getting providers to engage in the amount of professional development necessary to change their practices with families and getting them to complete evaluation measures. It is hoped that encouragement and support from their CFC and local leadership team will encourage their full participation. The state will continue to work with stakeholders and TA centers to access the resources necessary to support our implementation and evaluation activities.

CONCLUSION FOR ILLINOIS SSIP PRACTICE

Our second year of implementation has once again shown that our Plan was very ambitious. Given our resources and the time commitment required by local leadership team members and their supports, we have had to spend more time on child outcomes than originally anticipated. We continue to believe that the established processes and activities completed for our COS strategy will support our work as we focus on our family capacity-building strategy. We are confident that the investment in building LT capacity and commitment, relationship-building among LT members and supports, obtaining buy-in from local early intervention teams, engaging families in the COS discussion, defining a process for aligning professional development offerings with the elements of high quality PD, and supporting decision-making at both the local and state levels which is based on available data have enhanced our ability to address our second improvement strategy. We continue to believe that the leadership teams' support of desired outcomes is critical to the Plan's success. Implementation science has shown the benefits of local teams driving system change. We have made nice progress towards our short-term and intermediate outcomes related to the child outcomes summary process. We have furthered our teams' understanding of evaluation plans with assistance from the national TA partners and have collected a significant amount of data upon which to base decisions. We feel that we are continuing to improve our ability to plan for collecting the data that will help us determine the need for modifications. We continue to benefit from a group of stakeholders that are engaged and willing to help us make those decisions.

Appendix I - Chart

Narrative questions and responses:

Please comment on the meeting outcomes, and what you gained from participating in the meeting (e.g., knowledge, skills, motivation).

  • I feel that I have a deeper understanding of how the child outcomes process works because of the discussions lead by the presenter and ability to connect with both providers and service coordinators.
  • I learned a lot more about why Child Outcomes is a necessary process. Moreover, I learned how important it is for Service Coordinators to begin talking about the Child Outcomes process early to the families that they work with.
  • I learned it is important that both Providers and Service Coordinators need to be knowledgeable about the Child Outcomes Rating in order for it to be most effective and helpful to families.
  • The meaning and difference of immediate foundational skills and foundational skills was made more clear to me. Also, I better understand the decision tree and how to use it based on different situations.
  • I gained some new tips to help make the 3 outcomes more family friendly.
  • I think it focuses the providers more on what child can do vs what child cannot do. Follow cues from parents make sure they are not detaching from conversation and truly show them how what the kids are already doing can feed into what we will ALL be teaching them next!

Please describe one new thing you will try to incorporate into your practice as a result of participating in today's meeting.

  • For future exit meetings, I will take into consideration that if a child gets rated the same as entry, then that does not mean they did not make progress.
  • Learning more about typical child development in order to understand what an appropriate child outcome might look like.
  • Be mindful of parents' body language for when they might be feeling overwhelmed.
  • I will continue to try to keep the jargon out of my conversations with the family, and I will also continue to support the family to help make them more comfortable in contributing to the conversations about the child outcome ratings.
  • Continue to direct the conversation and questions towards the parent. Help them to be an active participant in our meetings

Narrative Questions and Responses - Chart

Please describe one new thing you will try to incorporate into your practice as a result of participating in today's meeting.

  • As a Service Coordinator, I will try to discuss the child outcomes summary with families in more detail during the intake meeting in order to give parents a heads up for what to expect.
  • As a Parent Liaison I feel that much of the language used regarding child outcomes can be helpful to use when I am meeting with families for a support visit or TPC with them. Families will benefit from hearing the same words and discussion throughout their time in EI.
  • I will try to apply the real life situations we talked about when working with my families and helping them better understand the child outcomes summary process.
  • Make sure that interpreters focus on the exact language that the SCs and providers are using.

Appendix IV - Chart

SSIP Leadership Teams

Benchmarks of Quality

The Benchmarks of Quality is used by a collaborative Leadership Team (LT) to assess progress and plan future actions to support the implementation of the IL SSIP within each CFC. The Benchmarks are grounded in the science of implementation, which bridges the gap between an evidence-based practice (EBP) and the actual high-fidelity implementation. Activities related to sustaining the effort are embedded throughout the process rather than being left until later. Benchmarks of Quality is a self-assessment tool that can be completed by the Leadership Teams as a whole or in small groups with the results from each group compiled into one consensus document to ensure all LT members are in agreement. The LT should use the data for planning future work and tracking progress.

If you require an alternate version of this report, please contact Jennifer Grissom at jennifer.grissom@illinois.gov.