Phase III (Year Three) - April 1, 2019

Illinois continues our work towards our state-identified measurable result (SiMR), to increase the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress in the acquisition and use of knowledge and skills in our pilot areas by .9 percentage points by 2018, as we are submitting our Year 3, Phase 3 report. As stated in our prior reports, these sites (Williamson County, Aurora, and East St. Louis) were selected to leverage resources and continue the work started during the Race to the Top Early Learning Challenge Grant. The majority of the work on the SSIP is now being completed by the evaluation team workgroups and the Leadership Team members in each of the pilot sites. The evaluation team continues to reference the Theory of Action (p.2) and the activities listed in the timeline (Appendix I). The evaluation team also continues to utilize the Theory of Change (p.3) when dividing the work into meaningful pieces. The three evaluation team workgroups- Leadership Team (LT), Professional Development/Technical Assistance (PD/TA), and Performance Support (PS) continue to support the work. Membership representation for each of these teams is detailed on page 7 of this report in the section Stakeholder Involvement in SSIP Implementation. The LT group focuses on aspects of the Plan related to Leadership Team development and support and assisted with refining our LT informational materials for new members. The PD/TA group focuses on aspects of the Plan related to professional development, coordinating curricula reviews and developing resource packages to support training. The PS group focuses on aspects of the Plan related to policy/procedure and implementation of practices, most recently focusing on the support needed to implement videotaping of the child outcomes summary process.

The primary focus for the third implementation year has been supporting the leadership teams (LT) in our three pilot areas as they provide professional development on the child outcomes summary (COS) process for their local teams, reviewing ongoing professional development offerings utilizing our rubric to ensure that they are of high quality, assessing fidelity to our desired COS process, and beginning our family engagement work. Although we recognize that changes to the COS process do not, on their own, improve children's acquisition and use of knowledge and skills, our stakeholders felt that it was imperative to improve the quality of COS data. We believe that better data will provide teams with better information for intervention planning and will allow us to better understand the impact of the family engagement practices we are supporting. We also feel that the process we have implemented engages families and team members in more collaborative decision-making, setting the stage for our second improvement strategy. The evaluation plan continues to undergo modification and refinement as we collect data on the utility of the professional development, assess fidelity, and examine the need for additional infrastructure improvements.

While increasing the field's knowledge about the child outcomes summary process has continued into this year, we have also begun planning and implementing strategies to build capacity around family engagement, continued to discuss ways to demonstrate the important changes being made to our infrastructure as a result of the SSIP, planned for how to assess fidelity to the desired child outcomes process (ensuring that outlined practices are being utilized), examined data to determine the need for adjustments/modifications to the COS process and professional development offerings, and surveyed leadership teams about their needs as they relate to supporting family engagement in their respective areas. In regard to the family engagement improvement strategy, we conducted our initial Leadership Team training, began developing our family engagement resource package, created a pre- and post-survey for professional development attendees to examine practice change, and modified family engagement offerings' evaluations. As outlined in previous submissions, Illinois continues to use multiple stakeholder groups to evaluate activities and refine the strategies intended to support the (SiMR).

SUMMARY OF PHASE 3, YEAR 3

The Theory of Action, Theory of Change, our Coherent Improvement Strategies, and our SiMR have not changed. We continue to use them to guide our SSIP work as well as general system improvement efforts. They are all included on the following pages as a reference.

As we shared previously, the evaluation team felt that we needed a graphic that more explicitly showed how the pieces of the plan and the Theory of Action related to each other and how each activity was intended to support the vision of improving outcomes for children. The Theory of Change shown below allowed us to break down the work listed in the Plan and assign tasks to new or existing workgroups. The Theory of Change also helps us think about what we need to be evaluating to determine the success of implemented activities.

State-identified Measurable Result:

To increase the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e., Summary Statement 1) in the acquisition and use of knowledge and skills in our pilot areas (i.e., Aurora, East St. Louis, and Williamson) by .9 percentage points by 2018.

SiMR Target and Data (FFY17)

FFY17/SFY18	2018
Target	78.5%
Data	74.7%

 SiMR Performance and Compliance by Pilot Site

Innovation Zone	SS1 for 3B (FFY17/SFY18)	# of eligible exits	# of Matched Entry-Exit Pairs	# of Matched Pairs with Complete Data	% of all eligible exits
East St. Louis	60.9%	38	27	24	63.2%
Aurora	83.3%	259	211	189	72.9%
Williamson Co.	53.7%	75	65	62	82.7%

Although we did not meet our target, we continue to see improvement. We have also noticed substantial fluctuations in the summary statements for the pilots. In at least two of the instances, this is likely due to the small number of matched pairs. Due to questions from stakeholders, we have included compliance information. The final column in the second chart indicates the percentage of the total number of children exiting that had a usable rating. We expect that continued data collection and improved family engagement will impact this percentage and the fluctuations due to small sample sizes.

The Coherent Improvement Strategies:

Our Coherent Improvement Strategies are:

• Implement effective training for leadership teams and EI providers that focuses on infant/toddler development and the Child Outcomes Summary (COS) Process and make related changes to state policy and guidance documents, so that early intervention teams implement the Child Outcomes Summary Process as desired.
• Implement effective training for EI providers that focuses on evidence based, family capacity-building practices, and make related changes to the local support structure by creating leadership teams, providing technical assistance and revising state policy and guidance documents, so that early intervention teams utilize practices that encourage the active participation of families in the intervention process by embedding intervention strategies into family/caregiver routines.

Progress in Implementing the SSIP

Although we have begun our family engagement efforts, we are farthest along with practice change related to the child outcomes process. We have been working to help teams understand the desired process, to develop/compile resources to support this process, and to offer opportunities to reflect on ways to continually improve. We have conducted evaluations of professional development activities and have begun assessing teams' use of desired practices. This information will inform future PD activities being conducted by the local leadership teams. With the exception of a single outlier at one training, over 70% of the participants say that they are benefitting from the training and support being provided through SSIP activities. We have now begun to also address our second improvement strategy. As with the first strategy, we have begun building leadership team capacity so that they feel capable of supporting the teams in their area (see leadership team capacity survey results in Appendix IV). Teams have indicated their need for ongoing training and support in this area. We have also started offering professional development on family-centered practices (one offering completed with three more linked series in progress). We continue to meet to refine our evaluation activities related to infrastructure improvements and implementation of the selected recommended practices (see pre-survey results about current practice use in Appendix IV).

Implementation progress is reflected in the timeline found in Appendix I. As part of our ongoing evaluation of implementation, the timeline continues to be updated to reflect progress and adjusted based on available resources and the needs expressed by local leadership team members. Each pilot site's activities continue to be driven by the Plan and the needs identified by the Local Leadership Teams.

As expected, many of our ongoing activities focused on improving the child outcomes process through professional development and reflection opportunities with local early intervention teams. While this work began in prior fiscal years, it has continued throughout the majority of this

fiscal year as well. In addition to focusing on family engagement this year, we continue to build the local leadership teams' capacity to use data to inform professional development. A detailed timeline with SSIP activities can be found in Appendix I.

The primary implementation activities for year 3 included: continued COS training for local early intervention teams, expanded reflection opportunities for early interventionists, implementation of the communication plan, quarterly reviews of select curriculum, refinement of the checklist to assess fidelity to our desired COS process, completion of the consents and procedures for examining COS process fidelity, and implementation of professional development for local leadership teams and their early intervention teams on the selected DEC Recommended Practices targeting family engagement. A list of activities that have been completed during the first three years of Phase 3 is provided below:

• Training rubrics reviewed and new rubric with guidance document developed - COMPLETED FFY15
• System developed to use rubric - COMPLETED FFY15
• Professional development panel members selected (EITP staff, students, and faculty supports); reviewers trained prior to utilizing the rubric; prioritized curriculum deemed most relevant to SSIP implementation activities - COMPLETED FFY15
• LTs in place with state supports (EI partners identified - EITP and Monitoring) - COMPLETED FFY15
• LT guidelines created: Guidance document, Benchmarks of Quality and Action Plan - COMPLETED FFY15; UPDATED FFY17
• General training about LT functioning and action planning - COMPLETED FFY15
• COS train the trainer event for LT members - COMPLETED FFY15
• Messaging rubric (with list of critical messages) - COMPLETED FFY15
• COS informational materials for parents for intake and IFSP - COMPLETED FFY15
• COS materials for stakeholders and providers - COMPLETED FFY15
• Development of COS resource guide - Tip Sheet for parents - COMPLETED FFY16
• COS resource guide distributed and advertised widely including monitors- COMPLETED FFY16
• Develop resource guides to support relevant training curriculum- COMPLETED FFY16
• Advertise availability of resource guides linked to curriculum- COMPLTED FFY16
• Include resource guides in supplements for appropriate trainings- COMPLETED FFY16
• Revision of existing monitoring tools and development of observation tools/checklists for COS - COMPLETED FFY16
• Establish process for coordinating ongoing communication- COMPLETED FFY16 AND ONGOING
• Training calendar for LT members - COMPLETED FFY16 AND ONGOING
• Viable LT COS training calendar for local EI teams - COMPLETED FFY 16 AND ONGOING
• Establish meetings for early interventionist to reflect on COS implementation- COMPLETED FFY16 AND ONGOING
• Support peer to peer observations, critique, and support- COMPLETED FFY16 AND ONGOING
• Assess fidelity to COS process- COMPLETED FFY17
• Determine leadership team capacity and ongoing needs related to family engagement- COMPLETED FFY17
• Team meets to identify topics for the focus of training on family engagement EBPs/DEC Recommended Practices - COMPLETED FFY17 AND ONGOING
• Creation and implementation of policy/procedure to define the "up to 20 hours" of EITP training - IN PROGRESS
• Revision of payee/provider agreement - IN PROGRESS

The outputs for year 3 include: COS training for local early intervention teams, focused conversations (reflection opportunities) for early intervention teams that have received training, Family Engagement resource package for use by local leadership teams, revised checklist to examine COS fidelity developed in conjunction with our ECTA and DaSy TA partners (see checklist in Appendix II), updated consents and written process for COS fidelity observations, and family engagement trainings for Leadership Teams and early intervention teams. Fidelity checklists were used for observations by Leadership teams and for scoring of video recordings by Early Intervention Training Program staff. Additional information about outputs and their relationship to implementation activities can be found in the section titled "Data on Implementation and Outcomes".

Stakeholder Involvement in SSIP Implementation

The large SSIP stakeholder group continues to receive quarterly written updates summarizing implementation. Each summary asks for recipients to contact staff if they have questions about reported activities or suggestions for future activities. The summary also contains a reminder that those receiving the summary should share widely with those whose interests they represent on the large stakeholder group. In addition, the large stakeholder group was reconvened in June 2018 in order to receive a progress update and provide input on potential evaluation strategies for the second coherent improvement strategy. Other stakeholders have been informed via information provided in the EI Partners' quarterly newsletters (EI Training, EI Clearinghouse, and Provider Connections) and postings on their websites. The SSIP work has also been discussed at each monthly CFC Managers Meeting, each quarterly Illinois Interagency Council on Early Intervention (IICEI) meeting, each monthly EI Partners' meeting, and through presentations at various professional conferences. In addition, the Early Intervention Training Program has a resource page dedicated to the State Systemic Improvement Plan (https://blogs.illinois.edu/view/6039/378910) and has created a specific resource page for materials that the local leadership teams are using.

In order to optimize stakeholder involvement and engagement, we continue to utilize the expertise of a variety of groups:

1. SSIP Core Team - consists of Illinois Department of Human Services (IDHS) Early Intervention staff and the EI Ombudsman
2. SSIP Stakeholder group - consists of the state interagency coordinating council members, direct service providers, contracted Child & Family Connections (CFCs which are regional entities that serve as points of entry for EI services) staff, parents, professional provider associations, the Part B/619 coordinator, IDHS planning/evaluation members, parent training and information center staff, and contracted EI partners for training, credentialing, monitoring and clearinghouse.
3. SSIP Evaluation Team - consisting of Bureau staff, Training staff, Monitoring staff, CFC personnel, EI Provider, and IDC TA representative
4. Leadership Team workgroup - consisting of Bureau staff, CFC managers, EI Training staff, EI Monitoring personnel, a parent liaison, and an external professional development representative
5. Professional Development/Technical Assistance workgroup- consisting of EI Ombudsman, EI Training, EI Provider and IDEA Data Center representative
6. Performance Support workgroup - consisting of EI Bureau, Local Leadership Team members (service coordinators and providers), ECTA representative, EI Training, EI Ombudsman, and EI Monitoring
7. Local Leadership Teams (in each of the three pilot sites) - consisting of CFC staff, EI Providers, EI Monitoring, EI Training and parents when possible.
8. Messaging workgroup - ad hoc team consisting of EI Bureau staff, EI Ombudsman, Local Leadership Team members, EI Clearinghouse, and Illinois Race to The Top representatives.

Cumulatively, these groups have been involved in all implementation activities and continue to inform the decision-making process and implementation timelines. The large stakeholder group primarily receives updates and they have requested an annual in person meeting to monitor progress. The evaluation team continues to meet, reviews evaluation data, determines need for adjustments, and plans for next steps/activities. The leadership team workgroup has revised the materials leadership team members receive and supports the work of the local leadership teams. The PD/TA workgroup continues to review curricula and prepared the plan for local leadership team training around family engagement. The performance support workgroup has been involved in the revision of the fidelity checklist and in finalizing the process for COS fidelity observations. The local leadership teams drive local efforts around training and support, monitor their performance in regard to the Benchmarks of Quality, and provide input around the evaluation of their activities. The messaging workgroup completed a major task of finishing a resource for primary referral sources to understand the EI philosophy and evidence-based practices. The EI Clearinghouse, our central directory, has been addressing the consistency of system messages and revising public awareness materials they publish and share to ensure they reflect desired messages about early intervention. While we continue to work on improving communication across the groups so that everyone can stay informed about, and provide input to, other groups' activities, this division of labor has allowed substantial stakeholder input without overburdening individuals. It has also ensured that information is constantly flowing between those responsible for implementation of the Plan and those working directly with children and families.

The evaluation team has continued to meet (June 26th 2018, December 18th 2018 and February 27th 2019) to plan, complete, and monitor the activities included in the Plan. In addition, technical assistance from ECTA has helped with a fidelity measure and process development. Our evaluation plan has been revised and refined based on an increased understanding of necessary plan components as a result of our participation in the TA centers' ongoing workshops on SSIP evaluation. In addition to the original outcomes and revised evaluation questions, we have now included information about infrastructure evaluation in order to better align our evaluation with our theory of action and to demonstrate the progress being made while we wait for the changes we have made to impact our long-term outcome- a change in our SiMR.

We continue to evaluate the three general types of activities being conducted during the implementation phase- leadership team planning, leadership team capacity building, and leadership team/EITP provided professional development. As during prior years, leadership team planning has minimal evaluation as these events are used to keep the process moving forward. Leadership team capacity building evaluations continue to examine the extent to which the intended outcomes of the activity are achieved. Leadership team/EITP provided professional development opportunities are conducted by the leadership teams or the Early Intervention Training Program for local leadership and EI teams and the evaluations of these events include determining whether or not the intended outcomes were achieved by event participants. These evaluations have been used to understand our progress and answer our evaluation questions. In addition to evaluating our professional development activities, we have also examined implementation fidelity for the outlined COS process through the use of videos and the revised checklist.

Additional, detailed descriptions of stakeholder involvement in reviewing SSIP evaluation data and making recommendations and decisions based on those data, are provided in the narrative under Data on Implementation and Outcomes.

Data on Implementation and Outcomes

The Plan below shows the outcomes we want to examine as well as evaluation questions, performance indicators, measurement plans, timelines, and related analyses.

Evaluation Plan for Coherent Improvement Strategy 1: Implement effective training for leadership teams and EI providers that focuses on infant/toddler development and the Child Outcomes Summary (COS) Process, and make related changes to state policy and guidance documents, so that early intervention teams implement the Child Outcomes Summary Process as desired.

Outcome Type	Outcome Description	Evaluation Question(s)	How Will We Know the Outcome Was Achieved?  (perform-ance indicator)	Measurement/ Data Collection Method	Timeline/ Measure-ment Intervals	Analysis Description	Data/ Results
Short term	Leadership teams will have the skills and knowledge to support local Early Intervention providers in implementing the desired Child Outcomes process using training and reflection activities	Did the individuals on the leadership teams acquire the skills and knowledge necessary to effectively provide training and reflection activities to support local early intervention teams in their implementation of the Child Outcomes process?	At least 75% of attendees report increase in the skills and knowledge acquired	Pre- survey of leadership team members about their knowledge of child outcomes process and ability to support others in implementation of the process Post-survey focused on what they now know about child outcomes	Pre-survey completed Oct-Nov 2016 Post-survey completed between Dec 2016- Jun 2017	Calculate the percentage of leadership team members who respond that they have the knowledge/skills on the pre-survey and the percentage who report acquiring knowledge/ understanding on post training surveys	85% of respondents reported that they had the knowledge and skills required on the pre-survey Despite the high level of competency reported on the pre-survey, 88% of the LT members agreed or strongly agreed that they had increased their knowledge and understanding of critical COS components.
Short term	Early Intervention providers have acquired the skills and knowledge necessary to understand how the Child Outcomes process is to be implemented	Do individuals have the skills and knowledge necessary to understand how the Child Outcomes process is to be implemented?	At least 75% of attendees report increase in the skills and knowledge acquired	Post training survey of early intervention team members about the knowledge they have acquired	Feb 2017- Feb 2018	Calculate the percentage of attendees who agree or strongly agree that they have increased knowledge and understanding	During this reporting period, 92% of participants agreed or strongly agreed that they had increased knowledge/ understanding
Intermediate	Leadership teams will utilize reflection and ongoing PD activities to support local Early Intervention teams in implementing the Child Outcomes Process	Are local EI teams benefitting from the support being provided by the leadership teams?	At least 75% of the attendees report gaining additional information or knowledge about implementing the COS process	Post reflection survey on information and knowledge acquired through participation in COS reflection activities	Jul 2017- Feb 2018	Percent of attendees who report that they have acquired knowledge/ information	For this reporting period, 50-100% of the attendees agreed or strongly agreed that they received additional information or knowledge about COS process
Intermediate	Early intervention teams utilize the recommended process for determining Child Outcomes ratings	Are providers implementing the Child Outcomes process as outlined in training and procedures?	75 % of the items on the child outcomes fidelity checklist are being completed	Fidelity assessed using recorded discussions & direct observations with a checklist developed based on IL policy/ procedure (25% of SCs in pilot areas to gain initial look at fidelity and to refine checklist and procedures)	Feb-Mar 2018	Recordings and direct observations will be used to score the fidelity checklist to determine if process is being implemented as desired and to inform areas needing additional LT support; LTs can then use COS resource package to offer additional support	For this reporting period using videos and revised checklists, 71-94% of the items are being completed
Long term	Children experience greater than expected growth in their acquisition and use of knowledge and skills in the pilot areas	Has the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e. SS1) in the acquisition and use of knowledge and skills in our pilot areas increased?	The percentage of children in summary statement 1 will increase by 0.9% over time in the three pilot areas	Indicator 3.b collected in Cornerstone will show a .9 percentage point increase by FFY18	Data collected for FFY18 reporting	Although change is not yet expected due to recent initiation of training and support, information about indicator 3B for summary statement 1 is already being collected for the three pilot areas	Results included each year with SSIP report; 74.7% for FFY17/ SFY18

Evaluation Plan for Infrastructure Improvements:

Outcome Type	Outcome	Evaluation Question(s)	How will we know? (Performance Indicator)	Measurement/ Data Collection Method	Timeline/ Measure-ment Intervals	Analysis Description/ Data Management	Date/ Results
Short term	CFCs will produce high quality child outcomes data	Do CFCs have the tools and information necessary to collect high quality child outcomes data?	More of the Quality Indicators in the State Child Outcomes Measurement System will be at least partially implemented each time we review.	State child outcomes measurement system data collection and transmission subcomponent	Baseline 2015 Post Jul 2018	Involve stakeholders in completion of the self-assessment. Compare self-assessment ratings from baseline and post and compute the percent of QI ratings for data collection and transmission that increased	10 of the 14 applicable Quality Indicators now have elements that are partially implemented.

The bulk of our evaluation measures examine the impact of training and technical assistance on practitioners' implementation of desired practices for the COS process and family engagement. While we feel strongly that the other two areas of our theory of action are critical to our success, we did not originally discuss evaluation of these areas. We now recognize that these areas, e.g. the impact of improved policies/procedures and whether internal and external knowledge of EI is changing, need consideration by our evaluation team to see if additional evaluation measures are necessary. We have included the evaluation information for the pieces of the COS measurement system that we have been able to put in place. Evaluating the increased knowledge of family-centered practices by internal and external stakeholders still needs to be addressed more fully. While internal knowledge will be examined through the evaluation of PD activities targeting our second improvement strategy, the primary way we are currently addressing increased external knowledge is through Messaging workgroup activities. The Messaging workgroup was charged with creating items to address the needs of the Local Leadership Teams (and all CFCs in general) to support the knowledge of EI to the local communities, the referral sources and families. These materials are intended to support improved internal and external knowledge of the early intervention system. This included updates to existing materials and creation of a new Referral Brochure resource focused on how to speak to community partners on what to expect when referring to EI.

Data sources, available baseline data, data collection procedures and timelines are all outlined in the chart above. In addition, sampling procedures (when applicable) and data comparisons are included. In terms of data management, the data related to professional development activities is collected and stored by the Early Intervention Training Program. Results are reviewed quarterly and shared with Local Leadership Teams so that they can use this information to inform next steps in their professional development efforts. COS fidelity checklist information is collected by the CFCs and securely stored by the Early Intervention Training Program. COS data is collected by early intervention teams and entered into Cornerstone by service coordinators. Early analyses of the professional development information have allowed us to determine whether or not participants feel that the trainings and reflection opportunities are achieving the desired outcomes of increasing knowledge and understanding. So far, the feedback has been generally positive, so minimal adjustments have been made to the curriculum. The one change has been in the focus of the reflection opportunities. They have expanded the focus beyond the COS and are addressing other practice challenges. Training participants report increased knowledge and understanding of the various COS components and indicate plans to use what they have learned in future COS discussions. In addition, we have used fidelity checklists through both observation and videotaping to determine if practices are actually being implemented. Both live observations and recordings indicate a high level of fidelity with the outlined COS process. As with the other evaluation data, the information collected from the checklists allow the local leadership teams to determine next steps in their professional development efforts. For example, many service coordinators are not using the "COS at IFSP" document created for families. Feedback from teams indicate that they are covering the material in the document verbally and are trying to refrain from overwhelming families with paperwork. While this may be a good approach for many families, some may benefit from visual support for what they are being told. This point will be made in upcoming meetings with leadership teams. Although we do not yet expect our improvement strategies to be impacting our SiMR in the way we anticipated, we have been collecting and reporting this information. It has allowed us to gauge natural fluctuations in performance and monitor data collection practices.

Evaluation data has been used to understand the impact of the professional development being provided by the local leadership teams. This includes evaluation of training, reflection activities, and fidelity to the desired COS process. With our broadening understanding of SSIP implementation and evaluation, we have realized that we need to collect more information about the changes implemented related to our infrastructure. While those changes have largely proceeded as planned, we did not initially realize the importance of monitoring or measuring their implementation. We have continued to use the ECTA/DaSy State Child Outcomes Measurement System to examine completed and needed changes to our child outcomes measurement system. Members of the Outcomes Workgroup have assisted with rating the various items and with setting priorities for future work. Results of the self-assessment can be found below.

State Child Outcomes Measurement (S-COMS) Self-Assessment

Data, in its broadest sense, have informed all changes to implementation and improvement strategies. For example, we had initially planned to use video recordings to assess COS implementation. Not all local leadership teams were prepared for recording or believed in the importance of this approach to assessing fidelity. This feedback led to the performance support workgroup developing additional support materials. Teams worked prepare service coordinators for recording by practicing with equipment and developing a plan for video sharing. Materials to be shared with the families and providers who would be recorded were also developed. Another example of data-informed change can be seen when evaluations of the initial family engagement training indicated that local leadership teams wanted additional support around the practices so that they could support others. They acknowledged the complexities involved in engaging families during all aspects of the IFSP process and recognized the need to further their own practice in order to be able to support the teams in their respective areas. Leadership teams have also determined which family engagement trainings to offer in their local areas based on the needs of the early intervention teams in their local areas, e.g. writing family-centered IFSP outcomes, engage and attune, family-centered institute.

Data on the PD opportunities indicated that the activities were achieving their desired outcomes, so they were not modified. In addition, the PD activity evaluation data and fidelity information will be used to guide future local leadership team offerings. These offerings will be supported by the activities outlined in our resource packages (the COS resource package is available on the LT resource page: https://blogs.illinois.edu/view/7582/592341. Another example of how data and feedback influence changes can be demonstrated by changes to the frequency and focus of reflective opportunities. As leadership teams noted decreased interest in, and attendance at, their focused conversations (opportunities to reflect on COS practice) they altered the offerings to provide more generalized system support rather than just focusing on the child outcomes process. The participants then mentioned that they felt like the event evaluation no longer reflected the content of the conversations and the evaluation was updated as well.

As reported last year, stakeholders (evaluation team and local leadership teams) reflected on the elements of high-quality PD, the ongoing trainings currently offered by EITP, and our credentialing process, and determined that a separate training on child development was not necessarily going to improve child outcomes ratings. While necessary information was included within the ongoing child outcomes trainings, the Early Intervention Training Program has also revised its Baby Steps Institute (linked PD opportunity on child development) to make more direct links to child outcomes and their measurement. This will help more teams understand the connection between typical child development and child outcomes as we scale up and sustain these efforts.

Beyond the examples already listed, most of our data have supported that SSIP implementation should proceed as planned. Data collected demonstrates improvements in the knowledge and skills of leadership team members, improvements in the knowledge of early intervention team members, and improved use of the desired components of the COS process. In addition to this evaluation data, we have also received anecdotal feedback from leadership team members that families are more engaged in the COS discussions, teams realize their earlier discussions were not appropriately age-anchored, and IFSP meetings are more streamlined with IFSP outcomes more closely tied to identified needs.

Each of the quarterly SSIP summaries sent to stakeholders included evaluation data with an accompanying description. The large stakeholder group was also convened in June to provide feedback on the evaluation plan. Their input on whether or not their informational needs were being met and ideas for measuring practice change were solicited. We also discussed what we have learned about evaluation, shared available data, and determined the need for revisions to our process and measures. As mentioned previously, the evaluation team (consisting of stakeholders) continues to meet and engage in decision-making about the ongoing evaluation of activities.

Data Quality Issues

We have not yet identified any concerns with the quality of the evaluation data we have collected. We have established systematic processes for collection. We realized last year, however, that we are potentially missing some evaluation measures, particularly those that would more extensively examine our infrastructure improvement strategies such as the Child Outcomes Measurement System. Although we knew we would use our Benchmarks of Quality as our primary measure, that tool does not reflect the breadth of changes being made to the statewide infrastructure. We need to continue to more fully develop these measures and are considering revisiting our original data analysis and review of system strengths and weaknesses. As mentioned previously, we are considering utilizing the ECTA/DaSY system framework to examine changes within the EI system. Although consistent review of these components demonstrates that there is still room for improvement, using the framework could provide a way to demonstrate the progress that we have made. We will continue working with our stakeholders in the coming year to determine which areas need examination and targets for improvement. We also still have concerns about the overall quality of our child outcomes data and its ability to accurately evaluate our progress toward the SiMR. Data system edits implemented in FFY15/SFY16 to prevent incorrect COS entries, have decreased the volume of errors. From FFY16 to FFY17, system edits impacted late entries, which decreased by 1.2% and early exits which decreased by 8.4%. Remaining errors have identified the need for additional technical assistance within the CFCs and additional improvements to the data system. Additional guidance will also be provided in the CFC Procedure Manual.

While we hope to further develop a plan for evaluating infrastructure improvements, we realize that many changes to our infrastructure will be difficult to evaluate despite their contribution to sustainability. Continued data entry issues are impacting our ability to utilize the full amount of COS data being collected and potentially impact our understanding of the progress being made on the SiMR. Although more families are participating in the process and more data are being entered, the usable data do not reflect this increase. There are a variety of reasons that data had to be excluded but the two primary issues are related to data entered outside of the established timelines, e.g. entry not collected at initial IFSP and exit entries that are completed more than 120 days prior to system exit. These reasons account for 43% (FFY16) and 33% (FFY17) of our missing data. We have not yet determined if the children for whom we have matched entries differ in important ways from those who had to be excluded, so our understanding of child outcomes may not reflect the breadth of what is occurring in the field. As we move forward, we have chosen to pilot increased data sharing with a few CFCs to examine a variety of performance data, including the COS, on a more frequent basis to better understand what additional supports are needed for better data quality.

The state, with the hiring of a qualified data manager, has begun to work closely with the CFCs to examine data more frequently and provide analyses to identify issues and trends in a more-timely fashion. We feel that this improved use of the available data will lead to improved practice. More complete and accurate data will allow teams to better understand the improvements made by the child. Enhanced data use will also help the local leadership teams identify specific data or system needs to be addressed prior to statewide implementation.

Continued use of the COMS self-assessment and review of other frameworks can help us determine next steps for system improvement as we recognize the importance of enhancing the infrastructure to ensure changes/improvements are sustainable.

Progress Toward Achieving Intended Improvements

To date, the primary infrastructure changes that support our SSIP, and in turn, our SiMR are related to enhancements to our child outcomes measurement system and the local leadership teams. Although the specific accomplishments have been highlighted earlier, it is important to note that we feel the changes to the Child Outcomes Measurement System will assist with sustainability and will be an important support when we begin scale-up. While these changes are a positive first step, we realize that more of the components of the child outcomes measurement system will need to be implemented in order to be assured that we are following a high-quality process that results in high quality data. The child and family outcomes workgroup members have identified some priority items that they will be addressing to assist with this work.

The local leadership teams are another vital part of our infrastructure change. They allow us to be responsive to local needs and provide local support for improved practices which is critical for a high quality professional development system. As noted earlier, these teams have been conducting professional development activities, participating in evaluation activities, and planning for future activities in their local area. The work of these teams has been guided by the Benchmarks of Quality established in our first year of implementation. All teams have demonstrated significant progress towards achieving the benchmarks necessary for a well-functioning leadership team.

The three local leadership teams continue to update their ratings on the Benchmarks of Quality on an annual basis. Each item on the Benchmark received a rating. The chart below shows the number of items on the Benchmarks that have shown progress since 2016 when the teams were first established. The Benchmarks of Quality guiding the teams' work can be found in Appendix V.

See Chart in PDF Version 

The Benchmarks of Quality (see Appendix V) have guided the local leadership teams' work. Implementing the items on the checklist has helped the teams serve as a support for practice change in their local area. As indicated previously, the areas that teams were asked to consider were:

• Leadership Team (creation, sustainability, objectives, membership, etc.)
• Action Plan (development, implementation, etc)
• Buy-In (communication, collaboration, feedback, etc.)
• Family Engagement (representation, communication, etc.)
• Communication (updating stakeholders, sharing data, routine schedule, etc.)
• Professional Development and Ongoing Technical Assistance (consistent attendance at offerings, self-assessment, mentoring, etc.)
• Data-Based Decision Making (data training, data entry, measuring improvement, etc.)

The charts below show the performance of the individual pilots for each of the identified areas. The blue information shows the teams' initial Benchmarks of Quality ratings when their teams were first convened. The orange bar shows where the teams were last year with grey indicating their current status. The progress over time is demonstrated side-by-side for each of the elements.  

See Charts in Printer-Friendly Version

It is evident from these charts, that although the teams started in different places, they have all made significant progress in creating a valuable infrastructure element. This progress on the Benchmarks demonstrates that our local leadership teams are implementing the pieces necessary to be a valuable support for practice change. Further evidence of progress is seen in our other evaluation information which shows that the teams' support activities are increasing early intervention teams' knowledge of system processes. Our use of the fidelity checklist has provided a look at whether increased knowledge and understanding is leading to a COS process conducted with fidelity. Though fewer videos were obtained than we hoped, we have video evidence that the process is being implemented with a high level of fidelity.

Our results indicate that we have achieved our short-term outcomes related to our first improvement strategy. Both leadership teams and early intervention teams have acquired knowledge and skills related to the COS process. We have also evaluated our intermediate outcome for this strategy in that early intervention teams have access to ongoing support related to the COS process. Video recordings and observations have shown that teams are demonstrating a high level of fidelity to the proposed process. We feel that the training and support provided have been necessary steps for moving our COS process forward and are very promising. Ensuring that we are engaging in a high quality COS process gives us the best chance for understanding the impact of our family engagement practices' on the SiMR.

We have just started seeing improvement in the SiMR (see charts on targets and performance on page 5), but we are not yet meeting our targets. As implementation has proceeded, we have realized that we were unaware of how many teams were rating children higher than they should and that we were too ambitious in our prediction of how much impact we could have on the SiMR during the SSIP timeframe. Although we suspected that a focus on the COS would alter our performance data, the initial decreases in the summary statement were even greater than we anticipated. We had also hoped that the focus on the COS would improve the number of usable matched entry-exit pairs in each of the pilot areas. While the total number of usable matched pairs increased this year, one pilot area showed a substantially larger increase due to new practices being implemented within the CFC. We know that there are many moving pieces to achieving our SiMR and we are excited to be addressing our second improvement strategy and to see how it will impact our SiMR. With each piece that gets implemented, we are moving closer to our goal. Although we know that the impact of this strategy will not yet be seen in our SiMR, we are eager to see how improved family engagement will impact intervention and outcomes.

PLANS FOR NEXT YEAR

As we have altered the timeline for some of the activities, at this time we do not plan on adding new activities, but rather plan on moving further with the implementation of already proposed activities. The primary activities in the coming year will be related to our family engagement strategy. Specifically, these activities are:

• Continued training for leadership team members regarding family capacity building through the use of selected DEC Recommended Practices (RPs)- 4th quarter SFY19 and throughout SFY20
• Resource package of activities that can be used by local leadership teams to support family engagement/capacity building professional development activities- 4th quarter SFY19 due to delayed responses to LT surveys
• Training and support for early intervention teams on family capacity building RPs- continuing through 1st quarter SFY20
• Additional training on using data to inform professional development-beginning 4th quarter SFY19
• Continue creation and implementation of policy/procedure to define requirements for "up to 20 hours" of EITP training- ongoing during SFY20
• Revision of payee/provider agreement and supporting documents to include language about new training requirements-beginning 3rd quarter SFY19
• Develop/modify materials related to evidence-based intervention practices (family capacity-building, family engagement, family decision-making, & family centered practices) utilizing recommendations for tailoring information to specific audiences (e.g. families/caregivers, providers, CFC staff)- ongoing throughout SFY20
• Continued refinement of our evaluation plan- Continuing during 3rd quarter SFY19 and throughout SFY20
• Continue building infrastructure components identified for a high-quality child outcomes measurement system- throughout SFY20
• Enhance use of resource guides and resource packages to support family engagement efforts- throughout SFY20
• Develop plan for scaling up desired practices- throughout SFY20

EVALUATION PLAN FOR SECOND COHERENT IMPROVEMENT STRATEGY: Implement effective training for EI providers that focuses on evidence based, family capacity-building practices, and make related changes to the local support structure by creating leadership teams, providing technical assistance and revising state policy and guidance documents, so that early intervention teams utilize practices that encourage the active participation of families in the intervention process by embedding intervention strategies into family/caregiver routines and activities.

Outcome Type	Outcome Description	Evaluation Question(s)	How Will We Know the Outcome Was Achieved? (perform-ance indicator)	Measure-ment/ Data Collection Method	Timeline/ Measure-ment Intervals	Analysis Description	Data/ Results
Short term	Leadership teams will have the skills and knowledge to support local early intervention providers in implementing the selected DEC Recommended Practices (RPs)	Did leadership team members acquire skills and knowledge to support local early intervention teams in their implementation of selected RPs?	At least 75% of attendees report increase in the skills and knowledge acquired	Pre- survey of leadership team members about their knowledge of and ability to support use of RPs Post-survey focused on what they now know about RPs	Pre-survey completed Apr 2018 Post-survey completed May 2018	Calculate percentage of LLT members who report they have the knowledge/skills on the pre-survey and the percentage who report acquiring knowledge/ understanding on post training surveys	75% of LT members reported having all the desired knowledge/ skills; 86% increased their confidence with implementing the selected RPs
Short term	Early intervention providers have acquired the knowledge necessary to implement selected RPs	Do individuals have the knowledge necessary to implement selected RPs?	At least 75% of attendees report increase in the skills and knowledge acquired	Administer pre-survey (RP checklists) about RP use prior to training Post training survey of early intervention team members about the knowledge they have acquired	Jan-Mar 2019	Calculate the percentage of attendees who agree or strongly agree that they have increased knowledge and understanding	Attendees demonstrate variability in their use of the RPs. 70-100% of attendees self-reported using the practices often or most of the time.
Intermediate	Leadership teams will utilize reflection and ongoing PD activities to support local Early Intervention teams in implementing RPs	Are local EI teams benefitting from the support being provided by the leadership teams?	At least 75% of the attendees report gaining additional information or knowledge about the RPs	Post reflection survey on information and knowledge acquired through participation in leadership team RP activities	Mar 2019- Jun 2020	Percent of attendees who report that they have acquired knowledge/ information	Not yet evaluated
Intermediate	Early Intervention teams utilize the selected RPs in their work with families	Are providers implementing the RPs when working with families?	75 % of surveyed providers will report greater use of family practices on the RP checklists	At least half the providers who received training will complete a checklist about their use of the RPs	Apr 2019-Jul 2020	Compare pre (acquired prior to Institute attendance) and post (need to determine timeline) checklists to determine change in practice	Not yet evaluated
Long	Families are able to help their children develop and learn	Are families able to help their children develop and learn?	75% of the families surveyed in the IZs will report that they can help their children develop and learn on the FOS-R	Families whose providers participated in training will be asked to complete a few questions from the FOS-R	Initial measurements Apr-Jul 2019	Percent of families that report a mean of 4 or higher on this section of the FOS-R	Not yet evaluated
Long term	Children experience greater than expected growth in their acquisition and use of knowledge and skills in the pilot areas	Has the percentage of infants and toddlers with disabilities who demonstrate greater than expected progress (i.e. SS1) in the acquisition and use of knowledge and skills in our pilot areas increased?	The percentage of children in summary statement 1 will increase by 0.9% over time in the three pilot areas	Indicator 3.b collected in Cornerstone will show a .9 percentage point increase by FFY18	Data collected for FFY18 reporting	Percent of children in SS1 for Indicator 3b	Results included each year with SSIP report; 74.7% for FFY17/SFY18

CONCLUSION FOR ILLINOIS SSIP PRACTICE

Our third year of implementation has once again shown that our Plan was very ambitious. We needed to continue the COS training that had begun last year in order to reach a larger number of team members. We also needed time to increase the capacity of our leadership team members so that they felt prepared to support others' growth related to family capacity-building. We are confident that these time investments have ultimately enhanced our ability to address our second improvement strategy. We continue to believe that the leadership teams' support of desired outcomes is critical to the Plan's success as implementation science has shown the benefits of local teams driving system change. We have tried to strike a manageable balance between local teams expressing autonomy and obtaining the evaluation data and forward momentum we need. Teams also struggle with time commitments to the amount of work needing to be done. We continue to grow our understanding of how data impacts decision-making with assistance from national TA opportunities. We feel that we are continuing to improve our ability to plan for collecting the data that will help us determine the need for modifications, but occasionally struggle to find "just right" tools. We do, however, continue to benefit from a group of stakeholders that are engaged and willing to help us make those decisions. Initial discussions regarding scale up have brought to light the variations in available infrastructure across Illinois. The stakeholders have offered great ideas for recognizing the individuality of each CFC while also ensuring consistent implementation of evidence-based practices for child outcomes and family engagement. Our work in this regard will continue over the next year as we finalize plans for scale up.

Appendix I

Illinois State Systemic Improvement Plan Timeline (SFY18/FFY17)

Legend for Timeline:

• Item in progress
• Item postponed

X  Item completed

0  Item eliminated

Colored in boxes indicates the period in which activities are either intended to be, or are being, addressed

• Blue items are those being addressed by the PD/TA group
• Pink items are those being addressed by the LT group
• Green items are those being addressed by the PS group
• Yellow items are those being addressed by the Messaging group

Appendix II

(Blank)

IL- Child Outcomes Summary Process Checklist (Revised)

Please indicate the extent to which the team used these practices:	Yes/No (Y/N)	Evidence/Notes
Including the family: The team Communicates with the family about the COS process in an unbiased way by:	(Y/N)
describing/defining the three outcomes describing the rating criteria, and explaining how the family will be involved in the process explaining why we collect this information, e.g. determine the impact of the program	(Y/N)
Shares what data is collected and that it is used to evaluate the program
Uses the "COS at IFSP" document or verbalizes its content to explain the COS process at the ISFP meeting
Discusses the types of information that will be useful for the process. Examples (as appropriate) may include: reports from parents and/or other caregivers, information from intake (RBI and ASQ:SE), information from the referral source, evaluations, observations, and progress reports.
Both listens to the family and shares information to build respectful partnerships that are responsive to the family's cultural practices and beliefs
Uses an EI enrolled interpreter when needed to ensure understanding
Routinely checks for understanding by team members before moving on
Understanding child functioning: The team Shares information they have about the child's functional skills for each of the 3 child outcomes
Discusses the child's current functioning in each outcome area, including gathering information from the family about the child's participation in everyday activities across settings.
Ensures that information from multiple sources is considered for each outcome, e.g., family input, other observations, assessment, progress monitoring, child care providers, specialists, etc.
Discusses the child's current use of skills related to age-expected development (incorporating age anchors as necessary), including skills the child has and has not yet developed.
Building Consensus: The team Uses available resources/materials to create a shared understanding of the child's development.
Discusses the rating for each outcome in descriptive terms, NOT using a number.
Reaches consensus for each outcome's rating.
Ensures that the ratings are consistent with the information shared and discussed, verifying that sufficient evidence has been discussed to support the rating.
Documentation: The team Documents the rating in Cornerstone (ASO3 screen)
Includes sufficient evidence for the rating in the documentation (e.g., captures the reason for the rating)

Appendix III

Cumulative Summary of Two Pilots Submitting Video

IL- Child Outcomes Summary Process Checklist-Revised

SC Name: _____________________________

Meeting Type (check one): 

Initial 

Annual 

Exit

Please indicate the extent to which the team used these practices:	Yes	No	N/A or Missing
Including the family: The team Communicates with the family about the COS process in an unbiased way by: *describing/defining the three outcomes	9	1
*describing the rating criteria, and	9	1
*explaining how the family will be involved in the process	9	1
*explaining why we collect this information, e.g. determine the impact of the program	9	1
Shares what data is collected and that it is used to evaluate the program	8	2
Uses the "COS at IFSP" document or verbalizes its content to explain the COS process at the ISFP meeting:	2	8
Discusses the types of information that will be useful for the process. Examples (as appropriate) may include: *reports from parents and/or other caregivers, *information from intake (RBI and ASQ:SE), *information from the referral source, *evaluations, *observations, and *progress reports	10	0
Both listens to the family and shares information to build respectful partnerships that are responsive to the family's cultural practices and beliefs	9	1
Uses an EI enrolled interpreter when needed to ensure understanding	1	0	9- N/A
Routinely checks for understanding by team members before moving on	10	0
Understanding child functioning: The team Shares information they have about the child's functional skills for each of the 3 child outcomes	10	0
Discusses the child's current functioning in each outcome area, including gathering information from the family about the child's participation in everyday activities across settings	9	1
Ensures that information from multiple sources is considered for each outcome, e.g., family input, other observations, assessment, progress monitoring, child care providers, specialists, etc.	10	0
Discusses the child's current use of skills related to age-expected development (incorporating age anchors as necessary), including skills the child has and has not yet developed	10	0
Building Consensus: The team Uses available resources/materials to create a shared understanding of the child's development	4	6
Discusses the rating for each outcome in descriptive terms, NOT using a number	10	0
Reaches consensus for each outcome's rating	9	1
Ensures that the ratings are consistent with the information shared and discussed, verifying that sufficient evidence has been discussed to support the rating	10	0
Documentation: The team Documents the rating in Cornerstone (ASO3 screen)	3	0	7 - N/A
Includes sufficient evidence for the rating in the documentation (e.g., captures the reason for the rating)	1	0	9 - N/A

Appendix V

SSIP Leadership Teams - Benchmarks of Quality

The Benchmarks of Quality is used by a collaborative Leadership Team (LT) to assess progress and plan future actions to support the implementation of the IL SSIP within each CFC. The Benchmarks are grounded in the science of implementation, which bridges the gap between an evidence-based practice (EBP) and the actual high-fidelity implementation. Activities related to sustaining the effort are embedded throughout the process rather than being left until later. Benchmarks of Quality is a self-assessment tool that can be completed by the Leadership Teams as a whole or in small groups with the results from each group compiled into one consensus document to ensure all LT members are in agreement. The LT should use the data for planning future work and tracking progress.

Team members participate in annual meetings with other leadership teams to share successes and challenges. Team members engage providers and other

• EI team members in learning activities and events.

Elements	Indicators	Not in Place 0	Partially in Place 1	In Place 2
Leadership Team	1.  Team has 5 to 8 members with majority of skills needed (see SSIP Leadership Teams document), decision making individuals, and a designated T/TA state individual. 2.  Team has facilitator, written purpose, decision making process, and a method to handle changing members (membership succession). 3.  Members of the team participate in self-assessment activities.  4.  Majority of members (80%) attends monthly meetings and written notes are recorded summarizing meeting proceedings. 5.  Team completes benchmark of quality tool annually and celebrates accomplishments.
Action Plan	8. Leadership Team develops an action plan that guides the work of the Leadership Team and includes short and long-term objectives related to these benchmarks. 9. As a part of each meeting, Leadership Team reviews the action plan and assesses progress of each component.
Buy-In	10. Leadership Team regularly communicates with CFC staff so they are aware of and supportive of the work. 11. Leadership Team regularly communicates with EI providers so they are aware of and supportive of the work.  12. CFC staff and EI providers' input and feedback are obtained throughout the process. Leadership Team provides updates on the process and data on the outcomes to program staff and EI providers on a regular basis.
Family Engagement	13. The Leadership Team has parent representation. 14. To maximize family engagement, the Leadership Team has multiple mechanisms for communicating with families about its work and invites them to all learning events.
Communication	15. Leadership Team (via the Communication Liaison or other designated Leadership Team member - see SSIP Leadership Teams document) provides updates on the action plan (process, data, outcomes) to statewide leadership partners on a regular basis. 16. Dissemination strategies are identified and implemented to ensure that local community stakeholders and providers are kept aware of activities and accomplishments (e.g., website, newsletter, community events). The dissemination is done via the Communication Liaison or other designated member.
Professional Development and Ongoing Technical Assistance	17. Leadership Team members attend training events including Train the Trainer sessions. 18. Leadership Team identifies training and technical assistance, capacity within its members and areas needing additional support. 19. Leadership Team engages in learning events to support EI teams using adult learning principles, including but not limited to reflective supervision, coaching and mentoring strategies. 20. A process is in place to ensure the implementation of EBP and to support EI teams, with State partners, as challenges arise. 21. A process is in place to ensure the Child Outcomes Process is done with fidelity and reliability and to support EI teams as challenges arise. (Resource packages will be available to support the process.) 19. Leadership Team engages in learning events to support EI teams using adult learning principles, including but not limited to reflective supervision, coaching and mentoring strategies. 20. A process is in place to ensure the implementation of EBP and to support EI teams, with State partners, as challenges arise. 21. A process is in place to ensure the Child Outcomes Process is done with fidelity and reliability and to support EI teams as challenges arise. (Resource packages will be available to support the process.)
Data-Based Decision Making	22. Training, materials, and support are available to Leadership Team members on what data to collect, why, and how to use the data for making decisions for improving outcomes for children, providers, programs, and communities as well as how to submit the data. 23. A process is in place for the Leadership Team to collect, enter and summarize needed data as well as training on how to use the data for system improvement. 24. A process is in place for the Leadership Team to access the data or summaries of the data. The Leadership Team uses these data as part of their action plan for regular evaluation and continuous improvement.

If you require an alternate version of this report, please contact Jennifer Grissom at jennifer.grissom@illinois.gov.