Introduction

This report presents the evaluation findings for the third full year of this project’s operations. This evaluation has three components: (1) an evaluation of process, or the extent to which the work of the project has proceeded as expressed in the work plan; (2) an evaluation of outcomes, or the extent to which anticipated milestone events and results and products have emerged as the consequence of the work of project staff and their leaders and various collaborators and (3) the lessons learned during the project.

Evaluation Findings

It should be noted here that data sources for much of the first two components of this report identified above have been created and administered by project staff for their own internal uses in tracking the progress of their work and assuring that critical tasks are carried out according to schedule. These tools routinely inform and guide decisions and actions in every respect. The diligence and managerial expertise with which this project has been conducted during this 12 month period is clearly evident and facilitates an accurate and complete evaluation of the project team’s work.

Process Findings

The major year three activities are presented in detail within the following matrix labeled Year Three Activities. This project task matrix was created at the outset, has guided discussions at the meetings of the project team and is updated periodically by the project director to reflect current status and planned activities. The project has completed or exceeded all its original tasks on time, within budget and in good order. The progress attained in each substantive area of work has been described in some detail within each of the Project Director’s semi-annual progress reports.

Outcomes Findings

Year three was primarily dedicated to disseminating information about the project, curriculum and findings from the promising practices polls through conference presentations with pilot site representatives, publications and hosting a Training Roundtable. Producing, disseminating and introducing this curriculum is the essence of this three-year project. Year one was dedicated to developing competencies and a draft curriculum that could be piloted. Year two was primarily dedicated to field-testing, evaluating and revising the curriculum, with the ultimate goal of producing a final curriculum for wide dissemination. Year three activities focused on creating useful versions of the curriculum, generating interest within the child welfare community about using the curriculum and training trainers on the curriculum.

All outcomes and products scheduled for year three were completed in good order and, as described below, often went beyond original expectations. Additional information about each outcome and product can be found on the project website and within each of the Project Director’s semi-annual progress reports.

Dissemination Approach

The dissemination approach relied on a variety of mechanisms to get the word about this curriculum out to the child welfare community—teleconferences, mailings, newsletters, publishing various versions of the curriculum and a web site, which contains the complete curriculum, state by state responses to the ASFA promising practices phone polls, the phone poll reports and information about the project work plan.

Perhaps the most valuable dissemination activity involved the continued participation of representatives from the pilot agencies in the project (the Department of Community Based Services, Kentucky, the Department of Children, Youth and Families, New Mexico, Cuyahoga County Children and Family Services, Cleveland, Ohio and the Division of Children and Family Services and the Training Partnerships, Wisconsin). Because child welfare agency personnel tend to learn best from other child welfare agency personnel, the dissemination approach used by the project team relied heavily on the involvement, support and participation of representatives from the agencies that piloted the curriculum in year two of the project. Each of the four pilot sites volunteered to remain active in the project and to spread the message about the value of this curriculum to their agency. The project director offered each pilot site the opportunity to co-present at a major child welfare conference on their experiences with the project, adaptation of the curriculum and impact of the training. During this year, the project director and pilot site reps presented at five major, national child welfare conferences. This peer to peer presentation approach was an effective strategy for communicating the benefits of using the curriculum and for generating interest in participating in another major component of this year’s work plan—the training of trainers.

Training of Trainers … and More

The original proposal for this grant stated that in year three of the project, the team would sponsor and host a train the trainer conference in Maine on the Bringing Together the Child Welfare Team curriculum. The team intended to invite child welfare trainers, supervisors, managers and data analysts from ten states to attend this train the trainer program. The plans expanded however when the project director received feedback from the year two pilot sites and participants in presentations at national child welfare conferences that the child welfare training community needed a forum to understand and discuss training issues raised by the Child and Family Services Reviews. To address that need, the project team expanded the audience and goals of the originally planned training of trainers and, in collaboration with the National Child Welfare Resource Center for Organizational Improvement (NCWRCOI) at the Institute for Child and Family Services, partnered to organize and host a Training Roundtable.

The aim of this Training Roundtable was to identify best practices and criteria for high performance training systems that support achievement of organizational mission, goals, and objectives. The Roundtable also accomplished the original objective by featuring an in-depth walkthrough of the Bringing Together the Child Welfare Team curriculum by the project staff and pilot site representatives.

On June 22 - 24, 2003, the Muskie project team and the National Child Welfare Resource Center for Organizational Improvement hosted a Training Roundtable. Fifty seven child welfare administrators, trainers and training and program managers from around the country, representing sixteen states, five counties, NYC, nine universities and several provider agencies, attended the Roundtable. These child welfare experts gathered to identify best practices and criteria for high performance training systems that support achievement of organizational mission, goals and objectives. The Training Roundtable also featured a presentation by Jerry Milner on the training implications and promising training practices identified during the Child and Family Services Reviews and an in-depth walkthrough of the Bringing Together the Child Welfare Team curriculum developed as part of this project. During the curriculum walkthrough, the four child welfare agencies that piloted the curriculum discussed how they adapted the curriculum to meet their needs, the benefit of participating as a pilot and how the training has impacted participants, supported their ASFA implementation and guided their future training.

Feedback from participants in the Roundtable indicated that they felt recharged, had received enough information to revamp and deliver the curriculum and have a better understanding of strategies to help them positively participate in their PIP. In fact in late 2003 and early 2004, two states (CA and CO) that attended the Roundtable launched their versions of the Bringing Together the Child Welfare Team training, with another state (VT) well into the curriculum design process.

Versions of the Curriculum

After publishing the Bringing Together the Child Welfare Team curriculum in September 2002, the project team received feedback that the curriculum book was ‘too big’ and bulky to be easily accessible—it is more than 150 pages! Taking that feedback to heart, the project team developed and published a set of short, bound, topic specific booklets based on the content of the curriculum. Like the Bringing Together the Child Welfare Team curriculum, this set of trainer’s guides is designed to communicate information on the Adoption and Safe Families Act (ASFA) that goes beyond introductory, compliance based topics. While these booklets are designed primarily for training purposes, each certainly is adaptable to other forums, such as internal or external workshops, presentations, newsletters or briefings on ASFA and could be successfully presented to child welfare administrators, supervisors, managers, foster parents, caseworkers, providers, teachers and other community stakeholders.

The trainer’s guides in this series are:

• The Adoption and Safe Families Act (ASFA) and the Child and Family Services Reviews (CFSR): Using Outcomes to Achieve Results
  This trainer’s guide highlights the major requirements of ASFA, presents federal outcomes and measures and systemic factors and provides an opportunity to discuss the philosophy, practice implications and results of the CFSR.
  
  
• Action Planning: A Problem Solving Tool
  This trainer’s guide introduces and demonstrates how the use of Action Planning can assist child welfare managers and supervisors in planning, managing and evaluating practice, systems and programs toward the achievement of desired outcomes.
  
  
• Collaboration with Native American Tribes: ICWA and ASFA
  In the child welfare system Native American children have different service delivery systems as well as laws that apply to them. Therefore, individuals must ask different questions and make different assumptions in their efforts to identify and work with Native American children and families. Because of the importance of the interaction between the agency and tribes, this trainer’s guide focuses on successful approaches to collaboration, the requirements of the Indian Child Welfare Act (ICWA) and ASFA and the practice considerations when working with Native American children and families.
  
  
• Using Data to Measure Success
  Child welfare managers and supervisors are increasingly expected to be able to use data, information and reports to guide decision making and to determine what is working and what isn't working in the organization, with practice and in the service delivery system. This trainer’s guide gives participants practice in analyzing reports and in using basic data tools for reading and interpreting data.
  
  
• Change is all Around Us: Tools to Build Commitment to Change
  In most organizations change occurs constantly. In order to be effective in leading and modeling change management skills, supervisors and managers must understand the dynamics of avoiding resistance to change and how to build commitment to it. This trainer’s guide includes a model for building commitment to change, provides an opportunity to build on these skills and includes use of a case study, Family Net: An Automated Child Welfare Information System which explores organizational and managerial issues when a major change in the workplace takes place.

Lessons Learned

Throughout the project period, the project team has had the opportunity to explore with the child welfare community training approaches, issues and concerns. Information about child welfare training emerged during the three annual phone poll of child welfare agencies conducted by the Muskie project team, at the training Roundtable (June 2003) sponsored by the Muskie School and during project related teleconferences with child welfare trainers, educators and managers. From these conversations and research, we have identified several lessons learned about the current field of child welfare training.

1. The Building the Child Welfare Team: Results of the 2003 Adoption and Safe Families Act Phone Poll report, published by the Muskie project team in September 2003, identified how the Adoption and Safe Families Act (ASFA) has changed the way that child welfare agencies do business. While many ‘promising administrative practices’ emerged from the 2003 poll data analysis, three findings that impact training stand out:
   
   
   • Agencies view the Adoption and Safe Families Act (ASFA) from a ‘good practice’ perspective. Both the 2002 and 2003 ASFA phone poll analyses indicate that child welfare agencies continue to actively implement the provisions of the Adoption and Safe Families Act (ASFA) and to feel the impact of the Child and Family Services Review (CFSR). Agencies tend to view ASFA requirements as ‘good practice’ and are taking steps, such as joint training, starting interagency workgroups, communicating the shared responsibility for meeting outcomes for children in care and setting up informational websites, to implement this approach throughout the agency and the child protective network. One respondent’s comment sums up what many others stated ‘…case planning and caseworker contact appear to be key to positive outcomes in implementing requirements of both ASFA and CFSR’.
     
     
   • Agencies continue to deliver ASFA training and to integrate ASFA related topics into on-going training, meetings and forums. Of the 37 child welfare agencies surveyed in 2003, 33 (89%) reported that their agency continued to provide ASFA training to their staff during the last year. The most frequently mentioned training topics were ASFA requirements, time frames for reunification, concurrent planning and permanency issues. Agency training unit staff continue to be mentioned most frequently (n =25 or 68% of states responding) as delivering the training. The percent of respondents reporting that ASFA training was incorporated into other training increased from 59% in 2002 to 76% in 2003. There was also an increase in the percent reporting ASFA training was incorporated into new worker training, from 51% in 2002 to 65% in 2003. 24 agencies (65%) continue to do ASFA related training with or for the courts on topics such as permanency issues, legal issues, ASFA requirements, timeframes for reunification and reasonable efforts. Fewer agencies, however, report doing ASFA related training with other partners in the child welfare network; agencies report doing ASFA related training for tribes (27%), mental health (19%) and health care (14%) providers, community stakeholders (11%) and schools (8%).
     
     
   • The managerial and supervisory skills needed to implement ASFA continue to change while the workers’ needed skill set remains steady. Interestingly in the 2001 poll, ‘casework’ was the key skill respondents said was needed by managers, supervisors and workers to implement ASFA. The results of the 2002 poll show that while ‘casework’ remains the 1st ranked skill for workers, ‘understanding the requirements of ASFA’ is ranked number one for managers and ‘communication’ is ranked first for supervisors. In the 2003 phone poll, ‘using data effectively’ was number one for managers, ‘collaboration’ and ‘understanding how to implement ASFA’ tied for 1st for supervisors and ‘casework’ remained number one for workers.

ASFA mandates regarding the achievement of outcomes and the use of data carry clear expectations for child welfare workers, supervisors and managers, as well as change the way that child welfare agencies need to be managed. The 2003 phone poll clearly indicates that continued attention is needed to crafting training programs that enhance the managerial and supervisory skills of using data to inform decision-making, improve practice and measure performance. Additionally, it is critical to build and support training systems that result in agency staff and key partners throughout the child protective network having an increased understanding of how to deliver effective, outcomes-based child welfare practice.

2. The 2003 phone poll report also indicates that agencies need to do more work to assure that their information system produces information and reports that support and inform managerial and supervisory day to day decision-making. On a scale of 1-5, with 1 being ‘poor’ and 5 being ‘outstanding’, the average rating was 3.14 in 2001, rose to 3.43 in 2002, and to 3.51 in 2003. As one agency explained “…as additional management reports continue to roll out, our SACWIS will become more and more useful. The data is in there; we’re just now figuring out how to get it out in a meaningful way.” At the same time, our experience working with child welfare agencies to adapt the Bringing Together the Child Welfare Team curriculum has shown us over and over again that to help child welfare supervisors become comfortable with information management skills:
   
   
   • Buy-in needs continual and planful assistance from managers
     
     
   • Data reports need to be easily available , timely and readable
     
     
   • The data needs to be meaningful (team, unit, area)
     
     
3. The value of training in a child welfare agency cannot be overstated. The first round of the Child & Family Services Review only served to heighten the importance of training in advancing the organizational and practice improvements that agencies are undertaking in order to achieve better outcomes for children and their families. In drawing lessons from the Review, one observer noted that “to change systems, we must change practice”. (Jerry Milner) Another noted that “child welfare agencies must translate the principles they adopt as their mission into the day to day child welfare practices in the field and set up a comprehensive training system within the state that assures that the workers (at every level of the agency) have the values, attitudes, and skills and are supported in implementing those practices”. (Elena Cohen)

There is a great deal of excellent work being done in the area of evaluating training curricula, designing curricula and establishing human services staff development and training programs. However, more work needs to be focused on the training system itself, specifically on the extent to which the training system contains components necessary to have a positive impact on children’s safety, well-being and permanency and its relationship with other systems that support that work.