Miguel Kagan

To cite this article: Kagan, M. Kagan Implementation Surveys. San Clemente, CA: Kagan Publishing. Kagan Online Magazine, Summer 2008. www.KaganOnline.com

Kagan workshops are terrific. They engage. They educate. They inspire. But that's not what really matters.

What really matters is not what happens at Kagan workshops. What matters is what happens after workshops. The whole point of Kagan workshops is to empower teachers with cooperative and engaging instructional strategies that can be easily implemented to make a positive difference for students. Sure, we can put on a world-class event. We can create the experiences so the knowledge is in the minds of teachers. We can provide a toolbox of tools in the hands of teachers. But, none of that matters if teachers don't apply what they learn to improve classroom practices.

So how do we measure whether the Kagan Structures and methods are being used and if they are really making a difference in schools? One way is to measure achievement gains. Many past issues have featured gains in the classroom, for schools, and for entire districts. For more information on achievement gains, see the research section of the Kagan Website: www.kaganonline.com/Research. While test scores remain the du jour barometer of professional development and school success, surveys can shine an additional spotlight on what is working and what we need to work on.

What Are Implementation Surveys?
The basic idea is to survey teachers about their implementation of the structures and strategies they learned: Are you using what you learned? Is it having a positive effect on instruction? Do you like it? Do students like it? After teachers respond, the results are tabulated and analyzed.

Who Administers the Survey?
Some schools or districts have a dedicated staff member in charge of data collection, and have their own survey format they perform in conjunction with their professional development efforts. Other times, when Kagan works closely with a school or district, Kagan administers an implementation survey as part of the long-term improvement plan.

When Are Surveys Administered?
Ideally, an implementation survey is given more than once. The first time is a few weeks after professional development. This offers insight to immediate implementation. Did teachers find what they learned helpful? Are they able to apply it right away?

The next logical point to administer the survey is several months later. Is there sustained implementation or has implementation waned? Do teachers need a refresher or are they ready to take the next step? Implementation is best sustained through ongoing support. The support can take the form of Kagan Coaching, certified Kagan school or district Trainers, or follow-up trainings.

Finally, an implementation survey may be administered a year or more after the initial professional development. At this point, we can use a survey like a check-up. Does everything look healthy? Are there areas for improvement? Have we structured for longevity?

How Are Surveys Administered?
For the implementation surveys that Kagan administers, Dr. Jacqueline Minor, Kagan's Director of Curriculum and Instruction, sets up a web-based survey. She e-mails the school or district contact person a link. The contact person then distributes the link to teachers and is responsible for ensuring that all teachers fill out the survey within a given timeframe. At their leisure, teachers log onto the web-based survey and fill it out. Usually the survey takes no more than 10 minutes. Paper-and-pencil surveys have also been administered. The advantage of the paper-and-pencil survey is we can collect all the surveys at once. The disadvantage is they require more data entry and manual calculations than web-based surveys.

Analyzing Results
The two most common results we look for are percents and ratings. For percents, we compare what percent of teachers feel this way versus that way. For example, what percent of teachers feel the training helped them improve instruction: a) a great deal, b) somewhat, or c) not at all?

Ratings are usually on a 1 to 5 scale from (1) Strongly Disagree to (5) Strongly Agree. For example, the statement may be: "My students are actively engaged." A rating of 1-2 shows disagreement, 3 suggests indifference or mixed responses, and 4–5 indicates agreement.

Oftentimes, in surveys we provide space for teacher comments, as they are often insightful. For example, a teacher comments: "I am better able to engage those hard-to-reach students." Or, "Great so far, but I feel I need more training."

What Do Implementation Surveys Tell Us?

Effect on Instruction
The results of the Kagan Implementation Surveys have been very positive. During a five-year implementation plan in Florida, teachers from kindergarten to high school were asked what effect did Kagan Cooperative Learning training have on instruction. Fifty percent responded that their instruction improved a great deal, 37% reported their instruction improved somewhat, 2% stated their instruction hadn't improved, and 11% did not respond. When we summarize perceived improvement for all respondents, the ratio is 87 to 2: 87% reported a positive impact versus only 2% reported no impact.

Results of Using Kagan Structures
In addition to overall results on instruction, we can get more specific. The table below summarizes surveys administered in four schools in Louisiana. Again, the results are very encouraging. Teachers agree or agree strongly that students are more actively engaged, are learning more academically, are developing teamwork skills, are improving communication skills, and teachers are more successful reaching some students. It is especially reassuring to see that no teachers disagree strongly with these statements, and a very low percent (from 0% to 3.9%) disagree. Interestingly, the question of reduced discipline resulted in a mixed result. To be sure, it is much more good than bad, but we'd like to see an even greater reduction of discipline problems. This suggests that while social and academic measures are improving, a more direct focus on social skills, and even a Win-Win Discipline training may be the next step for professional development and/or coaching.

Motivation and Liking
Survey questions are grouped by sections. The topic of this section is "motivation and liking." The questions enable us to examine the different facets of motivation and liking: Do students like doing Structures? Do they like class more? Do they like school more? Do teachers feel teaching is easier? Do teachers feel it is easier to be successful?

Early research on cooperative learning has found that students enjoy class and school more. Our surveys corroborate that perception among teachers. Take a look at the survey results from Louisiana below. What is terrific to see here is that no (0%) teachers disagreed in any form with the statements: Students enjoy doing the structures, students enjoy class more, and students enjoy school more, compared with 78% who agreed or strongly agreed. From this survey, we can conclude that teachers in these schools feel the Kagan professional development has positively influenced motivation and liking.

Classroom Arrangement
Another focus of the survey is classroom seating patterns. Are students in the traditional row arrangement or are they in teams? Obviously, sitting in teams is conducive to Kagan Structures, teamwork, and active engagement, while sitting in rows is not. Teachers with their class in rows most likely aren't doing much cooperative learning, but if they are in teams after training, chances are high that there is active engagement. The following are results from over two hundred teachers polled in New Jersey. The chart shows that more than 93% of respondents have students either sitting in their teams full time or they are usually in teams. We take this as a very good sign.

Demographic Information
Surveys are very helpful for collecting demographic information, especially when we survey a district with teachers at different grade levels and different levels of experience. You can get a good feel for the respondents at a glance.This chart indicates teachers' experience implementing Kagan Structures.

Conclusions
Implementation surveys allow us to collect and analyze teachers' perceptions on a number of areas pertaining to professional development. Notice the italics on the word, "perceptions." We must remember that surveys do not necessarily capture reality. They are teachers' perception of reality. But in meeting with administrators, we have found a high correlation between the survey results, administrators' reports, and hard measures of success. For example, teachers believe implementation of Kagan Structures is having a positive effect on student learning, and test scores back up these perceptions. So while surveys are inherently imperfect, they appear to be a good representation of reality. Further, classic studies on educational expectations and beliefs (think Pygmalion) demonstrate that teacher expectancies can have a profoundly positive (or negative) effect. If teachers think that implementing the engaging and cooperative instructional strategies that they learn at Kagan workshops result in better instruction, more active engagement, better academic achievement, and increased motivation, not only are they accurately describing actuality, but these positive expectations become a self-fulfilling prophecy.