Evaluating An Innovative Elementary ESL Program

Alan Seaman evaluates the differences between "pod" and "pull-out" programs organised for young language learners.

R ecently I attended a meeting where the principal of an elementary school in the Chicago metropolitan area spoke with parents of new
students. "This is a school with an international focus--the world comes to us," he proudly claimed as he described his innovative English as a second language and bilingual education programs. "We have programs which lead to academic success."

In a community where immigration is a controversial issue, his words were gratifying. But were they accurate?

The ESL and BE programs at this school are part of an experiment by a school system which is looking for alternatives to the ESL pull-out programs it has used for decades. As a member of the ESL task force for the district, my role as a researcher is to develop a systematic way to evaluate these innovative programs. Although the new programs at two elementary schools in the district seem to be working, are they truly effective? And if they are more effective than the traditional pull-out programs, how can we communicate this good news with administrators, school board members, and citizens in the community?

There are a number of ways to evaluate a new educational program, including test scores, interviews, and surveys. In addition to data such as test scores, our task force decided to conduct a systematic observational study of the ESL students in the context of their classrooms. The results of this evaluation study were both quantitative (presented as numbers) and qualitative (presented as case study vignettes). In assessing the quality of the experimental programs, we wanted to help our stakeholders see what was actually taking place in the schools__to have a window into the daily experiences of the English language learners.

The Experimental "Pod" Model
In response to a complex immigrant population (involving a number of refugees as well as a broad range of immigrant groups), this suburban school district has offered pull-out ESL and bilingual education programs in several elementary schools for over twenty years. These traditional programs usually involved one or two teachers at each school working with the students in 30-minute segments. During the remainder of the day, the students were mainstreamed in classes at their grade levels. As with many similar pull-out programs, the case load of the ESL teachers was heavy, and their facilities were small.

During the past two decades, the pathbreaking research of Cummins (1981, 1984) and Collier and Thomas (1997, 1999a,b) has provided school districts with insights into the different program models for ESL and bilingual education. Their findings support the movement away from the pull-out model toward self-contained ESL and bilingual classrooms with an enrichment emphasis.

Searching for alternatives to the pull-out model, in 1998 the school district formed a task force which decided to create an experimental ESL/bilingual program at two elementary schools. Currently, this "pod" model involves self-contained ESL classes at the K, 1, 2-3, and 4-5 levels. Each class is taught collaboratively by two teachers - an ESL or BE teacher and a grade-level teacher - and includes a full-time instructional aide.

In addition to their time in the ESL class, the students spend part of their day in a "buddy class" at their grade level. For instance, all of the 4th graders in the pod ESL class will leave their room for an hour with a teacher in order to participate in a science segment of another class containing students who are proficient in English. The buddy concept allows the ESL students to gain appropriate content-based instruction and provides contact with the other students in the school.

The advantages of the pod concept
are fairly obvious. These classes feature a low student-to-teacher ratio, instruction throughout the day tailored to the needs of the ESL/BE students, a strong emphasis on literacy and content-based instruction, and the opportunity for ESL students to interact with peers who are proficient in English. The primary disadvantages of this model are related to high personnel costs and the fact that some of the students were bussed away from their neighborhood schools to the pod schools.

The Evaluation Study
After the experimental pod programs were up and running in the two schools, the key question was whether or not the pod model should be expanded to all of the elementary schools in the district. Did the quality of this program justify its increased cost?

Using a matched-pairs design, we selected students at each grade level from the pod program and matched them with very similar students in the pull-out programs according to grade level, English proficiency, time in the U.S., and ethnic and educational background. Each pair of students was systematically observed over a four-week period by well-trained graduate students using standardized forms to record data. The observation data included narratives of each student’s classroom experiences, and specific quantitative data related to each student’s engagement in class activities (also known as "time on task"), teacher-student interaction, and peer interaction.
In addition to recording a cross-section of each student’s daily educational experiences, the observers also interviewed the teachers to obtain additional background information and they wrote detailed analyses of each pair of students. Data were collected during April 1999 and September-October 1999 for eighteen pairs - a total of 36 students.

The Results of the Observational Study
The results of this study were compelling. A statistical analysis (a paired sample t-test) showed that, overall, the students in the pod programs had significantly higher levels of engagement in classroom activities and significantly more frequent interaction with teachers than their peers in the pull-out programs had. Each of the students in the pod programs was "on task" more than 90 percent of the time__a level that less than half of the pull-out ESL students achieved. Most disturbingly, several students in the pull-out programs were engaged in classroom activities only 55-70 percent of the time. These students were missing a large amount of instruction time each day.

In a society where quantitative research is valued, these results, presented in tables, can be persuasive with parents, administrators, and school board members. In communicating the results, however, we also used qualitative data - detailed descriptions of each student - to provide a window into what was happening in the classrooms.

In one vignette, for example, a second grader from Russia in the pod program is "relaxed and focused as he works with other children in structured groups." The observer describes "Alexander" meeting with a reading group of six other students at his level of English proficiency. They work with the teacher on a book entitled My Garden, with Alexander reading a section of the book aloud to the others and then explaining it. The teacher reinforces the book with vocabulary cards before Alexander pairs up with a third-grader who is his "buddy reader" to work on a different book. At the end of this segment of the class, Alexander moves to one of several computers in the classroom, where he begins to type out a writing exercise.

The sense of relaxed concentration in this classroom, as the students move smoothly from one focused activity to the next, contrasts with the experience of "Lars" in one of the schools with pull-out ESL. Like Alexander, Lars is a second-grader from Northern Europe who is at a beginning level of English proficiency. In the school’s ESL pull-out class, Lars is quiet and attentive. After an half hour of instruction there, Lars enters his grade-level class partway through a hands-on science activity related to donut production. The teacher directs Lars and two other ESL students to work on a computer in the back of the room while the other children complete the activity in groups.

In the next segment of class, when faced with a literacy-related worksheet, Lars struggles to understand the instructions, first staring at the paper, then looking around. After several minutes, the teacher circulates to his desk and reiterates the instructions. The observer notes Lars’ anxiety as he watches the other students work on an assignment he can’t comprehend.

One observer summed up the differences in the two programs in the following way: "The ESL students in the pilot pod program seemed to be able to go more in depth with their activities because they were all together, working at a similar pace, and had longer time periods to devote to certain activities. The feeling in the pull-out program was that the students had disruptive, choppy days where they were involved in two classrooms with different field-trips, activities, and teachers. The pod teacher stated that her classroom works at a slower pace so that her students can learn important social and linguistic rules. She focuses on more basic, hands-on types of activities as well. But this class did not seem any slower in covering the required curriculum."

Conclusion
At the time of writing, the school district is deliberating over what direction to take in the future in their elementary ESL and bilingual education programs. Clearly, the inequitable current situation needs to be addressed, with all students (rather than some) receiving access to the same resources. Part of the debate involves the willingness of the community to invest in the education of immigrant children. The current pod model is not the only option being considered. Can these pod classrooms be altered somewhat to become less expensive without sacrificing their proven quality as an educational program? And can they also be modified to have a stronger bilingual educational component throughout the day?
These issues will eventually be resolved. Whatever the outcome, the members of the task force, the school administration, and the community will have detailed, vivid data about the ESL and bilingual education programs in their schools. These data, and the insights generated from them, will be critical in making an informed decision for the future.