FIPSE eWriting: ESL Writing Success
A project of
ESL/Foreign Language Department
Developed under a grant from the Fund for the Improvement of Postsecondary Education (FIPSE)
Report prepared by:
Barry Greenberg, Ph.D.
Feedback Technologies, Inc.
June 22, 2004
I. Introduction:
In September
2002,
Last year, following a six-month period of project roll-out, an initial evaluation was prepared. This document represents the year two evaluation of the project.
II. Purpose of the evaluation:
Several of the initial questions examined in the first evaluation will be considered along with additional issues relating to specific project accomplishments. Specifically, the following issues are explored: 1) Consideration of the extent to which intended strategies have been implemented and objectives are in process of being attained; 2) Difficulties experienced in the implementation of the strategies; 3) Factors which may have facilitated or inhibited progress in attaining project goals; and 4) Future evaluation plans.
III. Evaluation Findings:
1.
Implementation of intended strategies and
accomplishment of objectives
It is clear that a great deal of
activity has occurred with respect to each of the six year 2 objectives of the
project. An overview of the status of
each objective appears below.
Objective 1: Identify and train faculty teams for year 2 curriculum development
Using the results of an online survey conducted with year
1 writers, the grant team systematically gathered ideas and comments for the second year of operation. They used these ideas to create tools to facilitate the work of the faculty who would be retained for the project. As the management team noted, “One of the main outcomes came from the realization that the final product – an online LO - is a result of a step-by-step, labor-intensive process.” Thus not only was this objective achieved through the acquisition of the required numbers of faculty writing teams, but faculty efforts were to be supported by detailed instructions, timelines and clearly-stated requirements regarding the product they would be submitting. This change from year 1 strategies to year 2 approaches was very much appreciated by the writing teams. During a focus group moderated by the evaluator, writers were in agreement that their project efforts during year 2 were much more effective than they were in year 1. In the first year of the project, the participants reported a great deal of uncertainty. They report everything was new to them and the necessary feedback was not always provided. Now, they feel that things are much better. There is much more needed structure and tremendous support in all aspects of the job. The technical support is considered ‘great’.
A writer noted:
“The first year writing I found terribly difficult…
Second year I knew exactly what I wanted.”
Objective 2: Complete the content development for the six
levels
The project management team reports, “The eWriting project Year 2 was bustling with activities and innovations, and a significant part of the work was channeled towards completion of the content for the six levels and facilitating the work of the writers’ teams”. To accomplish this objective required training of the selected faculty writers and providing sufficient incentives to insure their active participation. A series of four workshops were conducted during the Fall of 2003 providing extensive coverage of LO development and enhancement. The project management team devised a system of released time for faculty as compensation for the considerable work required to complete the project. They worked closely with faculty during the year to provide whatever support was needed to complete these tasks. During the focus group conducted with the writers, they reported that not enough time had been set aside for them to complete the tasks. Participants estimated that it took them about 14 hours to write each LO, or about twice the amount of time estimated. Despite this, they enjoyed their efforts very much and, for the most part, would like to stay with the project:
In this regard, one participant noted:
“I would work in this project again; however, I would need to be given more time. We got one class off last semester and I felt that my classes actually suffered. I enjoyed writing the LOs so much that I spent more time writing the LOs than I did prepping and doing student stuff and I feel really bad for that. I would like to get more time if I were to do it again.”
The result of these efforts is that the level 1 LO’s are about
75% complete. The level 2 writers’ team met the deadlines and completed 100% of the curriculum, and LOs for levels 3, 4, 5 and 6 have all been completed. The very challenging job of creating the curriculum in LO format is almost complete.
Objective 3: Pilot, Revise and Edit the Content for the Six Levels
Classroom piloting of written materials is a key component in the development of materials which will be suitable for the target population. The piloting has been used to determine suitability of each LO’s content, length, clarity of materials, interest level and response of students. The evaluator supported this effort with a) a series of interviews of students using LOs and b) an assessment of pretest to posttest gains by students using LOs in their classrooms.
Personal interviews were conducted in one of the classrooms where LOs were in use. Students were asked about their reactions to using LOs including whether they were easy to use, fun to use, difficult to use. They were asked about specific activities within the LOs including whether they were explained sufficiently, easy to understand and whether enough time was available to complete them. They were asked their feelings toward the pretests and posttests. They were asked to compare learning with LOs with learning in a typical class, and finally they were asked for ideas on improving them. Highlights of findings include the following:
1. Overall, they enjoy using LOs. They find them interesting and not boring. They like the idea of knowing in advance what the professor is going to lecture about. Some find them more difficult because of their language problems. For them, the class is too fast.
2. They feel the activities help them to learn. They like the practice. For some, the examples are reported as not helpful because they are easier than the actual questions (on the posttest). They feel the activities are explained sufficiently.
3. Some students prefer the LO approach as they find everything explained in the materials and do not need to ask the teacher for help. For other students, the need is for both the teacher and the book.
4. Overall, students feel the LOs are a good way to learn.
Pre-and post performance of students were measured to assess extent of entry knowledge and amount learned. Thus far 50 LO’s have been piloted and the process is continuing on schedule.
More in-depth consideration of the results of the pretest-posttest
results reveals that the amount of gain has not always been substantial. This has usually been the case when the
scores on the pretest were higher than expected (greater than 60% correct on
the pretest leaves less room for growth on the posttest). Thus, even though more than 90% of students
generally report they “strongly agree” or “agree” that the activities in the LO
helped them to learn more about the topic, it has not always been the case that
this was reflected in pretest-posttest differences. Based on a classroom survey conducted by the
evaluator, it was determined that the lack of pre to post gain was due to
changes in pretest answers made by students as they worked at home and
discovered incorrect answers they had placed on their original exams. These students thought that scores on the
pre- and posttests would count toward their grade. This situation was clarified for the students.
The following table summarizes the
results across levels 2, 3 4, and 6:
|
|
N |
Pre-to Post Gains |
|
Level 2 |
185 |
23.5% |
|
Level 3 |
354 |
6.2% |
|
Level 4 |
17 |
47.6% |
|
Level 6 |
21 |
9.9% |
As a result of the low pre- to posttest gains, a special workshop was
held with project faculty to review the importance of pre and posttest
measurement and to review techniques of item construction.
Pre to post differences will be routinely monitored in year 3 of the
project as LOs go on-line and test data are routinely collected and made
available for analysis. During the
coming year, it will continue to be necessary to make determinations of how
much learning it is reasonable to expect will occur and be documented, via
pre-to-post analyses. In the new project
year it is expected that this data will lead to changes in the LOs as well as
changes in the measures themselves.
Objective 4: Prepare
the Programming Phase of the Project
The required combination of programming personnel, special software and added hardware support along with project personnel have been put together to accomplish this objective. Narrated Power Point (with Impatica) supplemented by Camtasia movies with WEB CT in the background appears to have met staff requirements for tools which are powerful, yet easy to use and suitable for generating exciting materials. Extensive training in all required materials has been presented. In the focus group, the writers expressed high levels of satisfaction with this phase of the project as they see their LOs take on a new format in preparation for use online.
Objective 5: Launch the programming of LOs (ongoing)
The team described above has completed significant portions of 28% of the project content including: 1)
level 1 – six LOs 2) Level 2 – 45 LOs (100%) 3) Level 3 – 34 LOs 4) level 6 –
20 LOs. What remains to be done are the
pretests and posttests which are in process, and the
An extensive amount of dissemination has already occurred with articles
published in academic periodicals, materials presented in newsletters and
available at websites.
2. Difficulties experienced in the
implementation of the strategies
The project management staff, with its clear sense of purpose and
commitment to project goals, maintains a heavy workload along with high
expectations of all involved in this effort.
During the focus group, several of the writers felt that they were not
always sure about what was expected of them and that more direction would have
helped them to get their job done. For
this group, the late start of the project meant they did not have sufficient
opportunity to ever catch up. They also
felt they had substantially underestimated the amount of time required for the
project. The management team put
together a series of workshops on all the programming tools that would be
available when presenting the LOs in an online format. Some of the writers were overwhelmed by the
presentation and did not understand exactly what the expectations of them would
be in regard to the programming tools.
Clarification of role expectations has taken some time, but is occurring
as events unfold.
Despite these problems, all of the writers maintain a very high level of
commitment to the project and a sense of excitement about the next phase when
their LOs go live, online. They report
that the problems they encountered have not prevented them from completing
their work and all of them are interested in continuing their efforts. In fact, they even report self-growth due to
project involvement.
“I would do it again because now I’m a better teacher and that’s a fact because I have these lessons that I had to sift through. How many times before you do a class do you really sift through and think about each thing that you’re going to say and that’s so much on my mind because I also typed it. I also did all the research. .. When I get up in front of the class now, it’s like this (snaps fingers…) and for me that’s been the best…and I think I was a good teacher before too…now I’m better…the process has made me better…”
3) Factors
which may have facilitated or inhibited progress in attaining project goals:
It is clear that the knowledge
base and work ethic of the project
management team is responsible for keeping this project on track and on
schedule. The team maintains high levels
of contact with each person involved in the project and strives to understand
and resolve any difficulties they are having.
Those involved with the project have a sense they are involved in
something special and that feeling is attributable to the efforts of the
management team. This is a very
extensive, time-consuming effort with broad goals and many diverse people
involved.
The pieces of the project are interlocking, in that what is done in phase
I must fit to phase II and so forth. The management team has had to think ahead to
anticipate what will be required, both in the short run and the long run. On occasion they may have given too much
information too soon, or too little information to those in need of greater
clarification. However, the team has
kept its focus and always looked for ways to improve. Their efforts have been rewarded with a
project that is on schedule and shows every sign of achieving its ultimate
objective.
IV. Future evaluation plans:
In the coming year, the LOs will be programmed and placed online for
actual student use. This will be done in
phases and it is anticipated that as data are collected and analyzed, changes
in future LOs will be made. The following data will be collected, analyzed and
presented:
5.
student
attitudes toward the various LO activities
6.
time to
complete each section of the LO
7.
pretest
and posttest results