Chapter 5: Evaluation

In Chapter One some general questions were raised with regard to the viability of using the WWW within a given context. Chapters Two and Three outlined the practicalities and theory involved in this process and Chapter Four described the development and implementation of materials suitable for a particular context. What is now necessary is to analyse the solutions in chapter four in terms of the points put forward in the first three chapters. To do this effectively it is necessary to turn first to the question of how and what to evaluate in more detail and in the light of the material and ideas described in the previous chapters.

5.1 Introduction

Evaluation, though essential for the effective development of CALL materials, is all too frequently ignored in favour of marketing: programmes are sold and promoted as being reliable solutions to any range of language learnersí needs without any serious research into how effective the packages really are. This has been seen very clearly recently in Spain, with the growth and subsequent collapse of computer-based language academies which promoted up-to-date technology, providing flexibility of time-tabling and attractive new material, which in reality failed to live up to promotional claims.

Such a failure to fulfil initial expectations has also been seen in terms of the WWW in general: the brief history of the WWW has been plagued by an endless succession of claims and promises, many of which have proved false. From financial to educational promise, what is being discovered is that the WWW may bring a range of benefits and new ideas but it is only through time that the nature of its full potential and weaknesses will be discovered. Initial claims are all too often little more than hypothesis and often not well-grounded. What the WWW really offers will be discovered through research and investigation over time.

Within such a context, any development of WWW based material at ELI must be tested, evaluated and continuously developed to prove, as the academy always has, that its use of technology is not simply for promotional or marketing reasons but to innovate within sound pedagogical frameworks. Involving teachers and students in this testing process will only further promote the validity of the procedure, while also continuing training and involvement in the project. The key question is then, how and what to evaluate, particularly in a project which is so extensive and also relatively open in terms of results and expectations. The experiment at ELI involved six centres, thirty nine teachers and potentially more than two thousand students. Devising a method of evaluating how specific material is implemented within this context is very complex.

5.2 The Theory of Evaluation

Warschauer, Shetzer and Meloni (2000: 104-5) provide a simple synthesis of the major areas of debate surrounding educational research. One area is that of experimental versus natural research, with the former reflecting a traditional, more scientific method, keeping as many variables as constant as possible in very controlled experimental situations. Naturalistic research is more holistic and embraces a wide range of factors and areas and, in the context of ELI with a large number of teachers, students and a complex collection of web lessons and potential, a more natural approach will be easiest. Nevertheless, this does not fully discount a more experimental angle, particularly in that the identification, recording and analysis of data does need to explore both the concrete and the more general.

Closely related to the last point is the difference between quantitative and qualitative research; the collection of statistics and data and the description of data which cannot easily be quantified. Again, given the nature of the website and range of material involved, it may be far easier to provide qualitative analysis from students and teachers but once more this does not necessarily mean the exclusion of quantitative data where available. The use of a set questionnaire aims at providing some quantitative and qualitative data through a combination of fixed options and more open sections. Feedback opportunities from teachers offering perspectives on their experiences and what they have observed from their students' experiences, combined with similar information from a focus group of five teachers will provide further qualitative data for analysis and comment.

Warschauer et al (2000: 104-5) continue to identify another area of interest, that of etic versus emic analysis. Etic research implies the definition of areas of interest and a later analysis to see if data and people's behaviour falls into these categories. Emic takes the opposite approach, with the data being examined and patterns and categories taken from this analysis. Exploring new material and areas it appears that an Emic approach will probably be the most logical in the initial stages at ELI, although once initial results and categories have been identified they may provide the framework for a more Etic based approach as a continuation.

One final distinction lies between product and process: what is produced as a result and the process involved. Again, for the purpose of evaluating the ELI web pages, a combination of both approaches would be useful, examining the product, the language generated and produced but also the process involved, how the WWW and the related activities helped produce a given product. Effectively, as Warschauer et al (2000: 105) conclude: "In many cases, educational research will fall somewhere between the various approaches." The general approach adapted in this thesis is very much that defined by the authors as action research, when a problem is defined, a solution sketched, tried and then evaluated.

Overall, the complexity of the material and number of people involved in its implementation and exploration at ELI lends itself to an ethnographic approach creating a "thick description" (Geertz 1973) as information from numerous sources and perspectives is drawn together for analysis. This in turn implies a triangulated approach, processing various sources of information in an attempt at gaining a wider, more global vision of what is occurring. As Müller-Hartmann (2000) says: "The triangulation of various forms of data (e.g., e-mail letters, interviews, questionnaires, field notes) illuminates classroom processes and allows conclusions across the researched learning environments." DuFon (2002) describes triangulation as the building of: "... layers of description, thus yielding a thicker description and increased credibility or validity." Triangulation can thus be a: "... vital part of any qualitative research as it helps to develop a deeper understanding of the subject in question." (O'Dowd 2003)

From the more specific point of view of language learning Chapelle (1997) outlines two key questions which also need to be taken into consideration:

- "What kind of language does the learner engage in during a CALL activity?"

- "How good is the language experience in CALL for L2 learning?"

This first question is descriptive, while the second is evaluative. Chapelle (2001: 52-53) developed the ideas further when discussing the differences between judgmental analysis of the software and tasks and empirical evaluation of the learner's performance. What is emphasised here is that participation and engagement by itself is not sufficient in determining the success of an activity; what must also be considered is the quality of the learning that is taking place within this interaction.

Summarising different approaches Levy (1997: 44) concludes: "As far as evaluation is concerned, early comparative studies that aim to establish the superior method, have given way to more 'atomistic' studies that recognise the complexity of interrelating factors associated with the characteristics of the media, the learner, and the participating learning context." Warschauer (1998) reaches a similar conclusion:

"We don't need a new monopoly paradigm of research, but rather a multiplicity of approaches that allows us to fully address the many questions that use of new technologies poses. Indeed, some of the best language and literacy research combines a variety of methods (e.g., quantitative and qualitative linguistic analysis, discourse analysis, interviews, participant-observation) or even approaches ...".

Through a combination of theoretical and practical issues, the evaluation of the WWW-based material developed at ELI has indeed relied on a multiplicity of approaches and factors.

5.3 Practicalities of evaluation

Having looked at the theory there remains the question of practicalities, what to evaluate and how, while bearing in mind that the involvement of both learners and teachers in the evaluation process is also intended at encouraging increasing participation, involvement and ultimately use of the project as a whole.

5.3.1 What to evaluate

Chapelle's framework has been used as a framework for the development of the material and will prove useful again for analysing it. The design itself must also be evaluated, as it has been identified as a key practical issue, while the key areas outlined in Chapter Two also need to be remembered: hypermedia, learner and teacher. The question of what to evaluate also forces a reminder of the key potential of and intention behind the material: that of exposing students to and encouraging awareness of, authentic language and as a result generating enthusiasm, boosting language production and increasing long term-learner autonomy. What also needs to be remembered is that the material is intended for both teachers and students and both these areas will need to be analysed. These are very general and wide goals and an analysis of the success or failure in achieving them will require both judgmental and empirical approaches.

5.3.2 How to evaluate

In terms of empirical research, students but perhaps particularly teachers, will need to record samples of their interaction with authentic language, considering whether or not this is deemed positive for and by the students, and whether or not language production and language skills are positively stimulated as a result. In attempting to analyse this, Higgins (1995:72) provides a list of tools for CALL assessment among which is the concept of attitude questionnaires. He does, however, make the point that such questionnaires may not be very accurate. Osuna and Meskill (1998: 77) recognise a related problem:

"Students generally associate language learning with explicit morphosyntactic teaching. Thus, when subjects were asked directly if language learning was taking place, some did not recognise it. This finding, however, is not surprising because as research has demonstrated, implicit language teaching and learning is difficult to assess ..."

Chapelle and Jamieson (1991: 45) also note that in terms of questionnaires, the internal validity depends on the truthfulness and accuracy of statements students make. Nevertheless, descriptive research involving surveys of student attitudes and observations of their behaviour remains central to any evaluation. Windeatt (1986:79) also describes the "Ö development of observation technique as part of a methodology for evaluation software Ö" and this, combined with questionnaires, will be used in the evaluation of the materials designed at ELI in terms of how students interact with and possibly benefit from the resources. It must also be remembered that, while observations may not be practical within the given context, there is also a clear need for questionnaires with teachers, in order to explore their perceptions of the web pages on various levels and assess their appreciation of their own students and the value of the material as a whole. All this information will provide the range of views and information necessary for triangulation.

For the evaluation of the material in class a concise but encompassing questionnaire was devised. A very general model was used in the initial evaluation period simply asking for comments under the headings of technical, task, language and general. This was soon seen as being useful in that it did allow for freer contributions, but too vague also in terms of the information provided. As a result, a more detailed questionnaire was devised providing for specific areas and options but also continuing to allow for more general comments if necessary. With simple modifications, essentially the same survey could be used by both teacher and student. This questionnaire dealt in a very general way with the key areas involved: the technical side, the tasks, how they were integrated and perceived language benefits. As a starting point it was felt the initial information generated may highlight specific areas for further research and that a more detailed list of questions or options might actually reduce or limit unnecessarily the range of feedback being produced at this early stage. Having both learners and students evaluating the same essential areas, but from differing perspectives, was also deemed as an interesting concept to explore at this point and one which aided the triangulation approach mentioned earlier.

As a complement, teacher observation of student behaviour was encouraged as an additional source of data to be evaluated. It was felt, however, that observations of teachers would be perceived as possibly threatening at this early stage but could be replaced to an extent with opportunities for more detailed feedback through training sessions where experiences were related and recorded.

Nevertheless, rather than rely completely on general information from such a breadth of sources, the need for more specific evaluation was also noted, both on a smaller scale, but also on the basis of more intensive rather than casual use. This was achieved through the involvement of a smaller number of teachers, who agreed to explore the material more intensively and effectively work as a focus group. Over the course of two meetings, these teachers reflected on the questionnaires they had completed, discussing how their responses varied over time. They exchanged opinions and experiences in a general way and also under the areas highlighted in the framework outlined in Chapter Three.

5.3.3 Procedure

As outlined in the previous chapter, the site was initially introduced to teachers at the end of the academic year October 2001 - June 2002, giving people time for some initial experimentation. In general, of the forty teachers involved, twenty seven did use the materials in some way at this early stage. Three teachers explored the materials in a more intense way, forming a focus group and keeping their own and student records of each class for later evaluation. This initial period of evaluation led to certain changes within the site, the support material for its exploitation and, as mentioned previously, changes in the questionnaire used for teacher and student feedback.

From the beginning of the academic year October 2002 - June 2003 the material was re-presented and teachers were gently pressurised into trying it out at least once. They were also encouraged to use initial surveys of student attitudes to and experience of the WWW and to complete the questionnaires for both students and the teachers themselves after each class.

Of the forty teachers potentially involved, twenty two returned their own personal feedback sheets. Thirteen of these teachers also included feedback forms from classes, some of them from more than one with the result that there were completed questionnaires from seventeen different classes, four adult and the remainder adolescent within an age -group of thirteen to seventeen years old. Overall, one hundred and forty seven questionnaires from students were collected.

Initial statistics and impressions were exchanged at a teacher training session for all teachers, where it was discovered that over ninety per cent of teachers had experimented even if they had not returned feedback forms. On the basis of this initial exchange of experiences, a further round of experimentation was initiated. More detailed meetings continued, with what was now a focus group of five teachers investigating the material on a more regular basis. They continued to complete questionnaires and take notes on their observations of student behaviour and interaction. They did not necessarily ask students to complete questionnaires continuously, as it was felt to be repetitive. More useful was the occasional opportunity for teachers to report on a series of classes, providing more global, long term impressions. This focus group of five teachers met twice over a period of six months to exchange information. In general, each teacher involved in the focus group was dealing with two or three classes of teenagers and on average took them to explore the www.eli.es/class materials between four and six times over this six-month period.

The plan for the academic year October 2003 - June 2004 is that the teachers who have used the materials more extensively will report on what has been found positive and encourage wider, more consistent use for the future. A further round of experimentation on a wider level will then be implemented.

5.4 Feedback on questionnaire / judgmental observations

Following the concept of triangulation outlined earlier, the data from the research is presented in the following manner. Firstly, focussing on the areas in the student questionnaire, learners' responses are summarised in a quantitative way, followed by a more qualitative analysis of comments and general observations. Likewise, initial teacher responses are outlined in a similar fashion. The results from the focus group are then reported, giving an idea of how impressions and usage developed over time. Finally, some more general statistics are examined, providing the data for further discussion and conclusions.

5.4.1 Learner Questionnaire:

1. Technical

This section of the questionnaire was intended to obtain some general impressions on how easy the material was to use on a technical level.

- The computer room: 21% Excellent 62% Good 17% Poor

- The WWW connection: 9% Excellent 43% Good 49% Poor

- How do you find using the Internet: 62% Very Easy 24% Easy

14% A bit difficult

- Did you need your teacher's help?: 11% Yes 31% A little 68% No

Comments and initial discussion

The majority of students believed that the Internet connection could have been faster and this was equally true of all centres, whether or not the connection was through cable or ADSL. Again, the majority of students valued their own ability to use the WWW highly and their reliance on the teacher's help as being low. These statistics differ, as will be seen later, from the teachers' points of view.

Of the students who commented on this section, "easy" was used in 70% of their comments, reflecting a confidence in their own skills and perhaps in the ease of use of the material they had been guided through. A comment such as "I didn't have any problems" was reflective of this group. 49% also used the word "slow" to describe the speed of the Internet connection and, indeed, this was the only negative adjective used in the responses. Noticeably, even among students who used the concept "easy" to describe their experience there were over ten instances where comments were made regarding specific difficulties with use of the mouse: "It was easy but I had problems with mouse"; or more general difficulties such as: "I didn't know where I must go."

Overall, students appeared to find the material easy to use but their initial confidence does appear tempered by a lack of practical experience and knowledge of how to navigate the WWW on a complex level. This conclusion will be referred to again later under teacher observations.

2. The task

This area of the questionnaire aimed to explore attitudes to the actual task students were asked to complete, including perceptions of how it was integrated into more general class-work and the extent to which multimedia elements of the WWW and support features of the site were appreciated and used.

- Did you prepare for it in class?: 43% Yes

- Did you enjoy it?: 89% A lot 8% A Little

- Did you find the task easy to do?: 65% Very 22% A little

10% A bit difficult

- Did you need teacher's help?: 72% No

- Did you use it later in class?: 88% Yes

- Use of these features:

MAPS 5% PICTURES 34%

TIPS 12% DICTIONARY 42%

VOCAB PRACTICE 27%

Comments and initial discussion

Overall, these statistics reflect a very positive attitude to the tasks and their level of difficulty. Noticeably, use of areas designed to aid student autonomy and development remains limited and use of the more graphical and multimedia aspect of the WWW was also explored to a very limited extent. Equally interesting is the concept that most students felt they had not prepared for the tasks beforehand in class, while in many cases, as will be seen later, teachers felt they had. Again, on a similar point, there is a difference between the studentsí opinion of whether or not they have needed the teacherís aid and teachersí personal feelings. What remains clear is that the vast majority of students enjoyed the activity, appeared to recognise it as being incorporated into standard classroom activities and found it relatively easy to achieve.

Typical comments from students in this area were: "It was easy", "It was normal", "Easy, nothing strange or difficult", "Easy , no normal is the best expression". When problems were mentioned they tended to focus on vocabulary: "I have a little problem with the vocabulary", "I didn't understand some words". Only one comment: "The problem was my film hasn't got a plot" seemed to reflect something more serious in terms of the site selected, the task and the suitability of the material to fulfil that task.

3. Language

This area of the questionnaire explored what participants felt were the language benefits of the on-line activity.

- 0% said the activities aided SPEAKING

- 73% said they aided READING

- 34% said they aided WRITING

- 0% said they aided LISTENING

- 82% said they aided VOCABULARY

- 24% said they aided GRAMMAR

When asked if they thought the Internet activity helped their English in general:

- 86% replied A LOT

- 3% replied A LITTLE

- 11% replied NOT REALLY

Comments and initial discussion

Students, in other words, clearly identified the areas of Vocabulary and Reading as the principal beneficiaries of their WWW tasks. Interestingly, as will be seen later, this contrasts with the opinions expressed by teachers, who identify gains in a wider range of areas. This is perhaps a natural disparity, as teachers are possibly more aware of the range of linguistic skills being used at any one moment or within a task which overtly appears to be practising one specific language area. What may also be reflected here is that the students are focusing on and analysing the very specific WWW task and not considering how that was incorporated into a wider range of activities in class before and after the specific computer-based work. What is interesting, is the studentsí overwhelming positive response to the next section of the questionnaire: although a significant percentage saw little overall benefit, a very large percentage did see the WWW activity as a means of practising their English.

There were no comments at all in the space "Why do you think the activities helped or did not help you to practice English?" which seems to illustrate a problem with the questionnaire: the concept is either too difficult or too wide to be answered adequately by learners. However, this may also illustrate the difficulties learners have when asked to assess their own learning experiences and progress.

Nevertheless, there were a range of reflective issues in the general COMMENTS section which do indicate both an awareness of how the WWW may be of importance on a cultural and also linguistic level, and how these classroom activities have been helpful in that respect. There were over twenty comments of this type: "Yes it was very useful for my English." There were eleven references to the value in terms of vocabulary learning and then a variety of other positive comments such as:

- "English is important for the future"

- "You know something about another place"

- "If you are reading only English for 15 minute itís very useful"

- "When you are on the WWW you can learn other news and practice English in another form"

- "I learn English based in command an interesting topics"

These comments appear to reflect a very positive appreciation of the WWW as a potential for exposure to both language and the intercultural implications of the language in context.

4. General

This final section of the questionnaire allowed for more open comments and responses, examining more global attitudes and responses to the experiences.

- 91% replied they would like to do more of these activities in class.

- 5% replied they would certainly use the pages outside of class. 21% replied "Maybe" to this question and the vast majority did not answer this question.

- When asked if they would use certain features of the website in the future, there were only twelve responses in total. Ten said they would use the DICTIONARY, eight the VOCAB. PRACTICE and two the EXAMS.

 

Comments and initial discussion

Response to these questions indicates a very positive attitude to continuing to use the material in class but a much more reserved one to using the material on a more autonomous level outside of class, without a teacher. This in turn possibly reflects a more generalised reluctance to studying English outside of class time, although what would be interesting to investigate long-term is whether or not the WWW could provide the stimulus for encouraging students to increase their exposure to English outside of formal class time and, in the process, improve their awareness of the importance of learner autonomy in general. Their experience with the WWW and English in class time certainly drew very positive responses, as can be seen again by the more general comments students contributed.

- "It was very interesting and good for my English"

- "It was funny and easy"

- "I'd like to be more than 15 minutes on the computer"

From a total of sixty-seven comments, the adjective "easy" was used thirty one times, "funny" sixteen and "very easy" twelve times. There were no negative adjectives used at all, with the only major complaint being that more time could be spent on the computers. There was one more limited criticism from an older adolescent: "I don't think the WWW is essential for studying anything".

Overall, the comments certainly highlight a positive attitude to the area and the materials. The students appear to be motivated by the experience of using the WWW in class because they perceive it to be both an entertaining and an effective language learning tool.

5.4.2 Teacher Questionnaire

1. Technical

Teacher evaluation of the technical aspects of the WWW tasks was as follows:

- The computer room: 24% Excellent 73% Good 3% Poor

- WWW connection: 5% Excellent 58% Good 37% Poor

- Ease of use: Very Easy 32% Easy 51% A Bit difficult 15%

- Students knew more than you did: 63% Yes 20% A little 11% No

Comments and initial discussion

Overall, the most typical responses in this area were "fine", "ok", "no problems". There were, however, numerous areas which needed consideration, particularly in terms of the studentsí navigational skills when using the WWW. "They got there but needed help"; or: "They knew their way around but it took patience to get the task done". These illustrate the mixture of confidence and limitations of browsing skills which typical students appear to possess. Overall, it appeared quite clear that studentsí evaluations of their own ease of use did not coincide with teachers' evaluations: in general students were far less confident and adept when surfing the WWW than they claimed.

At the same time, teachersí own self-confidence appeared low, as 62% claimed that students knew more then they did, even while 45% also admitted to having had to help their students with navigation. When compared to studentsí own reactions to this area in which 68% claimed that they did not require help from their teacher, there does seem to be a general indication that students do have more self-confidence than teachers when dealing with WWW technology in the classroom. Nevertheless, there are indications that studentsí familiarity with technology may be over-valued by both students and teachers: some students certainly do rely on their teachers for help on a technical level, which, for teachers who are less confident with the area, may prove daunting.

2. The Task

This section of the questionnaire required teachers to evaluate task itself in terms of level and appropriacy and also the use of different support features of the web site in accomplishing the task.

- 87% said the level was appropriate

- 67% said students found it easy and only 5% said it was difficult

- 92% did not alter the task

- 32% used the on-line dictionary; 5% the on-line grammar; and 2% the TIPS

- 40% said their students used visual aids such as pictures, to help their understanding of the texts

- 34% used the Vocabulary Practice and 8% the Exam Practice section

Comments and initial discussion

Quite clearly, teachers viewed the tasks as being relevant and of the appropriate level. Quite naturally, teachers did not, on the whole, alter tasks they were only beginning to experiment with and again, correlating completely with the feedback provided by students, use of additional on-line support elements is rather limited.

With regard to the comments made in this area, what stands out are the continued references to learner motivation and autonomy:

- "Didn't get distracted"

- "Very focused"

- "Worked well by themselves"

- "Paid more attention than usual"

- "Motivates them"

- "They were fascinated"

These comments indicate a very positive impression on behalf of the teachers involved with regard to studentsí ability, necessity and motivation to work by themselves. This in turn can be closely related to student comments in the same area, which were extremely positive. In other words, both students and teachers evaluated the task in a very positive manner and coincided in that the task was of the appropriate level and not unduly difficult to carry out.

It is also interesting to note that virtually no teachers altered the given task and, as noted earlier in student responses, use of additional support material existed but was, on the whole, quite limited, with the on-line dictionary being the most popular.

3. Integration:

Teachers were also asked to comment on how they saw the tasks as fitting into their overall lesson plans and on their use of the material provided to help achieve this integration.

- 57% used the initial student questionnaire

- 24% used the initial WWW introduction pack

- 98% of those who had used these said that they had been useful.

- Did you use the FIRST Section of the task?:

25% 30% no 45% used their own

- Did you use the AFTER section of the task?:

66% Yes 2% NO 32% Other

- Did the task complement the coursebook?: 89% replied affirmatively.

- Did the on-line material complemented normal class work and activities?:

A LOT 89% A LITTLE 11%

Comments and initial discussion

While over half the teachers did use the initial student questionnaire with regard to their experience and opinion of the Internet, it is somewhat strange that only a quarter then went on to use the introductory material for the specific site and materials. This reluctance to prepare students more fully for what they are going to experience appears a pity in the light of the overwhelmingly positive response of those teachers who had used the material.

Interestingly, a much higher proportion of teachers used the AFTER section of the tasks, continuing suggestions for follow-up to the computer based-activities in the normal classroom, than the FIRST section with suggestions as to how the on-line tasks could be introduced in class. This suggests that, once teachers had seen the material on-line, they recognised its potential for later class use, while it is more difficult to launch an activity in class which depends to a large extent on an upcoming novelty element such as the incorporation of WWW based material. This does, however, support the notion that the WWW activities themselves encouraged recognition of their potential for later classroom exploitation. Teachers certainly responded positively in their perception of how well they complemented normal class-work and activities.

"I did it as an end-of-month filler but it has lots of potential for integration into normal coursework" may be the most general summary of comments in this area, as many teachers used the material on an initial exploratory basis but did not follow it through over the long term. Various comments support this conclusion: of the teachers who reflected on this area, all appear to initially identify the material as a relevant and an interesting way of expanding normal classroom work. "Tied into book nicely" is one of the twenty-one positive comments in this section, while twelve specifically refer to the speaking that was generated in class afterwards. Writing was mentioned by eight as having benefited greatly from being given a "focus" and "genuine purpose".

4. Language

The following is a summary of teachersí views as to the value of the WWW tasks and their exploitation for language acquisition.

- 65% said the activities aided SPEAKING

- 89% said they aided READING

- 78% said they aided WRITING

- 12% said they aided LISTENING

- 92% said they aided VOCABULARY

- 19% said they aided GRAMMAR

Comments and initial discussion

Compared to the views expressed by students on the same area, there is agreement, in that Vocabulary and Reading are the areas everybody perceives to be the most immediately benefited. Teachers valued the impact on writing and listening much higher than students while students, interestingly, placed more emphasis on grammar but had no recognition of the speaking element at all, even though a significant majority of teachers did.

Overall comments on the last two sections of this questionnaire provide some generally very positive feedback while raising very practical issues to be dealt with: "excellent revision of vocabulary" (speaking about the VOCABULARY section), "very teacher-friendly" were typical comments. In a general sense, the tasks were seen as being "excellent revision" and all students "appeared to like it". The vocabulary possibilities were the most frequently mentioned, followed by reading and this correlates with studentsí opinions.

It is also interesting that the potential for self-study and autonomous learning were noticed and identified by nine teachers:

- "Very motivating"

- "Adults loved having their own page and have been accessing the pages from home and work"

Overall, teachers coincide with students in a positive perception of the value of these WWW-based activities for language acquisition with teachers, obviously perhaps, identifying a wider range of benefits and issues for further consideration.

5.4.3 Extended use:

The following information is compiled from meetings of the focus group of five teachers, who used the material up to five times with more than one class. The questionnaires used with students and teachers provided a framework for these discussions. Teachers kept the questionnaires they had completed with their classes and discussed how their evaluations had changed over time. The statistics quoted below are taken from the last questionnaire each of the five teachers completed, in other words a questionnaire completed after taking that class through a series of four to five lessons based on the material on www.eli.es/class.

1. Technical

- The computer room: 20% Excellent 80% Good 0% Poor

- WWW connection: 40% Excellent 60% Good 0% Poor

- Ease of use: Very Easy 60% Easy 40% A Bit difficult 0%

- Students knew more than you did: 20% Yes 20% A little 60% No

Comments and initial discussion

Neither the studentsí nor the teachersí overall perception of the Computer room altered over time, with the same perception remaining of the facilities being adequate but by no means exceptional and the overall impression remaining that the WWW connection could be faster.

Teachers agreed unanimously in that over time the technical demands the activities made on students diminished drastically. Having experienced the type of activity once, on the second encounter students could be reminded of the essential skills and they remembered. Once more, it was apparent that students' initial appraisal of their own skills was far more optimistic than that experienced by teachers in the computer room, although over time these core skills did improve. On two occasions this was forced by teachers, who warned their students beforehand to remember what they had learned in previous lessons and that the teacher would help only with language issues in the following computer sessions. This apparently encouraged students to become more actively aware of the skills they both had and needed.

It was also found that studentsí ability to navigate between windows improved remarkably, particularly after the third session. Similarly, the ability to make notes of the task required, rather than having to revert to on-line written instructions, was noted in many different students.

Overall, on a technical level, given adequate preparation and opportunities to get to know the material, the on-line activities were deemed simple to use and easily assimilated by the vast majority of students. What is more, students, having become familiar with the basic technological requirements, appeared to acquire additional confidence, which teachers speculated may even have positively affected their involvement with the language they encountered on-line.

2. The tasks

- 100% said the level was appropriate

- 60% said students found it EASY 40% A LITTLE DIFFICULT

- 80% did not alter the task

- 80% used the on-line dictionary; 20% the on-line grammar;

and 60% the TIPS

- 80% said their students used visual aids, such as pictures, to help their understanding of texts

- 60% used the Vocabulary Practice and 10% the Exam Practice section

 

 

Comments and initial discussion

Over time, it is noticeable that teachers continue to feel even more strongly that the material is appropriate and of the correct level, although that does appear to depend a lot on the actual class level: lower levels found the material more challenging, although, even so, it was not considered too difficult. Over time, they also appeared to have gained the skills to find the tasks easier. What is also clear is that teachers gradually began to explore other options provided by the site, from TIPS, to DICTIONARY, to encouraging students to use visual aids a lot more. Teachers also mentioned that students, who had to be encouraged the first time perhaps, quickly got used to using these support features automatically and by their third class they needed little reminder from their teacher that such features existed.

Overall, teacher comments in this area emphasise the area of learner autonomy and motivation, as students explore the WWW more thoroughly but within very specific parameters:

- "Students become more independent using the dictionary and pictures to help their understanding"

- "The content titillates and motivates teenagers of this level"

- "The WWW really focuses them on the real world and real usage."

in general, the feeling was that the material and tasks were motivating, productive and in line with normal classroom procedure. Indeed, there was a general feeling among the focus group that long term use of the material would provide very positive results in terms of motivation, learner autonomy, teachersí ability to focus their attention on individuals within the class and general language skills improvement and development.

3. Integration

- 100% used the initial student questionnaire

- 100% used the initial WWW introduction pack

- 100% of those who had used these said that they had been useful.

- Did you use the FIRST Section of the task?:

40% YES 20% NO 40% used their own

- Did you use the AFTER section of the task?:

60% YES 0% NO 40% OTHER

- Did the task complement the course book?: 100% replied affirmatively.

- Did the on-line material complement normal class work and activities?:

A LOT 100% A LITTLE 0%

Comments and initial discussion

The overall appreciation was that the on-line tasks could be easily integrated into normal course work and syllabus is reflected very positively. While it has to be remembered that this focus group was very small and because they were so actively involved in the project these five teachers would perhaps naturally react positively and also attempt to explore to their fullest the range of support material and resources available, their reactions remain extremely positive. What is more, their experiences appear to indicate that a fuller use of the supporting materials will indeed help prepare students and in turn increase the ease of the activity and the benefits obtained.

Interestingly, over time teachers, continue to use the FIRST activities least, often developing their own, while continuing to use the AFTER activities provided by the task to a large extent, although there was a degree of experimentation. Over time, teachers remain convinced that the tasks complement normal coursework and classroom activities.

This area drew limited comments but those that were noted included:

- "Generated lots of speaking in class"

- "Excellent when students get used to the pattern"

- "Worked much better than the text in the book does"

- "Gave nice background to the class theme"

Once more, the indication is that over time, with regular use, the materials do provide for effective integration opportunities. The one negative point mentioned in this discussion was that, while the material could be incorporated effectively without any problems, particularly once students were comfortable with the technical skills and the overall concept, having to reserve and then move classes to a computer room regularly became an imposition; as a result there was some debate about the potential for having a computer in each classroom with an WWW connection.

4. Language

- 60% said the activities aided SPEAKING

- 80% said they aided READING

- 80% said they aided WRITING

- 40% said they aided LISTENING

- 100% said they aided VOCABULARY

- 10% said they aided GRAMMAR

Vocabulary was the area all teachers felt benefited without any doubt: exposure to genuine English was seen as vital both in terms of the vocabulary presented and in terms of the motivational effect it was seen to have on students. Teachers continued to feel that speaking and, as a result, listening benefited from follow-on activities in class, in other words the fuller integration of the WWW tasks into normal class work.

Interestingly, and in contrast to initial reactions by teachers and students, not all teachers felt that there were long-term benefits in the area of Reading. The feeling was that, while students were having to deal with genuine texts, they were often ill prepared for dealing with this input and the WWW itself and support material did not help sufficiently. This in turn resulted in a debate about what reading skills are needed and how best they can be developed in class, off-line. Given sufficient practice, the feeling was that students can be helped to develop their reading skills in terms of skimming and scanning, so that they are less confused by large texts. The ability to use additional support material to help them in this respect, from dictionary to pictures and other graphical information available in a hypertext environment, was also noted as important.

Writing was seen to have benefited more specifically as a result of the tasks themselves, as they lent a degree of purpose and authenticity which is often lacking in standard classroom writing activities. What was also observed was the enormous improvement in terms of studentsí abilities to skim through on-line information. After two to three sessions they were beginning to take notes rather than simply copy and in turn, develop the skills, either oral or written, to generate coherent and comprehensible feedback to use later in class. All five teachers involved in the focus group stressed this move from the studentsí original copying down verbatim what they had read, to their learning to report coherently on language, information and ideas they had actually absorbed. The more the studentsí used the WWW lessons the better they became at this.

5.4.4 Additional statistics and observations

While the initial presentation to teachers implies a certain degree of forced collaboration and the focus group was comprised of teachers committed to the investigation, it has to be noted that over and beyond these more imposed conditions, usage of the WWW increased considerably over the academic year October 2002 - June 2003. Although statistics remain observational rather than factual, from virtually no usage in the previous year it became quite a regular occurrence to find the computer room used for WWW surfing, and in the majority of these cases the starting point was the www.eli.es/class site. Interestingly, it also appeared that several teachers did indeed adapt on-line exercises to suit their own needs, while others appeared to branch out, find their own sites and develop their own completely independent material. In this sense, the web pages appeared to have built teacher confidence to such an extent that certain individuals felt able to develop their own material within a similar framework.

There was also evidence that some students had explored the site from their homes. Teachers who have used the site in class have reported the fact that students were impressed by the fact that they could access the material from home and there is evidence that a certain, if limited, number have. This is true of adults and adolescents but also interesting have been the number of enquiries by parents with regard to the Vocabulary section and the material suitable for their younger children. Overall, the school appreciates the marketing value of this in that ELI is seen as a reliable source of information and quality control for parents who may wish to use the WWW for their childrensí English learning on-line.

5.5 Evaluation within the theoretical framework

In general, initial evaluation and feedback from students, teachers and indeed parents has been very positive. Overall, there is an appreciation of the simplicity and power of each level having its own homepage, an easy point to begin, a place which can be trusted, and which in turn implies a filtering, a huge saving of time for all three parties. Quite clearly, the site appears to be providing an initial solution to the problem at ELI, dealing effectively with the key areas and features of the WWW outlined in Chapter Two. It is now time to reflect on this initial information in the light of the framework proposed in Chapter Three.

5.5.1 Language Learning potential

The provision of vocabulary and simple grammar practice games, the role of computer as tutor is not very original and primarily uses WWW as a new platform for relatively old ideas. Nevertheless, particularly with younger and lower-level classes, there is a wide appreciation among students and teachers that they are both enjoyable and useful, both as reinforcement and as presentation of new lexical and grammatical items. WWW as a platform has of course the advantage that the material is available both in and outside of class and is not restricted to the school itself. Attracting students to the material outside of class, from home perhaps, should increase the language learning potential of the site, as students explore paths of more autonomous learning and perhaps, in the process, move from the vocabulary section to others.

Teachers immediately identified the increase in student motivation and the benefits of material which generated important skills practice and development in terms of reading, note taking, general writing and summarising skills, speaking when exchanging information / opinions and as a result listening also. Again, the potential may increase as students feel freer and more confident to explore the material from their own homes. There appears to be evidence that there is (Chapelle 2001: 70-73) a focus on form as students become aware of lexical and grammatical items; there is modified interaction as students interrupt their reception of input in order to use a dictionary, the teacher, a partner or a multimedia feature to aid comprehension; there is modified output as students, when note taking or reporting back in class, are encouraged to deal with errors, consider the language they are using etc. Nevertheless, there are also doubts with regard to how long it may take for such a focus on form to develop. While identified more clearly by teachers in the focus group, one-off lessons, do not seem to automatically provide any such focus. Even with repeated usage, the focus appears to be more on general skills improvement and development rather than on specific grammatical forms although there remains the question of how far these two areas may be separated.

Quite clearly, both students and teachers have reacted very favourably to the language learning potential of the tasks designed to incorporate on-line material as a complement to course book topics. Students quickly saw the exposure to vocabulary as useful and powerful, while, with increased use, they also began perhaps to appreciate a wider range of potential uses.

 

 

5.5.2 Learner fit

The range of material adequately provides for the range of ages and levels at ELI. Indeed, the success in complying with the learner fit is reflected in the continued use of the site: it is difficult to imagine either students or teachers spending the amount of time they have if the material had not been recognised as being relevant, useful and manageable. The correct task is essential for keeping the language level suitable for the learners involved. This implies careful selection of a site, careful construction of the task, how they are supported in accomplishing the task by the teacher, on-line aids or multimedia features. With a clearly defined task, and with the experience gained over time, students will feel confident enough to deal with quite large amounts of authentic input and take what is necessary or relevant to them or their given assignment. The big advantage is the freedom students gain to work on an individual, more autonomous level, regulating their own pace, the range and speed at which they process input and generate output.

Inclusion of internal exam information also aids the learner fit in that it encourages students to access the site and explore its contents. The addition of the TIPS section further assists such autonomous exploration, as students begin to discover how the range of material available can be best exploited for their individual needs and interests.

 

5.5.3 Meaning focus

The Vocabulary and grammar games section quite clearly focus specifically on exposure to, and practice of, form. Nevertheless, as has been pointed out, this is use of the WWW as a platform for traditional CALL activities, rather than the exploitation of existing authentic material from the WWW as a means of discovering and practising language. The coursebook-related tasks deal more specifically with this. Observation and notes taken by teachers during these sessions show a high degree of student interaction with the on-line material and a large number of questions, very often related to vocabulary or expressions frequently found in authentic situations but interestingly sparse in standard textbooks. This appears to highlight an attention to, and awareness of, meaning at a lexical level.

There was little evidence, however, of studentsí awareness of, or interest in, the grammatical structures encountered. What was equally interesting were the problems encountered by students when asked to take notes of interesting or important information; all too frequently students copied directly from the on-screen text, despite the fact that they could not understand certain words or even the general meaning of what was being copied. This seems to indicate a clear lack of focus on meaning. Even with practice and guidance, note taking remains for some students a very difficult task, perhaps because it really does force concentration on meaning, as key information is separated from unessential or unintelligible material. Student problems with this concept hint at an area for further exploration.

5.5.4 Authenticity

This has already been referred to above, in the description of how the site helps students deal with authentic input in an unthreatening way and, to some extent, develop the skills that are required by learners when exposed to an authentic environment, without a teacher or text book to grade the language. Spanish students of English studying in England will have to develop a range of useful skills which will allow them to read or listen effectively while avoiding the sensation that all input has to be completely and perfectly understood. The development of such skills is difficult to achieve in the artificial, limiting scope of a classroom.

The authentic input from the WWW, combined with the more sterile but laboratory type facility for analysis and controlled practice that is the classroom, is an interesting and powerful combination: now, real language can be sampled, students can be aided in developing strategies for dealing with it, but with the support and comfort of a more controlled environment. The site has a potentially very positive role in stimulating such an active learning environment, within which students are exposed to real language, allowed to work at their own pace and in the process encouraged to develop the skills necessary to become more autonomous.

5.5.5 Positive impact

There appears to be little doubt that the site and materials have had a positive impact on learners and teachers; indeed the popularity with students has always been crucial in ELI at encouraging teachers to continue exploring the field and material and not abandon it after one relatively successful or partially failed class. This has also been the result with this site. Even after initial difficulties or challenges, students have wanted to return and teachers have been under some pressure on the one hand, but stimulated on the other, to continue investigating. What also appears clear is that the more regular and consistent the exploration, the richer the rewards and the more positive the impact on teacher, learner and language gain. Indications that students and parents have investigated the material outside of the school further points to a positive evaluation of the material.

5.5.6 Practicality

This is of crucial importance. Given the opportunities to use computer rooms and technology, students will typically jump at the potential for variety and escape from the traditional class. Faced with the possible threat of a new classroom situation, a new role for the teacher and technology which they may not be comfortable with, teachers frequently balk at the challenge. In this sense, practicality is the pivot around which all the other areas revolve: if something does not work on a practical level the whole range of pedagogical potential will never be explored, as was discovered at ELI with its initial, relatively unguided exploration of the WWW for language learning purposes.

The site appears to have overcome these problems by making the use of WWW-based materials in class a practical possibility. The pre-selected material and pre-graded tasks, the range of additional support features, and variety of training and awareness raising activities, all combine to create a relatively friendly and practical package which students and teachers have found useable, and more importantly, accessible enough to return to and use more than once: in other words, it is feasible to use the material in a consistent manner. Thus, students and teachers begin to unlock the potential, while also gaining the confidence and empowerment necessary to continue moving forward on a more autonomous level.

Apart from the actual materials and tasks, the physical design of the site played an important role in achieving this apparent simplicity. Organised so that each level could have their own homepage, while also permitting the advertising of an overall homepage for all levels, immediately made the material appear more friendly, more controlled and more accessible.

Overall, the site design also appears to have achieved its aims of simplicity of navigation and clarity in terms of highlighting what is available: students and teachers reported virtually no problems with these areas. The inclusion of a START link at the end of each task reduced the need for students to type addresses, which had previously been a major problem. Moreover, while new links opening in new frames initially required training to move from one open window to another, long-term it was seen as a very useful feature, as the worksheet or homepage remained on-screen, providing a comforting reference point if people felt they were getting lost.

Also important was the provision of additional support material. Teachers did use student IT surveys before beginning to experiment and in general found the experience useful in focusing attention on what was involved in using the WWW, while also giving the teacher some idea of what to expect from students in the computer room. Both students and teachers valued this highly. Netmag and www.eli.es/teacher promoted continuous awareness of what was available and ultimately aided in encouraging teachers to explore their own, more personalised, approaches to given tasks or even the development of completely new ones in some cases. This has been observed on a number of occasions when teachers have developed their own tasks, very similar to those provided by www.eli.es/class, but using a site that had been found and advertised as potentially useful in Netmag.

 

5.6 Conclusions

The overall conclusion from this evaluation is that the material developed met the initial goal of successfully expanding ELI experience with CALL into the area of WWW, after having analysed with care what this implies and how it can be best achieved within a sound pedagogical framework. From the information drawn from observation, interviews and survey feedback, it appears that the concept of www.eli.es/class succeeds in many of its initial aims. Taking into account the nature of hypermedia, learner and teacher issues, and firmly established within a pedagogical framework which incorporates CALL and ELI methodology, the material succeeds in developing a deeper understanding and exploitation of the WWW for language learning within a specific context. On this basis, returning to the questions posed at the end of chapter one, some initial conclusions can be drawn.

What has the WWW to offer within the teaching context at ELI?

The initial response to this question has to be that the WWW potentially has a lot to offer students at ELI. Teachers and students evaluate the material and experiences positively with an equally positive rating for the perceived value in terms of language learning. The WWW offers the potential of accessing genuine English to students who rarely have this opportunity. Nevertheless, exposure by itself is no guarantee of effective use, which leads to the next question.

What activities / tasks would best exploit its potential?

Once again, the activities and tasks on www.eli.es/class appear to provide profitable WWW language learning experiences, as judged by both teachers and learners. On the one hand, using the WWW as a platform for traditional CALL vocabulary and grammar-based activities appears to work for lower levels. On the computer, through Flash or Java, simple matching activities or gap-fill exercises have an appeal which is far stronger than that of a similar paper-based exercise. What is more, using the WWW as a platform means the material is available to students outside of the classroom or school.

On another, more complex level, from a more Constructivist approach, other activities also appear to work well. Traditional coursebook material and activities are easily supplemented by on-line research. Thus, as opposed to reading the set text about a pop star in a coursebook, students can choose their own favourite pop star, search for information, read and record what they are interested in and later enjoy and benefit from writing up these initial notes and orally reporting back on them to the class. This is just one example of the numerous lesson plans designed to use the WWW as a tool for extending studentsí exposure to authentic language in a motivating way.

The essence of the www.eli.es/class material is the potential it has for integration into normal teaching practice at ELI. The on-line material provides authenticity, variety, novelty within a comfortable, recognisable framework which allows both teachers and students to exploit it to its fullest by incorporating the on-line material into what is happening in class. The nature of these activities remains embedded in teaching practice at ELI: within an overall communicative approach encouraging genuine communication and the practice of all language skills, they provide on-line tasks which differ from standard classroom tasks only in that the input or source of material is the WWW. Furthermore, on-line tasks are easily incorporated into pre- and post- computer tasks which also help blend the time spend with the WWW into normal class time.

What support would teachers and students need?

Initially, it is clear that both teachers and students require a great deal of support, much of it off-line in terms of presentation of the material and how to use it. Off-line support and reinforcement has been essential to introduce and keep teachers aware of the site, help them become comfortable with it and to introduce students to its workings and potential. On-line support material such as TIPS appear to have had less acceptance, although other features, such as on-line Dictionaries, have been very effective. It may be with more extensive use the on-line support features will become accessed and appreciated more.

Would the development and use of these materials aid consolidation of computer literacy as identified by Pennington in Phase Six and ultimately help pave the way towards Phase Seven and a networked world?

The answer to this question appears to reside in the experiences provided by longer-term usage by the focus group of five teachers and their students and the overall reply has to be positive. Nevertheless, what remains interesting is the question of how to generate a context within which all teachers retain the interest, dedication and perseverance of the teachers involved in the focus group: implementation and effective usage long term implies initial knowledge and time dedicated to exploring the potential. As has been noted repeatedly, the medium by itself provides no answers: effective and profitable usage will be the result of dedication on the part of teachers and students, combined with effective materials developed by people working closely with these groups.