Putting principles into practice
Joan Jamiesons Putting Principles into Practice an article summarized by:
Yogyakata State University student
This article is written by Joan Jamieson, Carol A Chappele, and Sherry Preiss
Purpose of the studyTo illustrate how a set of principles informed a series of processes in the evaluation of Longman English Online, a commercial CALL product for teaching intermediate English as a second or foreign language (ESL/EFL).
CALL Product to Evaluate:Longman English Online 3 (LEO 3). its the intermediate level of the video-based multimedia, integrated language skills program. The units not only contain listening comprehension practice, but also include practice with grammar, vocabulary, pronunciation, speaking, reading, research on the web, and writing.Quizz in every unit, longer tests at the end of every four units (module test), and one at the end of all twelve units (the end of-level test).
Nature of the evaluation :The evaluation was focused on the materials rather than on the tasks that the leaners actually carried out as they were using the materialsIt was judgmental evaluation (: based on the logical analysis of a CALL activity), NOT empirical evaluation (: based on the quantitative or qualitative analysis of a CALL activity through observed data which are summarized by the evaluator)
CALL Evaluation Principles and their associated criteria & vriables
PrinciplesCriteriaVariablesExamples 1. Language Learning potential should be the central concern when developing & evaluating
* to indicate CALL quality. (Chapele. 2001)Enhanced inputInput saliece, modification, elaborationText highlighted on the screeninteractionBetween people, between a person & computer, within a personLearner-learner communication tasks, grammar help availableproductionPlanning, correcting production, using helpTaking notes suggested, model for constructed responses
PrincipleCriteriaVariablesExamples of Operationalized variables2. CALL should be evaluated in terms of the quality assessment in addition to instruction
*to indicate the usefulness of CALL test. (Bachmer & Palmer)AuthenticityDomain SamplingCorrespondence between tasks on the test and in coursewareConstruct ValidityScore interpretationMeaningfulness of scores
PrincipleCriteriaVariablesExamples of Operationalized variables3. The criteria to conduct the analysis should be drawn from theory and practice in SLA & Lang. Assessment4. Judgemental evaluation through defined variables is valable for examining materials in a way that can improve future design
All of the previously mentioned criteria were examined, to find whether or not they had been operationalized in LEO 3
These criteria were investigated as they have a relatively strong theoretical, pedagogical, and empirical base: Theory and research on SLA converge to suggest that the likelihood of learners acquiring linguistic input increases if their attention is drawn to linguistic features, using techniques such as marking salience, modification, or elaboration. (Schmidt, 1990; Sharwood Smith, 1993; Robinson, 1995; Skehan, 1998). Benefits and types of interaction for langugage learning (Ellis, 1999; Kol & Schcolnik, 2000; Lantolf & Appel, 1994, Long, 1996, Pica, 1994)
Swain (1985): From the cognitive perspective, the benefits of producing language may be enhanced when learners have the opportunity to plan before speaking or writing and to correct linguistic output, which can be prompted by feedback from others or from self-evaluation.From sociocognitive perspective, Swain (1998) stated that the help from the interlocutor allows for production beyond what the learner can accomplish alone.
Authenticity is explained as the degree of correspondence between the characteristics of a test-task and the features of a task in target language use (Bachman & Palmer, 1996).Achievement test is considered authentic in terms of the degree to which its tasks & directions mirror the content & skills in an insructional unit (:domain sampling)Construct Validity dals with the appropriateness and meaningfulness of the interpretations which are made on the basis of a test score.
Summary of Judgemental Evaluation of LEO 3AreaCriteriaVariablesOperationaizationsEvidenceJudgementLanguage Larning PotentialEnanced inputsalienceFont size/type/color, highlighting, repetitionFont size/highlighting graphics, animation, audio repetitionGoodmodificationSimplification, repetition, clarification, L1 transl., L2 definitions, image/videoImages/video, transcripts, L2 definitionsGoodelaborationAdding grammatical phrases or clauses to textsAdding grammatical phrasesFair
Language Learning PotentialInteractionbetween people Amount/types of co-constructed and attention to formMeaningful tasksGoodPeople and computerPotential of providing enhanced input, obtaining help using the languageProgram correction of learners explanations, culture notesGoodan inner voicechoices
Language Learning PotentialProductionplanning Instructions to plan, directed activities, knowledge of expectationsDirections to plan, pre-writing exercisesGoodcorrectingAmount of self-correction, amount/type of signals from computer or another personSelf-correction of speaking/ pronunciation teacherFairhelpAccessing dictionary, grammar, modelsGlossaries, transcripts, grammar help, models of speaking and writingGood
CALL assessmentAthenticityConstruct validitydomain samplingCorrespondence between tasks on tet & in coursewareTable of specifications, directions, texts, item typesFairscore interpretationAppropriate, meaningfulReporting schemeGood
CONCLUSIONbased on judgments on the criteria of language learning potential as well as authenticity and construct validity of assessment, LEO 3 received a good evaluation. This evaluation also points to areas in which LEO 3 might be improved in the future and it suggests that LEO 3 might be worth pursuing further through the addition of an empirical evaluation.
=== end of the slides ===