Assessment Criteria
I ran a workshop at a BALEAP conference several years ago about assessment criteria. I was particularly interested in the difficulty of marking assignments, especially when the English course is part of a degree course and the marks contribute to the student’s degree classification.
The purpose of the workshop was to look at ways of using profile forms to assess such writing assignments. I was not satisfied that we came to a satisfactory solution at that time, and I’m still not happy with it.
Typically writing is marked using some kind of profile form where students are given marks for criteria such as grammatical accuracy, organisation, spelling, cohesion and coherence, referencing etc. There are many problems associated with using such profile forms and this is what I wanted discuss. These problems need to addressed formally when the course is part of a degree. Some of the problems are:
- what happens if the student ignores the question?
what happens if the student doesn’t answer the question completely?
how do we handle content? - what happens if the student writes too may words?
what happens if the student doesn’t write enough words? - what do we do about plagiarism?
- how important is accuracy?
how important is handwriting?
what about word-processing mistakes (failure to handle the computer)?
can one mark sheet be used to help markers to be consistent and also to give feedback to the student?
I did not really want to look at the problems of constructing such profiles, but the problem of what these profiles can and cannot handle. There have been several important publication justifying the use of these profile forms. Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981), for example show that holistic scoring works. But what criteria should we use and how should they be weighted? One of the earliest uses of such profile, the Foreign Services Institute, uses Accent, Grammar, Vocabulary, Fluency, Comprehension at 6 levels for spoken language (Oller, 1979, pp. 320-326; Valette, 1977, pp. 157-161). For writing the following criteria have been used:
Mullen (1980)
Control of English structure,
Compositional organisation,
Quality of writing,
Appropriateness of vocabulary
Hughes (1989, p. 95) ELTS
Communicative quality
Organisation
Argumentation
Linguistic accuracy
Linguistic appropriacy
Bachman & Savignon (1986) – Oral
Grammar
Cohesion
Sensitivity to register
Hamp-Lyons (1991a)
Content
Argument
Text structure features – cohesion
Evidence of planning – coherence
Language
Hamp-Lyons (1991a) MELAB
Topic Development
Organisation
Connections
Sentence Structure
Vocabulary
Mechanics
Hamp-Lyons (1991a) Michigan Writing Assessment
Ideas & Arguments
Rhetorical Features
Language Control
Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981)
Content
Organisation
Vocabulary
Language use
Mechanics
Hamp-Lyons & Henning (1991)
Communicative quality
Interestingness
Referencing
Organisation
Argumentation
Linguistic accuracy
Linguistic appropriateness
Hamp-Lyons (1991b)
Communicative quality
Organisation
Argumentation
Linguistic accuracy
Linguistic appropriacy
However, this is what EAP teachers think. In this context, Bridgeman & Carlson (1983) surveyed undergraduate English faculty and subject specialist faculty to find out what they thought were important..
They found that in order of importance, the English faculty thought the following criteria were important:
1. Paper organisation
2. Development of ideas
3. Paragraph organisation
4. Addresses topic
5. Overall writing
6. Sentence structure
7. Appropriate to audience
8. Assignment requirements
9. Quality of content
10. Vocabulary usage
11. Punctuation/spelling
12. Vocabulary size
Whereas the subject specialist faculty prioritised the criteria differently
1. Quality of content
2. Assignment requirements
3. Addresses topic
4. Development of ideas
5. Appropriate to audience
6. Paper organisation
7. Overall writing
8. Vocabulary usage
9. Paragraph organisation
10. Sentence structure
11. Punctuation/spelling
12. Vocabulary size
It is useful to compare these findings with the criteria used by the various authors mentioned above. These can be summarised:
| English Lecturers | Subject Lecturers | a | b | c | d | e | f | g | h | |
| 1 | 6 | Paper organisation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| 2 | 4 | Development of ideas | ? | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| 3 | 9 | Paragraph organisation | ✓ | ✓ | ✓ | ? | ✓ | ✓ | ✓ | ✓ |
| 4 | 3 | Addresses topic | ||||||||
| 5 | 7 | Overall writing | ✓ | ✓ | ✓ | ✓ | ||||
| 6 | 10 | Sentence structure | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| 7 | 5 | Appropriate to audience | ? | ✓ | ✓ | |||||
| 8 | 2 | Assignment requirements | ||||||||
| 9 | 1 | Quality of content | ✓ | ✓ | ||||||
| 10 | 8 | Vocabulary usage | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| 11 | 11 | Punctuation/spelling | ✓ | ✓ | ||||||
| 12 | 12 | Vocabulary size | ✓ | ✓ |
| English Lecturers | Subject Lecturers | a | b | c | d | e | f | g | h | |
| 9 | 1 | Quality of content | ✓ | ✓ | ||||||
| 8 | 2 | Assignment requirements | ||||||||
| 4 | 3 | Addresses topic | ||||||||
| 2 | 4 | Development of ideas | ? | ✓ | ✓ | ✓ | ✓ | ✓ | ||
| 7 | 5 | Appropriate to audience | ? | ✓ | ✓ | |||||
| 1 | 6 | Paper organisation | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| 5 | 7 | Overall writing | ✓ | ✓ | ✓ | ✓ | ||||
| 10 | 8 | Vocabulary usage | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| 3 | 9 | Paragraph organisation | ✓ | ✓ | ✓ | ? | ✓ | ✓ | ✓ | ✓ |
| 6 | 10 | Sentence structure | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| 11 | 11 | Punctuation/spelling | ✓ | ✓ | ||||||
| 12 | 12 | Vocabulary size | ✓ | ✓ |
a. Mullen (1980)
b. Hughes (1989, p. 95) ELTS
c. Hamp-Lyons (1991a)
d. Hamp-Lyons (1991a) MELAB
e. Hamp-Lyons (1991a) Michigan Writing Assessment
f. Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981)
g. Hamp-Lyons & Henning (1991)
h. Hamp-Lyons (1991b)
Similarly, Norton (1990) compares students’ understanding of what lecturers want in assignments with the lecturers own views. There is some disagreement, but mostly they are in agreement:
| Mean ranking of students’ essays and tutors’ criteria for assessment of | ||
| Mean ranking | ||
| Criteria | Students(N = 98) | Tutors(N = 6) |
| Answer the question | 1.5 | 1.0 |
| Content/knowledge | 1.8 | – |
| Relevant information | 2.3 | 2.0 |
| Understanding | 2.4 | 1.3 |
| Wide reading | 2.9 | 3.7 |
| Structure/organisation | 3.3 | 2.8 |
| Argument | 3.5 | 1.5 |
| Evaluation/own views | 3.9 | 3.5 |
| Presentation/style | 4.2 | 3.5 |
| English/spelling | 4.8 | 4.0 |
What is common to these findings is that that subject lecturers value content. I do not think that most EAP marking profiles that I have seen deal with this adequately. Having a category giving, say, 20% to content does not seem to me to be satisfactory, as it is possible to ignore the content – or the question – totally and still pass the English exam. When I talk to examiners about this they always agree with me and say that – of course – they take that into account when marking,but the marking profile forms they use do not take it into account. Should they? Can they? How?
References
Bachman, L. F. & Savignon, S. (1986). The evaluation of communicative language proficiency: A critique of the ACTFL oral interview. The Modern Language Journal, 70, 380-390.
Bridgeman, B. & Carlson, S. B. (1984). Survey of academic writing tasks. Written Communication, 1, 247-280.
Hamp-Lyons, L. & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.
Hamp-Lyons, L. (1991a).Scoring procedures for ESL contexts.. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
Hamp-Lyons, L. (1991b). Reconstructing “Academic writing proficiency”. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 127-153). Norwood, NJ: Ablex.
Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press.
Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F. & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Rowley, Mass.: Newbury House.
Mullen, K. A. (1980). Evaluating writing proficiency in ESL. In J. W. Oller & K. Perkins (Eds.), Research in language testing (pp. 160-170). Rowley, MA: Newbury House.
Norton, L. S. (1990). Essay writing: What really counts? Higher Education, 20, 411-442.
Oller, J. (1979). Language tests at school. London: Longman.
Valette, R. M. (1977). Modern language testing (2nd ed.). New York: Harcourt Brace Jovanovich Inc.