12. Scripts, marking and adjudication

12.1. Marking

Marking of scripts and adjudication on the merits of candidates will proceed in accordance with the principles in this section, section 10, section 14, and the particular conventions of the examination concerned (see 7.3 above). Supervisory bodies should frame their policy according to these University requirements and may wish to refer to Annex H: Consistency in marking and classification conventions for second public examinations for a fuller rationale of these requirements.

Particular problems may arise as follows:

    (i)        Illegible Scripts:  If a chair considers a script to be illegible, he or she must inform the Senior Tutor of the candidate’s college as quickly as possible. If there is a dispute between the Chair and the Senior Tutor as to the illegibility of a script or scripts, the question should be referred to the Proctors for a ruling. Chairs will need to return any illegible scripts, by hand, to the candidate’s college asking for them to be typed. The college will either make arrangements to use the Schools’ transcription service or else will contact the Proctors for permission to type the script(s) in house. Chairs will be informed about the arrangements. The cost of the typing and invigilation shall not be a charge on the University.

   (ii)        Missing or incomplete scripts: If an examiner or assessor finds that a script is missing from the delivered package, or that a script is conspicuously incomplete, the chair should be notified immediately, so that a check can be initiated with the Schools and other markers. The Proctors should be informed promptly if it is not found.

  (iii)        Scripts with inappropriate content:  Where examiners feel that the content of a candidate’s script reveals signs of mental instability or the need for professional help, the chair should contact the Proctors’ Office for advice.

The scripts and submitted work of candidates whose dyslexia or other Specific Learning Difficulty has been notified through the Proctors will each have a cover-sheet advising markers of this fact and indicating in what ways the condition may have affected candidates’ written work. The scripts are to be marked as normal, but the marks-sheet should show when a candidate has an SpLD, so that this information can be considered by examiners in adjudicating on the candidate’s performance.

12.2. Standardised expression of agreed final marks

Numerical marking, which must be expressed in whole numbers for agreed final marks, will be used for both undergraduate and graduate examinations. All examination candidates will be provided with a transcript showing their final agreed marks for the individual examination papers or other examined elements of their degree programme, and (for undergraduate degrees) how these marks relate to the final degree classification.

Undergraduate courses

The University requires all examiners in Honour Moderations and second public examinations to express agreed final marks for individual papers (including those for formally assessed coursework) in the following form on the basis of the following class boundaries.

 

70 ­- 100

First Class

60 – 69

Upper Second

50 - 59

Lower Second

40 – 49

Third

30 – 39

Pass in Finals/Honour Mods

29 - 0

Fail

 

Examiners should be encouraged to use the entire range of the marking scale.

The transcript will show the final agreed marks according to the common scale and indicate the basis by which the classification is achieved. This may be on average mark alone, or may include specifications as to mark distribution (5 papers in the 2.1 class, etc.).

As Moderations and Preliminary Examinations are not classified, an amended version of this scheme is required as follows.

70 - 100

Distinction (where relevant)

40 – 69

Pass

39 – 0

Fail

 

Postgraduate taught courses

Following extensive consultation with divisions/Continuing Education, Education Committee has concluded that, with appropriate explanation on any accompanying transcript, two models for the award of agreed final marks within the categories of distinction, pass and fail are acceptable. Agreed final marks must be expressed in numerical form.

(a)

70 – 100

Distinction

50 – 69

Pass

49 –  0 

Fail

 

(b)       

70 – 100

Distinction

60 – 69

Pass

59 – 0

Fail

 

An overall award of distinction may be made to candidates who have shown excellence over the whole examination.  Examination conventions should make clear the rules for the award and these should normally exclude from consideration any candidate who has initially failed an assessment.  Exceptionally, responsible bodies may approve conventions that allow examiners to consider for distinction otherwise excellent candidates who have initially failed a minor assessment item (<=10%). Conventions should specify the element(s) that may be disregarded.

There is currently no provision for the award of a degree ‘with Merit’, other than the special provision for the First and Second BM.

12.3. Double-marking

Double marking has been adopted as University policy for judging the performance of candidates in the Second Public Examination (see below).  This good practice identifies discrepancies of judgement between two markers, which must then be resolved. It is not proper to average the two marks; the markers need to identify the reasons for the difference and agree an appropriate mark. If reconciliation is difficult, a third marker should act as arbiter in agreeing the appropriate mark. Only if such academic expertise is not otherwise obtainable within the University should an external examiner be asked to act in this capacity (see section 4 above).

There must be a mechanism to verify the marking of all papers for undergraduate second public examinations and graduate examinations under the aegis of all boards, and the minimum standard must be as follows:

In the case of papers for which there is a precise model solution and marking scheme approved by the examiners for every question, each script must be marked by an examiner or assessor; and every script must be checked independently (not necessarily by an examiner or assessor) to ensure that all parts have been marked and the marks and part-marks have been correctly totalled and recorded.

For papers without a model solution,

(a)    There should be an explicit process in place to ensure that a student’s mark is not affected by relatively severe or lenient marking;

(b)    The majority of courses in the University use independent double marking for this purpose;

(c)    Alternative methods such as those used by Jurisprudence are permitted, if it can be demonstrated to Education Committee that they serve the same purpose.

Every script or dissertation must be marked independently by two examiners or assessors (unless another marking method has been approved). Each division should have a consistent method across disciplines for reconciling the marks awarded by two markers.

Where subjects permit averaging of marks (over a narrow range) in reconciliation between markers, the system used must be clear and justifiable, and not operated to the detriment of candidates. If reconciliation is difficult, a third marker (perhaps – but not necessarily - the external examiner, see section 4 above) should act as arbiter in agreeing the appropriate mark.

Markers should generally not write on scripts during the marking process. This can compromise the independence of the second marker. In some subjects, however, the nature of the examination answers (such as translations or calculations) may be such that it is appropriate to indicate on the script objective errors for which the mark should be reduced.  Comments should not be written on the scripts but on the mark sheets provided for the purpose. 

Under the Data Protection Act, the University is not obliged to return scripts to candidates, but is obliged, if requested, to provide a transcript of anything written on them or separately about a candidate’s performance.

12.4. Scaling of marks

Education Committee considers that it is appropriate to scale marks for a paper where it has been established that either:

(a)   a paper was more difficult or easy than in previous years, and/or

(b)   an optional paper was more or less difficult than other optional papers taken by students in a particular year, and/or

(c)   a paper has generated a spread of marks which are not a fair reflection of student performance on the University’s standard scale for the expression of agreed final marks, i.e. the marks do not reflect the qualitative marks descriptors.

In each case, examiners need to establish if they have sufficient evidence for scaling. Different considerations need to be taken into account for each of cases (a), (b) and (c).

(a) A paper was more difficult or easy than in previous years

Examiners may wish to consider scaling where a paper has a higher or lower median or mean mark for a paper relative to previous years as this may indicate that the paper was easier or more difficult than intended, especially in a core paper taken by a large cohort.  However, this would not in itself constitute sufficient evidence for scaling. Scaling is not a mechanistic process but one which requires academic judgement. Further evidence should also be identified, for example, via

-               examiners’ academic evaluation of the performance of the candidates (possibly guided by qualitative descriptors of each class);

-               a comparison with the questions set in previous years’ papers; and/or

-               an analysis of the spread of candidates’ performance in compulsory papers compared to their performance in the paper in question.

Scaling should not be used mechanistically to fit the spread of classes on a paper to historical norms (i.e. norm referencing).

(b) An optional paper was more or less difficult than other optional papers taken by students in a particular year

Again, a higher or lower median or mean mark for an optional paper relative to other optional papers would not in itself constitute sufficient evidence for this. The differences in mean or median scores of students taking different optional papers could simply be the result of natural variation in ability within the cohort of students. If the number of students taking options is small, statistical analysis (say of performance of students in optional versus compulsory papers) can be an unreliable tool.

(c) A paper has generated a spread of marks which are not a fair reflection of student performance against the University’s standard scale for expression of agreed final marks.

Boards should take all steps which they consider to be reasonable academically to set questions and mark schemes which seek to generate a spread of marks that are a fair reflection of the performance of the student cohort on the University’s scale for standard expression of agreed final marks and the class descriptors set out in the examination conventions for the course.  However, it is recognised that despite the very best efforts at the examination setting stage, an examination, particularly in a quantitative subject where there is a precise model solution and mark scheme, may not generate such a spread of raw marks.  Scaling, with qualitative checks, may then be needed to translate raw marks to marks that are a fair reflection of the performance of candidates on the University scale.  Again, academic judgement will be critical here.

In all cases, the general principles below must be followed by all boards of examiners when scaling is used:

(i)   Scaling should only be considered and undertaken after moderation of a paper has been completed;

(ii)   If it is decided that it is appropriate to use scaling, examiners should review a sample of papers either side of the classification borderlines to ensure that the outcome of scaling is consistent with academic views of what constitutes a paper in each class. External examiners should be asked to report on this stage of the process;

(iii)   All scaling of marks must be done in the year in which the paper(s) in question is/are taken. This point will be particularly pertinent for subjects with second-year examinations and for bodies considering initiating such examination arrangements;

(iv)   Detailed information about why scaling was necessary and how it was applied should be included in the examiners’ report;

(v)   All examiners and boards should seek expert advice on the construction and operation of algorithms, where appropriate;

(vi)   All algorithms used for the purposes of scaling must be transparent and justifiable, and must be published as appropriate for the information of all examiners and students.

Examiners should also satisfy themselves that, if a computer algorithm is used in the classification process, its rules are fully consistent with the current conventions, especially if changes are being made to the conventions (see 7.3 above for further detail on examination conventions).

12.5. Comment sheets

Education Committee policy requires that comment sheets are used for all substantial assessment items. Substantial summative assessment item is understood to mean any thesis, dissertation, project report, extended essay, portfolio, research proposal, and any other summative assessment item that carries weight broadly equivalent to an unseen written exam.

Departments and faculties are encouraged to include the marking criteria on the marking sheet or book: additionally subjects may wish to offer further guidance to examiners on the coverage of their comments.

Section 12.13 below sets out the responsibilities of the chair of examiners in respect of the retention of the comment sheets along with other examination material. All material must be lodged with the chair who must make arrangements for its retention for two years following the examination.

12.6. Adjudication on the merits of candidates

The chair must arrange for all examiners and assessors to report the marks for those scripts they have marked. Marks are entered against candidates’ numbers on the marks sheet, and the examiners must then be provided with complete lists of marks that will form the basis of their adjudication. (Assessors do not take part in the final adjudication process but may be present in an advisory capacity only – see Examination Regulations, 2014, Regulations for the Conduct of University Examinations, Part 7, cl. 7.6, p. 20, ll. 6-9.)

Attention must be paid to the accuracy of data entry into marks spreadsheets and to ensuring that any changes in the list of candidates do not lead to knock-on errors. (Withdrawals are the most likely changes but the reinstatement of withdrawn candidates can also happen.) It is good practice to test new software on a set of dummy results before it is used in the examination.

During the process of adjudication, the scripts of all candidates should be available to the examiners as a whole. All examiners must be present at all Examination Board meeting unless prior permission has been obtained from the Proctors on the basis of exceptional circumstances – see 5.2 above.

12.7. Medical certificates and other special circumstances

Under Part 13 of the Regulations for the Conduct of University Examinations, information about medical or other circumstances affecting a candidate’s performance may be submitted by the candidate. For applications made before the final examiners’ meeting, the candidate’s college will send a completed application form to the Examinations and Assessments Section which will forward this to the chair. A subset of the board of examiners should meet to discuss all information received. (If necessary, in smaller departments, this may need to be undertaken by the entire exam board.) At this meeting each candidate’s circumstances should be considered and banded on a 1-3 scale to indicate the seriousness of the impact of the circumstances on the candidate’s performance with 1 indicating minor impact, 2 indicating moderate impact, and 3 indicating very serious impact. Examiners should also determine whether all or a subset of papers have been affected. A formal note should be kept of these decisions. The banding information should be used at the final meeting of examiners to adjudicate on the merits of candidates. Annex B: The use of medical and other certificates in examinations and assessment provides further information on the courses of action open to examiners and the detailed procedures to follow.

Information received by the Examinations and Assessments Team after the final examiners’ meeting has taken place will be sent to the Proctors. The Proctors will only forward medical or other information to the chair if one of the following criteria is met:

  • The candidate’s condition was such as to prevent them from making an earlier submission;
  • The candidate’s condition was not known or diagnosed until after the final meeting of the examiners;
  • There has been a procedural error that has prevented the candidate’s information from being submitted.

As the use of information about candidates’ special circumstances is increasingly the subject of query or complaint, boards are asked to keep a brief, formal note confirming that special circumstances information has been taken into account where an application has been received under Part 13, and indicating the outcome. Further guidance on reporting is provided in Annex B: The use of medical and other certificates in examinations and assessment.

12.8. Incomplete examinations

If a candidate does not complete an examination, under Examination Regulations, 2014, Regulations for the Conduct of University Examinations, Part 13, cl. 13.5 (pp. 28-29), the Proctors can:

  • Authorise examiners to examine the candidate at another place or time under such arrangements as they deem appropriate. Or, if the candidate has already submitted sufficient other work, to act as if he or she had completed the part of the University examination which he or she was unable to attend, and to award an estimated classification; or use the words ‘declared to have deserved Honours’ in the case of a classified degree if the examiners are unable to classify the candidate but none the less judge that the candidate would have obtained Honours if he or she had been able to complete the examination; or else fail the fail candidate.
  • In the case of a pass/fail degree, the examiners must consider whether the candidate has submitted sufficient work of sufficient merit for them to judge whether it is appropriate to award a pass.

Where a candidate has missed one or more papers in a First Public Examination, chairs will be instructed to examine the candidate during the Long Vacation, i.e. when they would normally provide re-sits if a candidate had failed.

12.9. Use of vivas

Examiners (and, if invited, an assessor) may examine a candidate viva voce in a University examination only where the specific regulations make provision for the use of vivas. Examiners should be clear as to the purpose of a viva voce examination, for example it should not be used as a means of assessing suspicions about possible plagiarism, which should be referred directly to the Proctors.

If examiners, following Examination Regulations, intend to call some or all candidates for a viva voce examination, the dates should be included as accurately as possible in the chair’s circular to candidates early in the year of the examination.  When examiners have retained the option of vivas, any request from a candidate for dispensation from the possibility because it conflicts with travel or vacation plans will be refused; the Junior Proctor may, however, seek from the chair an indication of the probability of a viva voce examination, so that the candidate may judge the risk involved in travelling at the specified date. If examiners are certain that they will not hold vivas at all, this can be communicated to candidates.  A viva need not be held on a failing candidate if it is not specified in the requirements of the course and the failure is beyond any margin of doubt.

When examiners call candidates for viva voce examinations, the conduct of the viva should be sufficiently formal to ensure fairness of treatment for all candidates examined in this way.  Notes must be kept of the questions asked, together with an indication of the level of response, and assessment made at the time. This material must be given to the chair of examiners (see 12.11 concerning the Data Protection Act).

12.10. Sharing information between examiners

Marks should be transported between examiners and assessors and the chair or relevant administrative officer by hand, or sent by Recorded Delivery. They should not be sent electronically unless encrypted. Advice on the encryption and decryption of marks sheets may be obtained from IT Services. No electronic transmissions should be made without previously informing the Proctors. As part of a pilot study, some boards of examiners have been authorised by the Proctors to use WebLearn for sharing marks and submitting marks to the chair. This pilot scheme is now available to other boards of examiners, and the chair should contact the Examinations Manager to discuss the practicalities of implementation.

Any exception to these rules must be agreed in advance by the Proctors, who will need to be convinced (taking technical advice if necessary) that it will cause no breach in security.

It is not good practice to use e-mail as a method of communicating between examiners about examination matters. The condensed style of e-mail communication is open to ambiguity and can give rise to errors.  Examiners should note that email communications about individual students would be disclosable under the Data Protection Act.

12.11. Confidentiality and the Data Protection Act

Confidentiality

Examination scripts and raw marks (i.e. the marks from individual examiners before agreement or reconciliation) are strictly confidential and in no circumstances may be shown to or discussed with anyone other than examiners or properly appointed assessors.  Details of the discussions at examiners’ meetings are equally confidential.  Apart from the chair, only authorised administrative staff may process the entry of marks and otherwise assist in the handling of information.

Data Protection Act

Under the Data Protection Act 1998 candidates are entitled to access to personal data concerning them, including confidential information arising in the examining process. In order to obtain information under the Act, a candidate must make a formal application through the University’s Data Registration Officer: under no circumstances should examiners, assessors, or administrative staff respond to direct requests for disclosure of information relating to the examination. If a data subject access request is made, access must be provided either within five months of the request or forty days after the official release of results (whichever is the sooner). The University must disclose to the candidate on request all personal data generated as part of the examination process except where it can be established that a duty of confidentiality has been created in respect of a third party.  Under the Act, candidates are not entitled to obtain their scripts or copies of them, but if a valid application is made for disclosure they are entitled to receive:

  •  the published results and all marks held,
  •  copies of markers’ comments on their work, and
  • (if identifiable separately from other individuals) comments recorded about their performance, whether by name or candidate number, in material presented to or in the minutes of examiners’ meetings.

12.12. Disclosure of marks and transparency

The Proctors take the view that all agreed marks awarded should be disclosed to students, on the grounds of both the Data Protection Act (1998) and the educational value of feedback on assessed work. Evidence of recent medical problems etc. should have been considered at this stage in the process (see Annex B: The use of medical and other certificates in examinations and assessment). Degree classification will continue to be a matter for boards in accordance with their examination conventions using the complete set of standardised marks.

12.13. Retention of records

Supervisory bodies should ensure that all examiners acting on their behalf are aware of the Proctorial requirements relating to the retention of records.

In the light of the Data Protection Act, the Proctors instruct the examiners, once the examination is complete and the final list produced, to do the following as regards retention of records:

  • Each examiner and assessor, including external examiners, should hand over to the chair of examiners all material in their possession pertaining to the marking of the exam.  This includes mark sheets, mark books, and notes on individual written answers and on vivas. They must sign a declaration that they no longer have such material in their possession (see Annex B: The use of medical and other certificates in examinations and assessment).
  • The chair of examiners should retain these records, and records relating to the adjudication of candidates, including records of those scripts which were remarked or on which the external examiner’s advice was sought, and the outcome; all medical evidence including a note, which must be countersigned by external examiner(s), of any actions taken in reaching the final marks and degree results; the declarations of examiners. All this material must be deposited with the nominated administrative officer and retained for two years. Duplicates and confidential waste must be destroyed (this means shredded, not put intact in a bin).
  • Appeals against final results are normally only possible within 3 months after the publication of results, although it is prudent to keep all records for two years.Records should not be destroyed if an appeal to the Proctors is in progress.    
  • If, during an examination itself or during classification of results, any automated processing or weighting of results takes place, then the chair should be able to provide a formal statement that explains the logic behind any such processing.  If an assessment itself is based partly on automated means (e.g. multiple choice tests) which form only a part of a larger assessment, the procedure for combining different sets of marks should be available.
  • The above arrangements apply to each completed examination (including the previous parts of a multi-part FHS).  In a multi-part examination, scripts and other submitted work, and notes on performance, must be retained (for consideration by the following year’s examiners) until all parts of the examination are complete.

It remains the University’s policy that results (i.e. the final marks that have been agreed, scaled and moderated) should be made available to candidates through their colleges (and via the student record system) at the end of the examination process. Raw marks should not be disclosed to colleges or candidates.

12.14. Computer records

The Proctors remind examiners that, in accordance with the requirements of the Data Protection Act, personal data should not be retained unnecessarily after the purpose for which it was collected has been fulfilled. Items which it is necessary to retain in electronic form include only such data to which there would be no objection to a candidate’s having access, including his or her final class and agreed marks on individual papers. This data should be retained for two years after the final examiners’ meeting. These provisions have particular implications for chairs of examiners in Schools with multiple parts. In these cases, records should be retained for two years after the final examiners’ meeting for the final part.

Data from the examination may be kept in electronic form for subsequent purposes (e.g. evaluating proposed changes in examining procedures, such as changes in paper weightings) beyond the time limit mentioned above provided the information is depersonalised or does not contain items that should not be divulged (such as raw marks).

12.15. Records for the future

The nominated administrative officer should retain copies of all the agreed marks, and any notes relating to them, for two years after the final meeting.  This is necessary in order to deal with examination queries and complaints, and to handle cases in which candidates carry forward their marks in the written papers to the following year, for example, when resubmitting only their dissertations.

Outgoing chairs should keep records for the assistance of future chairs on matters such as marking conventions, special problems encountered, and arrangements made with the Head of Examinations and Assessments.  The records should take the form of a chair’s book, which can be passed on each year.  It is very important that chairs transmit to their successors any medical certificates and ongoing permissions (i.e. in relation to dyslexia arrangements) for candidates likely to be resitting the examination, or any element of it.  It is preferable if these materials are deposited with a member of the Faculty/Departmental secretarial staff who will arrange for onward transmission.

12.16. Security, deposit and retention of scripts

Scripts in the possession of examiners and assessors must be locked away, particularly before they have been marked.

Where examiners’ meetings are held at the Schools, scripts should be returned to the Schools (not using messenger post) after marking so that they are available when the examiners meet to adjudicate on candidates’ performance.

All examination scripts must be returned to the Examination Schools.  If the examiners’ meetings do not take place in the Schools, scripts should be returned as soon as the examination is completed and in any case no later than the end of 4th week of the following Michaelmas Term.  They should be packaged securely in boxes, clearly labelled with the name of the examination and the term and year, with the course code/paper numbers and in candidate number order.  They should also clearly indicate the date after which they can be destroyed (usually six months after the final examination board). In addition to the labels on the boxes, a list of all candidate numbers for each examination should be provided to the Examinations and Assessments team, preferably with a list of all papers that have been taken by the candidates whose scripts have been included in the boxes. This will enable easier retrieval in the case of queries.  If examiners require them at a later date (e.g. for consultation in Part II of an examination), the Head of Examinations and Assessments should be informed and the scripts must be clearly marked (e.g. ‘TO BE RETAINED FOR PART II’).  All scripts should be retained for six months after the examination so they are available to be checked in the event that a candidate complains to the Proctors about an examination result.

If a candidate submits a complaint or appeal to the Proctors about their examination, all their scripts must be held separately and retained for two years after the completion of the investigation or of the candidate’s course, whichever is the longer.

After the release of the results for the examination, copies of theses that are required to be deposited at the Bodleian Library should be delivered to the Head of Examinations and Assessments, who will record them and arrange delivery. Where theses are deposited with another library (e.g. in the department), chairs should make the necessary arrangements.