GROUP TWO
REPORT #1
INTRODUCTION
Group 2 has been asked to advise a university academic school on a number of issues relating to assessment.
We have identified two general themes that appear to be having an impact on assessment in the school:
- Quality assurance and quality enhancement
- Inclusivity
These general issues have manifested in a number of specific problems which we wish to consider under two headings:
- Internal inconsistency – variance in marking between different modules within the school
- External inconsistency – variance between grades awarded by the school and the sector average
1.0 GENERAL THEMES
1.1 QUALITY ASSURANCE AND QUALITY ENHANCEMENT
1.1.1 DISCUSSION
Bloxham and Boyd (2007) consider four purposes of assessment: certification, student learning, quality assurance and lifelong learning capacity. Quality assurance enables judgements to be made on the appropriateness of standards and assessment of learning (Gibbs 1999). However these purposes of assessment can often be in conflict with each other (Biggs & Tang 2011). When assessment tasks are designed to show students’ knowledge of intended learning outcomes, declarative knowledge can be assessed through essays and exams and assured through the use of rubrics for example (see 2.1.1) This offers strong evidence of quality assurance as measurable, comparable and consistent outcomes can be gained and moderated in line with the QAA indicators of sound practice (QAA 2013).
These methods are efficient and assume that assessment is objective and can accurately show learning has been achieved in a scientific manner (Eisner, 1983). While a reductionist approach may be possible in the natural sciences, which only look for standard outcomes (Koetting, 1988), it can be seen as over simplified when applied to assessment in areas such as arts and humanities (Eisner, 1983). Traditional assessment of learning methods which focus on the monitoring function of HE (McDowell 2012) and that learning has taken place, can lead to a reductionist and fragmented approach using solutions that do not take an informed pedagogical approach (Quinn 2010). However, an assessment for learning approach is thought more relevant (McDowell 2012) as knowledge acquisition is recognised as infinite and dynamic, with learners expected to constantly acquire new knowledge independently and solve problems (Birenbaum 1996).
A pedagogic method which incorporates assessment as learning is the constructive alignment method (Bigg & Tang 2011). This uses assessment as part of the teaching approach to promote students’ active engagement in appropriate learning for the intended learning outcomes of the course. Through the assessment of solving complex problems, team-work and effective communication it provides methods for assessing student learning in essential skills (Biernbaum, 1996) thus enhancing student engagement and quality enhancement (Bigg & Tang, 2011).
1.1.2 RECOMMENDATIONS
- A institutional wide approach should be taken to quality assurance and quality enhancement including;
- A holistic pedagogic approach to teaching, learning and assessment in line with the literature.
- Consideration of the QAA indicators of sound practice which incorporates the subjects outlined below.
1.2 INCLUSIVITY
1.2.1 DISCUSSION
The evidence obtained from the academic school indicates a lack of an inclusive teaching and assessment environment for students – specifically a reluctance to use rubrics or to share marking criteria with students.
Indicators of an inclusive environment include embracing student perspectives in developing rubrics and ensuring that all students, irrespective of their background, are capable of achieving good degrees. Before moving on to discuss how the School can adopt a more inclusive approach to assessment, we must first acknowledge that there is a problem with defining inclusivity (Wray, 2013) in HE. Some institutions use an older definition which argues that an inclusive teaching environment is only required for disabled students. More progressive institutions use a definition that includes a wide group of students when designing inclusive teaching and assessment environments (Hockings, 2010). This group includes students who are the first in family to go to university, international students, mature learners and students from religious and ethnic minorities.
Underpinning the concept of inclusive teaching are the values of fairness and equality: the UKPSF devotes two out of four values to inclusivity (UK PSF, V1 and V2), which highlight the importance of respecting and promoting equality of opportunity for a diverse student population.
1.2.2 RECOMMENDATIONS
- Develop a statement on inclusivity, involving both staff and students, to clarify expectations on what is meant by an inclusive approach to teaching and assessment. This should be based on existing best practice statements from other HE providers (Hanesworth, 2015). The School could then assess the level of inclusivity in existing teaching and assessment mechanisms (Ward, 2009).
- Review existing teaching and assessment mechanisms, to understand how inclusive existing practice is at a module level. This could include principles of Universal Design in assessment design (Hockings, 2010), which is a concept borrowed from designing buildings for disabled people, whereby the design principle is “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design.” (Center for Universal Design, 2008).
- Develop an understanding that moving to an inclusive agenda will be a difficult process, involving understanding and challenging the personal and professional assumptions we bring to teaching (Burke & Crozier, 2012). However, the School can learn from other HE institutions, who have revitalised their inclusivity agenda, and can provide practical recommendations (Thomas and May, 2010).
2.0 SPECIFIC ISSUES
2.1 INTERNAL INCONSISTENCY – VARIANCE IN MARKING BETWEEN
DIFFERENT MODULES WITHIN THE SCHOOL
2.1.1 DISCUSSION
Marking is a largely subjective and unreliable practice (Bloxham 2009; Bloxham, den-Outer, Hudson,& Price, 2015). This is clearly manifested in the variance in marking between different modules within the school which may be attributed to a range of factors including:
- Varying knowledge and experience of tutors (Read, Francis and Robson, 2005; Smith and Coombe, 2006) in addition to personal values (de Fries, 1999).
- Staff applying their own criteria and ignoring institution standards (Price & Rust, 1999).
- Difficulties in applying generic guidelines to specific pieces of work (Price, 2005)
- Some tutors judge the value of certain aspects as more important than others, for example, accurate grammar (O’Hagan & Wigglesworth, 2015).
The role each factor play requires more detailed examination to properly assess the impact on the different types of assessment used by the school. However, it is clear that the use of rubrics in assessments has been inconsistent across the school and contributed to the internal and external insistencies seen in the school’s performance.
Use of rubrics
A commonly used definition of a rubric is a document that articulates the expectations for an assignment by listing the marking criteria and describing levels of quality from excellent to poor (Reddy and Andrade, 2010). They can be used to improve reliability of assessment and the quality of learning.
Reliability of assessment:
The appropriate use of rubrics can ensure a greater consistency in marking practices. The implementation of rubrics can address some of the barriers to ensuring reliability of assessment including the lack of consistency of individual markers (intra-rater reliability) and the lack of consistency between markers (inter-rater reliability).
One of the major barriers to reliability of assessment is the lack of consistency of an individual marker themselves (Brown, Bull and Pendlebury, 1997 and Jonsson and Svingby, 2007). Although there are few studies on intra-rater reliability, those that have been undertaken indicate that rubrics seem to aid raters in achieving high internal consistency when scoring performance tasks (Jonsson and Svingby, 2007).
There are several studies which focus on inter-rater reliability. Whilst there is ample evidence of disagreement between assessors using rubrics (Oakleaf, 2006; Boulet et al, 2004) several studies have shown that rubrics can allow instructors to reliably assess performance (Reddy and Andrade, 2010). Factors that influence inter-rater reliability include benchmarking, training of staff and the use of topic specific rubrics.
Quality of students’ learning:
Students who have rubrics to guide their work generally show higher achievement and deeper learning (Reddy and Andrade, 2010). A major theme in the research has been that of transparency for students i.e. if students are clear about what is expected of them, the assessment will be more meaningful. Hence students need to develop a clear understanding of each element of the assessment task and how this is going to be assessed.
2.1.2 RECOMMENDATIONS
- Development and implementation of programme level and module level rubrics as agreed by subject specialists within the teaching team. It is possible for academic teams to develop shared standards (Elwood & Klenowski, 2002).
- Training, support and/or mentoring on marking for inexperienced or new academics.
2.2 EXTERNAL INCONSISTENCY – VARIANCE BETWEEN GRADES
AWARDED BY THE SCHOOL AND THE SECTOR AVERAGE
2.2.1 DISCUSSION
Concerns over quality assurance and standards in Higher Education became a focus of attention in the mid 1980s (Green, 1994) and led to increased efforts in assessment research especially in the area of measurement instruments, validity, reliability, and controlling bias (Joughin, 2010).
The target of a positivist researcher is to ensure maximum objectivity in assessment methods to ensure the quality of the process as well as the output (Orr, 2007). In order to ensure measurement of assessment as a reliable tool of for grading the overall performance of students, it must be inclusive in nature (Keating, Zybutz, & Rouse, 2012; Davies & Elliott, 2012).
This academic school is awarding 5-10% fewer good degrees (2:1 and first) than comparable courses and lower than the sector average. This indicates a lack of students’ awareness of and involvement in the assessment process (Boudrias, Bernaud, & Plunier, 2014). Research has shown Grade Point Average (GPA) is the major item of value that institutions can offer (Joughin, 2010) so addressing this external inconsistency in assessment could create an opportunity to students to graduate with high class of quality degrees.
Students missing out on higher classifications by 1% or less
A key feature of the inconsistency with the sector average appears to lie with the number of students missing out on higher classifications by 1% or less. This suggests a fundamental problem is a failure of assessment to produce sufficient distinctiveness resulting in students missing out on higher classifications by meaningless margins of 1% or less. The use of percentages as opposed to broader grades amplifies this issue. Snyder’s ‘hidden curriculum’ (1971) appears to be alive and well in this school. Joughin (2010) may challenge the validity of the concept of the ‘hidden curriculum’ but it seems likely there are staff expectations which exist over and above the formal, declared curriculum.
This gives rise to a discussion of the validity of percentages as a grade descriptor. Bloxham (2009) describes a ‘positivist‘ approach in which assessment is seen as a process of divining an objective truth measured against objective standards – this appears to be the underlying approach taken in this school where less than 1% difference is held to represent a meaningful and objective distinction between levels of student performance. While this may be the ambition a greater acceptance of the influence of what Bloxham describes as an ‘interpretivist’ approach may be useful in solving this specific problem, thereby accepting the role of tutor preconceptions, expectations (both declared and undeclared) and prejudices.
Finely applied percentage grades have superseded broad marking categories (A, B, C etc) but, at the margins, these simply codify subjectivity – a 39 is not good enough, a 40 is. Such fine degrees of distinction are unhelpful and suggest a process of dogmatic summative assessment which is unsupported by constructive formative assessment to help students decode that hidden curriculum.
2.2.2 RECOMMENDATIONS
This issue requires proactive management and a fundamental review of the assessment process. The response should involve:
- A recognition of the importance of inclusive assessment to Increase student involvement in the assessment exercise
- Providing timely formative feedback to students could enhance students’ performance
- Review marking policies to encourage distinctive assessment (e.g peg marking).
- Review moderation policies (e.g specifically moderate marks at or close to the grade boundary to make a clear judgement of classification). To develop inclusivity, this targeted moderation process could be done by students.
3.0 ACTION PLAN
Common themes have emerged through the consideration of the use of assessment in this academic school. These should form the basis for an action plan to address the internal and external inconsistencies identifies. These themes are:
QUALITY ASSURANCE
This is a fundamental issue and must be considered both in terms of staff and students. A process of consultation, training and refinement is required to create a shared understanding and common objectives among all school members (staff and students).
INCLUSIVITY
The information provided suggests tutors are applying their own specific assessment policies and these are not being made clear to students. There is an opportunity to embrace the opportunities offered by a more inclusive approach (detailed above) to enhance both consistency and performance.
MARKING POLICY
Rubrics have a significant role to play in reducing internal inconsistencies – the appropriate design of marking criteria and grade descriptors will assist in creating clear expectations.
BIBLIOGRAPHY
Biggs, J. B., & Tang, C. (2011). Teaching for quality learning at university: What the student does. Maidenhead: McGraw-Hill Education.
Birenbaum, M. (1996). Assessment 2000: Towards a pluralistic approach to assessment. In M. Birenbaum & F. Dochy (eds.). Alternatives in assessment of achievements, learning processes and prior knowledge. (pp. 3-29). doi: 10.1007/978-94-011-0657-3
Bloxham, S., & Boyd, P. (2007). Developing Effective Assessment in Higher Education. Maidenhead: Open University Press.
Bloxham, S. (2009). Marking and moderation in the UK: false assumptions
and wasted resources. Assessment & Evaluation in Higher Education, 34(2), 209-220. doi: 10.1080/02602930801955978
Bloxham, S., den-Outer, B., Hudson, J., & Price, M. (2015). Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assessment & Evaluation in Higher Education, (ahead-of-print), 1-16. doi: 10.1080/02602938.2015.1024607
Boudrias, J.S., Bernaud, J. L., & Plunier, P. (2014). Candidates’ integration of
individual psychological assessment feedback. Journal of Managerial Psychology,
29(3), 341–359. doi: 10.1108/JMP-01-2012-0016
Burke, P. J. & Crozier, G. (2012). Teaching Inclusively: Challenging Pedagogical Spaces. York: Higher Education Academy. Retrieved from http://www.roehampton.ac.uk/uploadedFiles/Pages_Assets/PDFs_and_Word_Docs/Research_Centres/CEREPP/Teaching-Inclusively-Resource-Pack.pdf
Davies, M., & Elliott, S. N. (2012). Inclusive Assessment and Accountability: Policy to
evidence-based practices. International Journal of Disability, Development and
Education, 59(1), 1–6. doi: 10.1080/1034912X.2012.654932
Eisner, E. W. (1983). Educational Connoisseurship and Criticism. In Evaluation models (pp. 335-347). Springer Netherlands.
Elwood, J., & Klenowski, V. (2002). Creating Communities of Shared Practice: The challenges of assessment use in learning and teaching. Assessment & Evaluation in Higher Education, 27(3), 243-256. doi: 10.1080/02602930220138606
The Higher Education Academy. (2011). The UK Professional Standards Framework for teaching and supporting learning in Higher Education, Universities UK. Retrieved from
Joughin, G. (2010). The hidden curriculum revisited: a critical review of research into the influence of summative assessment on learning. Assessment & Evaluation in Higher Education, 35(3), 335-345. doi: 10.1080/02602930903221493
Koetting, J. R. (1988). Educational Connoisseurship and Educational Criticism: Pushing beyond Information and Effectiveness. In: Proceedings of the Annual Meeting of the Association for Educational Communications and Technology : 14-19 January 1988, (pp.1-15). New Orleans.
McDowell, L. (2012). Assessment for Learning. In L.Clouder., C.Broughan., S.Jewell., G.Steventon (Eds.), Improving student engagement and development through assessment; theory and practice in higher education (pp. 73- 85).
Quinn, J. (2010). Learning communities and imagined social capital: Learning to belong. London; New York: Bloomsbury Publishing.
Smith, E., & Coombe, K. (2006). Quality and Qualms in the Marking of University Assignments by Sessional Staff: An Exploratory Story. Higher Education, 51(1), 45-69. doi: 10.1007/s10734-004-6376-7
Snyder, B.R. (1971). The hidden curriculum. New York: Knopf.
Green, D. (1994). What is Quality in Higher Education ? What is Quality in Higher Education?
Jonsson, A. and Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences, Educational Research Review 2, 130-144. Doi: 10.1080/00131911.2014.929565
Keating, N., Zybutz, T., & Rouse, K. (2012). Inclusive assessment at point-of-design.
Innovations in Education & Teaching International, 49(3), 249–256. doi:10.1080/14703297.2012.703022
Knight, P. T. (2006). The local practices of assessment. Assessment & Evaluation in Higher Education, 31(4), 435–52. doi: 10.1080/02602930600679126
O’Hagan, S. R., & Wigglesworth, G. (2015). Who’s Marking My Essay? The Assessment of Non-native-speaker and Native-speaker Undergraduate Essays in an Australian Higher Education Context. Studies in Higher Education, 40(9). doi:10.1080/03075079.2014.896890.
Orr, S. (2007). Assessment moderation: constructing the marks and constructing the
students. Assessment & Evaluation in Higher Education, 32(6), 645–656. doi:10.1080/02602930601117068
Price, M. (2005). Assessment standards: The role of communities of practice and the scholarship of assessment. Assessment & Evaluation in Higher Education, 30(3), 215–30. doi: 10.1080/02602930500063793
Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133–44. doi :10.1080/1353832990050204
Read, B., Francis, B., & Robson, J. (2005). Gender, ‘bias’, assessment and feedback: analyzing the written assessment of undergraduate history essays. Assessment & Evaluation in Higher Education, 30(3), 241-260. doi: 10.1080/02602930500063827
Reddy, Y. M. & Andrade, H. (2010). A review of rubric use in higher education. Assessment and Evaluation in Higher Education, 35(4), 435–448. doi: 10.1080102602930902862859
Hanesworth, P. (2015). Embedding equality and diversity in the curriculum: a model for learning and teaching practitioners. York: Higher Education Academy.
Hockings, C. (2010). Inclusive learning and teaching in higher education: a synthesis of research. York: Higher Education Academy.
Thomas, L. & May, H. (2010). Inclusive teaching and learning in higher education. York: Higher Education Academy.
Ward, C. (2009). DIY Toolkit for Alternative & Inclusive Assessment Practice, Nottingham Trent University
Wray, M. (2013). Developing an inclusive culture in higher education: final report. York: Higher Education Academy.