Rowin’s blog

You may remember a proposed standard for the transfer of qualification assessment data and evidence that was previously covered on this blog.

Work on this has been ongoing since then, and a draft standard is now available for public consultation and comment.  The public draft can be accessed via the BSI website, and comments may be submitted by following the instructions there.

All comments must be submitted by 30 November to be considered for the final version of the standard.  Depending on the nature and extent of comments received, the standard is likely to be released in the first quarter of 2010.

Assessment, Standards and specifications, e-Portfolio (Technology), portfolio

Around 35 delegates struggled through Wednesday’s sweltering heat and the baffling mysteries of Manchester Metropolitan University Business School’s internal layout to discuss a range of issues around competences for learning, assessment and portfolio.  Delegates represented a wide range of knowledge and expertise, from novices looking to find out ‘what it’s all about’ to experienced practitioners and developers.

It was an impressively international turn out, with delegates from Norway, Greece, Austria, Spain and Belgium joining the UK contingent, mainly representing the iCoper project which is exploring the linking of assessment with competences.  Assessment interests were also represented by the University of Southampton, who are working on the automatic construction of statements of competency from QTI XML, exploring the underlying modelling of competencies for machine processing.  The majority of delegates came from a strong (e)portfolio background, with interests in the movement of information into and out of eportfolios.  JISC and CETIS participants also highlighted the relevance of this work to JISC’s Curriculum Design projects.

The morning session featured a number of short presentations (all presentations from the day can be found here) on competences requirements in the field of medical education, an area which is relatively advanced in the use of competence frameworks.  Claire Hampshire (MMU), Julie Laxton (ALPS CETL), Karen Beggs (NHS Education for Scotland) and Jad Nijjar (iCoper and Synergetics) covered a range of topics, including the desire for non-hierarchic representations, the management of massive amounts of data, and addressing the various points in a student’s career at which  information can move between one system and another.  The ownership of data in portfolios, including competency information, is an ongoing issue that still is not clear, with at least three actors involved: the data subject, data controller and data processor.  Three main points of interoperability were identified: across time (for example, undergraduate to postgraduate), across specialities (for example, from psychiatry to gynecology), and from elearning experiences to portfolios.

After coffee, Paul Horner (Newcastle University), Shane Sutherland (PebblePad), Dave Waller (MyKnowledgeMap) and Tim Brown (NHS Education for Scotland) delivered short presentations on various tools for handling competence information.  One key issue that emerged from this session was the strong need for a specification to enable the sharing of profiles between systems: while evidence can be exported as HTML, entire profiles cannot be moved between systems except in unwieldy formats such as .pdfs.  Interoperability is needed for both import and export.  There is a noticeable move away by customers from monolithic approaches towards using a variety of (Web 2.0) tools, and developers are working on building open APIs to support this. 

What struck me most from both sessions was the way in which developments around eportfolios and competence recording are very firmly rooted in actual teaching and learning practice, with requirements emerging directly from real-world practice and tool developments directly benefiting teachers and learners.

In the afternoon the meeting split into four groups, ostensibly to work on identifying and representing information structures for a purported competences specification.  In practice, my group spent most of our time discussing widely around the whole area of competences, eportfolios and assessment, but as a newbie in this field I found this hugely helpful.  Overall conclusions from the groups identified the following requirements and issues:

• ability to transfer information between different tools and systems
• transition
• curriculum progression pathways
• relationship between competences and evidence versus qualifications
• repeatable pattern of description at the core
• fairly simple structure
• identifiers for defining authority
• a definable core structure enables extension for extra semantics
• able to express the relationship between a learning object and skills, competences and knowledge
• collection of outcomes
• architectural issues: data is created and needed in many locations instead of at a central point
• competences are highly context dependent

The meeting concluded with asking delegates what they want CETIS to focus on in taking forward work on competences.  Suggestions included:

• development of a data model
• business case for interoperability
• look beyond HE/FE to workplace standards, particularly in the HR domain
• look for connections to the HIRA progress reports due out by November
• look at what has failed so far in order to learn from past experiences
• look at defining competences in such a way that a specification can be combined with XCRI
• have loosely defined competences that can be moved between systems
• need a high level map of the competency domain in comparison with curriculum description and learning objects.

CETIS will be looking at how best we can take this work forward and, as always, we very much welcome input and suggestions from our community - please feel free to leave comments here, follow up via the wiki or contact Simon or me!

Assessment, General, ccomp, competences, portfolio

A recent joint meeting of the JISC CETIS Assessment, Enterprise and Portfolio SIGs drew a wide range of participants to discuss topics of interest to all three SIGs.  The morning sessions covered a range of topics that touched on all three domains, while the afternoon was given over to a special session on student retention.

John Winkley of AlphaPlus Consultancy, who has been working with JISC as an expert consultant in the area of assessment, opened the meeting by introducing delegates to a number of funding opportunities in the domain that JISC will be releasing in the next few weeks.  These opportunities include at least two and up to four demonstrator projects, funded to build on and further develop outputs from earlier JISC toolkit activities, and two Invitations to Tender for desktop research studies.  These studies will look at advanced eassessment techniques, and at quality concerns around eassessment.  The demonstrator projects must be led by a HEFCE-funded institution, while the ITTs will be open to all bidders including Scottish insitutions, FE colleges with less than 400 HE students, and the private sector.  All work is due to be completed by March 2009, and will add considerably to JISC’s portfolio of work in this area.

One project which has benefited from JISC funding for part of its lifetime is the WebPA project based at Loughborough University.  Nic Wilkinson presented the successful peer assessment system to delegates, illustrating some of the reasons for its success at the recent IMS Learning Impact Awards in Austin, Texas.  One of the most signficant factors in the system’s ongoing success is the effort the project team have put into attracting and supporting a signficiant number of participating organisations that have now integrated the system into their own teaching practice.  It was also extremely interesting to learn how positively the students themselves have responded to the system, and their attitudes towards the anonymity of peer marks: the system awards each member of a group an aggregated mark derived from the individual scores awarded by their peers, and students are reported to not want to receive individual marks in order to avoid potential clashes outside the classroom.

After the break, Karim Derrick of TAGLearning discussed a proposed British Standard for managing the transmission of coursemarks and portfolios of digital evidence of coursework between schools and awarding bodies.  Based on TAG’s extensive experience in this area, the proposed standard includes ‘an XML schema for describing the relationship between components, options and exam specifications’ and a ‘universal translator’ API to support data exchange between the various systems used by exam centres and awarding bodies.  Although the current focus for this work is firmly on the schools sector, if adopted it’s not hard to see how it could be extended to support the universities admissions process and external marking at all levels, particularly in vocational courses where a single accrediting body has to deal with substantial amounts of data. 

Alan Paull of APS Ltd closed the morning with a lively journey round the admissions domain landscape and the DELIA project.  DELIA enables the sharing of enhanced learner information as part of the admissions process, enabling admissions officers to make more informed decisions when evaluating borderline applications.  This not only improves the quality of the admissions process, enabling a closer matching between applicants and course requirements, but can have a positive impact on subsequent retetention of such students.

The afternoon featured a special session on student retention, looking at a range of issues around the topic and attempting to capture requirements for work in the domain.  Simon Grant of JISC CETIS and the Centre for Recording Achievement led an interactive session that asked participants to consider self-assessment of suitability for courses and the different personas we adopt as our contexts change.  Simon also touched on some of the problems that arise when our different personas come into conflict, a situation which can be exacerbated by the widespread use of social networking services and individuals’ lack of awareness of the potential implications of forgoing privacy when using them. 

Helen Richardson, also of JISC CETIS and the Centre for Recording Achievement, closed the day by discussing some of the findings of the STAR project and the National Audit Office’s report on student retention.  The STAR project produced a detailed series of guidelines to help support students both before and during their university careers, including the use of technologies such as SMS messaging to aid this.

We’re grateful to all our presenters for sharing their work with us and for being so willing to respond to questions and comments from the audience, and to all those who attended on the day and helped to make it a success.

Assessment, assentport_May08, enterprise, portfolio

The room’s booked, the agenda’s confirmed and lunch has been ordered, so it must be time for another SIG meeting.  This time, the Assessment SIG is joining up with the Enterprise and Portfolio SIGs on 22 May at the University of Strathclyde to look at issues that affect all three domains and areas of overlap between the domains. 

The agenda includes the usual mix of news and updates, project presentations and discussion sessions, plus a special themed requirements gathering session focused on the pressing issue of student retention.  Myles Danson of JISC opens the day with a heads-up on forthcoming Invitations to Tender in the assessment domain, a topic that is always of great interest.  Nicola Wilkinson of the WebPA project, based at Loughborough University, will introduce their Learning Impact Award-nominated system, while Alan Paull will discuss the University of Nottingham’s DELIA project on admissions.

The admissions process is also the focus of proposed BSI standardisation work for the transmission of digital evidence and assessment data between schools and awarding bodies to be presented by Karim Derrick of TAG Learning.

The afternoon will feature presentations and discussions on student retention aimed at gathering requirements, recommendations and priorities for future activities, led by our own Simon Grant and Helen Richardson and building on the work of the STAR project and the National Audit Office.

As always, the meeting is free to attend, with lunch and refreshments provided.  It’s open to all, and we just ask that you register in advance to secure your place.  We look forward to seeing you there!

Assessment, assentport_May08, cetis-standards, cetis-systems, enterprise, portfolio