Rowin’s blog

IMS invite developers to participate in an IMS Question and Test Interoperability (QTI) v2.1 implementation survey.

The survey is designed to do two things:

• establish the ‘state of play’ with regard to QTI v2.1 tool capabilities
• establish whether there is sufficient overlap in tool capabilities to define one or more profiles.

IMS will publish an anonymous summery of the survey outcomes to all participants.  Survey results will be strictly confidential and data from this research will be reported only in the aggregate.  Your information will be coded and will remain confidential, however if you wish you may supply your email address if you are willing to be contacted to follow up or for additional information.

The survey only poses questions about QTI v2.1 capabilities, which means that it is not relevant for developers of earlier versions of QTI.

The survey will only take around 20-30 minutes to complete.  Information about the QTI project group and results from this survey will be posted in the IMS QTI forum.

This is a great opportunity for QTI v2.1 developers to help steer the future of the specification.

Assessment, Standards and specifications, cetis-standards, standards

Charlie Balch at Arizona Western College has been working on a simple, web-based question system for networked computers and hand held devices.  AskClass.net currently functions as a poll tool, allowing the creation of multiple-choice, single-answer questions that can be used in a variety of educational settings and as a consensus building tool.  It’s still very much under development, so while it’s not possible at the moment to specify a correct answer, this and other functionality may emerge in the future depending on user feedback and viability.  A simple marking system would make this an extremely useful tool for both classroom and remote teaching support.

AskClass is written in ASP with a MS Access backend, although Balch is considering making future versions in PHP or Java with an XML backend to increase portability.  Other applications may be integrated with the tool over time.  This is intended as a learning aid, therefore security and multiple voting are not addressed in this version.

Contact details for requesting source code and suggesting features are available on the site’s FAQ.

Assessment, assessment (technology), elearning

You may remember a proposed standard for the transfer of qualification assessment data and evidence that was previously covered on this blog.

Work on this has been ongoing since then, and a draft standard is now available for public consultation and comment.  The public draft can be accessed via the BSI website, and comments may be submitted by following the instructions there.

All comments must be submitted by 30 November to be considered for the final version of the standard.  Depending on the nature and extent of comments received, the standard is likely to be released in the first quarter of 2010.

Assessment, Standards and specifications, e-Portfolio (Technology), portfolio

A Word in Your Ear 2009 - Audio Feedback is a one day conference on the use of audio for providing assessment feedback to university students being held at Sheffield Hallam University on Friday 18 December.  There has been some interesting work in this area in recently such as the JISC-funded Sounds Good project (Bob Rotheram who led that project is the keynote speaker at this event) and this event looks like an excellent opportunity to learn more about initiatives in this area.

Assessment, accessibility, innovation

There’s a lively discussion (beginning here) going on on the VLEs JISCMail discussion list around ‘innovative ways of online assessing’.  Although in some cases ‘innovative’ seems to be equated with ‘electronic’, there are some interesting activities and comments emerging:

• MS Word documents uploaded and marked by comments; Google Docs may provide greater preservation of comments.
• Ongoing submission process over the academic year managed by Moodle’s iterative assessment feature.
• ePortfolios - although Emma Duke-Williams observes that ‘mostly… they’re used as summative - and staff, rather than student, controlled repositories.’
• Awarding marks for posting in discussions on the VLE.
• Wiki contributions.
• Audio feedback.
• Video feedback.
• Peer assessment.
• Self-assessment.
• Need to think beyond just summative assessment to consider formative and diagnostic assessment.
• Ways of seeing whether learners are engaging with feedback provided and acting on it.
• Knowing one’s students.

Assessment

Eric Shepherd of Questionmark has been developing an assessment maturity model ‘that provides a way for organisations’ executives and managers to monitor, chart and improve their use of assessments’, and has recently begun to formalise this in an online model.

The model separates the entire assessment process into three key areas (development, delivery and reporting), each comprised of six measures (stakeholder satisfaction, security, strategic alignment, maturity, data management and communication).  Shepherd also identifies four phases of assessment maturity which can help identify the needs and requirements of an organisations (ad hoc, managed, refined and aligned).  Each of these elements is or will be expanded and further developed as the model itself matures, with ongoing development being focused on the project wiki.

There is also a great deal of useful information available on the site, such as learning resources to help assessment managers understand their own processes’ maturity and helpful links to relevant material.

The model is still in development but should already be of value to users and will continue to develop over time.

Thanks to Steve Lay for the heads up!

Assessment

A group of students have finally lost their case against plagiarism detection service Turnitin with the judgement that Turnitin does not breach students’ copyright by retaining copies of submitted essays.  In particular, The Chronicle reports that

the district-court judge said Turnitin’s actions fell under fair use, ruling that the company ‘makes no use of any work’s particular expressive or creative content beyond the limited use of comparison with other works.’  He also said the new use ‘provides a substantial public benefit.’

While the reasoning is clear, I have always felt a bit uncomfortable about the tension between the undoubted benefits to be gained from Turnitin and the fact that, as a commercial company, Turnitin’s parent company iParadigms is undeniably profiting, albeit indirectly, from other people’s intellectual property.  However, institutions frequently claim IPR rights to their students’ work, and understandably see considerable benefits from requiring the submission of essays to Turnitin.

Plagiarism is undoubtedly a serious issue facing HE, but I think my biggest issue with this is the way in which so many institutions choose to use Turnitin exclusively as a tool to detect and evidence plagiarism.  There is an assumpation inherent in this approach that all students may attempt to cheat.  Retaining papers once they’ve been checked for plagiarism for adding to the database also suggests an assumption that students cannot be trusted not to pass their work on to others.

Other institutions take a more collaborative approach, allowing students to submit their work multiple times to Turnitin and using it as a teaching aid to help students avoid indirect plagiarism and learn how to better develop their essay writing and argumentation skills and understand their relationship to their research sources.  In this approach, all students receive a direct benefit from the software, profiting significantly as learners and writers, and those who may be tempted to intentionally plagiarise may look for other ways to cheat instead.

Assessment, General, elearning, plagiarism

A post this morning on the WebPA discussion list raised an issue that I’ve long felt has a negative impact on the uptake of JISC project outputs: how to support the use of tools produced by JISC projects in an institutional environment that is not interested in supporting them.

WebPA is a great project success story, being adopted by a number of institutions and winning a Bronze Award at the 2008 IMS Learning Impact awards.  It provides an innovative approach to peer assessment evaluation, allowing individual marks for each participant in a piece of group work.  The system generated a lot of interest when they presented at a CETIS event last year.

The poster has identified WebPA as a possible tool to support his teaching, but says:

Unfortunately there is little if any chance of the application ever being hosted on my University servers, I won’t even waste my time trying to get this on their radar [...] I should say that while I am not completely IT illiterate I am not going to install the application myself since this is well beyond my personal skill level.

This highlights what I feel is a gap in the project lifecycle: bridging the support gap between the production of useable tools and enabling those outputs to be used in real educational contexts.  Although some lecturers have a high level of technical confidence and competence, this absolutely cannot be expected for the vast majority, and there seems to be a lack of support for those who are keen to use these innovative tools but lack the confidence or expertise to do so.  How do we encourage institutions to be willing to broaden their horizons and support those lecturers who wish to use what they feel are the best tools for their teaching practice?  The poster references commercial companies which host open source systems such as Moodle, but what about newer systems that lack the wide uptake that make providing support and hosting services commercially attractive?

So who should be responsible for supporting projects after the end of their formal funding period, and supporting lecturers and institutions in using these tools?  We’ve addressed this issue before in relation to supporting emerging developer communities in an open source model, but what about tools that are ready for use in actual teaching practice?

Assessment, cetis-support, innovation

Around 35 delegates struggled through Wednesday’s sweltering heat and the baffling mysteries of Manchester Metropolitan University Business School’s internal layout to discuss a range of issues around competences for learning, assessment and portfolio.  Delegates represented a wide range of knowledge and expertise, from novices looking to find out ‘what it’s all about’ to experienced practitioners and developers.

It was an impressively international turn out, with delegates from Norway, Greece, Austria, Spain and Belgium joining the UK contingent, mainly representing the iCoper project which is exploring the linking of assessment with competences.  Assessment interests were also represented by the University of Southampton, who are working on the automatic construction of statements of competency from QTI XML, exploring the underlying modelling of competencies for machine processing.  The majority of delegates came from a strong (e)portfolio background, with interests in the movement of information into and out of eportfolios.  JISC and CETIS participants also highlighted the relevance of this work to JISC’s Curriculum Design projects.

The morning session featured a number of short presentations (all presentations from the day can be found here) on competences requirements in the field of medical education, an area which is relatively advanced in the use of competence frameworks.  Claire Hampshire (MMU), Julie Laxton (ALPS CETL), Karen Beggs (NHS Education for Scotland) and Jad Nijjar (iCoper and Synergetics) covered a range of topics, including the desire for non-hierarchic representations, the management of massive amounts of data, and addressing the various points in a student’s career at which  information can move between one system and another.  The ownership of data in portfolios, including competency information, is an ongoing issue that still is not clear, with at least three actors involved: the data subject, data controller and data processor.  Three main points of interoperability were identified: across time (for example, undergraduate to postgraduate), across specialities (for example, from psychiatry to gynecology), and from elearning experiences to portfolios.

After coffee, Paul Horner (Newcastle University), Shane Sutherland (PebblePad), Dave Waller (MyKnowledgeMap) and Tim Brown (NHS Education for Scotland) delivered short presentations on various tools for handling competence information.  One key issue that emerged from this session was the strong need for a specification to enable the sharing of profiles between systems: while evidence can be exported as HTML, entire profiles cannot be moved between systems except in unwieldy formats such as .pdfs.  Interoperability is needed for both import and export.  There is a noticeable move away by customers from monolithic approaches towards using a variety of (Web 2.0) tools, and developers are working on building open APIs to support this. 

What struck me most from both sessions was the way in which developments around eportfolios and competence recording are very firmly rooted in actual teaching and learning practice, with requirements emerging directly from real-world practice and tool developments directly benefiting teachers and learners.

In the afternoon the meeting split into four groups, ostensibly to work on identifying and representing information structures for a purported competences specification.  In practice, my group spent most of our time discussing widely around the whole area of competences, eportfolios and assessment, but as a newbie in this field I found this hugely helpful.  Overall conclusions from the groups identified the following requirements and issues:

• ability to transfer information between different tools and systems
• transition
• curriculum progression pathways
• relationship between competences and evidence versus qualifications
• repeatable pattern of description at the core
• fairly simple structure
• identifiers for defining authority
• a definable core structure enables extension for extra semantics
• able to express the relationship between a learning object and skills, competences and knowledge
• collection of outcomes
• architectural issues: data is created and needed in many locations instead of at a central point
• competences are highly context dependent

The meeting concluded with asking delegates what they want CETIS to focus on in taking forward work on competences.  Suggestions included:

• development of a data model
• business case for interoperability
• look beyond HE/FE to workplace standards, particularly in the HR domain
• look for connections to the HIRA progress reports due out by November
• look at what has failed so far in order to learn from past experiences
• look at defining competences in such a way that a specification can be combined with XCRI
• have loosely defined competences that can be moved between systems
• need a high level map of the competency domain in comparison with curriculum description and learning objects.

CETIS will be looking at how best we can take this work forward and, as always, we very much welcome input and suggestions from our community - please feel free to leave comments here, follow up via the wiki or contact Simon or me!

Assessment, General, ccomp, competences, portfolio

There’s been an interesting discussion over the last couple of days on the Computer Assisted Assessment JISCMail list around delivery of multiple choice questions. 

The question of how long should be allowed for multiple choice questions produced a consensus of around ‘a minute per question plus a wee bit’ for a ‘typical’ MCQ, but that difficulty level or the use of negative marking or more sophisticated questions would impact on this.  Sandra Gibson cited research by Case and Swanson which suggests that

good students know the answers and … select the right one in very little time (seconds), poor students try and reason out the answers which takes longer. It depends how long you want to give the poorer students to try to work it out, which then impacts on the validity, reliability and differentiation of your assessment.

Discussion broadened to cover the issue of sequential delivery, i.e. when a candidate is unable to return to questions and revise their response once they have moved on to the next question in the test.  There were some compelling educational arguments in favour of this, for example, a series of questions building on or even containing the answers to previous questions; and less satisfactory justifications such as technical limitations on delivery software.    Fascinatingly, a number of posters reported the same (sadly anecdotal) finding that where students revise their response, the likelihood is that they’ve changed a correct answer to a wrong one.   It was also noted that tests which do not permit candidates to revise their responses required less maximum time than those that do.

It’s a good discussion that’s still going on, so well worth following or contributing to!

Assessment