Rowin’s blog

IMS invite developers to participate in an IMS Question and Test Interoperability (QTI) v2.1 implementation survey.

The survey is designed to do two things:

• establish the ‘state of play’ with regard to QTI v2.1 tool capabilities
• establish whether there is sufficient overlap in tool capabilities to define one or more profiles.

IMS will publish an anonymous summery of the survey outcomes to all participants.  Survey results will be strictly confidential and data from this research will be reported only in the aggregate.  Your information will be coded and will remain confidential, however if you wish you may supply your email address if you are willing to be contacted to follow up or for additional information.

The survey only poses questions about QTI v2.1 capabilities, which means that it is not relevant for developers of earlier versions of QTI.

The survey will only take around 20-30 minutes to complete.  Information about the QTI project group and results from this survey will be posted in the IMS QTI forum.

This is a great opportunity for QTI v2.1 developers to help steer the future of the specification.

Assessment, Standards and specifications, cetis-standards, standards

You may remember a proposed standard for the transfer of qualification assessment data and evidence that was previously covered on this blog.

Work on this has been ongoing since then, and a draft standard is now available for public consultation and comment.  The public draft can be accessed via the BSI website, and comments may be submitted by following the instructions there.

All comments must be submitted by 30 November to be considered for the final version of the standard.  Depending on the nature and extent of comments received, the standard is likely to be released in the first quarter of 2010.

Assessment, Standards and specifications, e-Portfolio (Technology), portfolio

Over the last few days a new notice has appeared on the IMS Question and Test Interoperability webpage in place of the QTI v2.1 draft specification:

The IMS QTIv2.1 draft specification has been removed from the IMS website. Adequate feedback on the specification has not been received, and therefore, the specification has been put back into the IMS project group process for further work.

QTI v2.1 was under public review for more than 2 years and did not achieve sufficient implementation and feedback to warrant being voted on as a final specification. Therefore it has been withdrawn for further work by the IMS membership. IMS cannot continue to publish specifications that have not met the rigors of the IMS process.”

IMS GLC has convened a set of leading organizations to take the lead on this new work - which will be considered to be in the CM/DN draft phase in the IMS process.  Therefore, we are very encouraged and hopeful that a new version will be available in due time, possibly a QTI v2.2, along with the necessary conformance profiles. However, we cannot assume that it will be a linear evolution from QTI v2.1.

Until that time the only version of QTI that is fully endorsed by IMS GLC is v1.2.1, that is supported under the Common Cartridge Alliance: http://www.imsglobal.org/cc/alliance.html . While QTI version 2.0 has been voted on as a final specification by the IMS members, it’s deficiencies are well known and IMS does not recommend implementation of it.

This was clearly completely unexpected, not only for us at CETIS but also amongst a number of commercial and academic developers who have been working with the specification as can be seen by posts to the technical discussion list hosted by UCLES.  In particular, I’d encourage you to read Wilbert’s response on behalf of CETIS.

Concerns from the developer community addressed a number of the issues raised in IMS’s statement.  In response to the claim that ‘adequate feedback on the specification has not been received’, several commentators argued that this is because of the high standard of the specification; while the suggestion that ‘QTI v2.1 … did not achieve sufficient implementation … to warrant being voted on as a final specification’ sparked the addition of a number of implementations to Wikipedia’s QTI page.

There is agreement that work will progress on the basis of the public draft, so it is still perfectly possible that the outcome will be a mildly amended version of the public draft with some small profiles.

CETIS will be following this up, and will of course keep you all informed about progress.  In the meantime, we’d be very keen to hear any thoughts or comments you have, although I would encourage you to sign up for both the UCLES list and the official IMS QTI list to ensure your voice is heard as widely as possible; it would be most beneficial for the wider QTI community I feel for discussion to be focused in one place, i.e. the UCLES list.

Assessment, Standards and specifications, cetis-standards

Developers of tools that implement IMS specifications will have an opportunity to participate in a formal IMS TestFest as part of the next IMS Quarterly Meeting, being held in Birmingham on 15-18 September.  Seven QTI v2.1 tools and products have already signed up for the event on Tuesday 16th September, and will also be covered in the open session on QTI the day before: RM, ASDEL, JAssess, QTI Constraints Editor, AQuRate, Onyx and the QTI Migration Tool. 

Amongst other items of interest on Wednesday 17th is the final of the regional Learning Impact competition - entries for which must be in by 25 August, so there’s still time to get your application in!  The meeting closes on the 18th with a summit on Interoperability Now and Next, covering a broad range of issues and featuring a number of high profile speakers.

Assessment, Standards and specifications, ims-sept08

Found all by myself this time :-) was a series of tweets detailing Pierre Gorissen and Steve Lay’s successful integration of the ASDEL QTI Playr with Moodle, meaning that Pierre is now able to run QTI 2.1 assessments within a Moodle course.  This is a signficant step forward for making the specification more attractive to users, given the huge popularity and dynamic community that surrounds Moodle.  It might also make QTI a more attractive alternative to OpenMark within the Moodle community - there are some interesting comments to be found on QTI in the Moodle forums.

Assessment, Standards and specifications

Thanks also go to Adam for pointing out RM’s Test Authoring System which claims to be fully compliant with IMS QTI 2.1, making it one of the earliest commercial products to implement the revised specification.  I couldn’t find a demo to try out, but it is good to see the specification finally being implemented in this type of system and market sector.  Also on the website are also a couple of research reports on the impact of ICT in the classroom which are well worth reading.

Assessment, Standards and specifications

Things may be quiet at the moment on the release of the final version of IMS QTI 2.1, but there’s been quite a bit of activity in the background around implementations and integration with other systems.

Keris, the Korea Education and Research Information Service (similar to the UK’s JISC), have been quite active within IMS and in February signed a formal memorandum of understanding with IMS to launch IMS Korea.  Amongst their activities is involvement with Teaching Mate, a commercial product which aims to support QTI export by the end of the year.  Also from Keris is a QTI 2.1 player that does import and export QTI 2.1 - it’s in Korean, but Adam cunningly pointed out that you can use the browser status bar to work out what each button actually does.

It’s worth having a look around the Keris site to see the extent to which they support ICT in education.  Of particular interest is EDUNET, the National Teaching and Learning Center established in 1996 which, amongst a range of other services, provides a large range of school-level teaching materials and ‘an online testing service to evaluate students’ achievements’.  Would teachers in the UK welcome a centrally provided eassessment service to support the government’s eassessment targets?

Assessment, Standards and specifications

Registrations are now open for our next Assessment SIG meeting, and you’re warmly invited to book your place for this event hosted by the University of Cambridge.  It’s a joint meeting with the CETIS Educational Content SIG, something we’ve been planning to do for some time, looking in particular at two standards of interest to assessment: IMS Common Cartridge and IMS Tools Interoperability.

Common Cartridge hasn’t even been released yet, but has already generated significant interest amongst content vendors and publishers and has been heavily promoted by IMS.  It combines profiles of a number of different standards, including IMS Content Packaging v1.1.4, IMS Question and Test Interoperability v1.2.1, IMS Tools Interoperability Guidelines v1.0, IEEE LOM v1.0 and SCORM v1.2 and 2004.  The resultant learning object package or ‘cartridge’ is intended to be fully interoperable across any compliant system allowing content to be delivered to any authorised individual.

The appeal of Common Cartridge coupled with authentication and digital rights management systems to content publishers is clear, and the specification is particularly suited to the American educational system where there is a closer relationship between content vendor and courses than in UK Higher Education; in the UK, its primary impact may be in the schools and Further Education sectors where there is more of a history of buying content from publishers than HE.  The inclination of many UK HE lecturers to produce their own content and the bespoke nature of many higher level courses are issues we’ve already encountered when looking at topics such as open content and item banking, but there is some interest within UK education, in particular from the Open University.  As a major content producer whose resources are used far beyond their own courses, Common Cartridge has clear potential, and Ross McKenzie and Sarah Wood of OU OpenLearn will offer an insight into their experiences of implementing the specification and developing cartridges.  There has been very little work to date on the delivery of assessment material through Common Cartridge, a topic which will be addressed by Niall Barr of NB Software.  Our own Wilbert Kraan and Adam Cooper will update delegates on the current position of Common Cartridge.

IMS Tools Interoperability has received rather less fanfare, but is a valuable specification which takes a web services approach to seamlessly integrating different tools.  It allows specialist tools to be ‘plugged in’ to a learning management system, such as integrating a sophisticated assessment management system with a VLE which only provides limited native support for assessment, or discipline-specific tools such as simulators.  It also supports accessibility requirements through the (optional) incorporation of the user’s IMS Accessibility Learner Information Package profile to allow silent interface configuration.  Warwick Bailey of Icodeon will be discussing his experiences with the specification.

In the morning, Steve Lay of CARET, University of Cambridge, will be providing an update on the current state of IMS QTI v2.1.  Steve is co-chair of the IMS QTI working group (with Pierre Gorissen of SURF and Fontys University).

The afternoon will feature a presentation by Linn van der Zanden of the Scottish Qualifications Authority on the use of wikis and blogs in education and assessment, picking up on an increasing interest in the use and potential of Web 2.0 technologies in this domain.

The meeting is colocated with a workshop by the three JISC Capital Programme projects focusing on assessment to which you are also invited…

Assessment, General, Standards and specifications, Web 2.0, cetis-content, cetis-standards, ecass_Feb08, elearning, innovation

Gales are howling, trains in chaos, so it must be January and time to look ahead to what 2008 has in store…

The final release of QTI v2.1 should be out this spring, and it’ll be interesting to see what uptake is like.  This will be the most stable and mature version of the specification to date, supported by a long public draft stage and a number of implementations.  Angel Learning are a significant commercial early adopter, and other vendors are bound to be looking at their experiences and whether Angel’s embracing of the specification has an impact on their own customer demand for QTI 2.1. 

Other significant implementors of 2.1 are the JISC Capital Programme projects which will be concluding around March.  AQuRate offers an item authoring tool, Minibix provides support for a range of item banking functions while ASDEL is an assessment delivery engine which supports both standalone use and integration with a VLE.    These projects should deliver quality resources to the community which will provide a firm foundation for use of the specification.  There was a sneak preview of these projects at our last SIG meeting.

Talking of SIG meetings, dates for the next two meetings can now be confirmed. 

On 19 February there will be a joint meeting with the CETIS Educational Content SIG in Cambridge.  This meeting will cover a range of shared concerns such as new content related specifications such as Common Cartridge and Tools Interoperability, and innovative approaches to educational material and assessment.  Information about this meeting and online registration will be available very soon.  This will be preceded by a workshop hosted by the Capital Programme projects discussed above.

The focus shifts from assessment as content to assessment as process with another joint meeting on 1 May in Glasgow.  This meeting will be a joint meeting with the CETIS Portfolio and Enterprise SIGs and will offer an opportunity to explore some of the shared issues in these domains.  Again, information on the event will be available on the mailing lists, on this blog and on the website in due course.

Another event of note is the annual International Computer Assisted Assessment Conference on 8 and 9 July at Loughborough.  The call for papers is already out, with submissions due by 29 February.  As always, this should be a lively and important event in the CAA calendar.  Alt-C 2008, Rethinking the Digital Divide, will be held in Leeds on 9 - 11 September; again, the closing date for submissions is 29 February.  There’s also a regularly updated list of non-CETIS assessment related events on the wiki.

And what about the trends for eassessment in 2008?  The results of Sheila’s poll, with a strong emphasis on Web 2.0 technologies and possibilities, do seem to reflect to some extent the comments on the last meeting’s evaluation forms which suggested increasing interest in innovative technologies, signficant concern with transforming and enhancing the assessment experience and direct engagement with teaching and learning rather than the more abstract issues of standards and specifications for their own sake.  It will be interesting to see how the more ‘traditional’ XML-based QTI v2.1 fares in the light of the increasing popularity of mashups and web services in 2008.

Assessment, Standards and specifications, cetis-content, cetis-systems, elearning

Academics and developers met in Glasgow recently to participate in the most recent Assessment SIG meeting. The very full agenda covered a range of topics, both technical and pedagogic, and presentations led to some lively discussions.

Myles Danson of JISC opened the day by presenting JISC’s views and priorities for eassessment, as well as pointing to some future work they will be undertaking in the domain.

Yongwu Miao of the Open University of the Netherlands discussed work undertaken by the TENCompetence Project, with a particular focus on the relationship between IMS QTI and IMS Learning Design and the work they have done in this area. Dick Bacon of the University of Surrey and the HEA discussed the relationship between different varieties or ‘dialects’ of QTI, exploring some of the implementation and interpretation issues that hinder or break interoperability between systems nominally implementing the same version of the specification. CAL Consultant Graham Smith pleased the audience with news that a new Java version of his QTI demonstrator will be available shortly with updated support for QTI 2.0 items, which should help in the identification and resolution of implementation problems.

Martin Hawksey of the University of Strathclyde presented the work of the Re-Engineering Assessment Practices project. With a focus on real world assessment experiences, including an impressive collection of case studies exploring the impact of transformation within assessment practices, the REAP project was of particular interest to participants. Also of great interest, and perhaps unsuprisingly sparking the greatest amount of debate, was the exploration of ‘Assessment 2.0′ presented by Bobby Elliott of the Scottish Qualifications Authority. Bobby looked at ways in which Web 2.0 technologies can be used to enhance and modernise assessment in ways which can engage and appeal to increasingly digitally literate learners.

The day also featured several demonstrations of tools under development. Niall Barr of NB Software demonstrated his current work, an assessment tool which utilises the IMS QTI, Content Packaging and Common Cartridge specifications, while Steve Bennett of the University of Hertfordshire demonstrated MCQFM, a JISC-funded tool which provides a simple text-based format for converting and editing items between formats. Two more JISC projects closed the day. AQuRate, presented by Alicia Campos and David Livingstone of Kingstone University, is an elegant item authoring tool while ASDEL, presented by Jon Hare of the University of Southampton, is an assessment delivery tool which builds on the R2Q2 project to provide a fuller test tool. A third project, Minibix (University of Cambridge) on item banking, is working closely with AQuRate and ASDEL.

Links to presentations (via slideshare), project websites and other information can all be found on our wiki: http://wiki.cetis.ac.uk/JISC_CETIS_Assessment_SIG_meeting%2C_26_September_2007.

Assessment, General, Standards and specifications, cetis-community, cetis-content, cetis-systems, elearning