Simon Grant of CETIS

The other European project I’m involved in for CETIS is called ICOPER. Over the last couple of weeks I’ve been doing some work improving the deliverable D2.2, mainly working with Jad Najjar. I flag it here because it uses some of the conceptual modelling work I’ve been involved in. My main direct contribution is Section 2. This starts with part of an adaptation of my diagram in a recent post here. It is adapted by removing the part on the right, for recognition, as that is of relatively minor importance to ICOPER. As ICOPER is focused on outcomes, the “desired pattern” is relabelled as “intended learning outcome or other objective”. I thought this time it would be clearer without the groupings of learning opportunity or assessment. And ICOPER is not really concerned with reflection by individuals, so that is omitted as well.

In explaining the diagram, I explain what the different colours represent. I’m still waiting for critique (or reasoned support, for that matter) of the types of thing I find so helpful in conceptual modelling (again, see previous post on this).

As I find so often, detailed thinking for any particular purpose has clarified one part of the diagram. I have introduced (and will bring back into the mainstream of my modelling) an “assessment result pattern”. I recognise that logically you cannot specify actual results as pre-requisites for opportunities, but rather patterns, such as “pass” or “at least 80%” for particular assessments. It takes a selection process (which I haven’t represented explicitly anywhere yet) to compare actual results with the required result pattern.

Overall, this section 2 of the deliverable explains quite a lot about a part of the overall conceptual model intended to be at least approximately from the point of view of ICOPER. The title of this deliverable, “Model for describing learning needs and learning opportunities taking context ontology modelling into account” was perhaps not what would have been chosen at the time of writing, but we needed to write to satisfy that title. Here, “learning needs” is understood as intended learning outcomes, which is not difficult to cover as it is central to ICOPER.

The deliverable as a whole continues with a review of MLO, the prospective European Standard on Metadata for Learning Opportunities (Advertising), to get in the “learning opportunities” aspect. Then it goes on to suggests an information model for “Learning Outcome Definitions”. This is a tricky one, as one cannot really avoid IMS RDCEO and IEEE RCD. As I’ve argued in the past, I don’t think these are really substantially more helpful than just using Dublin Core, and in a way the ICOPER work here implicitly recognises this, in that even though they still doff a cap to those two specs, most of RDCEO is “profiled” away, and instead a “knowledge / skill / competence” category is added, to square with the concepts as described in the EQF.

Perhaps the other really interesting part of the deliverable was one we put in quite a lot of joint thinking to. Jad came up with the title “Personal Achieved Learning Outcomes” (PALO), which is fine for what is intended to be covered here. What we have come up with (provisionally, it must be emphasised) is a very interesting mixture of bits that correspond to the overall conceptual model, with the addition of the kind of detail needed to turn a conceptual model into an information or data model. Again, not surprisingly, this raises some interesting questions for the overall conceptual model. How does the concept of achievement (in this deliverable) relate to the overall model’s “personal claim expression”? This “PALO” model is a good effort towards something that I haven’t personally written much about - how do you represent context in a helpful way for intended learning outcomes or competences? If you’re interested, see what you think. For most skills and competences, one can imagine several aspects of context that are really meaningful, and without which describing things would definitely lose something. Can you do it better?

I hope I’ve written enough to stimulate a few people at least to skim through that deliverable D2.2.

This conceptual model now includes basic ideas about what goes on in the individual, plus some of the most important concepts for PDP and e-portfolio use, as well as the generalised formalisable concepts processes surrounding individual action. It has come a long way since the last time I wrote about it.

The minimised version is here, first…

and that is complex enough, with so many relationship links looking like a bizarre and distorted spider’s web. Now for the full version, which is quite scarily complex now…

Perhaps that is the inevitable way things happen. One thinks some more. One talks to some more people. The model grows, develops, expands. The parts connected to “placement processes” were stimulated by Luk Vervenne’s contribution to the workshop in Berlin of my previous blog entry. But — and I find hard to escape from this — much of the development is based on internal logic, and just looking at it from different points of view.

It still makes sense to me, of course, because I’ve been with it through its growth and development. But is there any point in putting such a complex structure up on my blog? I do not know. It’s reached the stage where perhaps it needs turning into a paper-length exposition, particularly including all the explanatory notes that you can see if you use CmapTools, and breaking it down into more digestible, manageable parts. I’ve put the CXL file and a PDF version up on my own concept maps page. I can only hope that some people will find this interesting enough to look carefully at some of the detail, and comment… (please!) If you’re really interested, get in touch to talk things over with me. But the thinking will in any case surface in other places. And I’ll link from here later if I do a version with comments that is easier to get at.

This version of the conceptual model (of learning opportunity provision + assessment + award of credit or qualification) uses the CmapTools facility for grouping nodes; and it further extends the use of my own “top ontology” (introduced in my book).

There are now two diagrams: a contracted and an expanded version. When you use CmapTools, you can click on the << or >> symbols, and the attached box will expand to reveal the detail, or contract to hide it. This grouping was suggested by several people in discussion, particularly Christian Stracke. Let’s look at the two diagrams first, then go on to draw out the other points.

You can’t fail to notice that this is remarkably simpler than the previous version. What is important is to note the terms chosen for the groupings. It is vital to the communicative effectiveness of the pair of diagrams that the term for the grouping represents the things contained by the grouping, and in the top case — “learning opportunity provision” — it was Cleo Sgouropoulou who helped find that term. Most of the links seem to work OK with these groupings, though some are inevitably less than fully clear. So, on to the full, expanded diagram…

I was favourably impressed with the way in which CmapTools allows grouping to be done, and how the tools work.

Mainly the same things are there as in the previous version. The only change is that, instead of having one blob for qualification, and one for credit value, both have been split into two. This followed on from being uncomfortable with the previous position of “qualification”, where it appeared that the same thing was wanted or led to, and awarded. It is, I suggest, much clearer to distinguish the repeatable pattern — that is, the form of the qualification, represented by its title and generic properties — and the particular qualification awarded to a particular learner on a particular date. I originally came to this clear distinction, between patterns and expressions, in my book, when trying to build a firmer basis for the typology of information represented in e-portfolio systems. But in any case, I am now working on a separate web page to try to explain it more clearly. When done, I’ll post that here on my blog.

A pattern, like a concept, can apply to many different things, at least in principle. Most of the documentation surrounding courses, assessment, and the definitions about qualifications and credit, are essentially repeatable patterns. But in contrast, an assessment result, like a qualification or credit awarded, is in effect an expression, relating one of those patterns to a particular individual learner at a particular time. They are quite different kinds of thing, and much confusion may be caused by failing to distinguish which one is talking about, particularly when discussing things like qualifications.

These distinctions between types of thing at the most generic level is what I am trying to represent with the colour and shape scheme in these diagrams. You could call it my “top ontology” if you like, and I hope it is useful.

CmapTools is available free. It has been a great tool for me, as I don’t often get round to diagrams, but CmapTools makes it easy to draw the kinds of models I want to draw. If you have it, you might like to try finding and downloading the actual maps, which you can then play with. Of course, there is only one, not two; but I have put it in both forms on the ICOPER Cmap server, and also directly in CXL form on my own site. If you do, you will see all the explanatory comments I have made on the nodes. Please feel free to send me back any elaborations you create.

Last Thursday (2009-04-16) I went to a very interesting meeting in Leeds, specially arranged, at the Leeds Institute of Medical Education, between various interested parties, about their needs and ideas for interoperability with e-portfolio tools - but also about skills frameworks.

It was interesting particularly because it showed more evidence of a groundswell of willingness to work towards e-portfolio interoperability, and this has two aspects for the people gathered (6 including me). On the one hand, the ALPS CETL is working with MyKnowledgeMap (MKM) - a small commercial learning technology vendor based in York - on a project involving health and social care students in their 5 HEIs around Leeds. They are using the MKM portfolio tool, Multi-Port, but are aware of a need to have records which are portable between their system and others. It looks like being a fairly straightforward case of a vendor with a portfolio tool being drawn in to the LEAP2A fold on the back of the success we have had so far - without the need for extra funding. The outcome should be a classic interoperability win-win: learners will be able to export their records to PebblePad, Mahara, etc., and the MKM tool users will be able to import their records from the LEAP2A-implementing systems to kick-start their portfolio records there with the ALPS CETL or other MKM sites.

MKM tools, as suggested by the MKM name, do cover the representation of skills frameworks, and this forms a bridge between two threads to this meeting: first, the ALPS CETL work, and second, the more challenging area of medical education, where frameworks - of knowledge, skill or competence - abound and are pretty important for medical students and in the professional development of medical practitioners, and health professionals more generally.

In this more challenging side of the meeting, we discussed some of the issues surrounding skills frameworks in medical education - including the transfer of students at undergraduate level; the transfer between a medical school like Leeds and a teaching hospital, where the doctors may well soon be using the NHS Foundation Year e-portfolio tools in conjunction with their further training and development; and then on to professional life.

The development of LEAP2A has probably been helped greatly by not trying to do too much all at once. We haven’t yet fully dealt with how to integrate skills frameworks into e-portfolio information. At one very simple level we have covered it - if each skill definition has a URI, that can be referred to by an “ability” item in the LEAP2A. But at another level it is greatly challenging. Here in medical education we have not one, but several real-life scenarios calling for interoperable skills frameworks for use with portfolio tools. So how are we actually going to advise the people who want to create skills frameworks, about how to do this in a useful way? Their users, using their portfolio tools, want to carry forward the learning (against learning outcomes) and evidence (of competence) to another setting. They want the information to be ready to use, to save them repetition - potentially wasteful to the institution as well as the learner.

The answer necessarily goes beyond portfolio technology, and needs to tackle the issues which several people are currently working on: European projects like TENCompetence and ICOPER, where I have given presentations or written papers; older JISC project work I have been involved with (ioNW2, SPWS); and now the recently set up a CETIS team on competences.

Happily, it seems like we are all pushing at an open door. I am happy to be able to respond in my role as Learning Technology Advisor for e-portfolio technology, and point MKM towards the documentation on - and those with experience of implementing - LEAP2A. And the new competence team has been looking for a good prompt to hold an initial meeting. I imagine we might hold a meeting, perhaps around the beginning of July, focused on frameworks of skills, competence, knowledge, and their use together with curriculum learning outcomes, with assessment criteria, and with portfolio evidence? The Leeds people would be very willing to contribute. Then, perhaps JISC might offer a little extra funding (on the same lines as previous PIOP and XCRI projects) to get together a group of medical educators to implement LEAP2A and related skills frameworks together - in whatever way we all agree is good to take forward the skills framework developments.

I’ve been telling people, for quite a while, that I will write up suggestions for how to deal with abilities (competence, competencies, knowledge, etc. etc.) for many reasons, including particularly e-portfolio related uses. Well, here are my current ideas for the New Year.

They are expressed in a set of linked pages, each dealing with a facet of the issues. The pages are very much draft ideas, but serve the purpose of getting out the basic ideas and inviting other ideas for improvement. If you have an interest, please do at least leave a comment here, or e-mail me with ideas and suggestions.

As I’ve been involved with GMSA in various ways including through the ioNW2 project, I went to their seminar on 14th May introducing GMSA Advance.  This is to do with providing bite-sized modules of Higher Education, mainly for people at work, and giving awards (including degrees) on that basis - picking up some of the “Leitch” agenda. As I suspected, it was of interest from a portfolio perspective among others.

I’ll start with the portfolio- and PDP-related issues.

The first issue is award coherence. If you put together an award from self-chosen small chunks of learning (”eclectic”, one could call it), there is always an issue of whether that award represents anything coherent. Awarding bodies, including HEIs, may not think it right to give an award for what looks like a random collection of learning. Having awarding bodies themselves define what counts as coherent risks being too restrictive. An awarding body might insist on things which were not relevant to the learner’s workplace, or that had been covered outside the award framework. On the other hand, employers might not understand about academic coherence at all. A possible solution that strikes me and others is

• have the learner explain the coherence of modules chosen
• assess that explanation as part of the award requirement.

This explanation of coherence needs to make sense to a variety of people as well as the learner, in particular, to academics and to employers. It invites a portfolio-style approach: the learner is supported through a process of constructing the explanation, and it is presented as a portfolio with links to further information and evidence. One could imagine, for example, a video interview with the learner’s employer as useful extra evidence.

A second issue is the currency and validity of “credit”. Now I have a history of skepticism about credit frameworks and credit transfer, though the above idea of assessed explanation of award coherence at last  brings a ray of light into the gloom. My issue has always been that, to be meaningful, awards should be competence-based, not credit based. And I still maintain that the abilities claimed by someone at the end of a course, suitably validated by the awarding body, should be a key part of the official records of assessment (indeed, part of the “Higher Education Achievement Report” of the Burgess Group - report downloadable as PDF)

One of the key questions for these “eclectic” awards is whether credit should have a limited lifetime. Whether credit should expire surely should depend on what credit is trying to represent. It is just the skills, abilities or competences whose validation needs to expire - this is increasingly being seen in the requirement for professional revalidation. And the expiry of validation itself needs to be based on evidence - bicycle riding and swimming tend to be skills that are learned once for ever; language skills fall off only slowly; but the knowledge of the latest techniques in a leading edge discipline may be lost very quickly.

This is a clear issue for portfolios that present skills. The people with those portfolios need to be aware of the perceived value of old evidence, and to be prepared to back up old formal evidence with more recent, if less formal, additional evidence of currency. We could potentially take that approach back into the the GMSA Advance awards, though there would be many details to figure out, and issues would overlap with accreditation of prior learning.

Other issues at the seminar were not to do with portfolios. There is the question of how to badge such awards. CPD? Several of those attending thought not - “CPD”is often associated with unvalidated personal learning, or even just attendance at events. As an alternative, I rather like the constructive ambiguity of the phrase “employed learning” - it would be both the learners and the learning that are employed - so that is my suggestion for inclusion into award titles.

Another big issue is funding. Current policy is for no government funding to be given for people studying for awards of equal or lower level than one they have already achieved. The trouble is that if each module itself carries an award, then work-based learners couldn’t be funded for this series of bite-sized modules, but only one. The issue is recognised, but not solved. A good idea that was suggested at the seminar is to change and clarify the meaning of credit, so that it takes on the role of measuring public fundability of learning. Learners could have a lifetime learning credit allowance, that they could spend as they preferred. Actually, I think even better would be a kind of “sabbatical” system where one’s study credit allowance continued to build, to allow for retraining. Maybe one year’s (part time?) study credit would be fundable for each (say) 7 years of life - or maybe 7 years of tax-paying work?

So, as you can see, it was a thought-provoking and stimulating seminar.

I’ve only been hearing and thinking about plagiarism in the last few days - since going to the Assessment Think Tank in York in fact, but since then reading in many places. One of the debated ideas is encouraging students to use plagiarism detection services. Another, heard at York, is that the more adventurous students run more risk. Why? It is unlikely in some subjects (say Philosophy) to come up with entirely novel ideas, so if a student has an idea which was not represented on the reading list, they are less likely to know if someone has had it before, and thus more likely to be judged to have plagiarised - have passed off some ideas as theirs which actually came from someone else. They may not have known that, but they can’t prove it.

Those two ideas together spark off a bigger idea.

Sophisticated plagiarism detection services could be rebranded to be thought of as tracing the intellectual heritage of a piece of work. That would be very useful - I could write some thoughts down, submit them, and be returned a list of similar ideas, along with how my ideas relate to theirs (according to the software, which is not of course going to be perfect). Then I could look up the originals, and work them in properly: paraphrase and reference, for example. It would also be a powerful self-critical tool: instead of simply imagining the objections to one’s own supposedly new idea, one could see how others have argued against similar ideas in the past.

Incredibly useful in the field of patenting, as well, I would guess…

Have the anti-plagiarism people got on to patents yet? I’ll ask.

Assessment think tank, at The Higher Education Academy, York, 31st January to 1st February 2008

Several of these events appear to have been arranged, and this one was with the Subject Centres both for History, Classics and Archaeology (HCA), and for Philosophy and Religious Studies (PRS).

Around 20 or so delegates were present, mostly from representative subject areas, but including from the JISC’s Plagiarism Advisory Service. Previously, I only recognised Sharon Waller from the HEA, and had talked with Costas Athanasopoulos (PRS Subject Centre) at the JISC CETIS conference: he was the one who invited me.

I won’t try to document the whole event, but to pick out a few things which were highlights for me.

The discussion around plagiarism was inspiring. There was very little on the mechanics and technology of plagiarism detection (Turnitin is popular now) and plenty on good practice to avoid the motive for plagiarising in the first place. This overlaps greatly with other good practice - heartening, I felt. George MacDonald Ross gave us links to some of his useful resources.

Also from George MacDonald Ross, there was an interesting treatment of multiple-choice questions, for use preferably in formative self-assessment, avoiding factual questions, and focusing on different possible interpretations, in his example within philosophy.

As I’m interested in definitions of ability and competence, I brought up the issue of subject benchmarks, but there was little interest in that specifically. However, for archaeology fieldwork, Nick Thorpe (University of Winchester) uses an assessment scheme where there are several practical criteria, each with descriptors for 5 levels. This perhaps comes closest to practice in vocational education and training, though to me it doesn’t quite reach the clarity and openness of UK National Occupational Standards. Generally, people don’t seem to be yet up to clearly defining the characteristics of graduates of their courses, or they feel that attempts to do that have been poor. And yet, what can be done to provide an overall positive vision, acceptable to staff and student alike, without a clear, shared view of aims? Just as MCQs don’t have to test factual knowledge, learning outcomes don’t have to be on a prosaic, instrumental level. I’d be interested to see more of attempts to define course outcomes at relatively abstract levels, as long as those are assessable, formally or informally, by learner, educator and potential employer alike.

One of the overarching questions of the day was, what assessment-related resources are wanted, and could be provided either through the HEA or JISC? In one of our group discussions, the group I was in raised the issue of what a resource was, anyway? And at the end, the question came back. Given the wide range of the discussion throughout the day and a half, there was no clear answer. But one thing came through in any case. Teaching staff have a sense that much good, innovative practice around assessment is constrained by HEI (and sometimes wider) policies and regulations. Materials which can help overcome these constraints would be welcome. Perhaps these could be case studies, which documented how innovative good practice was able to be reconciled with prevailing policy and regulations. Good examples here, presented in a place which was easy to find, could disseminate quickly - virally even. Elements of self-assessment, peer assessment, collaboration, relevance to life beyond the HEI, clarity of outcomes and assessment criteria, etc., if planted visibly in one establishment, could help others argue the case for better practice elsewhere.

If Sam can do it (and at the same time claim that Scott and Adam have as well) then I guess we all can…

Which Programming Language are You?

It is very interesting to note how compulsive these kind of tests are: it seems like we all want to know how we are rated by others. Very natural. Perhaps we can get a hold of this and link it in to the domain of assessment and the issue of identity?

Also, there should be an easy way of presenting the results of such tests (OK, perhaps more serious ones) in an e-portfolio, and make that available to others to search on. Perhaps I’m saying no more than something about FOAF and another way in which it could be used: this certainly links to Scott’s approach to e-portfolios.