Notes on badges

If you haven’t heard about Mozillas Open Badge Initiative, a great explanation and round up lives on Rowins blog. As Rowin points out that badges ‘draw upon widespread use of badges and achievements in gaming‘ and as somebody who has many badges and achievements in various game systems I can’t help but wonder if some of the problems that have cropped up in games might cross over into the Open Badge Initiative. Some early thoughts:

  • Nobody wants to complete a level using only a hyperblaster
  • Badly designed meta-goals can ruin an experience, some players will attempt to do all the tasks asked of them to get as many as the badges, achievements or points as possible. Is it fun completing any levels in quake 4 with only a hyperblaster? No, but it gets you a badge. Would learners do pointless tasks just to get badges, should we worry about loss of intrinsic motivation?

  • Bribery
  • Early in life a Microsoft console a game publisher realised it had a bad game on its hands. The answer to get gamers to part with their cash was to it would give them a full set of achievements in 3 minutes. Would users go for a product because it’s the quickest way to reach get a badge?

  • Badge Inflation
  • Achievements just aren’t enough anymore, as soon as games started giving out easy achievements gamers wanted more. How about a virtual hat? Now gamers are checking that their new game has extra avatar awards as well as achievements.

  • Bypassing the rewards system or creating a new one
  • What happens when developers read on a blog that bribery and badge inflation are a problem on your host platforms badge system. Some developers just create their own.

  • As punishment
  • Although now I like the term “useful indicator for characterizing an unknown” (see comments)

I think that badges are a really interesting idea. But maybe its worth thinking about other reward systems and the effects badges/achievements have had after implementation.

Developing a web analytics strategy for a distributed organisation

For as long as I have been a web developer with CETIS we have relied on analysing server logs to give an indication of traffic sources and visitor trends. This approach existed long before I joined CETIS and seemed like a logical way of doing things, CETIS has had many web servers and many different developers have installed different tools and resources and since they were all using the same servers and producing the same style logs it has been a reasonable method of producing comparable stats.

While this method of collecting stats has stayed the same over the life of CETIS, the direction of CETIS and the environment that it finds itself in has changed over time and a need for a new strategy has become apparent.

Challenges from JISC CETIS and the environment

  • JISC CETIS is more distributed from a technical point of view

Historically CETIS has had access to physical servers that sat in a server room somewhere in a University. A recession later and shifts in University policies mean that the abundance of resource is no longer available. While there are lots of external providers are happy to help you produce a flexible service and tie you into their hosting packages it does raise issues. Do we have access to server logs?  Are the logs the same? If not then are the stats produced similar to the stats package we use? Can we even produce stats?

Similarly JISC CETIS is moving away from bespoke code when there are popular services that do the same thing and this raises similar questions.  What stats do the services produce, are they comparable with other services, is there an API and will we have to pay to access what we’ve collected down the line.

  • JISC CETIS is more distributed from a people point of view

Staff in JISC CETIS are technologically savvy and have our opinions on the services and techniques that we like. While I think it is a good thing to have such a technically diverse organisation trying new and exciting things it is also a problem from a stats analysis point of view. Are staff hosting their blogs, events and resources on cloud services and if so how do we measure the use of these resources?

  • A call for more sophisticated analytics

In JISC CETIS there is an increasing call to know more about the things we do and how they are used. It is important for any organisation to respond to its environment and the questions we are asking ourselves about our resources are becoming more and more complex. Log files can only give you so much information and it seems that Javascript solutions are needed to answer these questions. Recent improvements in solutions such as Google Analytics offer real in depth analysis of your web traffic and resource usage

Implementation Woes

A simple step that we have taken is to start to role out javascript tracking  with Google Analytics over the CETIS services, but even that simple act starts to highlight issues. The first thing we noticed was that visitor numbers were hardly comparable. Some early thoughts on why this might be:

  • Google Analytics is more intelligent when it comes to what is and isn’t a visitor or a bot
  • Google Analytics is Javascript based and will not count anything if the tracking code is not executed for some reason
  • The hacks for Google Analytics to track binary files and RSS are not very good.

A hybrid solution

Despite the changing environment and early hiccups I feel positive about working towards a new web analytics strategy. I wrote this in an attempt to get my head around the issue and I think now I have some key starting ideas. I think that a hybrid solution is required as javascript solutions are more portable and answer more complex answers but are difficult to implement in such a distributed organisation and are held back by some of the limitations of javascript. I feel that we have to become more intelligence about how we analyse the data, my view is that analytics should be taken with a pinch of salt and that it is not about how high the figures are but about trends in these figures and that a good strategy for CETIS would be to identify places in its online resources with stats that can be compared and trends identified.

Finally I think that as organisations become more distributed and stats become more personal a web analytics strategy becomes more of an individual responsibility. I’m not quite sure what an effective strategy where analysis of individuals resources trends is helped to steer the organization as a whole would look like.

More to come…