Designing A Degrading Learning Experience

Since Dolly is snowbound in Virginia without internet access right now, I decided to post this week without her... and she is sorely missed. 

Recently I was asked to perform some basic instructional design on an already created eLearning storyboard. One of the duties assigned to me was to edit existing content, and ensure the content was optimized for the engine the developers would use to build the course. As I went through the content, I decided to add a knowledge check interaction at the end of each module. Although these would not be mandatory interactions, they would provide an opportunity for the learner to self-check their learning, and to receive contextual feedback. Before I continue this story, let me digress just a bit about instructional design:

Earlier this year I was presenting to a group of technical writers and instructional designers on what instructional design is. This was a request from a group of technical writers who really felt like they were also performing instructional design duties when they created "tutorials" and/or "labs". They wanted clarification on what my idea of instructional design is.

I noted three primary responsibilities of a typical instructional designer:

1. Applies instructional strategies to aid in the acquisition, creation, retention and retrieval of knowledge.
2. Matches content type to strategy. Creates a learning environment by providing examples and non-examples for teaching concepts and facts.
3. Applies systematic approaches to instructional messaging to aid in learning (pedagogy).

There are a few key components that an instructional designer brings to a learning intervention:

• Designs the information to assist the learner in building on prior knowledge
• Scaffolds the information to enable new knowledge to be acquired
• Provides relevant, contextual feedback in the learning environment

OK, back to my main story... the course design assistance I was providing... Just remember the items I listed above, cause they come into play in this story.

So I was informed that new game templates had been created by the developer that I could leverage to make the knowledge checks more engaging. So, as a game designer, I thought great! Can't wait to get my hands on them. This was a basic "learning by listening" course presented in a PowerPoint-like template to begin with, so any form of interaction would help (I thought) -- until I lifted the hood on the "game" template and peered inside its functionality.

The game template provided was based on the standard Wheel of Fortune format. As you can see in the screenshot below (click the graphic to enlarge), it contains four primary components:

1. Question selection mechanism in the form of a wheel with a spinning arrow
2. A fixed region for the item types. Two types of items are available: multiple choice with single or multiple correct response options, and a true or false type
3. A timer
4. A score indicator

 

The template affords eight game questions. The learner clicks the "SPIN" icon, the arrow spins and randomly lands on a question number. The question is presented in the fixed question region and the timer begins to count down from five minutes awaiting the learner's response which requires selecting a choice, and then clicking a Submit button.

Your first thought may be similar to mine when I came upon this game-like interaction: a nice graphical interface and higher levels of interactivity affording the learner the chance to actually do something. Great!

But as I began to develop the test items, I dug a bit deeper into the functionality, and quickly realized this interaction was not just instructionally unsound, but detrimental to the learning opportunity at hand. A templatized interaction should first strive to be instructionally sound, and then be visually engaging (if appropriate to the learning context). This template has the following problems:

• A game should heighten emotion. It does not meet the basic criteria I establish for a game. A game has to contain an element of play. There are many definitions of play, but one component I look for is heightened emotion. To heighten emotion there must be a compelling desire for the learner to want to engage. Think of the standard Wheel of Fortune game: the choices on the wheel contain different types of elements, such as values (5000 points, 100 points, 300 points) that denote item difficulty. There are also spoilers, such as "Bankruptcy" (forcing the player to lose all their points). Those elements heighten player emotion after spinning the wheel.. "do I want the 5000 point question... oh gosh, I hope it doesn't land on bankruptcy..." This game simply has question numbers as wheel choices. It would not be difficult to change these choices to make them more game-like.
• A game should contain an element of unpredictability. This template displays the same questions in order each time it's played. It does not randomize items from a larger pool. So every time the learner plays, question 1 is question 1. This corrupts the basic tenet of the game: it should easily be able to provide a "new" experience each time it's played. This is not hard to program into the Flash engine. I've done it. This feature would encourage the player to "play again" which potentially increases their ability to learn.
• Visuals should always support the learning content. I wrote feedback for each item before discussing any limitations with the developer (I never thought my feedback would be limited). After the developer received my items with feedback I was told that I must limit the feedback to 108 characters (not counting spaces). By limiting the feedback space, I am forced to NOT display all the feedback I may need for the learner to gain a deeper understanding (which really is the point of the whole interaction, right?). By simply reducing the size of the wheel graphic, I could gain maybe another 100 characters for feedback. The wheel is only the conduit to display the question in this representation anyway, so having it take up MORE screen real estate than the question box itself is just bad design. Reduce the graphic, and give me more room for content. 
• Adult learners need to know there's a reason for what they're doing, otherwise, they will become distracted, bored, and/or question the authenticity of the intervention. To attract the learner to a richer level of engagement, an element of competition is desirable. With multiple rounds, for instance, the player can achieve a higher game level or at least compete against the game for points. This can also provide a scaffolding opportunity. In this template, the score is meaningless -- there is no leaderboard, no saving of the score, and no demonstrable representation of the value of the scoring algorithm. The learner needs to easily understand the "value" of their score. What does a score of "30,000" mean anyway?
• As an instructional designer, I have a better idea of the time element required for the item other than an arbitrary number created by a programmer. When I wanted to control the timing options for the game, I was told that the timer amount was fixed at five minutes. I may have question items in a True or False format that require just a bit of time, and not the whole five minutes. Sure the player can answer quick, click Submit and move on... but with 5 minutes on the timer, where is the challenge?
• Every item type should support learning outcomes. True or False item types simply do not. I can't defend them. This item type is just not instructionally sound. What data does it give the learner? It does provide them a 50% chance of gaining more points. These item types should never be used.

Games designed for learning can be powerful interactions to help attract and engage learners, and have shown to be powerful agents in building problem-solving skills. I have nothing against game "shells" or templates (I worked with Thiagi to design game shells years ago that are still in use today), however, if divorced from sound instructional design, they can be demotivating. If you decide to use templatized game engines be sure to validate the instructional integrity before you adopt and use them. The best is to create them from scratch and design them to your unique requirements. Templates can provide a solid foundation for reusable, rapid development, but they should never take precedence over the ability to create an instructionally sound learning intervention.

Comments

[Jim Fuller]

To add to the concerns, consider the research done on the topic of recall context. After taking an eLearning course, participants were asked to take a series of multiple-choice test questions to assess their learning and reinforce the information.

Unfortunately, the Skinnerian S-R connection had been violated. The learners could pass the test questions given the context of reading a question and being provided with multiple answers from which to choose. However, when confronted the the job based stimulus of having to parse a longer-winded verbal question from a customer and recall the correct answer without multiple choice cues, the learers performance collapsed in a rather spectacular manner.

This poses some intersting questions regarding the efficacy of game-based assessment or "learning reformancement" activities. Are we teaching the learners to play games, or achieve valued performance on the job? Yes, game-based testing may increase engagement motivation for audiences with marginal attention spans, but if it does not result in increased job performance, do we really care?

In stark contrast to the game-based testing results is learning simulations with problem-based challenges that require learning on the part of the participant, as well as dealing with performance cues (stimuli) that closely mirror actual work-based environments.

[Dolly Joseph]

Jim said: This poses some intersting questions regarding the efficacy of game-based assessment or "learning reformancement" activities.

It has nothing to do with "game-based assessment" it has everything to do with ASSESSMENT. That is the whole problem with these thiagian frame games. Game-based assessment does NOT have to include multiple choice prompts nor true false.

Jim said: Are we teaching the learners to play games, or achieve valued performance on the job? Yes, game-based testing may increase engagement motivation for audiences with marginal attention spans, but if it does not result in increased job performance, do we really care? In stark contrast to the game-based testing results is learning simulations with problem-based challenges that require learning on the part of the participant, as well as dealing with performance cues (stimuli) that closely mirror actual work-based environments.

I would mostly agree with this. Again, it's related to semantics and jargon. If you are using game to refer to only frame-games, this is true. If you are using Game in the true meaning, a game's assessment can use any reasonable assessment method.