The Total Learner Experience

Don't Be Half-Assessed

Assessment is used to measure learning outcomes, but if you see it only as a testing tool you’re missing half its value. Assessment should not be punitive; it should uncover the learner’s strengths and help identify areas for improvement. Well planned and integrated assessment creates opportunities for learners to reflect on their learning, apply new skills and knowledge, and also enables the business to recognize ROI on the learning intervention. Adapting existing models can take you much further than re-creating the wheel. In this post, we examine some guiding principles for effective assessment.

An excellent example of assessment as a way to support learning goals is Adobe’s Certified Associate practice tests. Learners can take a certification exam without any structured practice, but there is also an unlimited certification prep test package. The practice test closely mimics the structure of the final exam. Both combine multiple choice questions and simulations of the Adobe environment. The multiple choice questions require the learner to demonstrate knowledge of design concepts and the production process. The simulations require the learner to complete a task within the simulated software application interface. Keyboard shortcuts are disabled, but otherwise, learners can use any correct method they choose to complete the task, using menus or panels as appropriate.

Another good example is the Sun (now Oracle) Java certification paths. Each path contains a test prep “kit” that includes preparation recommendations, additional resources, a practice test, and a re-take policy. Each path is designed to prepare the learner to achieve a specific level of certification, and used as a benchmark against industry standards. These certifications are recognized by employers and can advance a person in their career.

Accreditation or certification can be used to validate mastery of a topic, but this is not the only way that assessment can be useful. By seeing assessment an integral part of instruction, we can support the learner’s career development while measuring true performance for the business.  

Factors that make an ASSESSMENT tool also a useful INSTRUCTIONAL tool are:

• Authenticity
  • The learner performs the task in context, not recalling theory, but actually demonstrating competency.
• Open ended
  • Because the end product is assessed, not the method used to get there, learners are able to use whatever menus or panels they choose.
• Learning while doing
  • Learners use contextual clues and critical thinking to complete tasks. They may not know how to adjust alpha levels in Photoshop, but they may know to investigate the color panel to find them.
• Self-reporting
  • Learners can mark questions that they’d like to return to if they have time or opportunity.
• Cumulative time
  • The test is timed with one master clock, not with individual times for certain sections or items.
• Feedback
  • Learners receive feedback on each item, with notes about the correct answer.
• Tracking
  • Performance from one practice test to another is tracked.

Assessment is not something that should only occur in a testing situation, indicating pass/fail rates, but it should be integrated throughout instruction to allow the learner to know how they are doing, so they can learn more effectively. Don’t leave your learner half-assessed!

 

Create Losing Designs

We spend a great deal of time planning for learner success. Do we ever create the ability for learners to lose? Should we?

Our society puts a premium on “winning” -- nowadays it seems like every team gets a trophy -- not just the winner. Americans are culturally bound to the ideal of winning. However, in the real world people lose all of the time: athletes lose games and tournaments, politicians lose elections. In high stakes situations like these, losing has a real consequence. How does this affect learning?

Low stakes losing occurs every day. When we lose, we tend to either give up, or continue to practice to overcome the failure. Any casual gamer is familiar with losing and attempting to overcome the loss by quickly replaying the game. Losing, in this context, can be a big motivator, driving the will to practice, which can lead to increased skill. Learning to lose effectively is actually a skill in itself. Resiliency in the face of losing and using strategies to improve is an important life lesson and professional skill. If losing always equates to complete failure, then learners stop striving, stop attempting creative solutions, and see themselves as incapable.

In learning design, a primary goal should be to create an atmosphere where losing is acceptable and intrinsically motivates the learner to try again. The potential learning value from this type of experience is measured not by how often the player loses, but by how much they improve through repeated practice.

In your training design include opportunities for low stakes losing before learners are thrust into their high stakes, real-world situations -- be it a qualifying exam or on the job performance.

Recommendations for a “losing” design:

• Make content available on demand so that learners can review material as needed
  • A common mistake is being overly controlling with content. If the goal is for learners to gain knowledge and skills, shouldn’t they have free access to the tools that can make that happen?
• Communicate how the learner is performing relative to passing scores or via peer-to-peer review as appropriate
  • By communicating how comparatively well they are doing, learners can better sense where they can improve. This also gets their competitive juices flowing.
• Allow plenty of practice opportunities before the final assessment occurs
  • If the only time that learners can demonstrate knowledge is a final graded test, their opportunities to “lose” are reduced to one high stakes moment. Build in low stakes assessment opportunities that prepare learners for the one that counts.

On Winning

“Training” and “competition” are not usually words that are associated with each other. However, competition is an innate motivator and humans by nature enjoy winning.

Certain aspects of winning are universal to all competitive activities, including learning. A recent Newsweek article about winning provides insight into how instructional designers can create more engaging training. The author notes that winning by itself is not the most compelling impetus, but that winning while a competitor loses is more satisfying (this would seem obvious to anyone with siblings).

Rather than using  a “task completion” metaphor, instructional designers should use a gaming and winning metaphor when designing training. Rote tasks can be made more engaging if instead of simply reading and reacting in a safe environment, the learner triumphs over a tension-filled activity. Similarly, you can provide competitive opportunities with other learners virtually.

In many K-12 situations “safe” learning environments take out so much of the excitement of competition and rating. You have an obligation to make sure every beginner learner succeeds, but you also have an obligation to groom the special talents that individuals have as well.

How do you turn cognitive tasks into challenge?

Learning objectives can remain the same, but it’s the way that it is presented that changes. You don't have to completely redesign your training to make it more challenging. Consider these simple ideas:

Allow learners to be wrong

• Allow and penalize for incorrect answers. If learners can complete a course by merely clicking through content, they have little reason to engage with the content. Activities that allow for “failure” can create “good tension”.

Add variable scoring

• Reward learners for learning more difficult material by acknowledging that all content is not equal.

Add timed components for some activities

• While not appropriate for all activities, it does create a feeling of tension.

Allow for replay opportunities

• This reduces some of the negative aspects of the capability for “failure” allowed by other competitive components.

Create opportunities to share leaderboard scores outside of the learning environment

• Rather than learning taking place within a silo, authentic learning events can bleed into the larger community.

A simple multiple choice game can be either boring or competitive with a simple design tweak:

Boring

• The learner answers a question and then views the answers tagged with "correct" and "incorrect" feedback.

Competitive

• The learner answers a question. They receive variable points based upon the correctness of the answer, difficulty of the challenge and the speed with which they answered. They are then shown their score relative to other learners. The feedback is also contextual and continues the gameplay.

Take some lessons from this short quiz. How many times did you feel compelled to play?

First 2011 Post: An ID Only Needs to Do 3 Things

Hi all, Happy New Year! Hope 2011 gives you what 2010 didn't... ;). 2010 was interesting on several fronts -- and work-wise was a year of learning and learning design for me! More on that soon.

To kick-off 2011 I've decided to make only one resolution: to make work simpler. That doesn't mean to make my work less engaging, interesting, or informative... it really means to undress it, to de-accessorize it. In that vein, it dawned on me that as an instructional designer I really only need to do three things:

1. Provide information before interaction. This is a traditional approach to instruction — inform the learner about the subject matter and communicate what it does (This is what the button does when you click it).
2. Offer feedback during interaction. This is how the object behaves so that it reinforces what it does, or what it’s doing (The button displays the help screen).
3. Encourage reflection after interaction. This is what the person should know after she has finished interacting.

By understanding how to "do" these three things, you can create more effective learning experiences. That's what we really strive for -- and there's been a lot of talk about "dressing up" what we do... build engagement, heighten emotion, ensure contribution... sure, there are valid points to make on all these fronts. But if you unwrap the glitz, you'll find that these three elements are the foundation to anything instructional. How you accessorize it is well... up to you and your audience.

Do You Have a Curriculum Strategy?

Many training organizations may think their curriculum is aligned to the company's business needs. Can you confidently say that yours is?

When you look holistically at your curriculum, does it:

• Meet definable and observable performance standards?
• Enable you to determine if learner achievement of the curriculum objectives occurs?
• Allow you to determine if the instructional elements effectively meet the curriculum objectives?

Do you evaluate the effectiveness of your instructional programs along the following criteria:

• The instructional content teaches to the objectives specified
• The time required for instructional programs are appropriately aligned to the student time available
• Instructional materials and resources are easily accessible and contextual to the student's physical and virtual environments
• Appropriate assessment mechanisms are included that provide performance-related remediation
• Performance-related data is tracked and stored
• The instructional activities are evaluated to determine effectiveness

Too often, training organizations become siloed and redundant in their efforts to train the workforce. Consistently measuring the effectiveness of your curriculum and ensuring it's aligned not only to identified business needs, but also to the organization's competencies, will help you provide a measurable and sustainable ROI to the business.

The question is: how do you intend to ensure your curriculum strategy stands up to the test of alignment and effectiveness? Consider these three actions to help you get started:

• Evaluate all curriculum and instructional components annually
• Leverage business intelligence software to gather usage data on your course catalog. Consider deeper analysis on courses and/or programs that are not being used or accessed frequently
• Ensure that all courses fit into a program or curriculum that is tied directly to a business need
• Question the request for training if it can't be easily shown what the desired outcome is or how it is aligned to a business need or goal
• Provide supplemental support services for post-course evaluation

The primary success factor resulting from a well-defined curriculum strategy is performance improvement. By focusing your efforts on an appropriate strategy, you will move one-step closer to demonstrating true ROI to the business and the audiences you support.

Building Motivation in Online Learning

When creating online learning, the primary goal is to create a course or program that fosters "a continuing motivation to learn," which is defined by Maehr as "voluntary engagement in continuing to learn more about a given topic."

The complexity of human motivation makes it hard to find a one-solution-fits-all model for learners. However, the ARCS model establishes basic criteria an instructional designer can leverage to increase the potential for establishing learner motivation while reducing learner malaise, frustration, and/or dropout. 

The ARCS model consists of these four components:

• Attention
• Relevance
• Confidence
• Satisfaction

Consider the following tactics for each ARCS component. However, there is no substitute for proper analysis to uncover the motivational elements specific to your audience. Without that analysis, you are likely to include too many, too few, or wrong motivational elements.

Attention:

Gain learner attention by integrating:

• High-resolution graphics
• Interactivity and animation
• Conflict and failure
• Problem-solving and inquiry

Relevance:

Provide relevance to the learner's situation:

• Describe clear goals that are consistent with learner expectations
• Connect content to the learner's prior experience and/or knowledge
• Include guided instruction and navigational systems (while allowing self-determination in the consumption of the content)
• Incorporate "authentic" activities based on learner's job or tasks

Confidence:

Make the learner feel confident and aware of learning outcomes:

• Communicate clear expectations for successful completion
• Teach to the learner's "Zone of Proximal Development" -- avoid tasks and/or activities that are too easy or too hard
• Include activities that require critical thinking and minimize guessing
• Provide positive and contextual feedback that emphasizes learner achievement

Satisfaction:

Design for happy learners by:

• Recognizing learner achievement through reward mechanisms and in-context encouragement
• Providing opportunities for the learner to apply what they're learning
• Designing the right amount of work for the subject material
• Creating consistency between the content and assessment mechanisms throughout the material

Also, consider these additional design strategies:

• Reduce the word count on the screen compared with printed or narrated text
• Improve the quality of quizzes (consider quizzes as a motivational tool)
• Incorporate more interactive elements
• Create an automatic messaging system triggered by learner progress that can send positive email providing encouragement, reminders, empathy, and advice. 

References:

• 1986. Maehr, M.L. The motivation factor: A theory of personal investment. Lexington Books.
• 2004. Keller, J.M. and Suzuki, K. Learner motivation and E-learning design: a multinationally validated process. Florida State University, USA, Iwate Prefectural University, Japan.

What the Apple/Adobe Flash Debate Means for Learning Developers

I've been following the "Apple vs. Adobe" debate about Flash for several weeks now, and have recently come to the conclusion that it's not really about Apple forcing its will on Adobe as many in the media seem to be reporting. A milestone occurred yesterday when Adobe's Flash Platform product manager, Dan Chambers blogged about Adobe giving up on creating further iPhone app development tools.

As a designer and developer of corporate training materials, I obviously have a vested interest in this debate. Why, you may ask? Well, if you scan the landscape of eLearning development you can be rest assured that a large majority of online educational material relies on Adobe's Flash platform for its delivery.

I go back all the way to the pre-Internet Authorware days... yes, Authorware 1.0 when it was Mac-only authoring software. I loved Authorware. It was designed specifically for development of computer-based training programs. As the Internet era dawned and Macromedia put its energy behind Flash, I saw the writing on the wall for Authorware. I mean, at first, it was great... Macromedia kept rolling out new versions, and even put a lot of effort into the Authorware web player. The dark days eventually came, and I knew when I talked to some friends at Macromedia what was happening when they lamented that no new Authorware engineers were being hired... and there was only one left at that time!

As Flash emerged, the learning developer community embraced it and began retrofitting it for educational interactivity. I never completely transitioned from the flowline to the timeline. I still remember Robert Milton's explanation of the flowline in Authorware training, and how dang logical it was. For me, the timeline was anything BUT logical.

When Flash became the de facto standard for learning development, chaos ensued:

• Constant player updates adversely affected courses designed in earlier versions of Flash
• Just TRY to get Flash to communicate with any LMS without having a programming wizard (if you could find one). I used to beg Andrew Chemey (a Macromedia engineer) to help, and he could always make it work, but he could never explain how.
• Say goodbye to Authorware's text handling capabilities and welcome Flash's completely horrible rendering of fonts.
• Throw out the idea of modular development, which was Authorware's coup de grace.
• Expect a new or heavily modified scripting language with major updates
• And the list can go on

All this for a "light web player" -- people complained about the 3-6mb Authorware web player (this was pre-broadband). Granted, Flash has made important progress as it relates to learning development, but this whole "Apple/Adobe" conversation got me to thinking about the state of tools in eLearning design and development.

Flash the parent, and all its unruly siblings such as Articulate, Captivate, Camtasia... all the tools that produce .swf files, are essential for various tasks in eLearning development. And Adobe seems to be whining that Apple is being a big bully about the Flash platform. I think it's actually the opposite. I think Adobe has been the bully over the last several years.

Apple is basically saying, "we don't want to have to adopt or promote a company's proprietary technology on our platform when it's not consistent with the technologies that serve as the foundation for our platform." Flash is anything but open.. it's locked technology that requires software available only from the vendor. It's not open source, or even open for that matter.

I remember the days before the Flash player (remember DHTML?). I remember coding HTML to work in various versions of browsers, and it wasn't pretty. Flash made life a whole lot easier on several fronts because it created its own "walled garden" that was guaranteed to operate consistently across multiple operating systems and browsers. I fear HTML5 will bring back the whole issue of compatibility and interoperability.

I also know from first-hand experience that creating a software product with the Flash platform is fraught with potential issues because any potential change in player versions could cripple features/functionality. And try getting updates from Adobe on changes, or roadmaps on functionality.

I tend to side more with Apple on this matter, only because they're really not bullying Adobe -- they're just saying, look we have our "walled garden" and we've decided what the best technologies are to support it. You have your proprietary technology that may be a standard with a specific group of developers, but it's not an open standard for the Internet.

What does this mean for learning developers? I think just as we all were forced to move away from Authorware several years ago, we need to have our eye on where the industry is going. I tend to think more open technologies will always win at the end of the day. But that must be countered with how we get our jobs done as well. Tools like Flash do allow developers to get work done in an efficient manner. Change is guaranteed, and learning new tools and languages will always be a task we must perform.

Social Play

Part 6 of 6 on an ongoing series about the six activity modes. Part 6 of 6 on an ongoing series about the six activity modes.

Read parts 1, 2, 3, 4, & 5.

Social Play takes several different forms, all centered around social interactions between players, avatars and opponents.

One form of social play is the control of characters in interactions with other computer generated characters, a la old school The Sims. This seems to have been replaced by the more interesting (maybe) interactions of avatars controlled by humans in online environments like Second Life. People can meet strangers, develop relationships, even get married in these virtual communities. These interactions are set apart by their representation of 3-D avatars which can interact in a virtual world.

Another kind of social interaction is the live chats which can take place in a variety of online forums. Again, here, the player can interact with a friends and strangers, but here it takes place with a text dialogue. Typically it is external to the game play.

Multiplayer games allow all sorts of cooperation and competition between players. Sometimes players will alternate between competing and cooperating with the same groups of players.

Often in digital games we concentrate on the controlled interactions that we create, but there are a great number of interactions that happen outside of the planned gaming environment. I often observe the coaching and tutoring that goes on between people in the same physical environment. While it is easy to design for one player per station, oftentimes players enjoy cooperating and sharing the same station. I also will model how to use a game or an activity on a station to elicit excitement and engagement with students. Let's not forget that a people still like to interact in real life together.

Designing for the Auto-Compulsion Loop (ACL)

Instructional Designers can take a lesson from the popularity of social games such as Farmville and Mafia Wars. These social games are played primarily through Facebook and allow players to play asynchronously in competition and cooperation with their friends. A recent study conducted by Popcap resulted in some interesting statistics (via Raph Koster's website):

• There are around 100 million social game players
  
• 24% of US and UK Internet users play social games at least once a week
  
• Slightly more women than men
  
• Average age of 43
  
• Only 6% of them are 21 and younger
  
• They’re not all housewives: 41% of them work full time, and only 11% are homemakers.
  
• 2/3 of them also play other sorts of games — casual or hardcore.
  
• And if they, do they spend more time on the social games than on the casual or hardcore games
  
• And they’re stuck: more than half have been playing for a year.
  
• A third play more than once a day
  
• 2/3 of them play over an hour a week — 12% report over 10 hours.
  
• And they report that their social game playtime is increasing
  
• For half of them, it’s the MAIN reason they use Facebook.
  
• And also, half of them say it’s very unlikely they will spend real money

The underlying phenomenon behind these statistics is in the factors that motivate players to repeatedly engage -- this audience isn't the stereotypical hardcore 20 hour a week WoW addict -- the motivating factors seem to be the granular achievements earned while playing combined with the inclusion of the player's real friends. 

8 Tips for Designing an ARG (as a Learning Experience)

In November 2009 myself, Brandon, and Dolly published an article about Alternate Reality Games in the eLearning Guild's Learning Solutions e-magazine (download a copy here).

Since the article was published, several instructional designers have asked us to pull out some practical tips on how they can get started designing their own ARG. For this post, we thought we would reference eight key tips from the article that provide a starting point for you to consider as you embark on your ARG design.

1. Establish Clear Goals	Document what are you trying to achieve with the game. What are you trying to get your audience to experiment with or learn? After all, participants should be learning something while having fun. The Threshold, a game designed by Cisco had clear goals from the beginning, generating very positive feedback from players: "The Threshold was a four-week long alternate reality game designed to build teamwork, build awareness of Cisco products and services, and to provide an entertaining and fun experience."
2. Tell a Meaningful and Entertaining Story	Make your story relevant to your audience and engage them in the unfamiliar by tapping into what is familiar to them. Don't get too carried away with the story to a point you forget it's all about the audience. The designers at LAMP (Laboratory for Advanced Media Production, Australia) felt that an intriguing story about the school's production studio would motivate new students to participate in their game. The "Safesets, Sabotage and Madame Blash" mini ARG (mARG) is an excellent example of a game that has a relevant setting and background story that is sure to entice players.
3. Don't Become Overly Attached to Your Story and Characters	Be quick to adapt to your audience's responses. Let them participate in the design process as the story unfolds. Interact directly with them (via non-player characters), respond to their reactions, be prepared to change strategies, puzzles, and/or storylines to conform to how the audience is changing the game. The Cloudmarkers were a self-organized group of players involved in the famous ARG "The Beast." They initiated a Yahoo! group in order to discuss the latest findings in the game. Their discussion was closely followed by the developers and, since ARGs are normally developed as they are delivered, the developers would often include members of the group in the game plot as it unfolded.
4. Plan, Plan, Plan	Even though you need to be flexible, lay out a plan of the events and outcomes. For instance, create a flowchart of the action items such as clues, puzzles, characters interactions with the players, next moves, etc. This will make it easier for you to visualize and strategize how the game is going to be played and will give you a better idea of when, what and how to adapt. Here is an example of a flowchart being used to plan the actions in Ascendency Point, a live event for the game Perplex City.
5. Create a Reward System	No matter how cool your game design is and how engaging your characters and story are, chances are you may still have difficulty getting people to participate. Instill competitiveness and collaboration by offering rewards. Make it fun. In the Duke's Quest ARG we created, it was a challenge to get players involved without a reward system. This can be a challenge when organizing small group ARGs which tend to involve even those who might not be interested in games just for "fun's" sake. A reward system might be that extra incentive they need to get involved. The Dark Knight ARG (developed for The Dark Knight movie), had a message from the Joker requesting players to submit photos of themselves in Joker make-up in order to receive a mystery package in the mail.
6. Leverage Social Media	Consider using social media utilities to both design and deliver the game. Services such as blogs, YouTube, Twitter, Facebook and others can serve as a medium for puzzle delivery and for players to interact with each other and non-player characters.
7. Have Closure	It's OK to have cliffhangers for a next instance of the game, but make sure your game has closure. Players need to feel like they've achieved something at the end. This can be a live event at which players gather with characters for a final announcement, conclude the game going over their achievements, distribute rewards or even set up expectations for a segue.
8. Play an ARG	It is true, the best way to design something effective might be to actually get your hands dirty. If you want to know how to design a good ARG, play an ARG. By actively joining a game you may learn how to relate to your audience's feelings and expectations when designing your own ARG. You can find active ARGs and read up on old games on websites and communities like ARGnet.

I hope these tips can help you get started designing your own ARG. The potential for active engagement is inherent to ARGs and may help you deliver a truly effective and experiential learning activity. I'm interested in hearing your thoughts on ARGs in general. Please feel free to contact me or add to the comments here with your responses to these questions:

• Have you designed or played an ARG before?
• Was it specifically designed to teach or motivate the audience to experiment with technology or a concept?
• What are your best practices and advice for those interested in using ARGs for learning?