Can you cheat on a simulation?

When simulations are used in an assessment context does it actually decrease the possibility of cheating? I think so. One classic way to cheat is to memorize the answers. In the context of a simulation-based assessment which requires performance isn't that the same as learning? If a test-taker memorizes the way to perform a simulation they have in fact, "learned" to perform the task. This obviously does not eliminate the problem of coaching but it does limit some of the risk.

What is a Simulation Anyway?

I have heard a wide range of uses for the term "simulation" lately with an equally wide range of meanings. Is a simulation a simple screen capture with a single correct path via an active hypertext link? Or is it a sophisticated representation of the real thing with complex programming, logic and multiple correct paths? The answer to that question has direct application to using simulations in learning and testing environments.

For example, a single path, screen capture type "simulation" might be an effective tool to teach a specific skill. Yet, that same simulation would be a very poor method to measure that same skill. Imagine, a single path simulation with only one active link to advance the simulation. If I used that in a testing environment I could argue that I am not testing the skill in question but the test takers ability to deduce the correct answer by play what I call "virtual bumper cars". Given enough time anyone could eliminate incorrect answers by simply trying to click on the response and discovering that the particular link was not active. In this case a more sophisticated simulation with multiple correct paths or the ability to follow an incorrect path might prove to be a better measurement of the skill.

So, as you consider incorporating simulations into your learning and testing programs remember that not simulations are the same and more importantly certain types of simulations lend themselves better to learning environments and others to testing applications.

Virtually better than the real thing

We hear a lot about simulations lately. It seems that they are popping up everywhere and there is no small buzz about them in the educational community. But are they really better than the real thing?

The answer to that question is an unequivocal “yes and no”. I think that whenever possible students should have hands-on learning opportunities with the real thing – in our world that means technology. The more experience they can get designing, planning, installing, configuring, maintaining and troubleshooting real technology the better. But that experience can be costly and even disastrous.

For example, not too long ago I was working with our SQL-based CRM system and I inadvertently erased the entire customer database! Oops! I must confess that it was a lesson that I will NEVER forget - but at what cost? Fortunately we were able to recover the vast majority of the data but it was expensive and disruptive to our business. So I am compelled to argue that although an extremely effective learning event, it was not an efficient or recommended learning path.

My point is although experience is often the best teacher it may not be the most efficient way to go about it. This is where simulated-labs come into focus.

Simulated labs, if done properly, can provide students with as much reality as is necessary to give them the hands on experience they need to learn real skills. The upside is that they do not have to erase the entire corporate database to learn one feature of SQL. They are other reasons that simulated-labs are a good idea. Here are a few:

First, they are simulated labs are scalable. Real technology comes in the size that it comes in with all the features and functionality intact. Simulated labs can focus on just one feature set or one aspect of the technology. I do not need an entire live network, for example, to add a user. With smaller “chunks” of functional technology hands-on learning become affordable and feasible. It can also be delivered from the desktop or laptop – something very difficult to do when you consider the size of you average computer lab, server rack or hardware lab.

Second, simulated labs are flexible. Because simulated labs are a digital recreation of the technology that they represent they can be configured in an infinite number of combinations, situations and technical configurations. In the real world to help students learn troubleshooting skills the instructor would have to literally break something on a machine so the student could diagnose the problem and then fix it. If the teacher student ratio were one-to-one and if there were an amazing hardware budget then this scenario might work. Unfortunately, this is not the case for most of us. To configure machines for 20 students for one simple exercise could take all day and that would be for only one exercise that would not even fill the entire lecture period. To adequately provide students with the hundreds if not thousands of hands-on learning experiences they need to adequately learn how to use technology simulated labs become a viable option.

Third, simulated labs can be portable. With the advent of the internet the ability to provide hands-on experience at a distance is now possible. Not too long ago, they only way to gain hands-on experience was to physically go to a computer lab. This has obviously limitations in both capacity and accessibility. Now with simulated labs, students can take a copy of their own computer lab home with them. This has significant ramifications for distance education, homework and those with disabilities.

Fourth, simulated labs are designed for learning. Real technology is not designed to teach. It is designed to accomplished the tasks it was intended to accomplish (i.e. route packets, process data, serve web content etc.). Simulated labs, on the other hand are instructional by design. Their purpose is to teach. Therefore they are designed to provide performance evaluation, feedback and even instruction. Learning scenarios can be reset and practiced over and over again. Technical processes can be artificially slowed down to demonstrate difficult concepts or time consuming processes can be sped up to not waste time waiting for the process to be completed. Try doing that with real technology!

Fifth, simulated labs are affordable. When you consider the cost of technology and the rapid rate of technical obsolescence and multiply it by the number of students that are in the program, the result is a significant budgetary issue. Unfortunately few schools are funded to the level they would like or need to be. Earlier simulations and simulated labs were very expensive and cost prohibitive to the average user. Now a online simulated labs can be found for the cost of a text book.

So, I must confess that although I am die hard technology fan and believe that hands-on experience with real technology is an essential component of the skills acquisition process it is not the only way to learn. In fact, I would dare say that hands-on learning via simulated labs is a much better way to prepare for more sophisticated experience with real technology. Let learning occur with hands-on experience with high fidelity simulated labs and wait to erase the corporate database until you are fully qualified.

Virtual Certification Testing

A Vision of the Future
You are the new site administrator for a major financial institution. Besides keeping the network running smoothly you must battle an assortment of hackers, crackers, thieves, and freaks with the goal of getting to your network data. There was a security breach only a month ago. Today you get alarming news: network throughput to headquarters in NewYork is down by 42%, packet collisions are up by 113% across the LAN, server utilization is nearing 100%, disk space is low. You receive a security alert from your audit software indicating that permissions and rights have been modified on certain secure folders. You’ve got problem–a big problem.

What if this was not only the cause of a full-scale migraine but also part of your certification exam? Science fiction? Nope. In the not-too-distant future, certification exams will do more than give a percentage or indicate a “pass” or “fail” on a single vendor technology. Certifications will present real-world, problem-based scenarios, measuring how many years experience you have and testing the skills required in a heterogeneous environment. Current strides in simulation technology make all of this possible.

Simulations Today
For years we have heard mention of simulations and how they will impact the certification industry someday. The reality is that simulations are here now and they are gaining a critical mass that will ultimately result in a quantum leap in how we are tested.

Certification providers such as Novell and Cisco have already incorporated simulations into their exams. Microsoft has announced its own plans to add simulations to its certification testing. Virtually every other major certification provider is exploring similar options.

Recently, Cisco began delivering simulations in their tests, requiring candidates to interact with simulated Cisco networks and perform real-world tasks. Cisco argues that–

“Because the scenarios used in simulations provide a more realistic environment, they are better able to measure degree of knowledge and skill; for example, that a candidate can complete a task with the right commands, in the right order, and understands when the task is complete” (emphasis added).

The Cisco announcement pitches the addition of simulation-based testing as part of the “continuous improvement” of the exam to “ensure that employers and employees continue to rely on Cisco certifications to meet the demand for technical skills in networking.” What employers demand–and what the certification industry is rushing to provide–is a certification process that certifies that students can perform the tasks of the trade rather than regurgitate answers about IT theory. The complaint of employers is that newly certified job candidates who have managed to pass certification exams are too often unable to deliver when it comes to the everyday tasks of running a network. Many employers have discovered a painful disconnect between the certification credential and the skills that only come from experience. The message is clear: “memorization is not enough,” and simulations are increasingly used to provide true “performance-based testing.”

The growing consensus among certification providers is that adding simulations to a certification program builds both the confidence of the individual and the employer in the meaning and reliability of the certification process. Test-takers seem to concur, enthusiastically embracing the addition of simulation exercises and regarding them as a more real experience.

It is important to understand, however, that not all simulations are created equal. Two very different types of simulation technologies can be used in testing environments—we call them Dumb Simulations and Intelligent Simulations.

Dumb Simulations
“Dumb” simulations (like Dumb Terminals without internal logic, rules or processing ability) basically run the user through a series of screen captures. Designers of dumb simulations simply link individual screens to recreate the simulated task. These types of simulations allow only a very finite and limited interaction with the technology, often allowing only one pre-determined path to the end result. Dumb simulations provide little logic, few rules, and limited functionality. For example, if your task were to add a new printer in Windows 2000, you would be presented with a Windows 2000 desktop with an active link to the Start button. At the extreme this button might be the only active spot on the entire desktop. After clicking the Start button, the simulation moves forward to the next screen, where a link would be defined for the Printers menu item. This process would continue until the very last screen of the simulation. Users progress through the simulation by taking the single path through the screens, clicking on the hotspots on each screen. Simulations created in this manner prevent users from deviating from the predetermined course.

Because “dumb” simulations are often created with screen captures, there are very real pressures to keep file sizes down and to keep the linking as simple as possible. In fact, most of these simulations do not define paths for wrong answers–a major limitation for evaluation purposes. If “dumb” simulations were implemented in a testing environment, users could easily discover the correct path through a process of elimination.

“Dumb” simulations are cheaper and easier to create in the short run. They lend themselves to a single program, a single task, and/or single path applications. Although “dumb” simulations may have some instructional value, they have serious limitations in the testing environment. Dumb simulations offer the designer a cheap short-cut to providing a simulation-like experience.

Intelligent Simulations
At the other end of the simulation spectrum you have what are called “Intelligent” simulations They distinguish themselves by the wealth of options presented to the user, by the dynamic nature of the simulation, and by the complexity of logic and rules that make the simulations work. These simulations generally rely on a simulation engine or simulator that is programmed to act like the real thing. For example, if you have a Windows 2000 simulator, you could have a multitude of individual simulations that work with it.

Let’s say a particular simulation asks you to add a printer. With the Windows 2000 simulator, you simply navigate through the simulated Windows environment following whichever path you desire. You could even do things that are completed unrelated to the original task.

The ability to do things other than the original task is very important in a testing environment. The system responds to your input exactly as a real system would respond. You can perform any task in the simulation that you could perform in the actual system.
With open simulations it is impossible to discover the correct path by trial and error. In fact, the path you take to achieve the task need not even be evaluated. Intelligent simulations emphasize the end result–not the path. This flexibility is a key characteristic of open simulations.

Creating open simulations typically requires a team of–
Designers, who identify the task to be simulated (usually setting up a specific network configuration), define how the simulation should work, and set criteria for successful completion.
Programmers, who take the design specifications and create the simulation programs that replicate the system environment.
Testers, who verify that the simulation functions as required.

This overhead makes producing the first simulation more expensive than producing a single “dumb” simulation. But once you have programmed the functionality, rules, commands and logic of the underlying simulators, additional simulations can be created easily with dramatically diminishing marginal costs.

“Intelligent” simulations lend themselves to simulating multiple programs, such as within a desktop or network operating system, or when integrating multiple technologies from different vendors (such as Microsoft, Cisco and Novell). The ability to test troubleshooting skills is simply not possible with any other approach. “Intelligent” simulations monitor the states of a variety of programs and respond appropriately to user input as they make changes to the system.
Simulations, the Driving Force of Change
With a basic understanding of the distinctions between simulations, there are four key trends that you should keep in mind.

Testing in a Problem-based Environment
Today, the dominant instructional design methodology for learning is based on a topical organization. Certification exams follow suit. In contrast, simulation technology allows for the learning and ultimately the testing methodology to be organized around a problem-based scenario. Dr. M. David Merrill of Utah State University, a leading instructional designer, has developed an instructional design approach he calls the “Pebble-in-the-Pond Model for Instructional Design.” This model is based on a sequence of progressively challenging simulation-based problems that engage the learner in real-world situations. This approach can easily be applied to the testing environment using a similar sequence of real-world problems. The ability to deliver real-world problems in a simulated environment will allow certification candidates to show their stuff in environments that are closer to the real thing. Creating certification exams that will be delivered in a simulated environment fuels the need to additional simulation types including hardware simulations.

Simulating Hardware
Many of the IT simulations you find on the market today simulate software, operating systems, and networks. That will not remain the case for long. Simulations are evolving to include hardware components and peripheral devices. Fueled by advances in 3D rendering, virtual reality, and gaming technologies, simulations will increasingly be developed for hardware. This should come as no surprise as you consider that the newest entrants to the greater IT certification community have grown up with Nintendo, Sega, and Sony Playstations. This group is not only comfortable navigating virtual worlds, but they are also sophisticated connoisseurs of graphical environments. Creating a wider array of simulations, including simulating hardware, opens the door to truly heterogeneous or multi-vendor simulations.

Simulating Multi-vendor Technologies
Simulations today, especially “dumb” simulations, are infamously homogeneous. They are developed to simulate a specific concept or task. “Intelligent” simulations, however, offer the opportunity for connecting simulations together to create a simulated network. If developed correctly, these linked simulations operate like the real thing. Imagine a Windows 2000 Desktop simulation connecting to a NetWare Server simulation communicating through a Cisco Router simulation. Settings can be reconfigured, things can be broken, and equipment and software can be added, all running from a DVD or CD on your laptop. This ability to create heterogeneous, simulated networks, with full-scale hardware, software, and operating systems, opens the door to new types of certification programs and training materials—products that more closely resemble the realities of the IT world and allow for the test to measure your real-world experience.

Measuring Experience
As we all know, certification score reports are articulated in percentages and pass/fail marks. Simulations will change all that. With simulation-based assessment technology and job task standards, we are not far from the time when students will not be given a pass/fail report, but rather they will be judged based on the amount of real-world experience they have. Score reports of the future will identify how many years experience students have in a particular field. You see this trend already beginning with experience-based prerequisites for courses.

Microsoft states that:

A Microsoft Windows 2000 MCSE candidate is expected to have at least one year of hands-on experience implementing and administering a network operating system to pass the exam.

Today, experience requirements are loose suggestions—in the future they will become quantifiable assets. Not only will prerequisites be stated in terms of experience, but also the outcome of training will be stated in years. Imagine your score report stating, “Congratulations, you have demonstrated 6.5 years of experience in Network Administration.”
Looking Forward
Simulations are changing the landscape of certification testing. Regardless of whether multiple-choice exams were adequate for the task, there is little doubt that the certification industry is quickly moving towards simulation-based testing. Initially driven by a concern over “Paper Certification,” the inevitable trend towards simulations is now driven by the need of certification providers to keep up with industry demands and technological innovations.

When you integrate the power of simulations into testing environments, the result is not just an alternative way of doing things. It is a fundamentally new and better way to be evaluated. IT vendors will benefit because simulation items are inherently more difficult to reproduce on brain dumps that traditional test items. Simulation items can also be created more quickly and cost effectively, without the need to create “distractors.” Candidates will benefit because simulations are a better measurement of their skills and they provide a more enjoyable testing experience that resembles their job. Employers will experience an increased confidence in the certification credential and in the people they hire. We are just beginning to see the influence of simulation technology on our industry, Similar to the magnitude of impact that photocopiers, e-mail, and cell phones have transformed the way we conduct our daily business, simulations will transform the certification industry in the near future.

The Lessons Only Experience Can Teach

I remember the first time my young son enthusiastically got on his bike. Training wheels in place, he roared off not fully understanding the physics of acceleration or momentum. This lasted about 10 seconds until he realized that in his eagerness to “ride his bike” he had neglected to properly prepare for unavoidable moment of actually stopping his bike. It had never occurred to him he would actually want to stop – until the bushes that is. As he came sniffling out of the bushes with his ego more bruised than his body he said, “How do I stop?”

The next few minutes of hands-on learning were priceless. He was not only motivated to learn but he was able to apply what I said directly to something that was meaningful to him. He has never forgotten that experience and neither have I.

From an instructional perspective we had created what I believe to be the ideal learning environment. Although I had explained to him the basics of riding his bike and he had passed my oral assessment of his understanding, he was not able to translate a conceptual understanding into practical, useful skills. In this case and many others, there are simply lessons that only experience can teach.

One of my favorite Spanish proverbs states that “you cannot learn to swim in a field”. I think this can be applied to nearly every education subject. Without hands-on application the theory is largely meaningless. An argument can be made for the need for theory or knowledge-based education as a foundation. But when too much attention is paid to rote memorization and not application of that knowledge in practical situations the learning is often perceived as meaningless. I would even go so far as to say that in those circumstances real learning has not occurred.

This dilemma of book-smart v. street-smart is very pronounced in IT education. It is one thing to “know a lot about computers” and to be able to “do a lot with computers”. This is common sense really. Would you want to fly in a plane with a pilot who did not have flight experience? How about a surgeon who had never dissected a cadaver? With networks its very similar. I may know a lot about networks – I might even be able to pass an industry recognized certification exam – but I may have never touched a network in my life. Scared? You should be.

The problem is not that we do not realize that hands-on learning is important. The problem is that it is so challenging and often expensive to provide hands-on experiences in a learning environment.

Take the classic computer lab. Sure it has the “real” hardware and software but the technology is expensive to keep up to date. Then with that technology, the instructor has to configure each machine to replicate the learning environment and then evaluate the students performance. This is time consuming and laborious. This has to be done for each task. Quite a challenge, but there are viable alternatives.

I think we should take a page out of the NASA program training book. Before they trusted the astronauts with multi-million dollar space crafts they put them through hundreds if not thousands of hours of training in simulators. These simulators were designed to recreate, down to the tiniest detail, what it would be like in the real space craft. Their simulators allowed the astronauts to be presented with every imaginable situation that they might encounter in space.

The use of this type of simulator has been deployed further into the aerospace industry, military and recently into the technical area. With the advent of the internet and affordable simulations, IT students are now beginning to reap the benefits of hands-on learning, in wide variety situations from anywhere they have an internet connection.

The future is here today. Simulators are used in distance learning and traditional classroom alike. They not only provide hands-on learning but decrease the cost of maintaining computer labs. When you add the benefit of tracking progress and instant evaluation and feedback you have a learning situation that is superior to the “real thing.”
However it is accomplished, there are some lessons that only experience can teach. If you don’t believe me just ask my son.

Simulations anyone?

Although I have been in the technology industry for over a decade even us old dogs can learn new tricks. I think that is the point. Always pushing the envelope, challenging our assumptions and trying new things. So this is my first official post. In the near future I will be posting my thoughts on simulations in educational settings and I welcome your input.