Reliability and validity within assessment, as well as all parts of education, is necessary in order to make the results of that educational work appeal to peers and those requesting the work done. Without reliability or validity in the work, the results of that work become useless. But in order to understand that impact of both aspects independently, it is necessary to understand the terms clearly.

Reliability
Reliability has been defined differently depending on the experts that have been consulted. Baer defined reliability as “the degree to which two observers viewing the same behavior at the same time agree on its occurrence and nonoccurrence” (Gresham, 2003). This means that in order to truly have a reliable result, it would need to be recognized by at least more than one observer of the same result at the same time. As a definition, this is perhaps the most widely accepted of applied behavior analysis, and remains so to this day (Gresham, 2003).

Johnston and Pennypacker defined the results very differently, as “the consistency with which measure of behavior yield the same results” (Gresham, 2003). This applies to the consistency of results based on the same behavior, and is perhaps more applicable for individual experiments and observations. This differs, because the first definition by Baer doesn’t take into consideration individual bias of the observer during the behavior observation. Therefore, in an educational environment, one teacher can see the results of a student’s behavior as being completely different than another teacher’s observation, even though it is the same behavior being observed. This definition provides for the actual results of the behavior, and not the interpretation of the behavior that produces that result.

Validity
Validity does not rely on hypothetical constructs for description, but on actual results (Gresham, 2003). According to Johnston and Pennypacker “if the behavior under study is directly measured, no question about validity exists” (Gresham, 2003). This leaves only indirectly measured results to be given validity, which is therefore validated by directly measured results. Of course, this view assumes that direct methods of measurement do not contain large amounts of error (Gresham, 2003). This of course does present a problem for our definition of validity, when the question of error is brought before us.

In answer to that problem, many behavior analysts consider the concept of accuracy to be much more important than validity (Gresham, 2003). While validity measures the results, accuracy measures the degree in which the results reflect the true state that the analysis is meant to measure (Gresham, 2003). This of course calls into consideration the content of the analysis that is being measured, and its reflection on the true state that is being measured.

In answer to that, it seems that content validity has become more relevant than any other types of validity (Gresham, 2003). Linehan has argued assessment procedures need to be focused on actual representative sampling before validity can be given to the results (Gresham, 2003). Others feel that multiple sources of results, as well as measures, provide validity to the overall assessment, as it can give a more complete picture of the analysis to be evaluated (Henderson-Montero et al., 2003). Both provide a more comprehensive understanding of the results through valid content, allowing for acceptable statistical error.

Reliability and Validity in Application
Now that we have a general feel for what we want in our assessments, how can we apply this knowledge to an actual assessment situation? Lane and Ziviani (2003) managed to address these particular points in their assessments of children’s mouse proficiency.

The first step was to determine what exactly the results were that they were looking for. This was particularly difficult, since in many areas their assessment was breaking new ground in this field. What they were looking for were measurable results that could be gathered through computer interaction using only the mouse. In order to provide variability in the testing scenario, they tested their subjects one week apart for each case. They then pooled the results that were measured for a more accurate assessment, as opposed to assessing each group individually. They then used standard measurement procedures and algorithms to allow for a standard that their peers could relate to when the findings were published.

In order to assess the reliability of the assessment, Lane and Ziviani conducted additional studies other than the initial one, from various pools. This provided for more accurate measurements of the results, and provides reliability based on Johnston and Pennypackers definition of reliability with regards to results (Gresham, 2003). They also tested in environments that were mutually available, convenient, and comfortable for those being valuated. This allows for a more accurate measurement.

In order to provide validity to the results, two aspects were considered: construct-related validity and criterion-related validity. Both focus on Linehan’s definition of representative sampling as a source of content validity (Gresham, 2003), and are used to validate their findings based on how valid the actual measurements would be.

With construct-related validity, Lane and Ziviani focused on the ability to complete aiming, tracking, drawing, and target selection tasks with maximum speed and efficiency (Lane, Ziviani, 2003). This provides a clear idea as to what is being assessed, and how the results should be measured. Therefore, the actual measurements should not be effected by content that contains unpredictable errors (Gresham, 2003). Criterion-based validity focused on the specifically on the predictability of the results based on a coefficient of 0.5, which is pretty standard for similar assessments (Lane, Ziviani, 2003). This also provides validity, as the criteria are made valid with the expected results reaching the predictable mean in the statistical review.

Conclusion
And so we see that once a definition of reliability and validity are reached, and our understanding of those terms are firmly set in the assessment, the assessment itself can provide valid results that are reliable within statistical means. The actual definitions that you select determine the direction of your assessment, as well as the general validity and reliability as seen by your peers.

Resources
Lane, Alison, and Ziviani, Jenny, Assessing Children’s Competence in Computer Interactions: Preliminary Reliability and Validity of the Test of Mouse Proficiency, OTJR, Winter 2003. Vol. 23, Iss. 1; pg. 18

Gresham, Frank M., Establishing the Technical Adequacy of Functional Behavioral Assessment: Conceptual and Measurement Challenges, Behavioral Disorders, Tempe: May 2003 Vol. 28, Iss. 3; pg. 282

Henderson-Montero, Dianne, Julian, Marc W., Yen, Wendy M. Multiple Measures: Alternative Design and Analysis Models, Educational Measurement, Issues, and Practice Washington, Summer 2003 Vol. 22, Iss. 2; pg. 7

With every assessment that is given, there needs to be a specific set of goals behind that assessment to make the results become meaningful and useful. Without those standards and recognized criteria, an assessment cannot be an accurate measurement of the abilities or skills possessed by the learner. While many instructors and students will spend most of their time focusing on the results, we as potential instructors would need to recognize the methods used to develop the standards by which results are measured.

The need of Standards
As previously stated, standards are required to make an accurate picture of the skill level of a learner through the results of an assessment. The actual assessment method is not necessarily important, as long as it can accurately show the performance of an individual with regards to a specific skill set.

The first order would be to define the standards that are to be identified by the assessment, and how performance indicators for these standards should be adapted to the target student population (Browder, 2003). The criteria being set need to be standard across the board, so that accurate results can be measured. Once set, various assessment methods can be applied to measure those particular performance requirements.

The example given by Browder would be methods of assessing the performance and skill level of disabled students. In this situation, passing out milk to other classmates in the morning can address standards in listening, speaking, number operations, and problem solving (Browder, 2003). This same can be said with a learning team within the University of Phoenix. Team behavior can be used to assess organization skills, team-building abilities, leadership qualities, and teamwork skills. The assessments in both examples are not standard written assessments, but yet have the same qualitative properties, if the criteria being measured are taken in context of the demonstrated skills in each activity.

With the understanding that standards for several criteria are being set and need to be reached with each assessment, it becomes necessary to define the criteria to the student. Otherwise the student will tend to become unaware of the standards they are required to reach, and thereby left to imagine their own requirements, right or wrong (Hinett, 1997). This can lead to misunderstandings that inhibit the student’s ability to perform under ideal conditions for proper assessment.

This is where rubrics become important to students. They define the standards that are required, and outline what criteria are assessed in the learning environment. This initial rubric can take the form of a complete course outline with grade expectations and assessment points that will be looked at, or it can be a simple set of instructions and rules to follow during the assessment. At each level certain standards are required identified and presented to the learners for clarification and guidance to what is expected of them.

Developing Standards
Now that we understand why standards are important and how they are implemented in an evaluation environment, it is now necessary to understand how such standards are developed. Black and Duhon (2003) identify a clear way to develop standard requirements and grant validity to assessment findings. They identify valid criteria as results to the extent which scores on the test are correlated with other variables that the instructing institution expects for associated test performance (Black, 2003). This method is generally developed as the results of previous experience of the educational institution with similar student reactions. Once the school has identified the standard they wish to set through their experience, they can then compare their findings with those of other schools with similar demographics. This presents an industry standard that is expected for all schools to reach. But suppose a new concept, technology, skill, or process is developed? How is one to identify a correct method of measurement that has the potential of standardizing criteria being assessed?

The method can easily be identified by first identifying the criteria of the assessment itself. Is there a skill that should be identified, and if so, how can it be measured? Once that concept is identified, similar methods can be used as control comparisons. The example that Black and Duhon use relates to the performance of business majors on the new Educational Service’s Test (ETS) Major Field Test in Business. The goal was to see how accurate common methods of assessing achievement would measure up to the ETS.
The criteria
The students being tested were being organized with the following criteria in mind:
1. GPA (both in Business specific courses, and overall)
2. ACT/SAT scores (Both accumulative and English/Math only)
3. Age difference
4. Gender
5. Major emphasis

The results
Once the material was gathered the following results were gathered:
1. For each Business GPA point increase, the average ETS score was 7.49 points higher.
2. For each Accumulative ACT score point increase, an average of 1.51 points increase was found for the ETS.
3. For every year increase in age, an increase of 0.71 points was average on the ETS.
4. As for Gender, males tended to score 3.79 points over women.
5. In respect to a major emphasis, those majoring in Management tended to score 3.57 points lower than all other majors, once all other criteria had been controlled.

Once the statistics have been gathered, it is now important to understand how they are significant. If there is a high correlation (+/- 0.70) between any pair of independent individuals, it indicates a statistical mean, with it’s corresponding distortions (Black, 2003). Once a mean or “collinearity” has been reached, it represents a valid, measurement that can be used as a standard towards additional results. It identifies the statistical predictions of where students will generally score based on previous experience, skill exposure, and educational background. Once that standard can be reliably measured, assessments become equally reliable.

Conclusion
So, through identifying statistical trends in scoring results, as well as the criteria that should be measured, evaluations and assessments can be used as a reliable tool for instructors to see what requirements need to be met in order to produce the best results in education. Students are also able to realize those requirements by following defining tools such as rubrics that are presented to guide them through their educational aims. It keeps them mindful of the standards required by the educational institution, and thereby keeps them focused on the skills that the course is supposed to teach them.

Resources
Hinett, Karen Review Symposium: Enhancing Learning through Self-assessment, Assessment in Education, Vol. 4, No. 2, 1997, p. 321

Browder, Diane; Spooner, Fred; Algozzine, Robert; Ahlgrim-Delzell, Lynn; Flowers, Claudia; Karvonen, Meagan What We Know And Need to Know About Alternative Assessment, Exceptional Children, Fall 2003, Vol. 70, Iss. 1, p. 45

Black, H. Tyrone; Duhon, David L. Evaluating and Improving Student Achievement in Business Programs: The Effective Use of Standardized Assessment Tests, Journal of Education for Business, Washington, Nov/Dec 2003. Vol. 79, Iss. 2, pg. 90

Your adrenaline is moving out of your system now, the class is over. How did you do? Most trainers can “sense” a general feel in the presentation and participation from the learners, but what were they really thinking? Did they get it? Can they do their job better now than they could before they started the training?

If you are a trainer, chances are you were hired for a specific job: making sure learners work better/faster/smarter. As with any other job, chances are your boss will want a full accounting of your performance in this area. How can you prove that you have accomplished your goal in a way that’s measureable, and easy to understand? You do this through evaluation.

There are a number of ways you can evaluate the success of your training, depending on how much time you have to prove your worth to the company. There are the direct, timely methods, and there are indirect methods as well. Let’s take a look at them both, and see which is best for you.

Direct Evaluation Methods
These are commonly called “Tests”, “Assessments”, and “Surveys”. Basically, you check to see how well the learner had performed at the beginning of the course, give them quick tests in the middle of the course to see if they understand each of the modules you are presenting, and then have a final exam that tests overall comprehension. This is probably the most traditional method of evaluation, and everyone is pretty much familar with it. But it only looks at a small snap-shot of the learner’s abilities. You don’t know if the targeted skills are going to be applied.

A real bonus from this method, particularly from the survey, is that you can get a feel for your development and implementation of the course. How did it appeal to your learners? How are you doing as a presenter? There are a number of things that you can learn that will add to your ADDIE development through this method, outside of just whether or not the analysis was correct.

Indirect Evaluation Methods
Indirect evaluation methods would include monitoring employee performance over a long period of time, focus on overall numbers and how they relate to the skill that needed to be taught. Is there an improvement? Did it warrant the devotion of resources?

For those who are familar with any type of research, this should be nothing new. Researching the results of a change is part of what analysts do, and makes them so valuable to companies (mostly because it’s so boring no one else wants to do it ^_^). But what do you analyze? Focus on the results as compared to your initial needs analysis. Did the numbers you focused on for your initial analysis change? Did they change for the better? Where there other factors involved that were not initially recognized?

For those trainers that are caught in the political arena within your company and were forced to create a training program to compensate for non-skill related issues, this is a perfect time to emphasize that while the skill became better known, the outcome did not improve because of the x and y factors. If you provide the information in a scientific way, showing that even though the training was a success the solution failed to be realized, the management will often concede, or let you go, which would also be an acceptable alternative. Who wants to be blamed for someone else’s incompetence?

Seriously though, it’s a good method to see how effective your training was, your analysis was, and how well each of the learners assimilated the information. You learn how well things are going, how you can improve your teaching style, and therefore increase your effectiveness as an instructor. A success here will validate your work, give you a great promotion, raise, and a chance to win a free 2 week vacation in the Bahamas! ^_^

When to Use Your Evaluation Style
Neither evaluation method is perfect on it’s own, so combining both is essential for a full view into how well you are doing. Use a quick assessment at the beginning of the course to find out where your learners are (if that is in question). Once you know, have them keep their scores for future comparisons and self-evaluation. Also have an after-class evaluation that is done anonymously away from the classroom environment. This way the instructor doesn’t have a presence to influence the outcome of the evaluation.

Then, send two more evaluations, once after 3 weeks, and one after 2 months. This way you can find out how well the content is remembered, and what the percentage of recall is for the learners. This is good long term data to be gathering. And finally, spend some time doing indirect evaluations by checking performance numbers. Of course, this assumes that you have access to the information. If you don’t, you may want to provide a quick spreadsheet to the company that contracted your services so that they can provide the final data to you. They can leave out any information that may be proprietary and still provide enough information to let you know if you have been successful in your endeavors.

So, that finishes this segment of the ADDIE program. I may post some additional information on the subject, but for now, I wish all of you good luck in your training development!

Now that the main body of the work has been accomplished, we need to identify the objectives. This outlines specific goals for the training session. What do we need to specifically accomplish? Well, let’s find out.

The Learning Objective
The learning objective outlines the problem, the results, the environment and conditions for success, and the resources available for success. What’s more, this is all in one sentence, so the use of commas are encouraged. It’s through these objectives that your overall success can be measured by, and therefore how to determine if the training was worthwhile. This doesn’t include evaluation methods, though they are closely related to this process. We will cover that in a later section.

So it is important that we identify what exactly identifies success. This is defined by the Input (problem) and the Output (results). The Input presents the issues that the training module is going to address, and is generally linked to the inability to perform the task at hand.

The Output outlines the ability to perform the task within the set measureable guidelines required by the training. These can be satisfaction, performance, productivity, or safety guidelines. Just anything that measures success for the learner.

Following that, the Aids (resources) and Conditions need to be recognized. Aids identify the experience or needs that the learner requires to perform the task. For example, an aid would be a diagram showing how to insert the key into the ignition. In other words, it can be reference material, access to support staff, and anything else that can assist with the performance of the task.

The Conditions outline the limiting factors within the performance of the job. If an Internet connection is required, and may not be 100% reliable, that needs to be taken into account. If access to the key locker is necessary, that needs to be taken into account. Basically, all factors not related to knowledge and skill are outlined here in order to set a reasonable expectation. If someone doesn’t have the correct tools, you can expect them to perform the task.

Once the sections have been outlined (I do this in the Task analysis document, directly above the inserted table), the objective can be created. So let’s outline the sections!

The Input and Output
First we need to define what the problem is for this task going into training, and what we expect to get out of it. For instance, if we start the training with the idea that our taxi driver doesn’t know how to start the car, we would assume that after the end of this module the driver can now start the car while meeting all performance guildelines. That is an example of input and output. Here is how you can write it:

INPUT: The driver is unable to use the key to start the ignition.
OUTPUT: The driver is now able to start the ignition using the automobile’s key to the extent that customer satisfaction and proper use guidelines have been met.

So what do we have here? We have the beginning and the end of the learning objective! That’s right, we can actually copy and paste this into the learning objective, which saves a lot of typing. Finally! A short cut!

Aids and Conditions
I format my aids and conditions in a similar manner. While continuing with the example:

AIDS: Access to automobile manual, keys to the vehicle, and support staff.
CONDITIONS: Assuming the vehicle is in good maintenance, the driver is already licensed, and is familiar with the vehicle in question.

Here we have the center portion of the learning objective. Again, we can copy and paste this directly into the objective, which will save us a lot of heartache (and sore hand joints) in the long run. But how does it all go together?

Putting It All Together
It’s time to look at the whole application of this work. Here is how it should look in your Task analysis document:

Task A: Starting The Vehicle
INPUT: The driver is unable to use the key to start the ignition.
OUTPUT: The driver is now able to start the ignition using the automobile’s key to the extent that customer satisfaction and proper use guidelines have been met.
AIDS: Access to automobile manual, keys to the vehicle, and support staff.
CONDITIONS: Assuming the vehicle is in good maintenance, the driver is already licensed, and is familiar with the vehicle in question.

So we have our task, we have the problem stated, the expectations, the resources, and the conditions that are outlined. So let’s put it togeter into the Learning Objective!

The learning objective would be formatted this way:

Given [input] and [conditions] with [aids], the learner will be able to [output].

For our example, it would look like this:
Given the driver is unable to use the key to start the ignition and assuming the vehicle is in good maintenance, the driver is already licensed, and is familiar with the vehicle in question with access to automobile manual, keys to the vehicle, and support staff, the learner will be able to start the ignition using the automobile’s key to the extent that customer satisfaction and proper use guidelines have been met.

Yes, it’s one long sentence and it’s probably not formatted correct grammatically, but it outlines each of the important steps in the training process. We finally have an objective that is specific enough to keep us on topic while developing and designing the material. But before we can get that done, we have one final step: The Assessment Methods.

Assessment or Testing
Yes, you need to test your learners in some manner to be sure they are learning what you are trying to teach them. No matter how well you think you are doing, chances are you have lost someone that is too afraid to speak up. And if you have lost one person, you probably have a few others that are just barely keeping up. Assessments are necessary in determining their success, and whether or not you are teaching properly. If you lose a lot of students, it’s time to rethink your approach.

In order to evaluate someone’s abilities in the most efficient manner, the best thing is to create an environment as close to the actual performing environment as possible. No matter what other instructors (or even professors) may think, Multiple Choice doesn’t do this. The real evaluation method is in practice.

That being said, if it’s not possible, or economically feasible, to do so then alternative assessment methods can be used. After all, all certification classes (with a few exceptions) are multiple choice tests. This posting isn’t meant to be a discussion on the virtues of each evaluation method, so you need to decide what is best for you.

When it comes to your Assessment methods, I would have at least three options selected: one for Tactile learners (hands on), one for Auditory learners (written exams), and one for Visual learners (presentations). This gives you a general pool to pull from while designing the course, and will give you a lot of flexibility on future implementations. For instance, when I created training curriculum for a certain company I worked for previously, I outlined assessment methods for both online and in class training. I envisioned a number of alternatives the instructor could implement, and therefore created possibliities for future development.

Putting it All Together
The Learning Objectives and testing methods I place on the same document, being separated from the Task analysis. This document is then used in conjunction with the task analysis to create the learning materials, and design methods for each training module. As a quick tip, if you notice that a lot of your material is the same for each task, use your copy and paste option. It’s an ideal solution for sore fingers.

Finally, we have finished Analysis! The next session in the ADDIE series will be Development. Fortunately, it’s not nearly as long as the analysis section, and a lot more fun!

I am, by nature, an analyst. I love to analyze everything from complex learning strategies to the movie I’m sitting through. Yes, I can safely say that analysis is a big part of my life (to the chagrin of my wife). And as such, you would think that instructional analysis would be right up my street. Well, you would be right, but only when I take it in short bursts.

Instructional analysis comprises a strong 75% of my overall design process, because of the need to get every detail worked out. The details are often so minute that they can sometimes be missed through initial surveys. So I developed my own system that adopts many other systems I have been exposed to, but works best for me. That being said, please don’t think this is the one size fits all scenario. The process itself may not work specifically for your situation, but the basic elements should apply everywhere.

What is the Problem?
The first step in any analysis process is to determine the problem. A problem would basically mean a need is not being met. In the corporate world, this generally means that a job is not being performed to the standard that is expected.

This doesn’t mean that a job isn’t being done in the way that is expected, but that the outcome of the job produces results that are not as expected. I want to be very clear on that point, as innovation can be throttled if a single process is the only process allowed.

Is “throttled” too strong of a word? Good! I want to impress in this posting that the job of training is not to produce conformity, but to instill a level of competence that allows the learner to not only do what is required, but find ways to do it more efficently. This, utlimately, is what makes a good employee: Someone that is able to innovate within their realm. It also makes for really good resume fodder.

Also important to note, I have found that many managers feel that training is the answer to everything. It’s not. As I’ve mentioned before, you can’t expect more knowledge to improve on poor management decisions. At best it insults the employees, and at worst it exposes the poor management style for what it is, ruin morale, and shorten the employment span of the employees.

Where Does the Problem Exist?
So, having established what it is the trainer is looking for, it’s necessary for the trainer to focus on causes of the problem within the context that it happens. Often this means going right to the source: sit with the employees that are expected to benefit from this training. Does everyone experience the same problem? What do they know? What don’t they know? What are they allowed to do? What are they not allowed to do? These are all really good questions to get you started.

The next step is to check with those that do not experience the problem, which generally are more senior members within the group. What makes them different? This is the key that will answer the problem riddle, and determine whether or not training is necessary. Are the senior employees more empowered? Do they have access to resources that others do not? Do they have more knowledge than those that continually run into the problem? Do they have any insight into what could be the problem? These questions should clarify where a problem can exist, or at what point the process fails. If it doesn’t, continue up the chain until someone gives an idea of the expectation and you have enough information to identify the problem.

Houston, We Have the Problem. Now What?
Once you have identified the problem, it’s time to identify the solution. What is the only problem that applies to training? “There is a lack of knowledge or skill that needs to be addressed.” That’s it. Not having the tools to work with doesn’t get resolved through training, it gets resolved through new tools. Poor management decisions doesn’t get resolved through employee training, it requires a better manager. Unclear expectations do not get more clear with training, they need to be clearly communicated by management to the employees.

While working for a previous internet company, I found a major problem. Employees didn’t know what critical updates had been rolled to the site, and therefore couldn’t support the users that had trouble with these new updates. What did management try to do? Give them more training. Did the employees need it? No! They knew how to resolve the issues, but they didn’t know what changes were made, and hence could not prepare properly. This was a classic example of a communication failure within the company. Training cannot resolve this issue.

Also, with the same company, I found a new project that was being rolled to the site. This project was complex, and required a complete rethink of the entire process to utilize on the site. Does this require more communication from the developers? No, because I already had all the information, it just needed to be distributed to the rest of the company to teach the employees the new skill. This is an excellent example of what training is all about.

So determining the need itself can be a long and comprehensive process, but this is a necessary step in order to determine if training can actually resolve the issue. If not, you don’t have to invest any more time into developing for training, and more time and resources into resolving the problem on another level.

Stay tuned for tomorrow’s entry: Analysis Day 2: The Skill Assessment. Same Bat-time, same Bat-channel!