Cadet online testing

[Senior]

I see from the 2009 Winter Board that they are considering on line testing
for cadets.  Can someone give a brief description how that would work.
It sounds like an interesting idea.

[Eclipse]

http://captalk.net/index.php?topic=6719.msg132727#msg132727

On the "test integrity" issue, the proposal has included several safeguards.

First, each cadet will be required to complete an Honor Statement as follows:

I certify that I am Cadet (Name). I will complete the following Cadet Achievement Test on my own, without help from another person. My only resource for this open-book test will be my cadet textbooks and class notes. [By clicking below,] I reaffirm my commitment to our Core Value of Integrity.

Second, the on-line version of the test draws at random from a bank of questions that have been written for the achievment.  Even a cadet who has taken the test before would see a substantially different set of questions.

Third, the questions will be designed to test for comprehension rather than  memorization to minimize the Google effect.

Fourth (and most importantly), as others have pointed out, the comprehensive milestone tests are "closed-book" and will still be locally administered and proctored.  IOW, even if the other integrity safequards fail to prevent cheating, then the cadet will hit a roadblock at the end of their current phase and will not be able to promote.

The NHQ crew has put a lot of time and effort into crafting this proposal. It was been published on the NHQ website (in the CP Proving Grounds) for a year and discussed widely with CP leaders and the various CACs at numerous conferences.  (As well as here on CT and CadetStuff.)  As a direct result of input from the field, we added additional integrity safeguards into the final proposal.

Please let your wing commander know about any of your concerns so we can address them at the NB.

Ned Lee

[DC]

Here is the White Paper describing how it will work.

http://members.gocivilairpatrol.com/media/cms/ONLINE_TESTING_PDF_6EC9563B4B11D.pdf

[Flying Pig]

 Wow....what a concept.  Tests online.  Cadets log in, test...automatically tracked, dates, records, no stacks of paper to shred.  No more of cadets taking multiple tests at once, etc.  I knew a Sq. that allowed  cadets to complete all of their exams up the the Mitchell in about 3 months of joining.  Then, the cadet just waited the time in grade and passed the PT test each time and pinned on each rank as the time in grade was met. No more of that nonsense.  Basically every meeting you would take a test.  This was about 17 years ago.....so dont worry.  They are all gone.   But Im sure it still goes on.

[Eclipse]

Wow....what a concept.  Tests online.  Cadets log in, test...automatically tracked, dates, records, no stacks of paper to shred.  No more of cadets taking multiple tests at once, etc. 

No way to tell if a cadet actually took the test themselves - all tests are now an open book situation, Google will insure no one ever fails again, and there will be no such thing as test control - within a month all the questions and answers will be available via torrents or similar means.  Before you start discussing "security" - there is no way to plug the analog whole of anything displayed on a computer screen.

One thing that remains to be seen is whether any of the proctor controls we suggested will be implemented (dual logins, etc.)

A discussion we had the other night also brought up the suggestion that while online testing may be an option, allowing the cadets to take tests at home without a proctor may not be a requirement.  If that turns out to be the case, look to seeing a lot of unit SOP's that require cadets take their online tests either at the unit during meetings or at the least with an approved unit proctor.

As with WBP, this is not a universally popular idea, and it remains to be seen if NHQ will be as successful in implementation with online testing as they were with the WBP (which is/was a success).

[DC]

Time will tell. Supposedly the questions will be written to require comprehension vs. memorization, relatively short time limits for the tests, and a very large test question bank to try to combat cheating.

I think that if National implements this properly, and with proper safeguards in place, it will work well enough.

[wuzafuzz]

Wow....what a concept.  Tests online.  Cadets log in, test...automatically tracked, dates, records, no stacks of paper to shred.  No more of cadets taking multiple tests at once, etc. 

No way to tell if a cadet actually took the test themselves - all tests are now an open book situation, Google will insure no one ever fails again, and there will be no such thing as test control - within a month all the questions and answers will be available via torrents or similar means. 

Like most of the NIMS and ICS courses, and the existing online CAP courses?  Are there cheat sheets for those yet?

If the tests are sufficiently difficult I don't have many concerns with the concept.  I've taken some wickedly difficult online tests at the college level.

Like the white paper says, the tests are only one part of promotion eligibility.  

Now if we could only put more senior program tests online.

[PHall]

Wow....what a concept.  Tests online.  Cadets log in, test...automatically tracked, dates, records, no stacks of paper to shred.  No more of cadets taking multiple tests at once, etc. 

No way to tell if a cadet actually took the test themselves - all tests are now an open book situation, Google will insure no one ever fails again, and there will be no such thing as test control - within a month all the questions and answers will be available via torrents or similar means.  Before you start discussing "security" - there is no way to plug the analog whole of anything displayed on a computer screen.

One thing that remains to be seen is whether any of the proctor controls we suggested will be implemented (dual logins, etc.)

A discussion we had the other night also brought up the suggestion that while online testing may be an option, allowing the cadets to take tests at home without a proctor may not be a requirement.  If that turns out to be the case, look to seeing a lot of unit SOP's that require cadets take their online tests either at the unit during meetings or at the least with an approved unit proctor.

As with WBP, this is not a universally popular idea, and it remains to be seen if NHQ will be as successful in implementation with online testing as they were with the WBP (which is/was a success).

You know, the FAA uses the same "Test Question Bank" system. i.e. there are 30 questions on the test, but the bank holds 600 questions. You never get the same test twice.
The Air Force uses the same system too. They even gave us a handout with all of the questions in the bank too.

This system seems to work for them, so why not give it a chance?

[Eclipse]

Now if we could only put more senior program tests online.

Other than ECI-13, what's not online that is being taken regularly by most seniors?  And ECI-13 is being redone and I believe they've said it'll be an online course.

Like most of the NIMS and ICS courses, and the existing online CAP courses?  Are there cheat sheets for those yet?

What you guys are missing here in comparing NIMS, ICS, GES, Yeager, just about anything we do online today, is that these are already, by design, open-book tests, so having the curriculum, or even Google open on another tab isn't really "cheating", per se.

The current cadet test are not, by any means, an "open book" situation.  They are under such tight test control that they require a special officer to manage and attest to their security, and when one gets out in the wild, an all-stop is put on the test until it is redone.

All that goes out the window when we allow cadets to take the test on their own time, at home, in private.  It significantly changes the way cadets will prepare, assuming they do, and what we will now see is cadets bunching up below the achievements that require in-face testing, because the whole dynamic will be different.  Instead of those milestones being just "another test" they will become the "hard tests", because they will require actual study and preparation instead of the F3 key.

As said, it remains to be seen how this is implemented, what options commanders have for tighter controls, and how secure the tests themselves are.  I am a huge proponent of digitizing everything, but this significantly changes the entire testing dynamic, and I am concerned that the convenience factor, coupled with trying to wheedle commanders not doing testing right today is being attained through water-down the program, something we can ill afford.

With deference to Ned and others whom I'm sure are working with the bext intentions, there are practical and cost limits to how secure these kinds of tests can be, especially if we aren't going to use PME or professional testing centers or dual-login proctor situations.

[MIKE]

Someone had mentioned before that one good thing about it is that it is one less thing you have to schedule each month or make cadets show up early for... Have done both... didn't much care for showing up early to mentor... particularly when the cadet forgets to show up early.

It's also one bad thing... 'cause now you have to fill that hour with something other than Drill/Testing.  

When can I join the CAP Talk eSquadron, where I can be be CDC or CC, hold seven other jobs and progress to Lt Col without leaving my swivel chair and 22" flat panel?

[Eclipse]

When can I join the CAP Talk eSquadron, where I can be be CDC or CC, hold seven other jobs and progress to Lt Col without leaving my swivel chair and 22" flat panel?

Yeah, no kidding - CAP is not a correspondence school.

[RiverAux]

Other than ECI-13, what's not online that is being taken regularly by most seniors?  

SIGNIFICANT portions of our ES training program could and should be put online.  The basic knowledge elements of the SQTRs could be made into online quizes.  

[Eclipse]

Other than ECI-13, what's not online that is being taken regularly by most seniors?  

SIGNIFICANT portions of our ES training program could and should be put online.  The basic knowledge elements of the SQTRs could be made into online quizes.  

Significant portions already are in the ICS training.  In-face tasking is a critical safety valve as well as an avenue for hands-on training.

The last thing we need is people reporting for missions with "ticket-punched" online SQTRs and no idea what they are supposed to be doing.

[RiverAux]

For tasks that have to be demonstrated you are correct, but for a lot of SQTRs at least 25% of the tasks could be done as quizes.  If they had these available as online quizes you could actually spend your valuable face to face time doing the real work instead of regurgitating lists of info.

[MIKE]

Can we not turn this into another [darn] ES thread please!

[wuzafuzz]

Now if we could only put more senior program tests online.

Other than ECI-13, what's not online that is being taken regularly by most seniors?  And ECI-13 is being redone and I believe they've said it'll be an online course.

Like most of the NIMS and ICS courses, and the existing online CAP courses?  Are there cheat sheets for those yet?

What you guys are missing here in comparing NIMS, ICS, GES, Yeager, just about anything we do online today, is that these are already, by design, open-book tests, so having the curriculum, or even Google open on another tab isn't really "cheating", per se.

The current cadet test are not, by any means, an "open book" situation.  They are under such tight test control that they require a special officer to manage and attest to their security, and when one gets out in the wild, an all-stop is put on the test until it is redone.

That's precisely why I suggested redesigning the tests to be appropriate for the online environment.  That's not missing something.  It's a suggestion for a paradigm shift.  Moving more administrative tasks online, including some tests, increases consistency and leaves meeting time available for hands-on activities, training, and evaluations.   Filling meetings with administrivia is boring and likely reduces attendance and member engagement.

[Always Ready]

Wow....what a concept.  Tests online.  Cadets log in, test...automatically tracked, dates, records, no stacks of paper to shred.  No more of cadets taking multiple tests at once, etc. 

No way to tell if a cadet actually took the test themselves - all tests are now an open book situation, Google will insure no one ever fails again, and there will be no such thing as test control - within a month all the questions and answers will be available via torrents or similar means.  Before you start discussing "security" - there is no way to plug the analog whole of anything displayed on a computer screen.

One thing that remains to be seen is whether any of the proctor controls we suggested will be implemented (dual logins, etc.)

A discussion we had the other night also brought up the suggestion that while online testing may be an option, allowing the cadets to take tests at home without a proctor may not be a requirement.  If that turns out to be the case, look to seeing a lot of unit SOP's that require cadets take their online tests either at the unit during meetings or at the least with an approved unit proctor.

As with WBP, this is not a universally popular idea, and it remains to be seen if NHQ will be as successful in implementation with online testing as they were with the WBP (which is/was a success).

I take college courses online. Most, if not all, of our exams are conducted online both open and closed book. How do you take a closed book test online? Well I'm not completely sure on the technical stuff. But I can tell you that during these closed book exams, if another window, tab, screen, or program is running in the background, the test will automatically end. They are also timed similar to most standardized testing. At the end of the time, you can no longer answer any questions. Depending on how the professor configures it, if you 'cheat' the test is usually voided and you must take an alternative test in its place. I've had one professor who threatened to give an automatic 'F' if this occurred and the only other option was a proctored exam.

That being said, I can tell you the tests are generally not easy. Most of the questions are focused on your comprehension of the material.  Also, if National creates these questions right, Google is not your friend in most cases. Most of my open book exams are killers. Google won't even get close to the answers. There are usually a few essay type questions thrown in. Because this is an online university, they are very stringent on plagiarism (more so than any other college I've attended). If you write a paragraph and none of your work is cited (it doesn't matter if it is all original), you will get questioned for plagiarism. I've been questioned twice, both of which I was answering the question with stories from my youth. Even though the professors thought it was clear that it was clear that it was my own work, I got questioned. I ended up having to add a citation for both questions. I think I cited The Weather Channel for the weather those days which was completely irrelevant to the topic . I ended up getting full credit in both instances . Now I just add a citation to everything just to cover my rear. That just shows you the other side of the spectrum.

Works cited:
The Story of My Life, 1989-2009

[Eclipse]

^ I don't think comparing college-level exams with most CAP achievement tests is really a fair comparison - they are mostly fairly common knowledge questions (in the CAP vernacular), and taken on what is likely a significantly more regular basis nationally, vs. a non-standardized test from a single college.

I promise you, Google will be your friend for the kinds of questions we're talking about here, not to mention the fact that the curriculum will be online, too - so the open "book" is already, or will be, a .pdf.  Yeager's birthday, CAP's birth, and the command structure, etc., aren't really open to "interpretation", especially in a multiple-guess situation.

Essy questions aren't going to fly because they defeat the purpose in making the unit CC's, etc., hands-off the grading.

As to securing the tests, yes, there are any number of interesting ways to secure the testing environment, however beyond the typical Javascript code security, common from CAP and FEMA, for example (and all but useless), I don't expect to see any of it.

[Always Ready]

^I believe that what National proposes and college exams is like comparing small apples to large apples. I'm just giving another side to the story. I always thought online tests were easy...until this school. National can make this more difficult for the cadets. All they have to do is create more questions like: 'What do you think about....?' and 'How does this relate to...?'. These type of questions can still multiple choice and measure their knowledge level by focusing on comprehension instead of memorization. "When you comprehend something you learned it, when you memorize something you'll forget it." -My former Econ professor

You'd be surprised how many college level exams are standardized. When I transferred to my current school, some of my classes didn't transfer so I had to retake them. Two of the classes used the same books and the same exams as the previous class I had taken. Thankfully, I kept my notes 

The point is that the current cadet tests are relatively easy. I've taken them fairly recently (two years ago). It took two, maybe three, hours of studying per test and that was it. I passed most of my tests first time. I know the main concern with this is cheating, but there is/was a website or two that had the test questions on them to study off of back in my day. I'm not sure it still exists, but I'm sure one like it exists. What's the difference? None. To curb cheating we need to focus on the cadets' character development instead of anti-cheating mechanisms that someone will work around....that goes for everything, no matter the medium.

[caprr275]

I am glad the online testing passed as far as I can tell with no objections.  There will be a big test bank so cadets will not get the same test every time and there is a honesty statement at the beginning of the test.  IF a cadet really wants to cheat they will just be hurting themselves because milestones will still be the old pencil and paper.

Just a side note NHQ is getting ready to publish a question pool for NCC panel and written quizzes.