Why aren't the Milestones proctored on a computer?

Started by Майор Хаткевич, December 13, 2012, 08:40:40 PM

0 Members and 1 Guest are viewing this topic.

Майор Хаткевич

As I'm sitting here thinking about it, I'm wondering why we decided to keep them all on paper.

As an ATO I have to sit there and watch the cadets take the test anyway. Why couldn't NHQ make secured USB that plugs into any machine, pulls 50 Qs out of a question bank, and the cadet takes the test, gets his results, and we can even throw in a printable reference sheet for what they had missed!

Too much tech?

I was thinking making it online, with an ATO/TO required to sign the cadet into the system, but I'm aware that internet is still spotty for many units out there.

A thumbstick, secured in a similar fashion, but basically being an internal program on the drive, would avoid connection issues while still maintaining test security and modernizing the process.


krnlpanick

While I am all for modernizing CAP as much as humanly possible - I think that the cost of moving to computer administered testing in this case I think far outweighs the benefits of doing so; at least in the way that it has been suggested. A more cost efficient way to modernize the process would be to require a dual authentication on the existing LMS in eServices. First a cadet logs in to eServices, selects LMS, then selects the Milestone test that he would like to take. Next the SQ/CC or delegate must enter their CAPID and password into the system to allow the cadet to begin taking the test. This ensures that the test is still taken in the manner expected while leveraging existing technology available to us to minimize the implementation cost.

Just MHO
2nd Lt. Christopher A. Schmidt, CAP

Майор Хаткевич

Agreed, the issue again is internet availability at the unit.

I could fire up my phone and have someone take a test, but that's not an option for everyone.

krnlpanick

Quote from: usafaux2004 on December 13, 2012, 08:48:19 PM
Agreed, the issue again is internet availability at the unit.

I could fire up my phone and have someone take a test, but that's not an option for everyone.

That is one area of modernization I think should be enforced - I believe that all squadrons should have some type of internet access available to them. Even if that means getting a supplement from NHQ to offset the cost of doing so I think it is well worth the expense. Squadrons that cannot get internet due to financial reasons should have to option of acquiring discount 3g/4g internet through NHQ.

Again, just MHO.
2nd Lt. Christopher A. Schmidt, CAP

MSG Mac

That would actually be a good recommendation for Cadet Programs. I'm sure the IT people at National could design a program when theyfinish their current projects.
Michael P. McEleney
Lt Col CAP
MSG USA (Retired)
50 Year Member

MSG Mac

Quote from: krnlpanick on December 13, 2012, 09:01:46 PM
Quote from: usafaux2004 on December 13, 2012, 08:48:19 PM
Agreed, the issue again is internet availability at the unit.

I could fire up my phone and have someone take a test, but that's not an option for everyone.

That is one area of modernization I think should be enforced - I believe that all squadrons should have some type of internet access available to them. Even if that means getting a supplement from NHQ to offset the cost of doing so I think it is well worth the expense. Squadrons that cannot get internet due to financial reasons should have to option of acquiring discount 3g/4g internet through NHQ.

Again, just MHO.

National had an Internet supplement program that nstopped about a year ago, because no one was using it.
Michael P. McEleney
Lt Col CAP
MSG USA (Retired)
50 Year Member

JeffDG

Quote from: MSG Mac on December 13, 2012, 09:03:43 PM
That would actually be a good recommendation for Cadet Programs. I'm sure the IT people at National could design a program when theyfinish their current projects.
LOL

Майор Хаткевич

Quote from: MSG Mac on December 13, 2012, 09:03:43 PM
That would actually be a good recommendation for Cadet Programs. I'm sure the IT people at National could design a program when theyfinish their current projects.

In terms of doing it on eServices, they basically just need to add the tests and an extra layer of security.

Eclipse

Quote from: krnlpanick on December 13, 2012, 09:01:46 PMThat is one area of modernization I think should be enforced - I believe that all squadrons should have some type of internet access available to them. Even if that means getting a supplement from NHQ to offset the cost of doing so I think it is well worth the expense. Squadrons that cannot get internet due to financial reasons should have to option of acquiring discount 3g/4g internet through NHQ.

And that money would come from where?  Exactly?

Not to mention that Hooterville composite might not have any broadband options, nor a permanent place to run wires.

I'm all for modernization, and for most units it could be accommodated on some level or another, but we don't need more unfunded mandates, and NHQ's pockets are down to the lint right now just like everyone else.

With that said, some of the wireless broadband providers officer service to 501c(3)s for free or greatly reduced cost.  I also ran across
this company recently, that offers "free" service if you buy (lease) their access point - the hitch being that they are hopeful (actually the whole
business model) is that the user goes over 500mb per month - nothing for a normal user, but just for meeting nights probably doable.
http://www.freedompop.com/

"That Others May Zoom"

krnlpanick

Quote from: Eclipse on December 13, 2012, 09:15:00 PM
And that money would come from where?  Exactly?

Not to mention that Hooterville composite might not have any broadband options, nor a permanent place to run wires.

I'm all for modernization, and for most units it could be accommodated on some level or another, but we don't need more unfunded mandates, and NHQ's pockets are down to the lint right now just like everyone else.

With that said, some of the wireless broadband providers officer service to 501c(3)s for free or greatly reduced cost.  I also ran across
this company recently, that offers "free" service if you buy (lease) their access point - the hitch being that they are hopeful (actually the whole
business model) is that the user goes over 500mb per month - nothing for a normal user, but just for meeting nights probably doable.
http://www.freedompop.com/

Fair points Eclipse, and perhaps I should have been more clear when I was speaking about discounted service - there are options both as a 501(c)3 and as a DoD "Contractor" that could be open to us to acquire said internet access at a greatly reduced cost on a national level. The trick is that someone at national has to start that process and negotiate the terms. I would also contest that the situations where getting internet access for a squadron is simply *NOT* possible are very few and far between. If we can get cheap internet to some tiny village in the middle of nowhere in some third world country that no one has heard of there is no reason that we can not get it to our cadets. In the worst case scenario, if your squadron is really in that remote of a location you still have options in town libraries and schools - negotating times to use the library or school library as a "testing center" could be an option as well. The real problem is not the availability itself but the lack of creativity and initiative to really seek it out in most cases I would guess.

2nd Lt. Christopher A. Schmidt, CAP

coudano

I'm sure you'll see all tests online, eventually.

The idea of not putting it online initially i'm sure, had to do with ensuring that it was close-book and proctored.
As some have pointed out, there are ways to deal with that (and those ways are already implemented on the Spaatz Exam, which is computer proctored)

Wheel of progress turns slow, patience :)

Eclipse

Quote from: krnlpanick on December 13, 2012, 09:23:16 PMThe real problem is not the availability itself but the lack of creativity and initiative to really seek it out in most cases I would guess.

No argument there.

"That Others May Zoom"

Майор Хаткевич

Ah, but now we're going to the blame game, which is why I was all for the thumbstick idea.

NHQ writes the code. Authorized folks can download the software on a spare $5 stick. Works with or w/o internet.

TJT__98

I think the reason is because they want to keep the milestones closed book, and if they were online they could say closed book but not enforce it.
C/MSgt
Wright Award 21322

Cool Mace

Quote from: TJT__98 on December 13, 2012, 09:31:55 PM
I think the reason is because they want to keep the milestones closed book, and if they were online they could say closed book but not enforce it.

Aux already covered how it could be an online secured test.

I would like to see them online myself. I think it will happen, all in due time.
CAP is what you make of it. If you don't put anything in to it, you won't get anything out of it.
Eaker #2250
C/Lt Col, Ret.
The cookies and donuts were a lie.

Майор Хаткевич

Quote from: TJT__98 on December 13, 2012, 09:31:55 PM
I think the reason is because they want to keep the milestones closed book, and if they were online they could say closed book but not enforce it.

This would happen at the unit as it does currently. Just on a PC.

jimmydeanno

When the idea of online testing was proposed, there was significant backlash from the CAP community (mostly non-cadet program officers), who said that going online and open-book was watering down the program and that cadets wouldn't learn what they needed to.  The compromise for this was to keep the milestones closed-book and on paper, proctored locally.

The majority of CPOs that I've talk to are in favor of online milestone awards (I am), but there is still a select group of people that used to be cadets or want to involve themselves with cadets (without having a vested interest in them), that are still against that move.  Hopefully, the resistance will die as those people move on.

The CAP world is an interesting community.  There are people who have nothing better to do than critique the use of Gandhi in the leadership books because "He isn't Christian and this is America."  So, you can imagine the feedback that comes in from moves like online testing.
If you have ten thousand regulations you destroy all respect for the law. - Winston Churchill

Eclipse

Quote from: jimmydeanno on December 13, 2012, 09:46:21 PM
When the idea of online testing was proposed, there was significant backlash from the CAP community (mostly non-cadet program officers), who said that going online and open-book was watering down the program and that cadets wouldn't learn what they needed to.  The compromise for this was to keep the milestones closed-book and on paper, proctored locally.

I frankly still feel this way - online / open book is a test of your Google skills, not your knowledge.  As least having them as practical
tests at the milestone level means someone might have to learn something at some point.

"That Others May Zoom"

johnnyb47

How about a hybrid solution?
PDF's of the individual test booklets opened on a squadron computer and an all purpose answer sheet onto which the cadet can write their answers?
No test storage required, no internet required, answer sheet still in hand for hand grading.
Just make sure the PDF's are secured on the testing officers or squadrons laptop.

Capt
Information Technology Officer
Communications Officer


Uploaded with ImageShack.us

Майор Хаткевич

Quote from: Eclipse on December 13, 2012, 09:50:52 PM
Quote from: jimmydeanno on December 13, 2012, 09:46:21 PM
When the idea of online testing was proposed, there was significant backlash from the CAP community (mostly non-cadet program officers), who said that going online and open-book was watering down the program and that cadets wouldn't learn what they needed to.  The compromise for this was to keep the milestones closed-book and on paper, proctored locally.

I frankly still feel this way - online / open book is a test of your Google skills, not your knowledge.  As least having them as practical
tests at the milestone level means someone might have to learn something at some point.

That would still happen.

Having come back to active status after a few years of online testing, I can already say that there is a cultural change in the cadet ranks. IMO not for the better.

lordmonar

Quote from: usafaux2004 on December 13, 2012, 09:40:27 PM
Quote from: TJT__98 on December 13, 2012, 09:31:55 PM
I think the reason is because they want to keep the milestones closed book, and if they were online they could say closed book but not enforce it.

This would happen at the unit as it does currently. Just on a PC.
Back in the day...I got a web based testing program...loaded the first 10 or so test into it.  And i proctored the cadets as the took the tests "on line".  When they were done....it instantly graded it and told them what questions they missed and what chapters to study....just like we are supposed to do on paper.

The reason why we don't do this...probably no one has ever proposed it to the CP guys as doable....and IIRC there was a lot of push back from the field about going to online testing.
PATRICK M. HARRIS, SMSgt, CAP

krnlpanick

Quote from: Cool Mace on December 13, 2012, 09:33:04 PM
Aux already covered how it could be an online secured test.

While I am not necessarily a glory hog, I would like to state for the record that Aux was agreeing with my idea I believe :)
2nd Lt. Christopher A. Schmidt, CAP

Pylon

The Marine Corps uses Questionmark Perception software for online proctored exams.  It requires the student log-in to the regular web course and select a test, at which point it prompts for a valid proctor's login, and then launches the Perception software.  The software locks the user from closing out or minimizing or otherwise exiting the test window (among other things) preventing the user from flipping between windows or programs or having the textbook or Google open in the background.  It obviously doesn't stop users from using notes, their phone, or another computer but that's why the proctor has to enter their login -- theoretically the proctor is present, but knowing the workstation is at least locked down, they don't have to be watching every single screen at every moment.  They really only have to ensure students don't have phones or notes or other devices out.  Makes it easier to proctor a roomful at once, if need be.
Michael F. Kieloch, Maj, CAP

coudano

Quote from: johnnyb47 on December 13, 2012, 09:52:21 PM
How about a hybrid solution?
PDF's of the individual test booklets opened on a squadron computer and an all purpose answer sheet onto which the cadet can write their answers?
No test storage required, no internet required, answer sheet still in hand for hand grading.
Just make sure the PDF's are secured on the testing officers or squadrons laptop.

You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

krnlpanick

Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

That being said, you have far better control of an electronic test than you do of a traditional paper and pencil test:

1) Question randomization - How many school exams have been passed by kids memorizing the order of answers from a leaked or stolen answer key?
2) Larger question bank - Further control can be gained by offering different wordings and formats of the same questions so that simple memorization can not be used to pass the test - requiring students/cadets to actually understand the material they are testing on
3) Maintainability - Curriculum that is static is stagnant. Knowledge is an organic lifeform of it's own that is ever evolving - having the test and testing materials available online allows easier maintenance of that material
4) Cost - Whether it be cost to distribute printed materials or perceived cost of time spent by the testing officer grading tests, it is simply more cost effective to proctor exams electronically.
5) Accuracy - Even testing officers make mistakes every once in a while, but a computer does exactly what it is programmed to do.

As for the security element of electronic testing, electronically proctured tests are no more or less secure than paper tests - the security issue with most tests is the human element in most cases. If the testing officer leaves the paper answer key sitting on his desk, that is just as much a security issue as that same testing officer leaving the laptop sitting on the desk unattended. Also, with the added control of the electronic test, even if someone sees the test before taking it, it is highly likely that if it is done correctly - it will be a completely different test when they take it.
2nd Lt. Christopher A. Schmidt, CAP

EMT-83

Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

krnlpanick

Quote from: EMT-83 on December 13, 2012, 11:35:55 PM
As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

Actually, to be entirely accurate ours is in it's own filing drawer in a completely different filing cabinet than the paper tests; but I digress.
2nd Lt. Christopher A. Schmidt, CAP

cm42

Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.

Майор Хаткевич

Quote from: krnlpanick on December 14, 2012, 12:25:22 AM
Quote from: EMT-83 on December 13, 2012, 11:35:55 PM
As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

Actually, to be entirely accurate ours is in it's own filing drawer in a completely different filing cabinet than the paper tests; but I digress.

Yep. It's an interesting rule, but it works.
Quote1-5. Testing Material Storage. All hard-copy testing materials will be stored in a lockable (key or combination) metal cabinet. It can be a safe, filing cabinet or standing cabinet. Nothing other than testing materials can be stored in the locked area.

lordmonar

Quote from: cm42 on December 14, 2012, 02:06:44 AM
Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.
assuming they are willing to do that....having paper tests locked up will not deter them from cheating.
PATRICK M. HARRIS, SMSgt, CAP

Extremepredjudice

Quote from: cm42 on December 14, 2012, 02:06:44 AM
Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.
Wouldn't be hard to get the paper tests, either.
I love the moderators here. <3

Hanlon's Razor
Occam's Razor
"Flight make chant; I good leader"

SarDragon

One of the problems I've seen locally is that units can have a meeting place, but with no facilities such as 'Net access, storage, etc. Bring everything in before the meeting; take it out at the end. That makes it really hard to do all these kool things everyone is talking about. In this case, nothing beats a paper test.
Dave Bowles
Maj, CAP
AT1, USN Retired
50 Year Member
Mitchell Award (unnumbered)
C/WO, CAP, Ret

Cool Mace

Quote from: krnlpanick on December 13, 2012, 10:24:04 PM
Quote from: Cool Mace on December 13, 2012, 09:33:04 PM
Aux already covered how it could be an online secured test.

While I am not necessarily a glory hog, I would like to state for the record that Aux was agreeing with my idea I believe :)

My apologies, good sir.

:AHEM!: I believe Krnlpanick already covered  how it could be an online secured test.
CAP is what you make of it. If you don't put anything in to it, you won't get anything out of it.
Eaker #2250
C/Lt Col, Ret.
The cookies and donuts were a lie.

Майор Хаткевич

Yes,

eServices is the EASIEST route to go. The issue becomes one if infrastructure.

Now, what they COULD do, is provide the test online to the Testing Officers as an option within every cadet's testing record. The TO would PRINT a generated test from a question bank, and once the cadet is done, the TO would enter it into the system. The system would grade the test, give out a score, and a printable reference page for missed questions. If the cadet did not pass, then the next time the test is printed, the questions are different. Since the Award tests are based on the Achievement tests, they could then use the cadet's past testing history, and even make the test MORE meaningful by throwing in the questions cadets may have missed in the past. This would ensure that they aren't "just passing" but studying up on what they miss as well. There's a lot that can  be done on the back end of a system like this to make the program stronger.

It may sound like a lot of work, but it's honestly no different than what I do now (Look at the key and mark their answer sheet), manually get their score, and manually write out what they need to work on.

This would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.

a2capt

They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

Майор Хаткевич

Quote from: a2capt on December 14, 2012, 05:53:51 PM
They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

I wasn't aware that the Achievement tests are graded manually by someone at NHQ...

krnlpanick

Quote from: usafaux2004 on December 14, 2012, 05:54:52 PM
Quote from: a2capt on December 14, 2012, 05:53:51 PM
They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

I wasn't aware that the Achievement tests are graded manually by someone at NHQ...

I believe that Aux was insinuating that the *system* would grade the tests, and the results would be locked up at NHQ (in the database) rather than in each TO's filing cabinet. I see no manual process here for NHQ at all other than implementing the process. I think this is a pretty good compromise wrt my suggestion as well. Either way would have the same benefits and make the program stronger. Really the big difference is keeping them as "paper tests" for the cadets or moving them online - the end result of either is exactly the same. The current weakness is that the tests are static, difficult to maintain, and subject to human err in grading. While there is still the possibility of human err in "fat fingering" the input by the TO into the system using Aux' strategy it is IMHO less than the possibility of human err in manually grading the test.

Just MHO.
2nd Lt. Christopher A. Schmidt, CAP

a2capt

Quote from: usafaux2004 on December 14, 2012, 05:54:52 PMI wasn't aware that the Achievement tests are graded manually by someone at NHQ...
No.. but..
Quote from: usafaux2004 on December 14, 2012, 05:50:47 PMThis would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.
You elluded to that kind of system here. So I responded, and "someone" to me, means a person, not a system/computer. Which means open for business the next day kinda delay.

Майор Хаткевич

Quote from: krnlpanick on December 14, 2012, 06:22:01 PM
While there is still the possibility of human err in "fat fingering" the input by the TO into the system using Aux' strategy it is IMHO less than the possibility of human err in manually grading the test.

Easy fix. After submitting, the system could ask the TO to check that they marked the correct (the cadet's actual) answers for whichever are wrong. That's a double check that should resolve 99% of that problem.

Quote from: a2capt on December 14, 2012, 06:25:37 PM
Quote from: usafaux2004 on December 14, 2012, 05:54:52 PMI wasn't aware that the Achievement tests are graded manually by someone at NHQ...
No.. but..
Quote from: usafaux2004 on December 14, 2012, 05:50:47 PMThis would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.
You elluded to that kind of system here. So I responded, and "someone" to me, means a person, not a system/computer. Which means open for business the next day kinda delay.


I assumed that when I'm talking of eService's tests, "graded by NHQ" implies the same way they grade current online tests. It's 2012. I don't expect anyone having an online system with a manual grader.

Cool Mace

QuoteEasy fix. After submitting, the system could ask the TO to check that they marked the correct (the cadet's actual) answers for whichever are wrong. That's a double check that should resolve 99% of that problem.

I like this idea. That would help a lot when it comes to making sure the "wrong" answers are really incorrect, and not just human error on the TOs part.
CAP is what you make of it. If you don't put anything in to it, you won't get anything out of it.
Eaker #2250
C/Lt Col, Ret.
The cookies and donuts were a lie.

Flyboy-F22

The reason we have the test in paper is because it is strongly suggested that an administrator, administer the test because it is impossible to enforce the closed book rule. You will find that is the way the milestone awards work. Also, you are doomed if you somehow cheat or hack the test when it comes to paper tests.
Thank You,
C/TSgt Gerot
Element Leader
Omaha Composite Squadron, NEWG

Майор Хаткевич

Have you read the thread or just doing a drive by?