Why aren't the Milestones proctored on a computer?

Started by Майор Хаткевич, December 13, 2012, 08:40:40 PM

0 Members and 1 Guest are viewing this topic.

lordmonar

Quote from: usafaux2004 on December 13, 2012, 09:40:27 PM
Quote from: TJT__98 on December 13, 2012, 09:31:55 PM
I think the reason is because they want to keep the milestones closed book, and if they were online they could say closed book but not enforce it.

This would happen at the unit as it does currently. Just on a PC.
Back in the day...I got a web based testing program...loaded the first 10 or so test into it.  And i proctored the cadets as the took the tests "on line".  When they were done....it instantly graded it and told them what questions they missed and what chapters to study....just like we are supposed to do on paper.

The reason why we don't do this...probably no one has ever proposed it to the CP guys as doable....and IIRC there was a lot of push back from the field about going to online testing.
PATRICK M. HARRIS, SMSgt, CAP

krnlpanick

Quote from: Cool Mace on December 13, 2012, 09:33:04 PM
Aux already covered how it could be an online secured test.

While I am not necessarily a glory hog, I would like to state for the record that Aux was agreeing with my idea I believe :)
2nd Lt. Christopher A. Schmidt, CAP

Pylon

The Marine Corps uses Questionmark Perception software for online proctored exams.  It requires the student log-in to the regular web course and select a test, at which point it prompts for a valid proctor's login, and then launches the Perception software.  The software locks the user from closing out or minimizing or otherwise exiting the test window (among other things) preventing the user from flipping between windows or programs or having the textbook or Google open in the background.  It obviously doesn't stop users from using notes, their phone, or another computer but that's why the proctor has to enter their login -- theoretically the proctor is present, but knowing the workstation is at least locked down, they don't have to be watching every single screen at every moment.  They really only have to ensure students don't have phones or notes or other devices out.  Makes it easier to proctor a roomful at once, if need be.
Michael F. Kieloch, Maj, CAP

coudano

Quote from: johnnyb47 on December 13, 2012, 09:52:21 PM
How about a hybrid solution?
PDF's of the individual test booklets opened on a squadron computer and an all purpose answer sheet onto which the cadet can write their answers?
No test storage required, no internet required, answer sheet still in hand for hand grading.
Just make sure the PDF's are secured on the testing officers or squadrons laptop.

You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

krnlpanick

Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

That being said, you have far better control of an electronic test than you do of a traditional paper and pencil test:

1) Question randomization - How many school exams have been passed by kids memorizing the order of answers from a leaked or stolen answer key?
2) Larger question bank - Further control can be gained by offering different wordings and formats of the same questions so that simple memorization can not be used to pass the test - requiring students/cadets to actually understand the material they are testing on
3) Maintainability - Curriculum that is static is stagnant. Knowledge is an organic lifeform of it's own that is ever evolving - having the test and testing materials available online allows easier maintenance of that material
4) Cost - Whether it be cost to distribute printed materials or perceived cost of time spent by the testing officer grading tests, it is simply more cost effective to proctor exams electronically.
5) Accuracy - Even testing officers make mistakes every once in a while, but a computer does exactly what it is programmed to do.

As for the security element of electronic testing, electronically proctured tests are no more or less secure than paper tests - the security issue with most tests is the human element in most cases. If the testing officer leaves the paper answer key sitting on his desk, that is just as much a security issue as that same testing officer leaving the laptop sitting on the desk unattended. Also, with the added control of the electronic test, even if someone sees the test before taking it, it is highly likely that if it is done correctly - it will be a completely different test when they take it.
2nd Lt. Christopher A. Schmidt, CAP

EMT-83

Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.
It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

krnlpanick

Quote from: EMT-83 on December 13, 2012, 11:35:55 PM
As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

Actually, to be entirely accurate ours is in it's own filing drawer in a completely different filing cabinet than the paper tests; but I digress.
2nd Lt. Christopher A. Schmidt, CAP

cm42

Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.

Майор Хаткевич

Quote from: krnlpanick on December 14, 2012, 12:25:22 AM
Quote from: EMT-83 on December 13, 2012, 11:35:55 PM
As long as it's in a separate drawer in that filing cabinet, and not secured with the tests.

Actually, to be entirely accurate ours is in it's own filing drawer in a completely different filing cabinet than the paper tests; but I digress.

Yep. It's an interesting rule, but it works.
Quote1-5. Testing Material Storage. All hard-copy testing materials will be stored in a lockable (key or combination) metal cabinet. It can be a safe, filing cabinet or standing cabinet. Nothing other than testing materials can be stored in the locked area.

lordmonar

Quote from: cm42 on December 14, 2012, 02:06:44 AM
Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.
assuming they are willing to do that....having paper tests locked up will not deter them from cheating.
PATRICK M. HARRIS, SMSgt, CAP

Extremepredjudice

Quote from: cm42 on December 14, 2012, 02:06:44 AM
Quote from: krnlpanick on December 13, 2012, 11:06:11 PM
Quote from: coudano on December 13, 2012, 10:44:21 PM
You are now making security of the laptop an issue.

It's quite frankly easier (and more secure) to lock paper tests in a file cabinet, than to try and guarantee a computer's security.

FWIW - there is no reason you couldn't lock that very same laptop up in the very same filing cabinet; as a matter of fact that is exactly where our squadron's laptop lives.

Much easier to hook the USB drive up, copy the PDFs, and be out before anyone notices. Brute force crunching on the PDFs and they'll be out in the open in no time.
Wouldn't be hard to get the paper tests, either.
I love the moderators here. <3

Hanlon's Razor
Occam's Razor
"Flight make chant; I good leader"

SarDragon

One of the problems I've seen locally is that units can have a meeting place, but with no facilities such as 'Net access, storage, etc. Bring everything in before the meeting; take it out at the end. That makes it really hard to do all these kool things everyone is talking about. In this case, nothing beats a paper test.
Dave Bowles
Maj, CAP
AT1, USN Retired
50 Year Member
Mitchell Award (unnumbered)
C/WO, CAP, Ret

Cool Mace

Quote from: krnlpanick on December 13, 2012, 10:24:04 PM
Quote from: Cool Mace on December 13, 2012, 09:33:04 PM
Aux already covered how it could be an online secured test.

While I am not necessarily a glory hog, I would like to state for the record that Aux was agreeing with my idea I believe :)

My apologies, good sir.

:AHEM!: I believe Krnlpanick already covered  how it could be an online secured test.
CAP is what you make of it. If you don't put anything in to it, you won't get anything out of it.
Eaker #2250
C/Lt Col, Ret.
The cookies and donuts were a lie.

Майор Хаткевич

Yes,

eServices is the EASIEST route to go. The issue becomes one if infrastructure.

Now, what they COULD do, is provide the test online to the Testing Officers as an option within every cadet's testing record. The TO would PRINT a generated test from a question bank, and once the cadet is done, the TO would enter it into the system. The system would grade the test, give out a score, and a printable reference page for missed questions. If the cadet did not pass, then the next time the test is printed, the questions are different. Since the Award tests are based on the Achievement tests, they could then use the cadet's past testing history, and even make the test MORE meaningful by throwing in the questions cadets may have missed in the past. This would ensure that they aren't "just passing" but studying up on what they miss as well. There's a lot that can  be done on the back end of a system like this to make the program stronger.

It may sound like a lot of work, but it's honestly no different than what I do now (Look at the key and mark their answer sheet), manually get their score, and manually write out what they need to work on.

This would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.

a2capt

They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

Майор Хаткевич

Quote from: a2capt on December 14, 2012, 05:53:51 PM
They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

I wasn't aware that the Achievement tests are graded manually by someone at NHQ...

krnlpanick

Quote from: usafaux2004 on December 14, 2012, 05:54:52 PM
Quote from: a2capt on December 14, 2012, 05:53:51 PM
They have spent the better part of the last 10 years trying to -get away- from grading tests.

You want to wait for WB results now? Then that makes the "Correct to 100%" part a multi week process, too, as when most units have meetings, NHQ is closed.

I wasn't aware that the Achievement tests are graded manually by someone at NHQ...

I believe that Aux was insinuating that the *system* would grade the tests, and the results would be locked up at NHQ (in the database) rather than in each TO's filing cabinet. I see no manual process here for NHQ at all other than implementing the process. I think this is a pretty good compromise wrt my suggestion as well. Either way would have the same benefits and make the program stronger. Really the big difference is keeping them as "paper tests" for the cadets or moving them online - the end result of either is exactly the same. The current weakness is that the tests are static, difficult to maintain, and subject to human err in grading. While there is still the possibility of human err in "fat fingering" the input by the TO into the system using Aux' strategy it is IMHO less than the possibility of human err in manually grading the test.

Just MHO.
2nd Lt. Christopher A. Schmidt, CAP

a2capt

Quote from: usafaux2004 on December 14, 2012, 05:54:52 PMI wasn't aware that the Achievement tests are graded manually by someone at NHQ...
No.. but..
Quote from: usafaux2004 on December 14, 2012, 05:50:47 PMThis would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.
You elluded to that kind of system here. So I responded, and "someone" to me, means a person, not a system/computer. Which means open for business the next day kinda delay.

Майор Хаткевич

Quote from: krnlpanick on December 14, 2012, 06:22:01 PM
While there is still the possibility of human err in "fat fingering" the input by the TO into the system using Aux' strategy it is IMHO less than the possibility of human err in manually grading the test.

Easy fix. After submitting, the system could ask the TO to check that they marked the correct (the cadet's actual) answers for whichever are wrong. That's a double check that should resolve 99% of that problem.

Quote from: a2capt on December 14, 2012, 06:25:37 PM
Quote from: usafaux2004 on December 14, 2012, 05:54:52 PMI wasn't aware that the Achievement tests are graded manually by someone at NHQ...
No.. but..
Quote from: usafaux2004 on December 14, 2012, 05:50:47 PMThis would also make the tests 100% more secure, as not even the TO would have the keys, only National. Still a paper test, but graded by NHQ.
You elluded to that kind of system here. So I responded, and "someone" to me, means a person, not a system/computer. Which means open for business the next day kinda delay.


I assumed that when I'm talking of eService's tests, "graded by NHQ" implies the same way they grade current online tests. It's 2012. I don't expect anyone having an online system with a manual grader.

Cool Mace

QuoteEasy fix. After submitting, the system could ask the TO to check that they marked the correct (the cadet's actual) answers for whichever are wrong. That's a double check that should resolve 99% of that problem.

I like this idea. That would help a lot when it comes to making sure the "wrong" answers are really incorrect, and not just human error on the TOs part.
CAP is what you make of it. If you don't put anything in to it, you won't get anything out of it.
Eaker #2250
C/Lt Col, Ret.
The cookies and donuts were a lie.