The Great California CAP Email Debacle of 2011

[FalconFour]

I think this warrants a thread, just to serve at least as a record of this momentous trainwreck of a distribution list debacle.

So, here's the rundown of the sequence of events thus far (names removed just in a small attempt to preserve their remaining dignity):

IT guy sends out an email to "all" saying:

To all CAWG Members
Distribution list services have been restored and I am currently in the process of updating all the various distribution lists.

I have automated the list publishing regarding the general distribution list at this time.  I am extracting the primary and secondary email address information from your e-services account.  Please ensure these are correct and kept up to date. This will be
performed on a regular basis to ensure new members are uploaded and to catch changes in your email addresses

If you do not want to be on the general distribution list (*Clickable link to ALL list email address*), then remove your email addresses from e-services.  This action will also result in the removal of your email address from your unit and group distribution lists.

The extraction process will also be done for the alerting system and anyone who wants to have their cell phone, smart phone, pager, etc... be part of the alerting list, please ensure that the device email address is added in the ALPHA PAGER field in e-
services.

If you have any questions, please let me know and please DO NOT HIT REPLY ALL to this message.

Regards
(Anon.)

Simple enough, I didn't think much of it myself. I got the message on both my email accounts, everything's working fine, I want to stay on the list, bleh, standard procedure as usual.

Then, a couple hours later, I get this odd email addressed to "all"...

To: "all" <all>
X-Mailer: iPhone Mail (7D11)
Date: Mon, 4 Jul 2011 17:37:40 -0700
Subject: Email removal request
From: *****@clearwire.net

Please remove the above email address from all CAP related announcements.

*thinking to myself* "... umm, I hope about a thousand people just smacked this guy upside the head..."
Maybe so, but about half an hour later... another similar message, "Remove from e-mail list" with that as both the subject and the message (how quaint...).

Then, slowly but surely, all hell breaks loose...


A couple choice emails for amusement:

Date: Tue, 5 Jul 2011 09:38:03 -0700
To: all
Subject: Please Remove
From: B**** Z**** <**uuuuu***@gmail.com>

Please remove my email from the list.

--
() ()
(o.o)
(>/\<)o

THIS IS BUNNY
HELP HIM FINISH HIS CARROT AND DOMINATE THE WORLD BY COPYING THIS TO YOUR SIGNATURE...!!! (created by Frost)

Date: Tue, 5 Jul 2011 12:39:21 -0400 (EDT)
To: e*****@sbcglobal.net, all
X-Mailer: AOL 9.6 sub 5004
Subject: Re: Removal
From: R*********@aol.com

Please remove me from the ALNOT distribution list.
Thanks

In a message dated 7/5/2011 9:37:41 A.M. Pacific Daylight Time, e*****@sbcglobal.net writes:

Please remove me from all list. As I am recieving notices on accounts  that
have been use before, I have included all poss accounts to be  removed
from. Sorry for the  'extras'.
E*****@sbcglobal.net
E****##@gmail.com
Again, please  remove me from all list on both my accounts. Thank you, SM H*** ######

Sent from Yahoo! Mail on Android

And my personal favorite...

Date: Tue, 5 Jul 2011 17:12:58 +0000
To: B****** S****** <b******@s******.com>, all
Subject: Re: REMOVE
From: "W. H. *******" <k#***@socal.rr.com>

Nice move, old son. You have just broadcast this to the entire wing -- including those of us who cannot and care not a royal rat's ass whether you can get unsubscribed or not.

---- B****** S****** <b******@s******.com> wrote:
Please remove my email from ALL CAWG emails - you folks need to do a better job of managing email.  All the reply alls are friggen disaster of a clusterflub.  Nice job!

I've gotta wonder what the thought process is that would make anyone think that they should be using any email address titled "All" to send a removal request!

And on the flip side of things, I also just have to wonder why the IT guys left "All" open to all members... I can sort of understand it for emergency reasons, but... well, I guess they never expected people to be dim enough to abuse it in such a mind-bogglingly massive way...

SO! What of it, any similar stories? Thoughts and LOLs? I wonder how much longer this will go on for...  :clap:

edit: Hmm, I didn't really edit anything, trying to figure out where my avatar and profile info is. *shrug* Probably because I added it after this post, oh well.

[a2capt]

LOL, I too just finished reading all that.. When I saw the first message, I thought "this is going to go over extremely well.. there's a lot of people who don't want to be on the distribution list" .. and just adding everyone is going to rile a lot of people.

Worse is, email communications to the ITO are a black hole.

I really hope the Winds of Change with new Command blow throughout the whole structure. There's a lot of folks that have had positions so long they've become recluses.

[arajca]

You know those who eventually are removed from the list will be the biggest complainers that no one tells when anything is happening, right?

[FalconFour]

It's why I've always hated old-style "mailing lists"... and really, email as a communications medium in general. I use my email strictly for notification, not for communication - in the ~20 years that email has been around in its current form, it's become just a total trainwreck of botched protocols and a spam cat-and-mouse game. Whenever someone tries to tack communication onto email (ironically, its originally intended purpose), it ends up being a total disaster like this. "Give a guy an inch..."

Discussion boards, of course, have their own issues... for example, on this forum, I only get notified of replies to my own topics, or ones I manually "enable". I wouldn't get notified of new topics, say, if someone posts something interesting. With a mailing list, I get a little notification every time someone says something. There's just no grey area...

But, as evidenced by the "aol.com" emails, some people are still stuck in the stone age and like their email communications. Really, I think the problem here is that people simply don't know how to use email! I really point the blame of this debacle on the clueless users behind the computer... sure, nobody should be *able* to send to "all" by accident, but people SHOULD be checking their addresses before sending! It's the most basic of email communications: who is it being sent to? How can this get overlooked... :/

[davidsinn]

It's why I've always hated old-style "mailing lists"... and really, email as a communications medium in general. I use my email strictly for notification, not for communication - in the ~20 years that email has been around in its current form, it's become just a total trainwreck of botched protocols and a spam cat-and-mouse game. Whenever someone tries to tack communication onto email (ironically, its originally intended purpose), it ends up being a total disaster like this. "Give a guy an inch..."

Discussion boards, of course, have their own issues... for example, on this forum, I only get notified of replies to my own topics, or ones I manually "enable". I wouldn't get notified of new topics, say, if someone posts something interesting. With a mailing list, I get a little notification every time someone says something. There's just no grey area...

But, as evidenced by the "aol.com" emails, some people are still stuck in the stone age and like their email communications. Really, I think the problem here is that people simply don't know how to use email! I really point the blame of this debacle on the clueless users behind the computer... sure, nobody should be *able* to send to "all" by accident, but people SHOULD be checking their addresses before sending! It's the most basic of email communications: who is it being sent to? How can this get overlooked... :/

The stupidity is on IT. They should not have had "all" be the response email. The response email should have gone straight to them. My unit's official email is set up this way. If they hit reply it gets kicked to my email not the unit's.

[FalconFour]

The stupidity is on IT. They should not have had "all" be the response email. The response email should have gone straight to them. My unit's official email is set up this way. If they hit reply it gets kicked to my email not the unit's.

Well, it's not technically set up improperly at all... all the emails sent to "all" get sent with the user's email address as the default "reply", so replies will go directly to the user, and only an explicit "reply all" will get the "all" address tacked on. The only real mistake is allowing anyone to post to "all" without requiring any sort of admin confirmation.

The original message specifically said not to use "reply all", however, the flaw was in typing out the full "all" email address in parenthesis. That caused peoples' email clients to turn that into a big, blue, bold clickable link... and anyone stupid enough to ignore the text of the email (specifically saying NOT to use that address) will just click it and send it.

Honestly, it's such an obvious mistake (and I, personally, haven't yet sent an email through the "All" address, even by accident), that I think these nutjobs sending "remove" emails should have their CAP activities suspended, just for the safety of the whole group... it takes some SERIOUS lack of logical judgement to consistently make such a dumb mistake in addressing these emails! :/

[RiverAux]

Why in the world would you encourage people to remove their email address from the national database as the best way to opt-out of a local email list?  What if, for example, national wanted to reach all the members about some issue? 

[a2capt]

Welcome to the Black Hole Logic that our IT directorate follows.

The last piece of fun was because a few people had the usual Outlook address book hijack happen, where it would send random messages with stuff about various drugs out, so the ability to send mail got shut off, and the direction was "send it to me and I'll send it out", so every single message for two days was a "Fwd:" and the original sender was no longer the reply-to, as a reply to would just go to "moderator@cawg..."

First, that looked really professional, and secondly, it meant delays because it was manually done. However, the email server has always been hit and miss anyway, so the delay wasn't that noticeable, it sometimes took an hour or two for your message to come out.  Of course, the system didn't discriminate either. The paging system was not free of the same issues either since it's the same service doing it.


Worse is, when you offer help, suggestions, or outright volunteer to do something, you're just completely ignored anyway. So ..

[NIN]

Why in the world would you encourage people to remove their email address from the national database as the best way to opt-out of a local email list?  What if, for example, national wanted to reach all the members about some issue?

My oh my, how far CA Wing has come from the days of "ca0408.cap.gov" running "Majordomo" (which was also the old "cap-talk@" et al mailing lists. 

(Hint: Majordomo was the software that managed the email list. Majordomo, found here: http://www.greatcircle.com/majordomo/, was written in part by a CAP member, Brent Chapman. If you've never used majordomo, its a) pretty simple; b) pretty foolproof. Tons of control and features for both users & administrators)

But then again, slurping the user list from the wing email list periodically is an alright idea, but telling people to just pull their email from eServices if they don't want to be on the list is plain lazy.

[FalconFour]

Huh. Y'know, being a software developer and all, seeing this kind of IT-services rot really makes me wonder if I could help out there somehow. The IT guys in our CA wing still haven't gotten around to moving our squadron website to a usable server (can't run PHP on that server), which has more or less turned me into a totally function-less CAP member, waiting on a reply from IT.  Maybe I can help with more than just our own website...

[Phil Hirons, Jr.]

But then again, slurping the user list from the wing email list periodically is an alright idea, but telling people to just pull their email from eServices if they don't want to be on the list is plain lazy.

Not to mention impossible. e-Services won't let you delete your primary e-mail. Edit, yes. So unless you want to change your primary to you@wish.com (my personal favorite when e-mail is "required") and have your e-services password resets, duty notifications, etc go into the Ethernet, you WILL get CAWG e-mail

[a2capt]

Heck, it's been since 2005 that PHP hasn't been running, but promised. I actually saw it half working on a citrix session. the interpreter was there, but IIS wouldn't have anything to do with it. A little research showed me that was an easy fix.. but nope, all offers to help. Ignored. Even when Wing issued a command to "get help", they added a rep from each group.. and zero communication has ensued.

Another thing, too. Even cadets are part of this email list, and are getting these nastygrams when people reply to 'all'.

This whole thing is a product of IT not wanting to add the people back manually, who actually want to be on, so the harvest the data from NHQ solution was used, and it's backfiring big time.

It would have been far easier to use a backup and restore the stuff, and find a solution to the problem of adding/removing people that are no longer members of CAP or the wing,  but... whatever.

Yes, the whole thing is run very poorly.

[a2capt]

It's frustrating, because a mail server isn't rocket science, and could be done a lot more efficiently than this. Even using Google's free enterprise stuff as CAP, as many other CAP commands have done, and set up an alternate system just for paging. 

..and getting PHP/MySQL to work on an IIS server .. come on, that shouldn't take 6+ years.

[FalconFour]

Now THAT'S the funny thing. Preferential treatment? Take a look what these guys have on their site: http://sq14.cawg.cap.gov/ - is that... could it... no, is... YES! It's PHP!

And it's also not IIS:

Code Select Expand

C:\Users\Falcon\Documents\tools>tinyget.exe -srv:sq14.cawg.cap.gov -uri:/ -h
HTTP/1.1 200 OK
Date: Tue, 05 Jul 2011 19:25:53 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10

Yeah! It's Apache for these guys... that does tie into what the IT guy mentioned about "moving to the LAMP server" (linux/apache/mysql/php). Definitely a better solution IMO than trying to rig PHP into IIS - I'd done that long ago, and it was never a happy setup.

I wonder where I could go to apply to "be" a CAP IT guy... even in my off time, I could probably turn the place around  :angel:

[Eclipse]

Running unmoderated email lists is unacceptable.  We have the same issue in my wing, though we bring some of it on ourselves
when we send notices about some former member's mother's mailman who died, or non-CAP information through the lists.

But with that said, anyone still running a physical email server in the days of free, robust, secure cloud services is clueless, anyway.

Does this mean the servers are back up?  Because they have been down for at least a week, knocking out email lists
all over the country.

The other thing that is sad / funny / indicative, is that most of the people who complain about too much spam, etc., are usually the ones
causing the issues by sending out ecards, hoaxes ("just had to send / try, you never know"), and forwarding every consarn "too cute, had to
forward" picture they get".

[a2capt]

I still get this: (when hitting a php item)

The page cannot be displayed

You have attempted to execute a CGI, ISAPI, or other executable program from a directory that does not allow programs to be executed.
Please try the following:

Contact the Web site administrator if you believe this directory should allow execute access.
HTTP Error 403.1 - Forbidden: Execute access is denied.
Internet Information Services (IIS)

So a little tiny bit more research shows me that there is indeed a second server running now. Perhaps when this "mess" is settled down, it may be worthwhile to ask to have the site moved to the one running on Linux.

I really do wish I had taken up that offer to be Wing IT before ITO existed.. Might not have lasted this long, but .. this stuff isn't rocket science.

[FalconFour]

That's the same thing I got when I tried PHP (as I thought I'd be able to use when I promised to re-design our site from its original 1990's plain text "under construction" page)... I asked about PHP about 3 months ago, and got a reply saying they were waiting on a reply from the "LAMP admin". Then the same a month later, and again, and again... it seems like it's something we can just "ask" to get, but may never be able to get even after asking. Heck, if the guy would just direct the DNS entry to my own server, I'd just host the thing myself! =P

Email mess seems to have settled down... the last few "all" messages have been from people at wit's end with the clueless "remove" mailers, so it seems people are finally getting the idea! Either that or IT just shut off the "All" list, which would be nice too. Hard to believe they let it go on as long as it did...

[FalconFour]

Well, just when we thought it came to an end with a gigantic crescendo of absolute insanity, people publicly flinging replies about the last guy's "All" email, a couple "ill words" (though completely well-deserved for the utter morons fellow CAP members sending the ALL emails), and a couple MORE braindead "Remove" emails to "All"... (I mean, why hasn't IT shut off this new "ALL" list at this point to put a stop to the mess?!)... a couple new email lists and addresses are set up: one for subscribing, one for unsubscribing, and one "important alerts" list, I guess (personally, I'm staying out of the whole mess and sort of "shouting from the stands" with individual replies).

FINALLY, the wing commander himself puts out a final "EVERYONE SHUT UP NOW I SWEAR TO GOD I WILL PULL THIS CAR OVER!" type of email. Serious business. And the emails stop. Silence, at last.

... yeah, for about 5 hours.

To: all
Date: Thu, 7 Jul 2011 00:08:49 -0400 (EDT)
X-Mailer: AOL Webmail 33912-STANDARD
Subject:
From: A** Z****** <a**z******@aol.com>

please remove me from this list you have downlow it a virus into my computer

i am extremelly upset

I cannot make this s&$@ up. 

[a2capt]

No, He's not making it up. I still have not stopped laughing.

I mean, it's just email. Whoopdie-freaking-doo!

You'd think rapture was coming.

[FalconFour]

I mean, we really need someone to post SOMETHING smart on there. It's a tough little PR situation they've got into here. Personally, this debacle has made me *really* second guess my confidence in CAP, and I'm not just saying this for my sake, but because I imagine a lot of other unspoken individuals have as well! Anyone watching this whole debacle unfold is probably at wit's end with the whole thing if they're even reading the emails... it's just *horribly* depressing to watch.

I just don't know of anything that could really be posted that would revive confidence after something like this... it's crazy to think those responsible would come forward and admit any kind of mistake or fault, which would certainly be a good start, but what else could even be done? 

Or is this kind of stuff more commonplace than I thought, and most people would just brush it off? Haven't even been in CAP a year myself, so still rusty on this stuff

edit: ohyoumean "not a big deal geez shut up kid", not "inorite?"... haha, boy do I feel stupid.