eServices problem?

[SunDog]

Uhhh...not too many IT professionals would build a custom system on spec, then offer it up, unsolicited, for free.  Tough to ambush a customer that way - most of them would like to actually be involved in the process.

My criticism is for eservices, which has definitley improved some in the last few years. But it remains a clumsy, obscure, disjointed, and buggy application.  Doesn't mean the IT folks at National aren't good professionals. More likely they don't have the staff and/or money to do more than limited scope, incremental enhancements and bug patching. It's a bad system. Really. Bad. Surprisingly so, for a paid membership organization of this size.  The system navigation is an excellent "how not to" example. I spent an hour with it today, entering some Ops Quals, which likely should have taken 10 minutes. Not good to annoy your user base, waste limited volunteer time, or to be surprised if they start hurrying through tasks (or putting them off ).

Someone is going to comment that "it's not so bad once you learn it".  That translates into "I spent long, frustrating hours struggling with this pig, but now I know the work-arounds. Until I need to use it for a new task".

[jimmydeanno]

Uhhh...not too many IT professionals would build a custom system on spec, then offer it up, unsolicited, for free.  Tough to ambush a customer that way - most of them would like to actually be involved in the process.

My criticism is for eservices, which has definitley improved some in the last few years. But it remains a clumsy, obscure, disjointed, and buggy application.  Doesn't mean the IT folks at National aren't good professionals. More likely they don't have the staff and/or money to do more than limited scope, incremental enhancements and bug patching. It's a bad system. Really. Bad. Surprisingly so, for a paid membership organization of this size.  The system navigation is an excellent "how not to" example. I spent an hour with it today, entering some Ops Quals, which likely should have taken 10 minutes. Not good to annoy your user base, waste limited volunteer time, or to be surprised if they start hurrying through tasks (or putting them off ).

Someone is going to comment that "it's not so bad once you learn it".  That translates into "I spent long, frustrating hours struggling with this pig, but now I know the work-arounds. Until I need to use it for a new task".

I had the opportunity to see e-services mapped out in a wire diagram.  It was on the wall of the IT guys' office.  Oy vey, is all I could say.  But, there are few of them and they get tasked with a bunch of stuff.  There is only so much they can do.

[Eclipse]

One >major< issue right now is that Ops Quals function does not match the regulations, which means that
even when you power through the poor UI and other technical issues, things still don't act the way they are supposed to,
which cause further confusion.

[SunDog]

I kinda figured the resources were limited for IT at CAP.  A system like eServices gets very "brittle" with age; constant patching, kludges, "good enough" work arounds, and after a few years you get a nightmare of convoluted code, with forgotten or hidden dependencies, that is very tough to enhance, or even maintain very well. Like an old pair of jeans with more patches than denim.

Gotta be tough for them to even test properly. Hence the mess with the SETS roll-out, and the gacked error messages that say one thing, but are caused by another.  One of our guys went round and round, trying to figure out why the system was telling him he wasn't SETS qualified to update a qual for a trainee. Only the real issue was his safety currency had lapsed the day before.  Not what the message said, of course, and the date of the qual was before his currency lapsed.  Just not a good implementation of new business rules. But, again, tough to do on a system in such rough shape.

Makes it hard on the membership, and turns off new folks, big time.  Hard to fathom why it doesn't get more attention and resources, since it's probably the only contact most members have with National - it is the mission critical, enterprise system for the organization.  I have seen worse, but not often, not for a customer-facing system.

[NIN]

I kinda figured the resources were limited for IT at CAP.  A system like eServices gets very "brittle" with age; constant patching, kludges, "good enough" work arounds, and after a few years you get a nightmare of convoluted code, with forgotten or hidden dependencies, that is very tough to enhance, or even maintain very well. Like an old pair of jeans with more patches than denim.
<snip>
Makes it hard on the membership, and turns off new folks, big time.  Hard to fathom why it doesn't get more attention and resources, since it's probably the only contact most members have with National - it is the mission critical, enterprise system for the organization.  I have seen worse, but not often, not for a customer-facing system.

You pretty much nailed it.

15+ years ago, NHQ was looking for a junior programmer. Coincidentally, I was a pretty junior programmer. Unfortunately, the salary was laughable (I was, at the time, making about $35,000 a year in Detroit. Even accounting for cost of living adjustments between Detroit and Montgomery, it was probably a 30% pay cut).  This was at the very beginning of eServices, too. 

Over the years, there has been a pretty serious brain drain as people are hired, work on a system, gain experience and then go someplace else that will actually pay them for their skills.  This leads to a lot of that patchworking.   That led to my joke about "Alabama .NET programmers." (Helps if you say it with a Jeff Foxworthy voice)

I once worked on a system that managed about $20M in annual revenue for a software company.  It had started out as a home-grown system, never intended to be used by people outside of the sales & marketing teams of a fairly small outfit.  Web-based, MS SQL server back end, etc.   That  company got acquired by a much larger software company (the one I worked for).  The system had grown, had functionality added to it and morphed into a multi-use system (sales, sales operations, marketing, license administration) that was also touched by 2-3 other systems or departments (tech support, for example), and then had to interface with the software ordering system (Oracle) at the Mothership in Detroit.

The system had been worked on (before me) by 4 different IT managers, it had been built using 5 different coding styles and a multitude of dependencies (the user provisioning, for example, was tied to an object-based Sales CRM the company had. I can't even remember the name of it. But setting someone up for an ID required major hoop-jumping), and eventually it was being used by sales people in other offices around the world via the WAN.  You know something is amiss when you're getting a phone call at home at midnight from a guy at the sales office in Sidney, Australia apologizing for bothering you...

My programming partner and I had made the case for re-writing the system from the ground up using .NET, eliminating a lot of the goofball dependencies, building a legitimate connector to Oracle , hosting the system in a more centralized location so the field offices could get to it, etc.  All of that was shot down due to "resources," so we spent the next 12-18 months continuing to patch kludgy code before I finally left the company and the rest of my group got laid off.  My old boss called me about 4 weeks after I'd left asking if I could shed any light on the system, as they were moving the whole thing to Detroit and had no idea why it wouldn't work.... LOL!

So eServices has behind it a 15+ year legacy of old code, an originally very dodgy data model, not well-understood and ever-changing business rules (that often don't make a lot of sense when compared to one another), and only probably the barest of institutional memory as to "why things were done the way they were in the code."   That the folks at NHQ can keep eServices running at all amazes me sometimes, and my hat is certainly off to them, because I know what it is like to have to inherit someone else's system and maintain, extend and improve it.

The Air Cadets Organisation in the UK many years ago implemented a nationwide system called "Project Bader". https://news.bader.mod.uk/default.aspx

It is now substantially based on Sharepoint (shudder!), but the idea was that it would be a hierarchical  document management system for paperwork and communications.  So when Cadet Blaggs needs his glider wings, Pilot Officer Smythe initiates a form (electronically) and it goes to Wing Commander Butler for (electronic) signature.  And Plt Ofcr Smythe gets an email or other electronic notification that Butler has received it. And Wing Cdr Butler gets an email saying he's got something for approval in his in-box. And they get tickled when it *doesn't* get acted on.  And I'm betting that Wing Cdr Butler's boss somehow finds out when Butler is living up to his nickname "Bottomless Briefcase Butler" and has 50 items more than a month old still waiting approval.. 

There was more to Project Bader than that, but at the time it was described to me (2002? 2003?) it made e-Services look like AOL.

It is a toss up as to what is more appropriate and effective: a commercially-based system like Sharepoint that can be modifed and extended with APIs and such to mostly meet the needs, or a custom-developed system.  The problem with custom developing a system is that you need more people (programmers, business analysts, report developers, interface designers, documentation & training specialists, infrastructure management people, etc), whereas with an off-the-shelf setup you can (maybe) more easily find developers, the interface is more homogenized, reporting is standardized, interfaces between systems are based on very well developed APIs, etc.

[Spaceman3750]

+1 to NIN. Lots of people like to beat up on the IT/programming people because it's easy, with no real idea of the man hours, costs, and 100 other issues associated with the system.

While the current state of eServices can be frustrating, keep in mind that there are way worse methods of doing things. One of my clients has an off-shore development company that has been deploying changes directly into production for months.

[SunDog]

Hope I didn't insult the IT folks, not my intent. . .I think the failing is in management, of course.  sServices is the brand, the storefront, the place members interact with National, and locally, for almost everything.  I wonder how long National would tolerate a broken front window at HQ, or a backed up toilet, or a long term power outage? Or a receptionist that was rude to visitors.

For sure, they could "work around" all those things. But they wouldn't, not for long. They'd get 'em fixed, and rightly so.  The "as is" with eServices shows a disturbing lack of vision, and consideration for members.  I appreciate the example (shoving changes straight into prod), but, respectfully, it's like telling me having the flu ain't so bad, because someone else has pneumonia.

Rant complete. And pointless, like as not.

[Ned]

Hope I didn't insult the IT folks, not my intent. . .

I'm sure they took the "spam" comment as a compliment.   

I think the failing is in management, of course.  sServices is the brand, the storefront, the place members interact with National, and locally, for almost everything.  I wonder how long National would tolerate a broken front window at HQ, or a backed up toilet, or a long term power outage? Or a receptionist that was rude to visitors.

If eServices could be improved with the effort it takes to unclog a toilet or retrain a receptionist in customer care procedures, I am confident that it would have happened long ago.

As you pointed out earlier, eServices has greatly improved over the years, but as Nin so graphically described, the current system is a legacy system with issues. 

And NHQ is severly resource-constrained.  There have been multiple rounds of staff reductions in recent years.  Mr. Rowland and his team have done terrific work trying to balance the missions and needs of the HQ.  I don't envy the resource allocation choices he has had to make.

But the bottom line is that to achieve the kind of modern eServices that you and other members would like will almost certainly take hundreds of thousands of dollars to develop and code a new system.  (Even in Sharepoint.)

Which obviously we don't have.  It would take a substantial dues increase to raise that kind of funds, which I suspect the membership would find unappealing.  Alternatively, perhaps you can identify which major programs should be eliminated to provide additional support to IT.

(It probably goes without saying that the Appropriations Fairy is very unlikely to contribute any money for IT upgrades.)

Before saying management has "failed," perhaps we should carefully consider the constraints under which they are operating.

The "as is" with eServices shows a disturbing lack of vision, and consideration for members.  I appreciate the example (shoving changes straight into prod), but, respectfully, it's like telling me having the flu ain't so bad, because someone else has pneumonia.

Rant complete. And pointless, like as not.

How strange.  Anonymous ranting on the internet is usually so productive.  Maybe you're not doing it right.

[JeffDG]

The problem is that NHQ isn't using the most abundant resource they have access to:  Volunteers.

I would venture to say that there are thousands of highly-skilled IT personnel in CAP, a large number of whom would be quite willing and able to contribute to a project to fix things, all without appropriations.

Yet, we have folks at NHQ who insist on spending time to not facilitate, but to actively discourage, member developed systems that could be of huge benefit to CAP, then plead poverty of resources when asked to provide that functionality themselves.

NHQ could produce a lot more results if they focused on building secure interfaces to the data, defining standards, and coordinating member-developed projects, rather than fighting them.

[Ned]

The problem is that NHQ isn't using the most abundant resource they have access to:  Volunteers.

I can't really disagree with that.  The question then becomes "why?"

Indeed, as a long-time CP guy I always wondered why NHQ didn't make more use of volunteers in their shop at NHQ.  (Like IT, CP is severely understaffed and under resourced to accomplish what the membership expects.)

So, what sort of issues might it present to have volunteers working on IT issues? 

Are there examples of other organizations that used volunteers to code some sort of "eServices - equivalent" product to serve over 60,000 members, a couple hundred employees, and track and maintain the qualifications for the folks who operate a couple of hundred million dollars worth of Federally-provided assets? 

For example, does the Red Cross or the Air Force Association use some of their many volunteers / members in their IT functions at the national level?

How about the BSA? 

NASA is certainly budget-constrained lately and has had some severe IT blunders that resulted in a satellite or two being lost.  Do they use volunteers for any of their coding?

Does the Wikipedia organization have any lessons for us here?

Do you really think that the IT folks at NHQ are biting the helping hands being offered them out of spite or closed-mindedness?  What reason could they have for doing that?

So let's talk about how to constructively engage volunteers in a cost-effective manner.  But let's start with some historical examples where it has been successful in a comparable project.

[Storm Chaser]

The technology is certainly available to allow for volunteers to work remotely and in collaboration with others across the country. The open source community is proof that this is feasible. As a software test engineer, I wouldn't mind helping out. I'm sure there are many more software professionals that would jump at the opportunity to help improve eServices.

[jeders]

But let's start with some historical examples where it has been successful in a comparable project.

If everyone waited for someone else to succeed, nothing would ever get done. I understand your desire for empirical evidence to show that this won't be a massive waste of time and resources, eventually creating a bigger mess. But I would point to the relative success of several member generated projects. Granted, WMU was very annoying at times, but when it came out it did what eServices, or MIMS at that time, couldn't. Given a reasonable amount of time and coordination, I believe that some of our own members could build a better eServices.

[JeffDG]

The technology is certainly available to allow for volunteers to work remotely and in collaboration with others across the country. The open source community is proof that this is feasible. As a software test engineer, I wouldn't mind helping out. I'm sure there are many more software professionals that would jump at the opportunity to help improve eServices.

Yep...Linux was my first thought when asked for an example, far, far more complex than eServices, and collaboratively built by a combination of professional, amateur and volunteer developers.

There needs to be a paridigm shift.  We don't necessarily need "eServices 2.0".  What is more needed is documentation of what information is available, what information is needed, and standards of how to access and modify that data securely.

This stuff doesn't necessarily need to be one big application, but rather a series of modules that accomplish different parts of the process.

Then you get into niche situations.  You have someone, now that he has a defined process for it, developing an app for NESA to record SQTR task completion, and builds it on the iPad.  So the instructor/evaluator has the task guide, SQTR, and student list on one screen, along with the evaluation criteria, and as the criteria are met for each student, he puts a check next to it, and it automatically uploads completion into OpsQuals.  We will never have sufficient staff at NHQ to do something like that, and the staff we have will spend more time trying to figure out what's needed than actually developing it.  An instructor who is an iOS developer in the real world however can build something that makes his job immeasurably easier, gives NHQ more up-to-date, more accurate, and just plain better, information, all for near zero cost to CAP.

Hell, look at the capability of something like IMU.  It's been developed by members.  NHQ personnel have actively sought to shut it down through various methods.  Yet, members have managed to keep it working and improving.  Take the time spent trying to block the thing, and instead build a secure API for it, and suddenly you have a tool that can be used across the organization.

[NIN]

The problem is that NHQ isn't using the most abundant resource they have access to:  Volunteers.

I would venture to say that there are thousands of highly-skilled IT personnel in CAP, a large number of whom would be quite willing and able to contribute to a project to fix things, all without appropriations.

Yet, we have folks at NHQ who insist on spending time to not facilitate, but to actively discourage, member developed systems that could be of huge benefit to CAP, then plead poverty of resources when asked to provide that functionality themselves.

NHQ could produce a lot more results if they focused on building secure interfaces to the data, defining standards, and coordinating member-developed projects, rather than fighting them.

Truly, there is (well, _was_) a fair amount of "not invented here" at NHQ for a lot of stuff that prevents them from letting the "volunteers" behind the magic curtain..

I had discussed with them many years ago the possibility of establishing some some directory services for remote authentication.  (plus, as part of that, I had the idea, if you search for it on here, of a HQ-based hierarchical CMS & email system for unit pages/domain names/domain administration/email administration that was substantially database back-end/directory authentication based)  Since the idea didn't originate from HQ, it was immediately shot down and pish-toshed.  I said "OK, cool, whatever.." and went back to doing things the brute force way.

Part of the issue is that we need to understand (very clearly) what the "business needs" are that we're satisfying/solving with our IT systems.  Those needs are certainly substantially NHQ's needs (first), but need to trickle down to the subordinate HQ's needs as well.  But if you think about it, much of the field's "business needs" are different from the HQ's needs, so how well does HQ understand how things get used in the field?  (to be fair, probably they probably have a much, much better understanding in the last few years than it was, say, 10 years ago)

Another thing is the highly customized nature of eServices makes "external help" very difficult.  You can employ volunteer subject-matter-experts to help you understand your use cases, business rules, etc, but how many *more* .NET wonks do you want putting their butterfingers in your already fairly fragile custom codebase?  (hint: you don't want me doing it. eServices would order you a pizza every time you went to print a 101 card...)

So if you want to rely on your volunteer base of expertise, then you should probably wind up on a highly standardized and well known/documented platform/framework instead of a heavily customized system.

But getting off the heavily customized system and onto a standards-based, well-known system is going to be a hell of an undertaking, too.  (and just about the time you do, the vendor decides to drop all support for that product and go in another direction!)

The answers are not easy nor are they very good.

[Eclipse]

I'd like to know who reviews the "business rules" before they are implemented, and who knows those rules better? A contractor or an experienced member.

If you need to hire coders to maintain things, fine, but volunteer staff and commanders should be at the head of all committees and teams.

The number things that I routinely dig out that "no one ever thought of" are, quite frankly, both amazing and disappointing on so many levels,
especially glaring conflicts between regulations and the way systems function as is currently the case with OPS Quals.

I am, and never was, a fan of the WMU, but certainly it was demonstrated that one, or a small group of people can create a system
that would serve our needs. 

60,000 records?   That's one spreadsheet.

Most of what eServices needs are a bunch of "if this then that" type checks.

Somewhere between that spreadsheet and eServices is where we should actually be, and in this day and age, if
we're paying much, if anything, for the software we're using, someone isn't paying attention.

[JeffDG]

But getting off the heavily customized system and onto a standards-based, well-known system is going to be a hell of an undertaking, too.  (and just about the time you do, the vendor decides to drop all support for that product and go in another direction!)

The answers are not easy nor are they very good.

You don't need to discard the whole customized system.

a)  You don't want outsiders mucking about in the internals of the system.  Period, full stop.
b)  You define interfaces to the system to perform functions that outsiders need

So, take IMU as an example.  One thing it needs to do is create Air/Ground Sorties in WMIRS.  OK, so you create an authenticated API that permits the user to do a "CreateSortie(<MissionNumber>,<Type>,<sortieSettingsXML>,<authToken>)" and define all of your fields in the XML spec.  So, now, if an external app needs to create a sortie, they can do so via that method.  How WMIRS stores that data is of no consequence to the person creating the sortie, nor is how to pull reports.  You don't need to rebuild WMIRS, just add an API front-end thereto.

Now, once that's built, someone else comes along and decides to do an Android app that lets the director of an ORide day create sorties for the day on the fly from his Galaxy Tab.  No work needs to be done by NHQ at all now, as the interface already exists and can be used for an additional purpose.

NHQ is also now free to modify the look-and-feel of WMIRS as necessary to take care of the "manual entry" folks, without breaking apps that have had to build elaborate "screen scraping" processes to work around blocks.  They can even completely restructure the underlying database, provided that they update the API interface to keep putting new sorties in the right place.

[Storm Chaser]

I'd like to know who reviews the "business rules" before they are implemented, and who knows those rules better? A contractor or an experienced member.

You're absolutely right. In every software project I've worked on, we've always had subject matter expects (SMEs) working closely with developers to ensure we had good requirements, business rules, performance, workflows and usability.

[Jaison009]

To answer your question about ARC we have paid IT and a disaster support technology function that is nearly all volunteers. Our IT and DST play different roles. IT is concerned with systems where DST is concerned with service delivery in the face of disaster. IT handles day to day, DST brings cell phones, laptops, satellite connectivity, IP printers/faxes/gateways/etc. to support disaster relief operations.  We use a system called Group/Activity/Position. These are functional. Groups are overarching concepts i.e. Ops Mgmt, Individual Client Services, Mass Care, Info and Planning, External Relations, Logistics, Staff Srvs, and DST. DST is then divided into a few different activities: networking, computer operations, communications, and customer service. DST has all of their procedures on every single computer on a DR so everything is written out on what needs done, how to do it, and is mostly OJT. Each activity then has a hierarchy of positions starting: service associate, supervisor, manager, and chief. Each has specific activity, DR requirements, life experience, and Red Cross training associated with each.

We are working harder to integrate IT and DST together and bridge a chasm like the one you are discussing. At the local level, in some areas DST volunteers are the IT group. In others DST does not exist. At national it is hit and miss. One of the ways this bridge is supported is through our disaster reengineering cycle.  There is a new concept we are applying called Blue Sky/Gray Sky. This concept holds that everyone has a daily job and during disaster, that job might/will change and have some disaster element to every job. During blue sky IT is focused on maintaining our daily operational systems, intranet, physical assets, etc. During gray sky, IT will be working much more closely with DST (who consequently is made up mostly of numerous IT professionals) and hopefully DST will be working to backfill IT. 

The problem is that NHQ isn't using the most abundant resource they have access to:  Volunteers.

I can't really disagree with that.  The question then becomes "why?"

Indeed, as a long-time CP guy I always wondered why NHQ didn't make more use of volunteers in their shop at NHQ.  (Like IT, CP is severely understaffed and under resourced to accomplish what the membership expects.)

So, what sort of issues might it present to have volunteers working on IT issues? 

Are there examples of other organizations that used volunteers to code some sort of "eServices - equivalent" product to serve over 60,000 members, a couple hundred employees, and track and maintain the qualifications for the folks who operate a couple of hundred million dollars worth of Federally-provided assets? 

For example, does the Red Cross or the Air Force Association use some of their many volunteers / members in their IT functions at the national level?

How about the BSA? 

NASA is certainly budget-constrained lately and has had some severe IT blunders that resulted in a satellite or two being lost.  Do they use volunteers for any of their coding?

Does the Wikipedia organization have any lessons for us here?

Do you really think that the IT folks at NHQ are biting the helping hands being offered them out of spite or closed-mindedness?  What reason could they have for doing that?

So let's talk about how to constructively engage volunteers in a cost-effective manner.  But let's start with some historical examples where it has been successful in a comparable project.

[NCRblues]

I don't really have a dog in this fight as I am not an IT guy...

But the Encampment Management Program was done by New York volunteers and its used today in many encampments and saves a lot of time and energy for those staffs...

[NIN]

I don't really have a dog in this fight as I am not an IT guy...

But the Encampment Management Program was done by New York volunteers and its used today in many encampments and saves a lot of time and energy for those staffs...

Agree. It rocks the darn casbah.

But how much of it is based on NHQ's data model? Almost none.  Does it actually interface with NHQ? No.

It is a standalone app that uses the old CAPWatch system to do its participant authentication. (Now that CAPWatch is no longer, you have to jump thru hoops to put your wing's data in there)