Main Menu

Former member survey

Started by MajTbird, February 26, 2018, 11:54:59 PM

0 Members and 1 Guest are viewing this topic.

MajTbird

Greeting all,

Five years ago at a wing conference I attended a breakout session on recruiting and retention.  After listening to much discussion I asked the question, "Why do people quit CAP?"

I got no answer.  Apparently no one at NHQ follows up with former members to understand why they gave up their membership.  I think this is an important--critically important--thing to track and understand.  If we don't we run the risk of focusing primarily on recruiting while ignoring retention.  At various conferences I keep asking the same question but no one has an answer.  (I did get one answer to my question:  "Well, members die."  No, seriously.  That was offered as a primary reason for losing members.)

Ok, I've worked with some researchers to construct a survey for former members.  It is short--only nine questions--and takes about five to eight minutes to complete.  But it will give us heretofore unknown insight into why we lose members.  And, as a byproduct, insight into how we can all improve our organization.

The survey was built with the help of academics who do this sort of thing all the time and have at their disposal substantial analysis tools.  Thus, this is an academic effort that will become part of a research project studying non-profit organizational leadership and strategy.  It is hosted by Auburn University.

It is an anonymous survey; there is no login required and it in no way tracks or identifies respondents.  The results will be reported here on CAPTalk after being compiled and analyzed.  I've attached a copy of the survey so you may review the contents.

If you know anyone who has left CAP or who has been long-term inactive, please email the survey link to them and ask them to complete it.  Too, and this is critically important and relies on our ethics, please help maintain the integrity of the survey by not answering it if you are still an active member.  If, as a member, you feel it would be beneficial to survey membership on similar topics please let me know by posting here or PM me.

Survey link:  https://auburn.qualtrics.com/jfe/form/SV_1MRXH7sLqO9SgUR

Many thanks and semper vigilens!

MajTbird


P.S.  Apparently I can't upload files; I don't see any way to do so.  I'll happily upload the survey--in PDF form; size of the file is 101k--if anyone can point out how to do so.

Fester

Ned has said before on this forum that they DO do exit surveys of former members.  I know they do for Cadets, not sure for Senior Members.
1stLt, CAP
Squadron CC
Group CPO
Eaker - 1996

MajTbird

You may be right, Fester.  I hope you're right for cadets.  But, if NHQ does do exit surveys they refuse to share the data or the results with us.  And apparently they don't share it with wing, regional or even their own recruiting & retention folks because for years no one can address why members leave.  Maybe they just don't want us to know.  But I'd prefer to believe it's the more innocent assumption that they just don't know versus wanting to keep us in the dark (not good as we're all loyal members).

If you (or anyone following this) know of former members or long-term inactives please share this with them and let's see what we can learn.  The survey has only been live for a couple of hours and it is already getting responses.

Fester

Here is what Ned (NHQ Cadet Programs Manager) said about retention rates and causes of loss on a thread last week:

Quote
I hope it won't surprise you to learn that we regularly survey former members asking these very questions.

In essence, most cadets leave feeling reasonably positive about the program, but the #1 reason listed is "lost / changed interest."  Other top reasons are "moved, new location with no nearby unit.  (About 10-15% of Americans move every year.). "Poor leadership" is listed, but midway down the list in single digits.

When we try to mine down on "lost / changed interest" responses, it is hard tease out meaningful data. Respondents mention competing activities like sports programs, other youth programs, church activities, and concentrating on academics.

Obviously, much of this is out of our control, but there are certainly things we can address:  ensuring that weekly meetings (>90% of our cadet contact time) are vital and engaging instead of "the AE instructor didn't come tonight, so drill around the parking lot."

And we do indeed use the data to drive retention strategies.  We have adjusted the PT program to make sure cadets are far less likely to "stall" at an early stage in their cadet career specifically to engage them in the promotion system and allow for new duties and positions.  We have significantly invested in the encampment program to make it more accessible because one of the strongest indicators for renewal is encampment attendance.

We have specifically addressed the "poor leadership" issue by revamping the TLC program and placing incentives in the system to encourage attendance by CP seniors.  We also made the TLC program itself more accessible by re-designing it into a one day course for most squadron-level CP officers.

We routinely task the NCAC to provide concrete ideas to improve retention and listen carefully to their input.

Obviously, despite a great deal of effort by dedicated CP officers from the local unit and higher, our first year retention numbers remain well below 50%.

I repeat my request:

What specific things can we do to improve retention?

Ned Lee
National Cadet Program Manager

(Currently attending the CAWG Cadet Programs Conference at Camp San Luis Obispo with nearly 400 enthusiastic cadets from every wing in PCR.  It is an amazing activity.  But these troops are not the retention problem, it is the 1100 CAWG cadets are not here having a great time.)
1stLt, CAP
Squadron CC
Group CPO
Eaker - 1996

Fester

I would, however, love to see the results of the survey.  Will you share from time to time?
1stLt, CAP
Squadron CC
Group CPO
Eaker - 1996

FW

There have been numerous exit surveys taken throughout the years.  Usual top reasons for members leaving (for seniors) are: Nothing worthwhile to do, lacking good leadership/training, not what expected, kids left...

Nothing has changed that much.  We still have the same problems which no one really wants to fix.  Our friend "NIN" has a good finger on the "correct button".  Targeted recruiting, improved training opportunities, and better resources for units would go a long way, however the old saying of "money talks and ## walks...." is a factor.  We must find a way of improving our contributor base.  In a world where the tax code now puts a barrier on charitable giving, this may prove more difficult.

Ned

#6
I don't think I have much to add beyond my answer in the other thread quoted above.

I'm a little concerned about the notion that we "refuse" to share the information, or that we are doing nothing with the information.  That's just silly and begins to sound a bit like Black Helicopter stuff.  I shared the CP perspective (CAP's largest mission in terms of manpower) above, and Col Weiss has commented on the Senior side. 

CP doesn't do the surveys or archive the information, or I'd be happy to share it with you.  I suspect NIN is probably the best CT forum member to ask if you are not content to use the CoC.

On a practical note, I'd be interested in your results.  AFAIK, we've never tried a natural language instrument because of the difficulty in processing and analyzing the results.  The nascent data scientist in me is interested in your analytics.

Ned Lee

MajTbird

Hi Ned,

And thanks for your thoughts and insights.  I will be happy to share what we learn. 

I have access to text analytics tools that are pretty amazing at clustering sentiment.  I made half the survey open-ended questions because I knew getting a solid sample would be tough given the unknown and likely unreachable population (those who have quit CAP).  There are methods that can be applied to such problems, such as the Heckman correction, to account for such and those techniques are open to us given the categorical data collected if it appears helpful.  But, I'm optimistic that the sentiment analysis will agree with the objective scoring to provide a reliable picture of why members leave CAP.

I have emails from CAP NHQ where I asked to have data (redacted) and it has been refused every time.  I was told to "run my request by the John Salvadore and the Nat'l Cmdr and Moose" and they'd make a decision (I think we both know what that means).  I was also asked to provide a letter from a university outlining the research if it were to be a research project.  Just recently CAP sent a survey of pilots and I asked for the data from that survey.  Again, I was told that it would go to certain people in CAP for their review and there are no plans to share it with anyone else.  If you're looking for silly Black Helicopters you might want to start in the basement of NHQ; there are none here.  If there's any way you could shake loose some data that'd be much appreciated!

So, I am left to conclude that there is no interest in sharing data with members.  They seem to only be interested in giving us their interpretation of whatever data is collected.  I think that's unfortunate and fails transparency.

- MajTbird


dwb

A few years ago when I was a Wing DCP, my Wing CC would periodically receive exit survey data from NHQ for the Wing, and he would share it with me. Don't assume that not sharing with whole membership is the same as not sharing at all. People might be less inclined to fill out member surveys if they knew every Tom, Dick, and Harry could read their (ostensibly anonymous) remarks.

Put another way, the HR department of your employer is probably not sharing the raw data from employee exit surveys with every single employee. Some of this data is protected for good reason.

That said, my memory of the exit survey data was that it wasn't terribly insightful. Cadets grow out of the program, seniors find other things to do, etc. Certainly some people have bad experiences, but a comfortable majority of the exit survey responses I read were along the lines of "I enjoyed my time in CAP, but I'm on to something else".

SarDragon

Posted at the OP's request.
Dave Bowles
Maj, CAP
AT1, USN Retired
50 Year Member
Mitchell Award (unnumbered)
C/WO, CAP, Ret

TheSkyHornet

Most members don't receive/fill out a survey when they leave. I'd say the majority walk away and give no indication that they were walking away, and none that they were returning. People just fall off the roster and fade out.

You're going to get better feedback if you can actually make contact with those individuals than the ones willing to openly share why they're leaving in anger or because of a job change; that's just a given.


NIN

Quote from: TheSkyHornet on February 27, 2018, 08:28:19 PM
Most members don't receive/fill out a survey when they leave. I'd say the majority walk away and give no indication that they were walking away, and none that they were returning. People just fall off the roster and fade out.

You're going to get better feedback if you can actually make contact with those individuals than the ones willing to openly share why they're leaving in anger or because of a job change; that's just a given.

This is certainly an issue with our present schema of exit surveying.  From the timeframe that the surveys are sent out (anywhere between 90 and 180 days after expiry) to the fact that some folks have CAP-issued email addresses in their eServices account which leads to the exit survey going to account that's no longer being checked, the response rate was pretty abysmal.  Thats the bad news.

The good news is that we're in the process of changing the way we do exit surveys in an attempt to get a better response and better data. There are a few people (data scientists, institutional researchers) who are looking at ways to improve this.

To the OP's point: its not that NHQ "refuses to share the results." Wing commanders get this information frequently. Its that the survey results are often inconclusive and kind of open ended. 

For example, in a recent survey departed members were asked to ID the *one* reason they left.  About 20% said they didn't have enough time for CAP, work conflicted.  17.5% said "Other" (and there were free-text responses, but they were all over the map: "A combination of reasons," "Multiple reasons," "I'm too old," "the treatment of the cadets by the squadron commander," etc. You get 5 people who marked "Other" and you get 5 different responses.. not really helpful, and definitely not statistically beneficial). 15% said their time and talents were not valued by the unit. 10% said it was for medical reasons.  The numbers go down from there precipitously, but my point is that data in and of itself isn't going to tell anybody much without some context around them.

If you've been around, you've heard me talk about these things.  We've done webinars on this. And really, the data doesn't change too much from year to year, but we need to translate that into what everybody can understand and act on.

For example, when 20% say "They didn't have enough time for CAP, work conflicted," that says to me that we need to talk with units and explain the importance of recruiting people for "fit" in the organization. And I talk about that pretty frequently.  If 20% of the people who responded to the survey had work conflicts, how much talking to them in the onboarding process did we do?  Did we not do a good enough job setting the expectation level and interviewing them to be sure they'd be the right fit for CAP before they forked over their fingerprints?

Same goes for the "time and talents not valued by the unit." This speaks to the importance of commander & staff training on member management, and encouraging units on engaging their membership and using their talents appropriately. And speaks back to the recruiting "fit" part, and not recruiting someone just because they fog a mirror and can write a check, and then leaving them to sit there wondering why they joined this outfit while the pilots drink coffee and tell flying stories. (sorry, pilots, I had to stereotype there...)  If someone a year later says "My time wasn't valued," thats a problem that is almost entirely fixable at the unit level first.  So we come up with training on member engagement and the importance of the first 6 months of a members time in CAP, etc.

Its not that CAP is hiding information. There are people actually acting on this information.  The availability of the exit data might change when we change our survey schema, thats really not for me to decide. But as Dan said, the raw data we have now sometimes "isn't ready for primetime."
Darin Ninness, Col, CAP
I have no responsibilities whatsoever
I like to have Difficult Adult Conversations™
The contents of this post are Copyright © 2007-2024 by NIN. All rights are reserved. Specific permission is given to quote this post here on CAP-Talk only.

RiverAux

I'm usually a fairly big fan of doing surveys and in my real job do a few every year.   However, I agree with the above comments about CAP exit surveys probably being less useful than one might think. 

However, it might be useful to have such data broken down by years of CAP service.  I'd suspect that most seniors that leave after a year or two fall more into the "it wasn't what I expected" or "it takes more time than I thought" categories.  While weeding those people out on the front end would be nice, there will always be some sort of component that falls into those categories and we just have to expect to lose some percentage of new folks to that every time no matter how great the program is.

What I would be more interested in is in the reasons for leaving among those that have been in CAP for 5-15 years, which is probably the golden period when most CAP members are most productive and useful to the organization.  There will be a certain percentage of these folks who leave due to personality conflicts with other CAP members (i.e., "politics), but if there are significant issues causing them to leave (that CAP has some influence over), then we really need to know them. 

Holding Pattern

Silly thought: What about an annual survey of CURRENT members regarding what they think are the greatest risks to them leaving CAP?

GaryVC

One thing that wasn't mentioned is that some senior members leave when their children (or grandchildren) leave. However, there are more than a few that stay.

TheSkyHornet

Quote from: Mordecai on February 28, 2018, 01:12:04 AM
Silly thought: What about an annual survey of CURRENT members regarding what they think are the greatest risks to them leaving CAP?

That's also fairly good data if you can get it.

You have to be cautious with this sort of thing, though, as it may be mis-perceived as if there is an issue. People start to get jumpy when you say "Are you on the verge of quitting," like you're either suggesting to them that things aren't going well or prying into their personal matters.

I think most units have a good indicator of the morale within the unit and can see what's bringing people, keeping them, or driving them away. The problem is, if this information was sent up to higher, getting units to report truthfully and not be on the "high horses" thinking their program isn't chasing people out. I would bet that most issues with people storming out are either going to be from issues within their immediate command or a personal discrepancy with CAP in general (whatever reason that may be). Most of the people who fade away likely do so from a general lack of interest.

MajTbird

Quote from: Mordecai on February 28, 2018, 01:12:04 AM

Silly thought: What about an annual survey of CURRENT members regarding what they think are the greatest risks to them leaving CAP?


Not a silly thought at all.  Developing one now and will start distributing it soon.  And yes, panel data (annual data from the same members over time) is an excellent way to track sentiment.  There'll be an option to opt-in for that while remaining anonymous, yes.

Quote from: NIN on February 27, 2018, 09:04:57 PM

Its not that CAP is hiding information. There are people actually acting on this information.  The availability of the exit data might change when we change our survey schema, thats really not for me to decide. But as Dan said, the raw data we have now sometimes "isn't ready for primetime."


This is still don't get.  If no one is hiding information why not just send it to a member when the request is made?  Obviously raw data isn't ready for prime-time.  But many, many members have the tools and are adept at analyzing data.  Not sharing it until it goes through some sort of "review" smacks of hiding something (I'm not accusing; just wondering out loud).

The fact that I've been directly told that they (NHQ) will not send me data, well, I'm not sure there's any way to look at that other than hiding data.  I suggested that they could send the data to my university and suddenly a myriad of really pointless (and made up, frankly) hurdles and roadblocks appeared.  Puzzling, as we are members and it sure seems like members who fund the organization 100% through dues and taxes should be able to get data--raw or polished, preferably both--from the organization we're funding.

But, it's a fairly easy fix as it's simple enough to just construct a separate data machine.

abdsp51

Quote from: MajTbird on March 01, 2018, 05:52:09 PM
Quote from: Mordecai on February 28, 2018, 01:12:04 AM

Silly thought: What about an annual survey of CURRENT members regarding what they think are the greatest risks to them leaving CAP?


Not a silly thought at all.  Developing one now and will start distributing it soon.  And yes, panel data (annual data from the same members over time) is an excellent way to track sentiment.  There'll be an option to opt-in for that while remaining anonymous, yes.

Quote from: NIN on February 27, 2018, 09:04:57 PM

Its not that CAP is hiding information. There are people actually acting on this information.  The availability of the exit data might change when we change our survey schema, thats really not for me to decide. But as Dan said, the raw data we have now sometimes "isn't ready for primetime."


This is still don't get.  If no one is hiding information why not just send it to a member when the request is made?  Obviously raw data isn't ready for prime-time.  But many, many members have the tools and are adept at analyzing data.  Not sharing it until it goes through some sort of "review" smacks of hiding something (I'm not accusing; just wondering out loud).

The fact that I've been directly told that they (NHQ) will not send me data, well, I'm not sure there's any way to look at that other than hiding data.  I suggested that they could send the data to my university and suddenly a myriad of really pointless (and made up, frankly) hurdles and roadblocks appeared.  Puzzling, as we are members and it sure seems like members who fund the organization 100% through dues and taxes should be able to get data--raw or polished, preferably both--from the organization we're funding.

But, it's a fairly easy fix as it's simple enough to just construct a separate data machine.

Maybe because not every rank and file member needs it.  Maybe your reasoning for wanting the data does not require for it to be released.

MajTbird

Quote from: abdsp51 on March 01, 2018, 07:06:31 PM
Maybe because not every rank and file member needs it.  Maybe your reasoning for wanting the data does not require for it to be released.

Fair enough.  But here's my reasoning:  It is an organization fully-funded by taxpayers and member dues.  There are zero monies going to CAP that are not supplied through taxes or dues.  I don't want to sound all legalistic--I'm not--but that's our information.  Obviously personal or national security data isn't.  And I don't want that data.  I just want to know what's going on in my organization, unfiltered.  Why, for example, would they not share the responses from the recent survey of member pilots?  What could possibly be so secretive about that?  Not sure why that would spook anyone.  But it sure isn't forthcoming and I was told firmly that it would not be shared.  And what's wrong--seriously--with sharing exit data from members who left CAP with anyone who wants to see it and not just with wing commanders?  Jeez, maybe one of the 60,000 members who isn't a wing commander or a privileged data recipient may have some good ideas to improve retention.  What a concept.

Thus, being we're members and taxpayers I think the more appropriate question might be to them:  What is CAP's reasoning for not sharing performance, survey, financial and asset use data with us?  It's not a need thing (and who gets to decide need anyway?).

Broader question:  Why are so many in CAP so irritated by questions?  Seriously, why?

abdsp51

Quote from: MajTbird on March 01, 2018, 07:53:40 PM
Quote from: abdsp51 on March 01, 2018, 07:06:31 PM
Maybe because not every rank and file member needs it.  Maybe your reasoning for wanting the data does not require for it to be released.

Fair enough.  But here's my reasoning:  It is an organization fully-funded by taxpayers and member dues.  There are zero monies going to CAP that are not supplied through taxes or dues.  I don't want to sound all legalistic--I'm not--but that's our information.  Obviously personal or national security data isn't.  And I don't want that data.  I just want to know what's going on in my organization, unfiltered.  Why, for example, would they not share the responses from the recent survey of member pilots?  What could possibly be so secretive about that?  Not sure why that would spook anyone.  But it sure isn't forthcoming and I was told firmly that it would not be shared.  And what's wrong--seriously--with sharing exit data from members who left CAP with anyone who wants to see it and not just with wing commanders?  Jeez, maybe one of the 60,000 members who isn't a wing commander or a privileged data recipient may have some good ideas to improve retention.  What a concept.

Thus, being we're members and taxpayers I think the more appropriate question might be to them:  What is CAP's reasoning for not sharing performance, survey, financial and asset use data with us?  It's not a need thing (and who gets to decide need anyway?).

Broader question:  Why are so many in CAP so irritated by questions?  Seriously, why?

One it is the organizations data, not yours or my data the organizations.  The organization is allowed to determine who has access to the data.  Simply because you pay dues and taxes does not entitle you to data. 

On the pilot survey have you stopped to think that maybe they are not done processing and analyzing the data for the CC? Is having that data going to stop your mission? 

And on sharing exit data quit simply not everyone needs to know why members leave.  I can tell you as a CP guy I don't care about ES mission, sortie, and comms data.  I care about CP data, and sure it would be nice to know why members especially cadets leave, but at the end of the day I am not going to have heartburn over it. 

If you have an idea on retention submit it up the chain and follow up.  You don't need data to come up with ideas and submit them.  Data would help yes but you don't need it. 

You made the request and someone at NHQ told you you don't need to know.  You can always submit an FOIA request. 
When it comes to information simply because you pay dues and taxes doesn't mean you have a right, need or entitlement to know everything.