Metrics

[FlyTiger77]

Has anyone developed a comprehensive suite of metrics to monitor the health of (a) squadron(s)?

I trying to determine what the critical information is to monitor all aspects of a CAP unit.

If you would care to share, I would be interested in seeing it.

[a2capt]

One could argue that the items tracked for the Quality Cadet Award are indicators of activity, progress, and the like.

Though they've not been as forthcoming this year so far with the quarterly data.

[Eclipse]

I think this is huge, and CAP essentially ignores this kind of data to their detriment.  The trouble is that the history is nearly impossible to get, so
it can take a couple of years+ to get meaningful numbers.

Unit strength.

Member churn.

Professional development completed / eligible / late.

Yeager completion (should be 100%)

Time it take new members to:
     Join (after first meeting)
    Complete Level I

ES ratings completed / open / expired

Cadet promotions completed / eligible/ late

Meeting attendance.

Other activity attendance.

Squadron of Merit Standing

Quality Cadet Unit Standing.

Individual member rankings in terms of PD/ES/Activities/meeting participation

[A.Member]

PM Sent

[FlyTiger77]

I think this is huge, and CAP essentially ignores this kind of data to their detriment.  The trouble is that the history is nearly impossible to get, so
it can take a couple of years+ to get meaningful numbers.

Unit strength.

Member churn.

Professional development completed / eligible / late.

Yeager completion (should be 100%)

Time it take new members to:
     Join (after first meeting)
    Complete Level I

ES ratings completed / open / expired

Cadet promotions completed / eligible/ late

Meeting attendance.

Other activity attendance.

Squadron of Merit Standing

Quality Cadet Unit Standing.

Individual member rankings in terms of PD/ES/Activities/meeting participation

What formula would you use for "Member churn"?

Automated metrics that can be populated from CAPWatch or some other database without having to hand jam a bunch of numbers would be most excellent.

[Eclipse]

What formula would you use for "Member churn"?

Automated metrics that can be populated from CAPWatch or some other database without having to hand jam a bunch of numbers would be most excellent.

Yes, they would, but good luck with that.

I don't really have a formula, but I'd say starting with anyone who quits within the first 6 months to a year would be considered churn.
And something reasonable for participation probably needs to be included (empty shirts should not be considered in your active numbers, etc.)

That's the problem with the way many commanders (and apparently NHQ), credit themselves with member strength.  They wear the raw number
with no accounting for churn.  A unit with 50 members at the beginning and end of a fiscal year might be seen as stable, until you look
deeper and see they lost 5 experienced members, and churned 15 within the same year, only to wind up with the same number.

It might be interesting to assign some sort of points value to a member's accomplishments, longevity, participation, etc.  Unless it was done
at a wing-level, it'd be arbitrary to the unit, so difficult to compare (though we're really talking about usable single-unit metrics, so that's where
it's important), but I bet an actuary could knock out something reasonable in a few hours.

Certainly a 10-year member with Level III, 2 ES ratings, active participation, and multiple PD ratings is more valuable to the organization
then someone with 20 years in who still has gold bars and shows up twice a year.

[lordmonar]

Certainly a 10-year member with Level III, 2 ES ratings, active participation, and multiple PD ratings is more valuable to the organization
then someone with 20 years in who still has gold bars and shows up twice a year.

I would say certainly a 3 moth member whit level I ACTIVE PARTICPATION is more valuable then anyone else with all the PD/ES/ST ratings who only shows up twice a year.

[FlyTiger77]

I don't really have a formula, but I'd say starting with anyone who quits within the first 6 months to a year would be considered churn.
And something reasonable for participation probably needs to be included (empty shirts should not be considered in your active numbers, etc.)

In the past I have used mean/median longevity as a rough indicator. As it moves to the right, the unit is more stable. It is subject to the influence of a mass influx of members, though, and also gives the 20-year member who only shows up when we are turning avgas into noise the same credit as the member that is there every time the doors are open.

[lordmonar]

You can't develop a metric for a CAP squadron as no one has established any standards for what a squadron is supposed to be producing.

Personnel/personnel turn over/training metrics are all good....and are going in the right directions....but are useless with out "production" goals to base them on.

This is basic TQM and a prime example of why TQM failed in the USAF.   They wanted us to produce and report metrics....but with out production goals they were useless.  (and as a Comm guy we didn't have a production goals.....we had up time rate.......24 hours/7days a week.....)

[Eclipse]

Certainly a 10-year member with Level III, 2 ES ratings, active participation, and multiple PD ratings is more valuable to the organization
then someone with 20 years in who still has gold bars and shows up twice a year.

I would say certainly a 3 moth member whit level I ACTIVE PARTICPATION is more valuable then anyone else with all the PD/ES/ST ratings who only shows up twice a year.

Well, that's where it gets nuanced.

No one with 3 months in can do much of anything of real value to CAP.  With the exception of the random member who joins and hits NESA, an encampment, or some other major training
activity that just happens to occur within weeks or months of joining, most members in the first 6 months to a year are spending their timing finding the bathrooms and otherwise figuring out
what CAP is (or isn't).

Now, the value of an experienced member with a bunch of active ratings who can only show up a few times a year, will depend on when and why he shows up
and for how long, etc.  If he's not a PITA the rest of the year, maintains currency, and generally stays under the radar, but comes in and flies 100 hours for a flight
academy, or provides major training, it's going to be quite a few years before the new guys is equal in "value".

If, during his downtime, he's a constant PITA, off currency, and generally a nuisance when he shows up to an SLS to "start telling everyone how CAP works...", different
equation altogether.

[Eclipse]

You can't develop a metric for a CAP squadron as no one has established any standards for what a squadron is supposed to be producing.

Of course you can.

Perhaps it won't be meaningful to other units, but you can always establish your own local metrics of success.

The total lack of any criteria beyond "self-actualization" is the reason CAP continues to struggle with viability.

Unless you have goals, mandates, and strategies, you can't hold people's feet to the fire for "failure".

You also never know when you're "done". 

The current SUI / CI situation is a great example.  It presupposes standard evaluations across the board, however
we all know that isn't remotely true.  Not only are the inspectors all over the map in regards to abilities and experience,
but the definitions of much of the inspection points are wildly subjective at best.

[lordmonar]

Well, that's where it gets nuanced.

No one with 3 months in can do much of anything of real value to CAP.  With the exception of the random member who joins and hits NESA, an encampment, or some other major training

BS!    And that is where we part ways on volunteerism.   Day one we are getting our Senior Members involved in the job.  Just pulling records for the admin officer, just helping the logistics officer hand out uniforms, or count boot,  Assiting the comm officer in reading the numbers of the radios.   Driving cadets to and from the pick up point.

No one has to have a butt load of training to be valuable contribution to the unit and the mission.

activity that just happens to occur within weeks or months of joining, most members in the first 6 months to a year are spending their timing finding the bathrooms and otherwise figuring out what CAP is (or isn't).

That's a mentoring problem.......and not one we have in our squadron.

Now, the value of an experienced member with a bunch of active ratings who can only show up a few times a year, will depend on when and why he shows up
and for how long, etc.  If he's not a PITA the rest of the year, maintains currency, and generally stays under the radar, but comes in and flies 100 hours for a flight
academy, or provides major training, it's going to be quite a few years before the new guys is equal in "value".

If, during his downtime, he's a constant PITA, off currency, and generally a nuisance when he shows up to an SLS to "start telling everyone how CAP works...", different
equation altogether.

Yes....just because a CAP member is as you would call them an "empty shirt" does not mean they are not valuable themselves.   If they show up and produce....then that's okay by me.

[lordmonar]

Of course you can.

Perhaps it won't be meaningful to other units, but you can always establish your own local metrics of success.

The total lack of any criteria beyond "self-actualization" is the reason CAP continues to struggle with viability.

Unless you have goals, mandates, and strategies, you can't hold people's feet to the fire for "failure".

You also never know when you're "done". 

The current SUI / CI situation is a great example.  It presupposes standard evaluations across the board, however
we all know that isn't remotely true.  Not only are the inspectors all over the map in regards to abilities and experience,
but the definitions of much of the inspection points are wildly subjective at best.

I think we agree here.....except in maybe semantics....but yes....squadron X cannot know how well they are fulfilling CAP's or the Wing's mission goals.....with out having those goals.

[JeffDG]

Of course you can.

Perhaps it won't be meaningful to other units, but you can always establish your own local metrics of success.

The total lack of any criteria beyond "self-actualization" is the reason CAP continues to struggle with viability.

Unless you have goals, mandates, and strategies, you can't hold people's feet to the fire for "failure".

You also never know when you're "done". 

The current SUI / CI situation is a great example.  It presupposes standard evaluations across the board, however
we all know that isn't remotely true.  Not only are the inspectors all over the map in regards to abilities and experience,
but the definitions of much of the inspection points are wildly subjective at best.

I think we agree here.....except in maybe semantics....but yes....squadron X cannot know how well they are fulfilling CAP's or the Wing's mission goals.....with out having those goals.

Well, the good news here is that the person asking for assistance is a Group Commander...

That said, goals need to be set forth, then you can determine what Key Performance Indicators (KPIs) can be used to measure progress towards that goal.

To the OP,. sorry, I've not had a chance to really dive into those spreadsheets you sent me...probably won't be able to until the CI is behind me...

[Eclipse]

Yes....just because a CAP member is as you would call them an "empty shirt" does not mean they are not valuable themselves.   If they show up and produce....then that's okay by me.

Me too.  Empty shirts don't do that.

[bflynn]

Has anyone developed a comprehensive suite of metrics to monitor the health of (a) squadron(s)?

I trying to determine what the critical information is to monitor all aspects of a CAP unit.

If you would care to share, I would be interested in seeing it.

The question that hasn't been asked - are you sure you really want that?  I ask that because I've seen many, many organization ruined by bad metrics.  I'm in the process of writing a white paper for work on metrics and using them to drive business change.  But part of my white paper is NOT using them in certain circumstances.

Once you implement metrics, people will begin to conform to them.  Then people will begin to game the metrics to make themselves look better.  If the metrics are bad then pretty soon, they are chasing the metrics and not doing what needs to be done to make a strong squadron.

My advice here is not to have a comprehensive set of metrics - just have a good set.  Track simple things that you know are good for the squadron - number of members, number of hours in air/ground training, etc.  Maybe track money spent.  Avoid complex metrics such as $ per hour per member because what that means becomes twisted and is easier to manipulate.  You can't game number of member, either you recruit and get more or you don't and have less.

Last - I'd say that it's fine if you want to include some metrics as indicators of personnel performance, but keep it light.  You do not want people trying to game any systems and you don't want metrics to be demotivating...they can be.  So, I'd say never compare people to others using the metrics.  There can be many, many reasons outside the metrics on why someone isn't "performing".  For example, I think its easy to say that the squadron in Oakridge Or is going to have a whole lot more trouble recruiting than say Portland will.  Or even than the previous commander in Oakridge had.

Metrics can be a great TRENDING tool, if used that way.

[RiverAux]

WIWASC I kept track of overall member retention, senior retention, cadet retention and did the same for retention of 1st year members. 
Those were the most useful to me. 

You could really go crazy tracking just about everything under the sun but frankly no one other than you is going to care about them.

Several years ago they implemented a "Commanders Dashboard" at the wing level that kept track of certain things but I'm not sure how much it has been utilized as a tool. 

[Eclipse]

Several years ago they implemented a "Commanders Dashboard" at the wing level that kept track of certain things but I'm not sure how much it has been utilized as a tool.

The numbers are so high-level as to be mostly useless.

[Walkman]

...I ask that because I've seen many, many organization ruined by bad metrics. 

This makes a lot of sense to me. People can have the tendency to focus on the wrong thing, like chasing a number, and lose sight of the bigger picture. It reminds me of the airline crash in the 80's in Florida. On approach the landing gear light didn't light up to show the gear was down, while they kept fiddling with it and trying to figure out if the landing gear was working they forgot to fly the plan and crashed, killing lots of people. Investigation later revealed the the light had burned out.

[Eclipse]

And I've seen more then a few fixed with good metrics, so, use the good ones.

Regardless, without them, there is nothing but status quo.

[Walkman]

Agreed. I think the lesson is that metrics are a tool not the mission itself.

[FlyTiger77]

That said, goals need to be set forth, then you can determine what Key Performance Indicators (KPIs) can be used to measure progress towards that goal.

Let me re-phrase the original question: Does anyone have a comprehenisive list of goals and corresponding metrics for (a) squadron(s)?



Please assume that I am fairly competent at managing an organization of a few hundred people and that I have thought through the ramifications of instituting a system of metrics.

[41839j]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

[mwewing]

I have had this same conversation with several members in the last few weeks. Setting a vision with SMART goals, and establishing KPIs to measure progress is an essential part of leading a team toward a long term vision. Unfortunately, I don't see much of an appetite for this within CAP. There are vision statements and plans that pop up at various levels, but almost none of it seems to be linked together in any meaningful way. IMO this process works best when senior leadership establishes a vision for an organization, and subordinate levels set SMART goals within their content area. This keeps consistency throughout a large organization, and hopefully manages the flow of resources in support of the goals and vision. That said, I think this type of long term planning can be done even at the local level with success. A unit can create a vision statement, set goals, and establish KPIs with or without guidance from higher command. In the absence of strong/consistent direction from more senior leadership, a unit commander can create a unit specific vision. It will be limited by local factors and available resources, but can still be effective in leading a unit toward desired outcomes.

There are glimmers of this type of process throughout CAP. They come by way of annual plans and reports required in some duty assignments/specialty tracks. I have also seen some commanders at various levels institute a vision or set goals in various ways. If more members begin working in this way, it could become the standard. While I don't have a system of metrics completed at this time, I hope to continue working in this direction within my areas of responsibility. I will gladly share my finished product with anyone when complete, and I would be very interested in seeing what others have created as well.

[Eclipse]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

You can't spend time answering micro questions for a macro problem.

The above is why units never get out of their groove of status quo, never grow, and frequently die because one person leaves the program.
Looking at individual pieces of the program, or individual successes,  and treating them as a systemic success is that allows poor unit CC's to sleep at night thinking they are doing their job.

We have one unit inn our wing which has been on life support for years and years, yet when you visit them for things like SUIs or SAVs, they still bring up the Spaatz cadet that left the program
over 12 years ago as if it was relevent today.

Similarly, we had a unit that was essentially a Spaatz factory - several in as many years - but it was all coincidental and circumstantial to the participants, and most
of the leadership were cadet parents.  When the cadets left, so did they, and the unit went from a top-player to almost shutting down in a single year.

One of the most important SMART goals, or at least questions to build a SMART goal, is "Are we growing?"

Because if you aren't, then by design, you are dying, since, at least on the cadet side, your "product" has a finite expiration date.

In my opinion, the primary reason that the organization is not interested in objective criteria and simple answers to direct questions is that the facts would make people sad.

[mwewing]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

This is what the conversation is about. We all understand the basic goal of CAP is to succeed in our 3 missions, but what does success look like? Without a clear vision, SMART goals, and measured progress toward desired outcomes, we can't really determine our success. Long term planning and goal setting, establishes a measurable standard for success. Using metrics to measure KPIs, gives a commander needed data to determine progress, and evaluate future goals and planning efforts.

For example, you state that success in Aerospace education is that cadets learn about aerospace. We know that AE tests are required to progress through most achievements, and units should have regular AE lessons and activities. You can appoint me as AEO and I can teach 1 lesson a month where I read chapters from the AE textbook. The cadets will be bored, won't retain information, and won't be any better prepared for their achievement tests. Strictly speaking, I conducted my sessions... they learned about AE and I was "successful."

Now, I know my example is a bit extreme (hopefully), but it does articulate the need for goals and metrics to measure progress. What does success look like for unit level AE?

• Goal: Cadets will perform better in achievement AE testing; KPI: Cadet AE testing scores will average 90%
• Goal: Squadron will place emphasis on Model Rocketry; KPI: 80% of cadets will complete the Model Rocketry Program within the year
• Goal: Squadron will conduct the AEX program; KPI: The squadron completed the program within the year

These are just some basic MEASURABLE goals that a unit can set based on their local priorities within AE, or based on the direction of Group, Wing, Region, or National AE plans. Of course, reaching these outcomes will require programs and lessons that support the established goals. This process may seem "more complicated" but in reality we are directing our efforts in more productive ways.

[41839j]

Well that is exactly my point.  We are on the same page here.  ANY and EVERY unit actually knows how they are doing if they are honest enough to look objectively at it and doing so will give them the answers that they need.  Not much that anyone else or national can do. 

This organization is like any other volunteer organization is as much as you get out of it what you are willing to put into it.  Now, do we do things together as a unit, or do we only do what we want to do and expect others to pick up the slack?

[jimmydeanno]

There are numerous books and articles that talk about metrics and how they impact your work environment and mission success. 

Here is a good one, I think: http://martinfowler.com/articles/useOfMetrics.html

People will shift their focus to meet the metrics, so you really need to make sure what you are measuring is what you want people to do. 

[bflynn]

My question above was intended to make sure you thought it through - I have seem this done wrong too many times not to react to it.

I think from the lack of response, I'm hearing that nobody here has any metrics.

I also think that getting them can be tricky - using the Spaatz example above, a squadron that has several exception events, then stops having exceptional event hasn't started failing...they've stopped being exceptional.  The failure comes from a different reason.  The two could be related.

From my view, I'd see these are important numbers, but this is just a first pass at it so I could be missing something obvious.

retention percentage
recruiting
membership
attendence
"activity" - hours, dollars, number of events...some measure.

Tracking these trends should give you the feel of what is going on.  I suppose you could do secondary metrics off these, for example a retention:recruiting ratio.  A positive number there indicates growth, a negative indicates shrinkage.

[NIN]

People will shift their focus to meet the metrics, so you really need to make sure what you are measuring is what you want people to do.

BITD (a Thursday, it was), when the old CAP-MAP program was the hot ticket, my squadron commander did a pretty good job of "chasing the metrics."

Now, did we do these things because they were the proximate result of a good program? Not precisely.  We did these things because the commander wanted to be a high-scoring squadron in CAP-MAP, and as a consequence we did all kinds of cool things that counted toward CAP-MAP and only a few things that did not. 

I was asked to hold off on my Earhart so that it would be on the next Calendar Year and apply against the next year's CAP-MAP, since we already maxed our Cadet Awards for that year, for example. Several cadets got asked to do similar things with the Mitchell.

We flew enough orientation flights to get the maximum score on CAP-MAP. And nothing more.

Due to the points-chase, we were doing really well at program elements that were being tracked.

If it wasn't tracked, we probably were only doing it because it "came along with" something that was being tracked.

Seriously.

[RiverAux]

The Coast Guard Auxiliary attempts to track just about every member activity -- I think there may be a code for time spent in the bathroom at flotilla meetings.  In general hours spent on various activities are tracked as are actual accomplishments (i.e., the number of hours spent trying to do vessel safety checks and the number of actual checks performed).  I think most actual "missions" are generally included, however most members fail to report basics such as time spent at flotilla meetings. 

So, despite a massive infrastructure, including probably several 200-500 members whose job it is to enter this data), it isn't really used for a whole lot by most.  It looks good on reports to the CG to justify our existence, but thats about it. 

Unfortunately, useful HR stuff like retention isn't really tracked as efficiently.

[Private Investigator]

The Catch-22 for most units is they happy to maintain but they do not look at the people involved. I.E., Member numbers is 50 on 1/1/12 and 60 on 1/1/13 so they 'think' they have grown 20% but they do not think that 25 members have left and they recruited 35 new ones who will leave eventually. But they think they are doing good.

It is obvious a Unit is not doing well when their numbers shrink. Like I mention in another thread, "Failing Squadron" (or something similar. Most Units have been failing for a few years until they wake up and see only three active members. 

[lordmonar]

The key here.....is what do we mean by "doing well"?

What are our goals for the average cadet squadron?
What are our goals for the average Senior Squadron?

Chasing the numbers can be a good thing....if those number reflect what NHQ considers "doing well".
As NIN pointed out we need to be careful that we don't put artificial road blocks into the system....but we can't have any system until we have a definition of "doing well".

[Eclipse]

Another phenomena is unit CC's who are happy to scoop up the empty shirts from everyone else in an effort to artificially pad their numbers, or
worse, keep the charter open.

Both miss the point.

[Private Investigator]

^ Roger that.

Or the Unit CC who takes in the problems from other Units, the classic "EPIC FAIL" in progress deal

[SarDragon]

Another phenomena phenomenon is unit CC's who are happy to scoop up the empty shirts from everyone else in an effort to artificially pad their numbers, or worse, keep the charter open.

Both miss the point.

FTFY.

[Eclipse]

I was, referring to it happening more then once, of course.

[SarDragon]

I was, referring to it happening more then once, of course.

Give it up, Bob. It's a losing effort. 

[Eclipse]

Worth a try..

[bflynn]

The key here.....is what do we mean by "doing well"?

What are our goals for the average cadet squadron?
What are our goals for the average Senior Squadron?

Chasing the numbers can be a good thing....if those number reflect what NHQ considers "doing well".
As NIN pointed out we need to be careful that we don't put artificial road blocks into the system....but we can't have any system until we have a definition of "doing well".

This.  What is meant by doing well?

[Critical AOA]

Metrics help you quantify the problems and to possibly explain things to others outside of the unit but anyone inside a failing unit with anything resembling a pulse will know it.