Metrics

[Walkman]

Agreed. I think the lesson is that metrics are a tool not the mission itself.

[FlyTiger77]

That said, goals need to be set forth, then you can determine what Key Performance Indicators (KPIs) can be used to measure progress towards that goal.

Let me re-phrase the original question: Does anyone have a comprehenisive list of goals and corresponding metrics for (a) squadron(s)?



Please assume that I am fairly competent at managing an organization of a few hundred people and that I have thought through the ramifications of instituting a system of metrics.

[41839j]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

[mwewing]

I have had this same conversation with several members in the last few weeks. Setting a vision with SMART goals, and establishing KPIs to measure progress is an essential part of leading a team toward a long term vision. Unfortunately, I don't see much of an appetite for this within CAP. There are vision statements and plans that pop up at various levels, but almost none of it seems to be linked together in any meaningful way. IMO this process works best when senior leadership establishes a vision for an organization, and subordinate levels set SMART goals within their content area. This keeps consistency throughout a large organization, and hopefully manages the flow of resources in support of the goals and vision. That said, I think this type of long term planning can be done even at the local level with success. A unit can create a vision statement, set goals, and establish KPIs with or without guidance from higher command. In the absence of strong/consistent direction from more senior leadership, a unit commander can create a unit specific vision. It will be limited by local factors and available resources, but can still be effective in leading a unit toward desired outcomes.

There are glimmers of this type of process throughout CAP. They come by way of annual plans and reports required in some duty assignments/specialty tracks. I have also seen some commanders at various levels institute a vision or set goals in various ways. If more members begin working in this way, it could become the standard. While I don't have a system of metrics completed at this time, I hope to continue working in this direction within my areas of responsibility. I will gladly share my finished product with anyone when complete, and I would be very interested in seeing what others have created as well.

[Eclipse]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

You can't spend time answering micro questions for a macro problem.

The above is why units never get out of their groove of status quo, never grow, and frequently die because one person leaves the program.
Looking at individual pieces of the program, or individual successes,  and treating them as a systemic success is that allows poor unit CC's to sleep at night thinking they are doing their job.

We have one unit inn our wing which has been on life support for years and years, yet when you visit them for things like SUIs or SAVs, they still bring up the Spaatz cadet that left the program
over 12 years ago as if it was relevent today.

Similarly, we had a unit that was essentially a Spaatz factory - several in as many years - but it was all coincidental and circumstantial to the participants, and most
of the leadership were cadet parents.  When the cadets left, so did they, and the unit went from a top-player to almost shutting down in a single year.

One of the most important SMART goals, or at least questions to build a SMART goal, is "Are we growing?"

Because if you aren't, then by design, you are dying, since, at least on the cadet side, your "product" has a finite expiration date.

In my opinion, the primary reason that the organization is not interested in objective criteria and simple answers to direct questions is that the facts would make people sad.

[mwewing]

I think you are making this more complicated than it really is.

CAP has three missions.  To what extent are these missions being fulfilled by any given squadron?  Are cadets learning about aerospace and how successful are they after moving on?  Does the air force rely on you for anything at all or even ever request anything from you through wing?  Do you ever get called for missions?  If the answer to any of these is no, then you have to find out WHY and do something about it.

This is what the conversation is about. We all understand the basic goal of CAP is to succeed in our 3 missions, but what does success look like? Without a clear vision, SMART goals, and measured progress toward desired outcomes, we can't really determine our success. Long term planning and goal setting, establishes a measurable standard for success. Using metrics to measure KPIs, gives a commander needed data to determine progress, and evaluate future goals and planning efforts.

For example, you state that success in Aerospace education is that cadets learn about aerospace. We know that AE tests are required to progress through most achievements, and units should have regular AE lessons and activities. You can appoint me as AEO and I can teach 1 lesson a month where I read chapters from the AE textbook. The cadets will be bored, won't retain information, and won't be any better prepared for their achievement tests. Strictly speaking, I conducted my sessions... they learned about AE and I was "successful."

Now, I know my example is a bit extreme (hopefully), but it does articulate the need for goals and metrics to measure progress. What does success look like for unit level AE?

• Goal: Cadets will perform better in achievement AE testing; KPI: Cadet AE testing scores will average 90%
• Goal: Squadron will place emphasis on Model Rocketry; KPI: 80% of cadets will complete the Model Rocketry Program within the year
• Goal: Squadron will conduct the AEX program; KPI: The squadron completed the program within the year

These are just some basic MEASURABLE goals that a unit can set based on their local priorities within AE, or based on the direction of Group, Wing, Region, or National AE plans. Of course, reaching these outcomes will require programs and lessons that support the established goals. This process may seem "more complicated" but in reality we are directing our efforts in more productive ways.

[41839j]

Well that is exactly my point.  We are on the same page here.  ANY and EVERY unit actually knows how they are doing if they are honest enough to look objectively at it and doing so will give them the answers that they need.  Not much that anyone else or national can do. 

This organization is like any other volunteer organization is as much as you get out of it what you are willing to put into it.  Now, do we do things together as a unit, or do we only do what we want to do and expect others to pick up the slack?

[jimmydeanno]

There are numerous books and articles that talk about metrics and how they impact your work environment and mission success. 

Here is a good one, I think: http://martinfowler.com/articles/useOfMetrics.html

People will shift their focus to meet the metrics, so you really need to make sure what you are measuring is what you want people to do. 

[bflynn]

My question above was intended to make sure you thought it through - I have seem this done wrong too many times not to react to it.

I think from the lack of response, I'm hearing that nobody here has any metrics.

I also think that getting them can be tricky - using the Spaatz example above, a squadron that has several exception events, then stops having exceptional event hasn't started failing...they've stopped being exceptional.  The failure comes from a different reason.  The two could be related.

From my view, I'd see these are important numbers, but this is just a first pass at it so I could be missing something obvious.

retention percentage
recruiting
membership
attendence
"activity" - hours, dollars, number of events...some measure.

Tracking these trends should give you the feel of what is going on.  I suppose you could do secondary metrics off these, for example a retention:recruiting ratio.  A positive number there indicates growth, a negative indicates shrinkage.

[NIN]

People will shift their focus to meet the metrics, so you really need to make sure what you are measuring is what you want people to do.

BITD (a Thursday, it was), when the old CAP-MAP program was the hot ticket, my squadron commander did a pretty good job of "chasing the metrics."

Now, did we do these things because they were the proximate result of a good program? Not precisely.  We did these things because the commander wanted to be a high-scoring squadron in CAP-MAP, and as a consequence we did all kinds of cool things that counted toward CAP-MAP and only a few things that did not. 

I was asked to hold off on my Earhart so that it would be on the next Calendar Year and apply against the next year's CAP-MAP, since we already maxed our Cadet Awards for that year, for example. Several cadets got asked to do similar things with the Mitchell.

We flew enough orientation flights to get the maximum score on CAP-MAP. And nothing more.

Due to the points-chase, we were doing really well at program elements that were being tracked.

If it wasn't tracked, we probably were only doing it because it "came along with" something that was being tracked.

Seriously.

[RiverAux]

The Coast Guard Auxiliary attempts to track just about every member activity -- I think there may be a code for time spent in the bathroom at flotilla meetings.  In general hours spent on various activities are tracked as are actual accomplishments (i.e., the number of hours spent trying to do vessel safety checks and the number of actual checks performed).  I think most actual "missions" are generally included, however most members fail to report basics such as time spent at flotilla meetings. 

So, despite a massive infrastructure, including probably several 200-500 members whose job it is to enter this data), it isn't really used for a whole lot by most.  It looks good on reports to the CG to justify our existence, but thats about it. 

Unfortunately, useful HR stuff like retention isn't really tracked as efficiently.

[Private Investigator]

The Catch-22 for most units is they happy to maintain but they do not look at the people involved. I.E., Member numbers is 50 on 1/1/12 and 60 on 1/1/13 so they 'think' they have grown 20% but they do not think that 25 members have left and they recruited 35 new ones who will leave eventually. But they think they are doing good.

It is obvious a Unit is not doing well when their numbers shrink. Like I mention in another thread, "Failing Squadron" (or something similar. Most Units have been failing for a few years until they wake up and see only three active members. 

[lordmonar]

The key here.....is what do we mean by "doing well"?

What are our goals for the average cadet squadron?
What are our goals for the average Senior Squadron?

Chasing the numbers can be a good thing....if those number reflect what NHQ considers "doing well".
As NIN pointed out we need to be careful that we don't put artificial road blocks into the system....but we can't have any system until we have a definition of "doing well".

[Eclipse]

Another phenomena is unit CC's who are happy to scoop up the empty shirts from everyone else in an effort to artificially pad their numbers, or
worse, keep the charter open.

Both miss the point.

[Private Investigator]

^ Roger that.

Or the Unit CC who takes in the problems from other Units, the classic "EPIC FAIL" in progress deal

[SarDragon]

Another phenomena phenomenon is unit CC's who are happy to scoop up the empty shirts from everyone else in an effort to artificially pad their numbers, or worse, keep the charter open.

Both miss the point.

FTFY.

[Eclipse]

I was, referring to it happening more then once, of course.

[SarDragon]

I was, referring to it happening more then once, of course.

Give it up, Bob. It's a losing effort. 

[Eclipse]

Worth a try..

[bflynn]

The key here.....is what do we mean by "doing well"?

What are our goals for the average cadet squadron?
What are our goals for the average Senior Squadron?

Chasing the numbers can be a good thing....if those number reflect what NHQ considers "doing well".
As NIN pointed out we need to be careful that we don't put artificial road blocks into the system....but we can't have any system until we have a definition of "doing well".

This.  What is meant by doing well?