Boys & Girls Club of America
Boys & Girls Clubs of America survey over 150,000 young people in 2700 clubs.
BGCA came to IRRE and our partner Youth Development Strategies, Inc. (YDSI) in 2014; they asked if we could help look at their annual survey results differently. They wanted to have the results more powerfully inform efforts to improve their practices and to communicate to external audiences what difference club experiences make to young people. We went to work on finding ways to get BGCA credible results that would be more meaningful and actionable.
Our work with BGCA has allowed individual clubs, regional networks of clubs and the national research and program offices to know how many and what percentage of their members are having positive and less positive club experiences in domains relevant to several key outcomes BGCA seeks to strengthen with their efforts.
To make the results more meaningful, we conducted analyses to find a small number outcomes that were potentially movable by members’ club experience. We identified survey items and created metrics that tapped “Healthy Living”, “Character and Citizenship” and “School Performance and Commitment”. We then identified survey items asking about experiences at the club that developmental and educational research indicate should predict these three outcomes. We created metrics for three “meaningful” club experiences: “Adult Relationships and Safety”, “Sense of Belonging” and “Fun” as well as a metric for “Overall Club Experience”.
To make results more actionable, we applied IRRE’s threshold-based, configural scoring to items used for these metrics. We set cut points on these items that indicate where members’ responses were Optimal (reflecting desirable responses) and Risk (reflecting undesirable responses). This approach allows us and BGCA to see how many members scored at these levels.
This threshold-based scoring provides much clearer, interpretable information than do item and sub-scale means/averages (where there is no absolute reference point only a sense of where the middle of the pack is); or, normative statistics like “top 10% of clubs” (where no absolute reference point is provided, just who is winning and losing).
With our configural scoring methods, we then combine items into higher order component metrics and then component metrics into the final outcome and club experience metrics. BGCA can now ask deeper questions of the survey results and design more specific actions in response to the answers. For example, “we see our club has a low percentage of members with strong levels of Adult Relationships and Safety. Do these results hold both for Safety and for Adult Relationships? And within Adult Relationships are the findings the same for Adult Expectations and for Adult Caring and Support”?
The configural scoring method also allows BGCA to know that component measures of club experience with stronger relations to the three outcome measures were weighted more heavily when components are combined into the “Overall Club Experience” metric and that when there are either particularly risky or optimal combinations of components these will be reflected in the scoring as well.
IRRE and YDSI are now involved in helping BGCA develop metrics to assess the impacts of their new programmatic emphases on older members’ college readiness and interest in the areas of Science, Technology, Engineering and Math (STEM).