With generous support from

The Professionalizing Field of Financial Counseling and Coaching Journal

ACCOUNTABILITY

THE FOUR KEY PILLARS OF PROFESSIONALIZATION

QUALITY:

Training, professional development, and certification for practitioners and programs.

COMMUNITY:

Local, state, and national stakeholder networks that support and develop practitioner efforts.

ACCOUNTABILITY:

Standardized outcome definitions and data collection to understand program impact.

CONSISTENCY:

Service delivery models and the underlying data systems that support them.

Standardizing Outcome Measures in a Client-Centered Field

J. Michael Collins is faculty director of the Center for Financial Security at the University of Wisconsin, Madison. He is an Associate Professor at the La Follette School of Public Affairs and at the School of Human Ecology. He is also a family economics specialist for UW-Extension, Cooperative Extension, and an affiliate of the Institute for Research on Poverty and Center for Demography and Ecology.

Hallie Lienhardt is the Outreach Specialist for the Center for Financial Security at the University of Wisconsin- Madison where she focuses on research dissemination and management of the Center’s financial coaching projects.

Regina Salliey is a Program Associate in the Center for Community and Economic Opportunity at the Annie E. Casey Foundation. She is responsible for leading the Foundation’s financial coaching portfolio, as a financial capability strategy that moves families towards economic well-being.

Annika K. Little serves as deputy director for the Asset Funders Network, a community of foundations and grantmakers advancing programs, policies, and strategies that promote poverty alleviation, economic opportunity, and economic security.  As deputy director, Annika develops and guides AFN’s programming strategy, strategic partnerships and special projects.

>

NEXT

<

PREV

 

Financial coaching is an approach to build financial capability and goal-focused financial behaviors for people across all income and wealth levels.  Growing bodies of research point to the coaching approach’s effectiveness in helping people make positive behavior changes in their financial lives.  Yet, drawing broad insights about the effects of coaching and comparing outcomes across studies is difficult as financial coaching programs collect and calculate client outcome measures in a wide variety of ways. Reflecting the field’s lack of standards of practice, the development of more robust measures for tracking client outcomes has not kept pace with the rapid growth that the financial coaching field has experienced in recent years.  Adopting a uniform set of measures will allow the field to demonstrate the power of the coaching model and is fundamental to other critical tasks like standardizing practice.  Harnessing the current enthusiasm for coaching through implementation of shared measures and standards is key to creating a fact-based case for broader support from policymakers, funders, and coaching practitioners.

 

A standardized set of measures collected at client intake and tracked over time would benefit the financial coaching field in four powerful ways.  First, it is often difficult for individual coaching programs to amass the amount of data needed for rigorous program assessment that adheres to strict standards and yields accurate and dependable data.  By aggregating data from multiple programs, the field can work collectively to demonstrate effectiveness and scale in a way that is not possible otherwise.  Second, the coaching field encompasses a wide variety of approaches and models.  Utilizing a common set of measures facilitates studies that compare outcomes within and between programs and target populations. These comparisons will unlock insights into which types of coaching models work best in various contexts and with various types of clients.  These insights will be invaluable as organizations move to develop and implement tailored financial coaching programs to meet the diverse needs of communities.

 

A third area that stands to gain from adoption of standardized metrics is that of consistency of data collection. It is critical that standardized measures are statistically reliable and valid, a feat that is difficult to accomplish without coordination.  While the collection of administrative and financial data over time is vital to capturing changes longitudinally, it is difficult to consistently record these data due to a variety of challenges, such as staff turnover.  Standardized measures that can be easily administered in a uniform way would relieve the burden that individual organizations face in investing in and developing measures.  And finally, the adoption of standardized measures has the potential to decrease the amount of resources dedicated to collecting client data and greatly improving efficiency.  Many organizations collect dozens of measures on individual clients; agreement on a standardized set of measures could allow organizations to reduce the number of data points they track.

 

Agreement and adoption of a field-wide set of standardized measures is not without challenges. The belief that we need a “perfect” measure before trying one is an area that can hinder progress. Acknowledging that there will never be a set of measures that is without flaw, yet appreciating that we should not let this prevent the field from making much needed strides in the area of outcome measurement, is a necessary compromise to move forward.

 

Another obstacle lies in implementation of another set of questions to assess clients.  Balancing strategies for collecting data is an art that has not yet been perfected in the field of financial capability. While it is a goal and projected benefit that standardized measures will help to eliminate unneeded data and streamline data collection, the initial addition of another client evaluation will no doubt cause many organizations to hesitate at the idea of adding to existing data collection practices. However, instituting a standardized measure across the field is an excellent step to creating a much simpler, yet focused data collection process for the long run.

 

A final hurdle to cross in the name of standardization and consistency is simply the decision of choosing one set of measures over another.  Unfortunately, a decision of this nature could incite rivalry over which set of measures is used. This is an unfortunate roadblock to progress that should not be the demise of cooperation in this area. The agreed-upon measures should be a starting place, a compliment to existing measurements, and a tool that is constantly open to improvement and refinements.

 

The Center for Financial Security’s (CFS) early efforts to evaluate financial coaching revealed a need for more consistent tracking of clients’ financial behavior, ultimately leading CFS to launch an effort to develop new outcome measures in 2011.  The mission of the project was to develop and test a small set of standardized measures. The project aimed to meet rigorous social science standards while taking into account the practical issues of data collection and analysis in a field setting.  For 18 months, four nonprofit community-based organizations collected data on a proposed set of client outcomes and shared the results to form a consistent database of measures.  The project resulted in the six-question, eight-point Financial Capability Scale (FCS).

 

The FCS proved practical to implement, but just as importantly, also held up in a variety of statistical tests. Organizations in the FCS pilot continued to collect a variety of other measures, often including data from credit reports, bank accounts, or workforce development records.  Thus, the FCS data could be compared to external sources of information, a process that tested the validity of the FCS—that is, the extent to which the set of questions accurately captured clients’ financial status. Among other findings, the FCS proved to be a relatively valid proxy for credit status, as it appears to be correlated with credit scores and payment delinquency rates.  Because the partner organizations also collected the FCS on the same clients over time, the scale’s reliability could also be tested. Reliability measures the extent to which an individual is consistent in his or her responses to the questions, along with the level of random response or “noise” that would make the scale volatile and difficult to use.  In addition, the FCS was tested for internal reliability. Internal reliability is a widely used test in scale development that ensures that each question included in the scale provides unique value and derives distinct information from the other questions.  This internal reliability test helped focus the FCS to the smallest number of questions, with each of the six questions gleaning a different aspect of financial capability. Overall, the FCS is a starting place for a growing field, a ready-to-use tool that complements existing measurements.

 

As the practice of financial coaching continues to grow, standardization is increasingly important for identifying and replicating effective coaching practices.  Through the development of the FCS and ongoing financial coaching research in partnership with The Annie E. Casey Foundation and the Asset Funders Network, CFS has come to understand that the field has reached the critical mass needed to begin tracking and measuring outcomes in a more unified way.  Numerous organizations are building, developing, and refining their financial coaching programs, making the field uniquely positioned to integrate and implement common measures.  Nonprofit practitioners, policy makers, funders and the research community all have an important role to play in driving standardization and effective replication of financial coaching.   By coalescing around a set of measures that organizations can readily integrate into their existing client tracking systems, whether that be the FCS or an alternative, stakeholders will be better equipped to understand and improve the outcomes and impact of financial coaching services.

 

OTHER ESSAYS ON ACCOUNTABILITY:

Standardizing Outcome Measures in a Client-Centered Field

 

The authors make the case for standardized outcome measures.

Hallie Lienhardt

Outreach Specialist, Center for Financial Security at the University of Wisconsin - Madison

Regina Salliey

Program Associate, Center for Community and Economic Opportunity at the Annie E. Casey Foundation

J. Michael Collins

Faculty Director, Center for Financial Security at the University of Wisconsin - Madison

Annika Little

Deputy Director, Asset Funders Network

<back to CFEFund.org

Financial coaching is an approach to build financial capability and goal-focused financial behaviors for people across all income and wealth levels.  Growing bodies of research point to the coaching approach’s effectiveness in helping people make positive behavior changes in their financial lives.  Yet, drawing broad insights about the effects of coaching and comparing outcomes across studies is difficult as financial coaching programs collect and calculate client outcome measures in a wide variety of ways. Reflecting the field’s lack of standards of practice, the development of more robust measures for tracking client outcomes has not kept pace with the rapid growth that the financial coaching field has experienced in recent years.  Adopting a uniform set of measures will allow the field to demonstrate the power of the coaching model and is fundamental to other critical tasks like standardizing practice.  Harnessing the current enthusiasm for coaching through implementation of shared measures and standards is key to creating a fact-based case for broader support from policymakers, funders, and coaching practitioners.

 

A standardized set of measures collected at client intake and tracked over time would benefit the financial coaching field in four powerful ways.  First, it is often difficult for individual coaching programs to amass the amount of data needed for rigorous program assessment that adheres to strict standards and yields accurate and dependable data.  By aggregating data from multiple programs, the field can work collectively to demonstrate effectiveness and scale in a way that is not possible otherwise.  Second, the coaching field encompasses a wide variety of approaches and models.  Utilizing a common set of measures facilitates studies that compare outcomes within and between programs and target populations. These comparisons will unlock insights into which types of coaching models work best in various contexts and with various types of clients.  These insights will be invaluable as organizations move to develop and implement tailored financial coaching programs to meet the diverse needs of communities.

 

A third area that stands to gain from adoption of standardized metrics is that of consistency of data collection. It is critical that standardized measures are statistically reliable and valid, a feat that is difficult to accomplish without coordination.  While the collection of administrative and financial data over time is vital to capturing changes longitudinally, it is difficult to consistently record these data due to a variety of challenges, such as staff turnover.  Standardized measures that can be easily administered in a uniform way would relieve the burden that individual organizations face in investing in and developing measures.  And finally, the adoption of standardized measures has the potential to decrease the amount of resources dedicated to collecting client data and greatly improving efficiency.  Many organizations collect dozens of measures on individual clients; agreement on a standardized set of measures could allow organizations to reduce the number of data points they track.

 

Agreement and adoption of a field-wide set of standardized measures is not without challenges. The belief that we need a “perfect” measure before trying one is an area that can hinder progress. Acknowledging that there will never be a set of measures that is without flaw, yet appreciating that we should not let this prevent the field from making much needed strides in the area of outcome measurement, is a necessary compromise to move forward.

 

Another obstacle lies in implementation of another set of questions to assess clients.  Balancing strategies for collecting data is an art that has not yet been perfected in the field of financial capability. While it is a goal and projected benefit that standardized measures will help to eliminate unneeded data and streamline data collection, the initial addition of another client evaluation will no doubt cause many organizations to hesitate at the idea of adding to existing data collection practices. However, instituting a standardized measure across the field is an excellent step to creating a much simpler, yet focused data collection process for the long run.

 

A final hurdle to cross in the name of standardization and consistency is simply the decision of choosing one set of measures over another.  Unfortunately, a decision of this nature could incite rivalry over which set of measures is used. This is an unfortunate roadblock to progress that should not be the demise of cooperation in this area. The agreed-upon measures should be a starting place, a compliment to existing measurements, and a tool that is constantly open to improvement and refinements.

 

The Center for Financial Security’s (CFS) early efforts to evaluate financial coaching revealed a need for more consistent tracking of clients’ financial behavior, ultimately leading CFS to launch an effort to develop new outcome measures in 2011.  The mission of the project was to develop and test a small set of standardized measures. The project aimed to meet rigorous social science standards while taking into account the practical issues of data collection and analysis in a field setting.  For 18 months, four nonprofit community-based organizations collected data on a proposed set of client outcomes and shared the results to form a consistent database of measures.  The project resulted in the six-question, eight-point Financial Capability Scale (FCS).

 

The FCS proved practical to implement, but just as importantly, also held up in a variety of statistical tests. Organizations in the FCS pilot continued to collect a variety of other measures, often including data from credit reports, bank accounts, or workforce development records.  Thus, the FCS data could be compared to external sources of information, a process that tested the validity of the FCS—that is, the extent to which the set of questions accurately captured clients’ financial status. Among other findings, the FCS proved to be a relatively valid proxy for credit status, as it appears to be correlated with credit scores and payment delinquency rates.  Because the partner organizations also collected the FCS on the same clients over time, the scale’s reliability could also be tested. Reliability measures the extent to which an individual is consistent in his or her responses to the questions, along with the level of random response or “noise” that would make the scale volatile and difficult to use.  In addition, the FCS was tested for internal reliability. Internal reliability is a widely used test in scale development that ensures that each question included in the scale provides unique value and derives distinct information from the other questions.  This internal reliability test helped focus the FCS to the smallest number of questions, with each of the six questions gleaning a different aspect of financial capability. Overall, the FCS is a starting place for a growing field, a ready-to-use tool that complements existing measurements.

 

As the practice of financial coaching continues to grow, standardization is increasingly important for identifying and replicating effective coaching practices.  Through the development of the FCS and ongoing financial coaching research in partnership with The Annie E. Casey Foundation and the Asset Funders Network, CFS has come to understand that the field has reached the critical mass needed to begin tracking and measuring outcomes in a more unified way.  Numerous organizations are building, developing, and refining their financial coaching programs, making the field uniquely positioned to integrate and implement common measures.  Nonprofit practitioners, policy makers, funders and the research community all have an important role to play in driving standardization and effective replication of financial coaching.   By coalescing around a set of measures that organizations can readily integrate into their existing client tracking systems, whether that be the FCS or an alternative, stakeholders will be better equipped to understand and improve the outcomes and impact of financial coaching services.

44 Wall Street, Suite 605     New York, NY 10005     646.362.1645 phone     646.590.8743 fax

44 Wall Street, Suite 605, New York, NY 10005
646.362.1645 phone   646.590.8743 fax