Session 4: What to measure?

The focus now shifts to determining what to measure based on the output, outcome and impact identified in the logic model.  The concept of indicator takes prominence in this step.  Indicators determine exactly what you are measuring, collectively they create a picture of the project’s output, outcome or impact.  Each indicator provides a snapshot of the reality and can draw upon both quantitative and qualitative information.  Indicators must also be easy to understand, be sensitive to change and can be repeatedly measured to allow for tracking of changes over an extended period of time to help determine longer-term trends.

Here, there are two main choices, either developing your own set of indicators, or adopting an existing set of indicators and where appropriate, making minor modifications and/or select the ones that are suitable for the programme under evaluation.  Some examples of indicator set available in the field are offered at the end of the current session.  While searching and reviewing available indicator sets, their relevance to what is expected to change as a result of the programme activities (i.e. output, outcome, impact identified through the logic model) should be the main consideration.   If there is an indicator set that is more or less suitable, adopting or adapting it would be less time and resource intensive than putting together a new set of indicators.

Regardless of whether an existing set of indicators is selected or a new set is being developed, the involvement of relevant stakeholders and indicator testing should be treated with great importance.   Involving local communities helps researchers to select indicators that assess locally relevant sustainability issues, avoiding the accusation towards expert chosen indicators of failing to capture locally important factors (Fraser et al. 2006; Poveda and Lipsett 2014).  It is recommended that potential indicators are evaluated with the target participants and/or relevant stakeholder representatives, followed by testing the selected indicators (Reed et al., 2006).  Testing the indicators may involve determining what kind of data is needed and trying to collect such data to see if there are any challenges in doing so.  For example, if questionnaire surveys are necessary, then pilot surveys should be conducted.  These steps should be conducted before finalising on a set of appropriate and feasible indicators. 

 

 

Starting with the HSBC Rural Sustainability Programme, our centre developed a set of indicators based on the five core dimensions of sustainability.  This study adopts a hybrid model where the development of indicators and sub-indicators under each core dimension would take into account both generic and case/context specific considerations (Gibson 2006a).  Using the generic categories as a starting point and gradually elaborating through considering case/context specific scenario would allow for evolution as greater understanding is developed on the case/context as the programme progresses, while also ensuring that a basic yet comprehensive framework is used at the outset.

Moving from defining sustainability through the five core dimensions to developing the indicators under each core dimension involved extensive literature review.  For example, in order to operationalise Core Dimension 4 – Precaution and Adaptation through indicators and sub-indicators, Gibson and Hassan (2005), the City resilience index (2014) provided an important basis for developing our definition for this core dimension, and to expand on it, the body of scholarly work on environmental literacy was consulted.  Pre and early-programme engagement with a number of Start-up scheme participants was conducted during this process to develop the indicators and sub-indicators.  Both the literature and discussions with the participants were necessary to decide on ways to collect data for the indicators and sub-indicators (see Session 4 for further details).

 

 

The indicators and sub-indicators were designed to be able to assess the impact of the programme at different levels.  For example, at the programme-level, the indicator ‘innovative solutions trialed and implemented’ helps to communicate the programme output, or more specifically the Start-up scheme’s output.  At the individual participants’ level, the changes experienced relate to their knowledge and competence through completing the Start-up scheme which is assessed through indicators 4.3 and 4.4.  At the societal level, all the projects incubated through the Start-up scheme collectively contribute to climate change adaptation and mitigation. 

To clearly demonstrate the ways in which the indicators and sub-indicators are linked to the output, outcome and impact, please see the figure below.  The output, outcome and impact from ‘Logic model for the Rural in Action Start-up scheme’ are repeated below.  

 

 

The next session answers the question of ‘how to measure’ by focusing on the sub-indicators under Indicator 4.3 Disposition and awareness and 4.4. Sustainability Literacy.    
 

Resources

City Resilience Index (2014) City Resilience Framework. The Rockefeller Foundation. Available online: https: //assets.rockefellerfoundation.org/app/uploads/20150530121930/City-Resilience-Framework1.pdf (accessed on 21 March 2020)

DeGraaf, M., Buck, L., Shames, S., & Zagt, R. (2017) Assessing Landscape Governance: A Participatory Approach. Tropenbos International and EcoAgriculture Partners: Wageningen, The Netherlands.

Gibson, R. B. (2006a). Beyond The Pillars: Sustainability Assessment As A Framework For Effective Integration Of Social, Economic And Ecological Considerations In Significant Decision-Making. Journal of Environmental Assessment Policy and Management (JEAPM), 8(03), 259-280.

Gibson, R., & Hassan, S. (2005). Sustainability assessment: Criteria and processes. London; Sterling, VA: Earthscan.

Reed, Mark S., Evan D. G. Fraser, and Andrew J. Dougill. 2006. 'An adaptive learning process for developing and applying sustainability indicators with local communities', Ecological Economics, 59: 406-18.

Takahashi, Y., Schauffele, N., Mengrani, S., Mader, A., Scheyvens, H., Dasgupta, R., & Lopez-Casero, F. (2019). Achievements, challenges and ways forward for the Satoyama Development Mechanism: A self-assessment by the SDM Secretariat. Institute for Global Environmental Strategies and United Nations University Institute for the Advanced Study of Sustainability, Japan.

UNU-IAS, Bioversity International, IGES and UNDP (2014) Toolkit for the Indicators of Resilience in Socio-ecological Production Landscapes and Seascapes (SEPLS). 

 

Continue to “Session 5: How to measure?”