Home » Session 5: How to measure?
Session 5: How to measure?
Once the indicators and sub-indicators are selected, the subsequent task is to find appropriate ways to collect data. When considering the type of data to be collected, both quantitative and qualitative data could be useful, as well as documentary record compiled throughout the programme. Some available indicator sets are accompanied by suggestion of methodologies for data collection and analysis (e.g. see UNU-IAS, Bioversity International, IGES and UNDP, 2014). Potential users will find these methodologies useful when these sets of indicators match their expected outcome and impact. Alternatively, the literature that is consulted during the indicator development stage is also important for identifying ways to measure it, in other words, what types of data to collect and how to do so.
For example, when incubating start-up groups, a potential impact considered is that they would not only gain knowledge and skills but could potentially develop the competence to become change agents or sustainability leaders in the society. To find a way to measure related changes, a literature review was conducted which led to the discovery of frameworks on environmental literacy. These frameworks were developed based on the North American Association for Environmental Education (NAAEE) framework, which comprises of three domains - knowledge, disposition and competencies (Hollweg et al., 2011). Scholars have been developing survey questions on environmental literacy (King and Franzen, 2017) or energy literacy (DeWaters and Powers, 2013) based on the NAAEE framework. For our Programme, the survey questions were modified to measure sustainability literacy. Simply put, the survey questions were divided into three parts, based on the NAAEE framework, assessing knowledge, affective/disposition and competence/action of the respondent in relation to sustainability. The following table summarizes key characteristics of sustainability change agents when assessing awareness and potential action:
Though receiving similar amount of funding and support from our Centre, the groups have all undergone a rather different experience in establishing their start-ups/initiatives. In addition, since the group of start-up participants is small, it was more effective to conduct interviews. The discussion allows us to delve into their experiences throughout the scheme and explore the extent to which changes in their sustainability literacy could be attributed to the scheme. It is surrounding the dimensions of sustainability literacy that the interview protocol was developed.
Interviews with Start-up scheme participants were then transcribed and coded based on the relevant sub-indicators, in this case, those under Indicator 4.3 Disposition and awareness and 4.4. Sustainability Literacy. Reviewing all the coded data helps to identify commonalities between several start-up groups. Some of these findings are summarised in the table below.
In general, common methods of data collection would involve focus group meetings, interviews, questionnaire surveys and programme records. Focus group discussion can be particularly useful in the earlier stage when stakeholders are engaged to develop or refine the logic model and in indicator selection. Key informant interviews held with programme participants, representatives of relevant stakeholder groups, or people in key positions in a local community are exceptionally important in gaining a deeper understanding of changes experienced by individuals, stakeholder groups or the community as a whole, ensuring greater participation and multiple perspectives. Where there is a significant number of participants undergoing similar interventions, it could be worthwhile to consider developing a participant survey or adapting available ones from the literature to help measure the attainment of the project from the participants’ perspective in a collective manner.
Resources
DeWaters, J., & Powers, S. (2013). Establishing measurement criteria for an energy literacy questionnaire. The Journal of Environmental Education, 44(1), 38-55.
Hollweg, K. S., Taylor, J. R., Bybee, R. W., Marcinkowski, T. J., McBeth, W. C., & Zoido, P. (2011). Developing a framework for assessing environmental literacy. Washington, DC: North American Association for Environmental Education. Available at http://www.naaee.net.
King, J., & Franzen, R. (2017) Environmental Literacy in Environmentally Themed Higher Education Courses. Journal of Sustainability Education, Vol 13, March Issue http://www.jsedimensions.org/wordpress/content/environmental-literacy-in-environmentally-themed-higher-education-courses_2017_03/