It is sometimes difficult to decide what data and/or methods should be used for M&E. There seems to be so many perspectives and issues to consider. I’ve found the following 4 points helpful to think through the range of issues, and as the basis seeking consensus among stakeholders:
- Reliability: Do the questions have the same meaning to the same respondents at different times?
- Validity: Do the answers tell us what we want to know? Do the questions have the same meaning to different respondents at the same time?
- Scalability: Can comparative analysis of the findings be carried out at all the required levels of dis-aggregation?
- Affordability: Can the method be implemented for reasonable cost?
The issues of reliability and validity are perennial challenges for social researchers dealing with human perceptions and qualitative issues. One way to ensure reliable and valid findings is to use established questions/instruments that have been piloted/tested in the field.
The issue of scalability involves a tension between generic and context-specific instruments. Context-specific instruments are more likely to yield meaningful information that is sensitive to the nuances of a particular situation and/or the changes fostered by a particular project. Generic instruments, while potentially insensitive to nuance, make comparison across different geographic areas and/or projects possible. Scalable data is more likely a product generic survey instruments.
The issue of affordability concerns the pragmatic reality that the capture, analysis and dissemination of M&E data has a cost that implementing organisations must manage diligently.
Different stakeholders tend to place different emphasis on these 4 factors. For example, for some ‘purests’ validity is the key concern; while for some ‘pragmitists’, the affordability/cost-effectiveness is the key concern.
I’ve found that laying out the inherent tensions within each of the 4 factors helps to stimulate discussion and lead to consensus. It sometimes helps to rate all the propose M&E data/methods as high/low against each of the 4 points of consideration.