How to evaluate methods?

To effectively evaluate methods within the context of planning and conducting an evaluation, you need to follow a structured approach that considers the purpose, context, and available resources. This involves more than just picking data collection tools; it's about strategically selecting and combining processes for every step of the evaluation.

Here's how to evaluate methods based on established principles:

Steps for Selecting and Applying Methods in Evaluation

Choosing the right methods is crucial for ensuring your evaluation provides credible and useful information. The process involves careful consideration of your goals and circumstances.

1. Choose Methods or Processes for Every Task

It's not enough to just select methods for data collection (like surveys or interviews). You need to choose methods or processes for every task in the evaluation. This includes methods for:

• Planning: How will you define the scope, stakeholders, and questions? (e.g., logic modeling, stakeholder workshops)
• Data Collection: How will you gather information? (e.g., surveys, interviews, focus groups, observations, document review)
• Data Analysis: How will you make sense of the collected data? (e.g., statistical analysis, thematic analysis, case studies)
• Reporting & Dissemination: How will you share findings? (e.g., written reports, presentations, infographics, workshops)
• Management & Quality Assurance: How will you ensure the process is efficient and rigorous? (e.g., project management tools, peer review processes)

Practical Tip: Create a detailed evaluation plan that maps out each task and the specific method chosen to accomplish it.

2. Analyse the Types of Key Evaluation Questions (KEQs) You Want to Answer

Your evaluation methods must directly align with the types of Key Evaluation Questions (KEQ) you want to answer. Different questions require different types of evidence and thus different methods.

• Implementation Questions: How was the program delivered? What happened? (Often require process documentation, observations, interviews)
• Outcome/Impact Questions: What results did the program achieve? What difference did it make? (Often require quantitative data analysis, comparisons, qualitative inquiry into experiences)
• Efficiency Questions: Was the program cost-effective? (Often require cost-benefit analysis, resource tracking)
• Relevance Questions: Was the program appropriate for the need? (Often require needs assessment data, stakeholder perspectives)

Example: If your KEQ is "Did the training program improve participants' skills?", you'll need methods that measure skill change, such as pre/post-tests or performance assessments. If your KEQ is "How did participants experience the training?", you'll need qualitative methods like interviews or focus groups.

3. Consider Your Particular Situation

The ideal methods in theory might not be feasible in practice. You must consider your particular situation, taking into account:

• Available Resources: What is your budget, staff capacity, and timeline?
• Context: What is the political, social, and organizational environment? Are there sensitivities or access issues?
• Available Data: Is there existing data you can use?
• Stakeholders: Who needs the information? What methods will be credible to them? What are their preferences or literacy levels regarding different data types?
• Ethical Considerations: Are the methods appropriate and ethical given the participants?

Practical Tip: Assess your constraints early in the planning phase. A complex experimental design might be ideal scientifically but impossible with limited time or budget.

4. Review the Advice Provided for Each Method

Before committing to a method, it's essential to review the advice provided for each method. This means understanding:

• Strengths and Weaknesses: What kind of information does the method provide well? What are its limitations?
• Appropriate Use: When is this method best applied?
• Requirements: What skills, resources, or conditions are needed to implement it correctly?
• Potential Biases: What are the risks of skewed results?

Example: Surveys can collect data from many people but rely on self-report and may have low response rates. Focus groups provide rich interaction data but are challenging to analyze and susceptible to groupthink.

5. Aim to Use a Complementary Mix of Methods

Relying on a single method can give a narrow perspective. A more robust approach is to aim to use a complementary mix of methods. This is often referred to as triangulation.

• Triangulation: Using multiple data sources, methods, theories, or evaluators to cross-check findings and increase confidence in conclusions.
• Mixed Methods: Combining qualitative and quantitative approaches to gain a more comprehensive understanding. Qualitative data can help explain quantitative findings, and vice-versa.

Practical Insight: Collecting both survey data (quantitative) on program participation rates and conducting interviews (qualitative) to understand why people participated (or didn't) provides a much richer picture than either method alone.

By systematically following these steps, you can effectively evaluate and select the most appropriate methods to ensure your evaluation is rigorous, relevant, and useful.

Summary of Method Selection Considerations:

Further Reading: You can explore various specific evaluation methods and tools on platforms like BetterEvaluation.