Blog: Is Developmental Evaluation the Right Measurement Strategy?
Monday, June 3, 2019
Note: The following post was originally published on NextBillion.net, which is managed by WDI.
In emerging economies, businesses and other organizations engage in work that can be really messy and staggeringly complex. They know it’s important to measure the social impact of their work, but they often struggle to find the right approach, spending money on measurement strategies that are designed for static environments. Like dieters looking for a quick weight loss solution, they try one approach or another, then grow frustrated when these strategies fail, delivering results that they won’t use – or worse, can’t use. After experimenting with evaluations that end up wasting their time and money, they might even decide that measuring their impact isn’t worth the effort.
These organizations may not realize that there’s a better approach, more geared toward the unique challenges they face. Called “developmental evaluation” – often just referred to as “DE” – it was created to support innovative programs that operate in complex environments, and are thus expected to adapt over time. Developed in response to the problem of under-utilized data and hit-or-miss evaluations, DE is different from traditional evaluation methods, such as Randomized Controlled Trials or pre/post-intervention tests. It’s methodologically agnostic and utilization-focused, employing data collection, analysis and feedback that are quick, ongoing and iterative. It can draw on many different evaluation methods to collect data, with the purpose of aiding decision-making – even when a program is still developing. It’s more similar to R&D in the product development process of private sector companies, because it provides program staff with data-based feedback in real time (or close to real time), thus facilitating a continuous development loop.
DE achieves this by having one (or more) Developmental Evaluators integrated into the program implementation team, on a full- or part-time basis. These Developmental Evaluators collect and analyze data, which helps the organization make modifications in program design and achieve targeted outcomes. They participate in team meetings, and document decisions, processes and dynamics, with the goal of helping staff make decisions based on the latest and most relevant data to their particular context.
To extend the weight loss analogy: From the perspective of evaluation participants, DE resembles not a fad diet, but a new evaluation lifestyle. It views measurement less as a temporary, occasional effort driven from outside, but rather as an integral part of an organization’s ongoing work. Let’s take a closer look at the approach, to explore how to make a DE successful – and to determine whether it would be a good fit for your organization.
A Healthy Measurement Routine
As any doctor will tell you, a healthy lifestyle generally includes frequent exercise. So let’s continue this analogy by comparing DE to a balanced workout: As it happens, DE includes many of the same essentials that a good, well-rounded workout would.
CHOOSE A GOOD PERSONAL TRAINER
DEs come equipped with their own personal trainer: the Developmental Evaluator. Like any good fitness coach, the Developmental Evaluator is there to personalize the evaluation methods and activities to your specific needs. Hiring a Developmental Evaluator can sometimes be the hardest part of a DE, because you want to choose someone who is well-qualified to lead you through the DE process, and who can help your team establish data-driven evaluation methods that you can actually use to inform strong program strategy. Top credentials to consider in a candidate include: experience in qualitative and quantitative methods, facilitation skills or a background in co-designed processes, and a proven ability to work in uncertain environments.
Just as it’s important to warm up before starting a strenuous workout, it’s essential to get warmed up before you jump into a DE. If you don’t, you might hurt the evaluation’s chances for success. To prepare for this process, DEs often include one or more activities, such as a kick-off workshop, to help teams understand what DE is and how it can be useful to them – including how to best leverage the role of the Developmental Evaluator.
As you kick off a DE, identify and target the areas where you think measurement will add immediate (and long-term) value. If you’re warming up for a 5K run, you would target your legs and not just limit yourself to upper-body stretches. Similarly, if you’re starting a DE that is focused on stakeholder engagement, you might start with a stakeholder mapping exercise as a quick win for your teams.
Like any effective workout regimen, a DE should include a balanced mix of activities. Some activities will be hard, heavy lifts that only need to be completed once or twice at the beginning or end of the DE (think: strength training). For instance, if you want to build strength in your DE, you need to work hard to fully integrate the Developmental Evaluator into your team. This is one of the first and most important weights you can lift in DE, and it may require a few heavy pushes. You’ll need to introduce the Developmental Evaluator to all DE stakeholders, set up standing meetings, get into the habit of cc’ing them on emails, and make space for them at your office. But once they are integrated, you’ll have built the strongest muscle you need to power through the DE.
Other activities and processes take more endurance and will need to be performed throughout the life of the DE to maintain their benefits (think: cardio). In these cases, perseverance and stamina will pay off, especially when it comes to building buy-in and stakeholder support for the DE. Because a team’s willingness to engage in a DE may change with each new evaluation question, or with the sharing of negative results, promoting buy-in for the DE takes both time and endurance – it is more a long distance run than a sprint. And even if you hit a nice stride early on, the level and pace of buy-in for the DE can change as you navigate new terrain, such as staff turnover, product launches or new leadership decisions. The Developmental Evaluator and DE stakeholders will need to foster a culture of open learning and adaptability throughout the life of the DE.
To properly end your DE workout, you’ll need to do some cool-down activities to ensure that you don’t harm your organization. Most importantly, when ending the DE, take the time to “de-integrate” the Developmental Evaluator from your team. This may include handing off the Evaluator’s responsibilities to permanent staff, ensuring that action plans are in place to carry DE recommendations forward, and preparing teams to continue their work with a focus on continuous learning – even without the added support of the Developmental Evaluator.
Last but not least, a healthy lifestyle requires good – and ongoing – nutrition habits. Likewise, if you want to get the most out of your DE, you have to continue with the healthy measurement habits it has established, even after the evaluation has ended. It’s important to pause and reflect regularly, to make sure your DE leads to data-driven decision-making and strategic thinking within your team in the long term. Schedule recurring meetings or workshops to help internalize the DE’s data and findings. This will ensure that your team has truly absorbed the learnings generated by the DE.
Diet or Lifestyle: Which Evaluation Approach Are You Choosing?
If you were to look at your organization’s current evaluation practices, would you say they fall more into the category of “fad diet” or “healthy lifestyle”? It’s an important question, and it’s essential to answer it through an honest reflection on your organization’s needs and capabilities.
After engaging in this self-assessment, you may decide that DE is not appropriate for your situation, organization or business – the approach is not for everyone. But it’s well worth considering, especially for programs operating in complex environments, or without an established theory of change. It can also be used alongside market-based approaches that embed the principles of social innovation and social enterprise into an organization.
DE could be just what you need to establish a healthy evaluation lifestyle: one that balances, in equal parts, data-driven evaluation methods and a strong program strategy.
Want to learn more about DE? Check out these resources – or connect with WDI’s Performance Measurement and Improvement teamto see if DE (or another area of continuous improvement) fits your needs:
- Developmental Evaluation in Practice: Lessons from Evaluating a Market-Based Employment Initiative: This document introduces the concept of DE and provides some tools to support its use.
- Developmental Evaluation in Practice: Tips, Tools, and Templates: This resource offers guidance for organizations, managers and evaluators that seek to implement DE.4
- 7 Hot Tips for Implementing a Developmental Evaluation: This webinar shares lessons learned, and showcases best practices in DE.
- Advancing the Use of Developmental Evaluation at USAID: This page shares information and a comprehensive list of project documents from the Developmental Evaluation Pilot Activity project that inspired this blog.
Author’s note: Almost five years ago, the United States Agency for International Development launched the Developmental Evaluation Pilot Activity (DEPA-MERL) under the U.S. Global Development Lab’s Monitoring, Evaluation, Research, and Learning Innovations program to test how developmental evaluation (DE) could be used in situations where traditional approaches to program evaluation were not a fit. This blog was inspired by findings from the three DEs conducted by the DEPA-MERL consortium from 2016-2019.
Yaquta Kanchwala Fatehi of the William Davidson Institute and David Yamron of Search for Common Ground contributed to this report.
Photo courtesy of renategranade0.