Assessing the Impact of Design

Welcome to The Design Loft #39.

In today’s issue, we are going to talk about impact and how we know we’re making it (or if we are at all). That means we are talking about something called evaluation.

Evaluation is the means for understanding what we are doing, what our efforts are producing, and the overall influence and impact of that work is on the world around us.

We make things because we seek to create positive influence in the world, but how do we know if this is true? How do we improve upon what we do and where does that input come from?

The answers to this are: evaluation.

Let me start with the only evaluation “joke” I have, but one I tell often:

Evaluation is the longest four-letter word in the English language.

Most people bristle at the term evaluation.

Evaluation often evokes memories of grade school report cards, university exams, performance reviews at work, or the feedback you get from your art instructor. Yet, evaluation is a vehicle for innovation and learning. It is the mechanism for providing and receiving feedback on what we make, do, and share. Unlike opinion, evaluation uses evidence. It’s systematic and often systemic, and unlike research (which evaluation is a form of), it also requires a particular form of thinking to execute, much like design.

Evaluation is a structured, multidisciplinary approach to evidence generation, use, and learning that uses data to assess the merit, worth, and significance of a designed thing.

That thing can be a program, service, product, policy, or system; it’s anything that we have put into the world that people derive value from.

Evaluation is a means to aid us in learning about what we made, documenting how we made it, and ascertaining what kind of influence our creations have on the world around us. It is a form of determining the return on investment for our designs.

It does this by generating evidence.

The common refrain from designers is that they work to the brief. Their job, as some see it, is to create what’s been asked for and that’s it. The way its used (or whether its used at all) and its effects are up to the buyer or client.

This is much like health researchers who might be interested in answering important questions, but feel the responsibility to ensure that the evidence they generate is used is beyond their remit.

Just as we looked at the role of science and creativity in our last issue, we must also confront the conceits of evidence in the role of design and health work. By that, it means looking at what designers do and what evidence does.

Designers are good at making claims about what they make. Whether you are a product designer, a non-fiction book author, a scientist doing applied research, or a service creator — everyone of these roles is about making claims about the product or “thing” being created. Evaluation is the means to test those claims with evidence.

Evaluators do this through many different means. This might be a design-driven evaluation approach, which connects evidence generation to the decisions made about something’s use and usefulness. It could be through any number of methods and approaches. For more on this, I’d recommend you check out the Better Evaluation platform for methods, theories, and approaches.

Whatever the methods and approaches you choose, evaluation generates evidence and makes it transparent to you as the designer and to those who are affected by or will use yoru creations. That’s important because not every design will yield positive benefits. Some may (unintentionally) cause harm. Some might have greater impact than anyone realized.

We won’t know unless we evaluate.

What does evaluation offer designers? Here is an incomplete list and some things to consider.

Decision making. Evaluation helps ask: What are we doing and what could we do next? Rather than being arbitrary, why not use data and a structured process to inform your decisions about next steps in the design process or where to continue innovating.

Seeing the future. An understanding of what we’re doing and the effects that arise from it allows us to apply foresight to better imagining what could happen, how that connects to our strategies and decisions today, and reinforce or support what we do today to influence tomorrow.

Design. Evaluation data provides the feedback necessary to undertake design work and design thinking. Without good data to support the design process of research, ideation, prototyping and implementation design thinking is an art project with a purpose . John Nash has illustrated how evaluation and design are linked in his book on design thinking and schools.

Asking better questions. Few things will engage people like good questions. Evaluation questions what we are doing, who we are, and how we undertake our work and live our values.

Creating conversation. The asking and answering of questions in a manner that is transparent, potentially practical, and relevant to people’s lives is what fosters engagement between people. This taps into what makes us distinct, what brings us together, and the curiosity and purpose people have about the world.

Share

Speaking truth to power. Data, democracy, rights, and responsibilities are all part of evaluation of the designs we make.

Honouring our work. Evaluation data allows us to recognize the effort that goes into making something and helping inform how to make it again (scale) or make it different.

Learning. Evaluation data provides a focus for attention and engagement. It is also the one outcome we can guarantee when innovating. When used across an organization, evaluation can be a catalyst for organizational transformation.

Leading system change. Complex systems evolve and adapt based on feedback. Evaluation is the means of providing and structuring that feedback. We can either have the winds of change blow us anywhere, or we can sail with the winds to where we want to go. Evaluation data also feeds and helps shape the systems models we need to inform any systemic change and design.

Telling stories about who we are as people. The stories we tell ourselves and each other is based on data (even if we don’t think of it as such). Evaluation provides the means of making that data a little more systematic to help translate the story of one person or group into meaning for another.

Promoting health and preventing harm. What contributes to our well-being? What actions promote harm? How do we know? Simple cause-and-effect situations can be easy to assess. What happens when we aren’t sure — like with drug education programs, the effect of eating certain foods on our health, or the role of social technologies on our mental health.

Reinforcing democratic ideals. Work from Katri Vataja and Pinka Parkkonen illustrate how Finland are using evaluation as a vehicle for fostering democratic engagement around social policy and what it means for the future. They argue that evaluation has an ethical imperative to understand people and the designs we create.

Design and innovation. Evaluation data is the lifeblood of innovation. It is the innovator’s secret advantage. Over at Cense.ca there is a whole series devoted to exploring this and why great innovation can’t succeed without good evaluation.

Provoking curiosity. It’s a big job, but evaluation provides a means to focus on what is happening and invoke curiosity about the world by giving us something constructive to contemplate.

Recognizing humanity. If it’s not obvious from the other points, one thing that evaluation does really well is recognize our humanity. It provides a means to allow us to celebrate, to improve, to recognize where we fall short and where we exceed our goals. Evaluators are looking at our place on the planet, how we can see into the future and be visionaries, and how we can lead and support change.

How about that for a sense of impact? Responsible design is as much about evaluation as it is creativity and creation. Link these together and you have real impact.

Thanks for reading. Go out there and make something great.

Connect Design and Evaluation Together

Thanks also to the work of Chris Lysy, the cartoon-laureate of evaluation for his work and creative contribution to the field. You can read more about him, his work, and his layered appreciation of both design and cartooning at https://www.freshspectrum.com