How can museums measure social impact? Interview with Johanna Jones

Johanna Jones is the Director of Evaluation and Visitor Insights at the Oakland Museum of California (OMCA).  Johanna leads the Museum’s audience research, evaluation, and social impact measurement work.  Before joining OMCA in 2016, Johanna was an evaluation consultant for 20 years.

In 2020, you wrote about the work OMCA has been doing to evaluate its social impact. What does social impact look like at OMCA?

Back in 2017, after collecting feedback from a lot of different stakeholders, we drafted an impact statement that we felt pretty good about: The Oakland Museum of California helps create a more equitable and caring city.

Then we took this statement to non-visitors. They asked questions like, “How does a museum make the city more equitable? Are you going to get me a better paying job? Are you going to change unfair housing policy?” Our board asked very similar questions.

I know why we landed on that statement. But we had to go back and ask ourselves, what can a museum really do? We aren’t going to change the minimum wage in Oakland. And we also wanted to be more than just a place to gather people, because a lot of places do that.

What is different about a museum is that we have content, we have objects, and we have stories that we’re trying to tell. At OMCA we tell under-told stories, so people’s experiences feel validated, and we make others aware of those issues. As we were talking to stakeholders it dawned on us that this museum can be the place where we bring people together to feel more connected, to develop empathy, to see other people’s perspectives and see the community challenges that they might not otherwise be aware of. Our content, programs, and exhibits can genuinely foster those conversations. So, we defined our social impact as fostering social cohesion: increasing people’s connectedness, their willingness to interact with one another and to take action together, valuing each other, getting to know each other, and enjoying being around each other.

In 2016, the United Nations came out with a report saying we’re never going to address world poverty if we can’t get people at the table to talk about world poverty. There are so many problems that OMCA can’t change. We are not going to change unfair housing policies; that’s not our wheelhouse. But we can bring people together and have exhibits and programs that get people talking about this problem, acknowledging that it’s a problem, and seeing the different perspectives. And in an ideal world, participants might decide, let’s work on this together and take action.

After you articulated the museum’s social impact, what came next?

After we articulated our social impact we created a framework and metrics. We spent a year collecting data in order to fine-tune those measurements.

Our social impact measurement is extremely specific to the work that we do. It is really a tool for us to use to improve our practice.

Every single week our post-visit survey was sent out to OMCA visitors. This allows us to see months where our social impact was really high, and ask, which programs did those folks attend? Which exhibits were on view?

We did that for a whole year, and we adjusted some of our language and our scales to improve the survey. And since 2019 we have been using the same instrument so that we can make comparisons across years.

What are some of your findings?

We have a Friday night program that used to be year-round (this year we have a winter hiatus). We’re open until 9PM and we have food trucks, live music, and art activities. Everything that’s outside is free, and you can purchase a ticket to visit the galleries, which are open. At these events we see a much more diverse audience than we see for other kinds of museum programming. We see strangers talking to strangers – families will put a blanket down in the garden next to another family. We hear from visitors that they have their Friday night friends who they meet and see regularly at this event. So that’s one example where we see social impact.

Once we started this work, staff started programming with these outcomes in mind. They want people talking to each other, so they ask, what can they include in their program that is going to get people talking to each other?

We have also found that our exhibits that tell under-told stories often have the strongest social impact on our visitors. This is true both for people who say, “Oh, that’s my story, and nobody’s ever told it before,” and for visitors who say, “I didn’t know that story, and I’m so glad that now I’m aware of it.”

We embed things like talkback exhibits. We ask a prompt and people respond, and that’s really effective at getting people talking. We also have “take action” cards with provocative statements on the front and resources on the back, to foster conversation both onsite and once visitors take the cards home.

We have one prompt for an artwork called Question Bridge: Black Males, which is a video installation. The talkback station asks the question, “What does it mean to be your true self?” We don’t ask about the work of art – we ask visitors a question about identity. The question is at a station that consists of little nails around the wall and nicely-printed cards on high-quality paper stock.  This station is in a hidden corner – a lot of nonsense could happen back there, because it’s not being closely monitored. But we don’t get nonsense: we get very little profanity, and very little that is off-topic. We get a lot of rich personal reflections in that space.

I think the quality of the paper and the experience matters. But even more important is that we ask people to reflect on their own experiences rather than asking their opinion of something we have created. I think this is much more conducive to those thoughtful responses.

Talkback stations generate a lot of responses on small pieces of paper. Occasionally staff notice a cool comment or say, “Oops, better take that one down, that’s profanity.” But, at least at our museum, no one was systematically analyzing and cataloguing responses. Now that is something that my staff do regularly. The Talkback Exhibit comments are proving to be a very rich data source that’s helping us understand how visitors respond to and connect with our content.

What is the most surprising thing you have learned doing this work?

I’m very interested in examining for whom social impact is stronger or weaker. When national museum studies don’t show differences by race, ethnicity or income, it’s usually because their samples are skewed – there is not a diverse enough audience response being captured.

Pre-pandemic we distributed all our surveys face to face, so we only got about 1,500 visitor surveys back each year. With that amount of data I was limited in how I could analyze it. I could run the data by BIPOC visitors vs White visitors, and low income vs high income, but I couldn’t get more granular than that. We consistently saw that BIPOC visitors and visitors with lower income rated us lower for social impact. They felt less of a sense of belonging, they felt less welcome, they felt less like their stories were reflected.

That was hard for staff to hear. But I also said to them that if we didn’t see differences I would question our metrics, because we know that museums have not historically served these audiences well. So of course they are not going to feel as connected to us and to museums in general. So, for me, this validated the work we are doing – it demonstrated that there is nuance in our scales.

After the pandemic we went to an all-emailed survey, and now we get almost 6,000 surveys a year. So now we are able to tease out data by specific racial and ethnic groups. We’re finding that our Asian American and Pacific Islander (AAPI) visitors are the ones who feel the least social cohesion at our museum. I like to look at other national trends when studying data. There are reports out there showing that Asian Americans feel the least sense of belonging in America. Those studies are helping us put our data in perspective, and letting us know where work needs to be done.

Sometimes it’s hard for OMCA staff to hear that not all of our visitors are feeling as strong a social impact as others. But then they turn around and say, “Okay, what are we going to do differently? What other community partners are we going to reach out to? What content might we include? We have a shared goal of changing the museum so the data looks different five years from now. Staff are able to pivot and address the challenge of making change.

For museums that are interested in this approach, what are a few important take-aways to keep in mind? 

1. Think carefully about what your impact really is. It might not be what you initially say. Take a step back and think about what your organization can uniquely do in your community, because you can’t do it all. Take time talking with internal and external stakeholders to get a sense of their perceptions of this place. Why do people come here? The more precise you can be about what your impact is, the more useful it will be for your staff.
2. The data that’s hard to hear might be the most useful. Data that shows that everything is great is probably bad or incomplete. Data showing that different groups are responding differently, and that different things work differently, is harder to hear but much more useful.
3. The point of evaluation is improvement. The evaluation process isn’t just about gathering data to say, this is what our impact is. You’re gathering data to show what your impact is and what it isn’t, and how you can get better. And that’s a full staff project.
4. Each museum needs to define their social impact for their own work in their particular context. OMCA is a local museum – 90% of our visitors come within a 50-mile radius, and we have a really diverse audience. You can’t bring different kinds of people together if there aren’t different kinds of people at your museum. Context matters.