Methodology Overview

StoryEngine is influenced by narrative research methods (Most Significant Change, Global Giving’s Storytelling Project, CognitiveEdge’s SenseMaker, and TASCHA’s evidence narratives) and participatory principles. It also draws on grounded theory, developmental evaluation, and utilization-focused evaluation. Our aim: to better understand what is different now (impacts), how change is happening, and how we might design or adapt our strategies and practices.

“Methods can be emergent and flexible; designs can be dynamic. Contrary to the usual practice in evaluation of fixed designs that are implemented as planned, developmental evaluation designs can change as an innovation unfolds and changes.” — Michael Quinn Patton, Kate McKegg, and Nan Wehipeihana (editors), Developmental Evaluation Exemplars: Principles in Practice, Guilford Publications, 2016, p.12.

Giving back

StoryEngine attempts to be sensitive to and useful for participants. We are not interested in extractive research or evaluation. We also understand that an interview is an important touchpoint and opportunity for engagement. So we look for opportunities to make connections and support participants’ work.

Unless otherwise noted, all stories are shared under a Creative Commons Attribution license, so that others can build on this work, including researchers who want to analyze datasets further.

Sampling

For Mozilla, we used purposeful sampling, focusing on foundation partners, fellows, volunteers, and people and organizations in the broader Mozilla ecosystem. We actively sought out respondents from different programs, initiatives, demographic groups, and geographies. Our sampling evolved during the pilot, and included staff tip submissions, individuals submitting their own stories, and snowball sampling.

StoryEngine was also designed to analyze stories drawn from content on the open web, such as community-generated blogs, podcasts, or videos. At this stage, however, we have only analyzed a corpus composed of participant interviews.

Consent

StoryEngine uses a two-step consent process to ensure privacy and security of participants and the org – the interview release and consent to publish. The interview release is sent to the participant and signed before the interview takes place and the consent to publish is sent to participants after they informally (email) say they are finished editing their story and approve it.

We are currently looking into ways for participants to revoke consent after their interview has been posted online. We learned about the gaps in our consent process during one of our interviews. Una Lee is a Mozilla grantee and a designer working on something called the Ripple Map — a way to capture the impacts of events like these — impacts that ripple out over many years. As Una and her partner, Dann Toliver, developed the digital version of the Map, they started to run into challenges around consent. They were capturing data about people and organizations — and the connections between them — that posed privacy and security risks, so Una and Dann proposed an approach based on Planned Parenthood's sexual consent framework — extended to think about how we might create consentful technologies and ensure that users maintain control over their digital bodies.

We're thinking about how we might to adopt this framework for StoryEngine. The challenge: it never seemed like “the right time” to implement practices around revoking consent, especially because it’s complicated logistically.

Six key steps

The StoryEngine methodology consists of six steps and work is designed to be an ongoing process — a loop. You’re listening and testing throughout the process, with the goal of building stronger and more systematic feedback loops with the people you serve.

Design → Listen → Process → Share → Analyse → Act + Loop Back

What follows is a summary of each of the six phases, each chapter ends with tools and templates to facilitate your work. That said, StoryEngine is flexible and meant to be customized, while staying true to core principles. We encourage adaptation to different uses; it has already been tailored for events and grant reporting.