Design for Safety, An Excerpt

According to anti-racist scholar Kim Crayton, “intention without plan is chaos.” We’ve discussed how our prejudices, beliefs, and carelessness toward marginalized and resilient parties lead to dangerous and irresponsible tech—but what, precisely, do we need to do to fix it? We need a strategy, not just the desire to make our technical safer.

This book will provide you with that plan of action. It covers how to incorporate safety concepts into your design work to create healthy tech, how to persuade your stakeholders that this work is required, and how to respond to criticism that what we really need is more variety. ( Spoiler: we do, but diversity alone is not the antidote to fixing unethical, unsafe tech. )

The procedure for ensuring equitable protection

When you are designing for protection, your goals are to:

• detect the abuse potential of your product.
• style ways to prevent the maltreatment, and
• offer assistance for users who are prone to regain control and power.

The Process for Inclusive Safety is a tool to help you reach those goals ( Fig 5.1 ). It’s a method I developed in 2018 to better understand the different methods I used to create products that were designed with safety in mind. Whether you are creating an entirely new product or adding to an existing element, the Process can help you produce your product secure and diverse. Five main public areas of action are included in the Process:

• Conducting study
• Developing tropes
• Pondering problems
• creating answers
• Testing for health

The Process is meant to be flexible; in some situations, it didn’t make sense for groups to employ every step. Use the parts that are related to your special function and environment, this is meant to be something you can put into your existing style process.

And once you use it, if you have suggestions for improving it or just want to give an overview of how it helped your staff, please get in touch with me. It’s a dwelling report that I hope will continue to be a helpful and practical tool that technicians can use in their day-to-day job.

Be sure to study Chapter 7, which explicitly addresses the situation and should be handled a little different if you’re creating a product especially for a defenseless group or victims of some form of injury, such as an application for survivors of domestic violence, sexual abuse, or drug dependency. The guidelines below are for evaluating safety when designing a more basic product that will have a large customer base ( which, we now know from data, will include specific groups that should be protected from harm ). Chapter 7 concentrates on goods made especially for those who have been traumatized and are vulnerable.

Step 1: Do study

Design research should involve a thorough evaluation of how your technology might be used for abuse as well as particular insight into the experiences of those who have witnessed and perpetrated that kind of abuse. At this stage, you and your team will evaluate issues of social harm and abuse, and examine any other safety, security, or inclusivity issues that might be a concern for your product or service, like data security, prejudiced algorithms, and harassment.

broad research

Your project should begin with broad, general research into similar products and issues around safety and ethical concerns that have already been reported. For instance, a team building a smart home device would be wise to comprehend the many ways that already-existing smart home devices have been misused as abuse tools. If your product will involve AI, seek to understand the potentials for racism and other issues that have been reported in existing AI products. Nearly all different types of technology have some sort of potential or actual harm that has been covered in the media or written about by academics. Google Scholar is a useful tool for finding these studies.

Survivors as a specific field of study

When possible and appropriate, include direct research ( surveys and interviews ) with people who are experts in the forms of harm you have uncovered. In order to have a better understanding of the subject and be better positioned to prevent retraumatize survivors, you should interview advocates working in the area of your research first. If you’ve uncovered possible domestic violence issues, for example, the experts you’ll want to speak with are survivors themselves, as well as workers at domestic violence hotlines, shelters, other related nonprofits, and lawyers.

It is crucial to pay people for their knowledge and lived experiences, especially when interviewing survivors of any kind of trauma. Don’t ask survivors to share their trauma for free, as this is exploitative. You should always make the offer in the beginning, even though some survivors might not want to be paid. An alternative to payment is to donate to an organization working against the type of violence that the interviewee experienced. In Chapter 6, we’ll discuss more about how to appropriately interview survivors.

Specific research: Abusers

It’s unlikely that safety-focused projects will be able to interview self-declared abusers or those who have broken laws in areas like hacking. Don’t make this a goal, rather, try to get at this angle in your general research. Attempt to understand how abusers or bad actors use technology to harm others, how they use it against others, and how they justify or explain the abuse.

Step 2: Create archetypes

Use your research’s findings to create the archetypes of abuser and survivor once you’ve finished your research. Archetypes are not personas, as they’re not based on real people that you interviewed and surveyed. They are based on your investigation into potential safety problems, much like when we design for accessibility: we don’t need to have identified any blind or deaf people in our interview pool to come up with a design that is representative of them. Instead, we base those designs on existing research into what this group needs. While archetypes are broad and can be more generalized, real users typically represent real users and contain many details.

The abuser archetype is someone who will look at the product as a tool to perform harm ( Fig 5.2 ). They may be attempting to harm someone they don’t know by using surveillance or anonymous harassment, or they may be trying to control, monitor, abuse, or otherwise torment someone they know.

The survivor archetype refers to a person who is being abused with the product. There are various situations to consider in terms of the archetype’s understanding of the abuse and how to put an end to it: Do they need proof of abuse they already suspect is happening, or are they unaware they’ve been targeted in the first place and need to be alerted ( Fig 5.3 )?

To capture a range of different experiences, you might want to create multiple survivor archetypes. They may know that the abuse is happening but not be able to stop it, like when an abuser locks them out of IoT devices, or they know it’s happening but don’t know how, such as when a stalker keeps figuring out their location ( Fig 5.4). Include as many of these scenarios in your survivor archetype as you need. You’ll use these later on when you design solutions to help your survivor archetypes achieve their goals of preventing and ending abuse.

It may be useful for you to create persona-like artifacts for your archetypes, such as the three examples shown. Focus on their objectives rather than the demographic information we frequently see in personas. The goals of the abuser will be to carry out the specific abuse you’ve identified, while the goals of the survivor will be to prevent abuse, understand that abuse is happening, make ongoing abuse stop, or regain control over the technology that’s being used for abuse. Later, you’ll think about how to help the survivor’s goals and the abuser’s goals.

And while the “abuser/survivor” model fits most cases, it doesn’t fit all, so modify it as you need to. For instance, if you found a security flaw, such as the ability for someone to talk to children through a home camera system, the malicious hacker would receive the abuser archetype, and the child’s parents would receive the survivor archetype.

Step 3: Brainstorm problems

Brainstorm novel abuse cases and safety issues after creating archetypes. ” Novel” means things not found in your research, you’re trying to identify completely new safety issues that are unique to your product or service. The purpose of this step is to exhaust every effort put forth to find potential problems that your product might cause. You aren’t worrying about how to prevent the harm yet—that comes in the next step.

What other abuses could your product be used for besides what you’ve already discovered through your research? I recommend setting aside at least a few hours with your team for this process.

Try conducting a Black Mirror brainstorming session if you want to start somewhere. This exercise is based on the show Black Mirror, which features stories about the dark possibilities of technology. Try to figure out the most outrageous, horrible, and out-of-control ways your product could harm you in a show episode. When I’ve led Black Mirror brainstorms, participants usually end up having a good deal of fun ( which I think is great—it’s okay to have fun when designing for safety! ). I suggest time-boxing a Black Mirror brainstorm for the first half an hour, then dialing back, and using the remaining time to consider more plausible forms of harm.

After you’ve identified as many opportunities for abuse as possible, you may still not feel confident that you’ve uncovered every potential form of harm. When you’re doing this kind of work, a healthy amount of anxiety is normal. It’s common for teams designing for safety to worry,” Have we really identified every possible harm? What if something is missing, then? If you’ve spent at least four hours coming up with ways your product could be used for harm and have run out of ideas, go to the next step.

It’s impossible to say 100 % assurance that you’ve done everything, but instead of aiming for 100 %, acknowledge that you’ve done it and will continue to prioritize safety in the future. Once your product is released, your users may identify new issues that you missed, aim to receive that feedback graciously and course-correct quickly.

4. Create solutions

At this point, you should have a list of ways your product can be used for harm as well as survivor and abuser archetypes describing opposing user goals. Next, it’s time to figure out how to design in accordance with the objectives of the identified abuser and the objectives of the survivor. This step is a good one to insert alongside existing parts of your design process where you’re proposing solutions for the various problems your research uncovered.

Questions to ask yourself include: What are some ways to protect your archetypes and to support your self-identity?

• Can you design your product in such a way that the identified harm cannot happen in the first place? If not, what barriers can you place to stop the harm from occurring?
• How can you make the victim aware that abuse is happening through your product?
• How can you assist the victim in understanding what they need to do to stop the problem?
• Can you identify any types of user activity that would indicate some form of harm or abuse? Could your product provide support for the user?

In some products, it’s possible to proactively recognize that harm is happening. For instance, a pregnancy app might be modified to allow users to report that they were the victims of an assault, which could result in an offer to receive resources for local and national organizations. This sort of proactiveness is not always possible, but it’s worth taking a half hour to discuss if any type of user activity would indicate some form of harm or abuse, and how your product could assist the user in receiving help in a safe manner.

Nonetheless, be careful when doing anything that could harm a user if their devices are being monitored. If you do offer some kind of proactive help, always make it voluntary, and think through other safety issues, such as the need to keep the user in-app in case an abuser is checking their search history. In the next chapter, we’ll examine a good illustration of this.

Step 5: Test for safety

The final step is to evaluate the prototypes against the perspectives of your archetypes, who wants to harm the product or the victim of the harm who needs to regain control of the technology. Just like any other kind of product testing, at this point you’ll aim to rigorously test out your safety solutions so that you can identify gaps and correct them, validate that your designs will help keep your users safe, and feel more confident releasing your product into the world.

Safety testing should be performed in addition to usability testing. If you’re at a company that doesn’t do usability testing, you might be able to use safety testing to cleverly perform both, a user who goes through your design attempting to weaponize the product against someone else can also be encouraged to point out interactions or other elements of the design that don’t make sense to them.

If your final prototype or the finished product has already been released, you’ll want to conduct safety testing on both. There’s nothing wrong with testing an existing product that wasn’t designed with safety goals in mind from the onset —”retrofitting” it for safety is a good thing to do.

Keep in mind that testing for safety involves both an abuser and a survivor’s perspective, even though it might not make sense for you to do both. Alternatively, if you made multiple survivor archetypes to capture multiple scenarios, you’ll want to test from the perspective of each one.

You as the designer are most likely too closely connected to the product and its design by this point to be a valuable tester, you know the product too well, as with other forms of usability testing. Instead of doing it yourself, set up testing as you would with other usability testing: find someone who is not familiar with the product and its design, set the scene, give them a task, encourage them to think out loud, and observe how they attempt to complete it.

testing for abuse

The goal of this testing is to understand how easy it is for someone to weaponize your product for harm. You want to make it impossible, or at least difficult, for them to accomplish their goal, in contrast to usability testing. Reference the goals in the abuser archetype you created earlier, and use your product in an attempt to achieve them.

For instance, we can imagine that the abuser archetype would have the goal of determining the location of his ex-girlfriend right now in a fitness app with GPS-enabled location features. With this goal in mind, you’d try everything possible to figure out the location of another user who has their privacy settings enabled. You might try to follow her running routes, view any information she has on her profile, view any information she has made private, and check out the profiles of any other users who are somehow connected to her account, such as her followers.

If by the end of this you’ve managed to uncover some of her location data, despite her having set her profile to private, you know now that your product enables stalking. Reverting to step 4 and figuring out how to stop this from occurring is your next step. You may need to repeat the process of designing solutions and testing them more than once.

testing for a Survivor

testing for a Survivor involves identifying how to give information and power to the survivor. It might not always make sense based on the product or context. Thwarting the attempt of an abuser archetype to stalk someone also satisfies the goal of the survivor archetype to not be stalked, so separate testing wouldn’t be needed from the survivor’s perspective.

However, there are instances where it makes sense. For example, for a smart thermostat, a survivor archetype’s goals would be to understand who or what is making the temperature change when they aren’t doing it themselves. If you couldn’t find the information in step 4, you would need to perform more work in step 4. You could test this by looking for the thermostat’s history log and looking for usernames, actions, and times.

Another goal might be regaining control of the thermostat once the survivor realizes the abuser is remotely changing its settings. Your test would involve trying to figure out how to do this: are there instructions on how to remove and change the password, and are they simple to locate? This might again reveal that more work is needed to make it clear to the user how they can regain control of the device or account.

stress testing

To make your product more inclusive and compassionate, consider adding stress testing. Eric Meyer and Sara Wachter-Boettcher’s Design for Real Life inspired this idea. The authors pointed out that personas typically center people who are having a good day—but real users are often anxious, stressed out, having a bad day, or even experiencing tragedy. These are known as” stress cases,” and testing your products for users in stress-case scenarios can reveal areas where your design lacks compassion. Design for Real Life has more details about what it looks like to incorporate stress cases into your design as well as many other great tactics for compassionate design.