Design for Safety, An Excerpt

According to antiracist analyst Kim Crayton, “intention without plan is chaos.” We’ve discussed how our prejudices, beliefs, and carelessness toward marginalized and resilient parties lead to dangerous and irresponsible tech—but what, precisely, do we need to do to fix it? We need a strategy, not just the desire to make our technical safer.

This book will provide you with that plan of action. It covers how to incorporate safety principles into your design work in order to make tech that’s secure, how to persuade your stakeholders that this work is important, and how to respond to the critique that what we really need is more diversity. ( Spoiler: We do, but diversity alone cannot solve unethical, unsafe technology. )

The procedure for diverse safety

Your objectives when designing for protection are to:

• determine way your product can be used for misuse,
• style ways to prevent the maltreatment, and
• offer assistance for users who are prone to regain control and power.

The Process for Inclusive Safety is a tool to help you reach those goals ( Fig 5.1 ). I developed this strategy in 2018 to better understand the various methods I used to create products that were designed with safety in mind. Whether you are creating an entirely new product or adding to an existing element, the Process can help you produce your product secure and diverse. The Process includes five basic areas of action:

• conducting exploration
• Creating themes
• pondering issues
• Designing options
• Testing for health

The Process is meant to be flexible; in some situations, it didn’t make sense for groups to employ every step. Use the parts that are related to your special function and environment, this is meant to be something you can put into your existing style process.

And once you use it, if you have an idea for making it better or simply want to give perspective of how it helped your staff, please get in touch with me. It’s a living document, which I hope engineers may use as a practical and useful tool throughout their day-to-day tasks.

If you’re working on a product especially for a resilient team or survivors of some form of injury, such as an application for survivors of domestic violence, sexual abuse, or drug addiction, be sure to read Section 7, which covers that position directly and should be handled a bit different. The purpose of this design is to prioritize safety when creating a more general product with a broad user base ( which, as we already know from statistics, will include some groups who need to be protected from harm ). Chapter 7 is focused on products that are specifically for vulnerable groups and people who have experienced trauma.

Step 1: Conduct research

Design research should include a thorough analysis of how your technology might be used for abuse as well as specific insights into the experiences of those who have witnessed and perpetrated that kind of abuse. At this stage, you and your team will investigate issues of interpersonal harm and abuse, and explore any other safety, security, or inclusivity issues that might be a concern for your product or service, like data security, racist algorithms, and harassment.

broad-based research

Your project should begin with broad, general research into similar products and issues around safety and ethical concerns that have already been reported. For example, a team building a smart home device would do well to understand the multitude of ways that existing smart home devices have been used as tools of abuse. If your product involves artificial intelligence, make sure to learn about the potential for racism and other issues that have been reported in other AI products. Nearly all types of technology have some kind of potential or actual harm that’s been reported on in the news or written about by academics. Google Scholar is a useful resource for locating these studies.

Specific research: Survivors

When possible and appropriate, include direct research ( surveys and interviews ) with people who are experts in the forms of harm you have uncovered. In order to have a better understanding of the subject and be better positioned to prevent retraumatize survivors, you should interview advocates working in the area of your research first. If you’ve uncovered possible domestic violence issues, for example, the experts you’ll want to speak with are survivors themselves, as well as workers at domestic violence hotlines, shelters, other related nonprofits, and lawyers.

It is crucial to pay people for their knowledge and lived experiences, especially when interviewing survivors of any kind of trauma. Don’t ask survivors to share their trauma for free, as this is exploitative. While some survivors may not want to be paid, you should always make the offer in the initial ask. Alternative to paying is to donate to a cause fighting the kind of violence the interviewee experienced. We’ll talk more about how to appropriately interview survivors in Chapter 6.

Specific research: Abusers

It’s unlikely that teams aiming to design for safety will be able to interview self-proclaimed abusers or people who have broken laws around things like hacking. Don’t make this a goal, rather, try to get at this angle in your general research. Describe the ways that abusers or bad actors use technology to harm others, how they use it to silence others, and how they justify or explain the abuse.

Step 2: Create archetypes

Use your research’s findings to create abuser and survivor archetypes once you’ve finished conducting your research. Archetypes are not personas, as they’re not based on real people that you interviewed and surveyed. Instead, they’re based on your research into likely safety issues, much like when we design for accessibility: we don’t need to have found a group of blind or low-vision users in our interview pool to create a design that’s inclusive of them. Instead, we base those designs on existing research and what this group requires. Personas typically represent real users and include many details, while archetypes are broader and can be more generalized.

The abuser archetype is defined as someone who views a product as a means of harm ( Fig. 5.2 ). They may be trying to harm someone they don’t know through surveillance or anonymous harassment, or they may be trying to control, monitor, abuse, or torment someone they know personally.

The survivor archetype describes a person who is being abused with the product. There are various situations to consider in terms of the archetype’s understanding of the abuse and how to put an end to it: Do they need proof of abuse they already suspect is happening, or are they unaware they’ve been targeted in the first place and need to be alerted ( Fig 5.3 )?

You may want to make multiple survivor archetypes to capture a range of different experiences. They may be aware of the abuse is occurring but not be able to stop it, such as when a stalker keeps figuring out where they are from ( Fig. 5.4), or they may be aware that it is happening but are unable to stop it ( such as when an abuser locks them out of IoT devices ). Include as many of these scenarios as you need to in your survivor archetype. You’ll use these later when you create solutions to help your survivor archetypes achieve their objectives of preventing and ending abuse.

It may be useful for you to create persona-like artifacts for your archetypes, such as the three examples shown. Focus on their objectives rather than the demographic information we frequently see in personas. The goals of the abuser will be to carry out the specific abuse you’ve identified, while the goals of the survivor will be to prevent abuse, understand that abuse is happening, make ongoing abuse stop, or regain control over the technology that’s being used for abuse. Later, you’ll think about how to help the survivor’s goals and the abuser’s goals.

And while the “abuser/survivor” model fits most cases, it doesn’t fit all, so modify it as you need to. For example, if you uncovered an issue with security, such as the ability for someone to hack into a home camera system and talk to children, the malicious hacker would get the abuser archetype and the child’s parents would get survivor archetype.

Step 3: Remind yourself of your issues

After creating archetypes, brainstorm novel abuse cases and safety issues. You’re trying to identify completely new safety issues that are unique to your product or service by using the term” Novel” in terms of things that are not found in your research. The goal with this step is to exhaust every effort of identifying harms your product could cause. You aren’t worrying about how to prevent the harm yet—that comes in the next step.

What other uses could your product be used for besides what you’ve already identified in your research? I recommend setting aside at least a few hours with your team for this process.

Try conducting a Black Mirror brainstorming if you’re looking for a place to start. This exercise is based on the show Black Mirror, which features stories about the dark possibilities of technology. Try to figure out how your product would be used in an episode of the show—the most wild, awful, out-of-control ways it could be used for harm. Participants in Black Mirror brainstorming typically end up having a lot of fun ( which I believe is great because having fun when designing for safety! ). I recommend time-boxing a Black Mirror brainstorm to half an hour, and then dialing it back and using the rest of the time thinking of more realistic forms of harm.

You may still not feel confident that you have found every potential source of harm after identifying as many opportunities for abuse as you can. A healthy amount of anxiety is normal when you’re doing this kind of work. It’s common for teams designing for safety to worry,” Have we really identified every possible harm? What if something is missing? If you’ve spent at least four hours coming up with ways your product could be used for harm and have run out of ideas, go to the next step.

It’s impossible to say 100 % assurance that you’ve done everything, but instead of aiming for 100 %, acknowledge that you’ve done it and will continue to prioritize safety in the future. Once your product is released, your users may identify new issues that you missed, aim to receive that feedback graciously and course-correct quickly.

Step 4: Design solutions

You should now be able to identify potential harm-causing uses for your product as well as survivor and abuser archetypes describing opposing user objectives. The next step is to identify ways to design against the identified abuser’s goals and to support the survivor’s goals. This is a good addition to existing areas of your design process where you’re making recommendations for solutions to the various issues your research has identified.

Some questions to ask yourself to help prevent harm and support your archetypes include:

• Can you design your product in such a way that the identified harm cannot happen in the first place? If not, what barriers can you place to stop the harm from occurring?
• How can you make the victim aware that abuse is happening through your product?
• How can you assist the victim in understanding what they need to do to stop the problem?
• Can you identify any types of user activity that would indicate some form of harm or abuse? Could your product help the user access support?

In some products, it’s possible to proactively detect harm that is occurring. For example, a pregnancy app might be modified to allow the user to report that they were the victim of an assault, which could trigger an offer to receive resources for local and national organizations. Although this kind of proactiveness is not always possible, it’s worthwhile to spend a half hour talking about how your product could help the user receive help in a safe manner if any kind of user activity would indicate some form of harm or abuse.

That said, use caution: you don’t want to do anything that could put a user in harm’s way if their devices are being monitored. If you do offer some kind of proactive help, always make it voluntary, and think through other safety issues, such as the need to keep the user in-app in case an abuser is checking their search history. In the next chapter, we’ll walk through a good illustration of this.

Step 5: Test for safety

The final step is to evaluate your prototypes from the perspective of your archetypes, who wants to harm the product and the victim of the harm who needs to regain control over the technology. Just like any other kind of product testing, at this point you’ll aim to rigorously test out your safety solutions so that you can identify gaps and correct them, validate that your designs will help keep your users safe, and feel more confident releasing your product into the world.

Ideally, safety testing happens along with usability testing. If you work for a company that doesn’t conduct usability testing, you might be able to use safety testing to deftly perform both. A user who uses your design while trying to use it against someone else can also be encouraged to point out interactions or other aspects of the design that don’t make sense to them.

You’ll want to conduct safety testing on either your final prototype or the actual product if it’s already been released. It’s okay to test an existing product that wasn’t created with safety goals in mind right away; “etrofitting” it for safety is a good thing to do.

Remember that testing for safety involves testing from the perspective of both an abuser and a survivor, though it may not make sense for you to do both. Alternatively, if you made multiple survivor archetypes to capture multiple scenarios, you’ll want to test from the perspective of each one.

You as the designer are most likely too closely connected to the product and its design by this point to be a valuable tester, you know the product too well, as with other forms of usability testing. Instead of doing it yourself, set up testing as you would with other usability testing: find someone who is not familiar with the product and its design, set the scene, give them a task, encourage them to think out loud, and observe how they attempt to complete it.

testing for abuse

The goal of this testing is to understand how easy it is for someone to weaponize your product for harm. Unlike with usability testing, you want to make it impossible, or at least difficult, for them to achieve their goal. Use your product to try to accomplish the objectives in the abuser archetype you created earlier.

For example, for a fitness app with GPS-enabled location features, we can imagine that the abuser archetype would have the goal of figuring out where his ex-girlfriend now lives. You’d make every effort to track down another user’s location who has their privacy settings turned on with this in mind. You might try to see her running routes, view any available information on her profile, view anything available about her location ( which she has set to private ), and investigate the profiles of any other users somehow connected with her account, such as her followers.

If by the end of this you’ve managed to uncover some of her location data, despite her having set her profile to private, you know now that your product enables stalking. Reverting to step 4 and figuring out how to stop this from occurring is your next step. You may need to repeat the process of designing solutions and testing them more than once.

Testing for Survivors

Testing for Survivors involves identifying how to give information and power to the survivor. It might not always make sense based on the product or context. Thwarting the attempt of an abuser archetype to stalk someone also satisfies the goal of the survivor archetype to not be stalked, so separate testing wouldn’t be needed from the survivor’s perspective.

However, there are cases where it makes sense. For instance, a survivor archetype’s goal would be to discover who or what causes the temperature to change when they aren’t altering it themselves. You could test this by looking for the thermostat’s history log and checking for usernames, actions, and times, if you couldn’t find that information, you would have more work to do in step 4.

Another goal might be regaining control of the thermostat once the survivor realizes the abuser is remotely changing its settings. Your test would involve trying to figure out how to do this: are there instructions on how to remove and change the password, and are they simple to locate? This might again reveal that more work is needed to make it clear to the user how they can regain control of the device or account.

stress testing

To make your product more inclusive and compassionate, consider adding stress testing. This concept comes from Design for Real Life by Eric Meyer and Sara Wachter-Boettcher. The authors noted that personas typically focus on happy people, but that happy people are frequently anxious, stressed out, unhappy, or even go through a bad day. These are called” stress cases”, and testing your products for users in stress-case situations can help you identify places where your design lacks compassion. More information about how to incorporate stress cases into your design can be found in Design for Real Life, as well as in many other effective methods for compassionate design.