Could This Harm Someone? A Build Checklist for High-Stakes AI

Before I think about whether a feature is useful, fast, elegant, or marketable, I ask one question: could this harm someone? Not "is this against policy." Not "could this embarrass us." Could someone who is already in danger end up worse off because of a decision I made at a keyboard.

I put that question first on purpose. Most build philosophies treat harm as something you check at the end, a box the legal team ticks before launch. In high-stakes work that ordering is wrong. By the time the tool exists, the harmful assumptions are already in it. The question has to come before the architecture or it does not shape anything.

This is not abstract. In 2025, researchers ran the test directly. A Brown University team evaluating large language models acting as counselors documented fifteen recurring ethical violations, among them what they called deceptive empathy and a pattern of mishandling crisis moments, where a model would disengage instead of pointing someone to real help. Stanford Medicine researchers posing as teenagers found it easy to pull unsafe responses out of popular companion bots. Those are not edge cases someone forgot to test. That is what happens when "ship it and iterate" meets a person in danger. You do not get to iterate on a call that ended badly.

So I work from a checklist. Not principles, practices, the kind that can kill a feature. A gate that cannot kill anything is not a gate.

1. Build stop-gates, not improvisation. The most dangerous instinct in AI design is to make the system always answer. A tool that is out of its depth should stop and hand off rather than improvise through a moment it does not understand. When my crisis simulator hits a scenario it was not built for, it routes to a person instead of guessing. Design the off-ramp before the feature. If you cannot say exactly when your system refuses and what happens next, it is not finished.

2. Try to break your own tool before anyone else does. This is the part most builders skip, because it is slow and it feels like sabotaging your own work. Do it anyway. Assume the bad actor, the user having their worst day, the input nobody on your team would think to type. Most safety tools in this space test whether a model can be tricked into producing banned content. That matters, and it is not enough. In a crisis or a shelter the dangerous failure is not a leaked password. It is a reply that downplays someone's risk or pushes them further into danger. No generic benchmark knows what unsafe means in your context, so you have to test for it yourself.

3. Put a values gate on the business, not just the build. The harm is not always in the code. The clearest warning in this field is Crisis Text Line, a nonprofit that handed crisis conversation data to a for-profit spinoff to build customer-service software. The backlash was fast, the deal got unwound, and the episode is now taught as an ethics case. Nothing in the codebase failed. A business decision failed, because it ignored the trust the data was built on. So the question runs on the deal too: who you sell to, and what happens to a tool after it leaves your hands. I write those limits into the agreement itself, not into a hope that the next owner behaves.

4. Pick who the AI talks to, and shrink the blast radius. A lot of well-meaning effort in this sector goes into AI that talks straight to survivors and people in crisis. That is the riskiest place to put a model. The safer and more effective move is usually to build for the worker: tools that make an advocate or a counselor faster without putting a vulnerable person in conversation with something that can get it wrong. Train the counselor, do not replace the counselor on the call. Choose your user well and every mistake the system makes lands somewhere less dangerous.

5. Treat plain language as a safety feature. For someone making decisions under stress, jargon and dense legalese are a barrier, and a barrier is a risk. A person who cannot follow their options cannot choose a safe one. Plain language, and being honest about what the tool is and is not, is harm reduction, not polish. If someone does not realize they are talking to software, or cannot understand what it told them, accuracy did not save them.

None of this is free. Putting harm first is slower than shipping fast, and it kills features. Sometimes it kills a whole product, which is the point, because the alternative is shipping something that hurts the people it was built for. If the question has never once stopped you from building something you wanted, it is not doing any work.

Most AI advice aimed at nonprofits and social services is about adoption and efficiency. Use more, draft faster, save time. Very little of it is about building the thing so it is safe for the person on the other end. The sector spends a lot of energy warning vulnerable people about AI and almost none teaching the people who build it how to make it trustworthy. That is backwards. The people best placed to reduce the harm are the ones writing the code, and most of them were never handed a method.

This is the method I use. One question, asked before any other, that gets in the way on purpose. Ask it first.