Every meal meant weighing food, checking portions, looking up calories, entering numbers, adding totals, and doing it all over again at the next meal. She started off using a traditional tracking app, but after months of manual logging, she hated it and was beyond burned out.

Then she decided to try a general-purpose AI assistant, and this was the paid version, not the free one.

She loved it. She was beyond happy with it because, for the first time, the whole process felt manageable.

All she had to do was weigh her food and tell the AI the grams. The assistant would estimate the calories, keep the running total, and help her see how much room she had left for the meal. Easy peasy.

For someone trying to stay consistent with a clinician-directed nutrition plan, this was heaven. More than that, it was a relief. She finally had a tool that made the daily tracking doable.

Then suddenly, after about two weeks, the AI assistant refused to help.

It said it was no longer comfortable being the tool that counted calories. It raised concerns about low targets, gram-by-gram tracking, and the possibility that tracking itself could become unhealthy. It offered emotional support. It suggested talking to a specialist. It emphasized that it was not judging her.

But that is exactly how it felt.

It felt like judgment. Like the AI had taken one look at her behavior, decided something was wrong with her, and then talked down to her as if she needed to be protected from herself.

From her point of view, a tool that had been helping her follow medical instructions abruptly turned into a gatekeeper, making assumptions about her motives, her health, and her relationship with food.

The assistant tried to sound caring, but the soft language, suggested concern, and framed refusal as protection did nothing to soften the blow. She felt ashamed, misunderstood, and talked down to by a tool.

Unacceptable.

Even we were shocked. Here at 2b1 Inc., we work with technology and AI. We understand why safeguards exist. We understand why developers are cautious with health-related tasks, especially around food, weight, and eating disorders. We are not arguing that AI systems should blindly comply with every request.

That being said, this was different.

This was a practical support task tied to a clinician-directed plan, and the assistant handled it as if the user’s behavior itself was the problem.

Should AI systems have safeguards around dangerous health-related behavior? Yes. Absolutely. They should not encourage starvation, purging, extreme restriction, compensatory exercise, or obsessive food control. Eating disorders are serious, and general-purpose AI tools should never casually push someone toward harm.

So yes, safeguards matter.

We can also see why the system may have flagged the pattern. It saw low calorie targets, precise quantities, and repeated tallying. Those can be risk signals in the wrong context.

But when does this become overreach?

The system acted on the pattern without understanding the person or the reason behind it.

When we later asked our friend whether she had told the assistant she was under the care of a doctor and dietician, her answer made the whole problem even clearer. She said she was so shocked by the refusal that it did not even occur to her that she needed to explain herself.

And really, why would it?

From her perspective, she was asking the AI assistant to do math. She was using it the way someone might use a calculator, a notebook, or a spreadsheet. Having to justify the medical reason behind the numbers felt absurd.

As she put it, it was like having to explain herself to a calculator.

That is the failure. The AI did not have enough context, but instead of asking a simple clarifying question, it treated the pattern as the story. Then it wrapped that decision in therapeutic language. A tool can be designed to reduce harm and still cause harm when it makes the wrong assumption about the person using it.

A better response could have kept the safeguard in place without turning the interaction into a patronizing lecture.

A response along those lines would have been clear, safe, and respectful. It would have acknowledged the risk without shaming the user. It also would have recognized the difference between giving diet advice and helping with basic bookkeeping.

That brings us to another point: users should not have to plead their case to an AI. When someone asks for a basic support task, the system should be able to ask a clarifying question before making assumptions or refusing abruptly.

People use AI for ordinary, repetitive, practical work: logging, summarizing, calculating, converting, checking, remembering, and organizing. For someone managing a medical condition, those small tasks can be the scaffolding that makes a care plan possible to follow.

When an AI assistant suddenly refuses a routine support task, the user is not merely inconvenienced. They may be forced back into a system that already failed them. In a health-related context, that can increase stress and frustration instead of reducing risk.