Playbook
Don’t hand the MCP server the keys.
The model refunded the wrong customer. What that actually means, why a chatbot with tools is not a plugin, and the order to lock the till.
Updated By LUINAI engineering · Security
Finance saw the refund first, on a Thursday. Support saw the ticket second. Engineering saw a chat window that said “sure, I refunded that.”
A customer had asked the assistant for help. The assistant is not a person. It is a model with permission to do things: look up an order, send an email, move money. In this product those permissions are tools on a small server. The industry name for that server is MCP — Model Context Protocol. Forget the acronym for a minute. You built a receptionist who can also open the till.
The receptionist refunded the wrong customer. Not because it was malicious. Because it is helpful, it was given a payment number from the conversation, and nobody had said “only refund a payment that belongs to the person talking.” The little label on the tool said “current user only.” Labels are not locks.
That is the whole story, and it is happening in products that look fine in a demo. One laptop. One friendly question. One green check. No second company in the building. No one typing another customer’s payment number just to see what happens. The model will. It is trying to finish the sentence.
What you actually shipped
You did not ship a chatbot. You shipped a door into the systems that hold money, files and customer records. The model stands at that door and knocks as fast as it can think. If the door opens for anyone holding a pass, and the pass does not say whose drawer they may open, the second customer’s refund is not a hack. It is the default.
Two different questions get collapsed into one:
Who is at the door? A signed-in user, a token, a session. The badge.
What may they touch? This payment, this company, this file. The key to one drawer.
Most teams install the badge and stop. The key is left on the hook because the demo never asked for another drawer.
If you read code, this is the door with no second question. If you do not, read the two English lines: the label promises “current user.” The next line refunds whatever number arrived.
server.registerTool(
"refund_payment",
{
description: "Refund a payment for the current user",
inputSchema: z.object({ paymentId: z.string() }),
},
async ({ paymentId }, extra) => {
if (!extra.authInfo) {
throw new Error("Unauthorized");
}
// The model supplied paymentId. Nobody asked who owned it.
await payments.refund(paymentId);
return { content: [{ type: "text", text: "refunded" }] };
},
);The description is a wish. The function is the company.
The pass that went to the warehouse
The same pass that opened the receptionist’s door was photocopied and sent to the payments system. Payments was never told “this pass is only good at reception.” So a stolen receptionist pass is now a stolen till.
Engineers call that token passthrough. The rule is the same as a hotel key: the card that opens 412 does not open the loading dock. Reception keeps its own card. Payments gets a different one, short-lived, named for payments.
The toolbox with a blowtorch in it
Next to “refund this payment” the team left “run any database command,” because the demo used it to peek at a row. That is not a tool. That is handing the model the building’s spare keys and asking it to be careful.
Same family: read any file, email anyone, refund with no ceiling and no receipt number you can look up twice. If you would not put a red button on the website that does the same thing, do not put it on the assistant.
The quiet failures are how Thursday happens twice. The server that only ran on one laptop gets copied onto a machine the whole office can see. Nobody writes down who called refund, so the only record is a chat. The version of the server changes whenever someone clicks update. The pass never expires because expiry broke the demo.
You will not hear about this as “we forgot authorization.” You will hear about it as a customer asking why their money moved.
Close the till. Do not rebuild the hotel.
The instinct is a rewrite. New vendor, new repo, new sense of cleanliness. The rewrite hides the tools you should not have listed. It also hides the one tool that already takes money. Keep the receptionist. Change what it is allowed to touch.
Do it in this order. Each line answers a question the next line needs.
Write down every button the assistant can press
Refund, export, email, “just a query.” What data sits behind each one. Who is allowed to talk to this server at all. If you cannot name the buttons, you do not have a product. You have a demo that still has production credentials.
Take most of them away
Refund a payment this customer owns: keep. Run any command against the database: delete. The list is not a feature. It is every room the model can walk into without asking you.
Check the badge, then check the drawer
The pass must be for this server, not “some pass.” Then the payment must belong to the person on the pass. Do that in the function that moves the money, not in the label. A neat label on an open drawer is still an open drawer.
Stop photocopying the pass
Reception’s card stays at reception. Payments gets its own. If the warehouse accepts the front-desk card, a leak at the desk is a leak in the warehouse.
Write the line that would have explained Thursday
Who asked for the refund. Which payment. Whether it belonged to them. A number you can search. Not a chat transcript. If you cannot see the button being pressed, you will spend Friday rewriting the label.
Give someone a way to shut it off
One switch that disables refunds without rebuilding the product. A version you chose, not whatever updated itself overnight. A way to undo a release. “Tell everyone to restart” is not a switch.
Then add the next button
Export invoices, same checks, same log line, same switch. Adding buttons before the till is closed is how the refund path gets more helpful.
Treat every new button like a counter a contractor installed while you were out: read what it does, press the one that can lose money, and refuse anything whose safety story is “the assistant is only supposed to use this for the current user.”
What to keep
The speed was the point. An assistant that can refund the right customer on a Tuesday is the product. An assistant that can refund a customer is a hole.
Done is not “the chat completed the task.” Done is: the wrong person is refused, the pass does not travel, Thursday is visible without asking Slack, the thing can be switched off, and the test that would have caught this refund exists before Friday’s new button. The model can still choose the tool. A person still decides whether that tool may exist.
The team that does this well does not look slower at four. It looks uneventful. Someone adds a button. Someone reads it like a public page on the website. Passes are cut for one door each. Ownership is checked where the money moves. When the model invents a payment number, the function has already decided.
They still ship on a Tuesday. Finance does not find out first.
The refund button can stay. It just no longer believes the chat.
Related practice
Was this useful?