One assistant, two inboxes (or more)
How we give one agent access to all your inboxes, making sure nothing leaks between them without asking the model to remember which is which
Almost everyone lives in more than one inbox. You’d want your assistant to read them all. For example, you’d want it to catch that your daughter’s school play from your personal inbox and the board prep from your work inbox both landed on Monday afternoon. Or say you run three businesses from three different addresses; you’d want one assistant to handle all three, with the right context each time.
So why not just connect them all? Most email clients let you do this, and many AI assistants list it as a feature. But a mail client only shows your inboxes side by side, while an AI agent reads all of them and then writes somewhere: an email, an invite, its own memory. That’s where the worlds blend, and we didn’t want to ship it until we could keep them apart.
What can go wrong
One agent juggling multiple inboxes in one context window will eventually leak details from one into another. A reply to a work thread mentions the medical appointment the agent saw on your personal calendar. Board notes end up with a family member. It’s Severance without the surgery: the same mind holds both sides, and nothing in the model keeps the wall up. A leak is just as bad whether the other inbox is “personal” or belongs to “the other company”. That’s why our implementation doesn’t know what “personal” means; it knows accounts.
Note that there’s no attacker in this story. The failure comes from within, and stems from (1) a model that sees content from two accounts mixed together, (2) an agent that can write to either account, and (3) nothing that can tell the two apart. This conflict-of-interest security problem was formalised by Brewer and Nash in 1989 as the Chinese Wall policy. The issue here is not “this data is secret”, but “if you read it from one side, you are not allowed to write it to the other”. Telling the LLM “personal content is private and should not be shared in a work context” is just an instruction. We do tell it that, but nothing guarantees it will comply. The approval-fatigue argument also applies here: if you had to review every cross-inbox action, you would eventually approve the wrong thing.
Our first solution was to make the personal inbox read-only. The assistant could read and draft, but ultimately the human was the one to send the email. But soon enough it became clear that this is not what users wanted. So the question became: how do we give the assistant full access to multiple inboxes and still keep them apart?
Colour every fact
There is no magic in our solution. We mark every item of data with the inbox it came from, and we call that mark its colour: each connected account gets its own. An email, a calendar event, a proactive task our agent produced, a contact, a memory, a draft: we colour each one according to the account it belongs to. Since the leak model is bidirectional (you wouldn’t want work-related information to reach your friends), asking “is this personal?” isn’t enough, so we give the primary account its own colour too, just like every connected account.
Each item is coloured when it is fetched from the email provider, and the colour is carried along with the data through every layer of our pipeline. This is what makes the separation described below possible: it is enforced by code, and holds even when the model does not follow the “keep personal stuff personal” prompt.
This is not a new idea. It is information-flow control with one label per account, the idea behind Myers and Liskov’s decentralised label model from the nineties. The same idea was applied to prompt injection by two recent agent-security systems: Google’s CaMeL tracks which values in an agent’s plan were derived from untrusted data, and Microsoft’s FIDES carries confidentiality and integrity labels through the planner and enforces policy on them deterministically.
Our labels are coarser: one per mailbox and not one per value’s history. Our policy is much simpler too, but the idea is the same: the label travels with the value. The check happens when the value is about to leave, as described below.
Writes: the subagent, again
Sending an email from one of your connected inboxes is where the boundary matters most, and we reuse the pattern from our last post: a subagent as a boundary. The main agent’s context is mixed by design, because it reads all your inboxes to know what you need. That’s why the main agent itself never sends from a connected inbox. It spawns a communication subagent that can send only from the relevant inbox, and hands over a brief.
The brief is the only bridge across the colour boundary, so the brief is what we check. The main agent doesn’t paste content into it; it cites the messages that the email draws on. Our code verifies that every citation, reply-to and forward carries the sending mailbox’s colour, then copies those messages into the brief. The subagent itself can’t search or read any mailbox: it sees the verified and enriched brief and the replies in its own conversation, and nothing else. A wrong citation or a contact mix-up is rejected before it reaches the subagent (i.e., before anything is sent).
Before the email leaves, an attribution check traces the identifiers, numbers, dates and domains in the body back to the brief or the conversation so far, and a second model judges the paraphrased claims that string matching can’t see. This is where a fabricated or leaked detail gets caught.
The shape of it
Nothing here is a new primitive. Google’s agent security framework calls it the hybrid approach: deterministic controls for what can be checked in code, reasoning-based checks for what can’t, and neither doing the other’s job. It’s the same idea as described in our last post, applied inward: make the boundary small enough to enforce, enforce it in code, and let the prompt be advice rather than the last line of defence.
References
- Brewer and Nash cs.purdue.edu
- Myers and Liskov's decentralised label model cs.cornell.edu
- CaMeL arxiv.org
- FIDES arxiv.org
- agent security framework research.google