How company data ends up in AI tools

In the public cases collected here, company information reached places it should not have along a small number of ordinary routes: a person pastes work into a tool, a feature publishes more than someone meant to share, a provider makes a mistake, a provider’s own vendor is breached, or a court tells a provider to keep what it would otherwise delete. This post walks through each route and keeps who did what separate. An employee’s choice is not a provider’s fault, and a provider’s bug is not the user’s.

Staff paste work into public tools

The first route is the least technical. Someone has a block of code, a draft or a meeting to write up, and a public AI tool that will summarize or fix it in seconds. In 2023, one Samsung employee reportedly uploaded confidential source code to ChatGPT while asking for help fixing a faulty database, and another shared an entire meeting to have minutes written. Samsung then temporarily restricted generative AI tools on company devices. Its memo noted that data sent to these tools is held on external servers and is difficult to retrieve and delete.

Nothing at OpenAI was breached. The tool did what it was built to do with what it was given. That is the part worth holding onto: once text is sent to an outside AI service, it sits on someone else’s servers, and the organization cannot simply delete it on its own.

Sharing features that publish too much

Some exposure comes from how a product is designed, combined with what users click. ChatGPT’s share feature once offered an opt-in box labeled “Make this chat discoverable.” In 2025 thousands of shared conversations that users had opted in turned up in Google search. OpenAI removed the option, and its chief information security officer, Dane Stuckey, said the feature “introduced too many opportunities for folks to accidentally share things they didn’t intend to,” Engadget reported. That case mixes a user action with a design choice.

In August 2025, TechCrunch, citing Forbes, reported that hundreds of thousands of conversations with xAI’s Grok chatbot had become searchable on Google through its share links. There was no opt-in step: pressing share created a public page that search engines could index, in many cases without users knowing. That case was a design choice. For an organization, either way, a share button is a publishing decision, and staff rarely treat it as one.

Bugs and exposures at the provider

Other cases are the provider’s fault, and no amount of staff training would have prevented them.

• In March 2023 a bug in an open-source library used by ChatGPT let some users see the titles of other users’ conversations. For about nine hours it may also have exposed partial payment details of some subscribers. OpenAI took ChatGPT offline and fixed it (The Register).
• In 2023 Wiz Research found that Microsoft AI researchers, while publishing open-source training data, had shared a storage link that granted access to an entire storage account: 38 TB of internal data, including passwords, secret keys and more than 30,000 internal Teams messages. Microsoft said no customer data was exposed (Wiz’s disclosure).
• In January 2025 Wiz found a DeepSeek database that anyone could reach without a password, holding user chat history, API secrets and backend details. DeepSeek secured it promptly after being notified (Wiz’s disclosure).
• The New York Times reported in 2024 that a hacker had accessed an internal OpenAI employee forum in 2023 and obtained details about the design of its AI technologies, without reaching the systems where its models are built and housed. OpenAI did not announce it publicly because no customer or partner information was taken (Reuters, via Insurance Journal).

A separate class of risk comes with assistants connected to company data. In 2025 researchers showed that a crafted email with hidden instructions could have made Microsoft 365 Copilot send internal data it had access to, such as emails, files and chats, to an attacker, with no action by the user (the researchers’ paper). Microsoft rated the flaw critical (CVE-2025-32711) and fixed it on its own servers, with no customer action required. It was a flaw found by researchers, not a reported breach. It shows that the more company data an assistant can reach, the more a flaw in that assistant could expose.

The vendors behind the vendor

An AI provider runs on other companies’ software too. In November 2025 an attacker got into Mixpanel, an analytics vendor OpenAI used, and exported data about some OpenAI API users: names, email addresses, approximate location, browser and operating system, and organization IDs. OpenAI said no chats, API requests, passwords, API keys or payment details were exposed, and it removed Mixpanel from production (BleepingComputer).

The fault here sat with neither OpenAI’s customers nor, directly, OpenAI’s own systems. It is the supply-chain point in its simplest form: every third party in a cloud AI stack is another place data can leak, including third parties the customer never chose.

Retention a court can order

The last route involves no mistake at all. In May 2025, in the copyright litigation brought by The New York Times and other news organizations, a US magistrate judge ordered OpenAI to “preserve and segregate all output log data that would otherwise be deleted” (the order), whether it would have been deleted at a user’s request or under privacy laws. The obligation ended going forward in September 2025, but data already preserved stayed subject to the case (Engadget).

Nobody leaked anything. But while the order stood, a delete button did not mean deletion. Data sent to a cloud AI provider can be held for a lawsuit for reasons that have nothing to do with the person who sent it.

Each route has a different owner, and a fix for one does nothing for the others.

What an organization can control

An organization cannot patch a provider’s bug, audit its analytics vendor or set aside a court order. It can decide three things.

• What may be pasted. A written policy that names the material that does not go into outside AI tools (source code, client files, personnel records, anything under a confidentiality obligation) addresses the first route directly. It works better when staff have an approved alternative.
• Which tools are approved. An approved list, with sharing features and retention settings reviewed before rollout, narrows the design-choice route and gives the organization a record of which providers hold its data.
• Where processing happens. Work processed on hardware the organization controls is not sitting on a cloud AI provider’s servers, so that provider’s bugs, vendors and preservation orders do not reach it. The organization’s own systems still need access control and patching, and its own legal obligations still apply.

That last option is what Exalt builds. On Exalt local AI hardware, questions and documents are processed on the box, not sent to an outside AI service for answering or training. What can still cross the wall is set out on the isolation page, and the cases above are part of the full, source-linked record.