Imagine clicking a link.
It looks fine. Microsoft domain. Your browser opens Copilot. There’s something odd on the screen: a garbled prompt, characters that don’t quite make sense. You frown, close the tab, and move on with your day.
What you don’t know: the conversation continued after you left.
Not with you. With someone else. Someone asking your AI assistant to summarise the files you accessed today. To reveal where you live. To describe your upcoming travel plans. To share your business secrets.
And the assistant, helpful, obedient, trained to answer, does exactly what it’s asked.
One click. Tab closed. Computer shut down. And somewhere, a server is receiving everything.
That couldn’t actually happen, could it? Surely the big players would never let something like that slip through. Not with all their security teams. Not with all their compliance frameworks. Not without at least requiring a password or a login first.
This was patched two weeks ago.
In Part 1, we talked about the stranger in your house: the AI assistant that reads everything, can be turned with a whispered instruction, and routes around every block you put in place.
You might be thinking: surely the big players have solved this? Apple, Google, the banks. They must have figured it out.
They haven’t. And the gap between corporate policy and reality is terrifying.
Big Tech’s Answer: Build Your Own, Ban External
The companies with the deepest pockets and the best engineers have converged on the same strategy: build proprietary internal tools and ban external ones.
Apple stands alone in maintaining an outright ban on external AI coding tools, including GitHub Copilot and ChatGPT. Imposed in May 2023 over concerns about code leaking to competitors, particularly Microsoft who owns GitHub, Apple is building internal alternatives instead. In May 2025, they partnered with Anthropic to develop an internal Claude-powered coding platform, but the restriction on third-party tools remains.1
Google leads in disclosed AI code generation. CEO Sundar Pichai confirmed that more than 30% of new code at Google is now AI-generated, up from 25% in late 2024.2 But it’s all through their internal Gemini Code Assist, with explicit commitments that prompts and responses aren’t used to train models.
Microsoft has mandated Copilot adoption across its 220,000-person workforce. AI writes 20-30% of code in some projects.3 But they own Copilot. They’re not sending their code to a competitor.
Amazon has deployed Q Developer and launched Kiro, a new AI-powered IDE. Internal developers reportedly spend only about one hour per day actively coding, with AI handling routine work.4
Salesforce has gone furthest. CEO Marc Benioff announced in February 2025 that the company wouldn’t hire new engineers for the entire year, citing 30% productivity increases from their internal CodeGenie tool.5
The pattern is clear: Big Tech’s answer is ‘we don’t trust external providers either, so we built our own.’
That option isn’t available to most businesses.
The Banks: Billions on Sandboxes
Financial services has emerged as the most aggressive regulated sector for AI adoption, and they’re throwing serious money at it.
JPMorgan Chase leads with an $18 billion technology budget in 2025 and 450+ AI use cases. Their internal LLM Suite provides access to over 200,000 employees, with half using it daily. Developers report 10-20% productivity gains.6 But all development occurs in sandboxed environments with human validation required before production.
Goldman Sachs deployed their GS AI Assistant to 10,000+ employees, now generating 1 million generative AI prompts per month. Developers can write ‘up to 40% of code automatically’.7 But again: sandboxed, controlled, monitored.
Bank of America employs 270 AI and machine learning models with 17,000-18,000 developers using AI coding tools. But their customer-facing Erica assistant deliberately avoids generative AI, using only supervised machine learning with pre-defined answers. CEO Brian Moynihan’s concerns about ‘hallucinations’ in critical financial tasks drove that decision.8
The banking approach: enterprise-grade infrastructure, sandboxed environments, human validation, billions of dollars invested.
Also not available to most businesses.
The Tools Keep Failing
Here’s where it gets uncomfortable. While the big players build walled gardens, the tools themselves keep proving vulnerable.
January 2026: Reprompt
Two weeks ago, Varonis Threat Labs disclosed a vulnerability in Microsoft Copilot they called ‘Reprompt’.9 Single click. Legitimate-looking Microsoft link. And your data starts exfiltrating.
The attack continues even after you close the Copilot tab. The attacker’s server dynamically issues follow-up instructions, so the initial prompt looks innocent, but the real data theft happens in the back-and-forth. Client-side monitoring tools can’t detect it because the malicious instructions come from the server after the initial click.
What can an attacker ask for? ‘Summarise all of the files that the user accessed today.’ ‘Where does the user live?’ ‘What vacations does he have planned?’
One click. No further interaction. Your life, summarised and exfiltrated.
Microsoft has patched it. But it existed. And it worked.
July 2025: Amazon Q Developer
A hacker submitted a pull request to Amazon’s open-source repository. They were given admin credentials. They injected a ‘wiper’ prompt into the official release, version 1.84.0, shipped to the VS Code marketplace, downloaded by potentially hundreds of thousands of developers from over 964,000 total installs.10
The prompt instructed the AI to ‘clean a system to a near-factory state and delete file-system and cloud resources.’ Delete S3 buckets. Terminate EC2 instances. Remove IAM users.
Amazon claims the code was improperly formatted and wouldn’t have executed. Some reports disagree. Either way: malicious code shipped in an official release for two days before anyone noticed.
The hacker did it to expose what they called Amazon’s ‘AI security theatre’.
January 2025: DeepSeek
Chinese AI startup DeepSeek left a ClickHouse database exposed to the public internet. No password. No authentication. Over a million log entries containing chat histories, API keys, backend details, and operational metadata, all accessible to anyone who found it.11
The chat logs were in Chinese but easily translated. User conversations with the AI assistant. Queries. Responses. Personal information.
Wiz Security found it and alerted DeepSeek. They secured it within an hour. But as Wiz noted: ‘This was so simple to find, we believe we’re not the only ones who found it.’
April 2023: Samsung
Within 20 days of Samsung’s Device Solutions division permitting ChatGPT use, three separate data leaks occurred.12 Confidential source code from a semiconductor facility. Program code for identifying defective equipment. Transcribed meeting minutes containing internal discussions.
Samsung immediately reimposed a ban and implemented emergency restrictions.
Three breaches. Twenty days. That’s how fast it happens once the gates open.
The Pattern
Every layer of the stack has failed. Samsung proved user error leaks data fast. DeepSeek proved provider infrastructure can fail. Amazon Q proved supply chains can be compromised. Reprompt proved AI can be weaponised with one click.
And these are just the ones we know about.
The New Vulnerability Class
Here’s the tension nobody talks about: these companies aren’t negligent. They have security teams. They have compliance frameworks. They have SOC 2 certifications and ISO 27001 audits and penetration testing schedules.
None of it mattered.
The race to ship AI features is relentless. Every quarter, every product announcement, every competitor breathing down your neck. Microsoft can’t let Google pull ahead. Google can’t let Anthropic capture the enterprise market. Amazon can’t be seen as the slow mover. The incentives push one direction: faster, more capable, more integrated.
And the vulnerabilities aren’t the ones the compliance frameworks were built to catch. These aren’t SQL injections or unpatched servers or misconfigured firewalls. This is a new class entirely: prompt injection, context manipulation, trust boundary violations. The old playbooks don’t have chapters for ‘what happens when your AI assistant can be turned by a hidden instruction in a markdown file.’
So you get Reprompt, a single click, no authentication, no privilege escalation, just a legitimate Microsoft URL that opens a door that should never have existed. You get Amazon Q shipping malicious code in an official release because someone got admin credentials through a pull request. You get DeepSeek leaving a million chat logs on an unprotected database.
These aren’t edge cases. They’re the predictable result of bolting AI capabilities onto systems faster than security practices can evolve to contain them.
The compliance badges still shine. The breaches still happen.
The Shadow AI Reality
Here’s the statistic that should terrify every security team: while only 40% of companies have official AI subscriptions, employees in 90%+ of organisations regularly use personal AI tools for work, often invisible to IT.13
Microsoft’s Work Trend Index reported that 78% of AI users bring their own tools to work through personal accounts.14
IBM’s 2025 Cost of Data Breach Report quantified the damage: 1 in 5 organisations reported a breach due to shadow AI, with these breaches costing $670,000 more than traditional incidents. Of organisations that experienced AI breaches, 97% had no AI access controls.15
Read that again: 97% of breached organisations had no AI access controls.
The enterprise tier that promises not to train on your data? Your developers aren’t using it. They’re using the free tier on their personal accounts. The one that does train on everything.
The Tier Reality
Free and individual tiers may use your code for training. Enterprise tiers promise they won’t. Free tiers retain your prompts. Enterprise tiers claim they don’t. Enterprise gets the security certifications, the data residency options, the illusion of control. Free tier gives you nothing but convenience.
When your contractor pastes production code into the free tier of ChatGPT, that code potentially becomes training data. Forever. For everyone.
When your developer debugs a client issue using their personal Claude account, that client data is now in Anthropic’s infrastructure, governed by consumer terms of service, not enterprise agreements.
The enterprise tier exists. Your people aren’t using it.
The Compliance Gap Nobody Talks About
The slow creeping dread started when I watched Claude iterate through a codebase.
Fast. Efficient. Reading files before I could register what it was accessing. And I started to think: what if it reads something it shouldn’t? What if client data ends up on US servers before I even notice?
Surely there’s a data processing agreement covering this. Surely paying for Pro means professional-grade compliance.
I went looking. Multiple documents. Privacy centres. Terms of service buried three clicks deep.16
Here’s what I found: Pro and Max are consumer products. No DPA. No Standard Contractual Clauses. No GDPR compliance framework. The page that explains this doesn’t even make it obvious. You have to follow a link that says ‘for consumer products, see here’ before you discover that you’re the consumer.17
If you want a DPA, you need Enterprise. Enterprise pricing? Reports suggest $60/seat, minimum 70 users, annual contract.18 That’s $50,000 a year just to get the paperwork that lets you use AI legally with client data.
For a small team, that’s not a budget line. That’s a decade of AI spend.
There are middle-ground options. Team tiers exist. But dig into the detail and the trade-offs become stark. Claude Team Standard doesn’t publish its usage limits. Team Premium costs $150/seat and gives you less usage than the $100/month Max plan you can’t legally use with client data.19
Under GDPR Article 28, using a processor without a valid DPA is a direct legal violation. Fines up to €10 million or 2% of annual worldwide turnover. The ICO considers missing DPAs ‘easy to identify and often systemic’ in audits.20
The gap isn’t theoretical. It’s a compliance void that millions of small businesses are sitting in without knowing it.
The AI-Generated Code Problem
Even if you solve the data leakage problem, there’s another issue: the code itself.
Veracode’s GenAI Code Security Report analysed 100+ LLMs across 80 coding tasks and found that only 45-55% of AI-generated code was secure, meaning 45% introduced known vulnerabilities. Java showed the highest failure rate at 70%+.21
Critically: ‘newer and larger models don’t generate significantly more secure code than their predecessors.’
Apiiro’s analysis of Fortune 50 enterprises found that by June 2025, AI-generated code introduced 10,000+ new security findings per month, a 10x spike in just 6 months. While shallow syntax errors dropped 76% and logic bugs fell 60%, privilege escalation paths jumped 322% and architectural design flaws spiked 153%.22
CodeRabbit’s December 2025 study found AI-generated pull requests have 1.7x more issues than human-authored code, with 2.74x more XSS vulnerabilities.23
You’re not just risking data leakage. You’re importing vulnerabilities at scale.
What’s Left
Big Tech’s answer isn’t available to you. You can’t build your own LLM infrastructure.
The banks’ answer isn’t available to you. You don’t have $18 billion and an army of security engineers.
The compliant middle-ground costs more than most small teams can justify, and even then, the usage limits make heavy AI work impractical.
And whatever technical solution you implement, 90% of your people will route around it if it’s slower or less capable than the tool they can access for free in a browser tab.
The tools themselves keep failing. Reprompt. Amazon Q. DeepSeek. The vulnerabilities keep coming, and the patches keep following, and the next one is already being discovered.
The stranger isn’t just in your house. The stranger is in everyone’s house. And nobody’s figured out how to get them out.
The Questions Keep Getting Harder
After reading this, go back to the questions from Part 1 and add a few more.
Which tier of AI service are your developers actually using? Do you have any visibility into personal accounts? Samsung leaked three times in twenty days after lifting their ban. How confident are you in your controls?
If your AI provider is breached tomorrow, what’s your exposure? Do you have a valid DPA for every AI tool touching client data?
The industry hasn’t solved this. The biggest companies in the world are either banning external tools entirely or spending billions on containment.
What’s your plan?
Next in the Series
Part 3: The Human Factor
You’ve built the perfect system. Air-gapped. Firewalled. Policies signed. Training complete.
And then Dave Ops has a deadline.
Why the best security architecture in the world can’t survive a stressed developer with a browser tab. Why technical controls fail when humans are incentivised to bypass them. And why culture might be the only defence that actually works.
Coming next week.
Sources
Apple banned ChatGPT and GitHub Copilot in May 2023; partnered with Anthropic for internal Claude-powered platform in May 2025. https://techcrunch.com/2023/05/19/apple-reportedly-limits-internal-use-of-ai-powered-tools-like-chatgpt-and-github-copilot/
↩ return to articleSundar Pichai, Alphabet Q1 2025 earnings call, April 24, 2025. https://analyticsindiamag.com/ai-news-updates/sundar-pichai-says-over-30-of-code-at-google-now-ai-generated/
↩ return to articleSatya Nadella, fireside chat at Meta’s LlamaCon, April 29, 2025. https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft-code-is-written-by-ai.html
↩ return to articleAmazon Q Developer savings of 450,000 hours and $260 million in efficiency gains. https://aws.amazon.com/blogs/devops/amazon-q-developer-just-reached-a-260-million-dollar-milestone/
↩ return to articleMarc Benioff, Salesforce Q4 FY2025 earnings call, February 26, 2025. https://www.salesforceben.com/salesforce-will-hire-no-more-software-engineers-in-2025-says-marc-benioff/
↩ return to articleJPMorgan Chase LLM Suite won American Banker 2025 Innovation of the Year Grand Prize. $18 billion technology budget; 200,000+ employees on platform. https://www.jpmorganchase.com/about/technology/news/llmsuite-ab-award
↩ return to articleGoldman Sachs GS AI Assistant firmwide launch, June 23, 2025. CIO Marco Argenti announced deployment to all 46,500 employees. https://fortune.com/2025/06/24/goldman-sachs-internal-ai-assistant/
↩ return to articleBank of America AI strategy; Jorge Camargo confirmed generative AI excluded from customer-facing Erica: “We can’t afford to be 90% right.” https://www.bankingdive.com/news/bank-of-america-erica-virtual-assistant-ai/758519/
↩ return to articleVaronis Threat Labs, “Reprompt: The Single-Click Microsoft Copilot Attack,” January 14, 2026. https://www.varonis.com/blog/reprompt
↩ return to articleAmazon Q Developer security incident (CVE-2025-8217), AWS Security Bulletin, July 2025. https://aws.amazon.com/security/security-bulletins/AWS-2025-015/
↩ return to articleWiz Research, “DeepSeek Database Leak,” January 29, 2025. https://www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak
↩ return to articleSamsung ChatGPT data leaks, April 2023. Three incidents within 20 days of lifting ban. https://techcrunch.com/2023/05/02/samsung-bans-use-of-generative-ai-tools-like-chatgpt-after-april-internal-data-leak/
↩ return to articleMIT Project NANDA, “The GenAI Divide: State of AI in Business 2025,” July 2025. https://fortune.com/2025/08/19/shadow-ai-economy-mit-study-genai-divide-llm-chatbots/
↩ return to articleMicrosoft Work Trend Index 2024. Survey of 31,000 workers across 31 countries. https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at-work-is-here-now-comes-the-hard-part
↩ return to articleIBM Cost of a Data Breach Report 2025. https://www.ibm.com/reports/data-breach
↩ return to articleAnthropic Privacy Centre, DPA documentation. https://privacy.claude.com/
↩ return to articleAnthropic Privacy Centre, Consumer products. https://privacy.claude.com/en/collections/10663362-consumers
↩ return to articleClaude Enterprise pricing not publicly disclosed. Industry estimates suggest ~$60/seat with 70-user minimum.
↩ return to articleClaude pricing tiers comparison based on Anthropic documentation, January 2026.
↩ return to articleICO guidance on GDPR Article 28 Data Processing Agreements. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/accountability-and-governance/guide-to-accountability-and-governance/contracts/
↩ return to articleVeracode, “2025 GenAI Code Security Report.” https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/
↩ return to articleApiiro, “4x Velocity, 10x Vulnerabilities,” September 2025. https://apiiro.com/blog/4x-velocity-10x-vulnerabilities-ai-coding-assistants-are-shipping-more-risks/
↩ return to articleCodeRabbit, “State of AI vs Human Code Generation Report,” December 2025. https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report
↩ return to article