AI Governance Challenges in 2026: Why It Starts With People

Most AI governance programs start with the technology: which models are approved, what an agent can access, how outputs are logged. Those questions matter. But in most companies the larger risk sits with people. It is the customer list pasted into a chatbot, the contract uploaded to a personal account, the personal AI subscription connected to the company ERP, and the AI-written answer sent to a client without anyone checking it.

The numbers back this up. IBM’s 2026 Cost of a Data Breach Report found that incidents involving employees’ shadow AI more than doubled in a year, to 43% from 20%, and cost $5.39 million on average against $4.63 million for other incidents. This article covers the AI governance challenges we see most often in 2026, starting with the people side, and the steps that make a real difference.

1. AI arrives before the controls do

In most companies, the order is the same: employees start using AI, then leadership writes a policy, and the controls come last, if at all.

The research shows how common this is. In ISACA’s 2025 AI Pulse Poll of more than 3,000 digital trust professionals, 81% said employees in their organization use AI whether it is permitted or not, while only 28% said their organization had a formal, comprehensive AI policy. Gartner found that 69% of cybersecurity leaders suspect or have evidence that employees use prohibited public generative AI tools. And in the KPMG and University of Melbourne study of more than 48,000 people, 44% of employees said they have used AI in ways that go against their organization’s policies, and only 34% reported any organizational policy or guidance on generative AI.

The cost shows up later. Among organizations in IBM’s 2026 report that had an AI-related incident, 68% lacked AI governance policies, and only 19% of all organizations reported coordination between their governance and security teams.

The fix is not to stop AI use until a perfect program exists. It is to put a small set of controls in place early: an approved tool, clear rules on data, and a way to see what is happening.

2. Sensitive data goes into prompts every day

Employees use AI because it helps them work faster. To get a useful answer, they give it context: a customer email, a spreadsheet, a block of source code, a draft contract.

Cyberhaven’s 2026 AI Adoption and Risk Report, based on data movement at 222 companies, found that 39.7% of the data employees moved into AI tools was sensitive, and that the average employee entered sensitive data into an AI tool about once every three days. The same report found that 32.3% of ChatGPT use and 24.9% of Gemini use at work happened on personal accounts.

Personal accounts are more than a visibility gap. Consumer AI accounts follow consumer terms, which decide how long conversations are kept and whether they can be used for training, and those terms change. In 2025, for example, Anthropic moved consumer accounts that allow model training to a five-year retention period. Company data in a personal account sits outside your retention policy, your legal hold and your ability to delete it, and when the employee leaves, it leaves with them.

3. A big-name enterprise AI plan is not a governance program

A common next step is to buy an enterprise plan from a well-known AI vendor and consider governance done. Those plans have improved a lot. Business data is not used to train models by default, and the top tiers offer audit logs, compliance APIs and ways to block personal accounts on the company network. That is a real step forward from consumer accounts.

But a big name doesn’t mean a complete set of controls, and the industry is still catching up. When we reviewed the vendor documentation for the major enterprise AI plans in November 2026, we found:

• Controls differ by vendor and by tier. Some vendors offer native data loss prevention (DLP) for prompts, such as Microsoft Purview for Microsoft 365 Copilot and Google Workspace DLP for Gemini, each with its own limits on what it covers. Others rely on third-party DLP tools or a security service the customer runs. Many of the strongest controls are only available on the top tier.
• Blocking is not the same as masking. In the documentation we reviewed, none of the major plans masks sensitive data and then sends the cleaned prompt to the model. They block the prompt or allow it. That leaves employees with a choice between a refusal and a workaround.
• Several models mean several sets of controls. Most companies now use more than one AI provider. Each has its own admin console, its own logs and its own gaps, so nobody sees the full picture of what data reached which model.

This is why many companies put one governance layer in front of all their AI models: a private AI platform where the company decides which models are approved, sensitive data is masked before any model sees it, and every request is logged in one place. Teams get the best models from several vendors, and IT keeps one set of rules.

4. Secure connectors, but whose AI is on the other end?

AI is no longer only a chat window. Through the Model Context Protocol (MCP), assistants such as Claude and ChatGPT connect directly to business systems. Many vendors now host their own MCP servers. NetSuite’s AI Connector Service, which we covered in our SuiteWorld 2026 recap, is a good example.

As a connector, it is well built. It uses OAuth sign-in, runs under the user’s NetSuite role with all of its permissions and restrictions, can’t be used with administrator roles, is off until an admin grants access, and logs each call with the user’s email.

The gap is on the other side of the connection. NetSuite can see that a request came from Claude and which employee authorized it. It can’t tell whether that was the company’s managed Claude workspace or a personal subscription, and it doesn’t check the device. Unless sign-in to NetSuite is protected by single sign-on with device checks, an employee could connect a personal AI account on an unmanaged, unpatched laptop, and the activity would look like ordinary approved use. Oracle’s own guidance tells customers to keep a list of approved AI agents and not let users connect others without approval, which leaves enforcement to each customer.

NetSuite is one example, not an exception. The same question applies to every system you open to AI assistants. The controls that close the gap are known:

• Single sign-on with device checks for the roles that can use the connector, so authorization from an unmanaged device fails.
• A narrow, dedicated role for AI access, read-only where possible.
• Approval for new AI client applications, and disabling the ones you don’t use.
• Regular review of authorized applications and connector logs. NetSuite keeps the connector log for 21 days, so export it.
• One governed path for AI access, so the company decides which AI tools reach which systems.

5. AI output leaves the building without review

Most governance programs focus on what goes into AI. Fewer look at what comes out of it, and that is where many of the costly mistakes happen.

The KPMG study found that 66% of employees use AI output without evaluating it, and 56% have made mistakes in their work because of AI. More than half don’t disclose when they used AI and have presented AI-generated content as their own. The consequences can be public: in 2025, Deloitte partly refunded an Australian government report that contained citations invented by AI.

A simple rule closes much of this gap: any AI-assisted work that goes to a client, a regulator or the public gets a human review first, by someone accountable for the result.

6. AI you didn’t buy, and agents you didn’t scope

Not all AI arrives through a new purchase. Vendors add AI features to the CRM, ERP, collaboration and HR tools you already pay for, sometimes switched on by default or with new terms on how your data may be used. These changes rarely go through procurement or security review, because nobody bought anything. Review vendor terms for AI and data use at every renewal, and keep a list of the AI settings in your main systems and who owns each one.

AI agents need the same discipline as any privileged account. IBM’s 2026 report found that 92% of organizations with an AI-related incident lacked proper AI access controls. Each agent should get only the access its task needs, through its own credentials. Sensitive steps such as payments, deletions and messages to customers should wait for a person. Content from email, tickets and the web should be treated as data, never as instructions, and every action should be logged.

7. The rules changed in 2026

AI regulation moved this year, and many articles still quote the old dates.

• EU AI Act. The Digital Omnibus, in force since July 27, 2026, moved the deadline for high-risk AI systems in areas such as hiring, credit and education to December 2, 2027, and for AI built into regulated products to August 2, 2028. The transparency duties, such as telling people they are dealing with AI, have applied since August 2, 2026. Rules for general-purpose AI models did not change.
• Colorado. In May 2026, Colorado repealed and replaced its 2024 AI Act with a narrower law on automated decisions that significantly affect people. It takes effect on January 1, 2027, with notice and record-keeping duties, and only the state Attorney General can enforce it.
• Other states. Several other states have AI laws in force or taking effect in 2027. Check which rules apply to where you operate and whom you serve.

For most mid-market companies in the US, the practical message is the same: know where AI makes or supports decisions about people, keep records, and be ready to explain those decisions.

Where to start

You don’t need a large program to make progress. These steps cover most of the risk above:

1. Take an inventory. List the AI tools in use, approved or not, the AI features in your main business systems, the MCP connectors you have enabled, and the agents you run.
2. Write one page of rules. Which tools are approved, which data can go into them, and how AI output is reviewed before it goes out.
3. Put one governance layer in front of your models. Company sign-in, approved models, masking of sensitive data and one log across every provider.
4. Lock down connectors. Single sign-on with device checks, narrow roles and regular review of which AI applications are authorized.
5. Set a review rule for external work. AI-assisted work for clients, regulators or the public gets a human review first.
6. Review on a schedule. AI tools, features and rules change every quarter, so the program should too.

Frameworks such as the NIST AI Risk Management Framework and ISO/IEC 42001 give this work a structure your auditors and customers will recognize.

How Boston SoftDesign helps

Many of the gaps above don’t show up in a policy review. They sit in how tools, accounts and connectors are actually set up. Our AI governance services start there: we find where AI is used and how it connects to your systems, identify the risky patterns, and put secure, governed setups in place, from policies and approvals to the technical controls that enforce them. We help you get in line with the NIST AI Risk Management Framework, ISO/IEC 42001 and the regulations that apply to you.

For the governance layer in front of your models, Ally Platform gives employees approved models from several providers in one company workspace, masks sensitive data before it reaches a model, and logs every action. Talk to our team about where AI is used in your company today.

Sources

• IBM, Cost of a Data Breach Report 2026, as summarized by Baker Donelson (July 2026): bakerdonelson.com
• ISACA, 2025 AI Pulse Poll (June 2025): isaca.org
• Gartner, GenAI blind spots for CIOs (November 2025): gartner.com
• KPMG and University of Melbourne, Trust, attitudes and use of artificial intelligence (2025): kpmg.com
• Cyberhaven, 2026 AI Adoption and Risk Report (February 2026): cyberhaven.com
• The Register, Anthropic consumer data retention (August 2025): theregister.com
• Microsoft Learn, DLP for Microsoft 365 Copilot: learn.microsoft.com
• Google Workspace, DLP for Gemini: knowledge.workspace.google.com
• OpenAI, ChatGPT Enterprise security and privacy: cdn.openai.com
• Anthropic, inference hooks for Claude Enterprise: platform.claude.com
• Oracle NetSuite, AI Connector Service risks and best practices: docs.oracle.com
• Oracle NetSuite, AI Connector Service FAQ: docs.oracle.com
• The Register, Deloitte refund over AI-generated report (October 2025): theregister.com
• K&L Gates via The National Law Review, EU Digital Omnibus on AI (July 2026): natlawreview.com
• Davis Wright Tremaine, Colorado AI Act repeal and replacement (May 2026): dwt.com