The enterprise software market is functioning like a protection racket. Incumbent vendors, leveraging the lock-in of their core platforms, are now charging clients a mandatory premium for AI features that are, by their own admission, not yet fit for purpose. This is an “AI Tax,” compelling you to fund their research and development (or just increase the margins they report to shareholders) under the guise of an upgrade.
This edition of The AI Equilibrium provides a framework for navigating this environment. We will explore the new legal fronts opening up around training data, and provide a pragmatic, evidence-based method for scrutinising third-party AI systems. The goal is to move beyond the marketing narrative and assess the operational reality.
The Briefing#
In 2025 we have been seeing the aggressive repositioning of enterprise AI around the concept of “agents.” Major software vendors are proclaiming a new era of automation, but a closer look at the facts reveals reality: customers are being asked to pay a significant premium for technology that is demonstrably immature. This premium functions as a mandatory “AI Tax,” forcing a captive customer base to subsidise the vendors’ ongoing research and development.
For example Salesforce has retired its “Einstein” brand in favour of a new “Agentforce” platform, accompanied by a 6% price increase for core customers while internal Salesforce research shows its new AI agents achieve a mere 58% accuracy on single-step tasks. The promise of automation is an illusion when the tool fails nearly half the time. The problem compounds with complexity. For a process involving just five steps, the probability of a successful outcome plummets to a dismal 6.5% (it’s an exponential phenomenon, assuming independent success of each step). This creates a new, expensive process of constant human verification.
Oracle is integrating the latest models from OpenAI and Google, SAP is promoting its “Joule Agents” across its entire suite, and ServiceNow is forecasting robust growth driven by its premium-priced “Pro Plus” and “Now Assist” AI products. The common thread is a strategy of leveraging entrenched market positions to extract capital for an experiment. They are selling the promise of a future state, and you are paying for it today.
I’m not saying that the technology will not bring the results you expect, maybe with time it will — it’s just not going to happen within the next couple of quarters.
Simultaneously, a new legal front has opened, targeting the very fuel of AI: training data. A wave of class-action lawsuits is challenging the widespread and often opaque practice of using customer and employee data for the secondary purpose of improving AI models.
The August 2025 lawsuit filed against Otter.ai, an AI transcription service, is a good example. The complaint alleges that Otter used the contents of private conversations to train its models without obtaining specific, informed consent from all participants, acting as an “unauthorized third party eavesdropper.” The legal theory is that general consent to use a service does not automatically extend to consent for one’s data to be used as a training set for the vendor’s commercial benefit.
This creates a significant and underappreciated liability. When you deploy a customer-facing AI tool, you are creating a data pipeline that may flow directly into your vendor’s model training architecture. If that vendor uses customer interaction data to retrain its global models, you could be held co-liable for supplying that data without securing explicit, specific, and unambiguous consent for the express purpose of AI training. This is a new and distinct category of data processing that standard privacy policies were not designed to cover. For any senior leader, “data for AI training” must now be treated as a high-risk activity, requiring its own dedicated governance framework and robust consent mechanisms. Failure to make this distinction is to invite litigation.
Procuring a Third-Party AI System: A Framework#
Procuring a third-party AI system is more complex than licensing a CRM. Now you are integrating a complex, semi-autonomous (this is what’s new!) piece of industrial machinery into core business processes. Traditional vendor questionnaires are inadequate for this task. and an evidence-based assessment of a partner’s engineering discipline is needed. This assessment rests on four pillars.
Pillar I: Foundational & Corporate Integrity#
Before scrutinising algorithms, one must establish that the vendor operates a stable and secure organisation. An innovative model from a company with poor security hygiene is an unacceptable liability. This requires a review of financial statements, SOC 2 or ISO 27001 certifications, and documented incident response plans.
Pillar II: Data Governance & Provenance#
Data is the feedstock of AI. The vendor must provide explicit, auditable proof of their legal right to use all data in the training set. This requires demanding artefacts like a “Datasheet for Datasets”—a comprehensive document detailing the motivation, composition, and collection process for every dataset used. Other necessary documents include the data provenance reports, and any Data Protection Impact Assessments (DPIAs).
Pillar III: Model Transparency & Robustness#
This pillar demands proof of how a model functions and where it fails. The central artefact is the “Model Card,” a document detailing the model’s architecture, training data, performance metrics, and intended use cases. This shoulf be supported by bias audit reports with performance metrics broken down across demographic subgroups, and documentation of any explainability features.
Pillar IV: Operational Security & Regulatory Readiness#
An AI model is a dynamic system operating in a hostile environment. A vendor must prove the model is resilient to attack and compliant with emerging law. This requires seeing summaries of red-teaming and penetration tests against AI-specific attacks like prompt injection. It also requires reviewing their MLOps policies for managing model drift and their formal compliance statement for regulations like the EU AI Act.
We are still early in the development cycle of AI, so many vendors will not be able to provide all of the above documents, and in my opinion, this should not automatically disqualify them — but it should inform the buyer, what areas of application are acceptable (i.e. internal processes with human oversight vs. automated sales and customer service)
A Field Guide to Ethics Washing#
“Ethics washing” is creating a superficial impression of good governance without the underlying processes. Certain phrases should trigger scepticism, as they are often used to obscure a lack of substance.
Vague & Unverifiable Claims#
Terms like “AI-powered,” “ethical by design,” and “trustworthy AI” are functionally meaningless without specific proof. An “AI-powered” workflow might be a simple set of if-then rules; true AI should be integral to the product’s core function.
Focus on Intent, Not Outcome#
Statements about a “commitment to fairness” are irrelevant. What matters are the systems and audits that demonstrate fair outcomes. A vendor’s good intentions are not a defence in front of a regulator.
Anthropomorphism#
Describing an AI as “understanding” or “thinking” is a marketing tactic to obscure the statistical nature of the technology. It signals a superficial grasp of the technology or an attempt to mislead.
Examples of Superficial Governance#
An Ethics Board with no Real Power: A vendor announces an “AI Ethics Advisory Board” populated with distinguished figures. The red flag is when the board has no actual authority, its recommendations are non-binding, and its proceedings are opaque. It is a public relations shield, not a governance mechanism.
Misleading “AI-Powered” Claims: The U.S. Securities and Exchange Commission (SEC) fined two investment firms, Delphia and Global Predictions, for making false AI claims. Neither could substantiate their assertions, resulting in $400,000 in civil penalties. Regulators are watching.
The “GPT Wrapper”: A vendor claims a proprietary AI solution, but has only built a user interface on top of a third-party model from a provider like OpenAI. These vendors have little control over the model’s behaviour, training data, or security. It does not mean their products should not be procured and used, but you need to understand who’s behind and how the actual model vendor trains and manages the models.
⠀Questions To Consider
These questions can support a vendor’s approach to AI.
1. The Data Provenance Challenge: “Can you provide a complete audit trail showing the legal basis for every piece of training data? How do you handle data subject access requests?”
2. The Model Drift Reality Check: “How will you notify us if model performance degrades? What constitutes a material change requiring our consent? Can you guarantee consistent outputs for identical inputs?”
3. The Liability Stress Test: “If your AI makes a decision that causes a regulatory violation, what is your liability coverage? Can you provide evidence of insurance that covers AI-specific risks?”
4. The Competitive Intelligence Probe: “How do you prevent our proprietary data from influencing models used by your other customers, including our competitors?”
Let me repeat — we can’t yet expect positive and satisfying answers to all the above questions from vendors, and… that’s OK — this just shows the development state of the GenAI technology in 2025. The answers you get should be considered something that automatically disqualifies vendors, but they should tell you a lot about what applications AI is currently applicable for.
The core of this work is about understanding risk, managing it intelligently, and creating a defensible, evidence-based process. This allows an organisation to innovate, moving from blind trust in a vendor’s promises to a state of earned trust, verified by auditable proof.
Until next time, build with foresight. Krzysztof