AI Is Getting Companies Into Trouble, But Not For The Reason You Think😬

AI training data and blind AI adoption are getting companies into trouble. Many people assume AI itself is the problem, but in many cases, the real issue is failing to understand how the technology works before trusting it with important decisions. In other cases, the root cause is still human error or human bias.

Let's use the lawsuit against Workday involving alleged applicant discrimination as an example. While the allegations have not been proven, the case provides an opportunity to understand the types of issues investigators may examine in situations like this.

If a company is training or fine-tuning AI using historical hiring decisions or similar organizational data, the system may learn patterns that reflect biased, outdated, or even unlawful practices if those patterns aren’t identified and addressed. For example, if a company has historically rejected applicants whose employment history or graduation dates suggest they are older, candidates with non-white sounding names, or applicants without degrees, those patterns may be reflected in its historical data. If that data is then used to train or fine-tune an AI system without proper safeguards, the system may learn those patterns and apply them at scale.

If a company is adopting a third-party AI system instead, the focus shifts to how that system was configured, tested, monitored, and relied upon. While the company may have no control over how the AI was trained, adopting it doesn’t eliminate responsibility. Organizations should perform due diligence to understand how the system was built, whether it has been tested for bias and reliability, and what guardrails and accountability measures exist to mitigate harm. Without that due diligence, a single AI tool can scale harmful outcomes across thousands of organizations.

Completely eliminating bias from technology built by humans is nearly impossible, which is why intentionally designing systems to identify, mitigate, and monitor it is imperative.

Then there's human error and bias, which still exist whether AI is involved or not. Even when AI is designed to assist with decision-making, humans often have the final say. If those decision-makers are biased, the outcome can still be discriminatory regardless of what the AI recommended. We've already seen the harm humans can cause long before AI existed. While human bias can be mitigated through thoughtful processes, guardrails, and accountability, it still can't be completely eliminated.

AI may seem like the bad guy, but AI doesn't understand fairness, ethics, or discrimination, it learns statistical relationships in data. Without explicit guardrails made by humans, it has no way of knowing whether a historical pattern reflects good business decisions or practices that should never be repeated.

AI shouldn't be judged by whether it's perfect, because no human-built system ever will be. AI should be judged by whether it's demonstrably fairer, more transparent, and more accountable than the process it replaces. Even then, the responsibility doesn't stop once an AI system is deployed. Organizations should continuously test, monitor, and improve these systems as new risks surface.