Before Judging AI, Make Operational Promises for Your Data

When an intelligent assistant gives a wrong answer, the common reaction comes quickly: “The model is not good.” Sometimes the model really is not good, the prompt is poorly written, or we have given it a problem that this tool cannot solve in the first place. But in many projects, the model becomes the main suspect too soon, while it is working with data whose last update is unknown, whose missing parts are unclear, and whose business definition is unsettled.

This problem takes a familiar form inside organizations. Someone asks why line downtime has increased. The intelligent agent connects to files, dashboards, an ERP, or a data warehouse and produces a polished answer. Then it turns out that the latest shift records have not arrived yet, some downtime events were logged under “other,” or production and maintenance do not share the same definition of downtime. The model’s answer may be excellent linguistically, but operationally it stands on unstable ground.

We should not end the discussion with the broad statement that “our data is not clean.” That sentence usually makes nobody responsible for anything and helps no decision. Clean data is a vague wish. What we need is a clear, observable commitment about the data that will feed AI.

This is where the idea of an SLO becomes useful: a defined operational target for a service level. Software teams have used this language for years around availability, response time, and service error rates. The same logic can be applied to a data channel, not to pretend that data is a software service, but to move its condition out of guesswork and complaint.

A Data SLO means that before expecting reliable answers from AI, we define the minimum operational condition that the data must meet. The commitment can cover freshness, record completeness, error rate, data lineage coverage, and the time to resolve defects. The important point is that every item must be measurable, observable, and assigned to a clear owner.

Freshness is the simplest place to start. If your assistant is expected to answer questions about inventory, orders, line downtime, or customer status, you need to know how much delay is acceptable before the data can be used. You do not need to pursue near-zero latency on day one. You need to understand what the real decision requires. Data from the previous day may be enough for a management report. For a response to line downtime, that same delay can make the decision useless.

A freshness SLO can be stated simply: line-downtime data must be usable in the analytical source within a defined period after an event is recorded. The value of this sentence is not in a seemingly precise number. It is in the fact that you can later see whether the commitment was met. If it was not, the intelligent agent should not speak with the same confidence about the current state of the line.

Completeness is the next step. In practice, not every column matters equally. A table may contain thousands of fields, but for downtime analysis only a few may be critical: start time, end time, line, reason, shift, and approval status. If one of these is empty, the record may technically exist while remaining incomplete for decision-making.

So instead of measuring the quality of an entire table in the abstract, define a threshold for critical fields. State what share of records in a period must contain those fields before the data is considered reliable enough for analysis. Again, the goal is not to create a decorative standard. The goal is for both AI and people to know where certainty should stop.

The third item is the record error rate. An error is not only a blank value. An end time before a start time, a reason code that is no longer valid, a negative value for an implausible quantity, a duplicate event, or an invalid combination of two statuses are all examples of errors. These rules must come from the reality of the process, not the preference of the data team. The person who understands the process needs to sit beside the person who sees the data.

The fourth SLO, often noticed too late, is lineage coverage. When an agent says, “The main reason for the increase in downtime was this issue,” it should be possible to understand which source, table, transformation, and definition that conclusion relies on. If the route from data to answer is unclear, reviewing the answer becomes an argument. Every department is right from its own perspective, and nobody can unpack the issue.

You do not need to build a massive data-lineage architecture from the beginning. For the most important source connected to AI, start with a simple map: where the data comes from, who records it, what transformations are applied, where it is stored, and which business definition is valid for interpreting it. The map needs to be practical enough to shorten the investigation when disagreement arises.

The fifth SLO is the committed time to resolve a defect. Many teams have alerts, dashboards, and even visibility into errors, but it is unclear who must do what after a defect is found and by when. An alert without an owner and a deadline is just another sound in the organization. For every important category of defect, define an operational owner and an escalation path.

This is usually where resistance begins. Departments say that data belongs to everyone, or, conversely, they place the entire responsibility on the technology team. Both views are risky. The owner of the recording system, the owner of business meaning, the owner of data transfer, and the AI consumer have different roles. One person may not hold all of them, but none of them should be left ownerless.

After defining the SLOs, the more important part begins: AI behavior when they are violated. If the data is not fresh, the assistant should not pretend that it knows the live situation. If completeness falls below the threshold, it should present the answer with an uncertainty warning, limit the scope of analysis, or refuse to draw a conclusion altogether. This retreat is not a weakness of the system; it is a sign of maturity.

For some decisions, the best AI response is simply to ask a clarifying question. It might say that the latest shift data is not yet complete and cannot support a reliable analysis; should it continue based on the last confirmed period? For more sensitive decisions, analytical output should stop and the issue should be referred to a person. These policies must be set before a crisis, not after a troublesome answer.

If you work with n8n or similar automation tools, the initial implementation of this idea is not complicated. One workflow can check the time of the latest update, calculate the share of missing critical fields, run record-error rules, and store the result in a monitoring table. Another workflow can notify the data owner when a threshold is crossed while updating a status flag for the intelligent assistant.

The important point is not to place quality control only at the end of the line. If the assistant discovers a data problem only when it is answering a question, you have noticed too late. Monitoring should be independent and continuous, much like watching the condition of a service before a customer calls. AI should consume this status, not be its only detector.

For the first week, you do not need to include every data source in this program. Choose one source that feeds an important decision and is already connected to AI or analytical reporting today. It might be line-downtime data, order status, inventory for a sensitive product group, or after-sales service cases. If you try to cover everything from the start, you will probably build a heavy governance project that nobody uses.

For that one source, write three to five SLOs: the maximum acceptable delay, the minimum completeness of critical fields, the ceiling for defined rule-based errors, the minimum coverage of the route from origin to consumption, and the resolution time for a breach. Next to each one, write how it is measured, who owns the action, and exactly how AI behaves when it is breached. If you cannot write these three things, you do not have an SLO yet; you only have a good intention.

After one or two weeks, instead of asking what percentage accurate the AI was, look at this first: how often were answers given using data outside its commitment? How often did the assistant issue the right warning? Which defect repeated most often? Who resolved it, and how long did it take? These questions may lack the appeal of demonstrating an intelligent answer, but they determine whether the system becomes reliable in operations.

Model accuracy still matters. Answer evaluation, prompt design, access control, human review, and tool selection all have their place. But when the input carries no commitment, evaluating the model is like judging a driver behind the wheel of a car with a broken dashboard. The driver may sometimes steer correctly, but you do not know what your trust is based on.

An organization that wants to bring AI seriously into its work should start with an uncomfortable question: has anyone made an operational promise for the data that feeds the model? Not a broad promise to improve, and not a slide about data quality, but a measurable commitment with an owner, an alert, and a defined response. Until the answer is clear, every conversation about model intelligence is a little premature.

AI systems often produce poor answers for reasons that begin before the model is even involved. The real work is to make the condition of input data visible, measurable, and owned.