Our team will be out of office on Friday, May 1, 2026. We’ll be back and ready to assist you starting Monday, May 4th.

If Nobody Opens the Dashboard Anymore, How Do I Know My AI Automations Are Still Working?

Contents

You handed the work to AI and the screens stopped getting opened. This article answers the question that follows: when nobody is looking at the dashboard, what is actually watching your business, and how do you build the noticing back in on purpose?

Suggested SEO meta title tag: AI Automation Monitoring: How to Know Your Automations Still Work

The Boat Was Fine for Thirty-Four Hours

On June 10, 1995, the cruise ship Royal Majesty ran aground on the Nantucket shoals with more than a thousand people aboard.

Nothing had broken on the bridge. The autopilot was working perfectly. It had been steering for thirty-four hours toward a position the ship had not occupied since fifty-two minutes out of Bermuda, when the GPS antenna cable came loose and the receiver quietly switched to dead reckoning. By the time the hull hit sand, the ship was seventeen miles west of where everyone believed it was.

Here is the direct answer to the headline. If nobody opens the dashboard anymore, you do not know your automations are working, and you will not find out until a customer tells you. The only thing that can tell you is something you deliberately built to speak up without being asked.

That is the thesis. When you move a process into AI, you move two things: the task, and the daily habit of glancing at a screen and thinking "huh, that number looks wrong." The first transfers. The second evaporates, and almost nobody rebuilds it.

Candidly, this is the failure mode showing up in more entrepreneurs' businesses than any prompt problem or tool problem. Not AI doing something dramatic and wrong. AI doing nothing at all, for weeks, while everybody assumes it has got this.

Key Takeaways

  • Automating a workflow also automates away the daily habit of looking at it, and that habit does not restore itself.
  • Human factors research going back to 1983 shows people cannot reliably monitor a system that almost never misbehaves.
  • The dangerous failure is not the loud one; it is the automation that reports success while producing nothing.
  • You have to build a noticing layer on purpose: something that tells you what did not happen, unprompted.
  • If your answer to "what is watching this?" is "the AI has it," nothing is watching the watcher.

The Problem: I Have Automations I Have Not Looked At in Weeks

I will go first, because it would be dishonest not to.

Right now, as I write this, there are automations running in my business that I have not laid eyes on in weeks. I could not tell you from memory whether every one of them ran yesterday. I built them, tested them, watched them work, felt the relief, and stopped visiting. That is not carelessness. That is a normal human being with a working automation.

Rachel Woods wrote about this from the other direction this week in AMP, and she named the mechanism better than I have seen anyone name it. HubSpot's co-founder recently said you will not need to log in to use their software, which most software companies would never admit. Her read: "the real story is a software company admitting its screens were never really the product."

Self-reported and unaudited, she says her team has "TRIPLED the number of projects we ship… and almost nobody on our team opens ClickUp anymore lol." ClickUp is still there underneath, doing its job. They just do not visit.

Then she says the part that should stop you cold: "Once people stop logging in… they kind of stop looking, too. Take the screen away and that habit goes with it. And now work sits there quietly going stale while everybody assumes 'the AI has it.'"

That is the failure.

Here is the thing about why this is hard. The old noticing was free. You opened the CRM because you had to, and while you were in there your eyes caught the deal that had not moved in three weeks. Noticing was a side effect of the work. Take the work away and the side effect goes with it, and now noticing is its own job, with its own cost, producing nothing visible when everything is fine.

Nobody wants to pay for a system whose entire output is "yep, still fine." Which is why so few build one.

So here is the reframe. Monitoring is not supervision you perform. It is a thing you build once, that reports to you.

The Evidence: We Have Known This Since 1983

This is not a new problem with a new technology. It is a forty-year-old problem in human factors research that AI has just handed to small business owners.

Lisanne Bainbridge said the quiet part out loud in 1983. In a five-page paper titled "Ironies of Automation," published in Automatica, she wrote that "it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour." Her conclusion: "it is humanly impossible to carry out the basic function of monitoring for unlikely abnormalities." Not difficult. Impossible. Then she asks the question that belongs on the wall of every business running agents: "who notices when the alarm system is not working properly?" She also dismissed the dashboard forty years early. Keeping a log fails, she wrote, because "people can write down numbers without noticing what they are."

It is not about discipline; it is about having other work. Raja Parasuraman, Robert Molloy and Indramani Singh tested this in a 1993 study in the International Journal of Aviation Psychology. When monitoring the automation was a person's only job, they caught its failures reliably. When monitoring was one job among several and the automation was consistently reliable, detection dropped sharply, showing up after roughly twenty minutes. Mica Endsley's out-of-the-loop research explained why: automation moves the operator from active to passive processing, where situation awareness dies. You are not the exception to that research. You are the condition.

The Royal Majesty shows what "quietly fine" looks like. The NTSB investigation (report NTSB/MAR-97/01) found the GPS had flagged its data invalid the whole time, on bits the bridge system was not listening for. The dead reckoning indicator was tiny, in the chart room, which nobody visits at sea. An hourly cross-check against a second navigation system was required by procedure and not done. And the depth alarm sat at zero meters, the harbor setting used to stop nuisance alarms, instead of the three meters that would have warned them in time. The probable cause named the watch officers' overreliance on automated features.

Silencing the annoying alarm is a documented way people die. The Joint Commission's Sentinel Event Alert 50, April 8, 2013, reported that between 85 and 99 percent of medical device alarm signals do not require clinical intervention. The result is alarm fatigue, named in that Alert as the most common contributing factor in the events it reviewed. Of 98 alarm-related events reported between January 2009 and June 2012, 80 resulted in death, and in 36 a contributing factor was that the alarms had been turned off. Too many alerts is not safer than too few.

Most organizations running agents are not watching them. An EY and AIUC-1 Consortium survey reported in March 2026 found only 38 percent of organizations monitor AI traffic end to end across prompts, tool calls and outputs, and 17 percent continuously monitor agent-to-agent interactions. The enterprise version of this problem has a security chief. Your version is you.

The System: Build the Noticing Layer

What changed for me was realizing I had been treating this as a discipline problem when it is an architecture problem. I was never going to win by promising to check more often. Bainbridge told me that forty years ago.

So here is the system, in five parts. It is not glamorous, and that is the point.

One: a written playbook per process, not a tool configuration. Rachel Woods uses playbook to mean "a written process an AI can follow, specific enough that you don't have to explain it every time." The tool underneath can change next year and the playbook comes with you, because "the part you actually OWN is the stuff only your team knows." Your monitoring is only as good as your written definition of normal.

Two: a heartbeat that lives outside the thing it watches. The most common mistake I see is building the alert inside the automation. If the automation does not run, the alert does not run, and silence looks identical to success. You need a separate scheduled check asking "did this fire in the last 24 hours, and did it produce a plausible number of results?" In n8n that is an Error Workflow plus a scheduled count check; in Zapier, error notifications plus a webhook-triggered handler Zap. The rule: the watcher cannot be the thing watched.

Three: alert on absence, not just on errors. Almost everyone misses this one. A workflow that runs, succeeds and returns zero rows gets logged as a success. So the check is not "did it error," it is "did it come in under the floor I set." Zero leads yesterday when a normal day is eleven is an alert, even though nothing failed. Bainbridge warned that automatic control can camouflage a failure by compensating for it, so trends stay invisible until they are past fixing. Automatic systems, she wrote, should fail obviously.

Four: the unasked-for brief. Rachel Woods' own fix is deliberately boring. An AI agent clears their task inbox every day at 5:52pm, and the AI assistant they work with tells them what is still open at the start of every conversation. "Not fancy. But it's the reason the work still gets looked at." Isar Meitis built the same idea differently on Leveraging AI: a fifteen-minute morning brief connecting Gmail, Claude, ChatGPT and ClickUp that surfaces unanswered proposals, unsigned contracts and overdue invoices, and that watches his own automations for broken integrations. The information arrives without anyone asking.

Five: a scheduled drift check with a human in it. Dr. JeanCarlo Bonilla, on Brian Piper's AI for U, argues that AI is an organizational design challenge rather than a tech issue, and offers three questions worth stealing verbatim. Is anyone checking what our AI tools produce against an authoritative source, and how often? Do we know which models our workflows run on, who chooses, and what it costs? Is there any rule about what our tools are allowed to connect to? His maturity model tops out where "drift is checked on a schedule." Calendar it or it does not exist.

Practical Steps: Build This in an Afternoon

  1. List every automation and when you last looked at it. Write every AI workflow, agent and integration you have running, with a date beside each: the last time you verified its output was correct. Any blank, or any date older than a month, is where you start.

  2. Name the numbers you used to see by accident. For each automated process, write the one or two numbers you used to notice while you were in the screen anyway. New leads, overdue invoices, open tickets. You lost those signals without deciding to.

  3. Write down normal for each number. Not a target, a range. "A normal week is eight to fifteen new leads." Without this, no alert can exist, because you have not defined what wrong looks like.

  4. Build one external heartbeat today. Pick the automation whose silent failure would cost you the most money or trust. Build a separate scheduled job that compares yesterday's output count to your floor and messages you when it comes up short.

  5. Turn error notifications on, then route them to one place. Zapier, Make, n8n and most agent platforms will email or webhook you on failure, and plenty of people never switch it on. Send all of them to one inbox, and resist muting anything.

  6. Stand up your own unasked-for weekly brief. Use this prompt, with the brackets filled in for your business.

[The Job]
Draft a weekly noticing brief for my business that tells me what my automations did and what they failed to do.
This is for: [ME, THE OWNER OF A [TYPE OF BUSINESS] WITH [NUMBER] PEOPLE].
It matters because: if an automation quietly stops, I want to learn it from this brief and not from an angry customer.

[The Background]
Here is what you need to know: my automations are [LIST EACH AUTOMATION AND WHAT IT SHOULD DO]. The numbers I used to see by opening a screen are [LIST 3 TO 5 NUMBERS]. A normal week for each looks like [NORMAL RANGE FOR EACH]. The system holding the real version of each number is [NAME THE TOOL FOR EACH].
Do not use: estimates, industry averages, or any number you cannot trace to a system I named.

[The Deliverable]
Return: a one-page brief under 300 words, in plain English, in three blocks titled What Ran, What Did Not Run, and What Looks Off.
Must include: every automation that produced zero results, every number outside my stated range, the date each number was last confirmed, and a closing list titled Things I Should Look At Myself.
Optimize for: accuracy.

[The Questions]
Ask me any questions you have.
  1. Put a recurring thirty-minute drift check on your calendar. Once a month, open the systems nobody opens anymore and confirm the real numbers match what your brief has been telling you. This is the only step that requires you, and you cannot delegate it to the thing being checked.

Frequently Asked Questions

How often should I check automations that have been running fine for months?

Set a recurring monthly check of thirty minutes where you open the underlying systems and compare the real numbers to what your reports claim. Daily and weekly watching should be automated and push-based. Reliable automation is exactly the kind that stops getting checked, so calendar it instead of trusting memory.

What is automation complacency in plain English?

It is the documented tendency to stop genuinely watching a system that almost never misbehaves. Parasuraman, Molloy and Singh found in 1993 that people juggling several tasks detected far fewer automation failures when the automation was consistently reliable. It is not laziness. It is how human attention works, and it has to be designed around.

Can one AI agent just monitor my other AI agents?

Partly, and carefully. An agent can collect and summarize, which is genuinely useful. But it cannot be the only record of its own behavior, and it has to run separately from what it watches, or one failure silences both. Keep one scheduled human check comparing the agent's report to the source system.

Is it bad that nobody logs into our project management tool anymore?

Not by itself. Rachel Woods reports tripling shipped projects while her team mostly stopped opening ClickUp, a figure that is self-reported and unaudited. The risk is not the empty tool, it is the lost habit of noticing. If nothing replaced that habit with a push notification or a daily brief, that is the bad sign.

What should I check first if I have not looked in months?

Start with anything touching money or customers: invoicing, lead routing, appointment confirmations, review requests. For each, verify the last seven days of output against what you expect. You are hunting for zero counts and suspiciously round numbers, which usually mean a workflow is reporting success while doing nothing.

The Close: Nothing Was Broken on That Bridge

The detail that stays with me about the Royal Majesty is not the grounding. It is the thirty-four hours.

Thirty-four hours of professionals doing their jobs on a bridge where nothing was broken, every screen said fine, and the autopilot held course beautifully toward a place the ship had never been. The one indicator telling the truth was small, in a room nobody entered, with an alarm that sounded briefly and then stopped. You have a room like that in your business. So do I. Writing this is what made me go look.

So I will be as direct as I know how to be. The reason to build this is not efficiency. Real people depend on things happening in your business, and you have handed a meaningful share of them to systems that are extraordinarily good at appearing fine. "The AI has it" is not a monitoring strategy. It is hope wearing the costume of a system, and hope does not message you when the leads stopped coming in on Tuesday.

Build the noticing back in. Write down what normal looks like. Put one watcher outside the thing it watches. Make absence as loud as error. Then go stand in the room nobody enters. If you want help building that layer, that is most of what we do inside the AI Insiders Membership at 247 dollars a month, and you are welcome there.

The autopilot will hold course perfectly all the way to the sand. The question was never whether it works. The question is whether anything in your business is still looking out the window.


About the Author

Jonathan Mast is the founder and CEO of White Beard Strategies, where he provides AI coaching and mentorship to entrepreneurs who are curious about AI but tired of being sold hype. He runs a Facebook community of more than 500,000 members, created the Perfect Prompt Framework 2.0, and speaks to business audiences about using AI without handing over your judgment. He also has automations running in his own business that he has not laid eyes on in weeks, which is most of why he wrote this.

About the Author