Your monitoring tells you something broke. It doesn't tell you why. Someone still has to open the dashboard at 3 AM, work out which alerts matter, log in to the host and find the cause. Sentinel is the part of 2501 built for that gap. It watches the alerts your monitoring sends, decides which ones deserve a closer look, and opens an investigation that works like a careful on-call engineer: diagnose first, fix only once the evidence shows the cause, then check that the fix worked. Sentinel is experimental today and off by default.

TL;DR

1. Alerts become patterns. Monitoring alerts arrive in 2501 as signals. Signals of the same kind, on any host, group into one pattern, so a hundred copies of one error are one thing to look at.
2. Five detectors look for unusual behavior. A sudden burst, a pattern that is new or back after going quiet, the same error spreading to another host, errors rising across the board, and severe alerts still waiting for a decision.
3. Every flagged pattern gets a call. Sentinel opens an investigation or snoozes the pattern. A snoozed pattern comes back early if a more severe alert arrives.
4. Diagnosis comes before any change. An investigation starts with read-only jobs. It moves to a fix only once a job's output shows the cause.
5. A separate review can refuse the fix. Before any job changes a host, a separate review checks the fix against the evidence and refuses it if the evidence does not hold up.
6. Every investigation ends one of three ways. Resolved, dismissed or escalated to a person. Your operational rules apply to every job along the way.

The gap between an alert and a cause

Monitoring is good at one thing: telling you that a threshold was crossed. It is the smoke alarm. What happens next is still manual in most teams. An engineer reads the alert, decides whether it is noise, checks whether other hosts show the same thing, logs in, reads logs, forms a theory and tests it. Only then does the fix start.

That work has three parts, and each one is a place where time goes:

• Sorting. Most alerts are repeats of something already known. The one that matters is easy to miss among them.
• Diagnosing. The alert names a symptom. The cause is on the host, and finding it means looking.
• Checking. A fix that was applied is not a fix that worked. Someone has to confirm the service came back.

Sentinel takes the first part. The investigation it opens takes the other two.

How Sentinel decides what deserves a look

Alerts from your monitoring tool arrive in 2501 and are stored as signals. Signals with the same alert rule and message form one signal pattern, whichever host they come from. The changing parts of a message, such as times, identifiers and numbers, are ignored when grouping, so the same error with a different timestamp is still the same pattern.

Sentinel runs on a schedule. On each run, five detectors look at the patterns that have no decision yet:

• Burst. A pattern suddenly arrives far more often than usual.
• New fingerprint. A pattern not seen recently: brand new, or back after going quiet.
• Host spread. The same pattern shows up on another host.
• Error volume. Errors across the organization rise sharply: all errors, every pattern.
• Untriaged severe. Severe signals are still waiting for a decision.

For each pattern a detector flags, Sentinel looks at its recent signals and at the conclusions of past investigations, most urgent first, and makes a call:

• Open an investigation. Related patterns can share one investigation.
• Snooze it. Sentinel picks how long. When the snooze ends, the pattern is looked at again only if new signals arrived. If a more severe signal arrives first, it is looked at early.

While a pattern is under investigation, Sentinel leaves it alone.

What this means for on-call teams: the sorting is done before a person has to look. A known flapper gets snoozed. A new error spreading across hosts gets an investigation.

How an investigation works

An investigation works on one problem. It keeps a current explanation of the cause in one sentence, how confident it is in that explanation, and the list of hosts it believes are affected. It then repeats three steps until it can close: assess what it knows, decide what to do next, and wait for the result.

Step 1: diagnose, read-only

The investigation starts read-only jobs, one per host. They look at the host without changing it. Each result updates the explanation and the confidence. If the same pattern starts showing up on another host, that host joins the list.

Step 2: fix, only on evidence

The investigation moves to a fix only when a job's output shows three things: what causes the fault, what the fix would touch and what depends on it, and a method that follows from those facts. No runbook is needed.

Then a separate review reads the planned fix and the saved output it rests on. It sees only the evidence, not the investigation's reasoning. It asks whether that output shows the cause each change fixes. If a change would delete data, it asks whether the change picks exactly the faulty data and spares what is in use. If the answer is no, the fix is refused, no job starts, and the investigation goes back to diagnosing with the reason in hand. Too many refusals and it escalates.

Step 3: check that it worked

A fix job makes its change and reports what it did. It does not grade its own work. After it ends, another read-only job watches the reported service or symptom for fresh success. The investigation resolves only after that check and once the alerts have stayed quiet.

What this means for on-call teams: nothing changes on a host on the strength of a guess. A fix needs evidence from that host, a second opinion on the evidence, and a check afterwards.

Three ways an investigation ends

• Resolved. The reported problem stopped. If it recovered on its own, the investigation says so and records the cause as uncertain.
• Dismissed. The problem was not real. If its alerts are still arriving, it cannot be dismissed and is escalated.
• Escalated. A person needs to look. This happens when the investigation chooses it, when its fixes were refused too many times, when it has used up its job rounds, or when it has been open too long.

An investigation that fails to fix a problem does not repeat the fix that failed. It diagnoses once more, then tries a different fix or escalates.

What stays under your control

• Operational rules apply to every job. A rule can narrow a fix or forbid it. The investigation then has to pick another method the facts support, or escalate.
• Execution mode still counts. A fix job uses the execution mode of each agent. Through an agent in Plan it only reads, so nothing changes on that host.
• Jobs stay on affected hosts. Jobs run only on the hosts the investigation has tied to the problem.
• Each organization is separate. Signals, Sentinel decisions and investigations belong to the organization whose signal key the alert was sent with.

What it looks like in practice

An error that spreads

At 09:00 your monitoring reports "connection refused" from a web server. Five minutes later the same error appears on a second one. Sentinel sees the pattern is new and spreading, and opens an investigation. Read-only jobs on both hosts point to the same cause: the disk of the service they both call is full. The investigation's confidence rises with each result. Once the output shows what filled the disk and what is safe to remove, the evidence review lets the fix through. A last read-only job confirms the service answers again, the alerts stop, and the investigation closes as resolved.

A known flapper

An alert that fires a few times every night fires again. Past investigations concluded it recovers on its own. Sentinel snoozes it. If it stays at its usual level, nobody hears about it. If a more severe alert arrives on the same pattern, Sentinel looks again before the snooze ends.

A fix the evidence does not support

An investigation believes a stale file is the cause and proposes to delete it. The evidence review reads the job output and finds nothing showing the file is unused. The fix is refused. The investigation goes back to diagnosing. If it still cannot show the cause, it escalates with what it found, and a person starts from a diagnosis in progress instead of a raw alert.

How to turn it on

1. Turn on observability for your deployment. It is a feature toggle, off by default, and 2501 reads it at start.
2. Give Splunk your organization's signal key. Splunk sends the key with each alert. The key tells 2501 which organization the alert belongs to.
3. Send your Splunk alerts to 2501. From then on they are stored as signals and Sentinel starts its runs.
4. Check your agents' execution modes. An investigation can fix a host only through an agent that is allowed to make changes. Through an agent in Plan it only reads.
5. Write the operational rules you care about. State what a fix must never touch. The rules apply to every job an investigation starts.
6. Look at escalated investigations. Nobody is notified yet, so make checking them part of the on-call routine.

Where we are going

Fewer dashboards at 3 AM, more problems that arrive already diagnosed. When Sentinel can fix the problem, you find a resolved investigation in the morning, with its conclusion and the job results behind it. When it cannot, the person who takes over starts with the hypothesis, the affected hosts and everything the read-only jobs found.

Caveats

• Sentinel is experimental. Its behavior may change in later releases. It is off by default and turned on per deployment.
• Splunk is the alert source today. Tickets do not become signals, and investigations do not read or update tickets.
• Escalation does not notify anyone yet. An escalated investigation changes its status. Someone has to check for it.
• You can read an investigation, not talk to it. Sending messages to an investigation is not available.
• Resolved means the problem stopped. It does not prove the cause.
• The evidence review checks evidence. It refuses fixes the evidence does not support. It does not prove a fix is safe, which is why operational rules and execution modes still apply.