There's a version of an SRE copilot that I find actively harmful. It surfaces when an alert fires at 2am, an engineer opens the incident channel, and the tool responds with: "I noticed this could be related to several things. Can you tell me more about your deployment frequency?" That is the worst possible response. The engineer knows their deployment frequency. They don't need the tool asking them to do the tool's job.
We spent a lot of time at Devtract thinking about what a copilot should and shouldn't do during an incident. Not just the feature set, but the moment-to-moment interaction design. What does the tool say first? What does it hold back? When does it ask a question versus just answer one? This post is an attempt to write down the principles we settled on, because they're not obvious and we got several of them wrong before getting them right.
Principle 1: Lead with context, not questions
The first thing an SRE needs during an incident is orientation. They want to know: what is on fire, what changed recently, and what else is affected. In those first 60 to 90 seconds, they are building a mental model. The job of the copilot is to accelerate that model-building, not to interrupt it with dialog.
What this means in practice: when Devtract fires a notification into a PagerDuty incident or Slack war room, the first message contains correlation data. It surfaces the three most likely hypotheses, each with a confidence score and the evidence supporting it. It lists what changed in the last correlation window. It identifies which other services are showing anomalous signals. It does not ask "would you like me to investigate?"
We tested an earlier version that prefaced its output with "I've detected a potential issue. Shall I run a deeper analysis?" The feedback from engineers was immediate and consistent: stop asking. Just tell me.
Principle 2: Confidence scores are not optional
A copilot that gives you a single answer with no uncertainty information is making a design decision that favors the tool's authority over the engineer's judgment. That's backward. Engineers need to know how much to trust any given hypothesis so they can decide whether to pursue it or hold it as a secondary possibility while they gather more data.
Our RCA ranking surface shows percentage-based confidence scores next to each hypothesis. "Deploy of auth-service v1.4.2 at 02:14 UTC is likely related (78%)" is more useful than "the auth-service deploy caused this." The second phrasing removes the engineer from the loop. The first keeps them in it.
We're not saying numerical confidence is always the right interface. There are moments where showing a precise percentage feels misleading because the underlying signal is sparse and the confidence interval is wide. In those cases we've experimented with qualitative labels (high/medium/low) or explicitly stating the evidence count. But some form of expressed uncertainty is non-negotiable. A tool that presents hypotheses as conclusions is a liability during incidents.
Principle 3: Natural language queries should answer from telemetry, not from general knowledge
The on-call copilot in Devtract handles natural-language questions like "what changed in the payment service in the last 20 minutes?" or "are there any OOMKilled pods in the payments namespace?" These questions have specific, answerable facts behind them. The telemetry either shows this or it doesn't.
An early failure mode we caught internally: the model was sometimes augmenting its answers with general knowledge about common failure patterns when the telemetry signal was thin. It would say something like "OOMKills are often caused by memory leaks in JVM services, which could explain..." That's not an answer grounded in this incident. That's a generic statement. Engineers are pattern-matchers and they will follow wherever you point them. If you point them at a general hypothesis instead of a telemetry-backed one, you've just introduced scope creep into an active incident.
The constraint we built: the copilot's answers are scoped to the incident's telemetry window. If the data doesn't support an answer, the tool says so explicitly: "No OOMKilled events in the payments namespace in the last 30 minutes per cluster event stream." That's the right answer. It closes a branch of investigation rather than opening a speculative one.
Principle 4: Don't surface actions, surface facts
This one caused the most internal debate. The natural instinct when building a copilot is to go one step further and suggest remediation: "Based on the correlation, consider rolling back the auth-service deploy." It feels like value-add. It feels like the copilot being useful.
We don't do this in the primary response, and here's why: the copilot does not know your risk tolerance for a rollback. It does not know whether auth-service v1.4.2 was deployed to fix a production data-loss bug that a rollback would reintroduce. It does not know your change freeze policy. It does not know whether three engineers already discussed and rejected the rollback option 10 minutes ago in a thread it isn't reading.
Surfacing remediation actions creates a pull toward them. Engineers under time pressure are susceptible to anchoring on the first actionable option presented. We think the correct boundary is: Devtract tells you what happened and what the evidence suggests. The decision about what to do belongs entirely with the on-call engineer. This is not modesty about the tool's capabilities; it's a deliberate choice about where accountability should live during a production incident.
Principle 5: Silence is a valid response
Not every alert needs a response from the copilot. Some alerts are self-evident. An "InsufficientMemory" node event with one pod OOMKilled in the preceding three minutes and no recent deploys doesn't need the copilot to say much. The evidence is right there. The on-call engineer can see it.
We've built suppression logic into Devtract's notification layer. When the correlation engine finds a high-confidence single-cause incident with no ambiguity in the evidence chain, the notification is brief. When the correlation is ambiguous, multiple services are involved, or the evidence contradicts the obvious hypothesis, the notification is detailed. The copilot does more work when the incident is harder, less work when the incident is easy.
Consider a scenario we encountered while testing: a staging environment at a logistics platform running roughly 80 microservices was generating 12 to 15 noise alerts per week for a single recurring pod restart cycle in their ingress layer. Every alert was firing the copilot into a detailed analysis mode even though any engineer who'd been on-call for two weeks knew exactly what it was. That's not a copilot being helpful. That's a copilot generating overhead. The right fix was to recognize the pattern and deliver a one-line acknowledgment with a link to the known issue, not a five-paragraph RCA summary.
Principle 6: Interrupt cost is the design constraint
Every time the copilot surfaces something, it costs the engineer attention. Attention is finite and it's most depleted exactly when incidents are most severe. The design constraint we keep returning to is: what is the interrupt cost of this feature? If the feature asks the engineer to read more, decide something, or respond to the tool, the interrupt cost is high. If it gives them a fact they can scan and incorporate in under three seconds, the interrupt cost is low.
This is why we've been resistant to adding "follow-up question" prompts to the copilot interface. "Would you like me to look at related services?" is a high interrupt cost question. It requires the engineer to stop, process the offer, decide, and then type a reply. During a P0 incident with a war room full of people watching, that's a meaningful tax. The alternative: just look at the related services and include the relevant signals in the initial response. That's zero interrupt cost.
The underlying principle here isn't that dialog is bad. It's that dialog initiated by the tool is bad during an incident. Dialog initiated by the engineer, asking a specific question, is the tool doing its best work. Those are different modes and they require different design responses.
Where we're still figuring things out
The postmortem phase is harder. Once the incident is resolved and the team is reconstructing the timeline, a more conversational mode makes sense. "Walk me through the sequence" is a reasonable ask. "What was the time between first anomaly and first alert?" is a reasonable ask. We're still calibrating how much the copilot should take initiative in the postmortem versus waiting for directed questions.
There's also an unresolved tension between verbosity and completeness. An incident with ambiguous evidence may have five plausible hypotheses. Surfacing all five risks overwhelming the engineer. Surfacing only the top two risks missing the real cause. We default to the top three, but that number is arbitrary and we don't have strong data yet on whether it's right.
If you're building or evaluating SRE tooling, the test I keep coming back to is this: when the tool responds, does the engineer's mental model get clearer or more cluttered? A copilot that clarifies is earning its place. A copilot that adds noise to an already noisy incident is making the on-call experience worse, not better. That's the bar we're trying to clear.