AI agents automate the initial investigation of production failures 24/7, leaving humans solely responsible for 'decision-making.'
The emergence of AI agents is dramatically changing the role of human engineers when incidents occur late at night. Traditionally, engineers were responsible for triage (identifying the cause and scope of impact) when an alarm was triggered, but the vision of AI taking over triage and allowing humans to focus on final decision-making is becoming a reality.
How We Learned to Trust an AI Agent to Triage Production Incidents - Kiro
At 2:33 AM local time on August 16, 2026, an availability alarm occurred in one of the Frontier models handled by KIRO , a provider of development platforms equipped with AI agents, causing the response in the production environment to stop midway through processing. Just 13 minutes and 35 seconds after the monitoring system automatically issued a ticket, the AI agent identified the cause and recorded a diagnostic result in the ticket, summarizing the evidence and recommended next actions. All that human engineers did was enter a sentence into the ticket such as 'Request escalation' or 'Address the bug in the production environment,' meaning the human role was to 'read the completed summary,' 'scrutinize the decision criteria outlined in the summary,' and 'make a decision.' Considering that this series of actions by the AI agent took place in the middle of the night, it is unlikely that humans alone could have responded as quickly.
The AI agent handling the issue runs on the Kiro CLI and is responsible for triaging the production ticket queue. Through this alarm, the KIRO Frontier team redesigned their workflow, focusing on the agent, and came to trust its capabilities. When an alarm occurs, someone needs to determine 'Is the alarm real?', 'What caused it?', and 'Who is affected?'. Proper triage involves 'hypothesis exploration,' finding the first few questions to ask from a multitude of hypotheses spanning different log groups, accounts, and query languages. The AI agent excels at this task, utilizing various tools. The operational procedures, alarm wiki, and post-incident notes accumulated over many years can be seen as an investment in the agent's context. By recording the insights gained from investigations, the agent continuously accumulates knowledge about the system. It would be difficult for an exhausted human to perform this same task in the middle of the night.
System improvements are made weekly through a loop in which engineers train agents and manage what the agents have learned.
There are four mechanisms that maintain the rotation of the loop.
• Learn from your mistakes : Each fix is saved once and will be reflected in all future sessions. Additionally, agents will correct the documentation that caused the prompt bug.
• Research into knowledge : Closed tickets are aggregated in the archive and are the first to be queried in future sessions.
- Agents create skills, humans review them : When a new incident occurs, agents create or update new playbooks, which are managed by engineers and night shift workers like a codebase.
- Share status between agents : Even when multiple agents are monitoring the same incident, only a single upstream ticket is created.
However, this learning pipeline can also fail. Previously, text that was held in reserve at the end of a session was written directly to the correction store, which could lead to it being learned as incorrect operational knowledge in future sessions. To prevent this failure, schema validation before writing and nightly pruning have been introduced. The lessons learned by the KIRO frontier team can be summarized as follows:
- AI agents take over document bugs using the computer's processing speed : If a session fails, it is important to thoroughly follow the process of 'fixing it on the spot,' 'investigating the cause of the misunderstanding,' and 'correcting the document.'
A confident half-answer costs more than a wrong answer : agents must test and eliminate competing hypotheses before drawing conclusions. Automated checks eliminate many errors that were previously detected by human review.
• Conciseness builds trust : Early agents posted detailed work steps in threads visible to the customer, but this was noisy, so now only short key points are posted in threads, and details are sent to the work log.
• There is no single autonomous setting : Every action requires its own rules.
Security starts with the infrastructure : All unattended sessions are read-only by default, and requests for administrator privileges will result in an error. Each credential is issued for one session and scoped to the task declared by the agent, so the session can only call the API it requested.
Currently, AI agents are responsible for triage, but in the future, they are expected to also handle drafting code reviews and remediation through threshold adjustments. On-call rotation will continue, but its role will change; engineers will review completed investigations rather than starting investigations in the middle of the night. On-call engineers' pagers are becoming decision queues for work already performed by agents. This series of changes marks the beginning of a new life for frontier engineers, and further evolution is expected in the future.
Related Posts:
in AI, Posted by log1c_sh