AWS Infrastructure Investigation Agent

Project overview

This agent gives an AI assistant structured, audited access to production AWS infrastructure for investigation purposes — querying EC2 instances, IAM roles, ECS tasks, CloudTrail events, S3 buckets, and more — while enforcing a strict read-only boundary, automatic secret redaction, and an immutable audit trail for every tool call.

The system is designed to answer questions like "why did this ECS task fail?", "what IAM permissions does this role actually have?", or "which S3 buckets have public access enabled?" — without ever risking a mutation of live infrastructure. Every tool call is logged, every response is scanned for secrets before being returned to the model, and the investigation lifecycle is enforced by a terminal-state gate that prevents further evidence collection once a conclusion is reached.

Problem and constraints

Investigating production AWS infrastructure with an AI assistant introduces several risks that a naive tool implementation fails to address:

• Mutation risk: A model could inadvertently call a tool that modifies infrastructure if the tool surface is not tightly bounded.
• Secret exposure: AWS API responses regularly contain sensitive values — access key IDs, secret ARNs, connection strings — that must not be returned verbatim to the model's context.
• Audit integrity: Without an immutable record of tool calls and their outputs, an investigation cannot be reproducible or defensible.
• Unbounded investigation: Without a terminal state, a model could continue querying indefinitely or attempt to re-open a concluded investigation.
• Evidence fabrication: A model could synthesize plausible-looking evidence rather than recording actual API responses.

The agent also needed to pass a rigorous test suite verifying every security invariant before any phase was marked as accepted.

My contribution

I designed the architecture, defined the security invariants, authored the MCP server implementation, and delivered the complete test suite. The project was a solo engineering effort spanning 20+ formal delivery phases, with each phase gated on a passing acceptance checklist before proceeding.

The formal phase structure — rather than an ad-hoc development process — was a deliberate engineering decision: each phase had defined acceptance criteria, a test baseline, and a locked acceptance record. Phase 10 locked with 1,689 passing tests, 18 security-specific tests, and zero regressions from previous phases.

Architecture

The system has three main layers:

MCP Server — tool layer

The server exposes eight investigation tools to the model via the Model Context Protocol. Each tool is read-only, audited before and after execution, and passes through the secret redaction layer before returning results. The tool categories span the full investigation lifecycle:

• Evidence gathering (IAM, EC2, ECS, CloudTrail, S3, SSM)
• Hypothesis testing (resource state queries, cross-service correlation)
• Conclusion recording (structured finding submission)

Audit database

Every tool invocation is written to an append-only SQLite audit database (audit.db) that is structurally isolated from the investigation write path. The audit log cannot be modified after a record is inserted. This gives each investigation a complete, reproducible evidence chain.

Security layer

Before any tool response reaches the model, it passes through the secret redaction module, which scans for patterns matching AWS access key IDs, secret access keys, session tokens, database connection strings, and ARNs containing sensitive path segments — replacing them with redaction markers rather than raw values.

Terminal-state gate

Once an investigation reaches a terminal state (a conclusion has been submitted), the gate blocks further evidence operations. This prevents both accidental continuation and deliberate attempts to re-open a concluded investigation through additional tool calls.

Key decisions

Read-only tool surface

Every tool was designed with zero mutation capability. This is not a runtime permission check — the tools simply do not have the AWS IAM permissions to modify anything. The IAM role used by the MCP server grants only Describe*, List*, Get*, and Lookup* actions across the relevant services.

Phase-gated delivery over iterative development

Delivering across 20+ formal phases with acceptance criteria and locked records — rather than a typical sprint-based iterative approach — provided several concrete benefits: regressions were caught immediately, the security invariant suite grew alongside the implementation, and the final acceptance record provides an auditable history of design decisions.

SQLite over a remote database for audit storage

A local SQLite database, rather than DynamoDB or RDS, was the right choice for the audit log: it is trivially portable, has strong append-only semantics achievable via trigger constraints, and avoids network dependencies that could prevent audit logging if the investigation target AWS account is unreachable.

Security and reliability

Eighteen dedicated security tests cover the following invariants at Phase 10:

• SI-21 — No fabricated evidence: Evidence records must reflect actual tool output, not model-generated content.
• SI-22 — Confidence is authoritative: Confidence level is computed from the evidence chain, not accepted from model input.
• SI-24 — Terminal state blocks evidence: No further evidence operations are permitted after a conclusion is submitted.
• G6 — Resource authorisation validation: Per-resource authorisation is checked on every tool call before execution.
• G0.5 — Terminal state check: Terminal state is verified at the earliest possible point in the tool call lifecycle.
• G14 — Evidence persistence with redaction: Evidence is persisted to the audit log, and redacted values are confirmed absent from the recorded content.

The agent does not claim to be a production security scanner. It is an investigation and diagnosis tool — appropriate for post-incident analysis, infrastructure audits, and structured inquiry. It is not a replacement for automated security tooling like GuardDuty, Security Hub, or Prowler.

Outcome and lessons

The project demonstrates that an AI-augmented infrastructure investigation tool can be built with strong safety guarantees — read-only boundaries, evidence integrity, secret redaction, and lifecycle enforcement — without making the tool prohibitively complex to use.

The most important lesson from the phase structure: defining acceptance criteria before writing code consistently produces better outcomes than writing code and then deciding what "done" means. Each security invariant was written as a failing test before the implementation was updated to satisfy it.

Phase 10 reached 1,689 passing tests with 18 security-specific tests, 7 pre-existing known failures (tracked and accepted), and zero regressions across all phases — a meaningful baseline for a system that operates in a production AWS environment.