LetsDefend: AWS PerSEStence — building a timeline from CloudTrail

Cloud incident response is a different discipline from host forensics, and this challenge is a good demonstration of why. There is no disk to image and no memory to capture. There is an enormous pile of JSON, and the entire job is turning it into a story.

The problem with CloudTrail#

CloudTrail logs every API call made in an account. That is exactly what you want during an investigation and exactly what makes it unusable raw: thousands of deeply nested JSON events, one per API call, most of them routine.

The first instinct is cat. It is the wrong instinct.

cat cloudtrail-events.json

That gives you a wall of minified JSON scrolling past faster than you can read, with the one field you care about buried four levels deep in each event.

jq is the actual tool#

jq '.' cloudtrail-events.json

Same data, formatted and indented. That alone makes the structure visible — and once you can see the structure, you can start asking questions of it rather than reading it.

The fields that matter in almost every CloudTrail investigation:

# What happened, when, by whom, from where
jq -r '.Records[] | [.eventTime, .eventName, .userIdentity.arn, .sourceIPAddress] | @tsv' \
  cloudtrail-events.json

That one command collapses each event to four columns. It is the difference between a file you cannot read and a table you can sort.

Do not overlook ls either. Before parsing anything, list what you were actually given — how many files, how they are split, what date ranges they cover. Investigators regularly analyse one file thoroughly and never notice there were six.

Reading CloudTrail like an investigator#

Some orientation that carries beyond this challenge:

• eventName is the API call. Persistence in AWS almost always shows up as identity activity — creating users, creating access keys, attaching policies, creating roles or editing trust relationships.
• userIdentity tells you which principal acted, and whether it was a root user, an IAM user, or an assumed role. An assumed role appearing where it never has before is worth a second look.
• sourceIPAddress is the pivot. Once you have one confirmed malicious event, its source address gives you every other action from the same origin.
• errorCode is underrated. Failed calls map the attacker's reconnaissance — what they tried before finding something that worked, which tells you what they were after.

Building the timeline#

This is the actual deliverable, and the skill the challenge is really testing.

Work in one direction: sort every relevant event by eventTime and write the sequence out in plain language. Initial access, then what they enumerated, then what they created or modified to keep their access, then what they did with it.

The output is not a list of suspicious events. It is a narrative with an order and a causal chain — this call succeeded, which made that next call possible. If two events could have happened in either order, say so rather than implying a certainty the logs do not support.

Why persistence is the interesting part#

The name is the hint. Getting in is one event; staying in is a pattern. In AWS that usually means the attacker created something durable — a new access key, an extra IAM user, a modified trust policy — so that losing the original foothold costs them nothing.

Which is why containment has to be scoped from the timeline rather than from the initial alert. Rotating the compromised credential and stopping there leaves whatever they created afterwards fully functional.

What to take away#

Five things this challenge drills, all of which transfer directly to a real cloud incident:

1. Constructing a timeline is the core deliverable of incident response, not a write-up formality.
2. jq over cat for any JSON at volume — and @tsv to make it sortable.
3. List your evidence with ls before you start analysing a subset of it.
4. CloudTrail is readable once you know which four fields carry the story.
5. The narrative — how they got in, and what they did to stay in — is the finished product.

References#

• LetsDefend — the platform this challenge runs on
• AWS CloudTrail record contents — every field, defined
• jq manual — worth thirty minutes of your life