CloudTrail Ingestion Service Failure

Incident Report for F5 App Infrastructure Protection

Resolved

Threat Stack’s CloudTrail ingestion service runs once every 8 minutes to retrieve our customer’s CloudTrail information from remote SQS queues, then fetches the individual files from the s3 buckets for ingestion into the Threat Stack Platform.

While performing an upgrade to our CloudTrail ingestion service today, we ran into a configuration issue in one of our RabbitMQ queues. This meant that all of the messages for that run were dropped. Unfortunately, because the CloudTrail ingestion service deletes messages from the remote SQS queue, we are unable to to retrieve them to process.

The impacted run occurred July 9th, 1:35PM - 1:45PM EST.

The configuration was immediately fixed and CloudTrail events are flowing into the Threat Stack Platform as normal.

Posted Jul 09, 2020 - 13:30 EDT