Analyze S3 CloudTrail Log using Boto

at first

I wanted to parse the CloudTrail log on S3 using Boto (Python SDK of AWS), but there were some addictive points and a sample of the beginning that would be very helpful for me. I couldn't find the Code well, so I'll put the Code on the back of the leaflet for your reference.

Where I got hooked

• Boto3 doesn't support Access via HTTP Proxy? Like, I ended up using Boto2
• CloudTrail log is gz-compressed by Default, so decompression is required to parse the contents.
• If you say "xx of S3 targets only files under Path of aa / bb / cc of Bucket", bucket.list (prefix ='aa / bb / cc')

In addition, the log of CloudTrail seems to be Analysis progresses by using Loggly.

About Python Code

Items that need to be set individually

ʻAws_access_key_id, ʻaws_secret_access_key, target_path, proxy, proxy_port need to be set individually.

• For ʻaws_access_key_id, ʻaws_secret_access_key, set the key of the IAM User created for AWS external access (don't forget to give the created IAM User the ReadAccess right to S3 (ʻAmazon S3 ReadOnlyAccess`)!)
• proxy, proxy_port are HTTP Proxy server settings
• target_bucket is the target bucket name, target_path is the path of S3 that you want to analyze, here I tried to target only the log of 2015/07 of us-west-2

Supplement

• The access key of the sample code below is an appropriate character string.
• I am using Python2.7, Boto v2.32.1

Outline flow of Code

The processing flow is as follows.

• List the files (Keys) under target_path of target_bucketBucket and download them
• Unzip the downloaded file (gz format) and read it as JSON
• If the RDS related Log matches (['eventSource'] =='rds.amazonaws.com'), output the contents to standard output.

import boto.s3.connection, gzip, StringIO, json

aws_access_key_id='AKKBUGOIU4434DDTT'
aws_secret_access_key='78oiupoiuh7++REugoiusGSEE'
target_bucket = 'your-backet-name'
target_path = 'CroudTrail/AWSLogs/1234567899999888/CloudTrail/us-west-2/2015/07'

def main():
  s3Instance = boto.s3.connection.S3Connection \
    (aws_access_key_id, aws_secret_access_key, proxy='your.proxy.server.com', proxy_port=8080)
  s3Bucket   = s3Instance.get_bucket(target_bucket)
  bucketList = s3Bucket.list(prefix=target_path)

  for count, itemOne in enumerate(bucketList):
    s3BucketKey = s3Bucket.get_key(itemOne.name)
    buffer_gz = s3BucketKey.get_contents_as_string()
    stringBuffer = StringIO.StringIO(buffer_gz)
    buffer_text = gzip.GzipFile(fileobj=stringBuffer)

    try:
      responseJSON = json.loads(buffer_text.read())
    except Exception, e:
      print e
    else:
      for count, itemTwo in enumerate(responseJSON['Records']):
        if itemTwo['eventSource'] == 'rds.amazonaws.com':
          print json.dumps(itemTwo, separators=(',', ':'), indent=2)
          print 'Event name = %s' % (itemTwo['eventName'])
          print '================================='

    stringBuffer.close()
    buffer_text.close()

if __name__ == '__main__':
  main()