Cloudfront logs analysis

I built a WEB site with AWS CloudFront + S3 becase of good cost performance. But too many access logs are saved on S3.

To analyze access, it is very convenient for me to save these data on single database or file. How about registering these logs to DynamoDB using AWS Lambda ?

The code is on github moriya9n/cflog2dynamodb. Becase I am new to DynamoDB, the table definition is not efficient.

Configuration

  1. Open AWS Lambda via AWS Management Console.
  2. Create a function with a name you like, python 3.9 as runtime, x86_64 as architecture and role with defualt.
  3. Select the created fucntion. Select “code” tab, select “upload from” .zip file, then select the zip file in the above repository.
  4. Select “configuration” tab / “Permissions”, then edit the Execution role. Allow the lambda function to access the CloudFront bucket and DynamoDB.
  5. Push “Add trigger”. Select S3, then select the bucket you are logging, “put” as event type, *.gz as suffix.

Search Logs

  1. Open DynamoDB via AWS Management Console.
  2. Select Tables. Check cflog, then select Explore items as Action. Records are shown.
  3. For filtering, open “Filters”, then specify the conditions you want. For example, user_agent Contains Googlebot.

Conclusion and todo

  • Cloudfront logs can be saved to dynamodb.
  • For search efficiency, table should be improved.