DynamoDB Stream With Lambda Event Filtering

In this article, we will take a look at a use-case that allows us to pre-filter events by certain conditions before they invoke our AWS Lambda functions. We will be leveraging a DynamoDB stream with multiple AWS Lambda functions attached and multiple Lambda Event Source Mappings. The entire PoC (proof-of-concept) uses IaC written with Serverless Framework.

AWS Architecture

Folder Structure

DynamoDB Table

Two Lambdas + IAM Role for DynamoDB Stream

Above we are creating our two Lambda functions and attaching an IAM policy with permissions to log to CloudWatch and interact with our DynamoDB stream. The index.js is a one line log.

AWS Lambda Event Source Mappings

Above we are attaching two separate Lambda event filters.

One for Lambda A which is looking for the following:

eventName of “INSERT”
PK prefix of “Message#”
SK prefix of “Channel#”
messageId exists

If “messageId” doesn’t exist, “PK” doesn’t start with “Message#”, SK doesn’t start with “Channel#”, and eventName is not “INSERT” while in the new DynamoDB record, it will not trigger Lambda A.

This event would trigger Lambda A as it meets our filter requirements

The second Lambda event filter which would trigger Lambda B is looking for the following:

eventName of “INSERT”
PK prefix of “Feed#”

If “PK” doesn’t start with “Feed#” and eventName is not “INSERT” in the new DynamoDB record, it will not trigger Lambda B.

This event would trigger Lambda B as it meets our filter requirements

If one of our conditions is not met then our Lambda functions with filters will not be invoked. The trickiest part of this setup is getting the Filter Pattern correct. For this, I leveraged the AWS documentation and trial + error.

Conclusion

By leveraging Lambda Event Mapping Filters on a DynamoDB Stream we are able to only trigger the necessary Lambda when our conditions are met. Which is a huge improvement over triggering every time DynamoDB has an event take place then filtering inside the Lambda itself.

By doing this filtering outside the Lambda we are saving money (as we are not billed for the Lambda Event Mapping Filter), saving engineering hours to write the filtering code and test, reducing overall noise from millions of non-relevant Lambda invocations happening, and ensuring downstream maintenance is minimal.

KloudVM