Leveraging DynamoDB for Application Architecture: A Real Life Example

Photo by Jan Antonin Kolar on Unsplash

If you’re looking for real world example feel free to scroll to the “Real World Example Using DynamoDB” section. Hope you’ll find it interesting.

Quick intro on what is DyanmoDB (feel free to skip)

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. It is a key-value store that uses primary keys to uniquely identify items and secondary indexes to provide more querying flexibility.

DynamoDB is designed to handle large amounts of data and support high levels of read and write throughput. It is ideal for applications that require a flexible data model and low latency, such as mobile, web, gaming, ad tech, IoT, and many other applications.

One of the key benefits of DynamoDB is that it automatically scales throughput and storage to meet the needs of your applications. You can also choose to provision throughput and storage resources according to your specific needs, or use on-demand capacity mode to automatically adjust your capacity as needed.

In addition, DynamoDB provides several features to help you manage your data, such as data backup and restore, point-in-time recovery, and encryption at rest. It also integrates with other AWS services, such as Lambda, for building serverless applications.

RDS or DynamoDB?

Amazon Relational Database Service (RDS) and Amazon DynamoDB are two different database services provided by AWS. Here are some key differences between the two:

1. Data model: RDS is a relational database that uses tables, rows, and columns to store data, while DynamoDB is a NoSQL database that uses a key-value store model.
2. Scaling: RDS allows you to scale up or down the compute and storage resources of your database instance, while DynamoDB automatically scales throughput and storage to meet the needs of your applications.
3. Consistency: RDS provides strong consistency for read operations, while DynamoDB offers eventual consistency for read-after-write operations by default. You can also choose strong consistency for read operations in DynamoDB if needed.
4. Performance: RDS is designed to handle a wide range of workloads and provides options for optimizing performance, such as read replicas and database caching. DynamoDB is optimized for high write and read throughput and low latency.
5. Pricing: RDS charges for the compute and storage resources you use, while DynamoDB charges for the read and write throughput you provision or consume.
6. Compatibility: RDS supports a variety of database engines, such as MySQL, Oracle, PostgreSQL, and others, while DynamoDB supports only NoSQL data.

In summary, RDS is a good choice for applications that require a traditional, scalable relational database with strong consistency, while DynamoDB is suitable for applications that need a flexible NoSQL database with high performance and scalability.

Photo by Campaign Creators on Unsplash

Limitions

DynamoDB has several limitations that you should be aware of when designing your application:

1. Item size limit: The maximum size of an item (including its attributes) in DynamoDB is 400 KB. If you need to store larger items, you can use DynamoDB Streams to store the data in another store, such as Amazon S3.
2. Maximum number of attributes per item: DynamoDB supports a maximum of 400 attributes per item.
3. Maximum item collection size: The maximum size of an item collection, such as a list or a map, is 10 GB.
4. Maximum number of tables per account: You can create up to 256 tables per AWS account in each region.
5. Maximum number of secondary indexes per table: A DynamoDB table can have up to 20 global secondary indexes and 5 local secondary indexes.
6. Maximum number of reads and writes per second: The maximum number of reads and writes per second depends on the capacity mode and capacity units you have provisioned or are consuming for your table or index.
7. Maximum number of write request units per second: The maximum number of write request units per second for a DynamoDB table is 1,000,000.
8. Maximum number of read request units per second: The maximum number of read request units per second for a DynamoDB table is 4,000,000.

It’s important to design your DynamoDB schema and capacity planning based on these limitations to ensure that your application can scale and perform as needed.

Features

Here is a list of some key features of Amazon DynamoDB:

1. Fast, predictable performance with single-digit millisecond latency
2. Seamless scalability to support millions of requests per second
3. Fully managed service with no infrastructure or administrative tasks to worry about
4. Flexible data model that allows you to store any data in any format
5. Multiple layers of security to protect your data, including encryption at rest and in transit
6. Integration with other AWS services, such as Amazon S3, Amazon EMR, and Amazon Redshift
7. Global tables for replication of data across regions
8. Time to Live (TTL) support to automatically delete expired items
9. Streams to capture data changes in real-time
10. Support for transactions to enable atomic operations across multiple items
11. ACID compliance to ensure data integrity and consistency
12. Integration with popular programming languages and tools, such as Python, Java, and the AWS SDKs
13. Ability to use secondary indexes to query data using non-key attributes
14. On-demand and reserved capacity modes to optimize costs
15. Serverless DynamoDB Auto Scaling to automatically adjust capacity based on workload patterns
16. Integration with AWS Identity and Access Management (IAM) to fine-tune access permissions.

DynamoDB & Kinesis

One of the coolest feature is that DyanmoDB can integrate with Kinesis to capture data changes in real-time and process them using Kinesis Streams or Kinesis Data Firehose.

To integrate DynamoDB with Kinesis Streams, you can enable DynamoDB Streams on your table. This will create a stream of data changes for your table, which can then be processed using Kinesis Streams. You can use the Kinesis Streams API to read data from the stream and process it using a Kinesis Streams application.

To integrate DynamoDB with Kinesis Data Firehose, you can use the DynamoDB Streams integration in Kinesis Data Firehose. This allows you to capture data changes in your DynamoDB table and load them into a destination, such as Amazon S3 or Amazon Redshift, using Kinesis Data Firehose.

Using these integrations, you can build real-time data pipelines to process and analyze data changes in your DynamoDB tables in near real-time.

Real World Example Using DynamoDB

Let’s say you have 1B users and you would like to support the following access patterns

1. Get user by id
2. Get user by Email
3. Get user by phone number
4. Get a user by its username (bare with me for the example)

Since DynamoDB only support partition key and secondary index we can only index by two value user id and email, which mean if we need to find a user by their phone number we will need to do a full scan.

To structure your DynamoDB schema to support login users with email and phone number, you can follow these steps:

1. Create a table with a primary key that uniquely identifies each user, such as a user ID. You can use a string type for the primary key.
2. Add both attributes email and the user’s phone number to the users table (no need adding secondary index).
3. Create another table with a primary key of the user email address (this will be out partition key), we can call this table `users-email`.
   We will do the same for the phone number. Create another table (`users-phone`) with partition key of the user phone number.
   This will help us to look up the user by their email or phone, get the user_id and with it we can fetch the user records from the users table.

The tables should be looking like this:

• users (main table): user_id (PK), email (secondary index), first_name, last_name,…
• users_phone: phone (pk), user_id
• users_username: username (pk), user_id

As you can see we can add as many table as needed and their goal is to help us get the user id when given a phone, username, etc… you name it.

Keep in mind that you may need to consider additional attributes and indexes based on your specific requirements and use case. For example, you may want to add attributes for user passwords, or indexes for searching by other user attributes such as name or location.

In summary, I am an avid supporter of DynamoDB and believe it is an excellent choice for businesses of all sizes, from small startups to large enterprises. If you have any questions or have noticed any errors in my article, please feel free to leave a comment below.

Always, happy to connect and help other startup, you can find me on Linkedin.

Happy Coding!