Amazon Kinesis & Data Streaming
7 questions foundWhat is Amazon Kinesis and what type of problems does it solve?
Beginner Amazon Kinesis is a family of services designed to collect, process, and analyze real time streaming data at scale, such as website clickstreams, application logs, or IoT sensor readings, allowing you to build applications that react to data within seconds of it being generated rather than waiting for a traditional batch processing job to run.
aws kinesis create-stream --stream-name my-stream --shard-count 1
Real-world example A ride sharing application uses Kinesis to continuously stream location updates from thousands of drivers, allowing the platform to calculate accurate estimated arrival times in near real time.
Common follow-ups: What are the different services within the Kinesis family?;How does Kinesis differ from a traditional message queue like SQS?
Amazon SQS (Simple Queue Service);Amazon EventBridge
What is the difference between Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics?
Beginner Kinesis Data Streams is the core service for capturing and storing streaming data that you process using your own custom consumer applications, Kinesis Data Firehose is a fully managed service that automatically loads streaming data into destinations like S3 or Redshift without you managing any consumer code, and Kinesis Data Analytics lets you run SQL or Apache Flink queries directly against streaming data to perform real time analysis and transformations.
aws firehose create-delivery-stream --delivery-stream-name my-firehose --s3-destination-configuration RoleARN=arn:aws:iam::123456789012:role/firehose-role,BucketARN=arn:aws:s3:::my-bucket
Real-world example A company uses Kinesis Data Firehose to automatically load raw clickstream events into S3 for long term storage, while a separate Kinesis Data Analytics application runs SQL queries on the same stream to detect unusual traffic patterns in real time.
Common follow-ups: When would you choose Firehose instead of writing your own Data Streams consumer?;What data destinations does Firehose support besides S3?
S3 & Storage;Amazon Redshift & Data Warehousing
What is a shard in Kinesis Data Streams, and how does it affect throughput?
Intermediate A shard is the base unit of capacity in a Kinesis Data Stream, with each shard supporting up to one megabyte per second of write throughput and up to two megabytes per second of read throughput, meaning the total throughput capacity of your stream scales directly with the number of shards you provision, and you can increase or decrease this number as your data volume changes.
aws kinesis update-shard-count --stream-name my-stream --target-shard-count 4 --scaling-type UNIFORM_SCALING
Real-world example A streaming analytics platform increases the shard count of its Kinesis stream from two to eight during a product launch event to handle the expected surge in incoming event volume, then scales back down afterward.
Common follow-ups: How is data distributed across shards within a stream?;What happens if a stream receives more traffic than its shards can handle?
Auto Scaling Groups;Monitoring (CloudWatch)
How does the partition key affect how records are distributed across shards in Kinesis?
Intermediate Each record written to a Kinesis stream includes a partition key, which Kinesis hashes to determine which specific shard should store that record, meaning that choosing a partition key with high variety, such as a unique device ID, spreads records evenly across all available shards, while a partition key with only a few possible values can overload just one or two shards, creating an uneven and inefficient distribution of data.
aws kinesis put-record --stream-name my-stream --data 'sensor reading' --partition-key 'device-12345'
Real-world example An IoT platform uses each device's unique serial number as the partition key when sending sensor data to Kinesis, ensuring an even spread of traffic across all shards regardless of how many devices are actively reporting data.
Common follow-ups: What happens if you use the same partition key for every record?;Can a single shard become a bottleneck even with a good partition key strategy?
Amazon DynamoDB;Auto Scaling Groups
How do consumer applications read and process data from a Kinesis Data Stream?
Intermediate Consumer applications typically use the Kinesis Client Library, which handles the complexity of tracking which shard each worker should read from, managing checkpoints so processing can resume after a failure, and automatically load balancing shard assignments across multiple worker instances, letting you focus on writing the actual business logic that processes each record rather than the low level mechanics of reading from shards.
// Simplified consumer logic using Kinesis Client Library concepts
for record in shard_iterator.get_records():
process(record)
checkpoint(record.sequence_number)
Real-world example A fraud detection system runs multiple worker instances using the Kinesis Client Library, which automatically distributes the stream's shards among the workers and tracks exactly which records each worker has already processed.
Common follow-ups: What is a checkpoint and why is it important for fault tolerance?;What is enhanced fan out and when should you use it?
Lambda & Serverless;Auto Scaling Groups
What is enhanced fan out in Kinesis Data Streams, and why is it useful for multiple consumers?
Advanced Enhanced fan out gives each registered consumer its own dedicated two megabyte per second throughput pipe for every shard using a push based delivery mechanism, meaning multiple consumer applications can read the exact same stream simultaneously at full speed without competing for the shared two megabyte per second read throughput that would otherwise need to be divided among all consumers using the standard polling model.
aws kinesis register-stream-consumer --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/my-stream --consumer-name my-analytics-consumer
Real-world example A company running both a real time fraud detection application and a separate real time reporting dashboard off the same Kinesis stream registers each as an enhanced fan out consumer, ensuring neither application's read performance is throttled by the other.
Common follow-ups: How many enhanced fan out consumers can be registered per stream?;What is the additional cost consideration of using enhanced fan out?
Amazon EventBridge;Monitoring (CloudWatch)
How does data retention and replay work in Kinesis Data Streams?
Advanced Kinesis Data Streams retains data for a default period of twenty four hours, which can be extended up to one year with extended retention enabled, and because records remain available for the entire retention period, consumer applications can replay historical data by resetting their shard iterator to an earlier position, which is extremely useful for reprocessing data after fixing a bug in your consumer logic or for backfilling a new analytics application.
aws kinesis increase-stream-retention-period --stream-name my-stream --retention-period-hours 168
Real-world example A data engineering team discovers a bug in its stream processing logic and, thanks to Kinesis's extended data retention, is able to replay the past three days of events through the corrected consumer application without any permanent data loss.
Common follow-ups: What is the cost difference between standard and extended retention periods?;How do you reset a consumer's position to replay historical data?
AWS Backup & Disaster Recovery;S3 & Storage