AWS › Amazon Kinesis & Data Streaming

How do consumer applications read and process data from a Kinesis Data Stream?

Intermediate

Answer

Consumer applications typically use the Kinesis Client Library, which handles the complexity of tracking which shard each worker should read from, managing checkpoints so processing can resume after a failure, and automatically load balancing shard assignments across multiple worker instances, letting you focus on writing the actual business logic that processes each record rather than the low level mechanics of reading from shards.

// Simplified consumer logic using Kinesis Client Library concepts
for record in shard_iterator.get_records():
    process(record)
    checkpoint(record.sequence_number)

Real-world example A fraud detection system runs multiple worker instances using the Kinesis Client Library, which automatically distributes the stream's shards among the workers and tracks exactly which records each worker has already processed.

Common follow-ups: What is a checkpoint and why is it important for fault tolerance?;What is enhanced fan out and when should you use it?

Lambda & Serverless;Auto Scaling Groups