All 7 Amazon Kinesis & Data Streaming questions
More in Amazon Kinesis & Data Streaming
What is a shard in Kinesis Data Streams, and how does it affect throughput? How does the partition key affect how records are distributed across shards in Kinesis? How do consumer applications read and process data from a Kinesis Data Stream? What is enhanced fan out in Kinesis Data Streams, and why is it useful for multiple consumers?
Advertisement
AWS › Amazon Kinesis & Data Streaming

How does data retention and replay work in Kinesis Data Streams?

Advanced

Answer

Kinesis Data Streams retains data for a default period of twenty four hours, which can be extended up to one year with extended retention enabled, and because records remain available for the entire retention period, consumer applications can replay historical data by resetting their shard iterator to an earlier position, which is extremely useful for reprocessing data after fixing a bug in your consumer logic or for backfilling a new analytics application.
aws kinesis increase-stream-retention-period --stream-name my-stream --retention-period-hours 168
Real-world example A data engineering team discovers a bug in its stream processing logic and, thanks to Kinesis's extended data retention, is able to replay the past three days of events through the corrected consumer application without any permanent data loss.

Common follow-ups: What is the cost difference between standard and extended retention periods?;How do you reset a consumer's position to replay historical data?

AWS Backup & Disaster Recovery;S3 & Storage
All 7 Amazon Kinesis & Data Streaming questions