The Future of Repair: Transparent and Incremental by Botond De?nes

Mar 5, 20250 likes75 views

Regularly run repairs are essential to keep clusters healthy, yet having a good repair schedule is more challenging than it should be. Repairs often take a long time, preventing running them often. This has an impact on data consistency and also limits the usefulness of the new repair based tombstone garbage collection. We want to address these challenges by making repairs incremental and allowing for automatic repair scheduling, without relying on external tools.

A ScyllaDB Community
The Future of Repair:
Transparent and Incremental
Botond Denes
Software Team Lead

$�� Consistent data is important �� Consistent tombstones is even more important �� tombstone_gc = {'mode':'timeout'} - at least one repair per GC_GRACE_SECONDS �� tombstone_gc = {'mode':'repair'} - only tombstones written before last repair can be garbage-collected Repair is important$

�� Slow
�� Large dataset
�� Mixed-shard cluster
�� Need external tools to schedule
�� nodetool - manual
�� scylla-manager - needs provisioning
Repair is challenging

Incremental Repair
Not every repair needs to repair all the data!
�� Repair only data written since last repair
�� The more frequent repairs are �C the less work they have to do
�� Repairing often is no longer a problem

Move scheduling into ScyllaDB core!
�� Operational simplicity - no need for external tools
�� External tools can only observe past/current state
�� Internal scheduling is also aware of plans
Automatic repair

�� Tablet only
�� Tablet repair is simpler - 1 peer shard on each peer node
�� Tablet repair happens via the tablet scheduler
�� User requests tablet(s) to be repaired via /storage_service/tablets/repair
�� Request saved in system.tablets
�� Repair of individual tablets is scheduled
�� Excludes with ongoing migrations
�� Consider load on individual nodes and shards
The way to incremental and automatic repair

Stay in Touch
Botond Denes
bdenes@scylladb.com
denesb

Predictable performance isn't just a feature��it's a must for anyone building serious, data-intensive applications at scale. ScyllaDB has often been labeled merely as a faster alternative to Cassandra, but is that all there is to it? In this talk, let's discuss the results of our head-to-head comparison between the long-delayed Apache Cassandra 5.0 release versus ScyllaDB and explain what truly sets ScyllaDB apart beyond just speed.

Efficient Deduplication in ReSys at Scale: How We Managed to Reduce Server Co...ScyllaDB

Deploying ScyllaDB as an On-Premise Replacement to DynamoDB by Chuck WilliamsScyllaDB

Scaling Cron at Slack by Claire Adams, SlackScyllaDB

Revolutionizing Sleep: Scaling IoT Telemetry to 30+ Billion Daily Events by D...ScyllaDB

ScyllaDB: 10 Years and Beyond by Dor LaorScyllaDB

There��s a common adage that it takes 10 years to develop a file system. As ScyllaDB reaches that 10 year milestone in 2025, it��s the perfect time to reflect on the last decade of ScyllaDB development �C both hits and misses. It��s especially appropriate given that our project just reached a critical mass with certain scalability and elasticity goals that we dreamed up years ago. This talk will cover how we arrived at ScyllaDB X Cloud achieving our initial vision, and share where we��re heading next.

Feature Store Evolution Under Cost Constraints: When Cost is Part of the Arch...ScyllaDB

Read- and Write-Optimization in Modern Database Infrastructures by Dzejla Med...ScyllaDB

Securely Serving Millions of Boot Artifacts a Day by Joa?o Pedro Lima & Matt ...ScyllaDB

Gmetrics: Processing Metrics at Uber Scale by Cristian VelazquezScyllaDB

Replacing RocksDB with ScyllaDB in Kafka Streams by Almog GavraScyllaDB

Using ScyllaDB to Implement Lists in Medium��s Feature Store by Andreas SaudemontScyllaDB

30B Images and Counting: Scaling Canva's Content-Understanding Pipelines by K...ScyllaDB

Evolving Atlassian Confluence Cloud for Scale, Reliability, and Performance b...ScyllaDB

Route It Like It��s Hot: Scaling Payments Routing at American Express by Benja...ScyllaDB

ScyllaDB��s Monstrous Engineering Advances by Avi KivityScyllaDB

Architecture for Extreme Scale by Avi KivityScyllaDB

How We Boosted ScyllaDB Data Streaming by 25x by Asias HeScyllaDB

Streaming, the process of scaling out/in to other nodes used to analyze every partition, one-by-one and was too slow and depended on the schema. File based stream is a new feature that optimizes tablet movement significantly. It streams the entire SSTable files without deserializing SSTable files into mutation fragments and re-serializing them back into SSTables on receiving nodes. As a result, less data is streamed over the network, and less CPU is consumed, especially for data models that contain small cells.

How Discord Indexes Trillions of Messages: Scaling Search Infrastructure by V...ScyllaDB

Inside Freshworks' Migration from Cassandra to ScyllaDB by Premkumar PatturajScyllaDB

Data Structures Handling Trillions of Daily Streaming Events by Evan ChanScyllaDB

Building a Scalable Event-Driven Architecture for Open Finance Brasil by Thi...ScyllaDB

DynamoDB Cost Optimization Considerations and Strategies by Alex DeBrieScyllaDB

How Discord Performs Database Upgrades at Scale by Ethan DonowitzScyllaDB

Discord relies on ScyllaDB to serve millions of reads per second across many clusters, so they needed a comprehensive strategy to sufficiently de-risk upgrades to avoid impact to our users. To accomplish this, they use what they call ��shadow clusters.�� This talk explains how testing with shadow clusters has been paramount to de-risking complicated upgrades for one of the most important pieces of infrastructure at Discord.

Telemetry Showdown: Fluent Bit vs. OpenTelemetry Collector by Henrik RexedScyllaDB

Pushing Your Streaming Platform to the Limit by Elad LeevScyllaDB

Overcome Redis Cluster Scale Bottlenecks with ScyllaDB & EloqKV by Hubert ZhangScyllaDB

Caching for Performance Masterclass: Caching at ScaleScyllaDB

Understanding Traditional AI with Custom Vision & MuleSoft.pptxshyamraj55

Understanding Traditional AI with Custom Vision & MuleSoft.pptx | ### �ݺ�ߣ Deck Description: This presentation features Atul, a Senior Solution Architect at NTT DATA, sharing his journey into traditional AI using Azure's Custom Vision tool. He discusses how AI mimics human thinking and reasoning, differentiates between predictive and generative AI, and demonstrates a real-world use case. The session covers the step-by-step process of creating and training an AI model for image classification and object detection��specifically, an ad display that adapts based on the viewer's gender. Atulavan highlights the ease of implementation without deep software or programming expertise. The presentation concludes with a Q&A session addressing technical and privacy concerns.

Formal Methods: Whence and Whither? [Martin Fr?nzle Festkolloquium, 2025]Jonathan Bowen

Alan Turing arguably wrote the first paper on formal methods 75 years ago. Since then, there have been claims and counterclaims about formal methods. Tool development has been slow but aided by Moore��s Law with the increasing power of computers. Although formal methods are not widespread in practical usage at a heavyweight level, their influence as crept into software engineering practice to the extent that they are no longer necessarily called formal methods in their use. In addition, in areas where safety and security are important, with the increasing use of computers in such applications, formal methods are a viable way to improve the reliability of such software-based systems. Their use in hardware where a mistake can be very costly is also important. This talk explores the journey of formal methods to the present day and speculates on future directions.

More Related Content

More from ScyllaDB (20)