Skip to content
AppDues
Log in
Contact salesSign up

Replicating a Write-Heavy Kvrocks Dataset in Real Time

When subscriber metadata changes constantly, a relational primary database can become a bottleneck: row updates move on disk, write amplification grows, vacuum pressure rises, and billable IOPS climb. Moving that write-heavy dataset to Kvrocks—which uses RocksDB under the hood and speaks the Redis protocol—relieves the pressure, but it creates a new problem: the warehouse still needs those changes. This guide walks through the architecture: reading RocksDB change streams, reconciling node-local sequence numbers with application-level epochs, packaging the reader as a sidecar service, and driving exports from a stateful job.

Why move write-heavy data to Kvrocks

PostgreSQL's write-ahead log and row-update mechanics make very high write volumes expensive. Each write can change a row's on-disk location, forcing references to be updated and producing write amplification. Frequent updates also create dead tuples that increase vacuuming work, and on RDS the added write volume shows up as billable IOPS.

Kvrocks sits on RocksDB, whose LSM tree is designed for write-heavy workloads. Because Kvrocks uses the Redis protocol, application code can use a familiar Redis client. The trade-off is that features needing warehouse aggregation—such as a chart showing the share of Android versus iOS users—no longer read directly from the primary database. A pipeline must ship changes from Kvrocks to the warehouse.

Use this decision checklist before committing to the pattern:

  • Is the dataset dominated by frequent small updates rather than large analytical scans?
  • Does the application already tolerate Redis-protocol access?
  • Do downstream features require warehouse aggregation that the operational store cannot serve?
  • Can the team accept a replication delay measured in seconds rather than zero?
  • Is there a place to stage exported files before warehouse ingestion?

If most answers are yes, a change-stream pipeline is a reasonable fit.

Streaming changes out of Kvrocks

Kvrocks itself does not expose hooks for reading incoming changes. The underlying RocksDB does: getUpdatesSince accepts a RocksDB sequence number—effectively a checkpoint—and returns an iterator over every change after that point. Because Kvrocks is an abstraction over RocksDB, some reverse engineering is required to parse Kvrocks key names and identify which RocksDB column families serve which purposes. Kvrocks is open source, so the implementation is available to inspect.

The harder problem is multi-node operation. A Kvrocks deployment may run several nodes to balance reads and writes and to provide redundancy. RocksDB is unaware of this topology; it is only the storage engine on each node. A sequence number on one node does not correspond to the same change on another node.

The workaround is an application-level epoch. The pipeline periodically writes an epoch to a special Kvrocks key. That epoch becomes the replication checkpoint. During export, the epoch is treated as the last-seen checkpoint, and the replication API plus local-instance RocksDB sequence numbers are used to scan for when that epoch was written. Changes after that write are then exported. The epoch gives a checkpoint that is consistent across Kvrocks nodes even though the underlying RocksDB sequence numbers are node-local.

Failure distinction to keep clear: a missing or stale epoch is not the same as a RocksDB read error. A RocksDB error means the change stream could not be read on that node; a stale epoch means the cross-node checkpoint has not advanced, so the exporter may re-scan or stall even though each node's local stream is healthy.

Packaging the reader as a sidecar service

The replication agent runs alongside each Kvrocks node. It uses the RocksDB Java library to connect to the local Kvrocks RocksDB database in read-only mode. The service is written in Kotlin with Dropwizard and exposes two endpoints:

  • Export endpoint — accepts the last-seen checkpoint identifier and returns a JSON response containing S3 paths to CSVs with the changes, plus the next checkpoint identifier to use on the following export.
  • Checkpoint endpoint — returns the current replication checkpoint, which is used to track how far behind replication is.

Kotlin was chosen because RocksDB offers two libraries: a C++ library and a Java wrapper around it. The team was most familiar with building in Java, so the Java wrapper was the practical choice.

In the current setup, changes are written to an S3 bucket, ingested by the replication system, and upserted into Snowflake. The design is generic enough to adapt to other data stores.

Field and control map for the sidecar:

  • Input to export: last-seen checkpoint identifier.
  • Output from export: JSON with S3 paths to CSV change files and the next checkpoint identifier.
  • Input to checkpoint: none.
  • Output from checkpoint: current replication checkpoint.
  • Access mode: read-only connection to the local RocksDB database.
  • State: none held by the agent itself.

Sending changes to the data warehouse

The replication agents are stateless, which matters because multiple nodes run in parallel. State lives in a job in the main codebase that tracks the current checkpoint.

The job uses Redis Sentinel—already used elsewhere to select appropriate Kvrocks servers—to obtain a Kvrocks reader. It then makes an HTTP request to export changes from the last-seen checkpoint. When the response arrives, the job copies the data into the data lake for the replication system to ingest into the warehouse.

A worked example using the captured evidence: suppose the job's stored checkpoint is epoch E1. It asks a reader for changes since E1. The sidecar scans the local RocksDB stream for the write of E1, collects subsequent changes, writes CSVs to S3, and returns their paths together with the next checkpoint E2. The job copies the CSVs into the data lake and records E2. If the job crashes before recording E2, the next run repeats from E1; the export is idempotent at the checkpoint level because the same epoch is used as the boundary.

In the original deployment, the end-to-end delay from a change landing in Kvrocks to replication was about 10–15 seconds. The checkpoint mechanism was described as somewhat hacky but robust in practice. Treat that latency figure as a dated reference from the original setup, not a guarantee for other deployments.

Frequently asked questions

Why not keep the write-heavy dataset in PostgreSQL?

High write volumes in PostgreSQL can change row locations on disk, force references to be updated, and produce write amplification. They also generate dead tuples that increase vacuuming and raise billable IOPS on RDS. Kvrocks on RocksDB uses an LSM tree that is better suited to frequent writes.

Why can't RocksDB sequence numbers be shared across Kvrocks nodes?

RocksDB is only the storage engine on each node and is unaware of the Kvrocks cluster topology. A sequence number on one node does not map to the same change on another node, so the pipeline uses an application-level epoch written to a special Kvrocks key as a cross-node checkpoint.

Does the sidecar service hold replication state?

No. The sidecar is stateless and connects read-only to the local RocksDB database. The current checkpoint is owned by a job in the main codebase, which uses Redis Sentinel to pick a Kvrocks reader and then calls the sidecar's export endpoint.

How much replication latency should I expect?

The original setup reported roughly 10–15 seconds from a change landing in Kvrocks to replication into the warehouse. That figure is a dated reference from that deployment; your latency will depend on export frequency, file sizes, and warehouse ingestion speed.