The Case for Hybrid Cache for Object Stores Author: Yingjun Wu Read Time: 9 minutes Overview: This article discusses the challenges of using Amazon S3 as a backend for modern data systems due to its high latency and how RisingWave developed Foyer, a hybrid caching library in Rust, to mitigate these issues by combining memory and disk cache layers. --- Why S3 Requires a Cache S3 is widely used for its unlimited capacity, durability, and low cost, making it ideal for cloud databases and data infrastructure. However, S3 has high latency due to network round-trips for every read request, causing performance bottlenecks. Systems relying solely on S3 face a tradeoff between cost efficiency and low latency performance. Streaming systems like RisingWave are especially sensitive because they require real-time processing and multiple S3 reads per second can degrade performance severely. Introducing Foyer: Hybrid Cache Architecture Foyer is a Rust library designed to bridge the gap between low-latency and scalable storage by implementing a unified hybrid cache system combining: Memory Cache (foyer-memory crate) Fastest layer, supports sharding for concurrency. Flexible eviction policies: LRU, FIFO, w-TinyLFU, S3-FIFO, SIEVE. Request deduplication avoids redundant fetches. Uses Rust zero-copy abstractions to minimize overhead. Disk Cache (foyer-storage crate) Larger capacity than memory, sits between memory and S3. Modular engines for different data types (block, set-associative, object engines). Flexible I/O backends supporting synchronous/asynchronous reads and writes. Can use different storage devices: files, raw block devices, directories. Coordinator (main foyer crate) Manages data movement between memory and disk caches. Promotes frequently accessed data to memory. Ensures consistent API regardless of data location. Consolidates concurrent fetches to prevent request storms. Modes of Operation Hybrid Mode: Both memory and disk caches are active. Hot data served from memory, warm data from disk. Data missing in cache is fetched from S3, then cached. Memory-Only Mode: Disk cache disabled; all caching is in memory. Useful for small workloads or testing. The API remains consistent across both modes for easier adoption. Benefits and Trade-offs Benefits: Significantly improved latency since most requests are served locally. Reduced costs by fewer direct S3 accesses. Scalable storage as disk supplements memory. Configurable eviction strategies, disk engines, and I/O methods. Built-in observability with Prometheus, Grafana, OpenTelemetry. Trade-offs: Increased system complexity compared to simple in-memory caches. Requires proper configuration and tuning. Disk slower than memory, so cache management is crucial. Monitoring vital to detect and prevent performance degradation. How RisingWave Uses Foyer RisingWave treats S3 as primary storage for state, views, logs, making it elastic and cost-efficient but latency-sensitive. Implements a three-tier storage architecture: Memory: Holds hottest data for fastest access. Local Disk: Intermediate cache for frequently accessed data not fitting in memory. S3: Durable scalable store accessed only when missing in caches. Foyer manages disk caching and reduces S3 requests, enabling real-time query performance with cost control. Several strategies optimize caching effectiveness including: Block-level reads to reduce S3 requests. Sparse indexes to skip unnecessary lookups. Prefetching blocks expected to be scanned. Cache hydration by preloading newly written data. Deep integration with RisingWave allows: Disk-level I/O and cache invalidation management. Feedback on access patterns to optimize caching. Cooperation with compaction to maintain relevant cache contents. ##