CDC File Transfer A public archive repository by Google containing tools for syncing and streaming files from Windows to Linux, based on Content Defined Chunking (CDC) techniques. --- Overview Originated to facilitate developers working on Stadia who needed to transfer game files rapidly from Windows to Linux cloud instances. Addresses problems with scp for large and frequently updated game content transfer, which copies full files without efficient delta transfers. --- Main Tools cdcrsync A Windows-to-Linux file synchronization tool optimized for: Skipping files quickly if timestamps and sizes match. Fast compression during data transfer. Transferring only changed parts of files using a CDC-based diffing algorithm (FastCDC), achieving up to 30x speedup vs. standard rsync used under Cygwin. Syncs recursively with support for wildcards. Local sync also supported. Key Features Uses variable-sized, content-defined chunks vs. fixed-size chunks (standard rsync). Prevents cascading chunk boundary shifts; only altered chunks transfer. Simplifies and speeds up chunk matching. Performance Syncing large builds (~40-45 GB) can be 3x faster than Cygwin rsync. Linux native rsync performance compared in documentation. cdcstream A Windows-to-Linux streaming tool resembling sshfs but optimized for read speed: Caches streamed data on Linux. Only streams file differences upon re-read. Fast directory metadata retrieval suitable for streaming. Uses the same CDC-based diffing algorithm as cdcrsync. Linux directory is read-only; no write support back to Windows. Performance Provides 2x to 5x speed improvements over sshfs in game startup times tests. --- Supported Platforms | Tool | From Platform | To Platform | |-------------|--------------------------|-----------------------------| | cdcrsync | Windows x8664 (✓) | Windows x8664 (✓ local only) / Ubuntu 22.04 x8664 (✓) | | | Ubuntu 22.04 x8664 (✗) | | | | Ubuntu 22.04 aarch64 (✗) | | | | macOS 13 x8664 (✗) | | | | macOS 13 aarch64 (✗) | | | Tool | From Platform | To Platform | |-------------|--------------------------|-----------------------------| | cdcstream | Windows x8664 (✓) | Ubuntu 22.04 x8664 (✓) | | | Ubuntu 22.04 x8664 (✗) | | | | Ubuntu 22.04 aarch64 (✗) | | | | macOS 13 x8664 (✗) | | | | macOS 13 aarch64 (✗) | | Note: Remote syncing support for Windows target is under development. See GitHub issues #61, #56, #62. --- Getting Started Installation Download precompiled binaries from the latest release. Linux binaries deploy automatically to ~/.cache/cdc-file-transfer by Windows tools. Building from source requires Bazel and SSH client (ssh.exe and sftp.exe). Prerequisites for Building Install Bazel. Clone repository and initialize submodules: Ensure SSH and SFTP tools are available on Windows. --- Building CDC RSync Build Linux server component on Linux: Build Windows client component on Windows: