Xet-backed storage benchmarks •
m5dn.24xlarge (96 vCPUs, 384GB RAM, 100 Gbps) in us-east-1
• huggingface_hub 1.7.1 + hf-xet 1.4.2
CLI (hf buckets upload) vs Rust xet-data direct (no Python overhead, no SHA256). Unique random file per run.
| CLI upload (hf buckets) | |||
|---|---|---|---|
| Size | Run | Time | Throughput |
| 100MB | 0 | 6.44s | 15.5 MB/s |
| 100MB | 1 | 2.28s | 43.9 MB/s |
| 100MB | 2 | 2.48s | 40.3 MB/s |
| 1GB | 0 | 6.70s | 152.9 MB/s |
| 1GB | 1 | 7.10s | 144.2 MB/s |
| 1GB | 2 | 6.69s | 153.0 MB/s |
| 5GB | 0 | 17.31s | 295.8 MB/s |
| 5GB | 1 | 16.70s | 306.6 MB/s |
| 5GB | 2 | 17.26s | 296.7 MB/s |
| 10GB | 0 | 33.88s | 302.3 MB/s |
| 10GB | 1 | 29.31s | 349.3 MB/s |
| 10GB | 2 | 30.11s | 340.1 MB/s |
| 50GB | 0 | 129.31s | 396.0 MB/s |
| 50GB | 1 | 127.90s | 400.3 MB/s |
| 50GB | 2 | 129.54s | 395.2 MB/s |
| 100GB | 0 | 252.90s | 404.9 MB/s |
| 100GB | 1 | 254.50s | 402.4 MB/s |
| 100GB | 2 | 249.75s | 410.0 MB/s |
| Rust direct upload (xet-data) | |||
|---|---|---|---|
| Size | Run | Time | Throughput |
| 100MB | 0 | 2.17s | 46.1 MB/s |
| 100MB | 1 | 2.21s | 45.3 MB/s |
| 100MB | 2 | 2.47s | 40.5 MB/s |
| 1GB | 0 | 6.43s | 159.2 MB/s |
| 1GB | 1 | 6.26s | 163.6 MB/s |
| 1GB | 2 | 6.19s | 165.5 MB/s |
| 5GB | 0 | 16.36s | 312.9 MB/s |
| 5GB | 1 | 16.06s | 318.9 MB/s |
| 5GB | 2 | 15.49s | 330.5 MB/s |
CDN cold = first download after upload (cache miss, full origin fetch). Warm = subsequent downloads from CloudFront edge.
| Size | Cold | Warm 0 | Warm 1 | Warm 2 | Speedup |
|---|---|---|---|---|---|
| 100MB | 1.54s (65.0 MB/s) | 1.31s | 0.51s | 0.67s (149.9) | 1.9x |
| 1GB | 3.92s (261.1 MB/s) | 3.67s | 2.11s | 1.91s (536.2) | 1.5x |
| 5GB | 7.51s (681.9 MB/s) | 7.72s | 4.34s (1180.2) | 4.46s | 1.4x |
| 10GB | 13.13s (779.9 MB/s) | 13.53s | 8.71s (1175.6) | 7.72s (1326.7) | 1.4x |
| 50GB | 46.12s (1110.2 MB/s) | 54.68s | 46.56s (1099.8) | 45.32s (1129.6) | 1.0x |
| 100GB | 93.92s (1090.3 MB/s) | 107.57s | 88.10s (1162.3) | 81.33s (1259.0) | 1.0x |
Upload a 500MB base file, then upload variants with N% bytes randomly modified. Only changed CDC chunks are transferred.
| Changed | Transferred | Saved | Duration |
|---|---|---|---|
| Base (100%) | 500 MB | — | 4.93s |
| 1% | 5.5 MB | 99% | 1.29s |
| 5% | 27.5 MB | 95% | 1.30s |
| 10% | 55 MB | 89% | 1.29s |
| 50% | 275 MB | 45% | 1.30s |
500MB file, 2% modified, re-uploaded. Only changed chunks transferred on update.
| Step | Duration | Transferred |
|---|---|---|
| Initial upload | 4.87s | 500 MB |
| Update (2% changed) | 1.08s | deduped |
100 files x 1MB, initial sync then modify 20% of files and re-sync.
| Step | Duration | Transferred | Modified |
|---|---|---|---|
| Initial sync | 5.48s | 100 MB | — |
| Incremental | 5.20s | 20 MB | 20 files |