HuggingFace Buckets Benchmarks

Xet-backed Buckets vs plain S3 • m5dn.24xlarge (96 vCPUs, 384GB RAM, 100 Gbps) in us-east-1 • huggingface_hub 1.7.1 + hf-xet 1.4.2 • Buckets-only view →

Upload Throughput S3 vs Buckets

Raw upload speed with unique random files per run (no dedup). S3 uses direct PUT, Buckets use CDC chunking + dedup.

Size	Run	S3	Bucket (CLI)
100MB	0	1.20s (83.6 MB/s)	6.44s (15.5 MB/s)
100MB	1	1.72s (58.3 MB/s)	2.28s (43.9 MB/s)
100MB	2	1.16s (86.1 MB/s)	2.48s (40.3 MB/s)
1GB	0	3.90s (262.8 MB/s)	6.70s (152.9 MB/s)
1GB	1	3.60s (284.2 MB/s)	7.10s (144.2 MB/s)
1GB	2	3.15s (325.2 MB/s)	6.69s (153.0 MB/s)
5GB	0	12.93s (395.8 MB/s)	17.31s (295.8 MB/s)
5GB	1	13.11s (390.4 MB/s)	16.70s (306.6 MB/s)
5GB	2	12.08s (423.7 MB/s)	17.26s (296.7 MB/s)
10GB	0	23.29s (439.7 MB/s)	33.88s (302.3 MB/s)
10GB	1	25.44s (402.5 MB/s)	29.31s (349.3 MB/s)
10GB	2	26.01s (393.8 MB/s)	30.11s (340.1 MB/s)
50GB	0	120.60s (424.5 MB/s)	129.31s (396.0 MB/s)
50GB	1	122.50s (418.0 MB/s)	127.90s (400.3 MB/s)
50GB	2	121.80s (420.3 MB/s)	129.54s (395.2 MB/s)
100GB	0	249.20s (410.9 MB/s)	252.90s (404.9 MB/s)
100GB	1	242.30s (422.6 MB/s)	254.50s (402.4 MB/s)
100GB	2	246.50s (415.4 MB/s)	249.75s (410.0 MB/s)

Download Throughput S3 vs Buckets

CDN cold = first download (cache miss, origin fetch). Warm = subsequent downloads from edge.

Size	Run	S3	Bucket	CDN
100MB	0-2	1.14s, 1.03s, 1.01s (avg 94.6 MB/s)	—	n/a
100MB	cold	—	1.54s (65.0 MB/s)	cold
100MB	warm 0-2	—	1.31s, 0.51s, 0.67s (avg 140.7 MB/s)	warm
1GB	0-2	3.30s, 3.15s, 3.47s (avg 310.5 MB/s)	—	n/a
1GB	cold	—	3.92s (261.1 MB/s)	cold
1GB	warm 0-2	—	3.67s, 2.11s, 1.91s (avg 433.9 MB/s)	warm
5GB	0-2	12.95s, 12.19s, 13.46s (avg 398.6 MB/s)	—	n/a
5GB	cold	—	7.51s (681.9 MB/s)	cold
5GB	warm 0-2	—	7.72s, 4.34s, 4.46s (avg 996.7 MB/s)	warm
10GB	0-2	24.62s, 25.10s, 24.85s (avg 412.0 MB/s)	—	n/a
10GB	cold	—	13.13s (779.9 MB/s)	cold
10GB	warm 0-2	—	13.53s, 8.71s, 7.72s (avg 1086.4 MB/s)	warm
50GB	0-2	121.5s, 123.8s, 122.1s (avg 418.1 MB/s)	—	n/a
50GB	cold	—	46.12s (1110.2 MB/s)	cold
50GB	warm 0-2	—	54.68s, 46.56s, 45.32s (avg 1055.3 MB/s)	warm
100GB	0-2	246.2s, 250.1s, 243.8s (avg 415.1 MB/s)	—	n/a
100GB	cold	—	93.92s (1090.3 MB/s)	cold
100GB	warm 0-2	—	107.57s, 88.10s, 81.33s (avg 1124.4 MB/s)	warm

Deduplication S3 vs Buckets

500MB base file, then upload variants with N% of bytes modified. S3 always re-uploads the full file.

Scenario	Changed	S3 transferred	Bucket transferred	Saved
Base upload	100%	500 MB	500 MB	—
1% modified	1%	500 MB	5.5 MB	99%
5% modified	5%	500 MB	27.5 MB	95%
10% modified	10%	500 MB	55 MB	89%
50% modified	50%	500 MB	275 MB	45%

Incremental Update S3 vs Buckets

500MB file, 2% modified then re-uploaded.

Scenario	S3	Bucket
Initial upload	2.14s	4.87s
Update (2% changed)	2.32s (full)	1.08s (deduped)

Directory Sync S3 vs Buckets

100 files x 1MB, sync then modify 20% of files and re-sync.

Scenario	S3	Bucket
Initial sync	2.08s	5.48s
Incremental (20% changed)	1.07s	5.20s

CDN Cache Performance Buckets

Cold vs warm CDN download. Warm = xorbs served from CloudFront edge.

Size	Cold (MB/s)	Warm avg (MB/s)	Warm peak (MB/s)	Speedup
100MB	65.0	140.7	195.9	2.2x
1GB	261.1	433.9	536.2	1.7x
5GB	681.9	996.7	1180.2	1.5x
10GB	779.9	1086.4	1326.7	1.4x
50GB	1110.2	1055.3	1129.6	1.0x
100GB	1090.3	1124.4	1259.0	1.0x

Dedup Efficiency Buckets

Bytes actually transferred as a function of file change percentage. CDC chunks are ~64KB.

Rust Direct Upload xet-data

Upload using xet-data crate directly (no Python, no SHA256). Unique file per run. CLI now nearly matches Rust on large files thanks to hf-xet 1.4.2.

Size	Run	CLI Python (hub 1.7.1)	Rust direct	Speedup
100MB	avg	3.73s (33.2 MB/s)	2.28s (43.9 MB/s)	1.3x
1GB	avg	6.83s (150.0 MB/s)	6.29s (162.8 MB/s)	1.1x
5GB	avg	17.09s (299.7 MB/s)	15.97s (320.8 MB/s)	1.1x

Upload Warm-up Effect Buckets

Upload throughput per run with unique random files. Run 0 includes cold start overhead (JIT, connection setup).