Your Iceberg Table Is Already a Pinot Table
A Netflix format, a LinkedIn engine, and why they fit
Gonzalo Ortiz · Apache Pinot PMC · StarTree
A Netflix format, a LinkedIn engine, and why they fit
Gonzalo Ortiz · Apache Pinot PMC · StarTree
FOR VERSION AS OFPuffin: a blob file format from the Iceberg spec. It stores DVs and statistics.
day(ts), bucket(16, user_id)tsday(ts) and bucket(16, user_id)user_id spread outts and user_id| Engine | Local copy | Remote scan + cache |
|---|---|---|
| Trino / Athena / Spark | — | ✓ |
| DuckDB | ✓ | ✓ |
| Dremio | ✓ Reflections | ✓ C3 cache |
| Snowflake / BigQuery | ✓ | ✓ |
| StarRocks / Doris / ClickHouse | ✓ | ✓ data cache |
| Databricks | ✓ | ✓ disk cache |
Official documentation of each engine, checked 2026-10-01.
StarTree benchmark: 12.2 B rows, 770 GB Parquet on S3, 4 × r6gd.4xlarge for every engine. Trino 476 (Iceberg connector), ClickHouse 26.2 (icebergS3Cluster). Indexes add 270 GB.
Inverted indexes don't help aggregations
SUM
| Engine | Local copy | Own indexes on remote data | Remote scan + cache |
|---|---|---|---|
| StarTree Pinot | ✓ native | ✓ External Tables + indexes | ✓ External Tables |
| Snowflake | ✓ | ✓ search optimization* | ✓ |
| Starburst (Warp Speed) | — | ✓ index layer | ✓ |
| BigQuery | ✓ | ? external: ✗ | ✓ |
| Databricks | ✓ | ? Reyden undisclosed | ✓ |
| Trino / Athena / Spark | — | ✗ | ✓ |
| DuckDB / Dremio | ✓ | ✗ | ✓ |
| StarRocks / Doris / ClickHouse | ✓ | ✗ | ✓ |
* Snowflake-managed Iceberg tables confirmed; externally managed not documented. Official docs, checked 2026-10-01.