Yes, but parquet hates small files. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		aynyc 17 days ago \| parent \| context \| favorite \| on: Apache Arrow is 10 years old Yes, but parquet hates small files.

dionian 16 days ago [–]

You can't compact? i.e. iceberg maintenance

aynyc 16 days ago | [–]

We might be doing something wrong, but we saw significant performance degradation for both ingestion and query when doing compaction when it comes to finance data during trading hours.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact