Skip to content

Update DataFusion instructions / Enable swap on small machines#804

Open
alamb wants to merge 2 commits intoClickHouse:mainfrom
alamb:alamb/fix_low_mem_machines
Open

Update DataFusion instructions / Enable swap on small machines#804
alamb wants to merge 2 commits intoClickHouse:mainfrom
alamb:alamb/fix_low_mem_machines

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Feb 27, 2026

Rationale

I would like to

  1. Get the ClickBench benchmark results reflecting the most recent version of DataFusion
  2. More easily run pre-release DataFusion benchmarks in the ClickBench harness so we can evaluate their impact (e.g. Analyze current ClickBench performance with DataFusion 52 apache/datafusion#20601)

Since we last successfully ran the benchmarks ourselves (47.0.0), new smaller machines (c6a.xlarge 4core, 8G RAM) where the existig scripts have struggled a lot. Specifically:

  1. rustc is OOM killed (due to the link flags we pass)
  2. datafusion-cli is OOM killed (due to issues such as datafusion-cli fails to run ClickBench queries with 8GB of RAM apache/datafusion#18473)

Also, the existing instructions are somewhat outdated

Changes

  1. Update README.md to reflect reality, and incorporate changes from @waynexia in Update Results for DataFusion 52.0.0 #749 (reverted by Revert "Update Results for DataFusion 52.0.0" #766)
  2. Automatically enable swap on low memory machines (as requested by @rschu1ze in Update Results for DataFusion 52.0.0 #749 (comment))
  3. Add in the make-json.sh script to help us evaluate the current status locally

Non Changes

Note this PR does NOT update any results (I will make a follow on PR with actual numbers, I want to get the scripts into shape first)

Testing

I tested on these machines following the instructions

  • c6.xlarge
  • c8g.4xlarge

@alamb alamb force-pushed the alamb/fix_low_mem_machines branch from e6e271a to 9d5f358 Compare February 27, 2026 20:07
@alamb
Copy link
Contributor Author

alamb commented Feb 27, 2026

FYI @pmcgleenon and @waynexia I wonder what you think of this approach (just turning on swap)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant