Skip to content

docs: expand profiling guide with JVM and async-profiler coverage#3628

Open
andygrove wants to merge 2 commits intoapache:mainfrom
andygrove:docs/profiling-guide
Open

docs: expand profiling guide with JVM and async-profiler coverage#3628
andygrove wants to merge 2 commits intoapache:mainfrom
andygrove:docs/profiling-guide

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Mar 4, 2026

Summary

Add detailed profiling guide - rendered version

Changes

  • Renames profiling_native_code.md to profiling.md and expands it into a comprehensive profiling guide
  • Adds a tool comparison table (async-profiler vs JFR vs cargo-flamegraph) with guidance on when to use each
  • Adds async-profiler section: installation, attaching to running Spark apps via asprof, java agent usage, event types, output formats, platform notes
  • Adds Java Flight Recorder section: spark-submit flags, jcmd dynamic recording, viewer options, useful JFR events for Comet debugging
  • Adds Tips for Profiling Comet section: wall-clock profiling for JNI overhead, alloc profiling around Arrow FFI, isolating Rust issues, correlating JVM/native frames
  • Preserves all existing cargo bench / cargo-flamegraph content
  • Links to benchmarks/tpc/README.md for integrated benchmark profiling

Rename profiling_native_code.md to profiling.md and add sections for
async-profiler (unified JVM + native flame graphs), Java Flight Recorder,
a tool comparison table, and practical tips for profiling Comet's mixed
JVM/Rust execution.
@andygrove andygrove marked this pull request as ready for review March 4, 2026 13:40
@andygrove andygrove requested a review from mbutrovich March 4, 2026 15:17
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove we might want to include a full working example

For example I use

TPCH Local Q4 (Profiler)
JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.5/libexec spark-submit --master="local[*]" --conf spark.driver.memory=8G --conf spark.executor.instances=1 --conf spark.executor.cores=4 --conf spark.cores.max=4 --conf spark.executor.memory=16g --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=16g --conf spark.comet.explainFallback.enabled=true --conf spark.comet.logFallbackReasons.enabled=true --conf spark.eventLog.enabled=true --jars $COMET_JAR --driver-class-path $COMET_JAR --conf spark.driver.extraClassPath=$COMET_JAR --conf spark.executor.extraClassPath=$COMET_JAR --conf spark.plugins=org.apache.spark.CometPlugin --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager --conf spark.comet.enabled=true --conf spark.comet.exec.shuffle.enableFastEncoding=true --conf spark.comet.exec.shuffle.fallbackToColumnar=true --conf spark.comet.exec.replaceSortMergeJoin=true --conf spark.comet.cast.allowIncompatible=true --conf spark.comet.scan.impl=native_iceberg_compat --conf "spark.executor.extraJavaOptions=-agentpath:/Users/xxx/Downloads/async-profiler-4.3-macos/lib/libasyncProfiler.dylib=start,event=cpu,file=profile-executor.html,tree" --conf "spark.driver.extraJavaOptions=-agentpath:/Users/xxx/Downloads/async-profiler-4.3-macos/lib/libasyncProfiler.dylib=start,event=cpu,file=profile-driver.html,tree" dev/benchmarks/tpcbench.py --name comet --benchmark tpch --data ../datafusion-benchmarks/tpch/data --queries ../datafusion-benchmarks/tpch/queries --output . --iterations 1 --query 4

And this flag helps a bit file=profile-driver.html,tree which builds an output as a tree hierarchy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants