feat: add micro benchmark runner and EC2 guide - #5374
Draft
andygrove wants to merge 6 commits into
Draft
Conversation
Add benchmarks/micro/run.py, a single entry point that installs the prerequisites, builds Comet in release mode, runs the 35 self-contained micro benchmark suites, collects the results with the environment they were produced in, and optionally opens a pull request with them. Results are published to benchmarks/results/micro, which is tracked. spark/benchmarks remains ignored and is used as scratch output. Add a contributor guide covering instance creation, manual prerequisite installation, running, collecting and publishing. Make the benchmark heap configurable through BENCH_HEAP so the Makefile target also works on instances with less than 20 GiB of memory.
The native build compiles the protobuf definitions with prost-build, which needs protoc on the PATH. Amazon Linux 2023 does not always provide it, and the build then fails with "Could not find protoc". Have setup fall back to the protobuf release archive when protoc is still missing after the package manager step, and document both the package and the manual install in the guide. Also treat a .git file as a checkout so that the clone check works in a git worktree.
Spark 4.x, the default profile, does not compile on JDK 11: scalac reports "Class java.lang.Record not found" and continues with a stub, which surfaces much later as a confusing compilation failure. Prefer the newest JDK under /usr/lib/jvm when JAVA_HOME is unset, rather than the first one alphabetically, and check the version before building or benchmarking so that the failure is immediate and explains itself.
The pom defaults java.version to 11 and only auto-activates a profile for JDK 11 or JDK 17 exactly. On JDK 21 neither matches, so the Spark 4.x sources compile against the Java 11 API and fail with "Class java.lang.Record not found" despite the JDK being new enough. Prefer a JDK 17 when detecting JAVA_HOME, and add -Pjdk17 to the build and benchmark invocations whenever the JDK in use is a different version.
The last lines of a failed suite log are Maven's epilogue and a NoClassDefFoundError from the Hadoop shutdown hook, which say nothing about the cause. Print the first exception in the log instead, keeping the tail for timeouts, where the end of the log is what matters. Also give CometShuffleBenchmark 180 minutes rather than the default 60, which it exceeds on a 4 vCPU instance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #1636.
Rationale for this change
Comet has around 35 micro benchmark suites, but running them is a manual, one-suite-at-a-time
exercise (
SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-<class>), and the results are writtento
spark/benchmarks, which is git ignored. That makes it tedious to produce a full set of numbersand impossible to publish them without force adding files.
Running the suites on a laptop also produces noisy numbers because of thermal throttling,
efficiency cores, and background load. A dedicated EC2 instance gives repeatable results, but the
existing EC2 guide only covers TPC-H and TPC-DS.
What changes are included in this PR?
benchmarks/micro/run.py, a dependency-free Python entry point withsetup,run,collect,publishandallsubcommands. It installs the prerequisites, clones or updates the repositoryat a chosen ref, builds in release mode, runs each suite in its own JVM with per-suite logs,
timings and timeouts, continues past failures, and copies the results into
benchmarks/results/microtogether with aRUN-INFO.mdrecording the instance type, CPU,memory, JDK, Rust and Comet commit that produced them. The script can be downloaded on its own
before the repository is cloned.
CometC2RIsolatedBenchare excluded from the default set, with thereason reported by
run --list. The default set is the 35 suites that generate their own data.benchmarks/results/microas the tracked location for published results.spark/benchmarksstays ignored and is used as scratch output.
benchmarking_micro_ec2.md, covering instance creation onm7i.xlarge, manual installation of every prerequisite (git, make, gcc, cmake,protobuf-compiler, JDK 17, Rust, and a note that Maven is provided by
./mvnw), running,collecting, publishing, guidance on getting comparable numbers, and troubleshooting.
BENCH_HEAPin theMakefile, defaulting to the previous20g, so that thebenchmark-%target can also be run by hand on an instance with less memory.
RUN-INFO.md, whose tables do not match prettier's column alignment and would otherwisefail the markdown lint job on every results PR.
How are these changes tested?
The runner was exercised locally against a release build: suite listing and
--only/--skipfiltering, a dry run of the Maven invocation, a real run of
CometArithmeticBenchmarkthatcompleted in 4m35s and produced a results file which
collectthen picked up, and the publish stepincluding the generated
RUN-INFO.mdand the git andghcommands. Documentation changes werechecked with prettier.