Skip to content

Run performance benchmarks on PRs and report the impact #343

Description

@sgerbino

Problem

Performance is a headline property of corosio and we have no automated
signal for it. perf/bench already measures nine categories
(socket_throughput, socket_latency, local_socket_*, timer,
http_server, fan_out, accept_churn, io_context) and writes
JSON, but it only runs when someone remembers to run it locally.
Regressions are found late, by hand, and improvements are asserted in
PR descriptions without evidence anyone can check.

What we need

  1. Every PR gets a performance verdict: PR head vs merge-base, per
    category, per backend, on Linux (epoll, io_uring, select), Windows
    (iocp) and macOS (kqueue), corosio only. IOCP and kqueue are
    separate implementations, not ports; a Linux-only signal says
    nothing about them.
  2. The verdict is trustworthy. A flagged regression must be real,
    and a clean result must mean "no change larger than the measured
    noise floor". The method has to state its noise floor and only flag
    deltas outside it.
  3. The result is visible where the review happens: a PR comment,
    updated in place on each push, with one table — category, metric,
    base, head, delta, and a marker on significant rows. Full JSON as
    a workflow artifact.
  4. Advisory, not blocking. A flagged row is a conversation starter
    on the PR, not a red check.

Dedicated hardware

Benchmarks run on dedicated hardware, one machine per platform, that
does nothing else. Hosted runners are too noisy to trust.

Method: build base and head, alternate base/head for N iterations,
report the paired delta. A row is flagged when its delta exceeds the
variance observed in base-vs-base runs on the same machine.

The trigger is gated so untrusted PRs cannot run arbitrary code on the
machine: automatic for maintainers, label-approved for everyone else.

Scope

  • Provision one runner per platform; document the setup
  • Measure base-vs-base variance per category, metric and platform;
    that is each platform's noise floor
  • Workflow: build base + head, interleaved run, JSON artifact per
    platform, one PR comment with a section per platform; gated
    trigger as above

Acceptance

  • A PR touching include/ or src/ gets a benchmark comment covering
    all three platforms, updated on each push.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Ready

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions