Skip to content

Implementation of Readable iteration helpers that does not rely on Async Iteration - #64429

Open
lukiano wants to merge 1 commit into
nodejs:mainfrom
lukiano:chore/faster-map-2
Open

Implementation of Readable iteration helpers that does not rely on Async Iteration#64429
lukiano wants to merge 1 commit into
nodejs:mainfrom
lukiano:chore/faster-map-2

Conversation

@lukiano

@lukiano lukiano commented Jul 11, 2026

Copy link
Copy Markdown

Hi, I'd like to offer an alternative implementation of Readable iteration helpers like find() that don't rely on for await (...) to consume items from the stream. This results in improved performance when the data is already available, such as when creating a Readable from an array, as shown in the attached screenshot (executed on a MacBook M2 Max).

The main con is increased duplication and code complexity. My understanding is that these helpers are still in the experimental phase, which I hope makes a change like this easier to accept.

There are many commits in the branch, but I'll squash them before merging.
Edit: I merged them to fix CI issues

The affected helpers are

  • map()
  • filter()
  • reduce()
  • find()
  • toArray()
  • some()
  • every()
  • drop()

Unaffected helpers:

  • flatmap()
  • take()

There's also a change to the from() method that increases the buffer watermark when the source is an array of data.
At first, I coded the changes manually, but eventually I had assistance from AI to keep backward compatibility. Still, there's a small break as can be seen in the updated test in test/parallel/test-stream-reduce.js.

The file benchmark/streams/operator-throughput.js allows interested parties to run the benchmarks on their computers or modify them to try other scenarios. I can remove it before merging.

stream-operator-throughput-summary

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/streams

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. stream Issues and PRs related to the stream subsystem. labels Jul 11, 2026
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.48748% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (9ad097b) to head (e751328).
⚠️ Report is 31 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/streams/operators.js 92.38% 43 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #64429      +/-   ##
==========================================
- Coverage   90.33%   90.14%   -0.19%     
==========================================
  Files         751      752       +1     
  Lines      250341   252245    +1904     
  Branches    47322    47459     +137     
==========================================
+ Hits       226145   227389    +1244     
- Misses      15575    16191     +616     
- Partials     8621     8665      +44     
Files with missing lines Coverage Δ
lib/internal/streams/from.js 91.55% <100.00%> (+4.79%) ⬆️
lib/internal/streams/operators.js 93.68% <92.38%> (-2.47%) ⬇️

... and 80 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lukiano
lukiano force-pushed the chore/faster-map-2 branch from fae33f0 to 17e0105 Compare July 12, 2026 18:24
@lukiano

lukiano commented Jul 12, 2026

Copy link
Copy Markdown
Author

I'm seeing #64447, which may change the measured performance of the current implementation.

@lukiano
lukiano force-pushed the chore/faster-map-2 branch from 17e0105 to 43335b2 Compare July 15, 2026 05:49
@lukiano
lukiano force-pushed the chore/faster-map-2 branch 3 times, most recently from e10c1f8 to 9050881 Compare August 2, 2026 22:49
@lukiano
lukiano force-pushed the chore/faster-map-2 branch from 9050881 to 64a5a60 Compare August 16, 2026 01:05
@lukiano

lukiano commented Aug 17, 2026

Copy link
Copy Markdown
Author

macbook-stream-operator-throughput-report.pdf
wsl2-stream-operator-throughput-report.pdf
linux-server-stream-operator-throughput-report.pdf
I've run the performance comparisons again with the latest changes from the main branch. macbook is the same M2 Max MacBook as before, linux-server is a 10900KF Intel CPU running on Debian 13, and wsl2 is a Windows machine with an AMD 9900x processor, and the tests ran within a WSL2 with Ubuntu 24.04

@lukiano
lukiano force-pushed the chore/faster-map-2 branch from 64a5a60 to e751328 Compare August 17, 2026 17:48
Signed-off-by: Luciano Leggieri <230980@gmail.com>
Assisted-by: Sol 5.6
@lukiano
lukiano force-pushed the chore/faster-map-2 branch from e751328 to a6618fd Compare August 17, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. stream Issues and PRs related to the stream subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants