Skip to content

requests: Add chunked response body support.#1135

Open
pablogventura wants to merge 4 commits into
micropython:masterfrom
pablogventura:requests-chunked-responses
Open

requests: Add chunked response body support.#1135
pablogventura wants to merge 4 commits into
micropython:masterfrom
pablogventura:requests-chunked-responses

Conversation

@pablogventura

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #1124, which added HTTP/1.1 with Content-Length streaming and
raised a ValueError on Transfer-Encoding: chunked responses. This PR adds
incremental decoding of chunked response bodies so requests can consume the
common case of servers that stream without a Content-Length.

A small ChunkedStream wrapper decodes the chunk framing on the fly. It keeps
the existing behaviour intact:

  • .raw.read(n) and .raw.readinto(buf) stream the decoded body without
    buffering everything in request() (works for mip).
  • .content stays lazy and materialises the full body on demand.
  • BodyStream (Content-Length) is untouched; chunked responses select
    ChunkedStream instead.

Chunk extensions after the size are skipped and trailers are discarded.

Testing

  • Unix port: micropython python-ecosys/requests/test_requests.py passes,
    including a replaced test that decodes a chunked response via .content and
    a new test that consumes it via .raw.readinto().
  • Hardware: ESP32_GENERIC (MicroPython 1.29 preview) over WiFi against
    http://httpbin.org/stream/2 (decoded via .content) and
    http://httpbin.org/stream-bytes/64 (consumed via .raw.readinto() with a
    16-byte buffer). Both returned the expected bytes.

Trade-offs and Alternatives

  • Trailers are discarded; there is no API to expose them.
  • Chunk extensions are ignored.
  • Case-insensitive header matching is out of scope here (tracked in aiohttp module case-insensitive headers #1126).
  • .content still materialises the whole body, consistent with the existing
    contract; readinto() is the incremental path.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the
code and is responsible for the code and the description above.

Decode Transfer-Encoding: chunked response bodies incrementally with a
ChunkedStream wrapper, keeping .raw streaming (read/readinto) and .content
lazy without buffering everything in request(). Chunk extensions are
skipped and trailers are discarded.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Replace the test that expected a ValueError for chunked responses with
coverage of decoding via .content and incremental .raw.readinto().

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Note chunked response decoding in the feature list and drop the
outdated limitation and follow-up entry.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
self._sock.close()


class ChunkedStream:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a lot of extra code. Can you find a way to reduce the code size, maybe combine this class with BodyStream?

To test code size, run mpy-cross __init__.py and look at the size of the resulting __init__.mpy file. Making that file smaller is the goal.

@pablogventura pablogventura Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review. I merged chunked decoding into BodyStream with a chunked flag and inlined the chunk header parsing in readinto(), removing the separate ChunkedStream class (commit 4822856).

__init__.mpy sizes measured with mpy-cross against upstream/master (no chunked support):

Version __init__.mpy vs master
master (no chunked) 2929 bytes -
this PR, separate ChunkedStream 3413 bytes +484 (+16.5%)
this PR, merged into BodyStream 3239 bytes +310 (+10.6%)

Merging the two classes saves 174 bytes (-5.1%) vs the separate-class version.

Unix tests pass. ESP32 (ESP32_GENERIC over WiFi) re-tested against httpbin chunked endpoints (.content on /stream/2, readinto on /stream-bytes/64).

Trailers are still discarded and chunk extensions are still ignored. The Content-Length code path is unchanged from master.

Combine ChunkedStream with BodyStream to reduce __init__.mpy size while
keeping incremental read/readinto behaviour for chunked responses.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants