Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@
```

[Impit](https://github.com/apify/impit) is the default HTTP client and is installed automatically. To use the
built-in [HTTPX](https://github.com/pydantic/httpx2) client instead, install the optional `httpx2` extra, which
built-in [HTTPX2](https://github.com/pydantic/httpx2) client instead, install the optional `httpx2` extra, which
provides Pydantic's maintained continuation of HTTPX, and pass `http_client=Httpx2HttpClient()` to
`ApifyClient.with_custom_http_client()`:

Expand Down Expand Up @@ -135,7 +135,7 @@ For a guided walkthrough — authenticating, running an Actor, and reading its r
- **Tiered timeouts** — short / medium / long tiers picked per endpoint, overridable per call ([Timeouts](https://docs.apify.com/api/client/python/docs/concepts/timeouts)).
- **Pagination and streaming** — iterate datasets, key-value store keys, or live logs without manual paging or buffering ([Pagination](https://docs.apify.com/api/client/python/docs/concepts/pagination), [Streaming](https://docs.apify.com/api/client/python/docs/concepts/streaming-resources)).
- **Convenience methods** — `call()`, `wait_for_finish()`, nested resource access, and other shortcuts that hide platform quirks ([Convenience methods](https://docs.apify.com/api/client/python/docs/concepts/convenience-methods)).
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
- **Pluggable HTTP layer** — use the default [Impit](https://github.com/apify/impit)-based client, opt in to the built-in [HTTPX2](https://github.com/pydantic/httpx2) client, or plug in any custom implementation ([HTTP clients](https://docs.apify.com/api/client/python/docs/concepts/custom-http-clients)).
- **Structured errors** — every API error surfaces as an [`ApifyApiError`](https://docs.apify.com/api/client/python/reference/class/ApifyApiError) with HTTP-specific subclasses for precise handling ([Error handling](https://docs.apify.com/api/client/python/docs/concepts/error-handling)).
- **Debug logging** — opt-in structured logging on the `apify_client` logger captures request URLs, status codes, retry attempts, and more ([Logging](https://docs.apify.com/api/client/python/docs/concepts/logging)).

Expand Down
2 changes: 1 addition & 1 deletion docs/01_introduction/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ For better request-body compression, opt in to `brotli`, which compresses better

For details, see [HTTP compression](../02_concepts/13_http_compression.mdx).

The client uses [Impit](https://github.com/apify/impit) as its default HTTP transport. To use the built-in [HTTPX](https://github.com/pydantic/httpx2) transport, install the optional `httpx2` extra, which provides Pydantic's maintained continuation of HTTPX:
The client uses [Impit](https://github.com/apify/impit) as its default HTTP transport. To use the built-in [HTTPX2](https://github.com/pydantic/httpx2) transport, install the optional `httpx2` extra, which provides Pydantic's maintained continuation of HTTPX:

<Tabs>
<TabItem value="PyPI" label="PyPI" default>
Expand Down
22 changes: 11 additions & 11 deletions docs/02_concepts/10_custom_http_clients.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import ArchitectureImportsExample from '!!raw-loader!./code/10_architecture_impo
import PluggingInAsyncExample from '!!raw-loader!./code/10_plugging_in_async.py';
import PluggingInSyncExample from '!!raw-loader!./code/10_plugging_in_sync.py';

The Apify API client uses a pluggable HTTP layer. It ships with an [Impit](https://github.com/apify/impit)-based default, offers [HTTPX](https://github.com/pydantic/httpx2) as an optional built-in alternative, and accepts custom synchronous or asynchronous implementations.
The Apify API client uses a pluggable HTTP layer. It ships with an [Impit](https://github.com/apify/impit)-based default, offers [HTTPX2](https://github.com/pydantic/httpx2) as an optional built-in alternative, and accepts custom synchronous or asynchronous implementations.

## Default HTTP client

Expand All @@ -45,11 +45,11 @@ You can configure the default client through the <ApiLink to="class/ApifyClient"
</TabItem>
</Tabs>

## Built-in HTTPX client
## Built-in HTTPX2 client

The package also provides <ApiLink to="class/Httpx2HttpClient">`Httpx2HttpClient`</ApiLink> and <ApiLink to="class/Httpx2HttpClientAsync">`Httpx2HttpClientAsync`</ApiLink>. They use the same request preparation, compression, retry policy, timeout tiers and growth, error handling, logging, and statistics as the default Impit clients, with [HTTPX](https://github.com/pydantic/httpx2) as the transport.
The package also provides <ApiLink to="class/Httpx2HttpClient">`Httpx2HttpClient`</ApiLink> and <ApiLink to="class/Httpx2HttpClientAsync">`Httpx2HttpClientAsync`</ApiLink>. They use the same request preparation, compression, retry policy, timeout tiers and growth, error handling, logging, and statistics as the default Impit clients, with [HTTPX2](https://github.com/pydantic/httpx2) as the transport.

HTTPX is an optional dependency provided by the `httpx2` package, Pydantic's maintained continuation of HTTPX. Install `apify-client[httpx2]`, then pass the appropriate client to <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink>. Impit remains the default even when the extra is installed.
The `httpx2` package, Pydantic's maintained continuation of HTTPX, is an optional dependency. Install `apify-client[httpx2]`, then pass the appropriate client to <ApiLink to="class/ApifyClient#with_custom_http_client">`ApifyClient.with_custom_http_client`</ApiLink>. Impit remains the default even when the extra is installed.

```bash
pip install "apify-client[httpx2]"
Expand All @@ -70,17 +70,17 @@ uv add "apify-client[httpx2]"
</TabItem>
</Tabs>

Configure retries, timeout tiers, default headers, and compression on the HTTPX client instance. The token passed to `with_custom_http_client` is applied automatically unless the HTTP client already has an `Authorization` header. The examples use the clients as context managers so their connection pools are closed deterministically. If a context manager doesn't fit your application's lifecycle, call `close()` on `Httpx2HttpClient` or `await aclose()` on `Httpx2HttpClientAsync` during shutdown.
Configure retries, timeout tiers, default headers, and compression on the HTTPX2 client instance. The token passed to `with_custom_http_client` is applied automatically unless the HTTP client already has an `Authorization` header. The examples use the clients as context managers so their connection pools are closed deterministically. If a context manager doesn't fit your application's lifecycle, call `close()` on `Httpx2HttpClient` or `await aclose()` on `Httpx2HttpClientAsync` during shutdown.

Timeout values are passed to the selected transport. Impit enforces them as a deadline for the whole request, body included. HTTPX applies them to each socket operation instead, so a response whose body arrives slowly keeps resetting the timeout and can outlast both the requested timeout and `timeout_max`. The `no_timeout` option disables HTTPX's timeouts.
Timeout values are passed to the selected transport. Impit enforces them as a deadline for the whole request, body included. HTTPX2 applies them to each socket operation instead, so a response whose body arrives slowly keeps resetting the timeout and can outlast both the requested timeout and `timeout_max`. The `no_timeout` option disables HTTPX2's timeouts.

## Architecture

Internally, the HTTP client hierarchy has three layers:

- A common internal base contains configuration and utilities shared by synchronous and asynchronous clients, including headers, request-body preparation, parameters, compression, and timeout tiers. It isn't a public extension point.
- <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> and <ApiLink to="class/HttpClientAsync">`HttpClientAsync`</ApiLink> add the synchronous or asynchronous request pipeline, retry loop, transport hooks, and lifecycle interface.
- The built-in Impit and HTTPX classes inherit directly from the corresponding sync or async class and adapt the underlying transport.
- The built-in Impit and HTTPX2 classes inherit directly from the corresponding sync or async class and adapt the underlying transport.

`HttpClient.is_timeout_error(exc)` and `HttpClientAsync.is_timeout_error(exc)` are the public, transport-neutral way to tell whether an exception is a timeout, so code built on the client, such as streamed logs, doesn't need to know which transport raised it.

Expand All @@ -103,7 +103,7 @@ The public `call` method provides the shared request pipeline. A concrete transp
- `is_timeout_error(exc)` identifies transport-specific timeout exceptions for higher-level client features. The default recognizes Python's `TimeoutError`. Timeout classification is independent of retryability, so a timeout the retry loop should retry has to be listed in `is_retryable_transport_error` too.
- `close()` or `aclose()` closes resources owned by the transport. The default does nothing, which is correct for a transport that owns no pool or session.

Decorate your implementations with `@override`, as the built-in Impit and HTTPX adapters do, so a type checker catches a misspelled or incompatible override.
Decorate your implementations with `@override`, as the built-in Impit and HTTPX2 adapters do, so a type checker catches a misspelled or incompatible override.

### The HTTP response protocol

Expand All @@ -124,9 +124,9 @@ Decorate your implementations with `@override`, as the built-in Impit and HTTPX
| `aiter_bytes() -> AsyncIterator[bytes]` | Iterate body in chunks (async) |

:::note
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX](https://github.com/pydantic/httpx2) already satisfy this protocol out of the box.
Many HTTP libraries, including our default [Impit](https://github.com/apify/impit) or for example [HTTPX2](https://github.com/pydantic/httpx2) already satisfy this protocol out of the box.

For a streamed response, consume the body inside the streaming context manager with `iter_bytes()` / `aiter_bytes()`, or call `read()` / `aread()` before accessing `content`. Some transports, including HTTPX, intentionally reject `content` on an unread streamed response.
For a streamed response, consume the body inside the streaming context manager with `iter_bytes()` / `aiter_bytes()`, or call `read()` / `aread()` before accessing `content`. Some transports, including HTTPX2, intentionally reject `content` on an unread streamed response.
:::

### Plugging it in
Expand Down Expand Up @@ -154,7 +154,7 @@ If you override `call` itself, your implementation becomes responsible for reque

## Use cases

Custom HTTP clients might be useful when the built-in Impit and HTTPX clients don't cover your requirements, for example when you need to:
Custom HTTP clients might be useful when the built-in Impit and HTTPX2 clients don't cover your requirements, for example when you need to:

- **Use a different HTTP library** - Integrate [requests](https://requests.readthedocs.io/), [aiohttp](https://docs.aiohttp.org/), or another transport.
- **Route through a proxy** - Add proxy support or request routing.
Expand Down
4 changes: 2 additions & 2 deletions docs/03_guides/05_custom_http_client.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import CustomHttpClientSyncExample from '!!raw-loader!./code/05_custom_http_clie

This guide implements a custom <ApiLink to="class/HttpClientAsync">`HttpClientAsync`</ApiLink> with [aiohttp](https://docs.aiohttp.org/) and a custom <ApiLink to="class/HttpClient">`HttpClient`</ApiLink> with [requests](https://requests.readthedocs.io/). Neither library satisfies the <ApiLink to="class/HttpResponse">`HttpResponse`</ApiLink> protocol, so both examples also show how to adapt a foreign response API.

For an overview of the architecture and the built-in Impit and HTTPX implementations, see [HTTP clients](../02_concepts/10_custom_http_clients.mdx).
For an overview of the architecture and the built-in Impit and HTTPX2 implementations, see [HTTP clients](../02_concepts/10_custom_http_clients.mdx).

## Installation

Expand Down Expand Up @@ -47,5 +47,5 @@ Each example has three parts:
</Tabs>

:::warning
These examples are compact integrations, not a replacement for all built-in client behavior. A production custom client should account for transport-specific details such as proxy configuration, TLS settings, redirects, and response resource cleanup. Timeout semantics differ per transport too: the aiohttp example passes the value as a budget for the whole request, while `requests` applies it to each socket read. Both example sessions also keep a shared cookie jar, which replays server cookies on later API requests. The built-in HTTPX client clears it instead.
These examples are compact integrations, not a replacement for all built-in client behavior. A production custom client should account for transport-specific details such as proxy configuration, TLS settings, redirects, and response resource cleanup. Timeout semantics differ per transport too: the aiohttp example passes the value as a budget for the whole request, while `requests` applies it to each socket read. Both example sessions also keep a shared cookie jar, which replays server cookies on later API requests. The built-in HTTPX2 client clears it instead.
:::
2 changes: 1 addition & 1 deletion docs/04_upgrading/upgrading_to_v2.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Support for Python 3.9 has been dropped. The Apify Python API Client v2.x now re

## New underlying HTTP library

In v2.0, the Apify Python API client switched from using [`httpx`](https://github.com/pydantic/httpx2) to [`impit`](https://github.com/apify/impit) as the underlying HTTP library. However, this change shouldn't have much impact on the end user.
In v2.0, the Apify Python API client switched from using [`httpx`](https://www.python-httpx.org/) to [`impit`](https://github.com/apify/impit) as the underlying HTTP library. However, this change shouldn't have much impact on the end user.

## API method changes

Expand Down
2 changes: 1 addition & 1 deletion src/apify_client/http_clients/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

_install_import_hook(__name__)

# `httpx2` is an optional extra, so the import is wrapped in try_import. Accessing the HTTPX clients without the
# `httpx2` is an optional extra, so the import is wrapped in try_import. Accessing the HTTPX2 clients without the
# extra installed raises a clear ImportError instead of failing at package import time.
with _try_import(
__name__,
Expand Down
Loading
Loading