Respect graceful_timeout and fix HTTP/2 shutdown cancellation

[emicuencac]

Fixes #308.

In the recent weeks our team faced several incidents related to this issue. When a worker has an HTTP/2 persistent connection the main process is unable to kill it.

Fix 1

Server.wait_closed() changed in Python 3.12.1 to wait for every active connection rather than just the listening socket (python/cpython#104344) –which was the original intended behavior– so a single connection that never finishes keeps the worker alive indefinitely and graceful_timeout is never reached.

Fix 2

Once Server.wait_closed() is removed, the forced cancel at graceful_timeout becomes reachable on 3.12.1+, and it exposes a second problem in HTTP/2, that was masked by the aforementioned wait_closed() change, but already existed before.

When the last stream on a terminated connection closes:

• H2Protocol sends GOAWAY and then Updated(idle=True)
• that makes the asyncio TCPServer restart its idle task inside a TaskGroup that is already being cancelled
• create_task raises RuntimeError
• the connection task fails instead of cancelling
• worker_serve raises
• and the non-zero exit makes run.py shut down the other workers too.

H11Protocol already sends Closed() in this situation, so we did the same in H2Protocol.

Fix 3

Testing the branch against a real client showed that the drain still hung, one line further down. wait_for() cancels the gathered connection tasks at graceful_timeout and then waits for the cancellation to complete, which an application that swallows CancelledError never does. The drain now uses asyncio.wait() in two bounded phases: wait graceful_timeout for the connections to finish, cancel the rest, and wait graceful_timeout again for the cancellation. asyncio.wait() also does not propagate a connection task's exception, so one failing connection no longer takes the worker (and with it the other workers) down.

Fix 4

A terminated connection answers new streams with RST_STREAM and drops the stream. The client may have already sent the DATA frame, so DataReceived arrives for a stream id that is no longer in self.streams and raises KeyError, which fails the whole connection. _handle_events now ignores DataReceived for an unknown stream, while still acknowledging the data so the flow-control window is returned.

Fix 5

The forced cancel path had not run on 3.12.1+ for two years, so a few more latent problems surfaced once it did, all of them failing a connection during the drain:

• TCPServer.protocol_send() and _close() only handled ConnectionError. drain() and wait_closed() re-raise whatever connection_lost() received, which is also ssl.SSLError (SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY #261, ssl.SSLError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2696) #291), TimeoutError from the SSL shutdown (SSL shutdown timed out #202) or a plain OSError such as EHOSTUNREACH (Plain OSError (EHOSTUNREACH) escapes TCPServer._close() on abrupt client disconnect — unhandled exception in client_connected_cb + propagates into the ASGI app #361). They now catch OSError, as _read_data() already did.
• AsyncioSingleTask.restart()/stop() swallowed the caller's own CancelledError while awaiting the cancelled idle task, so restart() went on to create_task() in a TaskGroup that was already shutting down. They now use asyncio.wait(), which does not raise the handle's CancelledError but lets the caller's through.
• H2Protocol._send_data() cleanup indexed stream_buffers and priority unguarded, so a stream that was already removed raised a second time and killed the connection's send task.

Each fix has a regression test that fails without it.