NullPointerException in NettyClientHandler.onHeadersRead when a HEADERS frame races concurrent stream teardown

[Sairam-Arpavur-Natarajan]

What version of gRPC-Java are you using?

1.69.0 (confirmed the same unguarded code is still present at HEAD of the latest release, 1.84.0)

What did you see instead?

A NullPointerException inside NettyClientHandler.onHeadersRead, wrapped by google-cloud-pubsub's StreamingSubscriberConnection:

com.google.api.gax.rpc.UnknownException: io.grpc.StatusRuntimeException: UNKNOWN
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:119)
	...
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
	at io.grpc.Status.asRuntimeException(Status.java:532)
	... 16 common frames omitted
Caused by: java.lang.NullPointerException: Cannot invoke "io.grpc.netty.NettyClientStream$TransportState.tag()" because "stream" is null
	at io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:400)
	at io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:981)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
	...

Observed on a com.google.cloud:google-cloud-pubsub:1.134.2 Subscriber (StreamingPull), during a burst where ~16 subscriber connections were being established concurrently within a ~10s window (a batch of pods restarting and each re-subscribing to many Pub/Sub subscriptions at once).

What did you expect to see?

No NPE — a stream whose TransportState has already been cleared should be handled gracefully, the same way it's handled elsewhere in this same class.

Root cause (traced from source)

• NettyClientHandler.java:399 — NettyClientStream.TransportState stream = clientStream(requireHttp2Stream(streamId));
• NettyClientHandler.java:400 — PerfMark.event("NettyClientHandler.onHeadersRead", stream.tag()); — dereferences stream with no null check.
• NettyClientHandler.java:930-931 — clientStream():

  private NettyClientStream.TransportState clientStream(Http2Stream stream) {
    return stream == null ? null : (NettyClientStream.TransportState) stream.getProperty(streamKey);
  }

  This returns null whenever the Http2Stream object is still live in the connection (so requireHttp2Stream doesn't throw) but its TransportState property was already cleared — e.g. a stream-cancel/RST race. That's exactly the condition hit when a HEADERS frame for that stream arrives concurrently with its own teardown.

The same unguarded clientStream(requireHttp2Stream(streamId)) → immediate dereference pattern also exists at:

• NettyClientHandler.java:436-437 (onDataRead)
• NettyClientHandler.java:448-450 (onRstStreamRead)

Suggested fix

This looks like the same class of bug as #10364, which was fixed server-side in #10384 by adding a null guard right after the serverStream(requireHttp2Stream(streamId)) call:

NettyServerStream.TransportState stream = serverStream(requireHttp2Stream(streamId));
if (stream == null) {
  return;
}

The same guard pattern would apply to NettyClientHandler's onHeadersRead (and likely onDataRead/onRstStreamRead, which have the identical shape).

Impact observed

Low — this was WARN-level in our application logs (not fatal), and the underlying gax retry logic (UNKNOWN status is retryable) recovered the stream automatically. Filing for visibility since the null-dereference itself is a real, reachable bug with a known-good fix pattern already established for the server-side sibling.