"Error deserializing trace from request body" errors

[Fodoj]

With the latest extension (78-next), we get spammed with logs like this:

DD_EXTENSION | ERROR | Error deserializing trace from request body: Error deserializing trace from request body: IO error while reading marker: failed to fill whole buffer

and this:

{"message":"Error deserializing trace from request body: Error deserializing trace from request body: IO error while reading marker: failed to fill whole buffer"} while sending 1 (size=12KB) traces. Total: 211, Received: 211, Sent: 210, Failed: 1. Status: 500 Internal Server Error (Will not log warnings for 5 minutes)

It also happened on v69 version. Lambda's are arm64, Java 21 runtime.

[astuyve]

Hi @Fodoj thanks for the note! This likely means we received some msgpack trace that was malformed in some way. Is this something you can reliably reproduce?

We may need to get someone from the Java team involved. Can you open a support ticket? just email support@datadoghq.com to start. Thanks!

[guil-jaco-scacap]

@astuyve Confirming that it is happening to us as well. We have a few Lambdas running Java 21 with both the DD Extension v81 and the dd-trace-java v19, and we get for all those Lambdas the same error logs as @Fodoj

Is there a status update on this?

[astuyve]

Hi @guil-jaco-scacap - thanks for the note. I'm still trying to track down a trace payload which causes this, DD_TRACE_DEBUG: true usually captures the output, so if you have a payload that would be super helpful! I've created a test function that runs a few 1000 invocations per day but haven't figured out the cause, and it seems we're only hearing this from Java users.

Thanks!