ci: guard the PR comment against GitHub's 65536-character limit - #183
ci: guard the PR comment against GitHub's 65536-character limit#183atassis wants to merge 2 commits into
Conversation
The aggregate body is posted unchecked. Once all four suites have reported for a head SHA it runs past the cap: on run 33550798970 it was over 265000 characters and the API returned 422 Validation Failed, so the PR received no results comment at all. Shed the per-suite details first, then truncate the summary, keeping the marker so the next run still updates the same comment. Claude-Session: https://claude.ai/code/session_014y7hzKhehcyoe74h2pRmfW
|
Any good way to test this? Maybe we could make a lot of tests fail on purpose in this PR so we can see it in action (and we can fix them before we merge for real)? |
CI Test Results7e49fd8 (2026_09_01_22_41_43) IRON - CI SummarySmalliron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Phoenix - SmallIRONTested on iron/operators/axpy
iron/operators/dequant
iron/operators/elementwise_add
iron/operators/elementwise_mul
iron/operators/gelu
iron/operators/gemm
iron/operators/gemv
iron/operators/layer_norm
iron/operators/leaky_relu
iron/operators/mem_copy
iron/operators/relu
iron/operators/repeat
iron/operators/rms_norm
iron/operators/rope
iron/operators/sigmoid
iron/operators/silu
iron/operators/softmax
iron/operators/strided_copy
iron/operators/swiglu_decode
iron/operators/swiglu_prefill
iron/operators/tanh
iron/operators/transpose
Trends: IRON Trendsiron/operators/axpytest_axpy[input_length_2048-num_aie_columns_1-tile_size_2048-scalar_factor_3.0]
test_axpy[input_length_2048-num_aie_columns_2-tile_size_1024-scalar_factor_3.0]
test_axpy[input_length_2048-num_aie_columns_4-tile_size_512-scalar_factor_3.0]
iron/operators/dequanttest_dequant[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-group_size_32]
test_dequant[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-group_size_32]
test_dequant[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-group_size_32]
test_dequant[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-group_size_32]
test_dequant[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-group_size_32]
test_dequant[input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-group_size_32]
iron/operators/elementwise_addtest_elementwise_add[input_length_2048-num_aie_columns_1-tile_size_2048]
test_elementwise_add[input_length_2048-num_aie_columns_2-tile_size_1024]
test_elementwise_add[input_length_2048-num_aie_columns_4-tile_size_512]
iron/operators/elementwise_multest_elementwise_mul[input_length_2048-num_aie_columns_1-tile_size_2048]
test_elementwise_mul[input_length_2048-num_aie_columns_2-tile_size_1024]
test_elementwise_mul[input_length_2048-num_aie_columns_4-tile_size_512]
iron/operators/gelutest_gelu[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048]
test_gelu[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024]
test_gelu[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024]
test_gelu[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512]
test_gelu[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512]
test_gelu[input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256]
iron/operators/gemmtest_gemm[M_192-K_384-N_64-num_aie_columns_4-b_col_maj_False-c_col_maj_False-m_48-k_96-n_16-trace_size_0-partition_N_1]
test_gemm[M_192-K_384-N_64-num_aie_columns_4-b_col_maj_True-c_col_maj_True-m_48-k_96-n_16-trace_size_0-partition_N_1]
test_gemm[M_2048-K_2048-N_2048-num_aie_columns_1-b_col_maj_False-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1]
test_gemm[M_2048-K_2048-N_2048-num_aie_columns_2-b_col_maj_True-c_col_maj_False-m_64-k_64-n_64-trace_size_0-partition_N_1]
test_gemm[M_384-K_1536-N_1792-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_32-k_48-n_64-trace_size_0-partition_N_1]
test_gemm[M_64-K_512-N_256-num_aie_columns_4-b_col_maj_True-c_col_maj_False-m_16-k_64-n_64-trace_size_0-partition_N_4]
iron/operators/gemvtest_gemv[M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128]
test_gemv[M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048]
test_gemv[M_2048-K_8192-num_aie_columns_2-tile_size_input_1-tile_size_output_1024]
test_gemv[M_2048-K_8192-num_aie_columns_4-tile_size_input_1-tile_size_output_512]
test_gemv[M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024]
test_gemv[M_8192-K_2048-num_aie_columns_2-tile_size_input_4-tile_size_output_1024]
test_gemv[M_8192-K_2048-num_aie_columns_4-tile_size_input_4-tile_size_output_1024]
test_gemv_batched[M_1024-K_1024-num_aie_columns_1-tile_size_input_1-tile_size_output_64-num_batches_2]
test_gemv_batched[M_1026-K_64-num_aie_columns_1-tile_size_input_1-tile_size_output_2-num_batches_2]
test_gemv_batched[M_256-K_128-num_aie_columns_1-tile_size_input_1-tile_size_output_256-num_batches_4]
test_gemv_batched[M_64-K_1536-num_aie_columns_1-tile_size_input_1-tile_size_output_64-num_batches_8]
test_gemv_gelu[M_128-K_128-num_aie_columns_1-tile_size_input_32-tile_size_output_128]No metrics available. test_gemv_gelu[M_2048-K_8192-num_aie_columns_1-tile_size_input_1-tile_size_output_2048]No metrics available. test_gemv_gelu[M_8192-K_2048-num_aie_columns_1-tile_size_input_4-tile_size_output_1024]No metrics available. iron/operators/layer_normtest_layer_norm[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048]
test_layer_norm[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024]
test_layer_norm[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024]
test_layer_norm[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512]
test_layer_norm[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512]
test_layer_norm[input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256]
iron/operators/leaky_relutest_leaky_relu[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-alpha_0.01]
test_leaky_relu[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-alpha_0.1]
test_leaky_relu[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-alpha_0.25]
test_leaky_relu[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-alpha_0.01]
test_leaky_relu[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-alpha_0.01]
test_leaky_relu[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-alpha_0.01]
test_leaky_relu[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-alpha_0.01]
test_leaky_relu[input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-alpha_0.01]
iron/operators/mem_copytest_mem_copy[input_length_2048-num_cores_1-num_channels_1-bypass_False-tile_size_2048]
test_mem_copy[input_length_2048-num_cores_2-num_channels_1-bypass_False-tile_size_1024]
test_mem_copy[input_length_2048-num_cores_2-num_channels_2-bypass_False-tile_size_1024]
test_mem_copy[input_length_2048-num_cores_4-num_channels_1-bypass_False-tile_size_512]
test_mem_copy[input_length_2048-num_cores_4-num_channels_2-bypass_False-tile_size_512]
test_mem_copy[input_length_2048-num_cores_8-num_channels_2-bypass_False-tile_size_256]
iron/operators/repeattest_cols_without_a_legal_split_is_rejected[cols_1031-why_prime > 1023: the only divisors are 1 and cols, neither legal]No metrics available. test_cols_without_a_legal_split_is_rejected[cols_2062-why_2 x 1031: the only word-aligned chunk leaves a 1031-wide chunk count]No metrics available. test_cols_without_a_legal_split_is_rejected[cols_513-why_odd: every divisor is odd, so no chunk is a whole 32-bit word]No metrics available. test_repeat[rows_4-cols_1024-repeat_2-transfer_size_None]
test_repeat[rows_8-cols_512-repeat_4-transfer_size_64]
test_repeat[rows_8-cols_64-repeat_4-transfer_size_None]
iron/operators/rms_normtest_rms_norm[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-weighted_False]
test_rms_norm[input_length_2048-num_aie_columns_1-num_channels_1-tile_size_2048-weighted_True]
test_rms_norm[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-weighted_False]
test_rms_norm[input_length_2048-num_aie_columns_1-num_channels_2-tile_size_1024-weighted_True]
test_rms_norm[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-weighted_False]
test_rms_norm[input_length_2048-num_aie_columns_2-num_channels_1-tile_size_1024-weighted_True]
test_rms_norm[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-weighted_False]
test_rms_norm[input_length_2048-num_aie_columns_2-num_channels_2-tile_size_512-weighted_True]
test_rms_norm[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-weighted_False]
test_rms_norm[input_length_2048-num_aie_columns_4-num_channels_1-tile_size_512-weighted_True]
test_rms_norm[input_length_2048-num_aie_columns_4-num_channels_2-tile_size_256-weighted_False]
iron/operators/ropetest_rope[rows_32-cols_512-angle_rows_32-aie_columns_1-method_type_0]
test_rope[rows_32-cols_512-angle_rows_32-aie_columns_2-method_type_0]
test_rope[rows_32-cols_512-angle_rows_32-aie_columns_4-method_type_0]
test_rope[rows_32-cols_512-angle_rows_8-aie_columns_1-method_type_0]
test_rope[rows_32-cols_512-angle_rows_8-aie_columns_2-method_type_0]
test_rope[rows_32-cols_512-angle_rows_8-aie_columns_4-method_type_0]
iron/operators/softmaxtest_softmax[input_length_32768-num_aie_columns_2-num_channels_2-tile_size_1024]
test_softmax[input_length_32768-num_aie_columns_2-num_channels_2-tile_size_2048]
test_softmax[input_length_32768-num_aie_columns_2-num_channels_2-tile_size_512]
iron/operators/strided_copytest_strided_copy[chunked_transfer]
test_strided_copy[contiguous]
test_strided_copy[four_channels]
test_strided_copy[kv_slot0]
test_strided_copy[kv_slot5]
test_strided_copy[kv_slot5_four_channels]
test_strided_copy[kv_slot5_two_channels]
test_strided_copy[kv_slot_last]
test_strided_copy[two_channels]
test_strided_copy[two_channels_chunked]
test_transfer_size_not_dividing_per_channel_share_is_rejected[iter0]No metrics available. test_transfer_size_not_dividing_per_channel_share_is_rejected[iter1]No metrics available. test_transfer_size_not_dividing_per_channel_share_is_rejected[iter2]No metrics available. test_transfer_size_not_dividing_per_channel_share_is_rejected[iter3]No metrics available. test_transfer_size_not_dividing_per_channel_share_is_rejected[iter4]No metrics available. iron/operators/swiglu_decodetest_swiglu_decode[embedding_dim_1024-hidden_dim_3584]
test_swiglu_decode[embedding_dim_2048-hidden_dim_2048]
iron/operators/swiglu_prefilltest_swiglu_prefill[seq_len_256-embedding_dim_2048-hidden_dim_2048-prio_accuracy_False]
iron/operators/transposetest_transpose[M_2048-N_64-aie_columns_1-channels_1-m_64-n_64-s_8-num_batches_1]
test_transpose[M_2048-N_64-aie_columns_1-channels_1-m_64-n_64-s_8-num_batches_2]
test_transpose[M_2048-N_64-aie_columns_1-channels_1-m_64-n_64-s_8]
test_transpose[M_2048-N_64-aie_columns_1-channels_2-m_64-n_64-s_8-num_batches_1]
test_transpose[M_2048-N_64-aie_columns_1-channels_2-m_64-n_64-s_8]
Phoenix - ExamplesIRONTested on Trends: IRON Trends |
|
Testing it from this branch won't work: Good news is we don't need broken tests. The failure reproduces on any SHA where all four suites have reported, a279467 does it every time. Bad news is you shouldn't merge this yet. I checked the comments on #150-#183: they run 122k to 142k characters and all of them posted fine, so 65536 isn't the limit GitHub actually enforces. As written my guard would strip the details off every comment, not just the huge ones. Let me redo it to shed only on a real 422, and drop |
65536 is the number the error message names, not the ceiling GitHub enforces: every aggregate comment on amd#150-amd#183 is 122k-142k characters and posted, so the guard would have stripped the details off every comment. Try the full body and drop a section only when the API rejects one, trends.md first.
|
Pushed as fe71df5. It now sheds only when the API actually rejects the body, trend tables first, which on the run that failed leaves an 84k comment with every pass/fail table intact. Body updated too. |
pr-comment.ymlposts the aggregate body in one call. Once all four suites have reported for a head SHA it can exceed what GitHub accepts: on run 33550798970 the body was 313,521 characters and the API returned422 Validation Failed: body is too long (maximum is 65536 characters), so the PR received no results comment. Earlier runs on the same SHA posted fine because fewer suites had finished.The 65536 in that message is not the enforced ceiling. Every aggregate comment on #150-#183 runs between 122,494 and 142,053 characters and all of them posted, so a guard comparing against a constant would strip the details off comments GitHub accepts today. Post the full body instead, and shed a section only when GitHub actually rejects one.
Sections go in size order. The trend tables are the bulk of an oversized body, so they go first: on run 33550798970 that takes the body from 313,521 to 83,961 characters and keeps every per-suite pass/fail table. After that the per-suite details, then the summary halved until it fits. The marker stays in the body on every rung, so the update branch keeps finding the existing comment instead of creating a new one each run.
Verified by replaying that run's four result artifacts through the script with a stubbed API rejecting above a given size: at 262144 it lands on the trends-dropped rung, at 65536 on the summary, and with nothing over the limit it makes a single call with the body unchanged.
Added
Comment on PRstep, with acore.warningwhen the comment had to be shortened.Changed
headerconstant so each rung can be rebuilt from it.post()helper.Removed
PR Merge Checklist
develcommit and pointing todevel.