Implement a `clitest` utility binary for testing CLIs by jviotti · Pull Request #2734 · sourcemeta/core

jviotti · 2026-08-16T19:09:07Z

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

github-actions

Benchmark (linux/llvm)

Details

Benchmark suite	Current: `6e58379`	Previous: `1bdc7aa`	Ratio
`Regex_Lower_S_Or_Upper_S_Asterisk`	`2.4952626828616307` ns/iter	`2.4614068886173266` ns/iter	`1.01`
`Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar`	`2.46237710032457` ns/iter	`2.465556433296174` ns/iter	`1.00`
`Regex_Period_Asterisk`	`2.4628495706344014` ns/iter	`2.459946443258057` ns/iter	`1.00`
`Regex_Group_Period_Asterisk_Group`	`2.4624725091176787` ns/iter	`2.4606565789410046` ns/iter	`1.00`
`Regex_Period_Plus`	`3.8709895576354714` ns/iter	`3.870499055688326` ns/iter	`1.00`
`Regex_Period`	`3.926336181204674` ns/iter	`3.8710999191297044` ns/iter	`1.01`
`Regex_Caret_Period_Plus_Dollar`	`3.870346416511841` ns/iter	`3.864916426004858` ns/iter	`1.00`
`Regex_Caret_Group_Period_Plus_Group_Dollar`	`3.868478563727115` ns/iter	`3.864960212911421` ns/iter	`1.00`
`Regex_Caret_Period_Asterisk_Dollar`	`2.828977543257055` ns/iter	`2.8184070019899603` ns/iter	`1.00`
`Regex_Caret_Group_Period_Asterisk_Group_Dollar`	`2.825920634267818` ns/iter	`2.829201901794657` ns/iter	`1.00`
`Regex_Caret_X_Hyphen`	`5.480784465074552` ns/iter	`5.628644921113485` ns/iter	`0.97`
`Regex_Period_Md_Dollar`	`26.214139050972324` ns/iter	`26.001834214967758` ns/iter	`1.01`
`Regex_Caret_Slash_Period_Asterisk`	`7.320333898100582` ns/iter	`7.196413850827649` ns/iter	`1.02`
`Regex_Caret_Period_Range_Dollar`	`4.219785332024488` ns/iter	`3.106762585200279` ns/iter	`1.36`
`Regex_Nested_Backtrack`	`38.30113517728612` ns/iter	`38.32253368622031` ns/iter	`1.00`
`JSON_Array_Of_Objects_Unique`	`442.4667953396457` ns/iter	`441.32030136125866` ns/iter	`1.00`
`JSON_Parse_1`	`4429.846093517548` ns/iter	`4599.14496295629` ns/iter	`0.96`
`JSON_Parse_Real`	`5200.329748745022` ns/iter	`5273.148264984219` ns/iter	`0.99`
`JSON_Parse_Decimal`	`7743.152875888697` ns/iter	`7884.738164976899` ns/iter	`0.98`
`JSON_Parse_Schema_ISO_Language`	`3392343.6105769593` ns/iter	`3448640.137930807` ns/iter	`0.98`
`JSON_Parse_Integer`	`3958.5849719818475` ns/iter	`3990.331585577707` ns/iter	`0.99`
`JSON_Parse_String_NonSSO_Plain`	`3052.931351061519` ns/iter	`3118.0240356148975` ns/iter	`0.98`
`JSON_Parse_String_SSO_Plain`	`2749.1047571429676` ns/iter	`2825.089134104851` ns/iter	`0.97`
`JSON_Parse_String_Escape_Heavy`	`12519.955310256544` ns/iter	`12547.923264530773` ns/iter	`1.00`
`JSON_Parse_Object_Short_Keys`	`7842.638406113545` ns/iter	`7851.936547627789` ns/iter	`1.00`
`JSON_Parse_Object_Scalar_Properties`	`4007.2836498900047` ns/iter	`4001.992996653089` ns/iter	`1.00`
`JSON_Parse_Object_Array_Properties`	`5555.322369516505` ns/iter	`5425.059486161915` ns/iter	`1.02`
`JSON_Parse_Object_Object_Properties`	`5640.679905967456` ns/iter	`5444.728637656447` ns/iter	`1.04`
`JSON_Parse_Nested_Containers`	`35272.61657609911` ns/iter	`35514.71793831037` ns/iter	`0.99`
`JSON_From_String_Copy`	`12.509650634709784` ns/iter	`13.873193590582096` ns/iter	`0.90`
`JSON_From_String_Temporary`	`10.39508572672463` ns/iter	`11.95994188922048` ns/iter	`0.87`
`JSON_Number_To_Double`	`23.823960512544204` ns/iter	`22.78984469438049` ns/iter	`1.05`
`JSON_Object_At_Last_Key/8`	`6.3366117862763005` ns/iter	`6.346492731392161` ns/iter	`1.00`
`JSON_Object_At_Last_Key/32`	`23.546450668160247` ns/iter	`23.56430204231684` ns/iter	`1.00`
`JSON_Object_At_Last_Key/128`	`91.06240077603388` ns/iter	`91.10852221440474` ns/iter	`1.00`
`JSON_Object_At_Last_Key/512`	`379.66490864659943` ns/iter	`381.04069833976666` ns/iter	`1.00`
`JSON_Fast_Hash_Helm_Chart_Lock`	`72.32799299330466` ns/iter	`72.21513547203324` ns/iter	`1.00`
`JSON_Equality_Helm_Chart_Lock`	`172.09883433780197` ns/iter	`159.89468127343073` ns/iter	`1.08`
`JSON_Divisible_By_Decimal`	`249.36044071167834` ns/iter	`246.58521118895277` ns/iter	`1.01`
`JSON_String_Equal/10`	`6.328746805497704` ns/iter	`5.625756910965577` ns/iter	`1.12`
`JSON_String_Equal/100`	`6.331842694228194` ns/iter	`6.333636193289884` ns/iter	`1.00`
`JSON_String_Equal_Small_By_Perfect_Hash/10`	`1.054110256553494` ns/iter	`1.054594863122733` ns/iter	`1.00`
`JSON_String_Equal_Small_By_Runtime_Perfect_Hash/10`	`12.343243307579325` ns/iter	`12.373883456479764` ns/iter	`1.00`
`JSON_String_Fast_Hash/10`	`2.4639426741761805` ns/iter	`2.466869733758363` ns/iter	`1.00`
`JSON_String_Fast_Hash/100`	`2.4645214178658366` ns/iter	`2.463024746318123` ns/iter	`1.00`
`JSON_String_Key_Hash/10`	`2.81570971092115` ns/iter	`2.81419559962278` ns/iter	`1.00`
`JSON_String_Key_Hash/100`	`8.083404546038272` ns/iter	`7.730922208533574` ns/iter	`1.05`
`JSON_Object_Defines_Miss_Same_Length`	`2.970024890772721` ns/iter	`2.9548481490530767` ns/iter	`1.01`
`JSON_Object_Defines_Miss_Too_Small`	`3.0391936194302307` ns/iter	`2.9644719484918713` ns/iter	`1.03`
`JSON_Object_Defines_Miss_Too_Large`	`3.2036603380188233` ns/iter	`3.2282487251083247` ns/iter	`0.99`
`Pointer_Object_Traverse`	`30.676719395084547` ns/iter	`30.616567233516108` ns/iter	`1.00`
`Pointer_Object_Try_Traverse`	`33.22980042408204` ns/iter	`32.86708151526342` ns/iter	`1.01`
`Pointer_Push_Back_Pointer_To_Weak_Pointer`	`184.29845425133908` ns/iter	`158.53553295238166` ns/iter	`1.16`
`Pointer_Walker_Schema_ISO_Language`	`1556395.1609978229` ns/iter	`1507770.0668104042` ns/iter	`1.03`
`Pointer_Maybe_Tracked_Deeply_Nested/0`	`1233864.4946997243` ns/iter	`1243630.36964285` ns/iter	`0.99`
`Pointer_Maybe_Tracked_Deeply_Nested/1`	`1606436.0668203298` ns/iter	`1614502.6572768744` ns/iter	`1.00`
`Pointer_Position_Tracker_Get_Deeply_Nested`	`707.0384327899147` ns/iter	`701.722324689787` ns/iter	`1.01`
`JSONPath_Descendant_Filter_Nested`	`1566.7350620800746` ns/iter	`1532.225731517003` ns/iter	`1.02`
`URITemplateRouter_Create`	`22137.649395988352` ns/iter	`21660.835062402304` ns/iter	`1.02`
`URITemplateRouter_Match`	`187.67264448710807` ns/iter	`187.3123159929642` ns/iter	`1.00`
`URITemplateRouter_Match_BasePath`	`222.96077087473287` ns/iter	`222.1038771737079` ns/iter	`1.00`
`URITemplateRouterView_Restore`	`9872.776420970327` ns/iter	`9729.892899791914` ns/iter	`1.01`
`URITemplateRouterView_Match`	`166.7580935050314` ns/iter	`166.43687295703907` ns/iter	`1.00`
`URITemplateRouterView_Match_BasePath`	`192.8127290161938` ns/iter	`187.75402967903503` ns/iter	`1.03`
`URITemplateRouterView_Arguments`	`452.002811227617` ns/iter	`466.91951183526913` ns/iter	`0.97`
`JSONL_Parse_Large`	`9026492.185183896` ns/iter	`8781320.153846722` ns/iter	`1.03`
`JSONL_Parse_Large_GZIP`	`10332592.220586723` ns/iter	`10404778.089550678` ns/iter	`0.99`
`JSONLD_Catalog_Annotation_List_Populate`	`615614.6911130224` ns/iter	`592976.9865546206` ns/iter	`1.04`
`JSONLD_Catalog_Materialize`	`3690909.417989575` ns/iter	`3624785.544041402` ns/iter	`1.02`
`HTML_Build_Table_100000`	`68038687.40000781` ns/iter	`67094392.19999923` ns/iter	`1.01`
`HTML_Render_Table_100000`	`1977991.0712164019` ns/iter	`1923156.4337015566` ns/iter	`1.03`
`GZIP_Compress_ISO_Language_Set_3_Locations`	`35344658.99999759` ns/iter	`35226796.500001` ns/iter	`1.00`
`GZIP_Decompress_ISO_Language_Set_3_Locations`	`4278525.049382557` ns/iter	`4269316.292683326` ns/iter	`1.00`
`GZIP_Compress_ISO_Language_Set_3_Schema`	`2098092.8742516995` ns/iter	`2093632.9401198116` ns/iter	`1.00`
`GZIP_Decompress_ISO_Language_Set_3_Schema`	`278498.0790937679` ns/iter	`277440.067749589` ns/iter	`1.00`
`JOSE_VerifySignature_RS256`	`63899.54175432195` ns/iter	`63957.97320940975` ns/iter	`1.00`
`JOSE_VerifySignature_ES512`	`2675166.0954200537` ns/iter	`2654959.410646363` ns/iter	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Benchmark (macos/llvm)

Details

Benchmark suite	Current: `6e58379`	Previous: `1bdc7aa`	Ratio
`Regex_Lower_S_Or_Upper_S_Asterisk`	`1.6812165182462555` ns/iter	`2.3511407750255007` ns/iter	`0.72`
`Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar`	`1.7410421599403412` ns/iter	`2.6401452932944327` ns/iter	`0.66`
`Regex_Period_Asterisk`	`1.8036228058340833` ns/iter	`2.534395479240884` ns/iter	`0.71`
`Regex_Group_Period_Asterisk_Group`	`1.7042657080787633` ns/iter	`2.3036077026916706` ns/iter	`0.74`
`Regex_Period_Plus`	`2.107016566336347` ns/iter	`2.851320956948415` ns/iter	`0.74`
`Regex_Period`	`2.0632674214757176` ns/iter	`2.5830216502129475` ns/iter	`0.80`
`Regex_Caret_Period_Plus_Dollar`	`2.051578431541414` ns/iter	`2.9671849136963067` ns/iter	`0.69`
`Regex_Caret_Group_Period_Plus_Group_Dollar`	`1.9794302331602447` ns/iter	`2.9646684698342107` ns/iter	`0.67`
`Regex_Caret_Period_Asterisk_Dollar`	`1.6774826742760902` ns/iter	`2.7847008143207814` ns/iter	`0.60`
`Regex_Caret_Group_Period_Asterisk_Group_Dollar`	`1.7325372618271295` ns/iter	`2.8201387527840005` ns/iter	`0.61`
`Regex_Caret_X_Hyphen`	`6.012942889049384` ns/iter	`8.179441124849733` ns/iter	`0.74`
`Regex_Period_Md_Dollar`	`18.178439158569944` ns/iter	`26.665061183543397` ns/iter	`0.68`
`Regex_Caret_Slash_Period_Asterisk`	`4.606619827543933` ns/iter	`6.035753250454431` ns/iter	`0.76`
`Regex_Caret_Period_Range_Dollar`	`1.9406936592637092` ns/iter	`3.0494217005687982` ns/iter	`0.64`
`Regex_Nested_Backtrack`	`25.647158897458617` ns/iter	`48.621327796436724` ns/iter	`0.53`
`JSON_Array_Of_Objects_Unique`	`374.8774696119433` ns/iter	`515.637234226989` ns/iter	`0.73`
`JSON_Parse_1`	`3150.7951892504025` ns/iter	`4593.291599502784` ns/iter	`0.69`
`JSON_Parse_Real`	`5470.393420599745` ns/iter	`7571.353937232248` ns/iter	`0.72`
`JSON_Parse_Decimal`	`6228.647754109023` ns/iter	`8381.51910405771` ns/iter	`0.74`
`JSON_Parse_Schema_ISO_Language`	`2303891.2440678347` ns/iter	`3423061.4889865406` ns/iter	`0.67`
`JSON_Parse_Integer`	`3105.465135684734` ns/iter	`5329.541629547672` ns/iter	`0.58`
`JSON_Parse_String_NonSSO_Plain`	`3046.613050020426` ns/iter	`5789.825000000518` ns/iter	`0.53`
`JSON_Parse_String_SSO_Plain`	`1652.1783878165031` ns/iter	`3012.7691375705444` ns/iter	`0.55`
`JSON_Parse_String_Escape_Heavy`	`16119.690046150447` ns/iter	`23919.377261228765` ns/iter	`0.67`
`JSON_Parse_Object_Short_Keys`	`4912.341796439137` ns/iter	`9756.486010233668` ns/iter	`0.50`
`JSON_Parse_Object_Scalar_Properties`	`2483.6290451999485` ns/iter	`5152.891249999811` ns/iter	`0.48`
`JSON_Parse_Object_Array_Properties`	`3141.0966972486276` ns/iter	`7442.908380407772` ns/iter	`0.42`
`JSON_Parse_Object_Object_Properties`	`3267.2849014925077` ns/iter	`5401.858598838992` ns/iter	`0.60`
`JSON_Parse_Nested_Containers`	`26339.56632258545` ns/iter	`43518.67357430457` ns/iter	`0.61`
`JSON_From_String_Copy`	`12.933260418714843` ns/iter	`17.302649155794747` ns/iter	`0.75`
`JSON_From_String_Temporary`	`6.219019065170001` ns/iter	`10.122686641160348` ns/iter	`0.61`
`JSON_Number_To_Double`	`30.506173123142105` ns/iter	`50.73501053824068` ns/iter	`0.60`
`JSON_Object_At_Last_Key/8`	`3.7794965101430393` ns/iter	`7.321552692734242` ns/iter	`0.52`
`JSON_Object_At_Last_Key/32`	`11.360375622224304` ns/iter	`17.605112296960435` ns/iter	`0.65`
`JSON_Object_At_Last_Key/128`	`51.51232895526946` ns/iter	`80.0853346789128` ns/iter	`0.64`
`JSON_Object_At_Last_Key/512`	`174.68582286305215` ns/iter	`272.5704923264795` ns/iter	`0.64`
`JSON_Fast_Hash_Helm_Chart_Lock`	`54.40584747687418` ns/iter	`82.70833538248662` ns/iter	`0.66`
`JSON_Equality_Helm_Chart_Lock`	`132.90541992114828` ns/iter	`213.58718640485776` ns/iter	`0.62`
`JSON_Divisible_By_Decimal`	`172.86033649575205` ns/iter	`247.6907791038247` ns/iter	`0.70`
`JSON_String_Equal/10`	`6.638501109699608` ns/iter	`8.780118425244137` ns/iter	`0.76`
`JSON_String_Equal/100`	`6.351953305578223` ns/iter	`8.39834825116721` ns/iter	`0.76`
`JSON_String_Equal_Small_By_Perfect_Hash/10`	`0.3306533028615711` ns/iter	`0.4026732093126597` ns/iter	`0.82`
`JSON_String_Equal_Small_By_Runtime_Perfect_Hash/10`	`3.156356520488854` ns/iter	`3.997017874240189` ns/iter	`0.79`
`JSON_String_Fast_Hash/10`	`2.201733897247004` ns/iter	`2.9949995518558294` ns/iter	`0.74`
`JSON_String_Fast_Hash/100`	`1.9061668943265258` ns/iter	`2.4039491511521667` ns/iter	`0.79`
`JSON_String_Key_Hash/10`	`1.5961365968747234` ns/iter	`1.8589538543715762` ns/iter	`0.86`
`JSON_String_Key_Hash/100`	`1.9822071764782447` ns/iter	`2.883787092523144` ns/iter	`0.69`
`JSON_Object_Defines_Miss_Same_Length`	`3.099792847846764` ns/iter	`3.958633544705866` ns/iter	`0.78`
`JSON_Object_Defines_Miss_Too_Small`	`3.2535660221350113` ns/iter	`3.2689556916647655` ns/iter	`1.00`
`JSON_Object_Defines_Miss_Too_Large`	`3.221257874535317` ns/iter	`3.2444044438446515` ns/iter	`0.99`
`Pointer_Object_Traverse`	`22.12962748707833` ns/iter	`26.2600591795629` ns/iter	`0.84`
`Pointer_Object_Try_Traverse`	`23.85511538139804` ns/iter	`29.356933214765007` ns/iter	`0.81`
`Pointer_Push_Back_Pointer_To_Weak_Pointer`	`137.4309858652673` ns/iter	`181.67225396924295` ns/iter	`0.76`
`Pointer_Walker_Schema_ISO_Language`	`1229448.095969342` ns/iter	`1667823.5684454984` ns/iter	`0.74`
`Pointer_Maybe_Tracked_Deeply_Nested/0`	`794007.288939089` ns/iter	`1056522.1100278578` ns/iter	`0.75`
`Pointer_Maybe_Tracked_Deeply_Nested/1`	`971946.7457864796` ns/iter	`1263295.6460674393` ns/iter	`0.77`
`Pointer_Position_Tracker_Get_Deeply_Nested`	`338.7246823753718` ns/iter	`393.9557066619676` ns/iter	`0.86`
`JSONPath_Descendant_Filter_Nested`	`1274.010968146494` ns/iter	`1351.6411792492922` ns/iter	`0.94`
`URITemplateRouter_Create`	`16490.95611535559` ns/iter	`21040.88795662437` ns/iter	`0.78`
`URITemplateRouter_Match`	`157.8810316331555` ns/iter	`203.50231275216805` ns/iter	`0.78`
`URITemplateRouter_Match_BasePath`	`184.7412769783113` ns/iter	`255.31151611031322` ns/iter	`0.72`
`URITemplateRouterView_Restore`	`9957.702196943977` ns/iter	`12634.377593680314` ns/iter	`0.79`
`URITemplateRouterView_Match`	`128.60816100112646` ns/iter	`175.84762738224433` ns/iter	`0.73`
`URITemplateRouterView_Match_BasePath`	`146.50238963819086` ns/iter	`180.4056747303957` ns/iter	`0.81`
`URITemplateRouterView_Arguments`	`510.3370028482002` ns/iter	`603.6321352724844` ns/iter	`0.85`
`JSONL_Parse_Large`	`6644683.057378061` ns/iter	`7956860.739725876` ns/iter	`0.84`
`JSONL_Parse_Large_GZIP`	`7531408.849999365` ns/iter	`11397846.09677375` ns/iter	`0.66`
`JSONLD_Catalog_Annotation_List_Populate`	`472169.74749367376` ns/iter	`679629.9790356472` ns/iter	`0.69`
`JSONLD_Catalog_Materialize`	`1939590.410958697` ns/iter	`3852089.8779071043` ns/iter	`0.50`
`HTML_Build_Table_100000`	`34810766.70000221` ns/iter	`58777173.58333712` ns/iter	`0.59`
`HTML_Render_Table_100000`	`1439670.0323231926` ns/iter	`2601867.926199118` ns/iter	`0.55`
`GZIP_Compress_ISO_Language_Set_3_Locations`	`24580331.857139625` ns/iter	`35353778.52631427` ns/iter	`0.70`
`GZIP_Decompress_ISO_Language_Set_3_Locations`	`3439549.10714217` ns/iter	`4541248.701863359` ns/iter	`0.76`
`GZIP_Compress_ISO_Language_Set_3_Schema`	`1410058.774390231` ns/iter	`1989378.8794518027` ns/iter	`0.71`
`GZIP_Decompress_ISO_Language_Set_3_Schema`	`254137.0152727144` ns/iter	`338864.87192891916` ns/iter	`0.75`
`JOSE_VerifySignature_RS256`	`20084.326065297606` ns/iter	`27026.993533488378` ns/iter	`0.74`
`JOSE_VerifySignature_ES512`	`1001813.0689206517` ns/iter	`1279781.1980033752` ns/iter	`0.78`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions

Benchmark (linux/gcc)

Details

Benchmark suite	Current: `6e58379`	Previous: `1bdc7aa`	Ratio
`JOSE_VerifySignature_RS256`	`21991.482355545537` ns/iter	`22058.95549555253` ns/iter	`1.00`
`JOSE_VerifySignature_ES512`	`575510.8259442186` ns/iter	`577008.6833881518` ns/iter	`1.00`
`GZIP_Compress_ISO_Language_Set_3_Locations`	`36380728.94736859` ns/iter	`36391862.3684245` ns/iter	`1.00`
`GZIP_Decompress_ISO_Language_Set_3_Locations`	`4173202.09523842` ns/iter	`4316511.312883193` ns/iter	`0.97`
`GZIP_Compress_ISO_Language_Set_3_Schema`	`2016501.5014407686` ns/iter	`2022194.011560898` ns/iter	`1.00`
`GZIP_Decompress_ISO_Language_Set_3_Schema`	`383085.1165207904` ns/iter	`382003.85464481474` ns/iter	`1.00`
`HTML_Build_Table_100000`	`56997660.909091376` ns/iter	`56193308.72727087` ns/iter	`1.01`
`HTML_Render_Table_100000`	`2063250.703703668` ns/iter	`1938205.9299720926` ns/iter	`1.06`
`JSONLD_Catalog_Annotation_List_Populate`	`674177.0502415327` ns/iter	`682481.3616383501` ns/iter	`0.99`
`JSONLD_Catalog_Materialize`	`3527663.8743716003` ns/iter	`3568597.1743591023` ns/iter	`0.99`
`JSONL_Parse_Large`	`9698183.62499887` ns/iter	`9855947.253520716` ns/iter	`0.98`
`JSONL_Parse_Large_GZIP`	`11075533.809524868` ns/iter	`11216482.645161545` ns/iter	`0.99`
`URITemplateRouter_Create`	`20848.78923044713` ns/iter	`20761.21577292574` ns/iter	`1.00`
`URITemplateRouter_Match`	`164.41062572580873` ns/iter	`165.0930180227342` ns/iter	`1.00`
`URITemplateRouter_Match_BasePath`	`185.52491538450468` ns/iter	`185.269450235789` ns/iter	`1.00`
`URITemplateRouterView_Restore`	`8339.406817196112` ns/iter	`8452.29097025726` ns/iter	`0.99`
`URITemplateRouterView_Match`	`127.8409233753247` ns/iter	`127.37641140898822` ns/iter	`1.00`
`URITemplateRouterView_Match_BasePath`	`145.251508955016` ns/iter	`145.1059821454249` ns/iter	`1.00`
`URITemplateRouterView_Arguments`	`467.9695891938853` ns/iter	`469.3950861459917` ns/iter	`1.00`
`JSONPath_Descendant_Filter_Nested`	`1642.4810103150155` ns/iter	`1617.6987546126659` ns/iter	`1.02`
`Pointer_Object_Traverse`	`23.109603317187137` ns/iter	`23.33777661920141` ns/iter	`0.99`
`Pointer_Object_Try_Traverse`	`23.259747364751576` ns/iter	`22.780609447719417` ns/iter	`1.02`
`Pointer_Push_Back_Pointer_To_Weak_Pointer`	`126.76090645451431` ns/iter	`130.64119010253356` ns/iter	`0.97`
`Pointer_Walker_Schema_ISO_Language`	`1407954.2572614958` ns/iter	`1408089.5682278876` ns/iter	`1.00`
`Pointer_Maybe_Tracked_Deeply_Nested/0`	`1191930.0645160906` ns/iter	`1202000.8688243711` ns/iter	`0.99`
`Pointer_Maybe_Tracked_Deeply_Nested/1`	`1721255.138349558` ns/iter	`1722890.5724816637` ns/iter	`1.00`
`Pointer_Position_Tracker_Get_Deeply_Nested`	`681.3300497210713` ns/iter	`633.250507966585` ns/iter	`1.08`
`JSON_Array_Of_Objects_Unique`	`427.8486255003114` ns/iter	`406.59681675241075` ns/iter	`1.05`
`JSON_Parse_1`	`5267.862604290796` ns/iter	`5265.589720578247` ns/iter	`1.00`
`JSON_Parse_Real`	`5649.816990882219` ns/iter	`5730.45702441552` ns/iter	`0.99`
`JSON_Parse_Decimal`	`10232.319765748061` ns/iter	`10367.652552597181` ns/iter	`0.99`
`JSON_Parse_Schema_ISO_Language`	`3605896.2694303286` ns/iter	`3602267.6145837097` ns/iter	`1.00`
`JSON_Parse_Integer`	`3727.230250917755` ns/iter	`3711.515046689525` ns/iter	`1.00`
`JSON_Parse_String_NonSSO_Plain`	`3287.4738771599223` ns/iter	`3269.189989203269` ns/iter	`1.01`
`JSON_Parse_String_SSO_Plain`	`2967.100650230142` ns/iter	`2988.0719873868943` ns/iter	`0.99`
`JSON_Parse_String_Escape_Heavy`	`16688.470487516` ns/iter	`16739.68806397875` ns/iter	`1.00`
`JSON_Parse_Object_Short_Keys`	`8444.809475637554` ns/iter	`8403.073402749314` ns/iter	`1.00`
`JSON_Parse_Object_Scalar_Properties`	`4308.722715243816` ns/iter	`4330.347777312024` ns/iter	`1.00`
`JSON_Parse_Object_Array_Properties`	`6533.207359323182` ns/iter	`6411.294816692167` ns/iter	`1.02`
`JSON_Parse_Object_Object_Properties`	`6584.954015465263` ns/iter	`6599.227945128157` ns/iter	`1.00`
`JSON_Parse_Nested_Containers`	`39241.25593078628` ns/iter	`39014.60878021781` ns/iter	`1.01`
`JSON_From_String_Copy`	`9.072930985091718` ns/iter	`8.79123115324379` ns/iter	`1.03`
`JSON_From_String_Temporary`	`8.02169958559615` ns/iter	`7.373179167957112` ns/iter	`1.09`
`JSON_Number_To_Double`	`20.88702177492899` ns/iter	`20.91408589461598` ns/iter	`1.00`
`JSON_Object_At_Last_Key/8`	`3.743858669800377` ns/iter	`3.7493106022758016` ns/iter	`1.00`
`JSON_Object_At_Last_Key/32`	`11.701388727262373` ns/iter	`11.886294644574793` ns/iter	`0.98`
`JSON_Object_At_Last_Key/128`	`48.6371878252631` ns/iter	`48.9775749913955` ns/iter	`0.99`
`JSON_Object_At_Last_Key/512`	`350.3085613756088` ns/iter	`357.9287303185175` ns/iter	`0.98`
`JSON_Fast_Hash_Helm_Chart_Lock`	`61.53925568487925` ns/iter	`61.194051332618365` ns/iter	`1.01`
`JSON_Equality_Helm_Chart_Lock`	`155.8132201214718` ns/iter	`160.26486138454266` ns/iter	`0.97`
`JSON_Divisible_By_Decimal`	`265.6290394742513` ns/iter	`265.3939302999428` ns/iter	`1.00`
`JSON_String_Equal/10`	`5.380480283503823` ns/iter	`5.3877211901685405` ns/iter	`1.00`
`JSON_String_Equal/100`	`6.001218530580423` ns/iter	`6.002727356078012` ns/iter	`1.00`
`JSON_String_Equal_Small_By_Perfect_Hash/10`	`0.6238438066908748` ns/iter	`0.6237671048640969` ns/iter	`1.00`
`JSON_String_Equal_Small_By_Runtime_Perfect_Hash/10`	`25.19986474113609` ns/iter	`25.216785083160087` ns/iter	`1.00`
`JSON_String_Fast_Hash/10`	`2.2574062828356545` ns/iter	`2.2609205216656423` ns/iter	`1.00`
`JSON_String_Fast_Hash/100`	`2.2581612172709167` ns/iter	`2.2614451168591447` ns/iter	`1.00`
`JSON_String_Key_Hash/10`	`1.2470686037026069` ns/iter	`1.248315604271231` ns/iter	`1.00`
`JSON_String_Key_Hash/100`	`12.447164175378244` ns/iter	`12.453708550136708` ns/iter	`1.00`
`JSON_Object_Defines_Miss_Same_Length`	`3.1165384392739885` ns/iter	`3.115958971128015` ns/iter	`1.00`
`JSON_Object_Defines_Miss_Too_Small`	`3.148499643541648` ns/iter	`3.1170238481014843` ns/iter	`1.01`
`JSON_Object_Defines_Miss_Too_Large`	`3.113238649628603` ns/iter	`3.1158208789449833` ns/iter	`1.00`
`Regex_Lower_S_Or_Upper_S_Asterisk`	`0.9350114017096591` ns/iter	`0.9366907691097417` ns/iter	`1.00`
`Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar`	`0.6227649151695557` ns/iter	`0.6237328177431622` ns/iter	`1.00`
`Regex_Period_Asterisk`	`0.9345720272487579` ns/iter	`0.9345286972545574` ns/iter	`1.00`
`Regex_Group_Period_Asterisk_Group`	`0.6231035105689782` ns/iter	`0.6236411458853984` ns/iter	`1.00`
`Regex_Period_Plus`	`0.9346019504133976` ns/iter	`0.935948700793869` ns/iter	`1.00`
`Regex_Period`	`0.6230767955940338` ns/iter	`0.623093824817489` ns/iter	`1.00`
`Regex_Caret_Period_Plus_Dollar`	`0.934795434611322` ns/iter	`0.9357971660666243` ns/iter	`1.00`
`Regex_Caret_Group_Period_Plus_Group_Dollar`	`0.6229211423222865` ns/iter	`0.6235622339746187` ns/iter	`1.00`
`Regex_Caret_Period_Asterisk_Dollar`	`0.9345917337710401` ns/iter	`0.9355061813529556` ns/iter	`1.00`
`Regex_Caret_Group_Period_Asterisk_Group_Dollar`	`0.6238312744996451` ns/iter	`0.6244228837158288` ns/iter	`1.00`
`Regex_Caret_X_Hyphen`	`4.4039134412132315` ns/iter	`4.372226845585045` ns/iter	`1.01`
`Regex_Period_Md_Dollar`	`28.900871800467783` ns/iter	`28.94017145393133` ns/iter	`1.00`
`Regex_Caret_Slash_Period_Asterisk`	`4.670291695011026` ns/iter	`4.673940617198296` ns/iter	`1.00`
`Regex_Caret_Period_Range_Dollar`	`1.245846700116018` ns/iter	`1.246270494368111` ns/iter	`1.00`
`Regex_Nested_Backtrack`	`37.814678218372684` ns/iter	`37.75618600373121` ns/iter	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

augmentcode · 2026-08-17T01:10:48Z

🤖 Augment PR Summary

Summary: Adds clitest, a TAP-producing interpreter for CLI test scripts.

Changes:

Introduces a DSL for running binaries in isolated temporary sandboxes.
Adds commands for fixtures, comparisons, filtering, filesystem operations, compression, checksums, and metadata.
Captures process output in a stable, stream-marked observation format.
Implements variable expansion, quoting, regex support, and script diagnostics.
Adds a static vacuity check for unasserted observations and unused generated artifacts.
Registers the utility and a broad CTest fixture suite, including a stub executable.

Technical notes: Process environments are explicitly assembled and scripts can be checked without execution using --check.

_{🤖 Was this summary useful? React with 👍 or 👎}

augmentcode

Review completed. 5 suggestions posted.

Comment augment review to trigger a new review at any time.

augmentcode · 2026-08-17T01:10:49Z

+      }
+
+      for (std::size_t index{0}; index + 8 < operands.size(); index += 1) {
+        clitest_collect(consumed, operands[index], bindings);


src/lang/test/clitest/clitest.h:695 — Treating every program argument as a consumed file lets a RUN bypass the vacuity check by passing its own INTO destination as an argument. For example, RUN result.txt ... INTO result.txt EXPECTING 0 is accepted without a comparison even though the child runs before the observation overwrites result.txt, so this run's output is never asserted.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

augmentcode · 2026-08-17T01:10:49Z

+  }
+
+  const auto expected_code{
+      sourcemeta::core::to_int64_t(command.at(trailer + 7))};


src/lang/test/clitest/clitest.h:1051 — The expected status is parsed without command.expand, despite the documented rule that operands expand. Consequently, a valid script such as ENV STATUS 0 followed by RUN ... EXPECTING $STATUS fails with not an exit code instead of accepting the configured expected code.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

augmentcode · 2026-08-17T01:10:49Z

+  for (const auto name : {"SYSTEMROOT", "COMSPEC", "TEMP"}) {
+    const char *const value{std::getenv(name)};
+    if (value != nullptr) {
+      essentials.emplace_back(value);


src/lang/test/clitest/clitest.h:1081 — child stores string_views into essentials, but a later emplace_back can reallocate that vector and dangle values already inserted for SYSTEMROOT or COMSPEC. On Windows, where multiple of these variables are normally present, spawn_and_capture can therefore receive corrupted environment values or access invalid memory.

Severity: high

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

augmentcode · 2026-08-17T01:10:49Z

+
+  // Silent when the target was never there to begin with
+  std::error_code error;
+  std::filesystem::remove_all(command.resolve(0), error);


src/lang/test/clitest/clitest.h:1329 — remove_all reports failures through error, but the result is discarded, so a REMOVE that cannot delete its target is reported as successful and the script can continue against stale files. The comment only permits a missing target to be silent, not permission or other filesystem failures.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

augmentcode · 2026-08-17T01:10:49Z

+    return false;
+  }
+
+  const auto problems{sourcemeta::core::clitest_check(statements)};


src/lang/test/clitest/main.cc:109 — The static checker is invoked without the command-line bindings, even though those bindings are already passed to run. Thus clitest --check -e OUT=result.txt accepts an unasserted RUN ... INTO $OUT ...; the known destination is treated as runtime-unknown and no vacuity problem is produced.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

cubic-dev-ai

1 issue found across 152 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/lang/test/clitest/clitest.h">

<violation number="1" location="src/lang/test/clitest/clitest.h:1224">
P2: SORT always writes a trailing newline because `clitest_join(lines.values)` uses the default `terminated=true`, while the sibling in-place filter DROP preserves the source file's flag via `clitest_join(lines.values, lines.terminated)`. Sorting a file that does not end with a newline therefore appends one, changing its byte content so a later `COMPARE` against an expectation produced the same way as DROP can diverge; the two filters should behave identically. Pass `lines.terminated` here as DROP does.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai · 2026-08-17T01:20:17Z

+  const auto content{command.read(0)};
+  auto lines{clitest_lines(content)};
+  std::sort(lines.values.begin(), lines.values.end());
+  command.write(0, clitest_join(lines.values));


P2: SORT always writes a trailing newline because clitest_join(lines.values) uses the default terminated=true, while the sibling in-place filter DROP preserves the source file's flag via clitest_join(lines.values, lines.terminated). Sorting a file that does not end with a newline therefore appends one, changing its byte content so a later COMPARE against an expectation produced the same way as DROP can diverge; the two filters should behave identically. Pass lines.terminated here as DROP does.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lang/test/clitest/clitest.h, line 1224: <comment>SORT always writes a trailing newline because `clitest_join(lines.values)` uses the default `terminated=true`, while the sibling in-place filter DROP preserves the source file's flag via `clitest_join(lines.values, lines.terminated)`. Sorting a file that does not end with a newline therefore appends one, changing its byte content so a later `COMPARE` against an expectation produced the same way as DROP can diverge; the two filters should behave identically. Pass `lines.terminated` here as DROP does.</comment> <file context> @@ -0,0 +1,1386 @@ + const auto content{command.read(0)}; + auto lines{clitest_lines(content)}; + std::sort(lines.values.begin(), lines.values.end()); + command.write(0, clitest_join(lines.values)); +} + </file context>

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

cubic-dev-ai

2 issues found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="test/test/clitest/run_carriage_return_script.clitest">

<violation number="1" location="test/test/clitest/run_carriage_return_script.clitest:1">
P2: This new fixture is committed with CRLF line endings so it exercises the runner's "\r\n"→"\n" normalisation, but unlike the other byte-sensitive fixtures it is not marked in .gitattributes. The repo already protects parser-input fixtures with `-text` and the comment there states their line endings "must reach the working directory exactly as committed on every platform". On a checkout where `core.autocrlf` is active (common on Windows) git can rewrite this file to LF, so the test silently stops testing CRLF handling even though it still passes. Add `test/test/clitest/run_carriage_return_script.clitest -text` to .gitattributes to keep the test meaningful.

(Based on your team's feedback about preserving fixture bytes.)</violation>
</file>

<file name="src/lang/test/clitest/main.cc">

<violation number="1" location="src/lang/test/clitest/main.cc:88">
P3: Normalizing `\r\n` over the whole script also rewrites WRITE bodies, so a body can no longer hold a literal carriage return before a line feed. Since COMPARE compares files byte-for-byte, a fixture or a program that legitimately produces CRLF content can no longer be matched by a WRITE-created expected file, and there is no escape hatch. The behavior is documented as intentional, but the loss of the previous "verbatim body" guarantee for CRLF bytes is worth confirming this is acceptable.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

cubic-dev-ai · 2026-08-17T13:16:25Z

@@ -0,0 +1,8 @@
+WRITE payload.txt UNTIL END


P2: This new fixture is committed with CRLF line endings so it exercises the runner's "\r\n"→"\n" normalisation, but unlike the other byte-sensitive fixtures it is not marked in .gitattributes. The repo already protects parser-input fixtures with -text and the comment there states their line endings "must reach the working directory exactly as committed on every platform". On a checkout where core.autocrlf is active (common on Windows) git can rewrite this file to LF, so the test silently stops testing CRLF handling even though it still passes. Add test/test/clitest/run_carriage_return_script.clitest -text to .gitattributes to keep the test meaningful.

(Based on your team's feedback about preserving fixture bytes.)

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At test/test/clitest/run_carriage_return_script.clitest, line 1: <comment>This new fixture is committed with CRLF line endings so it exercises the runner's "\r\n"→"\n" normalisation, but unlike the other byte-sensitive fixtures it is not marked in .gitattributes. The repo already protects parser-input fixtures with `-text` and the comment there states their line endings "must reach the working directory exactly as committed on every platform". On a checkout where `core.autocrlf` is active (common on Windows) git can rewrite this file to LF, so the test silently stops testing CRLF handling even though it still passes. Add `test/test/clitest/run_carriage_return_script.clitest -text` to .gitattributes to keep the test meaningful. (Based on your team's feedback about preserving fixture bytes.)</comment> <file context> @@ -0,0 +1,8 @@ +WRITE payload.txt UNTIL END +alpha +END </file context>

cubic-dev-ai · 2026-08-17T13:16:25Z

+    // asking every consumer to configure their checkout. Beyond that a line
+    // ends at a line feed and nothing else, which is what keeps a script
+    // meaning the same thing everywhere
+    contents = sourcemeta::core::replace(


P3: Normalizing \r\n over the whole script also rewrites WRITE bodies, so a body can no longer hold a literal carriage return before a line feed. Since COMPARE compares files byte-for-byte, a fixture or a program that legitimately produces CRLF content can no longer be matched by a WRITE-created expected file, and there is no escape hatch. The behavior is documented as intentional, but the loss of the previous "verbatim body" guarantee for CRLF bytes is worth confirming this is acceptable.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/lang/test/clitest/main.cc, line 88: <comment>Normalizing `\r\n` over the whole script also rewrites WRITE bodies, so a body can no longer hold a literal carriage return before a line feed. Since COMPARE compares files byte-for-byte, a fixture or a program that legitimately produces CRLF content can no longer be matched by a WRITE-created expected file, and there is no escape hatch. The behavior is documented as intentional, but the loss of the previous "verbatim body" guarantee for CRLF bytes is worth confirming this is acceptable.</comment> <file context> @@ -79,12 +79,14 @@ static auto run(const std::string &script, const bool check, + // asking every consumer to configure their checkout. Beyond that a line + // ends at a line feed and nothing else, which is what keeps a script + // meaning the same thing everywhere + contents = sourcemeta::core::replace( + sourcemeta::core::read_to_string(stream), "\r\n", "\n"); } catch (const sourcemeta::core::IOFileNotFoundError &) { </file context>

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

github-actions Bot reviewed Aug 16, 2026

View reviewed changes

jviotti force-pushed the clitest branch from ab7d4e1 to 5b150a9 Compare August 16, 2026 19:41

github-actions Bot reviewed Aug 16, 2026

View reviewed changes

jviotti force-pushed the clitest branch 6 times, most recently from 81996fc to d5d9103 Compare August 16, 2026 21:23

[WIP] Implement a clitest utility binary for testing CLIs

a3e5afa

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

jviotti force-pushed the clitest branch from d5d9103 to a3e5afa Compare August 17, 2026 00:00

jviotti changed the title ~~[WIP] Implement a clitest utility binary for testing CLIs~~ Implement a clitest utility binary for testing CLIs Aug 17, 2026

jviotti marked this pull request as ready for review August 17, 2026 01:07

augmentcode Bot reviewed Aug 17, 2026

View reviewed changes

Fixes

1d3d6a6

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

cubic-dev-ai Bot reviewed Aug 17, 2026

View reviewed changes

jviotti added 2 commits August 16, 2026 22:30

Fix

6e58379

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

Fix Windows

32b81c5

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

cubic-dev-ai Bot reviewed Aug 17, 2026

View reviewed changes

jviotti added 2 commits August 17, 2026 10:28

More

615dbef

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

Fix

a048172

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

Uh oh!

Uh oh!

Conversation

jviotti commented Aug 16, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Benchmark (linux/llvm)

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Benchmark (macos/llvm)

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Benchmark (linux/gcc)

Uh oh!

augmentcode Bot commented Aug 17, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jviotti commented Aug 16, 2026 •

edited by cubic-dev-ai Bot

Loading

github-actions Bot left a comment •

edited

Loading

github-actions Bot left a comment •

edited

Loading

github-actions Bot left a comment •

edited

Loading

augmentcode Bot Aug 17, 2026 •

edited

Loading

augmentcode Bot Aug 17, 2026 •

edited

Loading

augmentcode Bot Aug 17, 2026 •

edited

Loading

augmentcode Bot Aug 17, 2026 •

edited

Loading

augmentcode Bot Aug 17, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading