Fix jdbc-v2: lex heredoc strings as a single literal in the ANTLR4 grammar - #3032
Open
polyglotAI-bot wants to merge 1 commit into
Open
Fix jdbc-v2: lex heredoc strings as a single literal in the ANTLR4 grammar#3032polyglotAI-bot wants to merge 1 commit into
polyglotAI-bot wants to merge 1 commit into
Conversation
…ammar The ANTLR4 lexer had no '$' handling, so every '$' was dropped as an unrecognized character and a heredoc body was lexed as ordinary SQL: a body that still looked like valid SQL was silently mis-parsed, and a body containing ';' (or the empty heredoc $$$$) was reported as a parse error, which classifies an INSERT as a result-set-bearing statement with no values-list positions. Add a HEREDOC_LITERAL token (the untagged form ends at the first '$$', like the server does it) and accept it as a literal. '$' also becomes an identifier character so that an unterminated tag stays an identifier, which is how the server reads it too. Fixes: #3031
|
Client V2 CoverageCoverage Report
Class Coverage
|
JDBC V2 CoverageCoverage Report
Class Coverage
|
JDBC V1 CoverageCoverage Report
Class Coverage
|
Client V1 CoverageCoverage Report
Class Coverage
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3031.
The ANTLR4 lexer (
jdbc_sql_parser=ANTLR4andANTLR4_PARAMS_PARSER) had no$handling at all — no heredoctoken, and
$is not a token or an identifier character — so every$raised a token-recognition error and wassilently dropped, and the heredoc body was then lexed as ordinary SQL.
INSERT INTO t VALUES ($$abc$$, 1)reached the parser as
... VALUES ( abc , 1 ),$$a;b$$as( a ; b , 1 )and the empty$$$$as( , 1 ).For a body that happened to look like a valid SQL fragment the mis-lex was silent (statement classification,
table name and the VALUES-list positions were all derived from the wrong token stream); for a body that did not,
the statement came back with
hasErrors=true, which makesjdbc-v2classify the INSERT as a result-set-bearingstatement with no values-list positions (so the batch values template and the table-name based paths are
disabled).
The fix adds a
HEREDOC_LITERALtoken and accepts it as aliteral, the same seam the sibling JavaCC fix(#3030) uses —
literalfeeds bothassignmentValue(INSERT VALUES) andcolumnExpr(SELECT), the two entrypoints that reach this behavior.
Server behaviour, used to derive the expected values (verified against 26.5):
Changes
ClickHouseLexer.g4: newHEREDOC_LITERALtoken, placed beforeIDENTIFIER(a heredoc made of tagcharacters only, e.g.
$$abc$$, is also a valid identifier of the same length, and equally long matches areresolved by rule order). The untagged alternative is
$$ .*? $$, which ends at the first$$exactly likethe server, so a single
$inside the body is kept. The tagged alternative uses aHEREDOC_TAGfragment(
$+ tag characters +$).ClickHouseLexer.g4:IDENTIFIERnow accepts$, matching the server (and the JavaCC grammar, whichalready had it). This is also what keeps an unterminated tag harmless: ANTLR's lexer error recovery does not
rewind after a long failed match, so without an accepting rule at that position a stray
$tag$would consumethe rest of the statement.
ClickHouseParser.g4:literalacceptsHEREDOC_LITERAL.CHANGELOG.md: bug-fix entry.Known limitation, unchanged from the JavaCC grammar: the opening and closing tags of the tagged form are not
required to be equal (matching tags cannot be expressed by a plain lexer rule), so a statement that contains an
unterminated tag and a later
$word$can still be mis-lexed. Such a statement is rejected by the serveranyway.
Not in this PR: the hand-written placeholder scanner (
SqlParserFacade.parseParameters) is also heredoc-blind,so a
?inside a heredoc body is counted as a parameter. That is a different component, it affects the JavaCCbackend as well, and it is already tracked by #3009 / #3010.
Test
BaseSqlParserFacadeTest.testHeredocStatementsAntlr4Only— one TestNG@DataProviderrunning for both ANTLR4backends (guarded off for JavaCC, whose heredoc token is #3029/#3030), asserting
hasErrors, the INSERT /result-set classification, the table name, the parameter count and the exact VALUES-list substring for: a body
with
;, an empty body, a body with a single$, the tagged form, a body with parentheses and commas, twoheredocs in one values list, a heredoc next to a
?placeholder, a multi-line body, several value groups, and aheredoc as a
SELECTcolumn expression. Contrast rows pin the behaviour that must NOT change: an unterminatedtag and a
$inside an identifier stay identifiers ($foo$bar,a$b,t$$a$$), and quoted string literalskeep their existing handling, including the multi-group values-list positions.
Without the grammar change 12 of these rows fail (6 inputs × 2 backends) with
Query should parse without errors; with it the fulljdbc-v2unit suite is green:mvn -pl jdbc-v2 -DskipITs=true test→ 1343 tests, 0failures.
Pre-PR validation gate
hasErrors=truefor$$a;b$$and$$$$onmain, both ANTLR4 backends)$dropped by lexer error recovery → body lexed as SQL)literalrule), not the symptomjdbc-v2unit suite greenAGENTS.md(targeted Maven run,@DataProviderinstead of near-identicalmethods, no issue numbers inside test code, CHANGELOG updated). No public API change, so no
docs/features.mdupdate.