feat(pg-functions): implement PostgreSQL string functions - #407
Merged
sunng87 merged 5 commits intoSep 14, 2026
Merged
Conversation
Implement 14 string functions from the PostgreSQL built-in catalog as
DataFusion ScalarUDFs, organized into six modules under src/string/:
- convert: to_bin(int), to_oct(int) — integer-to-text base conversion
- quote: quote_literal(text), quote_nullable(text) — SQL quoting
- unicode: normalize(text[,form]), casefold(text), unicode_assigned(text),
unistr(text) — Unicode normalization and escape decoding
- format: format(fmt,...), sprintf(fmt,...) — PG %s/%I/%L text formatting
- regexp: regexp_substr(text,pattern,...), regexp_split_to_array(text,pat)
— regex extraction and splitting
- encoding: pg_client_encoding(), to_ascii(text) — encoding utilities
Each UDF follows the conventions from functions.md: ScalarUDFImpl with
PartialEq/Eq/Hash derives, NULL propagation, and unit tests covering
boundary cases. A string.slt integration test file exercises all functions
through the SQL → plan → execute path.
The 'string' Cargo feature now pulls in 'regex' and 'unicode-normalization'
as optional dependencies. functions.md is updated to mark 14 entries as 🔧.
Address every finding from the two-axis review of the string functions.
Spec (Postgres semantics):
- regexp_substr: 'start' is a 1-based CHARACTER position; resolve it via
char_indices so a start landing inside a multibyte UTF-8 char no longer
panics (was a crash on valid input).
- regexp_split_to_array: was implemented but never registered; now wired into
the string register() and given a working array/column path.
- to_bin/to_oct: negatives now render two's-complement (matching the to_hex
family) instead of sign-magnitude ('-1101').
- casefold: full Unicode case folding (CaseFolding.txt C+F), so casefold('ß')
-> 'ss' and long-s 'ſ' -> 's' (previously Rust to_lowercase, which kept 'ß').
- unicode_assigned: use ICU4X general-category tables via icu_properties;
Private-Use-Area chars (category Co) now correctly return true, and reserved
codepoints (Cn) correctly return false (was inverted on PUA, blind to Cn).
- quote_literal/quote_nullable: stop doubling backslashes to match
standard_conforming_strings = on (backslash is an ordinary char).
- to_ascii: transliterate Latin accented chars to their ASCII base (cafe,
Munchen) instead of replacing with '?'.
- format/sprintf: implement the exact Postgres grammar (%[position]s|I|L and
%%); reject width/flag specifiers, which the spec does not support.
Standards:
- Rename all UDF structs to the documented *UDF suffix (was *Udf).
- Add the previously-unregistered regexp_split_to_array to register().
- Document Postgres compatibility + link the manual at the top of each file.
- Add row-wise vectorized-batch unit tests for each function (convention datafusion-contrib#4).
- Replace 'Arc::new(...finish()) as _' with explicit 'as ArrayRef'.
- Clear all clippy warnings (borrowed-expression, map_or, match->?).
Add icu_properties behind the 'string' feature for the general-category
lookup. All 42 unit tests, the sqllogictest suite, and clippy pass.
Address the follow-up review findings:
- regexp: the 'c' flag had been mapped to (?i), making case-SENSITIVE
matching case-insensitive — the opposite of Postgres, where 'c' is
case-sensitive. Flags are now applied in order with last-one-wins
semantics ('i' enables, 'c' disables), matching PG ('ic' → sensitive,
'ci' → insensitive). Regression test covers all four combinations.
- casefold: replace the hand-enumerated fold set (which missed final
sigma ς→σ, micro µ→μ, capital sharp-s ẞ→ss, Greek symbols, Cyrillic
historical letters) with the ICU4X CaseMapper via icu_casemap — the
same tables Postgres uses. Tests added for ς, µ, ẞ.
- regexp_split_to_array: NULL input now returns a NULL of the declared
List type instead of untyped ScalarValue::Null.
- format: correct the %I documentation — reserved words are not checked
(documented deviation rather than an inaccurate claim).
- functions.md: flip the 12 string rows left as 🚧 by an earlier atomic
edit failure (casefold, format, normalize, pg_client_encoding,
quote_literal, quote_nullable, sprintf, to_ascii, to_bin, to_oct,
unicode_assigned, unistr) to 🔧, and fix the Pattern Matching summary
(2 implemented, 2 P2 remaining). The String summary (16 🔧 / 3 P2) and
TOTAL now match the section contents.
43 unit tests, the sqllogictest suite, clippy, and fmt all pass.
… behavior
Spin up a real PostgreSQL 18.4 and verify every string UDF against it;
correct four functions whose behavior differed from upstream (three of
which were over-corrected in an earlier review pass).
- to_ascii: replace the hand-written Latin-1/Ext-A table with Postgres'
actual LATIN1 -> ASCII table, extracted verbatim from the live server
(all 128 entries for 0x80-0xFF). Postgres' mapping is idiosyncratic -
ss -> 'B' (not 'ss'), AE -> 'A' (not 'AE'), (c) -> 'C', (pounds) -> 'L',
x -> 'x' - because it maps every byte to a single ASCII char and
replaces unmapped chars with a space. General libraries (any_ascii,
deunicode) produce different mappings (e.g. ss -> ss) and cannot be
substituted; the old hand table also mapped lowercase Ext-A letters to
uppercase ('a' with macron -> 'A') and dropped symbols Postgres maps.
- casefold: Postgres' casefold is SIMPLE per-character lowercase
(verified: casefold('ss')='ss' not 'ss', 'I-dotted' -> 'i', final
sigma not context-folded), so switch from CaseMapper::fold_string
(full folding) to CaseMapper::simple_lowercase, which matches exactly.
- quote_literal/quote_nullable: Postgres emits the E'...' escape-string
form with quotes AND backslashes doubled when the value contains a
backslash, and plain '...' otherwise (verified with
standard_conforming_strings=on). Restore backslash handling with the
E-prefix.
- format/sprintf: PG 18 DOES support width specifiers (verified
format('%10s','x') right-aligns; '%-10s' left-aligns; '%*s' takes the
width from the argument list; a leading 0 is part of the width). Restore
width/flag/'*' support that an earlier pass had removed as 'scope
creep'. sprintf is not in PG 18.4 - catalog note corrected to PG 19+.
- unistr: add Postgres' bare \XXXX escape form, require exactly 6 hex
digits after \+, and error on unrecognized escapes (\n, \\)
instead of passing them through - all verified against the live server.
All expectations in unit tests and string.slt updated to the verified
values. 44 unit tests, sqllogictest, clippy, and fmt pass. The any_ascii
evaluation dependency is removed (it does not match Postgres).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the String Functions category (the catalog's 2nd part after the math functions) as DataFusion
ScalarUDFs, covering 14 PostgreSQL built-ins listed as 🚧 P2 in `datafusion-pg-functions/functions.md`.Organized into six modules under `src/string/`:
The `string` Cargo feature now pulls in `regex`, `unicode-normalization`, and `icu_properties` (optional). `functions.md` is updated to flip these rows to 🔧.
Postgres compatibility notes
Each UDF matches documented Postgres semantics:
Testing
Follows the conventions in `functions.md`: `*UDF` structs, `ScalarUDFImpl` impls, PG manual links + compatibility docs at each file header, constructors wired into `register()`.