Cinematographer and AI creative technologist in Abu Dhabi. I work where film craft meets applied AI, and I publish open evaluation of the Arabic tooling that production depends on.

Each measures a failure that looks correct to anyone who does not read Arabic, which is exactly why it ships. All CC BY 4.0, with the test set, the scorer, the raw per-item results, and a plain statement of what the measurement does not establish.

Datasets: huggingface.co/syamjithnk

The numeral failure above is now a package anyone can install:

pip install arabic-tts-frontend

arabic-tts-frontend · PyPI — numerals, dates, currency, times and percentages converted to spoken Arabic before the text reaches a TTS engine. MIT, zero dependencies. Re-measured against the released build on the same 45 sentences, same engine and scorer: Arabic-Indic digits 0/15 → 11/15. The scorer ships with it, so the figures can be re-derived rather than trusted, and the release notes state the run-to-run noise (±3) instead of quoting the overall number precisely.

The Arabic fix everyone recommends is now the bug — the get_display(reshape(...)) recipe is in 1,180 indexed files on GitHub and now renders Arabic backwards on matplotlib 3.11, silently. Why the shaping half is detectable and the reordering half is not, and what happened when I filed it upstream.

More at syamjith-nk.github.io.

Filing the finding where the bug lives has reached far more people than publishing it ever did.

matplotlib — matplotlib 3.11 shapes Arabic itself, so the reshape+bidi recipe now runs twice and renders the label reversed. Nothing raises; it looks like Arabic to anyone who cannot read it. Measured as mean absolute pixel difference against a reference render: 7.75 for the raw logical string (antialiasing only), 76.28 pre-shaped. Three maintainers replied within two days, including the project lead — and one of them went and edited the StackOverflow answers to say the workaround is only for older versions. That is the widest-reaching outcome of any of this and it cost nothing but filing accurately. Docs PR #32263 is open.

The project lead proposed a fix I had not thought of — wrapping pre-processed text in Unicode LRO/PDF — and said he was not confident it was right for Arabic. I could check, so I did: across three fonts and nine strings it reads correctly in all of them, but does not always render identically, because the wrapped string draws the font's static presentation forms instead of the font's own shaping. That exchange is the argument for filing upstream rather than blogging first. He had the better fix; I had the only way to verify it.

Pillow — the same class of failure, different mechanism: Pillow's behaviour depends on whether it was built with Raqm, so the identical script produces correct text on one machine and broken text on another. On a Raqm-enabled build, 0 of 15 strings survive the recipe unchanged. The maintainer pushed back on my first draft for overclaiming a security angle. He was right, I dropped it, and the warning that remains is narrower and true.

python-arabic-reshaper — issue #102 and PR #103, adding the PIL.features.check("raqm") test to the README so readers can tell which case they are in. Still unanswered.

arabic-lint — pip install arabic-lint. Finds Arabic that was corrupted before it was stored, in JSON, localisation files, database exports and source. Zero dependencies, exit code 1 on a finding, so it drops into CI unchanged.

Detection is asymmetric, which is why this survived years of being copied: the shaping half leaves presentation-form codepoints that correctly authored Arabic never contains, so it is detectable — but the reordering half produces identical codepoints in a different order, so no general detector exists for it. And the damage is not cleanly reversible: lam-alef is one codepoint that decomposes to two, in logical order, while the text around it is in visual order. الإمارات goes 8 codepoints to 7, and the obvious NFKC repair turns السلام into السالم — a real but different word. The tool refuses to auto-fix those rather than guess.

In two of the three benchmarks the measurement contradicted the assumption I started with, and the write-ups say so. In arshape a pass/fail scorer reported a broken configuration as working; adding a middle band is what surfaced the real result. Designing a test that can prove you wrong is most of the work.

When I found four faults in my own scorer I published a documented correction with old-to-new figures rather than editing the numbers quietly.

Brand and product films, VFX compositing, and AI-assisted image pipelines where the craft still governs the tool — a generated frame is judged on light, framing and rhythm like a photographed one.

syamjithnk.com · Abu Dhabi, GMT+4