[RNE Rewrite] feat(models): point LLM constants at the re-published artifacts - #1385
Open
msluszniak wants to merge 1 commit into
Open
[RNE Rewrite] feat(models): point LLM constants at the re-published artifacts#1385msluszniak wants to merge 1 commit into
msluszniak wants to merge 1 commit into
Conversation
msluszniak
changed the title
msluszniak
marked this pull request as draft
August 26, 2026 12:23
msluszniak
added
model
msluszniak
changed the title
smollm2 now publishes a single `8da8w` file per size instead of a bf16 and an `8da4w` one. The removed variants are not worth a download: - The published `8da4w` files at 135m/360m were barely quantized: an unset group_size fell back to 128, and dims 576/960 are not divisible by it, so they shipped larger than bf16 and produced nonsense on device. - bf16 is dominated on every axis. At 360m int8 is 0.412 GB / 106 tok/s / 649 MiB against bf16's 0.725 GB / 11 tok/s / 935 MiB, with matching answers. At 1_7b bf16 decodes at 1.5-1.9 tok/s, which is not usable. - At 1_7b the published `8da4w` loses to the 360m int8 build on 5 of 10 prompts (degenerate repetition, and list-vs-tuple semantics stated backwards) while being 2.5x the download, so 4-bit is dropped there too. The 360m and 1.7B files were verified end to end in apps/nlp against locally served copies of exactly what is being published. Requires the matching re-export, published under the same version tag.
msluszniak
force-pushed
the
@ms/smollm2-int8-variants
branch
from
August 26, 2026 13:43
aeec673 to
a8470e1
Compare
msluszniak
marked this pull request as ready for review
August 26, 2026 17:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Points the LLM model constants at the re-published artifacts from export-scripts MR !16, which reconciles every LLM config against what is actually on the Hub and fixes the recipes that were wrong. All four families are re-exported and published.
8da8wfile per size; bf16 and8da4ware dropped because int8 is smaller, faster and answers better at every size.get_eos_ids [151645, 151643]; the bf16 builds currently emit<|im_end|>as visible text.8da4wstops on<|im_end|>like every other qwen2.5 file, rather than<|endoftext|>.8da4wfiles predate.Every LLM config also gains the corrected
get_n_layers(it wasnullin all 15) and the real 2047 sequence dim, but those are config-side and need no constant change here.Introduces a breaking change?
Type of change
Tested on
Testing instructions
nlpdemo app and generate with every model touched here (SmolLM2 135m/360m/1_7b, Hammer2.1 0_5b/1_5b/3b, Qwen2.5 3b, LFM2.5-text 350m/1_2b); check each loads, streams and stops cleanlyRelated issues
Export side: export-scripts MR !16.
Checklist
Additional notes
Do not merge until every family's artifacts are published: the new filenames do not exist at the version tag yet, so a premature merge 404s.