Tokenizers
Monthly
Out-of-bounds write in Hugging Face tokenizers v0.23.1 allows any actor who can supply a crafted tokenizer.json file to abort the loading process immediately, causing denial of service in Rust applications and FFI embeddings before any inference occurs. The defect in BpeBuilder::build (tokenizers/src/models/bpe/model.rs) allocates a scratch buffer sized to the longest vocabulary key, then writes concatenated BPE merge pairs into it without bounding the write to the buffer size; a merge whose result exceeds that length triggers a Rust panic and process abort (CWE-787). A secondary usize underflow at the same code site, activated when continuing_subword_prefix is set and a merge token is shorter than the prefix, produces a panic in debug builds and potential memory corruption in release builds, elevating the secondary impact beyond availability alone. No public exploit code or CISA KEV listing has been identified at time of analysis; a fix appears in the v0.23.2 source.
Out-of-bounds write in Hugging Face tokenizers v0.23.1 allows any actor who can supply a crafted tokenizer.json file to abort the loading process immediately, causing denial of service in Rust applications and FFI embeddings before any inference occurs. The defect in BpeBuilder::build (tokenizers/src/models/bpe/model.rs) allocates a scratch buffer sized to the longest vocabulary key, then writes concatenated BPE merge pairs into it without bounding the write to the buffer size; a merge whose result exceeds that length triggers a Rust panic and process abort (CWE-787). A secondary usize underflow at the same code site, activated when continuing_subword_prefix is set and a merge token is shorter than the prefix, produces a panic in debug builds and potential memory corruption in release builds, elevating the secondary impact beyond availability alone. No public exploit code or CISA KEV listing has been identified at time of analysis; a fix appears in the v0.23.2 source.