gpt-tokenizer
6
Versions
—
License
No
Install Scripts
Missing
Provenance
Supply chain provenance
Status for the latest visible version.
No SLSA provenance
npm registry signatures
gitHead linked
Without SLSA provenance there is no cryptographic link between this tarball and the public source, so a manually published version cannot be tied back to a reviewed commit.
Maintainers
niieani
Keywords
BPEencoderdecodertokenizerGPTGPT-2GPT-3GPT-3.5GPT-4GPT-4oNLPNatural Language ProcessingText GenerationOpenAIMachine Learningml
Accepted risks
Findings the reviewer chose to accept rather than block on.
| Source | Rule | Reason | Accepted by | When |
|---|---|---|---|---|
| semgrep | semgrep:dynamic-require | AI (semgrep): Dynamic require is in a test file loading model-specific encodings by name — standard test pattern, not production code, no malicious use. | ai | |
| semgrep | semgrep:base64-decode | AI (semgrep): Base64 decoding is used to parse BPE token tables in the tokenizer's code generation step — core, expected functionality for a GPT tokenizer package. | ai | |
| provenance | no-provenance | AI (provenance): Absence of Sigstore provenance is common (~88% of packages lack it); not a meaningful risk signal for this established package. | ai |