gpt-tokenizer

gpt-tokenizer is a Token Byte Pair Encoder/Decoder supporting all OpenAI's models (including GPT-5, GPT-4o, o1, o3, o4, GPT-4.1 and older models like GPT-3.5, GPT-4). It's the fastest, smallest and lowest footprint GPT tokenizer available for all JavaScript environments and is written in TypeScript.

Try it out in the playground!

This library has been trusted by:

• CodeRabbit (sponsor 🩷)
• Microsoft (Teams, GenAIScript)
• Elastic (Kibana)
• Effect TS
• Rivet by Ironclad

Please consider 🩷 sponsoring the project if you find it useful.

Features

It is the most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's tiktoken, with some additional, unique features sprinkled on top:

• Support for easily tokenizing chats thanks to the encodeChat function
• Support for all current OpenAI models (available encodings: r50k_base, p50k_base, p50k_edit, cl100k_base, o200k_base, and o200k_harmony)
• Can be loaded and work synchronously! (i.e. in non async/await contexts)
• Generator function versions of both the decoder and encoder functions
• Provides the ability to decode an asynchronous stream of data (using decodeAsyncGenerator and decodeGenerator with any iterable input)
• No global cache (no accidental memory leaks, as with the original GPT-3-Encoder implementation)
• Includes a highly performant isWithinTokenLimit function to assess token limit without encoding the entire text/chat
• Built-in cost estimation with the estimateCost function for calculating API usage costs
• Full library of OpenAI models with comprehensive pricing information (see src/models.ts and src/models.gen.ts)
• Improves overall performance by eliminating transitive arrays
• Type-safe (written in TypeScript)
• Works in the browser out-of-the-box

Installation

As NPM package

npm install gpt-tokenizer

As a UMD module

<script src="https://unpkg.com/gpt-tokenizer"></script>

<script>
  // the package is now available as a global:
  const { encode, decode } = GPTTokenizer_cl100k_base
</script>

If you wish to use a custom encoding, fetch the relevant script.

• https://unpkg.com/gpt-tokenizer/dist/o200k_base.js (for all modern models, such as gpt-5, gpt-4o, gpt-4.1, o1 and others)
• https://unpkg.com/gpt-tokenizer/dist/o200k_harmony.js (for open-weight Harmony models such as gpt-oss-20b and gpt-oss-120b)
• https://unpkg.com/gpt-tokenizer/dist/cl100k_base.js (for gpt-4 and gpt-3.5)
• https://unpkg.com/gpt-tokenizer/dist/p50k_base.js
• https://unpkg.com/gpt-tokenizer/dist/p50k_edit.js
• https://unpkg.com/gpt-tokenizer/dist/r50k_base.js

The global name is a concatenation: GPTTokenizer_${encoding}.

Refer to supported models and their encodings section for more information.

Playground

The playground is published under a memorable URL: https://gpt-tokenizer.dev/

Usage

The library provides various functions to transform text into (and from) a sequence of integers (tokens) that can be fed into an LLM model. The transformation is done using a Byte Pair Encoding (BPE) algorithm used by OpenAI.

import {
  encode,
  encodeChat,
  decode,
  isWithinTokenLimit,
  encodeGenerator,
  decodeGenerator,
  decodeAsyncGenerator,
  ALL_SPECIAL_TOKENS,
} from 'gpt-tokenizer'
// note: depending on the model, import from the respective file, e.g.:
// import {...} from 'gpt-tokenizer/model/gpt-4o'

const text = 'Hello, world!'
const tokenLimit = 10

// Encode text into tokens
const tokens = encode(text)

// Decode tokens back into text
const decodedText = decode(tokens)

// Check if text is within the token limit
// returns false if the limit is exceeded, otherwise returns the actual number of tokens (truthy value)
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

// Allow special tokens when needed
const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

// Example chat:
const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },
] as const

// Encode chat into tokens
const chatTokens = encodeChat(chat)

// Check if chat is within the token limit
const chatWithinTokenLimit = isWithinTokenLimit(chat, tokenLimit)

const chatWithinTokenLimitWithSpecial = isWithinTokenLimit(chat, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

// Encode text using generator
for (const tokenChunk of encodeGenerator(text)) {
  console.log(tokenChunk)
}

// Decode tokens using generator
for (const textChunk of decodeGenerator(tokens)) {
  console.log(textChunk)
}

// Decode tokens using async generator
// (assuming `asyncTokens` is an AsyncIterableIterator<number>)
for await (const textChunk of decodeAsyncGenerator(asyncTokens)) {
  console.log(textChunk)
}

By default, importing from gpt-tokenizer uses o200k_base encoding, used by all modern OpenAI models, including gpt-4o, gpt-4.1, o1, etc.

To get a tokenizer for a different model, import it directly, for example:

import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/model/gpt-3.5-turbo'

If you're dealing with a resolver that doesn't support package.json exports resolution, you might need to import from the respective cjs or esm directory, e.g.:

import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/cjs/model/gpt-3.5-turbo'

Lazy loading

If you don't mind loading the tokenizer asynchronously, you can use a dynamic import inside your function, like so:

const {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} = await import('gpt-tokenizer/model/gpt-3.5-turbo')

Loading an encoding

If your model isn't supported by the package, but you know which BPE encoding it uses, you can load the encoding directly, e.g.:

import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/encoding/cl100k_base'

Supported models and their encodings

We support all OpenAI models, including the latest ones, with the following encodings:

• o-series models, like o1-*, o3-* and o4-* (o200k_base)
• gpt-4o (o200k_base)
• gpt-oss-* (o200k_harmony)
• gpt-4-* (cl100k_base)
• gpt-3.5-* (cl100k_base)
• text-davinci-003 (p50k_base)
• text-davinci-002 (p50k_base)
• text-davinci-001 (r50k_base)
• ...and many other models, see models.ts for an up-to-date list of supported models and their encodings.

If you don't see the model you're looking for, the default encoding is probably the one you want.

API

encode(text: string, encodeOptions?: EncodeOptions): number[]

Encodes the given text into a sequence of tokens. Use this method when you need to transform a piece of text into the token format that the GPT models can process.

The optional encodeOptions parameter allows you to specify special token handling (see special tokens).

Example:

import { encode } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokens = encode(text)

decode(tokens: number[]): string

Decodes a sequence of tokens back into text. Use this method when you want to convert the output tokens from GPT models back into human-readable text.

Example:

import { decode } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
const text = decode(tokens)

isWithinTokenLimit(text: string | Iterable<ChatMessage>, tokenLimit: number, encodeOptions?: EncodeOptions): false | number

Checks if the input is within the token limit. Returns false if the limit is exceeded, otherwise returns the number of tokens. Use this method to quickly check if a given text or chat is within the token limit imposed by GPT models, without encoding the entire input. The optional encodeOptions parameter lets you configure special token handling.

Example:

import { isWithinTokenLimit, ALL_SPECIAL_TOKENS } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenLimit = 10
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

countTokens(text: string | Iterable<ChatMessage>, encodeOptions?: EncodeOptions): number

Counts the number of tokens in the input text or chat. Use this method when you need to determine the number of tokens without checking against a limit. The optional encodeOptions parameter allows you to specify custom sets of allowed or disallowed special tokens.

Example:

import { countTokens } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenCount = countTokens(text)

countChatCompletionTokens(request: ChatCompletionRequest): number

Counts the tokens that a function-calling chat completion request will consume, including message overhead, optional function definitions, and pinned function calls. This helper is only available on models that support the function_calling feature.

Example:

import {
  countChatCompletionTokens,
  type ChatCompletionRequest,
} from 'gpt-tokenizer/model/gpt-4o'

const request: ChatCompletionRequest = {
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Find the weather for San Francisco.' },
  ],
  functions: [
    {
      name: 'get_weather',
      description: 'Look up the weather for a city.',
      parameters: {
        type: 'object',
        required: ['city'],
        properties: {
          city: { type: 'string' },
          unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
      },
    },
  ],
}

const promptTokenEstimate = countChatCompletionTokens(request)

You can also access the helper from the module's default export:

import gpt4o from 'gpt-tokenizer/model/gpt-4o'

// Reuse the `request` defined above
const tokenCount = gpt4o.countChatCompletionTokens?.(request)

encodeChat(chat: ChatMessage[], model?: ModelName, encodeOptions?: EncodeOptions): number[]

Encodes the given chat into a sequence of tokens. The optional encodeOptions parameter lets you configure special token handling.

If you didn't import the model version directly, or if model wasn't provided during initialization, it must be provided here to correctly tokenize the chat for a given model. Use this method when you need to transform a chat into the token format that the GPT models can process.

Example:

import { encodeChat } from 'gpt-tokenizer'

const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },
]
const tokens = encodeChat(chat)

Note that if you encode an empty chat, it will still contain the minimum number of special tokens.

encodeGenerator(text: string): Generator<number[], void, undefined>

Encodes the given text using a generator, yielding chunks of tokens. Use this method when you want to encode text in chunks, which can be useful for processing large texts or streaming data.

Example:

import { encodeGenerator } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokens = []
for (const tokenChunk of encodeGenerator(text)) {
  tokens.push(...tokenChunk)
}

encodeChatGenerator(chat: Iterator<ChatMessage>, model?: ModelName): Generator<number[], void, undefined>

Same as encodeChat, but uses a generator as output, and may use any iterator as the input chat.

decodeGenerator(tokens: Iterable<number>): Generator<string, void, undefined>

Decodes a sequence of tokens using a generator, yielding chunks of decoded text. Use this method when you want to decode tokens in chunks, which can be useful for processing large outputs or streaming data.

Example:

import { decodeGenerator } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
let decodedText = ''
for (const textChunk of decodeGenerator(tokens)) {
  decodedText += textChunk
}

decodeAsyncGenerator(tokens: AsyncIterable<number>): AsyncGenerator<string, void, undefined>

Decodes a sequence of tokens asynchronously using a generator, yielding chunks of decoded text. Use this method when you want to decode tokens in chunks asynchronously, which can be useful for processing large outputs or streaming data in an asynchronous context.

Example:

import { decodeAsyncGenerator } from 'gpt-tokenizer'

async function processTokens(asyncTokensIterator) {
  let decodedText = ''
  for await (const textChunk of decodeAsyncGenerator(asyncTokensIterator)) {
    decodedText += textChunk
  }
}

estimateCost(tokenCount: number, modelSpec?: ModelSpec): PriceData

Estimates the cost of pr