Menu
PDF Tools
Image Tools
Converters & Utilities
Seller & Print Tools
Accounting & Business
Calculators
Text & Writing Tools
Developer & Web Tools
Creator & Social Tools
Browser Utilities & Games
Document & Design Makers
Text & Developer Tools
Exam & Form Tools

Text Chunker for LLMs

Paste a long text and split it into chunks of a size you choose, with overlap, cutting by tokens, words, characters, sentences or Markdown headings. Export the chunks as JSONL, JSON or text.

How to use Text Chunker for LLMs online

  1. Paste the text. Or open a file.
  2. Choose the unit, size and overlap. Tokens, words, characters, sentences, headings.
  3. Check the table. Look at the starts and lengths.
  4. Copy the output. JSONL, JSON or text.

Why use this free Text Chunker for LLMs

  • Five ways to cut
  • Exact token chunks
  • Overlap control
  • JSONL, JSON and text output
  • Table of chunks
  • Up to one million characters

Why split text

A model can read only so much at once, and search systems that fetch passages work better with pieces of a sensible size. Splitting a long document into chunks lets you feed it in turns or store pieces for retrieval. This page cuts the text in the way you choose and shows each chunk.

Five ways to cut

Cut by tokens for an exact size under an OpenAI-style tokenizer, by words or characters for a simple size, by sentences to keep whole sentences together, or by Markdown headings to keep sections together. For sentences and headings the size is counted in words, and a unit is never cut in the middle.

Overlap

Overlap repeats the end of one chunk at the start of the next, so a thought that crosses a boundary is not lost. For sentences it repeats the last whole sentences that fit within the overlap. The overlap must be smaller than the chunk size, and the page tells you if it is not.

Output

Choose JSONL, with one object on each line holding an id, the text and the character count, or a JSON list, or plain text with separators. A table lists the first fifty chunks with their length and start so you can check the cuts before you copy.

Limits

Token cuts use one tokenizer and are exact for it only. A section that is longer than the chunk size stays whole when you cut by headings, so check the longest chunk. Texts of up to one million characters are accepted, and nothing is uploaded.

Using the tool

Change the text or any option of the Text Chunker for LLMs and the answer moves straight away. Undo and redo step back through your earlier text.

Text Chunker for LLMs: frequently asked questions

Which unit should I choose for retrieval?
Sentences or headings keep meaning together; tokens give the most even sizes. Try both on your own text.
How large should the overlap be?
A tenth to a fifth of the chunk size is a common starting point, but test with your own data.
Why is a chunk larger than my size?
When cutting by headings or sentences, one unit can be longer than the size, and it is not cut.
Does it keep Markdown?
Yes. The text of each chunk is kept as it is.
Does the cutting send my text anywhere?
No. It runs in your browser.
Does the Text Chunker for LLMs keep a copy of my text?
Your text stays on this device when you use the Text Chunker for LLMs. The editor may keep the last text locally and offers to bring it back; you can dismiss it.
Can I chain the Text Chunker for LLMs with another tool?
Copy the Text Chunker for LLMs result, or save it as a text file from the result box.

Related searches for Text Chunker for LLMs

  • Text Chunker for LLMs online
  • free Text Chunker for LLMs
  • Text Chunker for LLMs online free
  • Text Chunker for LLMs for mobile
  • Text Chunker for LLMs without sign up
  • Text Chunker for LLMs – Text & Writing Tools

Text Chunker for LLMs is a free online tool on MAITBOX, in the Text & Writing Tools group. It runs in your browser, so what you type is not sent to a server. It works on Android and iPhone, and on Windows, Mac and Linux, with no download and no account.

More tools in Text & Writing Tools →

Search tools