Performance••13 min read

Working with Large JSON Files: A Survival Guide

Parse, stream, and query JSON documents that are too big for memory. NDJSON, streaming parsers, jq, simdjson, and the rules to avoid OOM.

By JSONEditor.net Team

What counts as "large"

The threshold depends on where you are running the code.

In the browser, a single tab gets between 2 and 4 GB of heap on modern Chrome. JSON.parse needs roughly 4× the document size in working memory because the parser builds intermediate strings and the resulting object graph carries V8's hidden-class overhead. A 500 MB JSON file will hit OOM in most tabs.

In Node, the default heap is 4 GB and can be raised with --max-old-space-size. Same 4× rule of thumb applies. You can comfortably parse documents up to about 800 MB on default settings.

In Python or Go, the OS limits apply — basically all your RAM. But the parser is still O(N) memory, and you still need somewhere to put the resulting tree.

So "large" starts around 100 MB if you care about speed and around 500 MB if you care about not crashing. Past 2 GB, parsing the whole thing is almost always wrong.

Why naive parsing breaks at scale

JSON.parse and its equivalents in every language are tree builders. They read the entire document, allocate a value for every literal, allocate a container (object or array) for every nesting level, and return the root. Memory cost scales with the document size; time cost is O(N) but with a large constant because of all the allocations.

For a 1 GB document of nested objects, V8 will allocate hundreds of millions of small objects. Garbage collection pauses dominate. You will see your event loop blocked for seconds.

The other failure mode is worse: you parse successfully, then iterate the result with .map or .filter, and each iteration allocates more intermediate arrays. A query that should produce 100 KB of output uses 5 GB of intermediate memory and crashes.

The solutions all share the same idea: never have the whole document in memory at the same time. Process it as a stream of records.

NDJSON: the format that solves most of this

NDJSON (Newline-Delimited JSON, also called JSON Lines or JSONL) is a convention, not a different format. Each line is an independent JSON document. There is no enclosing array.

{"id":1,"name":"Ada","age":36}
{"id":2,"name":"Bob","age":42}
{"id":3,"name":"Cha","age":29}

Every modern data tool understands it: BigQuery, Snowflake, Postgres COPY, jq, Spark, Pandas. The killer property is that you can stream it line by line with a 50-byte buffer — no parser state survives between records.

import { createReadStream } from 'fs';
import { createInterface } from 'readline';

const stream = createReadStream('events.ndjson');
const lines = createInterface({ input: stream });

for await (const line of lines) {
  const record = JSON.parse(line);
  if (record.type === 'click') process(record);
}

If you generate large datasets, write NDJSON instead of a giant array. If you receive large datasets, push back for NDJSON when you can. The rest of this guide is workarounds for the cases where you cannot.

Streaming parsers

A streaming parser reads the document as a stream of tokens — start-object, key("foo"), string("bar"), end-object — and lets you decide what to do with each one. You never hold the whole tree in memory.

In Node, the standard choice is JSONStream or the newer stream-json. They expose a pull-based API where you ask for "every element of the top-level array" and the parser emits one at a time.

import { parser } from 'stream-json';
import { streamArray } from 'stream-json/streamers/StreamArray';
import { createReadStream } from 'fs';

const pipeline = createReadStream('huge.json')
  .pipe(parser())
  .pipe(streamArray());

for await (const { value } of pipeline) {
  // value is each top-level array element, one at a time
  if (value.score > 100) await save(value);
}

In Python, ijson does the same:

import ijson

with open('huge.json', 'rb') as f:
    for record in ijson.items(f, 'item'):
        process(record)

In Go, the standard library's encoding/json has json.Decoder with Token() and Decode() methods that are stream-aware out of the box. Use dec.Decode(&v) in a loop for the array-of-records pattern.

Streaming with jq

jq can stream too, with the --stream flag. It emits a sequence of [path, value] pairs that you reassemble with fromstream.

# Get every top-level array item one at a time, low memory
jq --stream -c 'fromstream(1|truncate_stream(inputs))' huge.json

# Extract every "user.email" anywhere in the document
jq --stream 'select(length == 2) | select(.[0][-1] == "email") | .[1]' huge.json

The syntax is uglier than non-streaming jq because you are working at the token level, but it processes arbitrarily large files with constant memory. For one-off extraction from a multi-gigabyte log, it is unbeatable.

When the document fits but you want it faster

If the document fits in memory but parsing is the bottleneck, look at simdjson. It uses SIMD instructions to parse JSON at gigabytes per second — about 4× faster than the V8 parser, 10× faster than Python's stdlib. There are bindings for Node, Python, Rust, Go, Ruby, and PHP.

// Node — npm install simdjson
import simdjson from 'simdjson';
const parsed = simdjson.parse(text); // same API as JSON.parse, much faster

simdjson is worth it when you parse large documents repeatedly and the parse cost dominates your CPU time. For a single one-off parse, the startup overhead of loading the native binding eats the speedup. Profile before switching.

Large JSON in the browser

The browser has tighter limits than the server, and most tools that work in Node will not run unmodified.

For documents up to ~50 MB, JSON.parse works but blocks the main thread for hundreds of milliseconds. Move the parse to a Web Worker so the UI stays responsive:

// worker.js
self.onmessage = (e) => {
  const result = JSON.parse(e.data);
  // do something useful (extract fields, count items), don't post the whole tree back
  self.postMessage({ count: result.length });
};

// main.js
const worker = new Worker('worker.js');
worker.postMessage(largeJsonString);
worker.onmessage = (e) => updateUI(e.data);

For documents larger than ~100 MB, the browser will struggle even in a worker. The Streams API plus a streaming parser like oboe.js works around it — you can start displaying records before the download finishes.

For documents larger than the browser can handle, do not try. Render an upload form that sends the file to a backend that streams it through proper tooling. The browser is the wrong place for a 2 GB JSON file.

Storing large JSON

Two quick reminders that come up when JSON grows.

Postgres has both json (stored as text) and jsonb (stored as a binary tree). Use jsonb for anything you query — it deduplicates keys, supports indexes, and queries hundreds of times faster on large documents. Use json only if you need to preserve the exact byte stream including key order and whitespace.

S3 and object storage charge per request and per byte. Store NDJSON, not arrays. NDJSON can be split by line boundaries for parallel processing; an array forces every consumer to download the whole file before extracting anything. Compress with gzip or zstd — JSON compresses to about 10% of original size.

Avoid JSON for time-series. If you are writing one record per event, NDJSON of small records is fine. If you are storing millions of measurements, switch to a columnar format like Parquet or a time-series database. JSON's overhead per record is too high.

The five rules

To save the next person reading this from re-deriving the lessons:

  1. If you can change the format, switch to NDJSON. Solves 80% of the cases.
  2. If the document fits in memory and you need it for a millisecond, just use JSON.parse. Streaming is a tax on simple code.
  3. If the document fits but you parse it often, profile, then maybe switch to simdjson.
  4. If the document does not fit, stream it. Pick the parser that matches your language: stream-json, ijson, jq --stream.
  5. Never put a multi-gigabyte JSON in a browser tab. There is always a smaller derived form you can compute server-side.

For everything below 50 MB, the browser tools on this site handle it without breaking a sweat. The numbers in this post are about the territory past that — where the right answer is usually "do not load the whole thing in the first place."

JSONCSVConvertersData