Python JSON Streaming with ijson Cuts Memory Use on a 181 MB Dataset

For large JSON files, the main cost of Python’s built-in parser can be the objects it creates, not the file on disk. In a reported test on an 181 MB GeoJSON dataset, loading the full document with json.load() raised memory use to 808.9 MiB. Iterating through its records with ijson kept the measured peak at 46.1 MiB.

That difference matters when a task needs only a handful of fields from each record. The standard parser builds the entire object tree before analysis begins, including nested geometry that the task may never inspect. An iterative parser lets a program handle one record at a time and release it as it proceeds. In this example, that change made a large reduction in memory use; it did not make the run faster.

The dataset and the task

The example uses the San Francisco City Lots dataset, downloaded in a Google Colab environment from its GitHub source. The file is 181 MB and follows the GeoJSON FeatureCollection structure. Its root has a type field and a features array containing 206,560 parcels.

Each feature includes properties and geometry. The properties hold parcel identifiers and street information; geometry contains polygons and coordinates. The analysis counts records whose STREET property is MARKET and saves the first five matching BLKLOT identifiers. It doesn't need the coordinates, but a full parse still creates Python objects for them.

A basic load looks like this:

import json

with open("citylots.json") as f:
    data = json.load(f)

After that call, the complete document is available in memory as dictionaries, lists, strings, and numbers. The program can then read data["features"] and loop through the parcels. That workflow is simple, but its memory requirements include the full parsed dataset, not just the fields used in the calculation.

What the full parse costs

In the reported run, the full-load version found 808 Market Street parcels and returned the first five IDs as 0265003, 0265004, 0265005, 0290012, and 0291001. It processed the file in 4.01 seconds.

The memory profile shows where the cost appeared. Usage went from 43.4 MiB at the start to 808.9 MiB after json.load(f), an increase of 765.5 MiB on that line. The file’s size on disk is a poor guide to this peak: parsing represents JSON structures as many separate Python objects. Nested arrays and numeric coordinates add to that footprint, even when the analysis ignores them.

This is the central tradeoff in the example. A program that needs only two property values still pays to materialize the other fields if it loads the whole document. For data that fits comfortably in memory, that may be an acceptable convenience. As input grows, the object tree can become the limiting factor.

Iterating over records with ijson

ijson.items() can select objects at a given path in the JSON structure. For this file, the path features.item means each individual element in the features array. The file is opened in binary mode, and the parser yields one feature at a time:

import ijson

with open("citylots.json", "rb") as f:
    for feature in ijson.items(f, "features.item"):
        print(feature)
        break

The processing loop can then count records and inspect the same two properties without first building a list containing every feature:

total = 0
market_count = 0
first_five = []

with open("citylots.json", "rb") as f:
    for feature in ijson.items(f, "features.item"):
        total += 1
        properties = feature["properties"]

        if properties.get("STREET") == "MARKET":
            market_count += 1
            if len(first_five) < 5:
                first_five.append(properties.get("BLKLOT"))

For this pattern, memory is mainly needed for the current feature, the parser’s working buffer, and the small amount of accumulated output. Once a feature has been processed, the loop can move on rather than keeping all previously parsed records in a Python list.

Results and practical tradeoffs

The streaming version produced the same results: 206,560 total parcels, 808 Market Street parcels, and the same first five IDs. Its reported runtime was 4.58 seconds, compared with 4.01 seconds for the full-load version. In this measurement, streaming reduced memory use substantially but took slightly longer.

The measurements come from this dataset and execution environment; they don't establish a universal runtime or memory ratio. In particular, ijson isn't guaranteed to be faster. When sufficient RAM is available, parsing the complete document may be quicker for some workloads. The clearer benefit is that streaming avoids keeping the whole parsed document in memory at once.

Choosing the processing model

Use json.load() when the document is manageable in memory and convenient access to the complete structure matters. Consider ijson when the input is large and the work can be expressed as a pass over records: filter each one, update counters, or retain a small result set. Streaming helps most when downstream work doesn't require the full object tree to remain available.

The change is a shift from “load everything, then process” to “read, process, and discard.” For the parcel analysis, that cut the measured peak from 808.9 MiB to 46.1 MiB while preserving the output. If a workload only needs one record at a time, parsing records incrementally is a practical way to keep memory use from tracking the size of the entire dataset.

Kanwal Mehreen is a machine learning engineer and technical writer focused on data science and the intersection of AI with medicine. She co-authored the ebook “Maximizing Productivity with ChatGPT” and founded FEMCodes to support women in STEM. She has also received the Google Generation Scholar award for APAC, the Teradata Diversity in Tech Scholar recognition, the Mitacs Globalink Research Scholar award, and the Harvard WeCode Scholar recognition.