How to Stream Huge Files Using Laravel

Reading the contents of a multi-gigabyte file into memory with PHP is not possible with common tools such as file(), file_get_contents(), or even Laravel's standard collect(). PHP would attempt to allocate several gigabytes of memory for the result array, which would most likely exceed your PHP memory limit and result in an Allowed memory size exhausted error.

The most common approach is to split the file into smaller parts and save them on a temporary disk, which is far from being optimal. Instead, we can use native PHP generators and LazyCollections to implement a memory-efficient solution that does not require any additional disk operations.

In this tutorial, we'll dive into PHP generators and see how Laravel's LazyCollection wraps them in order to implement a memory-efficient solution for reading, filtering, transforming, and inserting gigabyte-scale CSV files with less than 10MB of memory.

The Memory Problem: Eager Collections

To understand why large files crash PHP, look at what happens with standard Laravel collections:

// Reading a 2 GB file into a standard collection
$lines = file(storage_path('imports/transactions.csv')); // Loads 2 GB into memory

$collection = collect($lines)->map(function ($line) {
    return str_getcsv($line);
});

When you call file(), PHP reads all lines into an array. When you call collect($lines), Laravel duplicates these references to memory in a collection class. The call to map() creates yet another array in memory for the result of the mapping. If your CSV has 2 GB, your script will require about 4–6 GB of RAM to perform all these operations.

How PHP Generators Work Under the Hood

A regular function in PHP returns a single value and destroys its internal execution state. A Generator function uses the yield keyword instead of return.

When a generator reaches the yield keyword, it produces a value, pauses execution, and saves its place in the code. Execution then resumes when the caller requests the next item in a loop:

function readLines(string $filePath): Generator
{
    $handle = fopen($filePath, 'r');
    while (($line = fgets($handle)) !== false) {
        yield trim($line); // Yields one line and pauses
    }

    fclose($handle);
}

As you iterate over the result of readLines() with a foreach loop PHP will only keep a single string in memory at any given time. Once the loop moves to the next line it will garbage collect the previous line.

Introducing Laravel LazyCollections

Native PHP generators are memory efficient, but offer no collection helpers. You can't simply call ->filter() or ->chunk() or even ->pluck() or ->map() on a native generator, without writing your own loops.

Laravel's LazyCollection aims to bridge this gap. It implements PHP's Enumerable and IteratorAggregate interfaces, wrapping a generator and providing all the standard collection methods. The collection methods are applied lazily, meaning each step is executed item by item, down the pipeline.

Step-by-Step: Streaming a Large CSV

Let's build an importer that reads a 5 GB CSV file of transaction records, parses the headers, normalizes amounts, removes invalid rows, and saves the data in batches.

1. Create a Lazy CSV Reader Method

We use LazyCollection::make(), passing a generator function that opens a file stream and yields parsed CSV rows:

namespace App\Services;

use Illuminate\Support\LazyCollection;

class LargeCsvReader
{
    /**
     * Stream a CSV file line-by-line as an associative array.
     */
    public function stream(string $filePath): LazyCollection
    {
        return LazyCollection::make(function () use ($filePath) {
            $handle = fopen($filePath, 'r');

            if ($handle === false) {
                return;
            }

            // Read the first line as headers
            $headers = fgetcsv($handle);

            if ($headers === false) {
                fclose($handle);
                return;
            }

            // Trim any whitespace from column headers
            $headers = array_map(fn ($h) => trim($h), $headers);

            // Stream each row one-by-one
            while (($row = fgetcsv($handle)) !== false) {
                // Ensure row column count matches header count
                if (count($headers) === count($row)) {
                    yield array_combine($headers, $row);
                }
            }

            fclose($handle);
        });
    }
}

2. Pipeline Transformation and Batch Database Insertion

Now, we can chain operations like filter(), map(), and chunk() directly on the LazyCollection. Data flows through each step item-by-item:

namespace App\Console\Commands;

use App\Models\Transaction;
use App\Services\LargeCsvReader;
use Illuminate\Console\Command;

class ImportLargeTransactionsCommand extends Command
{
    protected $signature = 'import:transactions {file}';
    protected $description = 'Import huge CSV files without memory spikes';

    public function handle(LargeCsvReader $reader): int
    {
        $filePath = storage_path('imports/' . $this->argument('file'));

        if (! file_exists($filePath)) {
            $this->error("File not found at: {$filePath}");
            return Command::FAILURE;
        }

        $this->info('Starting stream import...');

        // Stream and transform without loading the whole file
        $reader->stream($filePath)
            // Skip rows with missing or zero amount
            ->filter(function (array $row) {
                return ! empty($row['amount']) && (float) $row['amount'] > 0;
            })
            // Normalize columns for database insertion
            ->map(function (array $row) {
                return [
                    'reference'  => $row['transaction_id'],
                    'amount'     => (float) $row['amount'],
                    'currency'   => strtoupper(trim($row['currency'] ?? 'USD')),
                    'status'     => strtolower(trim($row['status'] ?? 'pending')),
                    'created_at' => now(),
                    'updated_at' => now(),
                ];
            })
            // Group stream into batches of 1,000 items
            ->chunk(1000)
            ->each(function ($batch) {
                // Inserts 1,000 records in a single query
                Transaction::insert($batch->all());
            });

        $this->info('Import completed successfully.');
        $this->line('Peak memory usage: ' . round(memory_get_peak_usage() / 1024 / 1024, 2) . ' MB');

        return Command::SUCCESS;
    }
}

Even if the CSV contains 10 million rows, memory usage remains virtually flat at around 4 MB to 8 MB for the entire run. Only 1,000 array items exist in memory during each insert() call.

Common Trap: Accidental Materialization

When working with LazyCollections, you must avoid calling methods that force materialization of the collection. Materialization is when you call a method that needs to examine all the items in the collection in order to return a result, which defeats the point of having a generator-based collection.

The following methods will cause materialization to occur:

all() : Will turn the stream into a normal PHP array.

count() : Needs to iterate the entire stream in order to return a count.

sortBy() : Needs to collect all the items in an array before it can sort them.

reverse() : Needs to collect all the items in order to output them in reverse order.

Compare how these behave:

// CRASH: sortBy() loads every line of the 5 GB file into RAM
$reader->stream($filePath)->sortBy('amount')->each(...);

// SAFE: chunk() processes items lazily in blocks of 1,000
$reader->stream($filePath)->chunk(1000)->each(...);

If you need to sort a multi-gigabyte dataset, perform the sort using database indexes after insertion, or use a command-line tool like the Unix sort utility before running the import.

Creating LazyCollections from Built-In Sources

Laravel also includes built-in factory methods on LazyCollection for reading file lines directly:

use Illuminate\Support\LazyCollection;

// Read lines from a plain text or log file
LazyCollection::make(function () {
    $handle = fopen(storage_path('logs/laravel.log'), 'r');

    while (($line = fgets($handle)) !== false) {
        yield $line;
    }

    fclose($handle);
})
->filter(fn ($line) => str_contains($line, 'ERROR'))
->take(10)
->each(fn ($line) => dump($line));

Notice take(10). Once 10 matching error lines are found, the LazyCollection stops iterating and closes the file stream immediately, skipping the rest of the file.

Standard Collection vs. LazyCollection

Feature	Illuminate\Support\Collection	Illuminate\Support\LazyCollection
Data Storage	Stores all items in an internal array in RAM.	Backing PHP Generator produces items on demand.
Memory Footprint	Grows linearly with dataset size ($O(n)$).	Constant memory usage ($O(1)$).
Evaluation Timing	Eager (executes immediately on method call).	Lazy (executes step-by-step during terminal loop).
Re-usability	Can be iterated multiple times freely.	Re-running restarts the generator from scratch.
Best Use Case	UI lists, small payloads (< 5,000 items).	Large CSVs, log file parsers, multi-GB exports.

Important Things to Remember

📌 Clean up resources: Always call fclose($handle) when reading files inside generators. If your loop has been broken out of using take(), PHP's garbage collector will clean up the generator instance, and any unreferenced stream handles it has opened will be closed.

⚡ Use database inserts wisely: Avoid creating full Eloquent models inside massive import loops (e.g. Transaction::create()) as hydrating model instances is an unnecessary overhead. Instead, use Transaction::insert($chunk) or upsert() on the raw array data.

⚠️ Generators cannot rewind: You cannot rewind (reset) a generator, like you can with an array, once you've exhausted it. You will have to re-execute the generator function.

Conclusion

Pairing PHP Generators with Laravel LazyCollections allows you to work with huge files cleanly without having to fiddle with PHP's memory limit or risking out-of-memory exceptions by streaming the file line by line and batching the database inserts

Thank you for reading this article 😊

For any query, do not hesitate to comment 💬