Are you a beginner and worried that your systems and applications will crash every time you load a huge data set and run out of memory?
Don’t worry. This quick guide will show you how to handle large data sets like a pro in Python.
Every data professional, novice or expert, has encountered this common problem – the “Panda out of memory error”. It’s because your dataset is too large for Pandas. Once you do this you will see a huge jump in RAM to 99% and suddenly the IDE crashes. Beginners will assume they need a more powerful computer, but the “pros” know that performance means working smarter, not harder.
So what is the real solution? Well, it’s about loading what’s necessary, not loading everything. This article explains how you can use it large datasets in Python.
Common techniques for working with large data sets
Here are some common techniques you can use if the dataset is too large for Pandas to get the most out of the data without crashing the system.
- Master the art of memory optimization
A true data scientist first changes the way they use their tool, not the tool entirely. By default, Pandas is a memory-intensive library that assigns 64-bit types where 8-bit types would be sufficient.
So, what do you need to do?
- Descending numeric types – this means that a column of integers ranging from 0 to 100 does not need an int64 (8 bytes). You can convert it to int8 (1 byte) to reduce memory requirements for this column by 87.5%
- Categorical advantage – here, if you have a column with millions of rows but only ten unique values, convert it to a dtype category. Replaces bulky strings with smaller integer codes.
# Pro Tip: Optimize on the fly
df(‘status’) = df(‘status’).astype(‘category’)
df(‘age’) = pd.to_numeric(df(‘age’), downcast=’integer’)
2. Reading data in bits and pieces
One of the easiest ways to use it Data for exploration in Python is processing them in smaller chunks, rather than loading the entire dataset at once.
In this example, let’s try to find the total revenue from a large data set. You need to use the following code:
import pandas as pd
# Define block size (number of lines per block)
chunk_size = 100,000
total_yield = 0
# Read and process the file in parts
for chunk in pd.read_csv(‘large_sales_data.csv’, chunksize=chunk_size):
# Process each chunk
total_revenue += chunk(‘revenue’).sum()
print(f”Total Revenue: ${total_revenue:,.2f}”)
This will only contain 100,000 rows, no matter how large the dataset is. So even if there are 10 million rows, 100,000 rows will be loaded at once and the sum of each block will be added to the total later.
This technique is best used for aggregations or filtering in large files.
3. Switch to modern file formats like parquet and feathers
The pros use Apache Parquet. Let’s understand it. CSVs are line-based text files that force computers to read each column to find it. Apache Parquet is a column-based storage format, which means if you only need 3 columns out of 100, the system will only touch the data for those 3.
It also comes with a built-in compression feature that will shrink even a 1GB CSV to 100MB without losing a single line of data.
You know that in most scenarios you only need a subset of the rows. In such cases, loading everything is not the right choice. Instead, filter during the loading process.
Here is an example where you can only consider transactions from 2024:
import pandas as pd
# Read chunks and filter
chunk_size = 100,000
filter_chunks = ()
for chunk in pd.read_csv(‘transactions.csv’, chunksize=chunk_size):
# Filter each piece before saving
filtered = chunk(chunk(‘year’) == 2024)
filter_chunks.append(filtered)
# Join the filtered chunks
df_2024 = pd.concat(filtered_chunks, ignore_index=True)
print(f”Loaded {len(df_2024)} rows from 2024″)
- Using Dask for parallel processing
Dask provides a Panda-like API for large datasets, along with automatic handling of other tasks such as sharding and parallel processing.
Here is a simple example of using Dask to calculate the diameter of a column
import dask.dataframe as dd
# Read with Dask (automatically handles tearing)
df = dd.read_csv(‘huge_dataset.csv’)
# Operations look like pandas
result = df(‘sales’).mean()
# Dask is lazy – compute() actually does the calculation
average_sales = result.compute()
print(f”Average sales: ${average_sales:,.2f}”)
Dask creates a plan to process the data in small chunks instead of loading the entire file into memory. This tool can also use multiple CPU cores to speed up the calculation.
Here’s a summary of when you can use these techniques:
|
Technical |
When to use |
Key benefit |
| Types of downcasting | When you have numeric data that fits into smaller ranges (eg age, rating, ID). | Reduces memory requirements by up to 80% without data loss. |
| Categorical conversion | When a column contains repeating text values (eg “Gender”, “City”, or “State”). | Dramatically speeds up sorting and shrinks DataFrames with large numbers of strings. |
| Chunking | When your data set is bigger than your RAM but you just need sum or average. | Prevents “Out of Memory” crashes by keeping only a portion of the data in RAM. |
| Parquet floors / feathers | When you frequently read/write the same data or only need specific columns. | Columnar storage allows the CPU to skip over unnecessary data and saves disk space. |
| Filtering during load | When you only need a certain subset (eg “Current Year” or “Region X”). | It saves time and memory by never loading irrelevant lines into Python. |
| Dask | When your dataset is massive (multi-GB/TB) and you need multi-core speed. | It automates parallel processing and handles data larger than your local memory. |
Conclusion
Remember, handling large data sets should not be a difficult task, even for beginners. You also don’t need a very powerful computer to load and run these huge datasets. You can do it with these common techniques large datasets in Python like a pro. By referring to the given table, you can know which technique should be used for which scenarios. Practice these techniques regularly with sample datasets for better knowledge. You can consider getting top data science certifications to learn these methodologies properly. Work smarter and with Python you can get the most out of your datasets without breaking a sweat.