{"id":598426,"date":"2025-02-25T15:56:16","date_gmt":"2025-02-25T20:56:16","guid":{"rendered":"https:\/\/towardsdatascience.com\/?p=598426"},"modified":"2025-02-25T15:56:17","modified_gmt":"2025-02-25T20:56:17","slug":"efficient-data-handling-in-python-with-arrow","status":"publish","type":"post","link":"https:\/\/towardsdatascience.com\/efficient-data-handling-in-python-with-arrow\/","title":{"rendered":"Efficient Data Handling in Python with Arrow"},"content":{"rendered":"\n

1. Introduction<\/h2>\n\n\n\n

We\u2019re all used to work with CSVs, JSON files\u2026 With the traditional libraries and for large datasets, these can be extremely slow to read, write and operate on, leading to performance bottlenecks (been there). It\u2019s precisely with big amounts of data that being efficient handling the data is crucial for our data science\/analytics workflow, and this is exactly where Apache Arrow comes into play. <\/p>\n\n\n\n

Why? The main reason resides in how the data is stored in memory. While JSON and CSVs, for example, are text-based formats, Arrow is a columnar in-memory data format (and that allows for fast data interchange between different data processing tools). Arrow is therefore designed to optimize performance by enabling zero-copy reads, reducing memory usage, and supporting efficient compression. <\/p>\n\n\n\n

Moreover, Apache Arrow is open-source and optimized for analytics. It is designed to accelerate big data processing while maintaining interoperability with various data tools, such as Pandas, Spark, and Dask. By storing data in a columnar format, Arrow enables faster read\/write operations and efficient memory usage, making it ideal for analytical workloads.<\/p>\n\n\n\n

Sounds great right? What\u2019s best is that this is all the introduction to Arrow I\u2019ll provide. Enough theory, we want to see it in action. So, in this post, we’ll explore how to use Arrow in Python and how to make the most out of it.<\/p>\n\n\n\n

2. Arrow in Python<\/h2>\n\n\n\n

To get started, you need to install the necessary libraries: pandas and pyarrow.<\/p>\n\n\n\n

pip install pyarrow pandas<\/code><\/pre>\n\n\n\n
Then, as always, import them in your Python script:<\/p>\n\n\n\n
import pyarrow as pa\nimport pandas as pd<\/code><\/pre>\n\n\n\n
Nothing new yet, just necessary steps to do what follows. Let\u2019s start by performing some simple operations.<\/p>\n\n\n\n
2.1. Creating and Storing a Table<\/h3>\n\n\n\n
The simplest we can do is hardcode our table\u2019s data. Let\u2019s create a two-column table with football data:<\/p>\n\n\n\n
teams = pa.array(['Barcelona', 'Real Madrid', 'Rayo Vallecano', 'Athletic Club', 'Real Betis'], type=pa.string())\ngoals = pa.array([30, 23, 9, 24, 12], type=pa.int8())\n\nteam_goals_table = pa.table([teams, goals], names=['Team', 'Goals'])<\/code><\/pre>\n\n\n\n
The format is pyarrow.table<\/em>, but we can easily convert it to pandas if we want:<\/p>\n\n\n\n
df = team_goals_table.to_pandas()<\/code><\/pre>\n\n\n\n
And restore it back to arrow using:<\/p>\n\n\n\n
team_goals_table = pa.Table.from_pandas(df)<\/code><\/pre>\n\n\n\n
And we\u2019ll finally store the table in a file. We could use different formats, like feather, parquet\u2026 I\u2019ll use this last one because it\u2019s fast and memory-optimized:<\/p>\n\n\n\n
import pyarrow.parquet as pq\npq.write_table(team_goals_table, 'data.parquet')<\/code><\/pre>\n\n\n\n
Reading a parquet file would just consist of using pq.read_table('data.parquet')<\/code>.<\/p>\n\n\n\n
2.2. Compute Functions<\/h3>\n\n\n\n
Arrow has its own compute module for the usual operations. Let\u2019s start by comparing two arrays element-wise:<\/p>\n\n\n\n
import pyarrow.compute as pc\n>>> a = pa.array([1, 2, 3, 4, 5, 6])\n>>> b = pa.array([2, 2, 4, 4, 6, 6])\n>>> pc.equal(a,b)\n[\n  false,\n  true,\n  false,\n  true,\n  false,\n  true\n]<\/code><\/pre>\n\n\n\n
That was easy, we could sum all elements in an array with:<\/p>\n\n\n\n
>>> pc.sum(a)\n<pyarrow.Int64Scalar: 21><\/code><\/pre>\n\n\n\n
And from this we could easily guess how we can compute a count, a floor, an exp, a mean, a max, a multiplication\u2026 No need to go over them, then. So let\u2019s move to tabular operations.<\/p>\n\n\n\n
We\u2019ll start by showing how to sort it:<\/p>\n\n\n\n
>>> table = pa.table({'i': ['a','b','a'], 'x': [1,2,3], 'y': [4,5,6]})\n>>> pc.sort_indices(table, sort_keys=[('y', descending)])\n<pyarrow.lib.UInt64Array object at 0x1291643a0>\n[\n  2,\n  1,\n  0\n]<\/code><\/pre>\n\n\n\n
Just like in pandas, we can group values and aggregate the data. Let\u2019s, for example, group by \u201ci\u201d and compute the sum on \u201cx\u201d and the mean on \u201cy\u201d:<\/p>\n\n\n\n
>>> table.group_by('i').aggregate([('x', 'sum'), ('y', 'mean')])\npyarrow.Table\ni: string\nx_sum: int64\ny_mean: double\n----\ni: [["a","b"]]\nx_sum: [[4,2]]\ny_mean: [[5,5]]<\/code><\/pre>\n\n\n\n
Or we can join two tables:<\/p>\n\n\n\n
>>> t1 = pa.table({'i': ['a','b','c'], 'x': [1,2,3]})\n>>> t2 = pa.table({'i': ['a','b','c'], 'y': [4,5,6]})\n>>> t1.join(t2, keys="i")\npyarrow.Table\ni: string\nx: int64\ny: int64\n----\ni: [["a","b","c"]]\nx: [[1,2,3]]\ny: [[4,5,6]]<\/code><\/pre>\n\n\n\n
By default, it is a left outer join but we could twist it by using the join_type<\/em><\/strong> parameter.<\/p>\n\n\n\n
There are many more useful operations, but let\u2019s see just one more to avoid making this too long: appending a new column to a table.<\/p>\n\n\n\n
>>> t1.append_column("z", pa.array([22, 44, 99]))\npyarrow.Table\ni: string\nx: int64\nz: int64\n----\ni: [["a","b","c"]]\nx: [[1,2,3]]\nz: [[22,44,99]]<\/code><\/pre>\n\n\n\n
Before ending this section, we must see how to filter a table or array:<\/p>\n\n\n\n
>>> t1.filter((pc.field('x') > 0) & (pc.field('x') < 3))\npyarrow.Table\ni: string\nx: int64\n----\ni: [["a","b"]]\nx: [[1,2]]<\/code><\/pre>\n\n\n\n
Easy, right? Especially if you\u2019ve been using pandas and numpy for years!<\/p>\n\n\n\n
3. Working with files<\/h2>\n\n\n\n
We\u2019ve already seen how we can read and write Parquet files. But let\u2019s check some other popular file types so that we have several options available.<\/p>\n\n\n\n
3.1. Apache ORC<\/h3>\n\n\n\n
Being very informal, Apache ORC can be understood as the equivalent of Arrow in the realm of file types (even though its origins have nothing to do with Arrow). Being more correct, it\u2019s an open source and columnar storage format. <\/p>\n\n\n\n
Reading and writing it is as follows:<\/p>\n\n\n\n
from pyarrow import orc\n# Write table\norc.write_table(t1, 't1.orc')\n# Read table\nt1 = orc.read_table('t1.orc')<\/code><\/pre>\n\n\n\n
As a side note, we could decide to compress the file while writing by using the \u201ccompression\u201d parameter.<\/p>\n\n\n\n
3.2. CSV<\/h3>\n\n\n\n
No secret here, pyarrow has the CSV module:<\/p>\n\n\n\n
from pyarrow import csv\n# Write CSV\ncsv.write_csv(t1, "t1.csv")\n# Read CSV\nt1 = csv.read_csv("t1.csv")\n\n# Write CSV compressed and without header\noptions = csv.WriteOptions(include_header=False)\nwith pa.CompressedOutputStream("t1.csv.gz", "gzip") as out:\n    csv.write_csv(t1, out, options)\n\n# Read compressed CSV and add custom header\nt1 = csv.read_csv("t1.csv.gz", read_options=csv.ReadOptions(\n    column_names=["i", "x"], skip_rows=1\n)]<\/code><\/pre>\n\n\n\n
3.2. JSON<\/h2>\n\n\n\n
Pyarrow allows JSON reading but not writing. It\u2019s pretty straightforward, let\u2019s see an example supposing we have our JSON data in \u201cdata.json\u201d:<\/p>\n\n\n\n
from pyarrow import json\n# Read json\nfn = "data.json"\ntable = json.read_json(fn)\n\n# We can now convert it to pandas if we want to\ndf = table.to_pandas()<\/code><\/pre>\n\n\n\n
Feather is a portable file format for storing Arrow tables or data frames (from languages like Python or R) that utilizes the Arrow IPC format<\/a> internally. So, contrary to Apache ORC, this one was indeed created early in the Arrow project.<\/p>\n\n\n\n
from pyarrow import feather\n# Write feather from pandas DF\nfeather.write_feather(df, "t1.feather")\n# Write feather from table, and compressed\nfeather.write_feather(t1, "t1.feather.lz4", compression="lz4")\n\n# Read feather into table\nt1 = feather.read_table("t1.feather")\n# Read feather into df\ndf = feather.read_feather("t1.feather")<\/code><\/pre>\n\n\n\n
4. Advanced Features<\/h2>\n\n\n\n
We just touched upon the most basic features and what the majority would need while working with Arrow. However, its amazingness doesn\u2019t end here, it\u2019s right where it starts.<\/p>\n\n\n\n
As this will be quite domain-specific and not useful for anyone (nor considered introductory) I\u2019ll just mention some of these features without using any code:<\/p>\n\n\n\n