Method Chaining in Pandas

A discussion of "method chaining" in pandas. Used for better readability, or harder debugging, depending on how you look at it.

When you work with pandas, you'll often perform multiple operations on a DataFrame. Some data cleaning and basic plotting for example, something like this:

import pandas as pd

df = pd.read_csv("my_data.csv")
df = df.drop("column_1", axis=1)
df = df.rename(columns={"column_2":"name", "column_3": "address"})
by_address = df.groupby("address")

That works fine, you're incrementally changing the DataFrame until you're ready for aggregation and plotting.

There is an alternative, which you might find more readable.

import pandas as pd

df = pd.read_csv("my_data.csv")
df.drop("column_1", axis=1)  \  
 .rename(columns={"column_2":"name", "column_3": "address"})  \  
 .groupby("address")["name"]  \  

Why can we keep chaining methods together like this?

In pandas, each of those functions returns a copy of the modified DataFrame, which is why we were setting it back to the df variable each time. By chaining methods together we're just calling the next method on the modified DataFrame until we're done.

Notice we have to break lines with a backslash character to allow the chain to go over multiple lines. You can avoid that by putting the entire chain in brackets:

    df.drop("column_1", axis=1)
        .rename(columns={"column_2":"name", "column_3": "address"})

Logically and computationally these examples are equivalent, so this is mostly just a stylistic consideration.



I'd argue the second method looks better, it actually reads left-to-right (or up-to-down I suppose) and you can understand logically what you're doing each time.

No Intermediate Variables

In the first example we had to either save back the modified DataFrame to the original df variable, or create a new one each time. This means you have to think about whether you want to store the DataFrame in each of its states and come up with descriptive names for them, and as we all know...

Avoids "inplace" Confusion

In my experience the fact that most DataFrame methods return a copy of the DataFrame is actually confusingly counterintuitive for pandas beginners. You either have to keep saving your DataFrame back to the same variable, or use the "inplace" keyword. Using method chaining means you only have to consider this problem once, i.e. set the final result to a variable, without ever accidentally throwing away any of your changes.


No Intermediate Variables

Not having access to the intermediate states of the method can also be a downside. If you want to reuse any of the intermediate steps in the process, you need to keep a copy of it so you might not want to use method chaining all the time.

There is a workaround. You could use the .pipe() method to call a custom function on your DataFrame, including creating a new variable. First, create a custom function:

def save_intermediate_df(df, variable_name):
    globals()[variable_name] = df
    return df

This function takes in a DataFrame and a variable name and creates a new global variable with the requested name. The variable becomes the current state of your DataFrame at that point in the chain.

You then use .pipe() to add your function to the chain:

    df.drop("column_1", axis=1)
        .rename(columns={"column_2":"name", "column_3": "address"})
        .pipe(save_intermediate_df, "df_2")

Now you have a global variable df_2, which represents the state of your DataFrame at that point in the chain, and which you can inspect. This is a slightly hacky way to add intermediate variables to a chain, but it gets the job done and skirts around this potential downside.

Debugging is Hard(er)

Debugging a problem in a long method chain is hard. If your chain consists of many intermediate steps and the final output is wrong, or you get an error message, it can be hard to retrace your steps to see what went wrong. If you had each command line by line, like in the first example, you could step through the code with a debugger or simply run the commands one at a time until you find the problem.

However, creating intermediate variables as shown above would greatly help the debugging process.

You Can Get Carried Away

You can take method chaining to extremes...

import pandas as pd

df = pd.read_csv("my_time_series.csv")

# take a deep breath...
        .drop("column_1", axis=1)
        .rename(columns={"column_2":"date", "column_3":"price"})
        .loc[df["price"] > 0, ["date", "price"]]

You could argue that's a bit much, but I'd also argue it's actually self-documenting analytical code.

Best of Both Worlds

There is a time and a place for method chaining.

If you don't care about intermediate steps, and just want a basic plot for example, it's a good option.

If you want to do complex operations that might need serious debugging, maybe it should be avoided (at least initially).

What I tend to do is avoid using it while I'm still exploring my data and the code isn't final, and refactor to use method chaining when I'm confident the code works. The idea is that it helps future readability, so I can better understand my code if I look back on it later.

This post was mostly inspired by the great Modern Pandas series. and the excellent Effective pandas book.

About David

I'm a freelance data scientist consultant and educator with an MSc. in Data Science and a background in software and web development. My previous roles have been a range of data science, software development, team management and software architecting jobs.

Related articles tagged Programming for Data Scientists

Method Chaining in Pandas

Programming for Data Scientists

A discussion of "method chaining" in pandas. Used for better readability, or harder debugging, depending on how you look at it.

SQL For Data Scientists

Programming for Data Scientists

SQL is a useful part of a data scientist's toolkit and it can feel like an intimidatingly big area to try and learn alongside all the other data science concepts. I want to present a few key concepts that are enough to get you up and running with SQL!

Turning Jupyter Notebooks into Reusable Scripts

Programming for Data Scientists

As part of my commitment to occasionally talk about "programming for data scientists", I want to share ideas that will facilitate this to help data scientists focus on important stuff. In this post I want to share some thoughts on how to make your Jupyter notebooks easier to "productionise".

Contact me

Please give me a brief idea of what you're looking for help with, and I'll get back to you very soon. If you're having trouble with the form below, you can also email me at

Twitter LinkedIn