BUG: `FutureWarning` on `concat`

[attack68]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd  # version 2.1.3
df1 = pd.DataFrame([[None, 1.0], [None, 2.0]])
df2 = pd.DataFrame([[3.0, None], [4.0, None]])
df3 = pd.concat([df1, df2], keys=["D1", "D2"])
df3.dtypes #  0:f64, 1:f64
df3
#        0    1
# D1 0  NaN  1.0
#    1  NaN  2.0
# D2 0  3.0  NaN
#    1  4.0  NaN

Issue Description

When moving to 2.1.0 a FutureWarning is created when concatenating DataFrames where columns in one may be empty.
The warning provided is:

FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

Expected Behavior

If you run the above code on 2.0.3 you will obtain exactly the same DataFrame, but without the Warning.

If you perform some analysis to exclude empty columns on 2.1.3 you achieve exactly the same DataFrame (columns re-ordered) but without the FutureWarning:

df3 = pd.concat([df1[[1]], df2[[0]]], keys=["D1", "D2"])

I cannot allow these FutureWarnings to propagate through to end users (pandas is internal working for my purposes), therefore I must either filter them out (which I dont want to add unneccessary code for) or perform checks to remove empty columns from DataFrames (which also adds code overhead, and actually can change the represenation of my output which I dont want).

Is there anything else that can be done - is this really a bug?

Installed Versions

Replace this line with the output of pd.show_versions()

[mroeschke]

cc @jbrockmendel

[jbrockmendel]

i think the 3.0 behavior here will retain the object dtypes from the input dataframes. The only way I can think to opt-in to the future behavior would be to implement a "future" option.

On your end, you could get the future behavior (and avoid the warning) by checking for object dtype and casting non-object entries to object.

[attack68]

Since the results seems to be same in all cases I opted for a simple short term solution

        with warnings.catch_warnings():
            # TODO: pandas 2.1.0 has a FutureWarning for concatenating DataFrames with Null entries
            warnings.filterwarnings("ignore", category=FutureWarning)
            _ = concat(dfs_filtered, keys=["leg1", "leg2"])

Feel free to close this if preferred.

[lithomas1]

This is closeable - unless we want to add that option?

[dcsaba89]

Does anyone know the workaround to get rid of this FutureWarning in 2.1.0?

I tried to exlude blank columns with adding

df1.dropna(axis=1, how='all', inplace=True)
df2.dropna(axis=1, how='all', inplace=True)

before concating, but it ends up with serious performance degradation (around 4x slower, determining if there is any blank column present makes it "only" 2.5 times slower).

[my-tien]

i think the 3.0 behavior here will retain the object dtypes from the input dataframes. The only way I can think to opt-in to the future behavior would be to implement a "future" option.

On your end, you could get the future behavior (and avoid the warning) by checking for object dtype and casting non-object entries to object.

In my case, the dataframes without all-NaN entries can have string or numerical types (unknown in advance) and those types should be retained after concatenation, so the change in 3.0 would be a problem for me.

It would complicate my code a lot if I had to search through all columns for the first non-empty column and determine the datatype, then go and apply it to corresponding all-NaN columns.

What is the best approach in this scenario?

Tbh, I don't understand what the purpose of this change is, as it breaks behavior a lot of people rely upon. Maybe a flag to retain the old behavior could reduce the friction when moving to 3.0?

[jbrockmendel]

@my-tien can you give an example?

[krishvk]

import pandas as pd

# Block with INT
df1 = pd.DataFrame({'A': [1], 'B': [4]})
df2 = pd.DataFrame({'A': [2], 'B': [None]})
print(len(pd.concat([df1, df2])))

# Block with FLOAT
df1 = pd.DataFrame({'A': [1], 'B': [4.0]})
df2 = pd.DataFrame({'A': [2], 'B': [None]})
print(len(pd.concat([df1, df2])))

Block with INT runs fine, while Block with FLOAT gives warning

Sample output

2
/home/kasula/lab/darpan/./testPoll.py:18: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  print(len(pd.concat([df1, df2])))
2

Is it expected?