DOC: User Guide Page on user-defined functions

[arthurlw]

• closes DOC: Write user guide page on apply/map/transform methods #61126
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[arthurlw]

Currently writing this, so I would appreciate any feedback on it!

[rhshadrach]

Thanks for the PR! I'm not opposed to a dedicated page on UDFs, but I am opposed to duplicating documentation that exists elsewhere in the user guide, as I think much of this does. Instead of e.g. examples of apply, I recommend linking to the appropriate section. This page can then focus on recommendations of when to use apply vs other methods.

[rhshadrach]

I think we should lead with Why _not_ User-Defined Functions. While performance is called out down below, I think the poor behavior of UDFs should be mentioned as well. Namely that pandas has no information on what a UDF is doing, and so has to infer (guess) at how to handle the result.

In particular, I think it should be mentioned that none of the examples on this page should be UDFs in practice.

[arthurlw]

Hi @rhshadrach thanks for the feedback! I agree with you and will push updates soon

[rhshadrach]

I think this is looking a lot better. Can we also link to https://pandas.pydata.org/pandas-docs/dev/user_guide/enhancingperf.html#numba-jit-compilation at the very bottom in a section titled something like "Improving Performance with UDFs".

[rhshadrach]

Can you also make a mention of resample, rolling, expanding, and ewm. Perhaps link to each section in the User Guide.

[rhshadrach]

Can we add the other objects to this note, it seems to me they all belong together.

Suggested change

	Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
	Some of these methods are can also be applied to groupby, resample, and various window objects. See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`, and :ref:`ewm()<window>` for details.

[rhshadrach]

nit: "change the data differently" sounds very close to mutating in a UDF, which we explicitly do not support. What do you think of "behave differently".

[arthurlw]

“Behave differently” sounds clearer and avoids implying mutation. I'll update it!

[rhshadrach]

I'm thinking we should remove groups of data here. DataFrame.apply that you're referencing doesn't operate on groups, and you mention groupby below.

[rhshadrach]

Can we add the other objects to this note, it seems to me they all belong together.

Suggested change

	Some of these methods are can also be applied to Groupby Objects. Refer to :ref:`groupby`.
	Some of these methods are can also be applied to groupby, resample, and various window objects. See :ref:`groupby`, :ref:`resample()<timeseries>`, :ref:`rolling()<window>`, :ref:`expanding()<window>`, and :ref:`ewm()<window>` for details.

[rhshadrach]

Things like .agg(["sum", "mean"]) aren't UDFs, so I don't think they should be mentioned here, and it could be make users think these types of usages are slow (they are not).

Suggested change

	When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
	functions across groups.
	When to use: Use :meth:`DataFrame.agg` for performing custom aggregations, where the operation returns a scalar value on each input.

[rhshadrach]

This isn't an example of a UDF. I really like your example of using linear regression - can we do that here? It's a bit unfortunate that groupby.transform does not allow operating on the entire group (only works column-by-column) here.

from sklearn.linear_model import LinearRegression

df = pd.DataFrame({
    'group': ['A', 'A', 'A', 'B', 'B', 'B'],
    'x': [1, 2, 3, 1, 2, 3],
    'y': [2, 4, 6, 1, 2, 1.5]
}).set_index("x")

# Function to fit a model to each group
def fit_model(group):
    x = group.index.to_frame()
    y = group
    model = LinearRegression()
    model.fit(x, y)
    pred = model.predict(x)
    return pred

result = df.groupby('group').transform(fit_model)

[datapythonista]

Excellent job here @arthurlw, thanks for taking care of this. I added a general comment about using examples to incrementally illustrate what it's explain here, and changing a bit the order of the sections.

Please let me know if it doesn't make sense or you have any comment. I'll review more in depth after the proposed changes are implemented or discussed. But in a first look, this is really nice.

[datapythonista]

I would move this before the groupby section. It feels more natural to me to explain first Series.apply and later explain groupby("col").apply.

[datapythonista]

Not sure if Sphinx is more flexible now, but this had to be the same exact length as the title before.

[datapythonista]

Suggested change

	Introduction to User-Defined Functions
	User-Defined Functions (UDFs)

This will be what will be shown in the index too, so better to be concise. Also, better for consistency to remove the Introduction to, which we could have in every other user guide too.

[datapythonista]

Maybe just personal opinion, but to me it makes more sense to explain what UDFs are in pandas before explaining when not to use them. This order seems reasonable assuming users already know what pandas udfs are in practice, but I'd personally prefer not to assume it in the user guide for UDFs.

In my opinion, after the previous introduction which is great, I'd show a very simple example so we make sure users reading this understand the very basics.

Something like:

def add_one(x):
    return x + 1

my_series = pd.Series([1, 2, 3])

my_series.map(add_one)

Building on top of this, like then showing the same with a DataFrame, at some point showing UDFs that receive the whole column with .apply... should help make sure users are following and understanding all the information provided here.