Comparing two data frame

[brindasanth]

I have two data frames having same same schema, Is there way to compare the two data frames ? so that it provide the added , deleted and modified rows. It may take some single/group of Key columns and Ignore columns.

[Jolanrensen]

Hi!
We don't have such functionality at the moment, but it might be a handy addition.

Tracking additions, deletions, and modifications, similar to how git would do it, requires a special algorithm. I suppose Myer's Differencing Algorithm could help.

I just tried this algorithm via https://github.com/andrewbailey/Difference on two dataFrames (as List<DataRow<*>>) which correctly provides the remove/move/add operations that likely occurred between the two dataframes.

We could wrap a library like that in the future to introduce this behavior to DataFrame natively, but in the meantime, you could try that library as well :)

[brindasanth]

Thanks for your comments and adding in Backlog.

[devcrocod]

We can consider adding a new compare method that will allow us to compare two dataframes flexibly
Inspired by https://capitalone.github.io/datacompy/spark_usage.html