Cfrgtkky

i have a pandas dataframe df with one column z filled with set values

i want to drop duplicated rows where 2 rows are considered duplicated version of one another when they have same column z values ( which are sets ).

import pandas as pd



lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) , ( 'b' , 'a' , { 'a' , 'b' } ) ]

lbls = [ 'x' , 'y' , 'z' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

Trying to drop duplicated rows based on column z values :

df.drop_duplicates( subset = 'z' , keep='first')

And i get the error message :

TypeError: unhashable type: 'set'

Is there a way to drop duplicated rows based on a unhashable typed column ?

It is true that a set is not hashable (it cannot be used as a key in a hashmap a.k.a a dictionary). So what you can do is to just convert the column to a type that is hashable - I would go for a tuple.

I made a new column that is just the "z" column you had, converted to tuples. Then you can use the same method you tried to, on the new column:

In [1] : import pandas as pd 

    ...:  

    ...: lnks = [ ( 'a' , 'b' , { 'a' , 'b' } ) , ( 'b' , 'c' , { 'b' , 'c' } ) 

    ...: , ( 'b' , 'a' , { 'a' , 'b' } ) ] 

    ...: lbls = [ 'x' , 'y' , 'z' ] 

    ...: df = pd.DataFrame.from_records( lnks , columns = lbls)                 



In [2]: df["z_tuple"] = df.z.apply(lambda x: tuple(x))                         



In [3]: df.drop_duplicates(subset="z_tuple", keep="first")                     

Out[3]: 

   x  y       z z_tuple

0  a  b  {b, a}  (b, a)

1  b  c  {c, b}  (c, b)

The apply method lets you apply a function to each item in a column, and then returns the values as a new column (a Pandas Series object). This lets you assign it back to the original DataFrame as a new column, as I did.

You can also remove the "z_tuple" column then if you no longer want it:

In [4] : df.drop("z_tuple", axis=1, inplace=True)                               



In [5] : df                                                                     

Out[5] : 

   x  y       z

0  a  b  {b, a}

1  b  c  {c, b}

2  b  a  {b, a}

I have to admit I did not mention the reason why I was trying to drop duplicated rows based on a column containing set values.
The reason is that the set { 'a' , 'b' } is the same as { 'b' , 'a' } so 2 apparently different rows are considered the same regarding the set column and are then deduplicated... but this is not possible because sets are unhashable ( like list )

Tuples are hashable but order of their elements matters... so when I build the tuples for each row i sort them :

import pandas as pd



lnks = [ ( 'a' , 'b' ) , ( 'b' , 'c' ) , ( 'b' , 'a' ) , ( 'a' , 'd' ) , ( 'd' , 'e' ) ]

lbls = [ 'x' , 'y' ]

df = pd.DataFrame.from_records( lnks , columns = lbls )

Building the tuple column (each tuple are sorted) :

df[ 'z' ] = df.apply( lambda d : tuple( sorted( [ d[ 'x' ]  , d[ 'y' ] ] ) ) , axis = 1 )

Droping duplicated rows (keeping first occurence) using the new tuple column :

df.drop_duplicates(subset="z", keep="first" , inplace = True )