Cfrgtkky

I have seen this question earlier here and I have took lessons from that. However I am not sure why I am getting an error when I feel it should work.

I want to create a new column in existing Spark DataFrame by some rules. Here is what I wrote. iris_spark is the data frame with a categorical variable iris_spark with three distinct categories.

from pyspark.sql import functions as F



iris_spark_df = iris_spark.withColumn(

    "Class", 

   F.when(iris_spark.iris_class == 'Iris-setosa', 0, F.when(iris_spark.iris_class == 'Iris-versicolor',1)).otherwise(2))

Throws the following error.

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-157-21818c7dc060> in <module>()

----> 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))



TypeError: when() takes exactly 2 arguments (3 given)





---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-157-21818c7dc060> in <module>()

----> 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))



TypeError: when() takes exactly 2 arguments (3 given)

Any idea why?

Correct structure is either:

(when(col("iris_class") == 'Iris-setosa', 0)

.when(col("iris_class") == 'Iris-versicolor', 1)

.otherwise(2))

which is equivalent to

CASE 

    WHEN (iris_class = 'Iris-setosa') THEN 0

    WHEN (iris_class = 'Iris-versicolor') THEN 1 

    ELSE 2

END

or:

(when(col("iris_class") == 'Iris-setosa', 0)

    .otherwise(when(col("iris_class") == 'Iris-versicolor', 1)

        .otherwise(2)))

which is equivalent to:

CASE WHEN (iris_class = 'Iris-setosa') THEN 0 

     ELSE CASE WHEN (iris_class = 'Iris-versicolor') THEN 1 

               ELSE 2 

          END 

END

with general syntax:

when(condition, value).when(...)

or

when(condition, value).otherwise(...)

You probably mixed up things with Hive IF conditional:

IF(condition, if-true, if-false)

which can be used only in raw SQL with Hive support.