Unsupported literal type class scala.runtime.BoxedUnit
0
I am trying to filter a column of a dataframe read from oracle as below
import org.apache.spark.sql.functions.{col, lit, when}
val df0 = df_org.filter(col("fiscal_year").isNotNull())
When I do it I am getting below error:
java.lang.RuntimeException: Unsupported literal type class scala.runtime.BoxedUnit ()
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:77)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at scala.util.Try.getOrElse(Try.scala:79)
at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:162)
at org.apache.spark.sql.functions$.typedLit(functions.scala:113)
at org.apache.spark.sql.functions$.lit(functions.scala:96)
at org.apache.spark.sql.Column.apply(Column.scala:212)
at com.snp.processors.BenchmarkModelValsProcessor2.process(BenchmarkModelValsProcessor2.scala:80)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at scala.Option.map(Option.scala:146)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:138)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:135)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at com.sp.MigrationDriver$.main(MigrationDriver.scala:135)
at com.sp.MigrationDriver.main(MigrationDriver.scala)
Any idea what am I doing wrong here and how to fix this?
scala apache-spark-sql datastax databricks
edited Nov 19 '18 at 13:29
hasumedic
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
|
show 5 more comments
0
I am trying to filter a column of a dataframe read from oracle as below
import org.apache.spark.sql.functions.{col, lit, when}
val df0 = df_org.filter(col("fiscal_year").isNotNull())
When I do it I am getting below error:
java.lang.RuntimeException: Unsupported literal type class scala.runtime.BoxedUnit ()
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:77)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at scala.util.Try.getOrElse(Try.scala:79)
at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:162)
at org.apache.spark.sql.functions$.typedLit(functions.scala:113)
at org.apache.spark.sql.functions$.lit(functions.scala:96)
at org.apache.spark.sql.Column.apply(Column.scala:212)
at com.snp.processors.BenchmarkModelValsProcessor2.process(BenchmarkModelValsProcessor2.scala:80)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at scala.Option.map(Option.scala:146)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:138)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:135)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at com.sp.MigrationDriver$.main(MigrationDriver.scala:135)
at com.sp.MigrationDriver.main(MigrationDriver.scala)
Any idea what am I doing wrong here and how to fix this?
scala apache-spark-sql datastax databricks
edited Nov 19 '18 at 13:29
hasumedic
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
1
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22
|
show 5 more comments
0
0
0
I am trying to filter a column of a dataframe read from oracle as below
import org.apache.spark.sql.functions.{col, lit, when}
val df0 = df_org.filter(col("fiscal_year").isNotNull())
When I do it I am getting below error:
java.lang.RuntimeException: Unsupported literal type class scala.runtime.BoxedUnit ()
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:77)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at scala.util.Try.getOrElse(Try.scala:79)
at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:162)
at org.apache.spark.sql.functions$.typedLit(functions.scala:113)
at org.apache.spark.sql.functions$.lit(functions.scala:96)
at org.apache.spark.sql.Column.apply(Column.scala:212)
at com.snp.processors.BenchmarkModelValsProcessor2.process(BenchmarkModelValsProcessor2.scala:80)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at scala.Option.map(Option.scala:146)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:138)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:135)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at com.sp.MigrationDriver$.main(MigrationDriver.scala:135)
at com.sp.MigrationDriver.main(MigrationDriver.scala)
Any idea what am I doing wrong here and how to fix this?
scala apache-spark-sql datastax databricks
edited Nov 19 '18 at 13:29
hasumedic
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
I am trying to filter a column of a dataframe read from oracle as below
import org.apache.spark.sql.functions.{col, lit, when}
val df0 = df_org.filter(col("fiscal_year").isNotNull())
When I do it I am getting below error:
java.lang.RuntimeException: Unsupported literal type class scala.runtime.BoxedUnit ()
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:77)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at org.apache.spark.sql.catalyst.expressions.Literal$$anonfun$create$2.apply(literals.scala:163)
at scala.util.Try.getOrElse(Try.scala:79)
at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:162)
at org.apache.spark.sql.functions$.typedLit(functions.scala:113)
at org.apache.spark.sql.functions$.lit(functions.scala:96)
at org.apache.spark.sql.Column.apply(Column.scala:212)
at com.snp.processors.BenchmarkModelValsProcessor2.process(BenchmarkModelValsProcessor2.scala:80)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.snp.utils.Utils$$anonfun$getAllDefinedProcessors$1.apply(Utils.scala:30)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at com.sp.MigrationDriver$$anonfun$main$6$$anonfun$apply$2.apply(MigrationDriver.scala:140)
at scala.Option.map(Option.scala:146)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:138)
at com.sp.MigrationDriver$$anonfun$main$6.apply(MigrationDriver.scala:135)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at com.sp.MigrationDriver$.main(MigrationDriver.scala:135)
at com.sp.MigrationDriver.main(MigrationDriver.scala)
Any idea what am I doing wrong here and how to fix this?
scala apache-spark-sql datastax databricks
scala apache-spark-sql datastax databricks
edited Nov 19 '18 at 13:29
hasumedic
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
edited Nov 19 '18 at 13:29
hasumedic
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
edited Nov 19 '18 at 13:29
hasumedic
1,981715
edited Nov 19 '18 at 13:29
hasumedic
1,981715
edited Nov 19 '18 at 13:29
hasumedic
1,981715
1,981715
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
asked Nov 19 '18 at 12:37
ShyamShyam
3001416
3001416
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
1
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22
|
show 5 more comments
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
1
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
1
1
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22
|
show 5 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53374838%2funsupported-literal-type-class-scala-runtime-boxedunit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53374838%2funsupported-literal-type-class-scala-runtime-boxedunit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
please add more information about versions of Spark, and Spark Cassandra connector...
– Alex Ott
Nov 19 '18 at 13:19
@AlexOtt , sir Here are the version details : scala - 2.11 spark - 2.3.1 cassandra - 3.11.1
– Shyam
Nov 19 '18 at 13:38
and spark-cassandra-connector version?
– Alex Ott
Nov 19 '18 at 13:58
what do you mean by "trying to filter a column of a dataframe"? can you elaborate that?
– Ramesh Maharjan
Nov 19 '18 at 15:08
1
check this stackoverflow.com/questions/39727742/… for filtering and you can check my answer too stackoverflow.com/questions/50478512/…
– Ramesh Maharjan
Nov 20 '18 at 4:22