Spark UDF could not find implicit value for parameter num: Numeric[Nothing]
I am trying to write generic add method for which can take any argument type and return result of that type
def addExactUDF[T](x: T, y: T)(implicit num: Numeric[T]): T = {
import num._
x + y
}
def addExact(value1: Column, value2: Column, dataType: String): Column =
dataType match {
case "Int" => expr(s"addExactUDF(cast($value1 AS INT), cast($value2 AS INT))")
case "Double" => expr(s"addExactUDF(cast($value1 AS DOUBLE), cast($value2 AS DOUBLE))")
}
Now when I try to register the UDF
object FilterFunctionsUtil extends MathFunctionsNameSpace with StringFunctionsNameSpace {
lazy val registerAsSparkUdf: UserDefinedFunction = {
val sqlContext = SparkSessionFactory.getSparkSession(Map(), Nil).sqlContext
sqlContext.udf.register("addExactUDF", addExactUDF _)
sqlContext.udf.register("subtractExactUDF", subtractExactUDF _)
}
}
It throw me error
Error:(36, 44) could not find implicit value for parameter num: Numeric[Nothing]
sqlContext.udf.register("addExactUDF", addExactUDF _)
Error:(36, 44) not enough arguments for method addExactUDF: (implicit num: Numeric[Nothing])Nothing.
Unspecified value parameter num.
sqlContext.udf.register("addExactUDF", addExactUDF _)
how do I add the implicit to make this work
scala apache-spark
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
add a comment |
I am trying to write generic add method for which can take any argument type and return result of that type
def addExactUDF[T](x: T, y: T)(implicit num: Numeric[T]): T = {
import num._
x + y
}
def addExact(value1: Column, value2: Column, dataType: String): Column =
dataType match {
case "Int" => expr(s"addExactUDF(cast($value1 AS INT), cast($value2 AS INT))")
case "Double" => expr(s"addExactUDF(cast($value1 AS DOUBLE), cast($value2 AS DOUBLE))")
}
Now when I try to register the UDF
object FilterFunctionsUtil extends MathFunctionsNameSpace with StringFunctionsNameSpace {
lazy val registerAsSparkUdf: UserDefinedFunction = {
val sqlContext = SparkSessionFactory.getSparkSession(Map(), Nil).sqlContext
sqlContext.udf.register("addExactUDF", addExactUDF _)
sqlContext.udf.register("subtractExactUDF", subtractExactUDF _)
}
}
It throw me error
Error:(36, 44) could not find implicit value for parameter num: Numeric[Nothing]
sqlContext.udf.register("addExactUDF", addExactUDF _)
Error:(36, 44) not enough arguments for method addExactUDF: (implicit num: Numeric[Nothing])Nothing.
Unspecified value parameter num.
sqlContext.udf.register("addExactUDF", addExactUDF _)
how do I add the implicit to make this work
scala apache-spark
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
add a comment |
I am trying to write generic add method for which can take any argument type and return result of that type
def addExactUDF[T](x: T, y: T)(implicit num: Numeric[T]): T = {
import num._
x + y
}
def addExact(value1: Column, value2: Column, dataType: String): Column =
dataType match {
case "Int" => expr(s"addExactUDF(cast($value1 AS INT), cast($value2 AS INT))")
case "Double" => expr(s"addExactUDF(cast($value1 AS DOUBLE), cast($value2 AS DOUBLE))")
}
Now when I try to register the UDF
object FilterFunctionsUtil extends MathFunctionsNameSpace with StringFunctionsNameSpace {
lazy val registerAsSparkUdf: UserDefinedFunction = {
val sqlContext = SparkSessionFactory.getSparkSession(Map(), Nil).sqlContext
sqlContext.udf.register("addExactUDF", addExactUDF _)
sqlContext.udf.register("subtractExactUDF", subtractExactUDF _)
}
}
It throw me error
Error:(36, 44) could not find implicit value for parameter num: Numeric[Nothing]
sqlContext.udf.register("addExactUDF", addExactUDF _)
Error:(36, 44) not enough arguments for method addExactUDF: (implicit num: Numeric[Nothing])Nothing.
Unspecified value parameter num.
sqlContext.udf.register("addExactUDF", addExactUDF _)
how do I add the implicit to make this work
scala apache-spark
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
I am trying to write generic add method for which can take any argument type and return result of that type
def addExactUDF[T](x: T, y: T)(implicit num: Numeric[T]): T = {
import num._
x + y
}
def addExact(value1: Column, value2: Column, dataType: String): Column =
dataType match {
case "Int" => expr(s"addExactUDF(cast($value1 AS INT), cast($value2 AS INT))")
case "Double" => expr(s"addExactUDF(cast($value1 AS DOUBLE), cast($value2 AS DOUBLE))")
}
Now when I try to register the UDF
object FilterFunctionsUtil extends MathFunctionsNameSpace with StringFunctionsNameSpace {
lazy val registerAsSparkUdf: UserDefinedFunction = {
val sqlContext = SparkSessionFactory.getSparkSession(Map(), Nil).sqlContext
sqlContext.udf.register("addExactUDF", addExactUDF _)
sqlContext.udf.register("subtractExactUDF", subtractExactUDF _)
}
}
It throw me error
Error:(36, 44) could not find implicit value for parameter num: Numeric[Nothing]
sqlContext.udf.register("addExactUDF", addExactUDF _)
Error:(36, 44) not enough arguments for method addExactUDF: (implicit num: Numeric[Nothing])Nothing.
Unspecified value parameter num.
sqlContext.udf.register("addExactUDF", addExactUDF _)
how do I add the implicit to make this work
scala apache-spark
scala apache-spark
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
asked Nov 21 '18 at 4:08
FreemanFreeman
5461826
5461826
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Specify the implicit type while registering the UDF. you may need to register separate UDF for each type when required.
import sqlContext.implicits._
val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")
df.createOrReplaceTempView("tempTable")
sqlContext.cacheTable("tempTable")
sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)
sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)
Result
+---------------------------------------------+
|UDF:addExtractIntUDF(val1,val2) |
+---------------------------------------------+
|3 |
|7 |
+--------------------------------------------+
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53405132%2fspark-udf-could-not-find-implicit-value-for-parameter-num-numericnothing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Specify the implicit type while registering the UDF. you may need to register separate UDF for each type when required.
import sqlContext.implicits._
val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")
df.createOrReplaceTempView("tempTable")
sqlContext.cacheTable("tempTable")
sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)
sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)
Result
+---------------------------------------------+
|UDF:addExtractIntUDF(val1,val2) |
+---------------------------------------------+
|3 |
|7 |
+--------------------------------------------+
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
add a comment |
Specify the implicit type while registering the UDF. you may need to register separate UDF for each type when required.
import sqlContext.implicits._
val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")
df.createOrReplaceTempView("tempTable")
sqlContext.cacheTable("tempTable")
sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)
sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)
Result
+---------------------------------------------+
|UDF:addExtractIntUDF(val1,val2) |
+---------------------------------------------+
|3 |
|7 |
+--------------------------------------------+
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
add a comment |
Specify the implicit type while registering the UDF. you may need to register separate UDF for each type when required.
import sqlContext.implicits._
val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")
df.createOrReplaceTempView("tempTable")
sqlContext.cacheTable("tempTable")
sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)
sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)
Result
+---------------------------------------------+
|UDF:addExtractIntUDF(val1,val2) |
+---------------------------------------------+
|3 |
|7 |
+--------------------------------------------+
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
Specify the implicit type while registering the UDF. you may need to register separate UDF for each type when required.
import sqlContext.implicits._
val df = sc.parallelize(Seq((1,2),(3,4))).toDF("val1","val2")
df.createOrReplaceTempView("tempTable")
sqlContext.cacheTable("tempTable")
sqlContext.udf.register("addExtractIntUDF",addExtractUDF[Int] _)
sqlContext.sql("select addExtractIntUDF(val1,val2) from temptable").show(false)
Result
+---------------------------------------------+
|UDF:addExtractIntUDF(val1,val2) |
+---------------------------------------------+
|3 |
|7 |
+--------------------------------------------+
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
answered Nov 21 '18 at 8:02
Balaji ReddyBalaji Reddy
3,13831636
3,13831636
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53405132%2fspark-udf-could-not-find-implicit-value-for-parameter-num-numericnothing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
