Can't initialize graph on Datastax using Spark
1
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
|
show 3 more comments
1
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 '18 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17
|
show 3 more comments
1
1
1
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
edited Nov 20 '18 at 16:30
Alex Ott
28.4k35273
28.4k35273
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
asked Nov 20 '18 at 15:52
BeginMohBeginMoh
6019
6019
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 '18 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17
|
show 3 more comments
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 '18 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
how do you submit job? via
dse spark-submit or you're using separate spark?– Alex Ott
Nov 20 '18 at 16:49
how do you submit job? via
dse spark-submit or you're using separate spark?– Alex Ott
Nov 20 '18 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17
|
show 3 more comments
1 Answer
1
active
oldest
votes
0
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53396768%2fcant-initialize-graph-on-datastax-using-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
add a comment |
0
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
add a comment |
0
0
0
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
answered Nov 21 '18 at 9:25
Alex OttAlex Ott
28.4k35273
28.4k35273
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
add a comment |
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 '18 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 '18 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 '18 at 13:39
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53396768%2fcant-initialize-graph-on-datastax-using-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 '18 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 '18 at 16:47
how do you submit job? via
dse spark-submitor you're using separate spark?– Alex Ott
Nov 20 '18 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 '18 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 '18 at 9:17