Is Spark’s map and reduce operations are different to Hadoop Map Reduce, If yes then how?
up vote
0
down vote
favorite
I was going through the spark document and found the below line
Hadoop MapReduce and does not directly relate to Spark’s map and reduce operations
1.Could you please help me to understand how the spark map reduce is different to the hadoop map reduce?
- How the RDD works in spark? Is it always converting the code to mapreduce like hive?
apache-spark pyspark hadoop2
asked Nov 14 at 7:52
ram
296
add a comment |
up vote
0
down vote
favorite
I was going through the spark document and found the below line
Hadoop MapReduce and does not directly relate to Spark’s map and reduce operations
1.Could you please help me to understand how the spark map reduce is different to the hadoop map reduce?
- How the RDD works in spark? Is it always converting the code to mapreduce like hive?
apache-spark pyspark hadoop2
asked Nov 14 at 7:52
ram
296
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I was going through the spark document and found the below line
Hadoop MapReduce and does not directly relate to Spark’s map and reduce operations
1.Could you please help me to understand how the spark map reduce is different to the hadoop map reduce?
- How the RDD works in spark? Is it always converting the code to mapreduce like hive?
apache-spark pyspark hadoop2
asked Nov 14 at 7:52
ram
296
I was going through the spark document and found the below line
Hadoop MapReduce and does not directly relate to Spark’s map and reduce operations
1.Could you please help me to understand how the spark map reduce is different to the hadoop map reduce?
- How the RDD works in spark? Is it always converting the code to mapreduce like hive?
apache-spark pyspark hadoop2
apache-spark pyspark hadoop2
asked Nov 14 at 7:52
ram
296
asked Nov 14 at 7:52
ram
296
asked Nov 14 at 7:52
ram
296
asked Nov 14 at 7:52
ram
296
asked Nov 14 at 7:52
ram
296
296
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
There are lots of resources available on the web for illustrating the difference between Hadoop MapReduce and Spark MapReduce which you can go through but still, below I am pointing key difference so that you can get an idea :
- Apache Spark is a framework for real-time data analytics in a
distributed computing environment. It executes in-memory
computations to increase the speed of data processing. It is faster
for processing large-scale data as it exploits in-memory
computations and other optimizations. Therefore, it requires high
processing power. while Hadoop MapReduce has to read from and write
to a disk. As a result, the speed of processing differs
significantly – Spark may be up to 100 times faster. However, the
volume of data processed also differs: Hadoop MapReduce is able to
work with far larger data sets than Spark.
Tasks Hadoop MapReduce is good for :
- Linear processing of huge data sets.
- Economical solution, if no immediate results are expected
Tasks Spark is good for:
- Fast data processing
- Iterative processing
- Near real-time processing etc.
Now lets jump to your second question Resilient Distributed Dataset (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
So in Spark two operations are performed on RDDs (Transformations, Actions)
the logic doesnt get executed unless you perform Action operation that's why it is called lazy evalutaion
answered Nov 14 at 18:12
VIN
14111
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
There are lots of resources available on the web for illustrating the difference between Hadoop MapReduce and Spark MapReduce which you can go through but still, below I am pointing key difference so that you can get an idea :
- Apache Spark is a framework for real-time data analytics in a
distributed computing environment. It executes in-memory
computations to increase the speed of data processing. It is faster
for processing large-scale data as it exploits in-memory
computations and other optimizations. Therefore, it requires high
processing power. while Hadoop MapReduce has to read from and write
to a disk. As a result, the speed of processing differs
significantly – Spark may be up to 100 times faster. However, the
volume of data processed also differs: Hadoop MapReduce is able to
work with far larger data sets than Spark.
Tasks Hadoop MapReduce is good for :
- Linear processing of huge data sets.
- Economical solution, if no immediate results are expected
Tasks Spark is good for:
- Fast data processing
- Iterative processing
- Near real-time processing etc.
Now lets jump to your second question Resilient Distributed Dataset (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
So in Spark two operations are performed on RDDs (Transformations, Actions)
the logic doesnt get executed unless you perform Action operation that's why it is called lazy evalutaion
answered Nov 14 at 18:12
VIN
14111
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
add a comment |
up vote
0
down vote
There are lots of resources available on the web for illustrating the difference between Hadoop MapReduce and Spark MapReduce which you can go through but still, below I am pointing key difference so that you can get an idea :
- Apache Spark is a framework for real-time data analytics in a
distributed computing environment. It executes in-memory
computations to increase the speed of data processing. It is faster
for processing large-scale data as it exploits in-memory
computations and other optimizations. Therefore, it requires high
processing power. while Hadoop MapReduce has to read from and write
to a disk. As a result, the speed of processing differs
significantly – Spark may be up to 100 times faster. However, the
volume of data processed also differs: Hadoop MapReduce is able to
work with far larger data sets than Spark.
Tasks Hadoop MapReduce is good for :
- Linear processing of huge data sets.
- Economical solution, if no immediate results are expected
Tasks Spark is good for:
- Fast data processing
- Iterative processing
- Near real-time processing etc.
Now lets jump to your second question Resilient Distributed Dataset (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
So in Spark two operations are performed on RDDs (Transformations, Actions)
the logic doesnt get executed unless you perform Action operation that's why it is called lazy evalutaion
answered Nov 14 at 18:12
VIN
14111
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
add a comment |
up vote
0
down vote
up vote
0
down vote
There are lots of resources available on the web for illustrating the difference between Hadoop MapReduce and Spark MapReduce which you can go through but still, below I am pointing key difference so that you can get an idea :
- Apache Spark is a framework for real-time data analytics in a
distributed computing environment. It executes in-memory
computations to increase the speed of data processing. It is faster
for processing large-scale data as it exploits in-memory
computations and other optimizations. Therefore, it requires high
processing power. while Hadoop MapReduce has to read from and write
to a disk. As a result, the speed of processing differs
significantly – Spark may be up to 100 times faster. However, the
volume of data processed also differs: Hadoop MapReduce is able to
work with far larger data sets than Spark.
Tasks Hadoop MapReduce is good for :
- Linear processing of huge data sets.
- Economical solution, if no immediate results are expected
Tasks Spark is good for:
- Fast data processing
- Iterative processing
- Near real-time processing etc.
Now lets jump to your second question Resilient Distributed Dataset (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
So in Spark two operations are performed on RDDs (Transformations, Actions)
the logic doesnt get executed unless you perform Action operation that's why it is called lazy evalutaion
answered Nov 14 at 18:12
VIN
14111
There are lots of resources available on the web for illustrating the difference between Hadoop MapReduce and Spark MapReduce which you can go through but still, below I am pointing key difference so that you can get an idea :
- Apache Spark is a framework for real-time data analytics in a
distributed computing environment. It executes in-memory
computations to increase the speed of data processing. It is faster
for processing large-scale data as it exploits in-memory
computations and other optimizations. Therefore, it requires high
processing power. while Hadoop MapReduce has to read from and write
to a disk. As a result, the speed of processing differs
significantly – Spark may be up to 100 times faster. However, the
volume of data processed also differs: Hadoop MapReduce is able to
work with far larger data sets than Spark.
Tasks Hadoop MapReduce is good for :
- Linear processing of huge data sets.
- Economical solution, if no immediate results are expected
Tasks Spark is good for:
- Fast data processing
- Iterative processing
- Near real-time processing etc.
Now lets jump to your second question Resilient Distributed Dataset (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
So in Spark two operations are performed on RDDs (Transformations, Actions)
the logic doesnt get executed unless you perform Action operation that's why it is called lazy evalutaion
answered Nov 14 at 18:12
VIN
14111
answered Nov 14 at 18:12
VIN
14111
answered Nov 14 at 18:12
VIN
14111
answered Nov 14 at 18:12
VIN
14111
14111
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
add a comment |
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
This has been answered couple of times. You can get a lot of answers and detailed explanation on stackoverflow itself. Try to search
– vikrant rana
Nov 15 at 1:05
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53295339%2fis-spark-s-map-and-reduce-operations-are-different-to-hadoop-map-reduce-if-yes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
