Yarns about YARN：Migrating to MapReduce v2

?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
Container Container
Container
?
?
?
?
?
?
?
?
?
?
?
?
?
Container Container Container
Container Container Container
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?$ spark-submit pi.py –master yarn-client
?
?$ spark-submit --master yarn-client pi.py 1000
?
spark-submit [options] [app options]
?
?
$ spark-submit --master yarn-cluster pi.py 1000
Error: Cluster deploy mode is currently not
supported for python applications.
Run with --help for usage help or --verbose for
debug output
?
$ spark-submit --master yarn-client pi.py 1000
…
Pi is roughly 3.132290
15/02/11 09:41:34 INFO SparkUI: Stopped Spark web UI
at http://wendang.chazidian.com:4040
15/02/11 09:41:34 INFO DAGScheduler: Stopping
DAGScheduler
?
? Symptom:
? Spark Driver WARN Messages
14/12/08 17:11:08 WARN scheduler.TaskSetManager: Lost task 205.0 in
stage 2.0 (TID 352, http://wendang.chazidian.com: ExecutorLostFailure (executor
lost)
? NodeManager Logs
2014-12-08 17:10:32,860 WARN org.apache.hadoop.yarn.server.nodemanager.
containermanager.monitor.ContainersMonitorImpl: Container
[pid=26842,containerID=container_1418059756626_0010_01_000093_01]
is running beyond physical memory limits.
Current usage: 26.2 GB of 26 GB physical memory used;
27.1 GB of 54.6 GB virtual memory used. Killing container.
? Workaround:
?Increase
spark.yarn.[executor|driver].memoryOverhead
? Test for your specific use case. 1GB to 4GB
? Spark 1.2 / CDH 5.3 : Prefer RDDs that are cached locally in HDFS
? Spark 1.2 : Dynamically release unused resources via spark.
dynamicAllocation.enabled
? Only support via YARN currently.
? Spark Streaming save incoming data to a WAL (write-ahead log) on HDFS,
preventing any data loss on driver failure.
?
?
?
?
?
?
?
? spark.shuffle.consolidateFiles=true
? spark.yarn.executor.memoryOverhead
? spark.yarn.driver.memoryOverhead
? spark.shuffle.manager=SORT
? http://www.wend angxiazai.compress=true
? spark.serializer=org.apache.spark.serializer.KryoSerializer
YARN Mesos
Asks for resources Offers resources
Evolved into a resource manager Evolved into managing Hadoop
Written in Java Written in C++
Locality aware More customizable