狠狠撸

狠狠撸Share a Scribd company logo

YARN overview

?
?
J
JEONGPHIL HAN

Apache Hadoop? YARN? ?? ??? ?????. ?? ?? ??? ??? ??? ???????.

1 of 34
YARN Architecture & Process ??? : ??? filfal2001@gmail.com ??? : 2017.12.06 On Hortonworks Data Platform
INDEX 01 02 03 04 05 06 YARN Architecture YARN Process YARN CLI Command YARN Parameter YARN High Availability YARN Capacity Scheduler
? YARN (Yet Another Resource Negotiator) – ?? scheduling ? Hadoop ????? ??? ??? ?? framework – Hadoop? ???? ? ????, HDFS? ?? ?? ???? ????? YARN? ??? ?? ?? ??? ??? ??? – Hadoop 1.0?? ?????? ???? MapReduce? ??? ???? ?? ???? ? 4000Node+ ?? ?? ????? ??(?? 10,000Node) ? ?? ???? ?? ??? ?? ? MapReduce ??? ?? ??? ???? ??(Hive, Pig, Spark ?) – YARN? ???? ??? MapReduce? Batch Processing ? ? ??? Batch, Interactive, Realtime, Streaming? ??? ?? ??? ?? ??? ?? ??? ? ?? ? 3 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) YARN??? 1. YARN Architecture
? YARN ???? MapReduce? JobTracker? ?? ???? ??? ??. ? ??? Hadoop? ?? ???? ????? ?? ?? ? ?? ???? ??? ????. ? ? ??? ???? ?? ?? ?? YARN?? YARN? ?? MapReduce ??? ????? ???? ???? ??? ??? ? ?? ?????? ?? ????? ??? ??? ? ?? MapReduce ??? ?????? ????? ???? ?? ??? ?? ???? ??? ?? ??? MapReduce? ???? ?? ?? ?? ? ???? ??? ??? ?? ? Hortonworks Data Platform ???? ?? Application? YARN? ??? ???(YARN Ready) 4 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) YARN??? 1. YARN Architecture
5 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) ?? – MRv1? YARN? ?? 1. YARN Architecture MRv1 YARN(MRv2) ?? ????? ????? Job ?? JobTracker Resource Manager ??? ?? ? ???? - Node Manager(N?) Job ?? ?? TaskTracker (N?, Task ?) Application Master (N?, Task ?) Task ?? HDFS HDFS ??? ?? ? ??
6 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
YARN? ?? ????? Resource Manager? Node Manager ? Resource Manager – ???? ??? ???? ???? ??, ??? Scheduler, Application Manager, Resource Tracker?? ????? ?? ? Scheduler : Node Manager?? ?? ??? ???? ??? ????? ????. Scheduler ? ????? ??? ????? ???? ?? ???, ???? ???? ??? ?? ? Application Manager : Application Master? ????, Application Master? ??? ?? ? Resource Tracker : Container? ?? ?? ??? ???? ???, Application Master ? ?? ?? ??, ??? Node Manager? ?? ??? ?? ? ??? ??? ?? ?? ??? ?? ?? ?? ??? ??? ??. 7 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
YARN? ?? ????? Resource Manager? Node Manager ? Node Manager – Container? ??? ???? ???? ??, ?? ??? Resource Manager?? ??? ??? ??. Application Master ? Container? ?? ?? ??. ? Application Master : ??? ????? ?? ??? ??? ????, Scheduler ?? ? ??? Container? ?? ??, ???? ?? ??? ?????? ????. ? Container : CPU, ???(Disk), ???(Memory) ?? ?? ???? ????. ? ?? ? ??? ??(Graph processing)? MPI? ?? ?? ?? ????? ????? ? ?? ??. ?? ??(job)? ?? ?? ?? ???? ?????, ? ???? ?? ? Container ??? ??? ??. ??? ??? ??? Application Master? ??? ?, ?? ??? Resource Manager? ????. 8 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
9 YARN(Yet Another Resource Negotiator)? ?? ??? ?? ?????. 1) YARN? Job ?? ?? 2. YARN Process
10 YARN(Yet Another Resource Negotiator)? ?? ??? ?? ?????. 1) YARN? Application Job ?? ?? 2. YARN Process 1. Initiate Job – Client?? Application? ????. 2. Request New Application to Resource Manager – Resource Manager? ??? App ?? 3. Copy Jars & Configuration to HDFS – App? ?? ???? ??? ???? HDFS? ????. 4. Submit Job to Resource Manager – Resource Manager?? ?? Application Job? ???? 5. Start Application Master – Resource Manager? ??? Node Manager? Application Master? ??? ? ??? Client?? ??? 6. Initialize Job – Resource Manager? ?? Application? Job? ??? Node Manager? ??? 7. Start Container – Resource Manager? ??? Node Manager?? Container? ??? ? Application Master?? ?? 8. Launch & Run – Application Master? Container? ???? Application? Job? ?? * Application - Spark, Hive?? Application? ?? * Job - ??? Application?? ???? Job?? ??(Hive Query, Spark Job ??)
? Hortonworks Data Platform? Ambari ?? ?? ? Zookeeper ?? 3EA ??? ?? ?? ① Ambari Web?? Services > YARN > Summary ? ?? ② Service Actions ???? Enable Resource Manager HA ? ??? ResourceManager HA wizard? ??? 11 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
③ Select Host ????? Standby ResourceManager? ??? Host ?? ? Next 12 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
④ Review Page? ??? YARN? ?? ????? ???? Next 13 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
⑤ Resource Manager HA ??? ??? Parameter ?? ??? ???? Next ⑥ HA ?? ?? ?? ?? 14 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
15 YARN(Yet Another Resource Negotiator)? HA(High Availability) ??? ??? ?????. 2) ResourceManager High Availability Works 3. YARN High Availability
? YARN Application Check Usage : yarn application 16 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 1) Yarn Command - applicaiton 4. YARN CLI Command COMMAND_OPTIONS Description -appStates <States> ?? ?? Application ??? ?? (ALL, NEW, NEW_SAVIN G, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILE D, KILLED) -appTypes <Types> -list ??? ?? ?? ???? ?? ??? Application ?? ? ?? -list ?? Application ??? ?? -kill <ApplicationId> Application? ?? ?? -status <ApplicationId> Application ?? ?? ??
? YARN Container Check Usage : yarn container 17 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 2) Yarn Command – Container & Queue 4. YARN CLI Command COMMAND_OPTIONS Description -list <Application Attempt Id> Application? ??? Container?? ??? ?? -status <ContainerId> Container? ?? ?? ?? ? YARN queue Check Usage : yarn queue COMMAND_OPTIONS Description -status <QueueName> Queue? ?? ?? ??
? YARN log Check Usage : yarn logs 18 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 3) Yarn Command - logs 4. YARN CLI Command COMMAND_OPTIONS Description -applicationId <application ID> ?? Application? ?? ?? -appOwner <AppOwner> Application Owner? ??? ??? ??, ???? ? ? ?? ?? ?????? ?? ?? -containerId <ContainerId> Container Id? ?? Container? ??? ?? -nodeAddress <NodeAddress> Nodemanager? Address? ?? ?? Nodemanager ? ??? ?? (Address:port)
? YARN set log level Usage : yarn daemonlog 19 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 4) Yarn Command – Set log level 4. YARN CLI Command COMMAND_OPTIONS Description -getlevel <host:httpport> <classname> YARN ??? log level ??? ?? ??. ??? ??? <host:httpport > ??? ???? ????? job? classname? ????. -setlevel <host:httpport> <classname> <level> YARN ??? log level? ????.
? YARN node Check Usage : yarn node 20 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 5) Yarn Command - node 4. YARN CLI Command COMMAND_OPTIONS Description -all ?? ?? ?? ?? -list ?? ???? ?? ?? ?? –states ??? ???? ????? ???? ???? ? ??. -states <States> Works with -list to filter nodes based on input com ma-separated list of node states. -status <NodeId> ?? ??? ???? ??
? Resource ?? Parameter (yarn-site.xml) 21 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (yarn-site.xml) 5. YARN Parameter Parameter Name Description yarn.nodemanager.resource.memory-mb ?? ???? ???? ??? ??? ? ?? ??? ??? yarn.scheduler.minimum-allocation-vcores ????? ?? ? ? ?? ?? Vcore ?? yarn.scheduler.maximum-allocation-vcores ????? ?? ? ? ?? ?? Vcore ?? yarn.scheduler.minimum-allocation-mb ????? ?? ? ? ?? ?? memory ?? yarn.scheduler.maximum-allocation-mb ????? ?? ? ? ?? ?? memory ?? yarn.scheduler.increment-allocation-vcores ????? ?? ?? ? ? ?? Vcore ? yarn.scheduler.increment-allocation-mb ????? ?? ?? ? ? memory ?? yarn.nodemanager.pmem-check-enabled ???? ??? ?? ? physical memory ?? ?? yarn.nodemanager.vmem-check-enabled ???? ??? ?? ? virtual memory ?? ?? yarn.nodemanager.vmem-pmem-ratio ???? ?? ??? ?? ??, default? pmem ???? 2.1? Virtual Memory ?? ??? ?? JIRA ?? ??? https://issues.apache.org/jira/browse/YARN-4714 ????? VMem Check ??? Disable ??? ?? ??? ?? ?? ??? ?? Test ?? ?
? Resource ?? Parameter Check List (yarn-site.xml) 22 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (yarn-site.xml) 5. YARN Parameter yarn.scheduler.maximum-allocation-vcores >= yarn.scheduler.minimum-allocation-vcores yarn.scheduler.maximum-allocation-mb >= yarn.scheduler.minimum-allocation-mb yarn.scheduler.minimum-allocation-vcores >= 0 yarn.scheduler.minimum-allocation-vcores <= HostsVCores yarn.scheduler.maximum-allocation-vcores >= 1 yarn.scheduler.maximum-allocation-vcores <= HostsVcores yarn.scheduler.minimum-allocation-mb < 1024 MB YARN ??? ?? ?? ?? Excel ??(Cloudera?? ??) https://www.dropbox.com/s/38reez543rh9xsv/yarn-tuning-guide.xlsx?dl=0
? Resource ?? Parameter (mapred-site.xml) 23 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (mapred-site.xml) 5. YARN Parameter Parameter Name Description yarn.app.mapreduce.am.resource.cpu-vcores MR Application Master? ?? ?? Vcore ?? yarn.app.mapreduce.am.resource.mb MR Application Master? ?? ?? memory ?? mapreduce.map.cpu.vcores Map ??? ???? Vcore ?? mapreduce.map.memory.mb Map ??? ???? memory ?? mapreduce.map.java.opts.max.heap Mapper? JAVA HEAP SIZE mapreduce.reduce.cpu.vcores Reduce ??? ???? Vcore ?? mapreduce.reduce.memory.mb Reduce ??? ???? memory ?? mapreduce.reduce.java.opts Reducer? JAVA HEAP SIZE mapreduce.task.io.sort.mb Split/Sort ??? ?? ?? ???
? Resource ?? Parameter Check List (mapred-site.xml) 24 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (mapred-site.xml) 5. YARN Parameter 1) Application Master ?? ??? yarn.app.mapreduce.am.resource.cpu-vcores >= yarn.scheduler.minimum-allocation-vcores yarn.app.mapreduce.am.resource.cpu-vcores <= yarn-scheduler.maximum-allocation-vcores yarn.app.mapreduce.am.resource.mb >= yarn.scheduler.minimum-allocation-mb yarn.app.mapreduce.am.resource.mb <= yarn.scheduler.maximum-allocation-mb yarn.app.mapreduce.am.resource.mb = ApplicationMaster Java Heap (must be close) 2) Map Task ?? ??? mapreduce.map.cpu.vcores >= yarn.scheduler.minimum-allocation-vcores mapreduce.map.cpu.vcores <= yarn-scheduler.maximum-allocation-vcores mapreduce.map.cpu.memory.mb >= yarn.scheduler.minimum-allocation-mb mapreduce.map.cpu.memory.mb <= yarn.scheduler.maximum-allocation-mb mapreduce.map.memory.mb = mapreduce.map.java.opts.max.heap (must be close) mapreduce.task.io.sort.mb << mapreduce.map.java.opts.max.heap 3) Reduce Task ?? ??? mapreduce.reduce.cpu.vcores >= yarn.scheduler.minimum-allocation-vcores mapreduce.reduce.cpu.vcores <= yarn-scheduler.maximum-allocation-vcores mapreduce.reduce.cpu.memory.mb >= yarn.scheduler.minimum-allocation-mb mapreduce.reduce.cpu.memory.mb <= yarn.scheduler.maximum-allocation-mb mapreduce.reduce.java.opts = mapreduce.reduce.memory.mb (must be close)
? YARN? Capacity Scheduler? ?????? Hadoop ??????? ????? ????? ???? ???? ?????? ?? ? ? ?? Capacity? ??? ?? ??? ??? ?? ex) OLAP Hive? 70% ??, ??? Spark 30% ?? ? ?? ????? ?? ?? ?? ???? ??? ????? ??? 25 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 1) YARN Capacity Scheduler Description 6. YARN Capacity Scheduler
? Ambari?? ???? YARN Queue Manager View, YARN Capacity Scheduler? Ambari ??? UI? ??? ? ??? ???. 26 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari 6. YARN Capacity Scheduler
? Queue ?? ?? ① YARN Queue Manager view ????? Add Queue ??? ????. 27 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
? Queue ?? ?? ② ??? Queue??? ??? ? ?? ??? ????. 28 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
? Queue ?? ?? ③ ???? ??? Queue? ???? ?? Queue? ?? ?? ??? ????. 29 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
? Queue ?? ?? ④ ??? ?? Queue? ????? ??? ????. 30 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
? Queue ?? ?? ① Capacity 31 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description Capacity Queue? ??? ? ?? ???? ???? ??, ?? Queue? ?? ?? Queue ???? ??? Max Capacity Queue?? ??? ? ?? ???? ???? ?? ??, ? ?? ? ? ?? ??? ? ?? ??? idle ??? ??? ? ?? ? Enable Node Labels Queue? Node Label? ????? ?? ?/?
? Queue ?? ?? ② Access Control and Status 32 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description Administer Queue ?? ??? ? ??? ??? ?? ?? Submit Applications Application? ?? ??? ? ??? ??? ? ??? ??
? Queue ?? ?? ③ Resources 33 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description User Limit Factor 1? ??? ?? ?? ???? Queue? ???? ??? ?? ?? 2? ??? ?? ?? ?? ???? Queue ??? ??? ??? 0.5? ??? ?? ???? Queue? ???? ??? ???? ??? ? ?? Minimum User Limit ????? ?? ?? ???? ?? ??? ?? Maximum Applications ? ??? ???? Scheduler Maximum Applications setting? ?? Maximum AM Resource ? ??? ???? Scheduler Maximum AM Resource setting? ?? Priority ? ?? ? ?? ???? ?? ??? ?? Queue? ? ?? ???? ???? ????
? https://docs.hortonworks.com/HDPDocuments/ Ambari-2.6.0.0/bk_ambari- views/content/ch_using_yarn_queue_manager_v iew.html ? http://www.popit.kr/what-is-hadoop-yarn/ ? http://blog.skcc.com/1883 ? https://hadoop.apache.org/docs/r2.7.1/hadoop- yarn/hadoop-yarn-site/YarnCommands.html 35 ???? ????

More Related Content

YARN overview

  • 1. YARN Architecture & Process ??? : ??? filfal2001@gmail.com ??? : 2017.12.06 On Hortonworks Data Platform
  • 2. INDEX 01 02 03 04 05 06 YARN Architecture YARN Process YARN CLI Command YARN Parameter YARN High Availability YARN Capacity Scheduler
  • 3. ? YARN (Yet Another Resource Negotiator) – ?? scheduling ? Hadoop ????? ??? ??? ?? framework – Hadoop? ???? ? ????, HDFS? ?? ?? ???? ????? YARN? ??? ?? ?? ??? ??? ??? – Hadoop 1.0?? ?????? ???? MapReduce? ??? ???? ?? ???? ? 4000Node+ ?? ?? ????? ??(?? 10,000Node) ? ?? ???? ?? ??? ?? ? MapReduce ??? ?? ??? ???? ??(Hive, Pig, Spark ?) – YARN? ???? ??? MapReduce? Batch Processing ? ? ??? Batch, Interactive, Realtime, Streaming? ??? ?? ??? ?? ??? ?? ??? ? ?? ? 3 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) YARN??? 1. YARN Architecture
  • 4. ? YARN ???? MapReduce? JobTracker? ?? ???? ??? ??. ? ??? Hadoop? ?? ???? ????? ?? ?? ? ?? ???? ??? ????. ? ? ??? ???? ?? ?? ?? YARN?? YARN? ?? MapReduce ??? ????? ???? ???? ??? ??? ? ?? ?????? ?? ????? ??? ??? ? ?? MapReduce ??? ?????? ????? ???? ?? ??? ?? ???? ??? ?? ??? MapReduce? ???? ?? ?? ?? ? ???? ??? ??? ?? ? Hortonworks Data Platform ???? ?? Application? YARN? ??? ???(YARN Ready) 4 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) YARN??? 1. YARN Architecture
  • 5. 5 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 1) ?? – MRv1? YARN? ?? 1. YARN Architecture MRv1 YARN(MRv2) ?? ????? ????? Job ?? JobTracker Resource Manager ??? ?? ? ???? - Node Manager(N?) Job ?? ?? TaskTracker (N?, Task ?) Application Master (N?, Task ?) Task ?? HDFS HDFS ??? ?? ? ??
  • 6. 6 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
  • 7. YARN? ?? ????? Resource Manager? Node Manager ? Resource Manager – ???? ??? ???? ???? ??, ??? Scheduler, Application Manager, Resource Tracker?? ????? ?? ? Scheduler : Node Manager?? ?? ??? ???? ??? ????? ????. Scheduler ? ????? ??? ????? ???? ?? ???, ???? ???? ??? ?? ? Application Manager : Application Master? ????, Application Master? ??? ?? ? Resource Tracker : Container? ?? ?? ??? ???? ???, Application Master ? ?? ?? ??, ??? Node Manager? ?? ??? ?? ? ??? ??? ?? ?? ??? ?? ?? ?? ??? ??? ??. 7 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
  • 8. YARN? ?? ????? Resource Manager? Node Manager ? Node Manager – Container? ??? ???? ???? ??, ?? ??? Resource Manager?? ??? ??? ??. Application Master ? Container? ?? ?? ??. ? Application Master : ??? ????? ?? ??? ??? ????, Scheduler ?? ? ??? Container? ?? ??, ???? ?? ??? ?????? ????. ? Container : CPU, ???(Disk), ???(Memory) ?? ?? ???? ????. ? ?? ? ??? ??(Graph processing)? MPI? ?? ?? ?? ????? ????? ? ?? ??. ?? ??(job)? ?? ?? ?? ???? ?????, ? ???? ?? ? Container ??? ??? ??. ??? ??? ??? Application Master? ??? ?, ?? ??? Resource Manager? ????. 8 YARN(Yet Another Resource Negotiator)? ?? ?? ??? ??? ?? ?????. 2) YARN? ???? 1. YARN Architecture
  • 9. 9 YARN(Yet Another Resource Negotiator)? ?? ??? ?? ?????. 1) YARN? Job ?? ?? 2. YARN Process
  • 10. 10 YARN(Yet Another Resource Negotiator)? ?? ??? ?? ?????. 1) YARN? Application Job ?? ?? 2. YARN Process 1. Initiate Job – Client?? Application? ????. 2. Request New Application to Resource Manager – Resource Manager? ??? App ?? 3. Copy Jars & Configuration to HDFS – App? ?? ???? ??? ???? HDFS? ????. 4. Submit Job to Resource Manager – Resource Manager?? ?? Application Job? ???? 5. Start Application Master – Resource Manager? ??? Node Manager? Application Master? ??? ? ??? Client?? ??? 6. Initialize Job – Resource Manager? ?? Application? Job? ??? Node Manager? ??? 7. Start Container – Resource Manager? ??? Node Manager?? Container? ??? ? Application Master?? ?? 8. Launch & Run – Application Master? Container? ???? Application? Job? ?? * Application - Spark, Hive?? Application? ?? * Job - ??? Application?? ???? Job?? ??(Hive Query, Spark Job ??)
  • 11. ? Hortonworks Data Platform? Ambari ?? ?? ? Zookeeper ?? 3EA ??? ?? ?? ① Ambari Web?? Services > YARN > Summary ? ?? ② Service Actions ???? Enable Resource Manager HA ? ??? ResourceManager HA wizard? ??? 11 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
  • 12. ③ Select Host ????? Standby ResourceManager? ??? Host ?? ? Next 12 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
  • 13. ④ Review Page? ??? YARN? ?? ????? ???? Next 13 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
  • 14. ⑤ Resource Manager HA ??? ??? Parameter ?? ??? ???? Next ⑥ HA ?? ?? ?? ?? 14 YARN(Yet Another Resource Negotiator)? HA(High Availability) ?? ??? ?? ?????. 1) Configure ResourceManager High Availability – In Hortonworks Data Platform 3. YARN High Availability
  • 15. 15 YARN(Yet Another Resource Negotiator)? HA(High Availability) ??? ??? ?????. 2) ResourceManager High Availability Works 3. YARN High Availability
  • 16. ? YARN Application Check Usage : yarn application 16 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 1) Yarn Command - applicaiton 4. YARN CLI Command COMMAND_OPTIONS Description -appStates <States> ?? ?? Application ??? ?? (ALL, NEW, NEW_SAVIN G, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILE D, KILLED) -appTypes <Types> -list ??? ?? ?? ???? ?? ??? Application ?? ? ?? -list ?? Application ??? ?? -kill <ApplicationId> Application? ?? ?? -status <ApplicationId> Application ?? ?? ??
  • 17. ? YARN Container Check Usage : yarn container 17 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 2) Yarn Command – Container & Queue 4. YARN CLI Command COMMAND_OPTIONS Description -list <Application Attempt Id> Application? ??? Container?? ??? ?? -status <ContainerId> Container? ?? ?? ?? ? YARN queue Check Usage : yarn queue COMMAND_OPTIONS Description -status <QueueName> Queue? ?? ?? ??
  • 18. ? YARN log Check Usage : yarn logs 18 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 3) Yarn Command - logs 4. YARN CLI Command COMMAND_OPTIONS Description -applicationId <application ID> ?? Application? ?? ?? -appOwner <AppOwner> Application Owner? ??? ??? ??, ???? ? ? ?? ?? ?????? ?? ?? -containerId <ContainerId> Container Id? ?? Container? ??? ?? -nodeAddress <NodeAddress> Nodemanager? Address? ?? ?? Nodemanager ? ??? ?? (Address:port)
  • 19. ? YARN set log level Usage : yarn daemonlog 19 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 4) Yarn Command – Set log level 4. YARN CLI Command COMMAND_OPTIONS Description -getlevel <host:httpport> <classname> YARN ??? log level ??? ?? ??. ??? ??? <host:httpport > ??? ???? ????? job? classname? ????. -setlevel <host:httpport> <classname> <level> YARN ??? log level? ????.
  • 20. ? YARN node Check Usage : yarn node 20 YARN(Yet Another Resource Negotiator)? CLI Command? ??? ?????. 5) Yarn Command - node 4. YARN CLI Command COMMAND_OPTIONS Description -all ?? ?? ?? ?? -list ?? ???? ?? ?? ?? –states ??? ???? ????? ???? ???? ? ??. -states <States> Works with -list to filter nodes based on input com ma-separated list of node states. -status <NodeId> ?? ??? ???? ??
  • 21. ? Resource ?? Parameter (yarn-site.xml) 21 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (yarn-site.xml) 5. YARN Parameter Parameter Name Description yarn.nodemanager.resource.memory-mb ?? ???? ???? ??? ??? ? ?? ??? ??? yarn.scheduler.minimum-allocation-vcores ????? ?? ? ? ?? ?? Vcore ?? yarn.scheduler.maximum-allocation-vcores ????? ?? ? ? ?? ?? Vcore ?? yarn.scheduler.minimum-allocation-mb ????? ?? ? ? ?? ?? memory ?? yarn.scheduler.maximum-allocation-mb ????? ?? ? ? ?? ?? memory ?? yarn.scheduler.increment-allocation-vcores ????? ?? ?? ? ? ?? Vcore ? yarn.scheduler.increment-allocation-mb ????? ?? ?? ? ? memory ?? yarn.nodemanager.pmem-check-enabled ???? ??? ?? ? physical memory ?? ?? yarn.nodemanager.vmem-check-enabled ???? ??? ?? ? virtual memory ?? ?? yarn.nodemanager.vmem-pmem-ratio ???? ?? ??? ?? ??, default? pmem ???? 2.1? Virtual Memory ?? ??? ?? JIRA ?? ??? https://issues.apache.org/jira/browse/YARN-4714 ????? VMem Check ??? Disable ??? ?? ??? ?? ?? ??? ?? Test ?? ?
  • 22. ? Resource ?? Parameter Check List (yarn-site.xml) 22 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (yarn-site.xml) 5. YARN Parameter yarn.scheduler.maximum-allocation-vcores >= yarn.scheduler.minimum-allocation-vcores yarn.scheduler.maximum-allocation-mb >= yarn.scheduler.minimum-allocation-mb yarn.scheduler.minimum-allocation-vcores >= 0 yarn.scheduler.minimum-allocation-vcores <= HostsVCores yarn.scheduler.maximum-allocation-vcores >= 1 yarn.scheduler.maximum-allocation-vcores <= HostsVcores yarn.scheduler.minimum-allocation-mb < 1024 MB YARN ??? ?? ?? ?? Excel ??(Cloudera?? ??) https://www.dropbox.com/s/38reez543rh9xsv/yarn-tuning-guide.xlsx?dl=0
  • 23. ? Resource ?? Parameter (mapred-site.xml) 23 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (mapred-site.xml) 5. YARN Parameter Parameter Name Description yarn.app.mapreduce.am.resource.cpu-vcores MR Application Master? ?? ?? Vcore ?? yarn.app.mapreduce.am.resource.mb MR Application Master? ?? ?? memory ?? mapreduce.map.cpu.vcores Map ??? ???? Vcore ?? mapreduce.map.memory.mb Map ??? ???? memory ?? mapreduce.map.java.opts.max.heap Mapper? JAVA HEAP SIZE mapreduce.reduce.cpu.vcores Reduce ??? ???? Vcore ?? mapreduce.reduce.memory.mb Reduce ??? ???? memory ?? mapreduce.reduce.java.opts Reducer? JAVA HEAP SIZE mapreduce.task.io.sort.mb Split/Sort ??? ?? ?? ???
  • 24. ? Resource ?? Parameter Check List (mapred-site.xml) 24 YARN(Yet Another Resource Negotiator)? ?? Parameter? ??? ?????. 1) Yarn Parameter Description (mapred-site.xml) 5. YARN Parameter 1) Application Master ?? ??? yarn.app.mapreduce.am.resource.cpu-vcores >= yarn.scheduler.minimum-allocation-vcores yarn.app.mapreduce.am.resource.cpu-vcores <= yarn-scheduler.maximum-allocation-vcores yarn.app.mapreduce.am.resource.mb >= yarn.scheduler.minimum-allocation-mb yarn.app.mapreduce.am.resource.mb <= yarn.scheduler.maximum-allocation-mb yarn.app.mapreduce.am.resource.mb = ApplicationMaster Java Heap (must be close) 2) Map Task ?? ??? mapreduce.map.cpu.vcores >= yarn.scheduler.minimum-allocation-vcores mapreduce.map.cpu.vcores <= yarn-scheduler.maximum-allocation-vcores mapreduce.map.cpu.memory.mb >= yarn.scheduler.minimum-allocation-mb mapreduce.map.cpu.memory.mb <= yarn.scheduler.maximum-allocation-mb mapreduce.map.memory.mb = mapreduce.map.java.opts.max.heap (must be close) mapreduce.task.io.sort.mb << mapreduce.map.java.opts.max.heap 3) Reduce Task ?? ??? mapreduce.reduce.cpu.vcores >= yarn.scheduler.minimum-allocation-vcores mapreduce.reduce.cpu.vcores <= yarn-scheduler.maximum-allocation-vcores mapreduce.reduce.cpu.memory.mb >= yarn.scheduler.minimum-allocation-mb mapreduce.reduce.cpu.memory.mb <= yarn.scheduler.maximum-allocation-mb mapreduce.reduce.java.opts = mapreduce.reduce.memory.mb (must be close)
  • 25. ? YARN? Capacity Scheduler? ?????? Hadoop ??????? ????? ????? ???? ???? ?????? ?? ? ? ?? Capacity? ??? ?? ??? ??? ?? ex) OLAP Hive? 70% ??, ??? Spark 30% ?? ? ?? ????? ?? ?? ?? ???? ??? ????? ??? 25 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 1) YARN Capacity Scheduler Description 6. YARN Capacity Scheduler
  • 26. ? Ambari?? ???? YARN Queue Manager View, YARN Capacity Scheduler? Ambari ??? UI? ??? ? ??? ???. 26 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari 6. YARN Capacity Scheduler
  • 27. ? Queue ?? ?? ① YARN Queue Manager view ????? Add Queue ??? ????. 27 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
  • 28. ? Queue ?? ?? ② ??? Queue??? ??? ? ?? ??? ????. 28 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
  • 29. ? Queue ?? ?? ③ ???? ??? Queue? ???? ?? Queue? ?? ?? ??? ????. 29 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
  • 30. ? Queue ?? ?? ④ ??? ?? Queue? ????? ??? ????. 30 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 2) Using YARN Queue Manager on Ambari - Setting up Queues 6. YARN Capacity Scheduler
  • 31. ? Queue ?? ?? ① Capacity 31 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description Capacity Queue? ??? ? ?? ???? ???? ??, ?? Queue? ?? ?? Queue ???? ??? Max Capacity Queue?? ??? ? ?? ???? ???? ?? ??, ? ?? ? ? ?? ??? ? ?? ??? idle ??? ??? ? ?? ? Enable Node Labels Queue? Node Label? ????? ?? ?/?
  • 32. ? Queue ?? ?? ② Access Control and Status 32 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description Administer Queue ?? ??? ? ??? ??? ?? ?? Submit Applications Application? ?? ??? ? ??? ??? ? ??? ??
  • 33. ? Queue ?? ?? ③ Resources 33 YARN(Yet Another Resource Negotiator)? Capacity Scheduler? ??? ?????. 3) Using YARN Queue Manager on Ambari – Configuring Queues 6. YARN Capacity Scheduler Parameter Description User Limit Factor 1? ??? ?? ?? ???? Queue? ???? ??? ?? ?? 2? ??? ?? ?? ?? ???? Queue ??? ??? ??? 0.5? ??? ?? ???? Queue? ???? ??? ???? ??? ? ?? Minimum User Limit ????? ?? ?? ???? ?? ??? ?? Maximum Applications ? ??? ???? Scheduler Maximum Applications setting? ?? Maximum AM Resource ? ??? ???? Scheduler Maximum AM Resource setting? ?? Priority ? ?? ? ?? ???? ?? ??? ?? Queue? ? ?? ???? ???? ????
  • 34. ? https://docs.hortonworks.com/HDPDocuments/ Ambari-2.6.0.0/bk_ambari- views/content/ch_using_yarn_queue_manager_v iew.html ? http://www.popit.kr/what-is-hadoop-yarn/ ? http://blog.skcc.com/1883 ? https://hadoop.apache.org/docs/r2.7.1/hadoop- yarn/hadoop-yarn-site/YarnCommands.html 35 ???? ????