(Edit: thanks Mostafa for the valuable feedback, I updated this post with explanation about the relationship between Yarn base and Java base memory settings)There are several related memory settings for jobs running in HDInsight cluster which most customers need to pay close attention to. When not correctly set, it will cause obscure failures in Hive/Pig/Mapreduce/Tez jobs.Note that HDInsight service provides default memory settings, however, the defaults may be subject to change as we tune the service for various workloads or as we move to different VM types and hardware.

1. Yarn Logs Hdfs Location

Yarn Logs Hdfs Location

It is advised that customers explicit set these settings once they know the right settings for their specific jobs. There are two ways to apply these settings: 1) provide it when creating HDInsight cluster using SDK. Or 2) set it on a per job basis. Use '-Define' option in Powershell SDK, or 'set property=value' in a hive script, or set it in mapreduce code directly. Note that the Yarn level settings can only be changed during cluster creation.1. Yarn memory settings:HDInsight 3.x service deploys Hadoop 2.x clusters.

Hadoop 2.x (Yarn) introduced the concept of containers. Unlike Hadoop 1.x where each node is assigned a fixed number of 'slots', in Hadoop 2.x each Yarn task (mapper, reducer or Tez task) is assigned a container which has an memory limit. This affects 1) how many containers can run in parallel in any given node; and 2) Yarn Node Manager will monitor the memory usage of the task and kill a container when the memory usage exceeds that limit.Yarn defines how much memory is available for allocation and what is the minimum and maximum container size:yarn.nodemanager.resource.memory-mb = 5376yarn.scheduler.minimum-allocation-mb = 768yarn.scheduler.maximum-allocation-mb = 5376Note that you can only ask for container size of multiples of minimum-allocation-mb. In the above example settings, if you ask for a container of size 1024MB, you will actually get a container of 1536MB.2. Mapreduce memory settings:These are the default Yarn container memory settings in HDInsight for the mapper, reducer and AM(Application Master):mapreduce.map.memory.mb = 768mapreduce.reduce.memory.mb = 1536yarn.app.mapreduce.am.resource.mb = 1536This is a typical error message in your job attempt log, if these limits are exceeded:Containerpid=container4510002,containerID=container4510002 is running beyond physical memory limits. Current usage: 519.1 MB of 512 MB physical memory used; 770.1 MB of 1.0 GB virtual memory used. Killing container.

Dump of the process-tree for container4510002: - PID CPUTIME(MILLIS) VMEM(BYTES) WORKINGSET(BYTES) - 49 -4712 952 - 607 - 4780 47840 - 454 Container killed on request. Exit code is 137Aside from the memory monitoring, each Java process has its own heap space settings. These are the default settings for the mapper, reducer and AM:mapreduce.map.java.opts = '-Xmx512m'mapreduce.reduce.java.opts = '-Xmx1024m'yarn.app.mapreduce.am.command-opts = '-Xmx1024m'This is a typical error message in your job attempt log if these limits are exceeded:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceededorError: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap spaceThe Yarn based and Java based settings are related.

The Java heap settings should be smaller than the Yarn container memory limit because we need reserve memory for Java code. The best practice is to reserve 20% memory for code. So if settings are correct, theoretically pure Java-based Hadoop tasks should never get killed by Yarn Node Manager unless there is a bug in your Java code somewhere. If the Yarn based error is seen, the cure is to either increase the Yarn container memory or decrease Java heap space.

If the Java heap error is seen, you can either increase both memory settings (in which case you'll get fewer tasks running in parallel), or bring down the memory usage if possible.For streaming jobs, the developer needs to make sure the streaming program don't exceeds Yarn container memory settings. This might require trial run and iterative tuning.Eventually the memory usage depends on the job you are doing. Anatomy for the artist jeno barcsay.

For example, if you are doing hive query 'CREATE TABLE AS SELECT' on huge tables your mapper may demand more memory than 512MB. To increase memory settings for the mapper, you can overwrite these settings in Mapreduce configuration:mapreduce.map.memory.mb = 1536mapreduce.map.java.opts = '-Xmx1024m'There is another memory setting that depends on the container memory size:mapreduce.task.io.sort.mb = 307This is the max memory a mapreduce task can use to sort data in buffer during the shuffle stage. This value should be 1/3 to 1/2 of the task heap size.This is an illustration of various memory settings for Mapreduce to help you visualize the relative size of them:3.

Tez memory settingsThese are the default Tez memory settings for Tez AM:tez.am.resource.memory.mb = 1536tez.am.java.opts = 1024For tez container, it uses mapper's memory settings:mapreduce.map.memory.mb = 768mapreduce.map.java.opts = '-Xmx512m'This is an illustration of various memory settings for Tez:4.