hadoop-env.sh.j2 in two separate locations

[PACordonnier]

hadoop-env.sh.j2 is stored in two locations in the repo

• roles/hadoop/common/templates
• roles/hdfs/common/templates

I'm not sure it is necessary ? From my understanding roles/hadoop/common/templates/hadoop-env.sh.j2 is exclusively used by hadoop/client. HDFS roles used their own hadoop-env.sh.j2

I think it makes sense to only manage one file (whether trough symbolic link or delete unused file)

[rpignolet]

If this file is rendered at the same location on target machine, I think it should only be templated by hadoop_client and deleted from hdfs role. We must check that hadoop-env content can be rendered by hadoop tdp_vars.

[PACordonnier]

That's the issue indeed.

hdfs's hadoop-env.sh.j2 uses vars such as hdfs_datanode_heapsize which are hdfs vars, not hadoop, and this would create an error hadoop_client_config.

[rpignolet]

This is always the problem with Hadoop which contains HDFS and YARN.

I think the hadoop-env needs to use hadoop variables from the tdp_vars to make the template happen at the hadoop_client level even though that contains the memory settings for HDFS components.

I don't understand what the java heaps configuration does in the hadoop-env...

Maybe we should discuss it at one of our meetings.

[PACordonnier]

It's getting worse, hadoop-env is actually in three locations since it's also in roles/yarn/common/templates 😱 Will update my recent PR

We can discuss it during meeting. Now that I investigate a bit further I think having 3 files, while not elegant, could actually be an adequate solution.