Late-bound variable replacement

[Oloremo]

Hi,

Just found this project and it looks very promising.
Could you please clarify the "Late-bound variable replacement" feature? So it's basically the same idea as the stack in pillarstack so you can re-use pillar data from the previously included pillar?

What I'm really looking for is a lazy variable interpolation, so I could do something like this:

var1: test
var2: ${var1}

In a single pillar file.
It's implemented in saltclass and reclass but they are too complex and require a really weird approach like defining a class for each minion.

[jgraichen]

Late-bound variable replacement by default only works in in tower top file. As this file is first passed through Jinja and Yaml before we can load any pillar data file there isn't any way to e.g. insert variables from the tower pillar with Jinja.

For regular pillar data files I do recommend to just place reused variables in a previously loaded file and use regular Jinja like this:

# tower.sls
base:
  - defaults.sls
  - pillar.sls

# defaults.sls
some_key:
  another_key: 1

# pillar.sls
special_key: {{ pillar.get('some_key:another_Key') }}

This would render into the following pillar result:

some_key:
  another_key: 1
special_key: 1

This is the recommended way for accessing pillar values that works very well. The pillar variable always contains all pillar values just like with pillarstack. It just does not work well to access values defined in the same file because of how salt itself passes the file through template engines one-by-one.

Nevertheless late-bound variables can kind of circumvent a few issues for specific cases.

---

The tower top file uses late-bound variables to handle a few specific edge cases we often had to simplify things. That does work because the top file has well defined semantics.

Imaging having to deploy an application for different flavors sharing most things but having a few different settings. You can first match e.g. the minion sub-domain to determine the flavor and assign e.g. a site and env pillar key. To use these key to extrapolate file paths you could not use Jinja (the file has long been parsed already) but late-bind variables:

# tower.sls
base:
  # All minions with `flavor-a` should load that flavor
  - '*.flavor-a.example.org':
    - site: a
  # flavor b, c ...

  # Assign environment, e.g. production, staging, testing
  - '*.prod.*':
    - env: production
  - '*.staging.*':
    - env: staging

  # Load shared and specific pillars
  - site/default/*.sls
  - site/{site}/default/*.sls
  - site/{site}/{env}/*.sls

A minion node1.staging.flavor-a.example.org would load the pillar files site/default/*, site/a/default/* and site/a/staging/* in this order. This can ease defining and loading setups that have a lot in common.

---

Last but not least you can do a pillar wide or partial late-bind variables replacement using an advanced python pillar file. Please be aware that this can be quite risky and you can easily crash your pillar if you do not care for all edge cases.

The tower pillar exposes an API you can call to "format" the full pillar or only some parts. This method will traverse all dictionaries and lists, take each string value and replace everything that looks like {...} with a matching pillar value. (See Advanced).

Here is an example for "formatting" the whole pillar:

# tower.sls
base:
  - load/lots/of/files/*.sls
  # Last apply the python pillar that can fully manipulate everything that has been loaded
  - format.py

#! py
# The above instructs salt to use the python renderer

def run():
  # Format and return the whole pillar
  # This will replace anything in any string that looks like `{...}` and where the expression
  # matches a pillar values, e.g.: `{pillar.key}`, `{list.0.key}`.
  return tower.format(pillar)

This can be risky if unconditional applied to the whole pillar as e.g. sections in configuration file blob could match too. Salt tower tries to ignore everything that isn't found or has an invalid expression inside the brackets but that cannot always be guaranteed.

If needed try to only apply it to partial trees you know work to keep possible impact as small as possible. We use it for example to merge and format some default settings into each application pillar (they do share a lot common structure but have e.g. own application name) but we did fail a lot while setting this up.

#!py

def run():
    result = {}
    defaults = __pillar__['default_app_config']
    # e.g. {
    #        'database': 'sqlite:///opt/{name}.sqlite'
    #        'listen': '0.0.0.0:{port}'
    # }

    for name, conf in __pillar__['applications'].items():
        # Merge defaults with conf into new dictionary
        conf = tower.merge({}, defaults, conf)

        # Format late-bind defaults with application config
        conf = tower.format(app_conf, name=name, port=conf['port'])

        result[name] = conf

    return {'applications': result}

Be reminded of some edge cases: If a late-bound variables refers some other pillar values that includes a late-bound variables you might end with having

[Oloremo]

Thanks for a very detailed explanation!