Problem with salt-ssh and mine.get

[manjiki]

Description of Issue/Question

Under some circumstances highstate fails when using salt-ssh and salt['mine.get'] with error:

Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'

Although I am able to successfully run each state individually, I get the above error when running highstate. The problem doesn't occure if we:

• remove the block/state where salt['mine.get'] is called
• change the order of states in top.sls

Setup

Roster file:

murray2:
  host: 10.10.10.56
  user: dan
  sudo: True
  priv: ~/.ssh/dan.pem
  thin_dir: /opt/salt-thin
  mine_functions:
    network.interfaces: []
    grains.item:
      - roles
      - fqdn
      - applications
      - ip_interfaces
  grains:
    roles:
      - backend
    applications:
      - chuck

states/test_hosts/init.sls:

/etc/hostname:
  file.managed:
    - contents: {{ salt['grains.get']('id') }}

{%- for host, data in salt['mine.get']('*', 'grains.item').items() %}
hostsfile_{{host}}:
  host.present:
    - names:
      - {{host}}
    - ip: {{data.ip_interfaces.eth0[0]}}
{% endfor%}

states/test_apps/init.sls

{%- for app in salt['grains.get']('applications',[]) %}
{%- set app_data = salt['pillar.get']('applications:%s' % app) %}
{%- from 'test_apps/macros.j2' import this_macro  %}
/tmp/{{app}}:
  file.managed:
    - contents: {{ salt['disk.blkid']('')}}

{{this_macro(app)}}
{% endfor %}

states/top.sls:

base:
  '*':
    - test_hosts
    - test_apps

Steps to Reproduce Issue

$ salt-ssh 'murray2' state.highstate

[ERROR   ] Rendering exception occurred: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
[CRITICAL] Rendering SLS 'base:test_apps' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
murray2:
    - Rendering SLS 'base:test_apps' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'

The last lines of the master.log are:

2016-10-05 17:19:03,239 [salt.template    ][PROFILE ][6241] Time (in seconds) to render '/tmp/salt/files/base/test_hosts/init.sls' using 'yaml' renderer: 0.00873517990112
2016-10-05 17:19:03,240 [salt.fileclient  ][DEBUG   ][6241] Could not find file from saltenv 'base', 'salt://test_apps.sls'
2016-10-05 17:19:03,242 [salt.fileclient  ][DEBUG   ][6241] In saltenv 'base', looking at rel_path 'test_apps/init.sls' to resolve 'salt://test_apps/init.sls'
2016-10-05 17:19:03,243 [salt.fileclient  ][DEBUG   ][6241] In saltenv 'base', ** considering ** path '/tmp/salt/files/base/test_apps/init.sls' to resolve 'salt://test_apps/init.sls'
2016-10-05 17:19:03,243 [salt.fileclient  ][DEBUG   ][6241] Fetching file from saltenv 'base', ** attempting ** 'salt://test_apps/init.sls'
2016-10-05 17:19:03,243 [salt.fileclient  ][DEBUG   ][6241] No dest file found
2016-10-05 17:19:03,244 [salt.fileclient  ][INFO    ][6241] Fetching file from saltenv 'base', ** done ** 'test_apps/init.sls'
2016-10-05 17:19:03,244 [salt.template    ][DEBUG   ][6241] compile template: /tmp/salt/files/base/test_apps/init.sls
2016-10-05 17:19:03,245 [salt.utils.jinja ][DEBUG   ][6241] Jinja search path: ['/opt/salt-thin/running_data/var/cache/salt/minion/files/base']
2016-10-05 17:19:03,256 [salt.utils.templates][ERROR   ][6241] Rendering exception occurred: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
2016-10-05 17:19:03,257 [salt.state       ][CRITICAL][6241] Rendering SLS 'base:test_apps' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
2016-10-05 17:19:03,362 [salt.utils.lazy  ][DEBUG   ][6232] LazyLoaded nested.output

Lastly, highstate runs successfully if I change states order in states/top.sls:

base:
  '*':
    - test_apps
    - test_hosts

Versions Report

Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul  1 2016, 15:12:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

[gtmanfred]

Thanks for reporting this @manjiki I am working on replicating this now.

[gtmanfred]

I was able to replicate this issue, thanks for reporting it.

[gtmanfred]

@manjiki has this worked in previous versions of salt?

[manjiki]

@gtmanfred I can't say, I haven't used mine with salt-ssh before.

[gtmanfred]

I just double checked and this is also not working on 2015.8

[ptitdoc]

I don't know if it is related but I have a bug when using salt-ssh and using such code inside a pillar:
{% set ca_cert = salt.saltutil.runner('mine.get', tgt='master.abc.local', fun='x509.get_pem_entries') %}

I tracked down the problem inside saltutils.py, in the runner function:

    if 'master_job_cache' not in __opts__:
        master_config = os.path.join(os.path.dirname(__opts__['conf_file']),
                                     'master')
        master_opts = salt.config.master_config(master_config)
        rclient = salt.runner.RunnerClient(master_opts)
    else:
        rclient = salt.runner.RunnerClient(__opts__)

When using salt-ssh, __opts__['master_job_cache'] does not directly exists. In this case, using __opts__['__master_opts__']['master_job_cache'] works by replacing the if condition and running:
rclient = salt.runner.RunnerClient(__opts__['__master_opts__'])

[ptitdoc]

It fixed both cases (with salt or with salt-ssh) by updating to the following code, but maybe there is a better solution:

    if '__master_opts__' in __opts__:
        master_opts = __opts__['__master_opts__']
        master_config =  os.path.join(os.path.dirname(__opts__['__master_opts__']['conf_file']),
                                     'master')
    else:
        master_opts = __opts__
        master_config = os.path.join(os.path.dirname(__opts__['conf_file']),
                                     'master')

    if 'master_job_cache' not in master_opts:
        master_opts = salt.config.master_config(master_config)

    rclient = salt.runner.RunnerClient(master_opts)

EDIT: Another solution is to fix client/ssh/init.py in run_wfunc:
The pillar is compiled by passing master opts in opts_pkg['__master_opts__'], which causes the pillar renderer to loose track of salt master options.
Replacing opts_pkg['__master_opts__'] = self.context['master_opts'] with opts_pkg.update(self.context['master_opts']) is sufficient to makes the pillar able to use runners.

[ptitdoc]

In fact the same problem arise when salt['publish.publish'] is called when using salt-ssh (for instance, this occurs in x509.create_certificate when the public key is published to the ca_server)
It ends with the following error when debugging salt-call manually in the ssh minion:

[DEBUG   ] LazyLoaded publish.publish
[ERROR   ] Cannot run publish commands without a connection to a salt master. No command sent.

I feel like something is not working properly with client/ssh/wrapper/publish.py where both runner and publish function are wrapped. These wrapper should call runner or publish with __opts__['__master_opts__'] instead of __opts__, which apparently does not occurs when running salt-ssh. Sadly I fail to understand the wrapping logic properly.

[ptitdoc]

The first bug I reference has an history:
#34835

But in my case, it's already fixed. Somehow the ssh wrapper of mine.get is not called when I use salt-ssh

[ptitdoc]

The problem related to publish.publish is now referenced here: #40943

[ptitdoc]

Here are my temporary fixes for mine.get in pillars not working properly when using salt SSH.
For instance, the following code present in a pillar is not working when using salt-ssh:

{% set ca_mine = salt.saltutil.runner('mine.get',
    tgt=ca_minion,
    fun=pem_mine)
%}

This is partially fixed in the following commits:
2899855

However, the dict merge order matters and master_opts needs to take precedence in order to mine.get over salt-ssh to work properly.

[thomasrossetto]

Hi guys, i have a similar problem with pillar.get ... I'm trying to use this formula https://github.com/saltstack-formulas/logrotate-formula with salt-ssh, but i was received this error

Rendering SLS 'base:logrotate.jobs' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'pillar.get'
      /var/tmp/.root_4a38c3_salt/running_data/var/cache/salt/minion/files/base/logrotate/map.jinja(6):
      ---
      # -*- coding: utf-8 -*-
      # vim: ft=jinja
      
      {% import_yaml 'logrotate/defaults.yaml' as default_settings %}
      
      {% set os_family_map = salt['grains.filter_by']({    <======================
              'Arch': {
                  'service': 'logrotate.timer',
                  'default_config': {
                      'tabooext': '+ .pacorig .pacnew .pacsave',
                  },
      [...]
      ---

This is my salt report :

Salt Version:
           Salt: 2016.11.5
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed
 
System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.9.27-moby
         system: Linux
        version: Ubuntu 16.04 xenial

Thanks for any help!

[gtmanfred]

I would be very surprised if that was related. Because I have used pillar.get in salt ssh before, can you open a new bug with steps to reproduce what you are seeing.

…

On Mon, May 29, 2017 at 3:39 AM Thomas Rossetto ***@***.***> wrote: Hi guys, i have a similar problem with pillar.get ... I'm trying to use this formula https://github.com/saltstack-formulas/logrotate-formula with salt-ssh, but i was received this error Rendering SLS 'base:logrotate.jobs' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'pillar' /var/tmp/.root_4a38c3_salt/running_data/var/cache/salt/minion/files/base/logrotate/map.jinja(6): --- # -*- coding: utf-8 -*- # vim: ft=jinja {% import_yaml 'logrotate/defaults.yaml' as default_settings %} {% set os_family_map = salt['grains.filter_by']({ <====================== 'Arch': { 'service': 'logrotate.timer', 'default_config': { 'tabooext': '+ .pacorig .pacnew .pacsave', }, [...] --- This is my salt report : Salt Version: Salt: 2016.11.5 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.4.2 docker-py: Not Installed gitdb: 0.6.4 gitpython: 1.0.1 ioflo: Not Installed Jinja2: 2.8 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.12 (default, Nov 19 2016, 06:48:10) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: Not Installed RAET: Not Installed smmap: 0.9.0 timelib: Not Installed Tornado: 4.2.1 ZMQ: Not Installed System Versions: dist: Ubuntu 16.04 xenial machine: x86_64 release: 4.9.27-moby system: Linux version: Ubuntu 16.04 xenial Thanks for any help! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36796 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAssoVoFoxqSh4U2oY3degwTlcdEjj4jks5r-pJcgaJpZM4KO4jH> .

[thomasrossetto]

Ok @gtmanfred , I'll do it soon, thanks!

[lastmikoi]

I am suffering from the very same issue under 2016.11.8. It would seems that any call to grains.get after usage of mine.get with salt-ssh becomes broken. Is there any known workaround ?

[gtmanfred]

Can you give me a better example to replicate that on?

Because i am not able too.

[root@salt ~]# salt-ssh \* test.ping
minion1:
    True
[root@salt ~]# salt-ssh \* mine.get \* test.ping
minion1:
    ----------
    minion1:
        True
[root@salt ~]# salt-ssh \* grains.get os
minion1:
    CentOS

[lastmikoi]

Sure, I've managed to reproduce the issue on a small codebase that I've pushed there: https://github.com/lastmikoi/saltstack-bug-36796 (just make sure to replace the absolute paths I haven't found a way to make relative yet, and the roster of course)

In this codebase I am able to apply my dummy mine.get-using salt-ssh -i '*' state.apply test_mine as well as apply my dummy grains.get-using salt-ssh -i '*' state.apply test_get. But when trying to apply the topstate, with test_mine applied before test_get it breaks with the following error that is identical to the originally reported one:

$ salt-ssh -i '*' state.apply
[ERROR   ] Rendering exception occurred: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
[CRITICAL] Rendering SLS 'base:test_get' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'
salt-master:
    - Rendering SLS 'base:test_get' failed: Jinja variable 'salt.utils.templates.AliasedLoader object' has no attribute 'grains.get'

As mentioned by the original reporter, this issue doesn't trigger if the mine.get-using state is applied after all grains.get-using state.

[gtmanfred]

Ok, cool I am able to replicate this.

As a side not, you can use {{grains.get('os')}} and it will work, that just uses the grains dictionary that is loaded into the renderer, instead of using using the grains.get salt module.

@saltstack/team-ssh does anyone have an idea of why this might be happening?

Initially i thought it might have been because mine.get was in the wrappers, but grains is also in there.

Thanks,
Daniel

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[OrangeDog]

Looks like this was fixed on develop but never ported.

[anilsil]

Folks, Can this be tested using 3006.3 packages?

[lkubb]

I think there are at least two issues lumped together here:

The initial one

I could not reproduce it using the config from #36796 (comment) - neither with the test suite nor in RL (I suspect this might have been fixed by the loader improvements since). I'm reasonably sure the originally linked PR does not fix it though.

saltutil.runner

This is the one where using saltutil.runner to call the mine runner during pillar compilation fails (the linked PR that was not migrated to master):

The underlying problem is valid and possibly affects many modules: Since the minion options take precedence over master options in pillar compilation, some options that are present in both minion and master context are set wrong for pillar compilation. Examples include [cache|config|sock]dir, caused by root_dir. It should be / on the master, but becomes /var/tmp/.root_*_salt/running_data. This also causes #60002 since the default gpg_keydir is set wrong. Also anything that relies on the master cache dir will likely not work, including requesting the mine data via a runner. Nearly complete list of differences vs usual pillar compilation (I checked):

• interface
• sock_dir
• root_dir
• pki_dir
• cachedir
• pillar_cache
• pillar_cache_ttl
• utils_dirs
• file_buffer_size
• conf_file
• auto_accept
• log_file
• log_level
• pidfile
• loop_interval
• default_include
• sign_pub_messages
• salt_cp_chunk_size
• __role
• __cli
• id (this should in fact be the master ID that's passed in to salt.pillar.get_pillar, it is preserved for ext_pillar_opts and overridden inside the pillar class)
• config_dir

Reversing the .update order (prioritizing master opts) makes the resulting opts nearly identical to the regular rendering and should thus fix a couple of issues.

The original PR (#50489) is a bit weird since it takes specific care to reset these pillar opts if the current invocation is not pillar.items or mine.get, but these opts are discarded after pillar compilation and not used for the following wrapper calls (this function is also never called during pillar rendering since the execution modules there are just the regular ones).

The fix is insufficient in the current state of salt-ssh since pillars are always re-rendered with the minion opts only during state rendering (fix in #65484). I will include the findings above in that PR as well.

[dwoz]

Closing as fixed per #65067 (comment)