hiera vault lookups slowing down puppet runs

[quanghiem]

I have vault setup and installed the hiera-vault gem on my puppetserver. When I enable the vault backend, my puppet apply runs go from a few seconds to a few minutes. The client is suck on Info: Loading facts. The puppetserver logs show hundreds of api calls to vault by every hiera enabled module multiplied by how many hierarchy I have listed. Is this normal?

Here is my hiera.yaml:

---
:backends:
  - yaml
#  - vault
:vault:
  :addr: https://vault.xxx.io
  :token: xxx
  :default_field: value
  :default_field_behavior: only
  :mounts:
    :generic:
      - secret
:yaml:
  :datadir: /etc/puppetlabs/code/
:hierarchy:
  - environments/%{environment}/hieradata/"nodes/%{::trusted.certname}"
  - environments/%{environment}/hieradata/common
  - environments/%{environment}/hieradata/users
  - workspace/%{environment}/hieradata/"nodes/%{::trusted.certname}"
  - workspace/%{environment}/hieradata/common
  - workspace/%{environment}/hieradata/users
  -

Here is a excerpt of the puppetserver logs:

2016-06-17 14:02:40,721 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/ntp::service_name: permission denied
2016-06-17 14:02:40,773 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/ntp::service_name: permission denied
2016-06-17 14:02:40,882 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//ntp::stepout: permission denied
2016-06-17 14:02:40,933 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/ntp::stepout: permission denied
2016-06-17 14:02:40,985 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/ntp::stepout: permission denied
2016-06-17 14:02:41,094 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//ntp::tinker: permission denied
2016-06-17 14:02:41,145 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/ntp::tinker: permission denied
2016-06-17 14:02:41,198 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/ntp::tinker: permission denied
2016-06-17 14:02:41,306 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//ntp::udlc: permission denied
2016-06-17 14:02:41,357 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/ntp::udlc: permission denied
2016-06-17 14:02:41,410 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/ntp::udlc: permission denied
2016-06-17 14:02:41,518 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//ntp::udlc_stratum: permission denied
2016-06-17 14:02:41,569 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/ntp::udlc_stratum: permission denied
2016-06-17 14:02:41,622 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/ntp::udlc_stratum: permission denied
2016-06-17 14:02:41,901 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//resolv_conf::searchpath: permission denied
2016-06-17 14:02:41,954 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/resolv_conf::searchpath: permission denied
2016-06-17 14:02:42,005 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/resolv_conf::searchpath: permission denied
2016-06-17 14:02:42,114 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//resolv_conf::options: permission denied
2016-06-17 14:02:42,166 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/environments/production/hieradata/users/resolv_conf::options: permission denied
2016-06-17 14:02:42,220 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret/workspace/production/hieradata/users/resolv_conf::options: permission denied
2016-06-17 14:02:42,486 INFO  [qtp643197702-62] [puppet-server] Puppet hiera(): [hiera-vault] Could not read secret secret//firewall::ensure: permission denied

[jsok]

This is expected behaviour, see https://github.com/jsok/hiera-vault/blame/master/README.md#L138 for details.

[abooitt]

Yes. And this is exactly the reason for #10

[jovandeginste]

We solved this for our installation in a different way by writing our own intermediate hiera-backend: https://rubygems.org/gems/hiera-router

The hiera-router basically redirects some hiera requests to other hiera-backends (like vault) based on the content of the yaml-entry... Constructive feedback is appreciated.

[mindriot88]

I have a solution for this problem which was suited to our particular requirement, it maybe applicable to others #33