Local ruleset patches

[mk-surf]

Hi all,

Does anyone use local ruleset patching? (docs)

I'm trying to get rid of our manual 'post-yoda-customization-playbook' :)

The file I'm trying to patch is /etc/irods/irods-ruleset-uu/uuGroup.r

So, I did 'git diff' and added this to group_vars/full.yml

core_rulesets:
  - name: yoda-ruleset
    repo: https://github.com/UtrechtUniversity/yoda-ruleset.git
    ruleset_name: rules-uu
    version: "{{ yoda_version }}"
    patch: |
      diff --git a/uuGroup.r b/uuGroup.r
      index ce64b9f..916f1e8 100644
      --- a/uuGroup.r
      +++ b/uuGroup.r
      @@ -870,7 +870,7 @@ uuExternalUser(*userName) {
        *nameAndDomain = split(*userName, "@");
        if (size(*nameAndDomain) == 2) {
                *domain = elem(*nameAndDomain, 1);
      -         if (*domain != "uu.nl" && *domain not like "*.uu.nl") {
      +         if (*domain != "") {
                        # is external
                        true;
                } else {
    install_scripts: true
  - name: core
    ruleset_name: core
    path: /etc/irods/core.re
    install_scripts: false

( including tabs )

However, it does not seem to work:

TASK [yoda_rulesets : Run make install for each ruleset] **********************************************************************************************************************************************************************************
changed: [example-yoda.irods.surfsara.nl] => (item=[{u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.7', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'}, {u'skip_reason': u'Conditional result was False', u'item': {u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.7', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'}, u'skipped': True, u'ansible_loop_var': u'item', u'changed': False}, {u'ansible_loop_var': u'item', u'item': {u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.7', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'}, u'changed': False, u'remote_url_changed': False, u'failed': False, u'invocation': {u'module_args': {u'force': False, u'track_submodules': False, u'reference': None, u'dest': u'/etc/irods/repo_yoda-ruleset', u'verify_commit': False, u'clone': True, u'gpg_whitelist': [], u'accept_hostkey': False, u'update': True, u'ssh_opts': None, u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'bare': False, u'archive': None, u'refspec': None, u'executable': None, u'remote': u'origin', u'recursive': True, u'separate_git_dir': None, u'umask': None, u'depth': None, u'version': u'release-1.7', u'key_file': None}}, u'after': u'57922dffc62c18364c64ca8529bfc47c34e77930', u'before': u'57922dffc62c18364c64ca8529bfc47c34e77930'}])
skipping: [example-yoda.irods.surfsara.nl] => (item=[{u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'}, {u'skip_reason': u'Conditional result was False', u'item': {u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'}, u'skipped': True, u'ansible_loop_var': u'item', u'changed': False}, {u'skip_reason': u'Conditional result was False', u'item': {u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'}, u'skipped': True, u'ansible_loop_var': u'item', u'changed': False}])

TASK [yoda_rulesets : Install requirements for ruleset] ***********************************************************************************************************************************************************************************
ok: [example-yoda.irods.surfsara.nl]

TASK [yoda_rulesets : Ensure all rulesets are in server config] ***************************************************************************************************************************************************************************
ok: [example-yoda.irods.surfsara.nl]

TASK [yoda_rulesets : Ensure Python rules are loaded] *************************************************************************************************************************************************************************************
ok: [example-yoda.irods.surfsara.nl]

TASK [yoda_rulesets : include_tasks] ******************************************************************************************************************************************************************************************************
skipping: [example-yoda.irods.surfsara.nl] => (item={u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'})
fatal: [example-yoda.irods.surfsara.nl]: FAILED! => {"reason": "Could not find or access '/var/ansiRods/yoda/yoda-ansible-1.7/yoda-ruleset.yml' on the Ansible Controller."}

[lwesterhof]

Hi Marius,

Looks like something is out of date. The playbook is looking for yoda-ruleset.yml, which only exists in development and not in release-1.7. What version are you using in this deployment?

fatal: [example-yoda.irods.surfsara.nl]: FAILED! => {"reason": "Could not find or access '/var/ansiRods/yoda/yoda-ansible-1.7/yoda-ruleset.yml' on the Ansible Controller."}

[mk-surf]

Hi Lazlo

Looks like something is out of date. The playbook is looking for yoda-ruleset.yml, which only exists in development and not in release-1.7. What version are you using in this deployment?

Yes, noticed that non-defined/existing file and did not get how/why Ansible was looking for it.
This is 1.7, branch release-1.7 and commit 200fd3f

Env: instance: full, yoda_version: release-1.7, yoda_environment: production

[lwesterhof]

What is the result of 'git show' in your clone of yoda? You are running Ansible from your local machine against example-yoda.irods.surfsara.nl?

[mk-surf]

What is the result of 'git show' in your clone of yoda? You are running Ansible from your local machine against example-yoda.irods.surfsara.nl?

We run ansible from a 'control node' against example-yoda and it looks like the local repo there was not on the right commit (see output from git cmds below).

But now the ruleset doesnt get installed at all and the play fails shortly after:

TASK [yoda_rulesets : Install requirements for ruleset]
<...> 
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/etc/irods/irods-ruleset-uu/requirements.txt

Which is makes sense, because dir /etc/irods/irods-ruleset-uu did indeed not exist at all.

After commenting out the ruleset patch (core_rulesets) from group_vars the ruleset gets installed and the playbook runs OK again.

---

After a 'git pull' :

ansiRods$ [git:release-1.7 u=] show
commit b4f74eba332a14810066f69456ebf7a3534496d9
Author: Lazlo Westerhof <[email protected]>
Date:   Tue Dec 14 10:54:58 2021 +0100

    Fix ip of development environment.

diff --git a/Vagrantfile b/Vagrantfile
index 0b7f932..a083b62 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -16,7 +16,7 @@ NETWORK = "192.168.56."
 NETMASK = "255.255.255.0"

 HOSTS = {
-  "combined" => [NETWORK, CPU, RAM, GUI, BOX],
+  "combined" => [NETWORK+"10", CPU, RAM, GUI, BOX],
 }

 # Hosts for full environment.

ansiRods$ [git:release-1.7 u=] git branch --list -v
  4.2.10      802f918 forced git to overwrite checkout for our other changes
  list        743192c Apache: disable mod_lua
  master      91354e7 Update README.md.
* release-1.7 743192c Apache: disable mod_lua

Before:

ansiRods$ [git:release-1.7 u=] git show
commit 200fd3fba08da31df3ab9a8c94823babf913fd01
Author: Lazlo Westerhof <[email protected]>
Date:   Fri Dec 3 15:47:43 2021 +0100

    Run make install for new ruleset checkout.

diff --git a/roles/yoda_rulesets/tasks/main.yml b/roles/yoda_rulesets/tasks/main.yml
index aee947e..df166e7 100644
--- a/roles/yoda_rulesets/tasks/main.yml
+++ b/roles/yoda_rulesets/tasks/main.yml
@@ -38,7 +38,7 @@
     update: "{{ update_rulesets }}"
   with_items: "{{ extra_rulesets + core_rulesets }}"
   when: "'repo' in item and 'patch' in item"
-  register: repochanges
+  register: patch_repochanges


 - name: Sync rulesets with repos before applying patches
@@ -77,19 +77,22 @@
   when: "'repo' in item and 'patch' in item and item['patch'] not in ['']"


-- name: Find temporary rule files left behind
+- name: Find temporary rule and cache files left behind
   find:
     paths: /etc/irods
-    patterns: '*.re.*'
+    patterns: '*.re.*,*.pyc'
+    recurse: yes
+  when: repochanges.changed or patch_repochanges.changed
   register: temporary_rules


-- name: Remove temporary rule files left behind
+- name: Remove temporary rule and cache files left behind
   become_user: "{{ irods_service_account }}"
   become: yes
   file:
     path: "{{ item.path }}"
     state: absent
+  when: repochanges.changed or patch_repochanges.changed
   with_items:
     - "{{ temporary_rules.files }}"

@@ -104,6 +107,7 @@
   with_together:
     - "{{ extra_rulesets + core_rulesets }}"
     - "{{ repochanges.results }}"
+    - "{{ patch_repochanges.results }}"


 - name: Install requirements for ruleset

git remote -v
origin  https://github.com/UtrechtUniversity/yoda.git (fetch)
origin  https://github.com/UtrechtUniversity/yoda.git (push)

git branch --list -v
  4.2.10      802f918 forced git to overwrite checkout for our other changes
  master      91354e7 Update README.md.
* release-1.7 200fd3f Run make install for new ruleset checkout

[mk-surf]

FYI:
tried again with release-1.8 and now the apply was applied correctly 🥳

TASK [yoda_rulesets : Check out rulesets with local patches from Github] ******************************************************************************************************************************************************************
changed: [example-yoda.irods.surfsara.nl] => (item={u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.8', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'})
skipping: [example-yoda.irods.surfsara.nl] => (item={u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'})

TASK [yoda_rulesets : Sync rulesets with repos before applying patches] *******************************************************************************************************************************************************************
changed: [example-yoda.irods.surfsara.nl -> example-yoda.irods.surfsara.nl] => (item={u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.8', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'})
skipping: [example-yoda.irods.surfsara.nl] => (item={u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'})

TASK [yoda_rulesets : Create patch files for rulesets] ************************************************************************************************************************************************************************************
changed: [example-yoda.irods.surfsara.nl] => (item={u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.8', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'})
skipping: [example-yoda.irods.surfsara.nl] => (item={u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'})

TASK [yoda_rulesets : Apply patches to rulesets] ******************************************************************************************************************************************************************************************
changed: [example-yoda.irods.surfsara.nl] => (item={u'repo': u'https://github.com/UtrechtUniversity/yoda-ruleset.git', u'install_scripts': True, u'name': u'yoda-ruleset', u'version': u'release-1.8', u'ruleset_name': u'rules-uu', u'patch': u'diff --git a/uuGroup.r b/uuGroup.r\nindex ce64b9f..916f1e8 100644\n--- a/uuGroup.r\n+++ b/uuGroup.r\n@@ -870,7 +870,7 @@ uuExternalUser(*userName) {\n \t*nameAndDomain = split(*userName, "@");\n \tif (size(*nameAndDomain) == 2) {\n \t\t*domain = elem(*nameAndDomain, 1);\n-\t\tif (*domain != "uu.nl" && *domain not like "*.uu.nl") {\n+\t\tif (*domain != "") {\n \t\t\t# is external\n \t\t\ttrue;\n \t\t} else {\n'})
skipping: [example-yoda.irods.surfsara.nl] => (item={u'path': u'/etc/irods/core.re', u'install_scripts': False, u'ruleset_name': u'core', u'name': u'core'})