Skip to content

Commit

Permalink
merge letter change in
Browse files Browse the repository at this point in the history
  • Loading branch information
Alex Fuerst committed Dec 6, 2023
2 parents 76ca73e + b6b7687 commit 42de8d9
Show file tree
Hide file tree
Showing 183 changed files with 198 additions and 202 deletions.
8 changes: 0 additions & 8 deletions .idea/.gitignore

This file was deleted.

15 changes: 10 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
Ilúvatar is an open Serverless platform built with the goal of jumpstarting and streamlining FaaS research.
It provides a system that is easy and consistent to use, highly modifiable, and directly reports experimental results.

<!-- <img src="./imgs/logo1.jpeg" alt="Ilúvatar orchestrating functions" width="300"/> -->
<center>
<img src="./imgs/logo1.png" alt="Ilúvatar logo" width="300"/>
</center>

## Try it Out

Expand All @@ -16,20 +18,20 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install non-apt dependencies and build Ilúvatar, assumes python3 exists
cd src/Ilúvatar/docs/examples
./sample_setup.sh
sudo ../../target/debug/ilúvatar_worker -c ../../ilúvatar_worker/src/worker.dev.json &
sudo ../../target/debug/iluvatar_worker -c ../../iluvatar_worker/src/worker.dev.json &
worker_pid=$(echo $!)
```

Register a function with the worker.

```sh
../../target/debug/ilúvatar_worker_cli --address "127.0.0.1" --port 8079 register --name "hello" --version 1 --image "docker.io/alfuerst/hello-iluvatar-action:latest" --memory 128 --cpu 1
../../target/debug/iluvatar_worker_cli --address "127.0.0.1" --port 8079 register --name "hello" --version 1 --image "docker.io/alfuerst/hello-iluvatar-action:latest" --memory 128 --cpu 1
```

Invoke the newly registered function, passing custom arguments.

```sh
../../target/debug/ilúvatar_worker_cli --address "127.0.0.1" --port 8079 invoke --name "hello" --version 1 -a name=`whoami`
../../target/debug/iluvatar_worker_cli --address "127.0.0.1" --port 8079 invoke --name "hello" --version 1 -a name=`whoami`
```

Kill the worker running in the background.
Expand Down Expand Up @@ -63,12 +65,15 @@ It also does not have the research-first features implemented in Ilúvatar.

See Ilúvatar in action for research [right here](./src/Ilúvatar/docs/RESEARCH.md)

## More documentation
## More documentation / Questions

Ilúvatar supports a large variety of customization in configuration and setup, and methods of load generation and experimentation.
It is easy to create custom functions that run new Python programs, [information is in this document](./src/Ilúvatar/docs/FUNCTIONS.md#preparing-code-to-be-functions).
Detailed documentation can be found [closer to the code](./src/Ilúvatar/README.md).

If you have questions, want help working with Ilúvatar, or anything else, we have a [Discord server](https://discord.gg/r4CZ7bZhmJ).
Definitely gives faster answers than opening an issue for a question.

## Why 'Ilúvatar'?

The name 'Ilúvatar' comes from J.R.R. Tolkien's mythology, published in [*The Silmarillion*](https://tolkiengateway.net/wiki/The_Silmarillion), a history of the world prior to the events of his *Hobbit* and *Lord of the Rings* books.
Expand Down
Binary file added imgs/logo1.odg
Binary file not shown.
Binary file added imgs/logo1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions src/Ilúvatar/.gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
ilúvatar_worker_cli/src/worker.dev*json
ilúvatar_worker_cli/tests/resources/worker.dev*json
ilúvatar_worker/src/worker.dev*json
ilúvatar_worker_library/tests/resources/worker.dev*json
ilúvatar_controller/src/controller.dev*json
iluvatar_worker_cli/src/worker.dev*json
iluvatar_worker_cli/tests/resources/worker.dev*json
iluvatar_worker/src/worker.dev*json
iluvatar_worker_library/tests/resources/worker.dev*json
iluvatar_controller/src/controller.dev*json
target/
**.swp
.vscode/
16 changes: 8 additions & 8 deletions src/Ilúvatar/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
[workspace]
members = [
"ilúvatar_worker",
"ilúvatar_worker_cli",
"ilúvatar_load_gen",
"ilúvatar_controller_library",
"ilúvatar_controller",
"ilúvatar_library",
"ilúvatar_worker_library",
"ilúvatar_energy_mon"
"iluvatar_worker",
"iluvatar_worker_cli",
"iluvatar_load_gen",
"iluvatar_controller_library",
"iluvatar_controller",
"iluvatar_library",
"iluvatar_worker_library",
"iluvatar_energy_mon"
]
resolver = "2"

Expand Down
2 changes: 1 addition & 1 deletion src/Ilúvatar/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ tiny:
@rustup toolchain install nightly
@rustup component add rust-src --toolchain nightly
@cargo +nightly build -Z build-std=std,panic_abort --target x86_64-unknown-linux-gnu -Z build-std-features=panic_immediate_abort --profile tiny
@upx --best --lzma ./target/x86_64-unknown-linux-gnu/tiny/ilúvatar_*
@upx --best --lzma ./target/x86_64-unknown-linux-gnu/tiny/iluvatar_*
debug:
@echo "Building debug"
@cargo build -j $(NPROCS) $(cargo_args)
Expand Down
8 changes: 4 additions & 4 deletions src/Ilúvatar/ansible/controller.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
vars:
host_group: "{{ groups['controller'] }}"
host_group_size: "{{ host_group | length }}"
bin_dir: "/tmp/ilúvatar/bin"
bin_dir: "/tmp/iluvatar/bin"
exe_dest: "{{bin_dir}}/{{ controller.bin_name }}"
config_dest: "{{bin_dir}}/{{ controller.config_name }}"
__worker_address: "{{ worker_address | default(servers[ansible_host].internal_ip) }}"
Expand Down Expand Up @@ -36,15 +36,15 @@

- name: copy controller executable
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{controller.bin_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{controller.bin_name}}"
dest: "{{exe_dest}}"
mode: "preserve"
when: mode == "deploy"
become: yes

- name: copy controller config
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{controller.config_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{controller.config_name}}"
dest: "{{config_dest}}"
mode: "preserve"
when: mode == "deploy"
Expand All @@ -64,7 +64,7 @@
"ILUVATAR_CONTROLLER__name": "{{ controller.host }}"
"ILUVATAR_CONTROLLER__port": "{{ controller.port }}"
"ILUVATAR_CONTROLLER__address": "{{ __worker_address | default('localhost') }}"
"ILUVATAR_CONTROLLER__logging__directory" : "{{ controller_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_CONTROLLER__logging__directory" : "{{ controller_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_CONTROLLER__logging__level" : "{{ controller_log_level | default('info') }}"
"ILUVATAR_CONTROLLER__load_balancer__algorithm" : "{{ controller.algorithm }}"
"ILUVATAR_CONTROLLER__load_balancer__load_metric" : "{{ controller.load_metric }}"
Expand Down
6 changes: 3 additions & 3 deletions src/Ilúvatar/ansible/group_vars/all.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
ilúvatar_home: "{{ lookup('env', 'ILU_HOME') | default(playbook_dir ~ '/..', true) }}"
iluvatar_home: "{{ lookup('env', 'ILU_HOME') | default(playbook_dir ~ '/..', true) }}"
mode: deploy
target: release

controller:
bin_name: "ilúvatar_controller"
bin_name: "iluvatar_controller"
config_name: "controller.json"
host: "{{ hostvars[ groups['controller']|first ].ansible_host }}"
port: "{{ controller_port | default(8089) }}"
algorithm: "{{ controller_algorithm | default('LeastLoaded') }}"
load_metric: "{{ controller_load_metric | default('loadavg') }}"

worker:
bin_name: "ilúvatar_worker"
bin_name: "iluvatar_worker"
config_name: "worker.json"
port: "{{ worker_port | default(8070) }}"

Expand Down
10 changes: 5 additions & 5 deletions src/Ilúvatar/ansible/worker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
host_group: "{{ groups['workers'] }}"
worker_index_base: 0
assigned_port: "{{ (worker.port | int + host_group.index(inventory_hostname)) }}"
bin_dir: "/tmp/ilúvatar/bin"
bin_dir: "/tmp/iluvatar/bin"
exe_dest: "{{bin_dir}}/{{ worker.bin_name }}"
config_dest: "{{bin_dir}}/{{ inventory_hostname }}.json"
__worker_address: "{{ worker_address | default(servers[ansible_host].internal_ip) }}"
Expand All @@ -24,14 +24,14 @@

- name: copy worker executable
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{worker.bin_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{worker.bin_name}}"
dest: "{{exe_dest}}"
mode: "preserve"
become: yes

- name: copy worker config
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{worker.config_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{worker.config_name}}"
dest: "{{config_dest}}"
mode: "preserve"
become: yes
Expand Down Expand Up @@ -96,7 +96,7 @@
"ILUVATAR_WORKER__port": "{{ assigned_port }}"
"ILUVATAR_WORKER__address" : "{{ __worker_address }}"
"ILUVATAR_WORKER__logging__basename": "worker_{{ inventory_hostname }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_WORKER__logging__level": "{{ worker_log_level | default('info') }}"
"ILUVATAR_WORKER__load_balancer_url" : "http://{{ controller.host }}:{{ controller.port }}"
"ILUVATAR_WORKER__timeout_sec" : "{{ worker_timeout_sec | default(6000) }}"
Expand Down Expand Up @@ -135,7 +135,7 @@
"ILUVATAR_WORKER__energy__ipmi_freq_ms" : "{{ worker_ipmi_log_freq_ms | default(0) }}"
"ILUVATAR_WORKER__energy__ipmi_pass_file" : "{{ worker_ipmi_pass_file | default('') }}"
"ILUVATAR_WORKER__energy__ipmi_ip_addr" : "{{ worker_ipmi_ip_addr | default(servers[ansible_host].ipmi_ip) }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
# Energy cap config
"ILUVATAR_WORKER__energy_cap__power_cap" : "{{ worker_power_cap | default(0.0) }}"
"ILUVATAR_WORKER__energy_cap__power_cap_version" : "{{ worker_power_cap_version | default('V0') }}"
Expand Down
14 changes: 7 additions & 7 deletions src/Ilúvatar/ansible/worker_laptop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
host_group: "{{ groups['workers'] }}"
worker_index_base: 0
assigned_port: "{{ (worker.port | int + host_group.index(inventory_hostname)) }}"
bin_dir: "/tmp/ilúvatar/bin"
bin_dir: "/tmp/iluvatar/bin"
exe_dest: "{{bin_dir}}/{{ worker.bin_name }}"
config_dest: "{{bin_dir}}/{{ inventory_hostname }}.json"
__worker_address: "{{ worker_address | default(servers[ansible_host].internal_ip) }}"
Expand Down Expand Up @@ -46,14 +46,14 @@

- name: copy worker executable
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{worker.bin_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{worker.bin_name}}"
dest: "{{exe_dest}}"
mode: "preserve"
become: yes

- name: copy worker config
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/{{target}}/{{worker.config_name}}"
src: "{{iluvatar_home}}/target/{{target}}/{{worker.config_name}}"
dest: "{{config_dest}}"
mode: "preserve"
become: yes
Expand Down Expand Up @@ -94,7 +94,7 @@
"ILUVATAR_WORKER__port": "{{ assigned_port }}"
"ILUVATAR_WORKER__address" : "{{ __worker_address }}"
"ILUVATAR_WORKER__logging__basename": "worker_{{ inventory_hostname }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_WORKER__logging__level": "{{ worker_log_level | default('info') }}"
"ILUVATAR_WORKER__load_balancer_url" : "http://{{ controller.host }}:{{ controller.port }}"
# Container resources
Expand All @@ -110,7 +110,7 @@
"ILUVATAR_WORKER__energy__ipmi_freq_ms" : "{{ worker_ipmi_log_freq_ms | default(0) }}"
"ILUVATAR_WORKER__energy__ipmi_pass_file" : "{{ worker_ipmi_pass_file | default('') }}"
"ILUVATAR_WORKER__energy__ipmi_ip_addr" : "{{ worker_ipmi_ip_addr | default(servers[ansible_host].ipmi_ip) }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
register: worker_output
- debug:
var: worker_output
Expand Down Expand Up @@ -145,7 +145,7 @@
"ILUVATAR_WORKER__port": "{{ assigned_port }}"
"ILUVATAR_WORKER__address" : "{{ __worker_address }}"
"ILUVATAR_WORKER__logging__basename": "worker_{{ inventory_hostname }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_WORKER__logging__level": "{{ worker_log_level | default('info') }}"
"ILUVATAR_WORKER__load_balancer_url" : "http://{{ controller.host }}:{{ controller.port }}"
# Container resources
Expand All @@ -161,7 +161,7 @@
"ILUVATAR_WORKER__energy__ipmi_freq_ms" : "{{ worker_ipmi_log_freq_ms | default(0) }}"
"ILUVATAR_WORKER__energy__ipmi_pass_file" : "{{ worker_ipmi_pass_file | default('') }}"
"ILUVATAR_WORKER__energy__ipmi_ip_addr" : "{{ worker_ipmi_ip_addr | default(servers[ansible_host].ipmi_ip) }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
register: worker_output
- debug:
var: worker_output
Expand Down
10 changes: 5 additions & 5 deletions src/Ilúvatar/ansible/worker_perf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
host_group: "{{ groups['workers'] }}"
worker_index_base: 0
assigned_port: "{{ (worker.port | int + host_group.index(inventory_hostname)) }}"
bin_dir: "/tmp/ilúvatar/bin"
bin_dir: "/tmp/iluvatar/bin"
exe_dest: "{{bin_dir}}/{{ worker.bin_name }}"
config_dest: "{{bin_dir}}/{{ inventory_hostname }}.json"
__worker_address: "{{ worker_address | default(servers[ansible_host].internal_ip) }}"
Expand Down Expand Up @@ -54,14 +54,14 @@

- name: copy worker executable
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/debug/{{worker.bin_name}}"
src: "{{iluvatar_home}}/target/debug/{{worker.bin_name}}"
dest: "{{exe_dest}}"
mode: "preserve"
become: yes

- name: copy worker config
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/debug/{{worker.config_name}}"
src: "{{iluvatar_home}}/target/debug/{{worker.config_name}}"
dest: "{{config_dest}}"
mode: "preserve"
become: yes
Expand Down Expand Up @@ -111,7 +111,7 @@
"ILUVATAR_WORKER__port": "{{ assigned_port }}"
"ILUVATAR_WORKER__address" : "{{ __worker_address }}"
"ILUVATAR_WORKER__logging__basename": "worker_{{ inventory_hostname }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_WORKER__logging__level": "{{ worker_log_level | default('info') }}"
"ILUVATAR_WORKER__load_balancer_url" : "http://{{ controller.host }}:{{ controller.port }}"
"ILUVATAR_WORKER__timeout_sec" : "{{ worker_timeout_sec | default(6000) }}"
Expand All @@ -134,7 +134,7 @@
"ILUVATAR_WORKER__energy__ipmi_freq_ms" : "{{ worker_ipmi_log_freq_ms | default(0) }}"
"ILUVATAR_WORKER__energy__ipmi_pass_file" : "{{ worker_ipmi_pass_file | default('') }}"
"ILUVATAR_WORKER__energy__ipmi_ip_addr" : "{{ worker_ipmi_ip_addr | default(servers[ansible_host].ipmi_ip) }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
# Invocation config
"ILUVATAR_WORKER__invocation__queue_policy" : "{{ worker_queue_policy | default('fcfs') }}"
"ILUVATAR_WORKER__invocation__retries" : "{{ worker_invoke_retries | default(0) }}"
Expand Down
10 changes: 5 additions & 5 deletions src/Ilúvatar/ansible/worker_perf_affinity.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
host_group: "{{ groups['workers'] }}"
worker_index_base: 0
assigned_port: "{{ (worker.port | int + host_group.index(inventory_hostname)) }}"
bin_dir: "/tmp/ilúvatar/bin"
bin_dir: "/tmp/iluvatar/bin"
exe_dest: "{{bin_dir}}/{{ worker.bin_name }}"
config_dest: "{{bin_dir}}/{{ inventory_hostname }}.json"
__worker_address: "{{ worker_address | default(servers[ansible_host].internal_ip) }}"
Expand Down Expand Up @@ -54,14 +54,14 @@

- name: copy worker executable
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/debug/{{worker.bin_name}}"
src: "{{iluvatar_home}}/target/debug/{{worker.bin_name}}"
dest: "{{exe_dest}}"
mode: "preserve"
become: yes

- name: copy worker config
ansible.builtin.copy:
src: "{{ilúvatar_home}}/target/release/{{worker.config_name}}"
src: "{{iluvatar_home}}/target/release/{{worker.config_name}}"
dest: "{{config_dest}}"
mode: "preserve"
become: yes
Expand Down Expand Up @@ -111,7 +111,7 @@
"ILUVATAR_WORKER__port": "{{ assigned_port }}"
"ILUVATAR_WORKER__address" : "{{ __worker_address }}"
"ILUVATAR_WORKER__logging__basename": "worker_{{ inventory_hostname }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__logging__directory": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
"ILUVATAR_WORKER__logging__level": "{{ worker_log_level | default('info') }}"
"ILUVATAR_WORKER__load_balancer_url" : "http://{{ controller.host }}:{{ controller.port }}"
"ILUVATAR_WORKER__timeout_sec" : "{{ worker_timeout_sec | default(6000) }}"
Expand All @@ -135,7 +135,7 @@
"ILUVATAR_WORKER__energy__ipmi_freq_ms" : "{{ worker_ipmi_log_freq_ms | default(0) }}"
"ILUVATAR_WORKER__energy__ipmi_pass_file" : "{{ worker_ipmi_pass_file | default('') }}"
"ILUVATAR_WORKER__energy__ipmi_ip_addr" : "{{ worker_ipmi_ip_addr | default(servers[ansible_host].ipmi_ip) }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/ilúvatar/logs') }}"
"ILUVATAR_WORKER__energy__log_folder": "{{ worker_log_dir | default('/tmp/iluvatar/logs') }}"
register: worker_output

- name: debug print
Expand Down
8 changes: 4 additions & 4 deletions src/Ilúvatar/docs/ANSIBLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ If using ssh, Ansible will expect an ssh key to be set up that allow connection

## Configuration via Ansible

Most of the configuration uses the default json files for both [controller](../ilúvatar_controller/src/controller.json) and [worker](../ilúvatar_worker/src/worker.json).
Most of the configuration uses the default json files for both [controller](../iluvatar_controller/src/controller.json) and [worker](../iluvatar_worker/src/worker.json).
Configuration of both can be overloaded at _runtime_ using environment variables as described in both [worker](./WORKER.md) and [controller](./CONTROLLER.md) sections, and Ansible can take advantage of that.
The tasks that start the executables and set their environment, allowing injection through Ansible.
The startup task (`run worker executable`) for the [worker](../ansible/worker.yml) has a number of examples of this.
Expand Down Expand Up @@ -63,15 +63,15 @@ A simple command to run Ilúvatar on the local machine.

```sh
# clean the environment to remove any possible leftover parts
ansible-playbook -i environments/local/hosts.ini ilúvatar.yml -e mode=clean -e "@./group_vars/host_addresses.yml"
ansible-playbook -i environments/local/hosts.ini iluvatar.yml -e mode=clean -e "@./group_vars/host_addresses.yml"
# Deploy the system
ansible-playbook -i environments/local/hosts.ini ilúvatar.yml -e mode=deploy -e "@./group_vars/host_addresses.yml"
ansible-playbook -i environments/local/hosts.ini iluvatar.yml -e mode=deploy -e "@./group_vars/host_addresses.yml"
```

### Tear down

To clean up services without starting them again, just pass `mode=clean`.

```sh
ansible-playbook -i environments/local/hosts.ini ilúvatar.yml -e mode=clean -e "@./group_vars/host_addresses.yml"
ansible-playbook -i environments/local/hosts.ini iluvatar.yml -e mode=clean -e "@./group_vars/host_addresses.yml"
```
Loading

0 comments on commit 42de8d9

Please sign in to comment.