startup-script

Description

This module creates a startup script that will execute a list of runners in the order they are specified. The runners are copied to a GCS bucket at deployment time and then copied into the VM as they are executed after startup.

Each runner receives the following attributes:

• destination: (Required) The name of the file at the destination VM. If an absolute path is provided, the file will be copied to that path, otherwise the file will be created in a temporary folder and deleted once the startup script runs.

• type: (Required) The type of the runner, one of the following:

  • shell: The runner is a shell script and will be executed once copied to the destination VM.

  • ansible-local: The runner is an ansible playbook and will run on the VM with the following command line flags:

    ansible-playbook --connection=local --inventory=localhost, \
      --limit localhost <<DESTINATION>>

  • data: The data or file specified will be copied to <<DESTINATION>>. No action will be performed after the data is staged. This data can be used by subsequent runners or simply made available on the VM for later use.

• content: (Optional) Content to be uploaded and, if type is either shell or ansible-local, executed. Must be defined if source is not.

• source: (Optional) A path to the file or data you want to upload. Must be defined if content is not. The source path is relative to the deployment group directory. Scripts distributed as part of modules should start with modules/ followed by the name of the module used (not to be confused with the module ID) and the path to the script. The format is shown below:

  source: ./modules/<<MODULE_NAME>>/<<SCRIPT_NAME>>
  

  For more examples with context, see the example blueprint snippet. To reference any other source file, an absolute path must be used.

• args: (Optional) Arguments to be passed to shell or ansible-local runners. For shell runners, these will be passed as arguments to the script when it is executed. For ansible-local runners, they will be appended to a list of default arguments that invoke ansible-playbook on the localhost. Thereforeargs should not include any arguments that alter this behavior, such as --connection, --inventory, or --limit.

Runner dependencies

ansible-local runners require Ansible to be installed in the VM before running. To support other playbook runners in the HPC Toolkit, we install version 2.11 of ansible-core as well as the larger package of collections found in ansible version 4.10.0.

If an ansible-local runner is found in the list supplied to this module, a script to install Ansible will be prepended to the list of runners. This behavior can be disabled by setting var.prepend_ansible_installer to false. This script will do the following at VM startup:

• Install system-wide python3 if not already installed using system package managers (yum, apt-get, etc)
• Install python3-distutils system-wide in debian and ubuntu based environments. This can be a missing dependency on system installations of python3 for installing and upgrading pip.
• Install system-wide pip3 if not already installed and upgrade pip3 if the version is not at least 18.0.
• Install and create a virtual environment located at /usr/local/ghpc-venv.
• Install ansible into this virtual environment if the current version of ansible is not version 2.11 or higher.

To use the virtual environment created by this script, you can activate it by running the following commmand on the VM:

source /usr/local/ghpc-venv/bin/activate

You may also need to provide the correct python interpreter as the python3 binary in the virtual environment. This can be done by adding the following flag when calling ansible-playbook:

-e ansible_python_interpreter=/usr/local/ghpc-venv/bin/activate

NOTE: ansible-playbook and other ansible command line tools will only be accessible from the command line (and in your PATH variable) after activating this environment.

Staging the runners

Runners will be uploaded to a GCS bucket. This bucket will be created by this module and named as ${var.deployment_name}-startup-scripts-${random_id}. VMs using the startup script created by this module will pull the runners content from a GCS bucket and therefore must have access to GCS.

NOTE: To ensure access to GCS, set the following OAuth scope on the instance using the startup scripts: https://www.googleapis.com/auth/devstorage.read_only.

This is set as a default scope in the vm-instance, SchedMD-slurm-on-gcp-login-node and SchedMD-slurm-on-gcp-controller modules

Tracking startup script execution

For more information on how to use startup scripts on Google Cloud Platform, please refer to this document.

To debug startup scripts from a Linux VM created with startup script generated by this module:

sudo DEBUG=1 google_metadata_script_runner startup

To view ouputs from a Linux startup script, run:

sudo journalctl -u google-startup-scripts.service

Example

- id: startup
  source: ./modules/scripts/startup-script
  settings:
    runners:
      # Some modules such as filestore have runners as outputs for convenience:
      - $(homefs.install_nfs_client_runner)
      # These runners can still be created manually:
      # - type: shell
      #   destination: "modules/filestore/scripts/install_nfs_client.sh"
      #   source: "modules/filestore/scripts/install_nfs_client.sh"
      - type: ansible-local
        destination: "modules/filestore/scripts/mount.yaml"
        source: "modules/filestore/scripts/mount.yaml"
      - type: data
        source: /tmp/foo.tgz
        destination: /tmp/bar.tgz
      - type: shell
        destination: "decompress.sh"
        content: |
          #!/bin/sh
          echo $2
          tar zxvf /tmp/$1 -C /
        args: "bar.tgz 'Expanding file'"

- id: compute-cluster
  source: ./modules/compute/vm-instance
  use: [homefs, startup]

License

Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name	Version
terraform	>= 0.14.0
google	>= 3.83
local	>= 2.0.0, < 2.2.0
random	~> 3.0

Providers

Name	Version
google	>= 3.83
local	>= 2.0.0, < 2.2.0
random	~> 3.0

Modules

No modules.

Resources

Name	Type
google_storage_bucket.configs_bucket	resource
google_storage_bucket_object.scripts	resource
local_file.debug_file	resource
random_id.resource_name_suffix	resource

Inputs

Name	Description	Type	Default	Required
debug_file	Path to an optional local to be written with 'startup_script'.	string	null	no
deployment_name	Name of the HPC deployment, used to name GCS bucket for startup scripts.	string	n/a	yes
labels	Labels for the created GCS bucket. List key, value pairs.	any	n/a	yes
prepend_ansible_installer	Prepend Ansible installation script if any of the specified runners are of type ansible-local	bool	true	no
project_id	Project in which the HPC deployment will be created	string	n/a	yes
region	The region to deploy to	string	n/a	yes
runners	List of runners to run on remote VM. Runners can be of type ansible-local, shell or data. A runner must specify one of 'source' or 'content'. All runners must specify 'destination'. If 'destination' does not include a path, it will be copied in a temporary folder and deleted after running. Runners may also pass 'args', which will be passed as argument to shell runners only.	list(map(string))	[]	no

Outputs

Name	Description
compute_startup_script	script to load and run all runners, as a string value. Targets the inputs for the slurm controller.
controller_startup_script	script to load and run all runners, as a string value. Targets the inputs for the slurm controller.
startup_script	script to load and run all runners, as a string value.