Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC-3872 Argopm Upgrade #45

Draft
wants to merge 11 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
node_modules
.github
.git
.idea
.vscode
.vscode
dist
tests
16 changes: 14 additions & 2 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -1,14 +1,26 @@
{
"parserOptions": {
"ecmaVersion": 2017
"ecmaVersion": 2022,
"sourceType": "module",
"ecmaFeatures": {
"jsx": true,
"modules": true,
"experimentalObjectRestSpread": true
}
},
"env": {
"es6": true,
"node": true,
"jest/globals": true
},
"plugins": ["jest"],
"extends": ["eslint:recommended", "plugin:jest/recommended", "prettier"],
"extends": [
"eslint:recommended",
"plugin:jest/recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"prettier"
],
"rules": {
"no-unused-vars": "warn"
}
Expand Down
37 changes: 19 additions & 18 deletions .github/workflows/npm-publish.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,42 @@
name: npm-publish
on:
push:
branches:
- master # Change this to your default branch
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
- '[0-9]+.[0-9]+.[0-9]+rc[0-9]+'

jobs:
npm-publish:
name: npm-publish
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@master

- name: Set up Node.js
uses: actions/setup-node@master
with:
node-version: 10.0.0
node-version: 18

- name: Publish if version has been updated
uses: pascalgn/npm-publish-action@06e0830ea83eea10ed4a62654eeaedafb8bf50fc
with: # All of theses inputs are optional
tag_name: "v%s"
tag_message: "v%s"
commit_pattern: "^Release (\\S+)"
workspace: "."
env: # More info about the environment variables in the README
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Leave this as is, it's automatically generated
NPM_AUTH_TOKEN: ${{ secrets.NPM_AUTH_TOKEN }} # You need to set this in your repo settings
- uses: actions/[email protected]
uses: JS-DevTools/npm-publish@v1
with:
tag: "v%s"
token: ${{ secrets.NPM_AUTH_TOKEN }

- uses: actions/checkout@v3
with:
fetch-depth: 0

- name: 'Get Previous tag'
id: previoustag
uses: "WyriHaximus/github-action-get-previous-tag@master"

- run: echo "REPOSITORY_NAME=`echo "$GITHUB_REPOSITORY" | awk -F / '{print $2}' | sed -e "s/:refs//"`" >> $GITHUB_ENV
shell: bash
- name: Set up Buildx
id: buildx
uses: docker/setup-buildx-action@v1

- name: Set up Buildx
uses: docker/setup-buildx-action@v2

- name: Login to GitHub Registry
uses: docker/login-action@v1
Expand All @@ -44,8 +46,7 @@ jobs:
password: ${{ secrets.MY_PAT }}

- name: Build and push docker image
id: docker_build
uses: docker/build-push-action@v2
uses: docker/build-push-action@v3
with:
context: .
file: ./Dockerfile
Expand Down
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
lib/static
.github/
.vscode/
dist/
package-lock.json
package.json
16 changes: 11 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
FROM node:14-alpine
ARG BUILD_IMAGE=node:18-alpine

RUN mkdir /app
FROM $BUILD_IMAGE as builder
WORKDIR /app
COPY . /app
RUN npm install && npm run build

FROM $BUILD_IMAGE as runtime
WORKDIR /app

COPY . /app
COPY --from=builder /app/dist/argopm.mjs /app/
COPY ./lib/static/ /app/static/

RUN npm link
# Shelljs, a dep of K8s client, does not work well with bundlers
RUN npm i shelljs

ENTRYPOINT ["argopm"]
ENTRYPOINT [ "./argopm.mjs" ]
119 changes: 53 additions & 66 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,82 +1,69 @@
# argopm - A package manager for Argo
# Getting started with argopm - Atlan's Package Manager

This is an Argo package manager that helps you find, install and publish packages to your Argo cluster.
This package was bootstrapped using argopm

![npm-publish-status](https://github.com/atlanhq/argopm/actions/workflows/npm-publish.yml/badge.svg)
- Package Name: argopm

### Requirements
## Package Structure

1. Node.js `v16.9.0`
```
.
├── README.md
├── index.js
├── package.json
├── configmaps
│   ├── README.md
│   ├── default-semaphore.yaml
│   └── default.yaml
├── cronworkflows
│   ├── README.md
│   └── default.yaml
├── pipelines
│   ├── README.md
│   └── default.yaml
├── secrets
│   ├── README.md
│   └── default.yaml
├── static
│   ├── README.md
│   └── data.json
└── templates
├── README.md
└── default.yaml

6 directories, 16 files
```

Please ensure you have `kubectl` access to the Kubernetes cluster where Argo is deployed and the AWS credentials for writing files to AWS S3.
## Available Commands

### Installation
- `argopm install`
- `argopm list`
- `argopm run`
- `argopm info`

```bash
npm i -g argopm
```
For more details on these commands run `argom --help`

### Usage

```bash
$ argopm --help
argopm <command>

Commands:
argopm install <package> Install a package. Package name can be of the format package@version [aliases: i]
argopm info <package> [template] Get info of the installed package or a specific template in the package
argopm run <package> [template] Run the package or the package template. Pass in arguments using --
argopm uninstall <package> Uninstall a package. Uninstalls all dependencies associated with the package. [aliases: u, r]
argopm init Initializes an Argo package inside the current working directory
argopm list List all the packages installed in the namespace [aliases: l]

Options:
--version Show version number [boolean]
--namespace, -n Kubernetes namespace. Packages will be installed in this namespace [string] [default: "argo"]
--registry, -r Argo Package Registry [string] [default: "https://packages.atlan.com"]
--pipeline, -p Enable Argo Pipeline type [boolean] [default: false]
--cluster, -c Install the template at cluster level [boolean] [default: false]
--help Show help [boolean]
```
## Pre-requisites

### Package Structure
- To run `argopm install` you must have `kubectl` access to the cluter where Argo is installed
- Files added in the `static` subdirectory are uploaded to the configured S3 artifactory bucket, make sure you have
the AWS credentials setup in your shell to upload static files.

This is the structure of a new package created with `argopm`. The
## Grafana Dashboards

```bash
.
├── README.md
├── index.js
├── package.json
├── configmaps # configmaps to be installed in the k8s cluster
│ ├── package-config.yaml
│ └── semaphore-config.yaml
├── cronworkflows # Cron Workflows to be installed on Argo
│ └── package-cronworkflow.yaml
├── pipelines # Argo Dataflow pipelines to be installed
│ └── package-pipeline.yaml
├── secrets # Secrets to be created in the k8s cluster
│ └── package-secret.yaml
├── static # Static data to be uploaded on S3
│ └── data.json
└── templates # Workflow templates to be installed on Argo
└── package-template.yaml

6 directories, 10 files
```
Argopm supports declaring and uploading grafana dashboards as a part of the package. Files available in `dashboards/grafana/` will be uploaded to
a specified Grafana instance.

### Static Files
Declare the following environment variables if you want to upload dashboards to grafana. You can put these in the `.env` file too for
local development.

Everything present in the `static` subdirectory of a package will be uploaded to AWS S3 with the following location prefix :
- `GRAFANA_URL` - URL of the grafana instance
- `GRAFANA_API_TOKEN` - Grafana API Token

```
<bucket-name>/argo-artifacts/argopm/<package-name>/<version>/static/
```
A post request is sent to `GRAFANA_URL` with the JSON file content in `dashboards/grafana` and `GRAFANA_API_TOKEN` is sent in the `Authorization`
header as a bearer token. Refer to the [Grafana HTTP API Docs](https://grafana.com/docs/grafana/latest/http_api/) for more information.

### Salient Features:
## Helpful Documentation

1. Built as a `npm` package
2. Works on marketplace built on [verdaccio](https://verdaccio.org). Verdaccio is an open-source _npm-like_ marketplace
3. Uses the K8s Custom resources to install packages into your Argo cluster
4. Support for uploading static files to the artifactory (available for AWS S3)
5. Manages versions and installed packages using K8s labels on Argo workflow templates
- https://argoproj.github.io/argo-workflows/
- https://www.npmjs.com/package/argopm
Loading