Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Secrets SDK with Rust keyring bindings #222

Merged
merged 46 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
a85090c
WIP Start adding secrets sdk
t1m0thyj Oct 30, 2023
c0fbaec
feat(secrets): Create PoC Python bindings that use secrets-core
traeok Oct 31, 2023
9e60ae3
WIP Move secrets_core dep to CLI repo
t1m0thyj Oct 31, 2023
52a465e
Fix working directory in GitHub workflow
t1m0thyj Oct 31, 2023
dcf52a4
Try adding script that configures cross
t1m0thyj Oct 31, 2023
c5401a4
Add executable permission to shell script
t1m0thyj Oct 31, 2023
6de5e87
Install libsecret-1-dev in cross script
t1m0thyj Oct 31, 2023
1e2f591
Try to use Ubuntu for cross-compilation
t1m0thyj Oct 31, 2023
924018b
Try again to fix architectures
t1m0thyj Oct 31, 2023
8a92ece
Try using GITHUB_ENV for cross-compile options
t1m0thyj Nov 1, 2023
36e8b10
Fix passing env vars to Docker container
t1m0thyj Nov 1, 2023
1e01a09
Forgot to pass target to configure-cross script
t1m0thyj Nov 1, 2023
1023951
Try to fix env vars in docker options
t1m0thyj Nov 1, 2023
64d671b
Don't set rust flags in configure-cross script
t1m0thyj Nov 1, 2023
d635e2f
Install pkg-config in Docker containers
t1m0thyj Nov 1, 2023
c1c9b7e
secrets: update errors to use secrets-core error info
traeok Nov 1, 2023
bebffa5
Force amd64 version of pkg-config
t1m0thyj Nov 1, 2023
03ef98b
Try removing CROSS_DEB_ARCH
t1m0thyj Nov 1, 2023
6f2d47c
Try to fix rustflags env var definition
t1m0thyj Nov 1, 2023
1522236
Remove quotes from vars eched to GITHUB_ENV
t1m0thyj Nov 1, 2023
bdd9ce5
Try to install libsecret for different arch
t1m0thyj Nov 1, 2023
505de34
Ignore missing apt packages
t1m0thyj Nov 1, 2023
f740f97
Ignore apt-get update errors
t1m0thyj Nov 1, 2023
15bd9d5
Try to fix apt sources for cross-compile
t1m0thyj Nov 1, 2023
78a6660
Add libglib2.0-dev package for s390x
t1m0thyj Nov 1, 2023
4bce88e
Fix invalid PKG_CONFIG_PATH
t1m0thyj Nov 1, 2023
d6f512c
Add py3.7 wheel and remove pypy ones
t1m0thyj Nov 1, 2023
c977594
Make keyring a submodule of secrets sdk
t1m0thyj Nov 2, 2023
7f588e9
List interpreters for windows and macos
t1m0thyj Nov 2, 2023
7524876
Add windows arm64 build
t1m0thyj Nov 2, 2023
3b8e6c5
Try abi3-py37 feature to fix windows arm64 build
t1m0thyj Nov 2, 2023
5eca570
Restructure secrets sdk as namespace package
t1m0thyj Nov 3, 2023
a6e58c9
Replace keyring package with secrets sdk
t1m0thyj Nov 3, 2023
7bff9bb
Try to fix maturin and add py3.12
t1m0thyj Nov 3, 2023
3e798cd
Fix unit tests for credential manager
t1m0thyj Nov 3, 2023
bab3ed5
Update flake8 version and change workflow triggers
t1m0thyj Nov 3, 2023
71cbbce
Add Rust to ReadTheDocs config
t1m0thyj Nov 3, 2023
3114108
Remove secrets sdk from doc requirements
t1m0thyj Nov 3, 2023
70d62ee
Remove logging call in credential manager
t1m0thyj Nov 3, 2023
82eaf4a
Enable release workflows to publish prerelease to PyPI
t1m0thyj Nov 6, 2023
9f676db
Fix secrets sdk version constraint
t1m0thyj Nov 6, 2023
fa37fce
Add package metadata and update changelog
t1m0thyj Nov 6, 2023
f2f361a
Fix Python binding and split out _set_credential method
t1m0thyj Nov 6, 2023
917ebd7
Fix nul byte not trimmed from win32 creds
t1m0thyj Nov 7, 2023
8ea8fb4
Merge pull request #223 from zowe/ci/publish-dev-release
t1m0thyj Nov 10, 2023
f21170e
Update commit hash for secrets_core package
t1m0thyj Nov 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/dev-release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Zowe SDK Release

on:
pull_request_target:
types:
- closed
branches:
- main

jobs:
release:
if: ${{ github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release-dev') }}
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
with:
ref: main
token: ${{ secrets.ZOWE_ROBOT_TOKEN }}

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.10"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

- name: Increment dev version
id: update-version
shell: python
run: |
import sys
sys.path.append("src")
from _version import __version__
prerelease_tag = "dev"
tag_end_index = __version__.index(prerelease_tag) + len(prerelease_tag)
new_version = __version__[:tag_end_index] + str(int(__version__[tag_end_index:]) + 1)
with open("src/_version.py", 'w') as f:
f.write("__version__ = \"" + new_version + "\"\n")
print("::set-output name=version::" + new_version)
print("::set-output name=cargo-version::" + "-".join(new_version.rsplit(".", 1)))

- name: Increment dev version (cargo)
run: cargo install cargo-edit && cargo set-version ${{ steps.update-version.outputs.cargo-version }}
working-directory: src/secrets

- name: Build dist wheels
run: bash build.sh

- name: Commit version update
uses: stefanzweifel/git-auto-commit-action@v4
with:
branch: main
commit_message: "Bump version to ${{ steps.update-version.outputs.version }} [ci skip]"
commit_options: "--signoff"
commit_user_name: ${{ secrets.ZOWE_ROBOT_USER }}
commit_user_email: ${{ secrets.ZOWE_ROBOT_EMAIL }}
file_pattern: "src/_version.py src/secrets/Cargo.*"
tagging_message: v${{ steps.update-version.outputs.version }}

- name: Publish packages to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_ROBOT_TOKEN }}
69 changes: 0 additions & 69 deletions .github/workflows/dev-release.yaml.old

This file was deleted.

66 changes: 33 additions & 33 deletions .github/workflows/sdk-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,41 +13,41 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
env:
OS: ${{ matrix.os }}
PYTHON: ${{ matrix.python-version }}

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 ./src --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 ./src --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
coverage run -m pytest ./tests/unit
- name: Generate a coverage xml file
run: |
coverage xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
if: ${{ matrix.python-version == '3.11' }}
with:
directory: ./
env_vars: OS,PYTHON
fail_ci_if_error: true
files: ./coverage.xml
flags: unittests
name: codecov-umbrella
verbose: true
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 ./src --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 ./src --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
coverage run -m pytest ./tests/unit
- name: Generate a coverage xml file
run: |
coverage xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
if: ${{ matrix.python-version == '3.12' }}
with:
directory: ./
env_vars: OS,PYTHON
fail_ci_if_error: true
files: ./coverage.xml
flags: unittests
name: codecov-umbrella
verbose: true
144 changes: 144 additions & 0 deletions .github/workflows/secrets-sdk.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# This file is autogenerated by maturin v1.3.1
# To update, run
#
# maturin generate-ci github
#
name: Secrets SDK CI

on:
push:
paths:
- "src/secrets/**"
- ".github/workflows/secrets-sdk.yml"
pull_request:
paths:
- "src/secrets/**"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
linux:
runs-on: ubuntu-latest
strategy:
matrix:
target: [x86_64, x86, aarch64, armv7, s390x]
fail-fast: false
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Set environment variables
run: src/secrets/scripts/configure-cross.sh ${{ matrix.target }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
args: --release --out dist --skip-auditwheel
sccache: "true"
manylinux: auto
docker-options: -e PKG_CONFIG_SYSROOT_DIR -e PKG_CONFIG_PATH
working-directory: src/secrets
before-script-linux: |
if command -v yum &> /dev/null; then
yum update && yum install -y libsecret-devel.${{ env.CROSS_DEB_ARCH }} pkgconfig
else
dpkg --add-architecture ${{ env.CROSS_DEB_ARCH }}
sed -i "s/deb /deb [arch=amd64] /g" /etc/apt/sources.list
echo "deb [arch=${{ env.CROSS_DEB_ARCH }}] http://ports.ubuntu.com/ubuntu-ports/ jammy main universe" >> /etc/apt/sources.list
echo "deb [arch=${{ env.CROSS_DEB_ARCH }}] http://ports.ubuntu.com/ubuntu-ports/ jammy-updates main universe" >> /etc/apt/sources.list
apt-get update && apt-get install -y libsecret-1-dev:${{ env.CROSS_DEB_ARCH }} pkg-config
fi
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: src/secrets/dist

windows:
runs-on: windows-latest
strategy:
matrix:
target: [x64, x86, aarch64]
fail-fast: false
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
if: ${{ matrix.target != 'aarch64' }}
with:
python-version: "3.10"
architecture: ${{ matrix.target }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
args: --release --out dist
sccache: "true"
working-directory: src/secrets
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: src/secrets/dist

macos:
runs-on: macos-latest
strategy:
matrix:
target: [x86_64, aarch64]
fail-fast: false
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.target }}
args: --release --out dist
sccache: "true"
working-directory: src/secrets
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: src/secrets/dist

sdist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build sdist
uses: PyO3/maturin-action@v1
with:
command: sdist
args: --out dist
working-directory: src/secrets
- name: Upload sdist
uses: actions/upload-artifact@v3
with:
name: wheels
path: src/secrets/dist

release:
name: Release
runs-on: ubuntu-latest
if: ${{ startsWith(github.ref, 'refs/tags/') }}
needs: [linux, windows, macos, sdist]
steps:
- uses: actions/download-artifact@v3
with:
name: wheels
- name: Publish to PyPI
uses: PyO3/maturin-action@v1
env:
MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_ROBOT_TOKEN }}
with:
command: upload
args: --non-interactive --skip-existing *
working-directory: src/secrets
1 change: 1 addition & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"editor.formatOnSave": true,
"python.formatting.provider": "black",
"rust-analyzer.linkedProjects": ["./src/secrets/Cargo.toml"],
"[python]": {
"editor.codeActionsOnSave": {
"source.organizeImports": true
Expand Down
23 changes: 14 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@ All notable changes to the Zowe Client Python SDK will be documented in this fil

## Recent Changes

- Bug: Fixed profile merge order to match Node.js SDK
- Feature: Added method to load profile properties from environment variables
- Bugfix: Fixed exception handling in session.py [#213] (https://github.com/zowe/zowe-client-python-sdk/issues/213)
- Feature: Added a CredentialManager class to securely retrieve values from credentials and manage multiple credential entries on Windows [#134](https://github.com/zowe/zowe-client-python-sdk/issues/134)
- Feature: Added method to Save profile properties to zowe.config.json file [#73](https://github.com/zowe/zowe-client-python-sdk/issues/73)
- Feature: Added method to Save secure profile properties to vault [#72](https://github.com/zowe/zowe-client-python-sdk/issues/72)
- Bugfix: Fixed issue for datasets and jobs with special characters in URL [#211] (https://github.com/zowe/zowe-client-python-sdk/issues/211)
- Feature: Added method to load profile properties from environment variables
- BugFix: Validation of zowe.config.json file matching the schema [#192](https://github.com/zowe/zowe-client-python-sdk/issues/192)
### Enhancements

- Added method to save secure profile properties to vault [#72](https://github.com/zowe/zowe-client-python-sdk/issues/72)
- Added method to save profile properties to zowe.config.json file [#73](https://github.com/zowe/zowe-client-python-sdk/issues/73)
- Added CredentialManager class to securely retrieve values from credentials and manage multiple credential entries on Windows [#134](https://github.com/zowe/zowe-client-python-sdk/issues/134)
- Added method to load profile properties from environment variables [#136](https://github.com/zowe/zowe-client-python-sdk/issues/136)
- Added validation of zowe.config.json file matching the schema [#192](https://github.com/zowe/zowe-client-python-sdk/issues/192)
- Added Secrets SDK for storing client secrets in OS keyring [#208](https://github.com/zowe/zowe-client-python-sdk/issues/208)

### Bug Fixes

- Fixed profile merge order to match Node.js SDK [#190](https://github.com/zowe/zowe-client-python-sdk/issues/190)
- Fixed issue for datasets and jobs with special characters in URL [#211](https://github.com/zowe/zowe-client-python-sdk/issues/211)
- Fixed exception handling in session.py [#213](https://github.com/zowe/zowe-client-python-sdk/issues/213)
2 changes: 1 addition & 1 deletion docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
sphinx_rtd_theme>=0.5.1
sphinxcontrib-spelling==5.4.0
sphinxcontrib-spelling==8.0.0
-e ./src/core
-e ./src/zos_console
-e ./src/zos_files
Expand Down
Loading