Skip to content

Commit

Permalink
Merge branch 'feature/v3/jcl' into user/markackert/zwe-tests
Browse files Browse the repository at this point in the history
Signed-off-by: MarkAckert <[email protected]>
  • Loading branch information
MarkAckert committed Jul 10, 2024
2 parents 862aac5 + 2bb8121 commit e2369e3
Show file tree
Hide file tree
Showing 63 changed files with 604 additions and 49,683 deletions.
22 changes: 22 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
version: 2

updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: weekly
groups:
all-version-dependencies:
applies-to: version-updates
patterns:
- "*"
update-types:
- "patch"
- "minor"
all-security-minorpatch:
applies-to: security-updates
patterns:
- "*"
update-types:
- "patch"
- "minor"
4 changes: 4 additions & 0 deletions .github/scripts/cicd_test/make_matrix.sh
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ case $install_test_choice in
test_force_system="zzow08"
;;

"z/OS node v20")
test_file="$ZOS_NODE_V20_TESTFILE"
;;

"Non-strict Verify External Certificate")
test_file="$NON_STRICT_VERIFY_EXTERNAL_CERTIFICATE_TESTFILE"
;;
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/cicd-test-readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Background: CICD testing relies on a `zowe.pax` or `zowe-smpe.zip` (for SMPE ins
When running CICD integration tests during RC stage, the following string will be parsed into the Github Actions matrix. As a result, a total of 21 independent jobs will be spawned.
```
basic/install.ts(zzow06,zzow07,zzow08);basic/install-ptf.ts(zzow06,zzow07,zzow08);basic/install-ext.ts(zzow07);extended/keyring.ts(zzow06,zzow07,zzow08);extended/node-versions/node-v18.ts(zzow06,zzow07,zzow08);extended/certificates/nonstrict-verify-external-certificate.ts(zzow06)
basic/install.ts(zzow06,zzow07,zzow08);basic/install-ptf.ts(zzow06,zzow07,zzow08);basic/install-ext.ts(zzow07);extended/keyring.ts(zzow06,zzow07,zzow08);extended/node-versions/node-v18.ts(zzow06,zzow07,zzow08);extended/node-versions/node-v20.ts(zzow06,zzow07,zzow08);extended/certificates/nonstrict-verify-external-certificate.ts(zzow06)
```
Total elapsed time when running in parallel is approximately 3.5 hours on paper idealy if all parallel jobs are executing at the same time. In reality, from numerous tests performed, total elapsed time is around 4 hours.
Expand All @@ -122,6 +122,7 @@ Selected test running elapsed time:
| Convenience Pax | 53m |
| SMPE PTF | 68m |
| z/OS node v18 | 45m |
| z/OS node v20 | 45m |
| Keyring | 53m |
| Non-strict Verify External Certificate | 51m |
| Extensions | 67m
Expand Down
7 changes: 6 additions & 1 deletion .github/workflows/cicd-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ on:
- Extensions
- Keyring
- z/OS node v18
- z/OS node v20
- Non-strict Verify External Certificate
- Install PTF Twice
- VSAM Caching Storage Method
Expand Down Expand Up @@ -74,14 +75,15 @@ env:
EXTENSIONS_TESTFILE: basic/install-ext.ts
KEYRING_TESTFILE: extended/keyring.ts
ZOS_NODE_V18_TESTFILE: extended/node-versions/node-v18.ts
ZOS_NODE_V20_TESTFILE: extended/node-versions/node-v20.ts
NON_STRICT_VERIFY_EXTERNAL_CERTIFICATE_TESTFILE: extended/certificates/nonstrict-verify-external-certificate.ts
INSTALL_PTF_TWICE_TESTFILE: extended/install-ptf-two-times.ts
VSAM_CACHING_STORAGE_METHOD_TESTFILE: extended/caching-storages/vsam-storage.ts
INFINISPAN_CACHING_STORAGE_METHOD_TESTFILE: extended/caching-storages/infinispan-storage.ts
CONFIG_MANAGER_TESTFILE: extended/config-manager/enable-config-manager.ts
GENERAL_API_DOCUMENTATION_TESTFILE: basic/install-api-gen.ts
ZOWE_NIGHTLY_TESTS_FULL: basic/install.ts(all);basic/install-ptf.ts(all)
ZOWE_RELEASE_TESTS_FULL: basic/install.ts(all);basic/install-ptf.ts(all);basic/install-ext.ts(any);extended/keyring.ts(all);extended/node-versions/node-v18.ts(zzow08):extended/certificates/nonstrict-verify-external-certificate.ts(any);extended/caching-storages/infinispan-storage.ts(any);extended/config-manager/enable-config-manager.ts(any)
ZOWE_RELEASE_TESTS_FULL: basic/install.ts(all);basic/install-ptf.ts(all);basic/install-ext.ts(any);extended/keyring.ts(all);extended/node-versions/node-v18.ts(zzow08);extended/node-versions/node-v20.ts(any);extended/certificates/nonstrict-verify-external-certificate.ts(any);extended/caching-storages/infinispan-storage.ts(any);extended/config-manager/enable-config-manager.ts(any)

jobs:
display-dispatch-event-id:
Expand Down Expand Up @@ -292,6 +294,9 @@ jobs:
case 'z/OS node v18':
eta = 45
break;
case 'z/OS node v20':
eta = 45
break;
case 'Infinispan Caching Storage Method':
eta = 51
break;
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/diff-schema.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Generate schema diff
on:
push:
branches:
- v1.x/master
- v2.x/master
- v3.x/master

workflow_dispatch:
inputs:
FROM_COMMIT:
description: 'Specify tag (v prefix) to diff from'
required: false
TO_COMMIT:
description: 'Specify tag (v prefix) to diff to'
required: false

jobs:
diff:
runs-on: ubuntu-latest
steps:
- name: '[Prep 1] Checkout'
uses: actions/checkout@v2
with:
fetch-depth: '0'
- name: '[Prep 2] Setup Node'
uses: actions/setup-node@v2
with:
node-version: 18
- name: '[Prep 3] Get to version'
run: node -e "const fs = require('fs'); const currentVersion = JSON.parse(fs.readFileSync('./manifest.json.template', 'utf8')).version; if ('${{ github.event.inputs.TO_COMMIT }}'.length > 0) { console.log('TO=${{ github.event.inputs.TO_COMMIT }}')} else { console.log('TO=v'+currentVersion) }" >> $GITHUB_ENV
- name: '[Prep 4] Get from version'
run: node -e "if ('${{ github.event.inputs.FROM_COMMIT }}'.length > 0) { console.log('FROM=${{ github.event.inputs.FROM_COMMIT }}') } else { let parts = '${{ env.TO }}'.split('.'); parts[1]--; console.log('FROM='+parts.join('.')); }" >> $GITHUB_ENV
- name: '[Build] Make diff'
run: git diff ${{ env.FROM }} ${{ env.TO }} -- schemas > schemas.diff
- name: '[Upload]'
uses: actions/upload-artifact@v3
with:
name: schemas.diff
path: schemas.diff
if-no-files-found: error
41 changes: 41 additions & 0 deletions .github/workflows/diff-yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Generate example yaml diff
on:
push:
branches:
- v1.x/master
- v2.x/master
- v3.x/master

workflow_dispatch:
inputs:
FROM_COMMIT:
description: 'Specify tag (v prefix) to diff from'
required: false
TO_COMMIT:
description: 'Specify tag (v prefix) to diff to'
required: false

jobs:
diff:
runs-on: ubuntu-latest
steps:
- name: '[Prep 1] Checkout'
uses: actions/checkout@v2
with:
fetch-depth: '0'
- name: '[Prep 2] Setup Node'
uses: actions/setup-node@v2
with:
node-version: 18
- name: '[Prep 3] Get to version'
run: node -e "const fs = require('fs'); const currentVersion = JSON.parse(fs.readFileSync('./manifest.json.template', 'utf8')).version; if ('${{ github.event.inputs.TO_COMMIT }}'.length > 0) { console.log('TO=${{ github.event.inputs.TO_COMMIT }}')} else { console.log('TO=v'+currentVersion) }" >> $GITHUB_ENV
- name: '[Prep 4] Get from version'
run: node -e "if ('${{ github.event.inputs.FROM_COMMIT }}'.length > 0) { console.log('FROM=${{ github.event.inputs.FROM_COMMIT }}') } else { let parts = '${{ env.TO }}'.split('.'); parts[1]--; console.log('FROM='+parts.join('.')); }" >> $GITHUB_ENV
- name: '[Build] Make diff'
run: git diff ${{ env.FROM }} ${{ env.TO }} -- example-zowe.yaml > example-yaml.diff
- name: '[Upload]'
uses: actions/upload-artifact@v3
with:
name: example-yaml.diff
path: example-yaml.diff
if-no-files-found: error
2 changes: 1 addition & 1 deletion .github/workflows/snyk-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,5 +55,5 @@ jobs:
uses: snyk/actions/docker@master
# we don't have original Dockerfile, cannot generate and upload sarif
with:
image: ${{ matrix.image }}
image: ${{ matrix.images }}
command: test
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@
All notable changes to the Zowe Installer will be documented in this file.
<!--Add the PR or issue number to the entry if available.-->

## `2.16.0
## `2.17.0`

## New features and enhancements
- Enhancement: `zwe components install` and `zwe components upgrade` now checks the schema validity of any application framework plugins present within a component, so that you will know better if a component is valid prior to running Zowe. [#3866](https://github.com/zowe/zowe-install-packaging/pull/3866)
- Enhancement: Added zowe.network.server.tls.attls and zowe.network.client.tls.attls as booleans for controlling global or per-component way to tell Zowe servers that they should operate in a mode compatible with an AT-TLS setup. [#3463](https://github.com/zowe/zowe-install-packaging/pull/3463)

## `2.16.0`

## Minor enhancements/defect fixes
- Enhancement: Reduced resource consumption by removal of one shell process per server that was used when starting each server. (#3812)
Expand Down
6 changes: 3 additions & 3 deletions INSTALLATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ When the JCL is prepared, the following jobs can be submitted to perform the fol
|Task|Description|Sample JCL|
|---|---|---|
|Create Instance Datasets|**Purpose:** Create datasets for Zowe's PARMLIB content and non-ZFS extension content for a given Zowe Instance<br><br>**Action:**<br>1) Allocate PDSE FB80 dataset with at least 15 tracks named from Zowe parameter `zowe.setup.dataset.parmlib`<br>2) Allocate PDSE FB80 dataset with at least 30 tracks named from Zowe parameter `zowe.setup.dataset.authPluginLib`<br>3) Copy ZWESIP00 member from `zowe.setup.dataset.prefix`.SZWESAMP into `zowe.setup.dataset.parmlib`|[ZWEIMVS](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIMVS)|
|APF Authorize privileged content|**Purpose:** Zowe contains one privileged component, ZIS, which enables the security model by which the majority of Zowe is unprivileged and in key 8. The load library for the ZIS component and its extension library must be set APF authorized and run in key 4 to use ZIS and components that depend upon it.<br><br>**Action:**<br>1)APF authorize the datasets defined at `zowe.setup.dataset.authLoadlib` and `zowe.setup.dataset.authPluginLib`.<br>2) Define PPT entries for the members ZWESIS01 and ZWESAUX as Key 4, NOSWAP in the SCHEDxx member of the system PARMLIB.|[ZWEIAPF](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIAPF)|
|APF Authorize privileged content|**Purpose:** Zowe contains one privileged component, ZIS, which enables the security model by which the majority of Zowe is unprivileged and in key 8. The load library for the ZIS component and its extension library must be set APF authorized and run in key 4 to use ZIS and components that depend upon it.<br><br>**Action:**<br>1) APF authorize the datasets defined at `zowe.setup.dataset.authLoadlib` and `zowe.setup.dataset.authPluginLib`.<br>2) Define PPT entries for the members ZWESIS01 and ZWESAUX as Key 4, NOSWAP in the SCHEDxx member of the system PARMLIB.|[ZWEIAPF](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIAPF)|
|Grant SAF permissions|The STC accounts for Zowe need permissions for operating servers, and users need permissions for interacting with the servers.<br><br>**Action:** [Set SAF permissions for accounts](https://docs.zowe.org/stable/user-guide/assign-security-permissions-to-users#security-permissions-reference-table)|RACF: [ZWEIRAC](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIRAC)<br><br>TSS: [ZWEITSS](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEITSS)<br><br>ACF2: [ZWEIACF](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/SZWIACF)|
|(z/OS v2.4 ONLY) Create Zowe SAF Resource Class|This is not needed on z/OS v2.5+. On z/OS v2.4, the SAF resource class for Zowe is not included, and must be created|RACF: [ZWEIRACZ](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIRACZ)<br><br>TSS: [ZWEITSSZ](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEITSSZ)<br><br>ACF2: [ZWEIACFZ](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEIACFZ)|
|Copy STC JCL to PROCLIB|**Purpose**: ZWESLSTC is the job for running Zowe's webservers, and ZWESISTC is for running the APF authorized cross-memory server. The ZWESASTC job is started by ZWESISTC on an as-needed basis.<br><br>**Action**: Copy the members ZWESLSTC, ZWESISTC, and ZWESASTC into your desired PROCLIB. If the job names are customized, also modify the YAML values of them in `zowe.setup.security.stcs`|[ZWEISTC](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWEISTC)|
Expand Down Expand Up @@ -164,7 +164,7 @@ Among the choices is for it to use a VSAM dataset of your choice.
|Task|Description|Sample JCL|
|---|---|---|
|Create VSAM Dataset for Caching Service|**Action**: Create a RLM or NONRLM dataset for the caching service, and set the name into the YAML value `components.caching-service.storage.vsam.name`|[ZWECSVSM](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWECSVSM)|
|Create VSAM Dataset for Caching Service|**Action**: Create a RLS or NONRLS dataset for the caching service, and set the name into the YAML value `components.caching-service.storage.vsam.name`|[ZWECSVSM](https://github.com/zowe/zowe-install-packaging/tree/feature/v3/jcl/files/SZWESAMP/ZWECSVSM)|

JCL samples for removing Zowe configuration also exist.
|Action|Sample JCL|
Expand Down Expand Up @@ -245,7 +245,7 @@ Among the choices is for it to use a VSAM dataset of your choice.

|Task|Description|Sample JCL|Doc|
|---|---|---|---|
|Create VSAM Dataset for Caching Service|Creates a RLM or NONRLM dataset for the caching service using the YAML values in `zowe.setup.vsam`|`zwe init vsam`|[Doc](https://docs.zowe.org/stable/appendix/zwe_server_command_reference/zwe/init/zwe-init-vsam)|
|Create VSAM Dataset for Caching Service|Creates a RLS or NONRLS dataset for the caching service using the YAML values in `zowe.setup.vsam`|`zwe init vsam`|[Doc](https://docs.zowe.org/stable/appendix/zwe_server_command_reference/zwe/init/zwe-init-vsam)|

<br>
<br>
Expand Down
14 changes: 14 additions & 0 deletions bin/commands/components/install/extract/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,20 @@ export function execute(componentFile: string, autoEncoding?: string, upgrade?:
common.printErrorAndExit(`Error ZWEL0167E: Cannot find component name from ${componentFile} package manifest`, undefined, 167);
}
common.printDebug(`- Component name found as ${componentName}`);

// If the component has appfw plugins, their validity should be checked against appfw plugin schema.
// If invalid, the installation will exit with an error message.
if (manifest.appfwPlugins) {
manifest.appfwPlugins.forEach((appfwPlugin: {path: string})=> {
let result = component.getPluginDefinition(pathoid.resolve(tmpDir, appfwPlugin.path), true);
//Normally, getPluginDefinition would quit upon failure. But we want to cleanup the tmpDir before that.
// So, we pass true to allow it to continue, check for null, and then remove the tmpdir and exit if so.
if (result === null) {
fs.rmrf(tmpDir);
std.exit(1);
}
});
}

const destinationDir = pathoid.resolve(targetDir, componentName);
const bkpDir = pathoid.resolve(targetDir, `${componentName}_zwebkp`);
Expand Down
33 changes: 33 additions & 0 deletions bin/commands/init/.help
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,39 @@ If you pass `--dry-run` with this command, each subcommand will print out the
JCL that it would submit, without doing the submission. This can be used to
review what Zowe would do before doing it.

The following are the JCL submitted by each command as needed

- `zwe init mvs`
- ZWEIMVS
- (If `--allow-overwrite` specified and old datasets exist) ZWERMVS
- (If `zowe.setup.dataset.authLoadLib` is not named prefix + ZWELOAD) ZWEIMVS2
- (If above is true, and the dataset exists, and `--allow-overwrite` is set) ZWERMVS2
- `zwe init vsam`
- ZWECSVSM
- (If `--allow-overwrite` specified and old datasets exist) ZWECSRVS
- `zwe init apfauth`
- ZWEIAPF2
- `zwe init security`
- (If RACF) ZWEIRAC
- (If RACF and z/OS 2.4) ZWIRACZ
- (If TSS) ZWEITSS
- (If TSS and z/OS 2.4) ZWITSSZ
- (If ACF2) ZWEIACF
- (If ACF2 and z/OS 2.4) ZWIACFZ
- `zwe init certificate`
- (If creating keyring and certificate for RACF) ZWEIKRR1
- (If creating keyring and connecting certificate for RACF) ZWEIKRR2
- (If creating keyring and importing certificate for RACF) ZWEIKRR3
- (If creating keyring and certificate for TSS) ZWEIKRR1
- (If creating keyring and connecting certificate for TSS) ZWEIKRT2
- (If creating keyring and importing certificate for TSS) ZWEIKRT3
- (If creating keyring and certificate for ACF2) ZWEIKRA1
- (If creating keyring and connecting certificate for ACF2) ZWEIKRA2
- (If creating keyring and importing certificate for ACF2) ZWEIKRA3
- `zwe init stc`
- ZWEISTC
- (If `--allow-overwrite` specified and old members exist) ZWERSTC

If you pass `--skip-security-setup` with this command, `zwe init apfauth` and
`zwe init security` steps will be skipped.

Expand Down
2 changes: 1 addition & 1 deletion bin/commands/init/certificate/.parameters
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
allow-overwrite,allow-overwritten||boolean|||||Allow overwritten existing MVS data set.
update-config||boolean|||||Whether to update YAML configuration file with initialization result.
ignore-security-failures||boolean|||||Whether to ignore security setup job failures.
security-dry-run,dry-run||boolean|||||Generates and prints JCL but does not execute
security-dry-run,dry-run||boolean|||||Generates and prints JCL but does not execute.
8 changes: 8 additions & 0 deletions bin/commands/init/certificate/index.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
# Copyright Contributors to the Zowe Project.
#######################################################################

print_level1_message "Generate certificate"

###############################
# validation
require_zowe_yaml
Expand All @@ -20,6 +22,12 @@ else
CONFIG_TO_WRITE=${ZWE_CLI_PARAMETER_CONFIG}
fi

# Keytool is needed
require_java

# Node is needed for read_yaml
require_node

export ZWE_PRIVATE_TMP_MERGED_YAML_DIR=$(create_tmp_file)
mkdir -p ${ZWE_PRIVATE_TMP_MERGED_YAML_DIR}
_CEE_RUNOPTS="XPLINK(ON),HEAPPOOLS(OFF),HEAPPOOLS64(OFF)" ${ZWE_zowe_runtimeDirectory}/bin/utils/configmgr -script "${ZWE_zowe_runtimeDirectory}/bin/commands/internal/config/output/cli.js"
Expand Down
2 changes: 1 addition & 1 deletion bin/commands/init/generate/.errors
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
ZWEL0143E|143|Cannot find data set member %s. You may need to re-run zwe install.
ZWEL0157E|157|%s (%s) is not defined in Zowe YAML configuration file.
ZWEL0318E|318|Configuration stored in PARMLIB must use member name ZWEYAML when using generate action.
ZWEL0319E|319|Configuration stored in PARMLIB must use member name ZWEYAML when using generate action.
Loading

0 comments on commit e2369e3

Please sign in to comment.