IJMP-1710 - Validate connection once during zosmf declarative step ex…

…ecution (#83) * implement connection validation Signed-off-by: Anatoli Kalbasin <[email protected]>
zowe · Sep 18, 2024 · fd4f407 · fd4f407
1 parent 880cf4b
commit fd4f407
Show file tree

Hide file tree

Showing 5 changed files with 296 additions and 68 deletions.
diff --git a/README.md b/README.md
@@ -3,12 +3,13 @@
 ## About the plugin
 The Zowe zDevOps Jenkins Plugin by [IBA Group](https://ibagroupit.com/?utm_campaign=IBA_W-Mainframe&utm_source=jenkins&utm_medium=referral&utm_content=description_zdevops) is an open-source, secure , and reliable agent-less Jenkins plugin that makes it possible to perform most of the actual tasks on the mainframe, managing it with a modern native mainframe zOSMF REST API and the capabilities of available zOSMF SDKs.
 
-## Advantages
+## Main features
 - Secure and modern connection of Jenkins to the mainframes through the use of zOSMF REST API.
 - The functionality is based on the Kotlin SDK methods, such as JCL jobs submission, download, allocate, write to the dataset, etc., with a log collected upon completion.
-- Multiple connections to various mainframes—z/OS Connections List where you can save all the necessary systems and credentials; all data is safely stored under the protection of Jenkins Credentials manager.
-- Agent-less.
+- Multiple connections to various mainframes — z/OS Connections List where you can save all the necessary systems and credentials (all data is safely stored under the protection of Jenkins Credentials manager).
+- Agent-less solution.
 - z/OSMF connection validation.
+- Convenient user interface panels for working with the mainframe
 - Fast execution and functional extensibility.
 
 ## About us
@@ -24,6 +25,254 @@ Please feel free to contact us or schedule a call with our Mainframe DevOps expe
 
 Thank you for considering IBA Group for your mainframe needs.
 
+## Before use - Plugin configuration
+After successfully installing the plugin, you need to configure it for further work - this will require a minimum of actions.
+1. Move to “Manage Jenkins” -> “Configure System / System” -> scroll to the very bottom of the list of installed plugins and find the panel with the name - <b>“z/OS Connection List”</b>
+2. This setting allows you to add all necessary z/OS systems and configure access to them.
+   It is necessary to set the connection name (it is also the ID for declarative methods in the code). For the example: ```z/os-connection-name```
+3. The URL address and port of the required mainframe to connect via z/OSMF. Example: ```https://<ip-addres>:<port number>```
+4. Add credentials (Mainframe User ID + Password) under which you can connect to the system.
+
+You can save as many connections as you like, the system will keep the corresponding user IDs/passwords.
+
+## Declarative methods brief list
+```groovy
+stage ("stage-name") {
+  steps {
+    // ...
+    zosmf("z/os-connection-name") {
+        submitJob "//'EXAMPLE.DATASET(MEMBER)'"
+        submitJobSync "//'EXAMPLE.DATASET(MEMBER)'"
+        downloadDS "EXAMPLE.DATASET(MEMBER)"
+        downloadDS dsn:"EXAMPLE.DATASET(MEMBER)", vol:"VOL001"
+        allocateDS dsn:"EXAMPLE.DATASET", alcUnit:"TRK", dsOrg:"PS", primary:1, secondary:1, recFm:"FB"
+        writeFileToDS dsn:"EXAMPLE.DATASET", file:"workspaceFile"
+        writeFileToDS dsn:"EXAMPLE.DATASET", file:"D:\\files\\localFile"
+        writeToDS dsn:"EXAMPLE.DATASET", text:"Write this string to dataset"
+        writeFileToMember dsn:"EXAMPLE.DATASET", member:"MEMBER", file:"workspaceFile"
+        writeFileToMember dsn:"EXAMPLE.DATASET", member:"MEMBER", file:"D:\\files\\localFile"
+        writeToMember dsn:"EXAMPLE.DATASET", member:"MEMBER", text:"Write this string to member"
+
+        writeToFile destFile: "u/USER/myfile", text: "Write this string to file"
+        writeFileToFile destFile: "u/USER/myfile", sourceFile: "myfile.txt"
+        writeFileToFile destFile: "u/USER/myfile", sourceFile: "myfile.txt", binary: "true"
+
+        deleteDataset dsn:"EXAMPLE.DATASET"
+        deleteDataset dsn:"EXAMPLE.DATASET", member:"MEMBER"
+        deleteDatasetsByMask mask:"EXAMPLE.DATASET.*"
+    }
+    // ...
+  }
+}
+```
+
+## Declarative Methods Detail Description
+
+### allocateDS - Represents an action for allocating a dataset in a declarative style
+```groovy
+allocateDS dsn:"EXAMPLE.DATASET", dsOrg:"PS", primary:1, secondary:1, recFm:"FB"
+```
+**Mandatory Parameters:**
+   * ```dsn:"EXAMPLE.DATASET"``` - The name of the dataset to be allocated
+   * ```dsOrg:"PS"``` - The dataset organization (could be only PO, POE, PS, VS)
+   * ```primary:"1"``` - The primary allocation size in cylinders or tracks
+   * ```secondary:"1"``` - The secondary allocation size in cylinders or tracks
+   * ```recFm:"FB"``` - The record format (could be only F, FB, V, VB, U, VSAM, VA)
+
+**Optional parms:**
+   * ```volser:"YOURVOL"``` - Volume serial number where the dataset will be allocated.
+   * ```unit:"SYSDA"``` - Specifies the type of storage device. SYSDA is a common direct access storage device.
+   * ```alcUnit:"TRK"``` - Allocation units (CYL for cylinders, TRK for tracks).
+   * ```dirBlk:"5"``` - Directory block records.
+   * ```blkSize:"800"``` - BLKSIZE=800: Block size of 800 bytes.
+   * ```lrecl:"80"``` - Logical record length.
+   * ```storClass:"STORAGECLASS"``` - Storage class for SMS-managed datasets.
+   * ```mgntClass:"MGMTCLASS"``` - Management class for SMS-managed datasets.
+   * ```dataClass:"DATACLASS"``` - Data class for SMS-managed datasets.
+   * ```avgBlk:"10"``` - Average block length.
+   * ```dsnType:"LIBRARY"``` - Specifies the type of dataset, LIBRARY for a PDS or PDSE.
+   * ```dsModel:"MODEL.DATASET.NAME"``` - Data set model is a predefined set of attributes that can be used to allocate new data sets with the same characteristics ("LIKE" parameter).
+
+
+### deleteDataset - Represents an action for deleting datasets and members in a declarative style
+```groovy
+deleteDataset dsn:"EXAMPLE.DATASET"
+```
+**Mandatory Parameters:**
+   * ```dsn:"EXAMPLE.DATASET"``` - Sequential or library dataset name for deletion
+   * ```member:"MEMBER"``` - Dataset member name for deletion
+
+**Expected behavior under various deletion scenarios:**
+
+* To delete a member from the library, the dsn and member parameters must be specified:
+    ```
+    deleteDataset dsn:"EXAMPLE.DATASET", member:"MEMBER"
+    ```
+
+* You cannot delete a VSAM dataset this way. Otherwise, you will get output similar to:
+    ```
+    Deleting dataset EXAMPLE.VSAM.DATASET with connection <ip-address>:10443 
+    ISRZ002 Deallocation failed - Deallocation failed for data set 'EXAMPLE.VSAM.DATASET'
+    ```
+
+* What do you get if a dataset does not exist?
+
+    ```
+    Deleting dataset EXAMPLE.DS.DOES.NOT.EXIST with connection <ip-address>:10443
+    ISRZ002 Data set not cataloged - 'EXAMPLE.DS.DOES.NOT.EXIST' was not found in catalog.
+    ```
+
+* What do you get if a dataset is busy by a user or a program?
+
+    ```
+    Deleting dataset EXAMPLE.DS.ISUSED.BY.USER with connection <ip-address>:10443
+    ISRZ002 Data set in use - Data set 'EXAMPLE.DS.ISUSED.BY.USER' in use by another user, try later or enter HELP for a list of jobs and users allocated to 'EXAMPLE.DS.ISUSED.BY.USER'.
+    ```
+
+## Use case example
+Here you can find an example of a minimal declarative Jenkins pipeline for execution, testing and further modification for your personal needs.
+Pipeline can be used either directly inside the ```Pipeline``` code block in the Jenkins server, or in a ```Jenkinsfile``` stored in Git
+This pipeline example uses all currently available methods and functionality of the Zowe zDevOps plugin.
+
+**Steps to Execute the Pipeline:**
+1. Add a zosmf connection in settings (<b>“Manage Jenkins” -> “Configure System / System” -> z/OS Connection List</b>). Enter a connection name, zosmf url, username and password.
+2. Create a new Jenkins item -> ```Pipeline``` and open its configuration.
+3. In the ```Pipeline``` section, paste the code from the example below and replace all the necessary variables with your data
+4. Done, enjoy the minimal ready-made pipeline template!
+
+```groovy
+pipeline {
+    agent any
+
+    environment {
+        // Define environment variables
+        GIT_REPOSITORY_URL = 'https://github.com/your-username/your-repo.git'  // Replace with your GitHub URL
+        GIT_BRANCH = 'main'                                     // Replace with your GitHub branch name
+        GIT_USER_CREDENTIAL_ID = 'jenkins-cred-key'             // Replace with your Jenkins GitHub credential ID
+        ZOS_CONN_ID = 'z/os-connection-name'                    // Replace with your z/OS Connection ID from zDevOps plugin settings
+        HLQ = 'HLQ'                                             // Replace with your z/OS high-level qualifier (HLQ)
+        PS_DATASET_1 = "${HLQ}.NEW.TEST1"                       // OPTIONAL: Replace with the dataset names you need
+        PS_DATASET_2 = "${HLQ}.NEW.TEST2"                       // OPTIONAL
+        PO_DATASET = "${HLQ}.NEW.TEST3"                         // OPTIONAL
+        PO_MEMBER = "NEWMEM"                                    // OPTIONAL
+        JCL_JOB_TEMPLATE = "jcl_job_example"                    // Replace with the name of your file that contains the JCL job code
+        JIRA_URL = 'https://your-jira-instance.atlassian.net'   // Replace with your Jira URL
+        JIRA_USER = '[email protected]'               // Replace with your Jira user email
+        JIRA_API_TOKEN = 'your-jira-api-token'                  // Replace with your Jira API token
+        JIRA_ISSUE_KEY = 'PROJECT-123'                          // Replace with your Jira issue key
+    }
+
+    stages {
+        stage('Checkout') {
+            steps {
+                // Checkout the source code from Git
+                checkout scmGit(
+                    branches: [[name: "${GIT_BRANCH}"]],
+                    userRemoteConfigs: [[credentialsId:  "${GIT_USER_CREDENTIAL_ID}",
+                    url: "${GIT_REPOSITORY_URL}"]])
+            }
+        }
+
+        stage('Allocate DSs') {
+            steps {
+                zosmf("${ZOS_CONN_ID}") {
+                    allocateDS dsn:"${PS_DATASET_1}", dsOrg:"PS", primary:1, secondary:1, recFm:"FB"
+                    allocateDS dsn:"${PS_DATASET_2}", dsOrg:"PS", primary:1, secondary:1, recFm:"FB", alcUnit:"TRK"
+                    allocateDS dsn:"${PO_DATASET}(${PO_MEMBER})", dsOrg:"PO", primary:1, secondary:1, recFm:"FB"
+                }
+            }
+        }
+        
+        stage('Add JCL content') {
+            steps {
+                script {
+                   // Read the content of the JCL job into a variable
+                   env.JCL_CONTENT = readFile("${JCL_JOB_TEMPLATE}").trim()
+                   // Print the content of the file (for debugging purposes)
+                   echo "JCL job content:\n${env.JCL_CONTENT}"
+                }
+                zosmf("${ZOS_CONN_ID}") {
+                    writeFileToDS dsn:"${PS_DATASET_2}", file:"${JCL_JOB_TEMPLATE}"
+                    writeFileToMember dsn:"${PO_DATASET}", member:"${PO_MEMBER}", file:"${JCL_JOB_TEMPLATE}"
+                    writeToDS dsn:"${PS_DATASET_1}", text:"${env.JCL_CONTENT}"
+                }
+            }
+        }
+        
+        stage('Add USS content') {
+            steps {
+                zosmf("${ZOS_CONN_ID}") {
+                    writeToFile destFile: "u/${HLQ}/test_file1", text: "${env.JCL_CONTENT}"
+                    writeFileToFile destFile: "u/${HLQ}/test_file2", sourceFile: "${JCL_JOB_TEMPLATE}", binary: "true"
+                }
+            }
+        }
+        
+        stage('Submit JCL jobs') {
+            steps {
+                zosmf("${ZOS_CONN_ID}") {
+                    submitJob "//'${PS_DATASET_1}'"
+                    submitJobSync "//'${PO_DATASET}(NEWMEM)'"
+                }
+            }
+        }
+
+        stage('Download datasets') {
+            steps {
+                zosmf("${ZOS_CONN_ID}") {
+                    downloadDS "${PS_DATASET_1}"
+                    downloadDS dsn:"${PS_DATASET_2}"
+                }
+            }
+        }
+        
+        stage('Clean up') {
+            steps {
+                zosmf("${ZOS_CONN_ID}") {
+                    deleteDataset dsn:"${PS_DATASET_1}"
+                    deleteDatasetsByMask mask:"${HLQ}.NEW.*"
+                }
+            }
+        }
+    }
+
+    post {
+        always {
+            script {
+                def jiraStatus = currentBuild.currentResult == 'SUCCESS' ? 'Build Successful' : 'Build Failed'
+                def jiraComment = """
+                {
+                    "body": "Jenkins build ${jiraStatus} for Job ${env.JOB_NAME} - Build #${env.BUILD_NUMBER}.
+                    [View the build here|${env.BUILD_URL}]"
+                }
+                """
+
+                httpRequest acceptType: 'APPLICATION_JSON',
+                            contentType: 'APPLICATION_JSON',
+                            httpMode: 'POST',
+                            requestBody: jiraComment,
+                            url: "${JIRA_URL}/rest/api/2/issue/${JIRA_ISSUE_KEY}/comment",
+                            authentication: 'jira-credentials-id'
+            }
+        }
+        
+        success {
+            // Notify success (example: send email)
+            mail to: '${JIRA_USER}',
+                 subject: "SUCCESS: Build ${env.BUILD_NUMBER}",
+                 body: "The build ${env.BUILD_NUMBER} was successful."
+        }
+        
+        failure {
+            // Notify failure (example: send email)
+            mail to: '${JIRA_USER}',
+                 subject: "FAILURE: Build ${env.BUILD_NUMBER}",
+                 body: "The build ${env.BUILD_NUMBER} failed. Please check the Jenkins logs for more details."
+        }
+    }
+}
+```
+
 ## Manual plugin installation by the .hpi executable file
 The plugin are packaged as self-contained <b>.hpi</b> files, which have all the necessary code, images, and other resources which the plugin needs to operate successfully.
 
@@ -45,50 +294,6 @@ Assuming a <b>.hpi</b> file has been downloaded, a logged-in Jenkins administrat
 6. Next you need to login into the Jenkins, move to the <b>“Manage Jenkins” -> “Manage Plugins” -> “Advanced (tab)” -> “Deploy Plugin”</b> (You can select a plugin file from your local system or provide a URL to install a plugin from outside the central plugin repository) <b>-> Specify the path to the generated .hpi/.jpi file</b> (or by dragging the file from Intellij IDEA project to the file upload field in the Jenkins).
 7. Click <b>“Deploy”</b>, reboot Jenkins after installation. The Plugin is ready to go!
 
-## Plugin configuration
-After successfully installing the plugin, you need to configure it for further work - this will require a minimum of actions.
-1. Move to “Manage Jenkins” -> “Configure System” -> scroll down and find the panel with the name - <b>“z/OS Connection List”</b>
-2. This setting allows you to add all necessary z/OS systems and configure access to them.
-   It is necessary to set the connection name (it is also the ID for the call in the code). For the example: ```z/os-connection-name```
-3. The URL address and port of the required mainframe to connect via z/OSMF. Example: ```https://<ip-addres>:<port number>```
-4. Add credentials (Mainframe User ID + Password) under which you can connect to the system.
-
-You can save as many connections as you like, the system will keep the corresponding user IDs/passwords.
-
-## Use case
-- Add a zosmf connection in settings (<b>Manage Jenkins -> Configure System -> z/OS Connection List</b>). Enter a connection name, zosmf url, username and password.
-- Create a new item -> ```Pipeline``` and open its configuration.
-  Create a <b>zosmf</b> section inside the <b>steps</b> of the <b>stage</b> and pass the connection name as a parameter of the section. Inside the zosmf body invoke necessary zosmf functions (they will be automatically done in a specified connection context). Take a look at the example below:
-```groovy
-stage ("stage-name") {
-  steps {
-    // ...
-    zosmf("z/os-connection-name") {
-        submitJob "//'EXAMPLE.DATASET(JCLJOB)'"
-        submitJobSync "//'EXAMPLE.DATASET(JCLJOB)'"
-        downloadDS "USER.LIB(MEMBER)"
-        downloadDS dsn:"USER.LIB(MEMBER)", vol:"VOL001"
-        allocateDS dsn:"STV.TEST5", alcUnit:"TRK", dsOrg:"PS", primary:1, secondary:1, recFm:"FB"
-        writeFileToDS dsn:"USER.DATASET", file:"workspaceFile"
-        writeFileToDS dsn:"USER.DATASET", file:"D:\\files\\localFile"
-        writeToDS dsn:"USER.DATASET", text:"Write this string to dataset"
-        writeFileToMember dsn:"USER.DATASET", member:"MEMBER", file:"workspaceFile"
-        writeFileToMember dsn:"USER.DATASET", member:"MEMBER", file:"D:\\files\\localFile"
-        writeToMember dsn:"USER.DATASET", member:"MEMBER", text:"Write this string to member"
-
-        writeToFile destFile: "u/USER/doc", text: "Hello there"
-        writeFileToFile destFile: "u/USER/doc", sourceFile: "myfile.txt"
-        writeFileToFile destFile: "u/USER/doc", sourceFile: "myfile.txt", binary: "true"
-
-        deleteDataset dsn:"USER.DATASET"
-        deleteDataset dsn:"USER.DATASET", member:"MEMBER1"
-        deleteDatasetsByMask mask:"USER.DATASET.*"
-    }
-    // ...
-  }
-}
-```
-
 ## How to run Jenkins plugin in Debug mode in a local Jenkins sandbox
 
 For debugging purposes run following Maven command from plugin project directory:

diff --git a/src/main/kotlin/org/zowe/zdevops/config/ZOSConnection.kt b/src/main/kotlin/org/zowe/zdevops/config/ZOSConnection.kt
@@ -8,7 +8,7 @@
  * Copyright IBA Group 2022
  */
 
-package org.zowe.zdevops.config;
+package org.zowe.zdevops.config
 
 import com.cloudbees.plugins.credentials.CredentialsMatchers
 import com.cloudbees.plugins.credentials.CredentialsProvider

diff --git a/src/main/kotlin/org/zowe/zdevops/declarative/AbstractZosmfAction.kt b/src/main/kotlin/org/zowe/zdevops/declarative/AbstractZosmfAction.kt
@@ -21,12 +21,9 @@ import hudson.tasks.BuildStepDescriptor
 import hudson.tasks.Builder
 import jenkins.tasks.SimpleBuildStep
 import org.zowe.kotlinsdk.zowe.client.sdk.core.ZOSConnection
-import org.zowe.zdevops.Messages
-import org.zowe.zdevops.config.ZOSConnectionList
-import org.zowe.zdevops.utils.validateConnection
+import org.zowe.zdevops.utils.getZoweZosConnection
 import java.io.PrintWriter
 import java.io.StringWriter
-import java.net.URL
 import java.nio.charset.StandardCharsets
 
 abstract class AbstractZosmfAction : Builder(), SimpleBuildStep {
@@ -39,20 +36,7 @@ abstract class AbstractZosmfAction : Builder(), SimpleBuildStep {
 
   override fun perform(run: Run<*, *>, workspace: FilePath, env: EnvVars, launcher: Launcher, listener: TaskListener) {
     val connectionName = workspace.read().readBytes().toString(StandardCharsets.UTF_8)
-    val connection = ZOSConnectionList.resolve(connectionName) ?: let {
-
-      val exception = IllegalArgumentException(Messages.zdevops_config_ZOSConnection_resolve_unknown(connectionName))
-      val sw = StringWriter()
-      exception.printStackTrace(PrintWriter(sw))
-      listener.logger.println(sw.toString())
-      throw exception
-    }
-    val connURL = URL(connection.url)
-    val zoweConnection = ZOSConnection(
-      connURL.host, connURL.port.toString(), connection.username, connection.password, connURL.protocol
-    )
-
-    validateConnection(zoweConnection)
+    val zoweConnection = getZoweZosConnection(connectionName, listener)
 
     runCatching {
       perform(run, workspace, env, launcher, listener, zoweConnection)

diff --git a/src/main/kotlin/org/zowe/zdevops/declarative/ZosmfStepDeclarative.kt b/src/main/kotlin/org/zowe/zdevops/declarative/ZosmfStepDeclarative.kt
@@ -13,17 +13,24 @@ package org.zowe.zdevops.declarative
 import hudson.EnvVars
 import hudson.Extension
 import hudson.FilePath
-import hudson.model.*
 import hudson.model.Run
+import hudson.model.TaskListener
 import org.jenkinsci.plugins.workflow.steps.Step
 import org.jenkinsci.plugins.workflow.steps.StepContext
 import org.jenkinsci.plugins.workflow.steps.StepDescriptor
 import org.jenkinsci.plugins.workflow.steps.StepExecution
 import org.kohsuke.stapler.DataBoundConstructor
+import org.zowe.zdevops.utils.getZoweZosConnection
+import org.zowe.zdevops.utils.validateConnection
 
 
 class ZosmfStepDeclarative @DataBoundConstructor constructor(private val connectionName: String) : Step() {
   override fun start(context: StepContext): StepExecution {
+    val listener: TaskListener? = context.get(TaskListener::class.java)
+    val zosConnection =  getZoweZosConnection(connectionName, listener)
+
+    validateConnection(zosConnection)
+
     return ZosmfExecution(connectionName, context)
   }