Custom tasks not being executed when using mlDeploySecurity

[mwarnes]

A custom task is defined that clears the required certificates from the system, however, it is not run when called as a dependency of mlDeploySecurity but is called when made a dependency for mlRedeploy.

plugins { 
id"com.marklogic.ml-gradle"version"4.3.2"
}

taskclearTvmCertificates(type: com.marklogic.gradle.task.ServerEvalTask) {
xquery =
"xquery version '1.0-ml';\n" +
"import module namespace pki='http://marklogic.com/xdmp/pki' at '/MarkLogic/pki.xqy';\n" +
"declare namespace x509='http://marklogic.com/xdmp/x509';\n" +
"for \$tvm-cert in pki:get-certificates(pki:get-trusted-certificate-ids())[contains(lower-case(x509:cert/x509:subject/x509:commonName), 'macpro-4698.home')]\n" +
"let \$cert-id := \$tvm-cert/pki:certificate-id\n" +
"return pki:delete-certificate(\$cert-id)\n"
}

mlDeploySecurity.dependsOn clearTvmCertificates

Running gradle mlRedeploy with debugging shows that the clearTvmCertificates task is initialised, but is never called so the certificate is not actually deleted.

2021-11-24T10:08:08.798+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Realize task :clearTvmCertificates' started
2021-11-24T10:08:08.798+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Realize task :clearTvmCertificates'
2021-11-24T10:08:08.798+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Realize task :clearTvmCertificates' completed

However, the certificate template properties are updated.

2021-11-24T10:08:10.230+0200 [DEBUG] [org.springframework.web.client.RestTemplate] HTTP PUT http://localhost:8002/manage/v2/certificate-templates/15266967747338245194/properties
2021-11-24T10:08:10.230+0200 [DEBUG] [org.springframework.web.client.RestTemplate] Accept=[text/plain, application/json, application/*+json, */*]
2021-11-24T10:08:10.230+0200 [DEBUG] [org.springframework.web.client.RestTemplate] Writing [<certificate-template-properties xmlns="http://marklogic.com/manage">
 <template-name>ssl-project-template</template-name>
 <template-description>This template is used to demonstrate how ml-gradle can enable SSL for an app server</template-description>
 <key-type>rsa</key-type>
 <key-options />
 <req>
 <version>0</version>
 <subject>
 <countryName>UK</countryName>
 <organizationName>MarkLogic</organizationName>
 <organizationalUnitName>MarkLogic Server Support</organizationalUnitName>
 </subject>
 </req>
</certificate-template-properties>] as "application/xml"

When this happens the template version is bumped up +1 which is causing the CSR Request to be generated

<pki:template-version>2</pki:template-version>

As such we are left with an updated Template, the original certificate and a dangling CSR Request.

Changing the build.gradle slightly so that clearTvmCertificates is now a dependency of mlRedploy instead and we see a different story. This time the clearTvmCertificates task is called and the certificate is deleted.

mlDeploySecurity.dependsOn clearTvmCertificates

2021-11-24T10:10:34.830+0200 [INFO] [org.gradle.execution.plan.DefaultPlanExecutor] :clearTvmCertificates (Thread[Execution worker for ':',5,main]) started.
2021-11-24T10:10:34.830+0200 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Creating new cache for executionHistory, path /Users/mwarnes/git/ml-gradle-airbus/.gradle/7.2/executionHistory/executionHistory.bin, access org.gradle.cache.internal.DefaultCacheAccess@23c20e00
2021-11-24T10:10:34.831+0200 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Acquiring file lock for execution history cache (/Users/mwarnes/git/ml-gradle-airbus/.gradle/7.2/executionHistory)
2021-11-24T10:10:34.831+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire exclusive lock on execution history cache (/Users/mwarnes/git/ml-gradle-airbus/.gradle/7.2/executionHistory).
2021-11-24T10:10:34.831+0200 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on execution history cache (/Users/mwarnes/git/ml-gradle-airbus/.gradle/7.2/executionHistory).
2021-11-24T10:10:34.832+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Task :clearTvmCertificates' started
2021-11-24T10:10:34.833+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Executing task ':clearTvmCertificates'' started
2021-11-24T10:10:34.833+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute serverEval for :clearTvmCertificates' started
2021-11-24T10:10:34.844+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 starting : OkHttp ConnectionPool
2021-11-24T10:10:34.844+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 run again after 125 s : OkHttp ConnectionPool
2021-11-24T10:10:34.845+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 finished run in 320 µs: OkHttp ConnectionPool
2021-11-24T10:10:34.847+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 starting : OkHttp ConnectionPool
2021-11-24T10:10:34.847+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 run again after 125 s : OkHttp ConnectionPool
2021-11-24T10:10:34.847+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 finished run in 316 µs: OkHttp ConnectionPool
2021-11-24T10:10:34.974+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 starting : OkHttp ConnectionPool
2021-11-24T10:10:34.974+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 run again after 125 s : OkHttp ConnectionPool
2021-11-24T10:10:34.974+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Execute serverEval for :clearTvmCertificates'
2021-11-24T10:10:34.974+0200 [DEBUG] [okhttp3.internal.concurrent.TaskRunner] Q10000 finished run in 483 µs: OkHttp ConnectionPool
2021-11-24T10:10:34.975+0200 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Executing task ':clearTvmCertificates''

The certificate template properties are updated and the template-version bumped +1 and a CSR Request generated.

2021-11-24T10:10:36.175+0200 [DEBUG] [org.springframework.web.client.RestTemplate] Writing [<certificate-template-properties xmlns="http://marklogic.com/manage">
 <template-name>ssl-project-template</template-name>
 <template-description>This template is used to demonstrate how ml-gradle can enable SSL for an app server</template-description>
 <key-type>rsa</key-type>
 <key-options />
 <req>
 <version>0</version>
 <subject>
 <countryName>UK</countryName>
 <organizationName>MarkLogic</organizationName>
 <organizationalUnitName>MarkLogic Server Support</organizationalUnitName>
 </subject>
 </req>
</certificate-template-properties>] as "application/xml"

This time the certificate is inserted, so the CSR Request is removed and everything is as expected.

2021-11-24T10:10:36.400+0200 [DEBUG] [org.springframework.web.client.RestTemplate] HTTP POST http://localhost:8002/manage/v2/certificate-templates/9883452173588630404
2021-11-24T10:10:36.400+0200 [DEBUG] [org.springframework.web.client.RestTemplate] Accept=[text/plain, application/json, application/*+json, */*]
2021-11-24T10:10:36.400+0200 [DEBUG] [org.springframework.web.client.RestTemplate] Writing [{"operation":"insert-host-certificates","certificates":[{"certificate":{"cert":"-----BEGIN CERTIFICATE-----\nMIIDUjCCA...\nRk/1QMF1A1alY0J04Qx18SOScPjf2sKq/KwAz7iCISlqisCrzPeDpniwEULKgnVZ\nT9vrWQ/3a1Ky/dliqN+UOW2dl4O9R+aK1Ke0A/FMJsecDUcm3+E=\n-----END CERTIFICATE-----\n","pkey":"-----BEGIN RSA PRIVATE KEY-----\n...\n30Yl6gJqrlU6Wm1NVG5DBadHJlQT0QCsNKskVCIfFAaXdi2rDidj5NSSIoFlxiRJ\nZTK8mdDtsCj31V2Pn7suL29Mu9v5vWd/gP3LCezmkL1AoPp0ZpPTOAbOG4AZqYJ6\nEDjkeQKBgDmUD2RWMCdcg6mrVLIPK9RQgkxo75iBkwX2KbRIjJ5klMMpqfsLfO1Y\neUCACklVnJDch9wgh6SMivaJ7L6oY5X4VLpKk+yW7WWzyOgDcbNryuIWX8Id+bkJ\n1WXX6hsR6W2R0pwK6sH1pjYLnwbv5AeBmtpUc1nCFf8hV4pFiJ0g\n-----END RSA PRIVATE KEY-----\n"}}]}] as "application/json;charset=UTF-8"

[grtjn]

The custom task was written by me for my current employer. Slight nuance, we deliberately hooked it up to mlDeploySecurity, as we don't always need to fully reload the certs. If we run mlDeploySecurity, everything seems clean, but when we run mlDeploy or mlRedeploy without clearing stuff first, we get the dangling CSR's. It still seems to work, but it is very confusing at the least, particularly for someone who is unaware of the nitty gritty details..

[rjrudin]

@mwarnes @grtjn I've opened #621 to address the need for describing how to attach custom tasks to existing tasks. The reason why the behavior in this ticket is occurring is because mlDeploy and mlRedeploy do not depend on mlDeploySecurity. Each mlDeploy* task is a separate entry point that selects the resource types to deploy to ML. So if you make mlDeploySecurity depend on myTask, that won't make mlDeploy depend on myTask because mlDeploy does not invoke mlDeploySecurity.

This is somewhat apparent if you look at the logging of running mlDeploy - Gradle logs every task that's invoked, and you won't see mlDeploySecurity logged anywhere. You'll instead see all the commands logged.

The reason for this is due to the fact that it's the underlying ml-app-deployer library that knows the order in which to deploy resources in - and that order is critical knowledge, as there are all sorts of deployment errors that can occur if resources aren't deployed in the right order (for example, an app-server being deployed before a dependent database is created). And it's important for ml-app-deployer to have this knowledge so that it can be reused without depending on Gradle - for example, DHF / Hub Central make use of ml-app-deployer to deploy stuff to ML without depending on Gradle. If ml-gradle instead had this knowledge, we wouldn't be able to reuse any of it outside of Gradle.

Because #621 now exists, I'm closing this ticket because the behavior is actually correct, and #621 will provide guidance on how to know which ml-gradle tasks should be configured to depend on a custom task.