Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deletion of VMs before build finishes #475

Closed
GeiserX opened this issue Oct 9, 2024 · 1 comment
Closed

Deletion of VMs before build finishes #475

GeiserX opened this issue Oct 9, 2024 · 1 comment

Comments

@GeiserX
Copy link

GeiserX commented Oct 9, 2024 •
edited
Loading

Jenkins and plugins versions report

Environment 
Jenkins: 2.462.1
OS: Linux - 6.1.85+
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:12.4.0
ansicolor:1.0.4
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-110.v77252fb_d4da_5
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
blueocean:1.27.14
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.14
blueocean-commons:1.27.14
blueocean-config:1.27.14
blueocean-core-js:1.27.14
blueocean-dashboard:1.27.14
blueocean-display-url:2.4.3
blueocean-events:1.27.14
blueocean-git-pipeline:1.27.14
blueocean-github-pipeline:1.27.14
blueocean-i18n:1.27.14
blueocean-jira:1.27.14
blueocean-jwt:1.27.14
blueocean-personalization:1.27.14
blueocean-pipeline-api-impl:1.27.14
blueocean-pipeline-editor:1.27.14
blueocean-pipeline-scm-api:1.27.14
blueocean-rest:1.27.14
blueocean-rest-impl:1.27.14
blueocean-web:1.27.14
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1169.va_f810c56e895
build-monitor-plugin:1.14-883.vf620a_44eb_ec1
build-user-vars-plugin:166.v52976843b_435
caffeine-api:3.1.8-133.v17b_1ff2e0599
categorized-view:1.13
checks-api:2.2.0
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_
cloudbees-folder:6.928.v7c780211d66e
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-lang3-api:3.14.0-76.vda_5591261cfe
commons-text-api:1.12.0-119.v73ef73f2345d
compact-columns:1.199.v61a_f51712072
conditional-buildstep:1.4.3
config-driven-pipeline:1.3
config-file-provider:973.vb_a_80ecb_9a_4d0
configuration-as-code:1836.vccda_4a_122a_a_e
conventional-commits:0.11.2
copyartifact:749.vfb_dca_a_9b_6549
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
data-tables-api:2.0.8-1
dependency-check-jenkins-plugin:5.5.1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.6-90.ve7c5c7535ddd
docker-workflow:580.vc0c340686b_54
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
embeddable-build-status:487.va_0ef04c898a_2
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
extended-read-permission:53.v6499940139e5
external-monitor-job:215.v2e88e894db_f8
favorite:2.218.vd60382506538
font-awesome-api:6.5.2-1
forensics-api:2.4.0
git:5.2.2
git-client:5.0.0
git-server:126.v0d945d8d2b_39
github:1.39.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1790.v5a_7859812c8d
gitlab-api:5.3.0-91.v1f9a_fda_d654f
gitlab-branch-source:704.vc7f1202d7e14
gitlab-oauth:1.19
gitlab-plugin:1.8.1
google-compute-engine:4.575.v6969b_7c435eb_
google-metadata-plugin:0.5
google-oauth-plugin:1.330.vf5e86021cb_ec
google-storage-plugin:1.360.v6ca_38618b_41f
gradle:2.12
gson-api:2.11.0-41.v019fcf6125dc
h2-api:11.1.4.199-30.v1c64e772f3a_c
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.36
http_request:1.18
ignore-committer-strategy:1.0.4
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jacoco:3.3.6
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.14
jenkinslint:0.14.0
jersey2-api:2.42-147.va_28a_44603b_d5
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jobConfigHistory:1229.v3039470161a_d
jobcacher:481.v15f51ca_4c6b_7
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
junit-attachments:239.v9e003a_c80a_8c
kubernetes:4253.v7700d91739e5
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
lockable-resources:1255.vf48745da_35d0
mailer:472.vf7c289a_4b_420
material-theme:0.5.2-rc100.6121925fe229
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
maven-info:0.3.1
maven-plugin:3.23
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.1-117.v2f1a_b_66ff91d
mina-sshd-api-core:2.13.1-117.v2f1a_b_66ff91d
monitoring:1.99.0
nodejs:1.6.1
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
parameterized-scheduler:277.v61a_4b_a_49a_c5c
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-config-history:1.6
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:727.ve832a_9244dfa_
pipeline-input-step:495.ve9c153f6067b_
pipeline-maven:1421.v610fa_b_e2d60e
pipeline-maven-api:1421.v610fa_b_e2d60e
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2205.vc9522a_9d5711
pipeline-model-definition:2.2205.vc9522a_9d5711
pipeline-model-extensions:2.2205.vc9522a_9d5711
pipeline-multibranch-defaults:2.1
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-usage-plugin:4.5
plugin-util-api:4.1.0
prism-api:1.29.0-15
prometheus:773.v3b_62d8178eec
pubsub-light:1.18
remote-file:1.24
resource-disposer:0.23
run-condition:1.7
scm-api:690.vfc8b_54395023
script-security:1341.va_2819b_414686
slack:722.vd07f1ea_7ff40
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.27
ssh-agent:367.vf9076cd4ee21
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
theme-manager:262.vc57ee4a_eda_5d
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
warnings-ng:11.3.0
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1322.v857eeeea_9902
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3908.vd6b_b_5a_a_54010
workflow-durable-task-step:1360.v82d13453da_a_f
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:783.787.v50539468395f
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
ws-cleanup:0.46

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins deployed in Kubernetes v1.29.6-gke.1254000 with the official Helm chart, currently still using 2.462.1-lts-jdk17
Latest actual version of the plugin is being used google-compute-engine:4.575.v6969b_7c435eb_

Reproduction steps

Recently, we have many builds that fail like this:

11:24:18  Cannot contact jenkins-gce-dynamic-agent-jdk21-01b4ii: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4a81fa50:jenkins-gce-dynamic-agent-jdk21-01b4ii": Remote call on jenkins-gce-dynamic-agent-jdk21-01b4ii failed. The channel is closing down or has closed down
11:25:13  Agent jenkins-gce-dynamic-agent-jdk21-01b4ii was deleted; cancelling node body
11:25:13  Could not connect to jenkins-gce-dynamic-agent-jdk21-01b4ii to send interrupt signal to process
11:25:13  [Pipeline] }
11:25:13  [Pipeline] // configFileProvider
11:25:13  [Pipeline] }
11:25:13  [Pipeline] // stage
11:25:13  [Pipeline] stage
11:25:13  [Pipeline] { (Contract Test)
11:25:13  Stage "Contract Test" skipped due to earlier failure(s)
11:25:13  [Pipeline] getContext
11:25:13  [Pipeline] }
11:25:13  [Pipeline] // stage
11:25:13  [Pipeline] stage
11:25:13  [Pipeline] { (Version update)
11:25:13  Stage "Version update" skipped due to earlier failure(s)
11:25:13  [Pipeline] getContext
11:25:13  [Pipeline] }
11:25:13  [Pipeline] // stage
11:25:13  [Pipeline] stage
11:25:13  [Pipeline] { (SonarQube)
11:25:13  Stage "SonarQube" skipped due to earlier failure(s)
11:25:13  [Pipeline] getContext
11:25:13  [Pipeline] }
11:25:13  [Pipeline] // stage
11:25:13  [Pipeline] stage
11:25:13  [Pipeline] { (Build Image)
11:25:13  Stage "Build Image" skipped due to earlier failure(s)
11:25:13  [Pipeline] getContext
11:25:13  [Pipeline] }
11:25:14  [Pipeline] // stage
11:25:14  [Pipeline] stage
11:25:14  [Pipeline] { (Create Branch in DEV Repo)
11:25:14  Stage "Create Branch in DEV Repo" skipped due to earlier failure(s)
11:25:14  [Pipeline] getContext
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // stage
11:25:14  [Pipeline] stage
11:25:14  [Pipeline] { (Update DEV Main Branch)
11:25:14  Stage "Update DEV Main Branch" skipped due to earlier failure(s)
11:25:14  [Pipeline] getContext
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // stage
11:25:14  [Pipeline] stage
11:25:14  [Pipeline] { (Declarative: Post Actions)
11:25:14  [Pipeline] junit
11:25:14  Error when executing always post condition:
11:25:14  Also:   hudson.model.Computer$TerminationRequest: Termination requested at Tue Oct 08 09:25:13 UTC 2024 by Thread[Computer.threadPoolForRemoting [#17367],5,main] [id=1943946]
11:25:14  		at hudson.model.Computer.recordTermination(Computer.java:242)
11:25:14  		at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:85)
11:25:14  		at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:220)
11:25:14  		at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
11:25:14  		at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
11:25:14  		at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
11:25:14  		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
11:25:14  Also:   hudson.model.Computer$TerminationRequest: Termination requested at Tue Oct 08 09:25:13 UTC 2024 by Thread[Computer.threadPoolForRemoting [#17367],5,main] [id=1943946]
11:25:14  		at hudson.model.Computer.recordTermination(Computer.java:242)
11:25:14  		at jenkins.model.Nodes$5.run(Nodes.java:283)
11:25:14  		at hudson.model.Queue._withLock(Queue.java:1409)
11:25:14  		at hudson.model.Queue.withLock(Queue.java:1283)
11:25:14  		at jenkins.model.Nodes.removeNode(Nodes.java:278)
11:25:14  		at jenkins.model.Jenkins.removeNode(Jenkins.java:2257)
11:25:14  		at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:91)
11:25:14  		at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:220)
11:25:14  		at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
11:25:14  		at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
11:25:14  		at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
11:25:14  		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
11:25:14  Also:   hudson.model.Computer$TerminationRequest: Termination requested at Tue Oct 08 09:25:13 UTC 2024 by Thread[Computer.threadPoolForRemoting [#17367],5,main] [id=1943946]
11:25:14  		at hudson.model.Computer.recordTermination(Computer.java:242)
11:25:14  		at hudson.model.Computer.disconnect(Computer.java:512)
11:25:14  		at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:816)
11:25:14  		at jenkins.model.Nodes$5.run(Nodes.java:284)
11:25:14  		at hudson.model.Queue._withLock(Queue.java:1409)
11:25:14  		at hudson.model.Queue.withLock(Queue.java:1283)
11:25:14  		at jenkins.model.Nodes.removeNode(Nodes.java:278)
11:25:14  		at jenkins.model.Jenkins.removeNode(Jenkins.java:2257)
11:25:14  		at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:91)
11:25:14  		at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:220)
11:25:14  		at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
11:25:14  		at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
11:25:14  		at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
11:25:14  		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
11:25:14  		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
11:25:14  Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 84dc4246-a48a-4e8c-b0a2-79235c904646
11:25:14  org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for jenkins-gce-dynamic-agent-jdk21-01b4ii; jenkins-gce-dynamic-agent-jdk21-01b4ii was marked offline: Node is being removed
11:25:14  	at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:188)
11:25:14  	at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
11:25:14  	at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$Translator.get(ExecutorStepDynamicContext.java:153)
11:25:14  	at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:170)
11:25:14  	at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
11:25:14  	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:95)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
11:25:14  	at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75)
11:25:14  	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.StepDescriptor.checkContextAvailability(StepDescriptor.java:263)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:300)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
11:25:14  	at jdk.internal.reflect.GeneratedMethodAccessor533.invoke(Unknown Source)
11:25:14  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
11:25:14  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
11:25:14  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
11:25:14  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
11:25:14  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
11:25:14  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
11:25:14  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
11:25:14  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
11:25:14  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
11:25:14  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:180)
11:25:14  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
11:25:14  	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:163)
11:25:14  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
11:25:14  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
11:25:14  	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:105)
11:25:14  	at WorkflowScript.run(WorkflowScript:252)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:138)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:768)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:402)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:400)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:767)
11:25:14  	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2125)
11:25:14  	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2110)
11:25:14  	at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2151)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy:757)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.runPostConditions(ModelInterpreter.groovy)
11:25:14  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executePostBuild(ModelInterpreter.groovy:735)
11:25:14  	at ___cps.transform___(Native Method)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
11:25:14  	at jdk.internal.reflect.GeneratedMethodAccessor427.invoke(Unknown Source)
11:25:14  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
11:25:14  	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
11:25:14  	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:136)
11:25:14  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
11:25:14  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
11:25:14  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
11:25:14  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
11:25:14  	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
11:25:14  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
11:25:14  	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
11:25:14  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
11:25:14  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
11:25:14  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
11:25:14  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
11:25:14  	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
11:25:14  	at java.base/java.lang.Thread.run(Unknown Source)
11:25:14  
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // stage
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // timeout
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // withEnv
11:25:14  [Pipeline] }
11:25:14  [Pipeline] // node
11:25:14  [Pipeline] End of Pipeline
11:25:14  Agent was removed
11:25:14  org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 33a2b86d-db77-4231-a2a9-3e12f3cd4f30
11:25:14  [GitLab Pipeline Status] Notifying branch build status: CANCELED ...(omitted) #5: Something is wrong with the build of this commit.
11:25:15  [GitLab Pipeline Status] Notified
11:25:15  Finished: ABORTED

And in Google Logging this is what I can see:

NOTICE 2024-10-08T09:23:42.922137Z [protoPayload.serviceName: compute.googleapis.com] [protoPayload.methodName: v1.compute.instances.delete] [protoPayload.resourceName: projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii] [protoPayload.authenticationInfo.principalEmail: jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com] audit_log, method: "v1.compute.instances.delete", principal_email: "jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com"
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "authenticationInfo": {
        "principalEmail": "jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com",
        "serviceAccountKeyName": "//iam.googleapis.com/projects/...(omitted)/serviceAccounts/jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com/keys/7d2e2723ddc7f154d7a1fa1de4f934d6c95d51cf",
        "principalSubject": "serviceAccount:jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com"
      },
      "requestMetadata": {
        "callerIp": "...(omitted)",
        "callerSuppliedUserAgent": "jenkins-google-compute-plugin Google-HTTP-Java-Client/1.42.2 (gzip),gzip(gfe)",
        "callerNetwork": "//compute.googleapis.com/projects/...(omitted)/global/networks/__unknown__",
        "requestAttributes": {
          "time": "2024-10-08T09:23:43.205441Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "compute.googleapis.com",
      "methodName": "v1.compute.instances.delete",
      "authorizationInfo": [
        {
          "resource": "projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii",
          "permission": "compute.instances.delete",
          "granted": true,
          "resourceAttributes": {
            "service": "compute",
            "name": "projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii",
            "type": "compute.instances"
          },
          "permissionType": "ADMIN_WRITE"
        }
      ],
      "resourceName": "projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii",
      "request": {
        "@type": "type.googleapis.com/compute.instances.delete"
      },
      "response": {
        "@type": "type.googleapis.com/operation",
        "user": "jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com",
        "id": "1544391138048820464",
        "status": "RUNNING",
        "name": "operation-1728379422904-623f3b18a0f75-5e42e360-811796a8",
        "zone": "https://www.googleapis.com/compute/v1/projects/...(omitted)/zones/...(omitted)",
        "selfLinkWithId": "https://www.googleapis.com/compute/v1/projects...(omitted)/zones/...(omitted)/operations/1544391138048820464",
        "targetLink": "https://www.googleapis.com/compute/v1/projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii",
        "insertTime": "2024-10-08T02:23:43.140-07:00",
        "selfLink": "https://www.googleapis.com/compute/v1/projects/...(omitted)/zones/...(omitted)/operations/operation-1728379422904-623f3b18a0f75-5e42e360-811796a8",
        "targetId": "5587014708678444962",
        "startTime": "2024-10-08T02:23:43.158-07:00",
        "progress": "0",
        "operationType": "delete"
      },
      "resourceLocation": {
        "currentLocations": [
          ...(omitted)
        ]
      }
    },
    "insertId": "5julepdd27s",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "project_id":...(omitted),
        "zone": ...(omitted),
        "instance_id": "5587014708678444962"
      }
    },
    "timestamp": "2024-10-08T09:23:42.922137Z",
    "severity": "NOTICE",
    "labels": {
      "compute.googleapis.com/root_trigger_id": "75ef9974-a437-469f-a584-a201c4e4865f"
    },
    "logName": "projects/...(omitted)/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "operation-1728379422904-623f3b18a0f75-5e42e360-811796a8",
      "producer": "compute.googleapis.com",
      "first": true
    },
    "receiveTimestamp": "2024-10-08T09:23:43.273498961Z"
  }
> NOTICE 2024-10-08T09:24:39.146122Z [protoPayload.serviceName: compute.googleapis.com] [protoPayload.methodName: v1.compute.instances.delete] [protoPayload.resourceName: projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii] [protoPayload.authenticationInfo.principalEmail: jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com] audit_log, method: "v1.compute.instances.delete", principal_email: "jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com"
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "authenticationInfo": {
        "principalEmail": "jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com",
        "serviceAccountKeyName": "//iam.googleapis.com/projects/...(omitted)/serviceAccounts/jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com/keys/7d2e2723ddc7f154d7a1fa1de4f934d6c95d51cf",
        "principalSubject": "serviceAccount:jenkins-compute-agent-deployer@...(omitted).iam.gserviceaccount.com"
      },
      "requestMetadata": {
        "callerIp": "...(omitted)",
        "callerSuppliedUserAgent": "jenkins-google-compute-plugin Google-HTTP-Java-Client/1.42.2 (gzip),gzip(gfe)",
        "callerNetwork": "//compute.googleapis.com/projects/...(omitted)/global/networks/__unknown__",
        "requestAttributes": {},
        "destinationAttributes": {}
      },
      "serviceName": "compute.googleapis.com",
      "methodName": "v1.compute.instances.delete",
      "resourceName": "projects/...(omitted)/zones/...(omitted)/instances/jenkins-gce-dynamic-agent-jdk21-01b4ii",
      "request": {
        "@type": "type.googleapis.com/compute.instances.delete"
      }
    },
    "insertId": "p19y57d141s",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "project_id": ...(omitted),
        "zone": ...(omitted),
        "instance_id": "5587014708678444962"
      }
    },
    "timestamp": "2024-10-08T09:24:39.146122Z",
    "severity": "NOTICE",
    "labels": {
      "compute.googleapis.com/root_trigger_id": "75ef9974-a437-469f-a584-a201c4e4865f"
    },
    "logName": "projects/...(omitted)/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "operation-1728379422904-623f3b18a0f75-5e42e360-811796a8",
      "producer": "compute.googleapis.com",
      "last": true
    },
    "receiveTimestamp": "2024-10-08T09:24:39.525325425Z"
  }

Expected Results

Builds to finish to completion without being abruptly interrupted.

Actual Results

Builds are left dangling in the middle of it after the issue of the Agent deletion - without any reason - is done.

Anything else?

I tried setting the node timeout period to 10 minutes, but this is not happening, because in the case I posted, it was only after 3 minutes that the ServiceAccount was used to execute the deletion from the plugin itself.
@GeiserX
Copy link
Author

GeiserX commented Oct 9, 2024

Seems like #467 (comment) helped. Why is this the default behaviour in Jenkins when cloning clouds? It's irritating.

@GeiserX GeiserX closed this as completed Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GeiserX