Specify 2GB max heap for tests #1474

iloveeclipse · 2023-10-25T12:52:09Z

-Xmx2g is needed for Jenkins where RAM is low.

SDK doesn't specify -Xmx anymore, see
#1463

iloveeclipse · 2023-10-25T14:00:30Z

@laeubi : could you please review? I'm not a maven/tycho expert.

laeubi · 2023-10-25T14:12:15Z

Build verification works, so it can't be that bad ;-)

iloveeclipse · 2023-10-25T14:13:21Z

However it looks like maven-war-plugin is also affected (or is the one that causes troubles)???

See https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-767/1/console

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.3.2:war (default-war) on project infocenter-app: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 2]

But that one shouldn't run with SDK / shouldn't be affected by SDK eclipse.ini changes?

iloveeclipse · 2023-10-25T14:17:49Z

Build verification works, so it can't be that bad ;-)

Not sure if any tests are running that use the option? At least they don't say that, searching -Xmx2g doesn't show up in the console... https://ci.eclipse.org/platform/job/eclipse.platform.releng.aggregator/job/PR-1474/1/consoleText

laeubi · 2023-10-25T14:22:56Z

However it looks like maven-war-plugin is also affected (or is the one that causes troubles)???

See https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-767/1/console
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.3.2:war (default-war) on project infocenter-app: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 2]
But that one shouldn't run with SDK / shouldn't be affected by SDK eclipse.ini changes?

That's what makes me really wonder, especially as it seems the Github Actions are not affected in the same way, so for me it looks like maybe an infra change that maven itself has insufficient resources now.

Not sure if any tests are running that use the option?

Yes tests are disabled here (because they run very long) ... my bad, so to verify one needs to run it locally or we need to temporarily remove -DskipTests to see what happening.

laeubi · 2023-10-25T14:24:40Z

But it could also be some kind of meommoryleak... I wonder if we can get a heapdum or if that will blow the Jenkins @fredg02 ?

Enabled tests to see effect of eclipse-platform#1474

iloveeclipse · 2023-10-25T14:29:01Z

or we need to temporarily remove -DskipTests to see what happening.

I've pushed bad commit on top (actually wanted to have different PR). Let see.

iloveeclipse · 2023-10-25T14:32:14Z

so for me it looks like maybe an infra change that maven itself has insufficient resources now.

The only way to know that would be revert #1471 and if OOM's are still there, we have a new infra/maven/tycho issue.

I will do that if the experiment above (with disabling skipTests) would not show anything useful.

iloveeclipse · 2023-10-25T14:33:18Z

And yes, we seem to have yet another infra issues: [ERROR] Caused by: repo.eclipse.org:443 failed to respond... OMFG

akurtakov · 2023-10-25T14:39:42Z

This seems like another pros towards redoing releng as mentioned here #1472 . If projects are really split the resources needed for each will be sufficient more often and a lot less extra configuration will be needed.

iloveeclipse · 2023-10-25T14:51:07Z

This seems like another pros towards redoing releng as mentioned here #1472 . If projects are really split the resources needed for each will be sufficient more often and a lot less extra configuration will be needed.

Not sure it's related. We have some build tasks & some tests running (sometimes in parallel) on a very limited Jenkins VM with only 4GB RAM. Some tasks just don't get enough memory or have some memory leaks - how that can be related to releng/project structure? Even if we would have a single project to build and test it can get OOM if maven/tycho/whatever has a memory leak.

akurtakov · 2023-10-25T14:54:48Z

This seems like another pros towards redoing releng as mentioned here #1472 . If projects are really split the resources needed for each will be sufficient more often and a lot less extra configuration will be needed.

Not sure it's related. We have some build tasks & some tests running (sometimes in parallel) on a very limited Jenkins VM with only 4GB RAM. Some tasks just don't get enough memory or have some memory leaks - how that can be related to releng/project structure? Even if we would have a single project to build and test it can get OOM if maven/tycho/whatever has a memory leak.

If JDT and PDE build and generate their own repos and run their own tests in their own builds that would free releng.aggregator one from 100+ bundles (some of them heavy ones) thus reduce the hardware requirements to execute it.

iloveeclipse · 2023-10-25T15:02:15Z

If JDT and PDE build and generate their own repos and run their own tests in their own builds that would free releng.aggregator one from 100+ bundles (some of them heavy ones) thus reduce the hardware requirements to execute it.

This is not the point here, we are probably talking about different issues.
The OOM's were not reported by PDE or JDT tests.

akurtakov · 2023-10-25T15:06:47Z

If JDT and PDE build and generate their own repos and run their own tests in their own builds that would free releng.aggregator one from 100+ bundles (some of them heavy ones) thus reduce the hardware requirements to execute it.

This is not the point here, we are probably talking about different issues. The OOM's were not reported by PDE or JDT tests.

Maven doesn't always build modules in strict order but rather takes the first dependency order it figures which can differ from build to build, thus if you have OOM it can happen in one bundle in one run and in another on the second one and etc. Thus the solving is usually to reduce the number of modules in order to get things more predictable and manageable.

iloveeclipse · 2023-10-25T15:53:04Z

Looks like JDT model tests are unhappy due ${tycho.surefire.argLine} argument from their pom file. Interestingly, that doesn't seem to be a problem for them running in Jenkins.

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-surefire-plugin:4.0.4-SNAPSHOT:test (default-test) on project org.eclipse.jdt.core.tests.model: An unexpected error occurred while launching the test runtime (process returned error code 1). Command-line used to launch the sub-process was /opt/tools/java/openjdk/jdk-17/latest/bin/java -Dosgi.noShutdown=false -Dosgi.os=linux -Dosgi.ws=gtk -Dosgi.arch=x86_64 -Xmx1G -Djdt.default.test.compliance=1.8 -DDetectVMInstallationsJob.disabled=true ${tycho.surefire.argLine} -Dosgi.clean=true -ea -jar /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/equinox/bundles/org.eclipse.equinox.launcher/target/org.eclipse.equinox.launcher-1.6.600-SNAPSHOT.jar -data /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/eclipse.jdt.core/org.eclipse.jdt.core.tests.model/target/work/data -install /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/eclipse.jdt.core/org.eclipse.jdt.core.tests.model/target/work -configuration /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/eclipse.jdt.core/org.eclipse.jdt.core.tests.model/target/work/configuration -application org.eclipse.tycho.surefire.osgibooter.headlesstest -testproperties /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/eclipse.jdt.core/org.eclipse.jdt.core.tests.model/target/surefire.properties in working directory /home/jenkins/agent/workspace/atform.releng.aggregator_PR-1474/eclipse.jdt.core/org.eclipse.jdt.core.tests.model```

This reverts commit b0b7e79. It is possible that Jenkins test execution is affected by that change (we see some stranhe OOM errors), see comments on eclipse-platform#1474. To make sure the change doesn't cause any side effects, let revert it.

iloveeclipse · 2023-10-25T16:13:01Z

I will do that if the experiment above (with disabling skipTests) would not show anything useful.

See #1475

Bananeweizen · 2023-10-25T16:24:19Z

Maven doesn't always build modules in strict order but rather takes the first dependency order it figures which can differ from build to build

That's worded wrongly, even though the end result will be the same: Maven always enforces a partial sorting that reflects all explicit dependencies, no matter which builder is used later. For other modules that don't have an explicit dependency relation, Maven will not change the order (therefore those remain in the order of declaration in the POM). Therefore the build graph (and dependency order) that Maven core dictates is the same each time. However, in multithreaded execution the builder can choose to iterate the build graph in different ways, which is what then leads to different build order when modules finish later or earlier than in other runs.

My experience with OOMs in non eclipse Maven builds is that this happens especially when you have forked JVM processes and your modules are of similar size, causing parallel builds to trigger identical goals roughly at the same time. E.g. I have another Maven build with 8 threads in parallel, and the worst memory usage happens when all modules run Spotbugs or Asciidoc generation (both with forked JVM) at the same time. I'm not aware of anything that can limit the number of parallel executions of a single maven plugin (e.g. use 8 threads, but at most 4 asciidoc calls in parallel). Could this be similar? Any good ideas how to have a "semaphore" on a specific maven goal?

This reverts commit b0b7e79. It is possible that Jenkins test execution is affected by that change (we see some stranhe OOM errors), see comments on #1474. To make sure the change doesn't cause any side effects, let revert it.

Fixes eclipse-platform#1463

iloveeclipse · 2023-10-27T12:25:20Z

Re-opening this PR, as we see OOM's also without this change.

I plan to push it once https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3896 is fixed, to not mix possible regressions coming from this PR with "other" OOM issues we see right now.

iloveeclipse · 2023-10-27T12:26:35Z

Re-opening this PR, as we see OOM's also without this change.

I plan to push it once https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3896 is fixed, to not mix possible regressions coming from this PR with "other" OOM issues we see right now

Sorry, github mixed everything, wrong PR, wrong diff, wrong comment.

iloveeclipse requested a review from laeubi October 25, 2023 12:52

iloveeclipse mentioned this pull request Oct 25, 2023

Use at least 2GB heap to run org.eclipse.core.tests.resources eclipse-platform/eclipse.platform#767

Closed

iloveeclipse added a commit to iloveeclipse/eclipse.platform.releng.aggregator that referenced this pull request Oct 25, 2023

DO NOT MERGE ME!

5c59e6c

Enabled tests to see effect of eclipse-platform#1474

iloveeclipse marked this pull request as draft October 25, 2023 14:29

iloveeclipse mentioned this pull request Oct 25, 2023

Revert "Don't specify JVM memory limits in eclipse.ini" #1475

Merged

Don't specify JVM memory limits in eclipse.ini

c856903

Fixes eclipse-platform#1463

iloveeclipse force-pushed the issue_1463 branch from 5c59e6c to c856903 Compare October 27, 2023 12:22

iloveeclipse added this to the 4.30 M3 milestone Oct 27, 2023

iloveeclipse closed this Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Specify 2GB max heap for tests #1474

Specify 2GB max heap for tests #1474

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

laeubi commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023 •

edited

Loading

laeubi commented Oct 25, 2023

laeubi commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

Bananeweizen commented Oct 25, 2023

iloveeclipse commented Oct 27, 2023

iloveeclipse commented Oct 27, 2023

Specify 2GB max heap for tests #1474

Specify 2GB max heap for tests #1474

Conversation

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

laeubi commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023 • edited Loading

laeubi commented Oct 25, 2023

laeubi commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

akurtakov commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

iloveeclipse commented Oct 25, 2023

Bananeweizen commented Oct 25, 2023

iloveeclipse commented Oct 27, 2023

iloveeclipse commented Oct 27, 2023

iloveeclipse commented Oct 25, 2023 •

edited

Loading