Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not able to communicate through authenticated proxy #381

Open
SebasHein opened this issue Nov 6, 2023 · 19 comments
Open

Not able to communicate through authenticated proxy #381

SebasHein opened this issue Nov 6, 2023 · 19 comments
Labels
ecf transport

Comments

@SebasHein
Copy link

I had a discussion about oomph not able to communicate through proxy anymore with Ed Merks already - see here: https://www.eclipse.org/forums/index.php/m/1861818/#msg_1861818

He suggests to open an issue here and to reproduce the behaviour with the latest eclipse build.

I downloaded I-Build of Eclipse from here https://download.eclipse.org/eclipse/downloads/drops4/I20231105-1800/

My environment is behind a proxy that requests to authentificate with user and password.
In the eclipse preferences I set the same settings as was working well with Eclipse 4-29 ( 2023-09).
Active Provider: Manual
HTTP and HTTPS: Host and Port, user and password according to what was working in former release.
The SOCKS is activated (checkbox is checked) but host, port etc. is empty.
No proxy bypasses except localhost and 127.0.0.1

I then tried to install new software with help → install new software
The update sites I tried were

I immediately get Proxy authentication required exceptions
org.eclipse.equinox.p2.core.ProvisionException: HTTP Proxy Authentication Required: https://download.eclipse.org/releases/2023-12/content.xml
at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache(CacheManager.java:250)
at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.getLocalFile(SimpleMetadataRepositoryFactory.java:69)
at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.load(SimpleMetadataRepositoryFactory.java:89)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:63)
at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791)
at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:689)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:110)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:105)
at org.eclipse.equinox.p2.ui.ProvisioningUI.loadMetadataRepository(ProvisioningUI.java:498)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getMetadataRepository(MetadataRepositoryElement.java:138)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getRepository(MetadataRepositoryElement.java:127)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getQueryable(MetadataRepositoryElement.java:120)
at org.eclipse.equinox.internal.p2.ui.QueryProvider.getQueryDescriptor(QueryProvider.java:92)
at org.eclipse.equinox.internal.p2.ui.model.QueriedElement.fetchChildren(QueriedElement.java:104)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.fetchChildren(MetadataRepositoryElement.java:86)
at org.eclipse.equinox.internal.p2.ui.model.RemoteQueriedElement.fetchDeferredChildren(RemoteQueriedElement.java:38)
at org.eclipse.ui.progress.DeferredTreeContentManager$1.run(DeferredTreeContentManager.java:219)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required
at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:292)
at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:71)
... 1 more

Please let me know what else I can do or information I can share that helps you track down the root cause of this issue.
As far as I understood you switched to Java 11 version of HttpClient.
Is there anything I can do on my side that would be helpful?
@merks
Copy link
Contributor

merks commented Nov 6, 2023

Thank you for testing and confirming the problem is reproducible purely with the Eclipse SDK 4.30.

The issue likely related to, or caused by, the same underlying problem as this one:

#357

@laeubi laeubi changed the title Not able to communicate through proxy Not able to communicate through authenticated proxy Nov 6, 2023
@merks merks mentioned this issue Nov 8, 2023
Merged
@l3-g5
Copy link

l3-g5 commented Nov 9, 2023 •
edited
Loading

I did a few tests with different kinds of proxies (2023-12 M2):

  • SOCKS with and without auth seems to work
  • Https Proxy without authentication works
  • Https Proxy with authentication (digest) not working
  • Https Proxy with authentication (basic auth) not working

All kind of proxies are working with 2023-09, so it seems that we introduced a severe problem with ECF's new httpclientjava.

@jonahgraham
Copy link
Contributor

What is the way out here? IIUC the default changed to be java http client (via ECF) and the alternative apache implementation (also via ECF) is still in place. Can we change the default back to apache for 2023-12?

M3 has already been published, so we need to resolve this ASAP to have a working RC1.

cc: @laeubi @akurtakov

@jonahgraham
Copy link
Contributor

Note for those that are only following this issue by email, @l3-g5's full comment was (side rant about how GitHub email notifications are insufficient):

I did a few tests with different kinds of proxies (2023-12 M2):

  • SOCKS with and without auth seems to work
  • Https Proxy without authentication works
  • Https Proxy with authentication (digest) not working
  • Https Proxy with authentication (basic auth) not working

All kind of proxies are working with 2023-09, so it seems that we introduced a severe problem with ECF's new httpclientjava.

@sratz
Copy link
Contributor

sratz commented Nov 10, 2023

Can we change the default back to apache for 2023-12?

I would also suggest to go for this, otherwise this would be a highly critical issue:

Since the updater itself would be affected by this regression it would be impossible to get out of a broken setup by updating.
And since the default update sites contain /releases/latest, people with basic-auth proxies will end up in such broken setups.

@merks merks mentioned this issue Nov 11, 2023
Open
@laeubi
Copy link
Member

laeubi commented Nov 12, 2023

@merks @akurtakov @jonahgraham I think the safest way forward would be to include both httpjava+http5

@scottslewis mentioned somewhere that one can set a system property to select the provider used but I can't found that comment right now.

So one both are there we only need a N&N entry to show how to switch between those, so people that are affected have a way out but we get qualified feedback if there are any problems.

@merks
Copy link
Contributor

merks commented Nov 12, 2023

In my environment with both ECF implementations present, I tested that this option works:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

@laeubi laeubi mentioned this issue Nov 12, 2023
Merged
@laeubi laeubi mentioned this issue Nov 12, 2023
Merged
@jonahgraham
Copy link
Contributor

In my environment with both ECF implementations present, I tested that this option works:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

Should we make that the default for Oomph and EPP product files and that would allow a soft launch of the backend change for the generally more advanced users that get platform or SDK directly?

@merks
Copy link
Contributor

merks commented Nov 12, 2023

Yes I think so. We will have to hold off until rc1 because it probably breaks m3 disabling the only provider. And we should consider a future where we maybe need a touch point to remove it.

I will do more testing with the installer tomorrow.

@laeubi
Copy link
Member

laeubi commented Nov 12, 2023

I don't think making this default makes much sense because then no one will use it and no one will test it "in the wild" adding it to the eclipse ini in case of problems seems easy enough and everyone without special authenticated proxy won't notice anyways.

@jonahgraham
Copy link
Contributor

Our advanced users will test it first since Eclipse SDK and Eclipse Platform downloads will use it.

But we know that the new version doesn't work in existing cases. I don't think we can ship such a product.

Once we have fixed the known bugs in the Java http version we can revisit changing the default in EPP / Oomph.

@laeubi
Copy link
Member

laeubi commented Nov 12, 2023

@jonahgraham of course its the decision of EPP / Oomph, I just would then ask to adjust the N&N accordingly with whatever you think fits to encourage people to try that out in regular EPP / Oomph setups so we get better feedback, because even if we can fix this there might be more to discover:

@merks merks mentioned this issue Nov 13, 2023
Closed
@jonahgraham jonahgraham mentioned this issue Nov 14, 2023
Closed
jonahgraham added a commit to jonahgraham/simrel.build that referenced this issue Nov 14, 2023
@jonahgraham
Include ecf.filetransfer.httpclient5 in SimRel
7c4a957 
This is to workaround:

- eclipse-equinox/p2#381

And to enable:

- eclipse-packaging/packages#81

in 2023-12 M3.

The Eclipse Platform is expected to contribute this for RC1
but including this from ECF allows us to start testing this
in 2023-12 M3
@jonahgraham jonahgraham mentioned this issue Nov 14, 2023
Merged
jonahgraham added a commit to eclipse-simrel/simrel.build that referenced this issue Nov 14, 2023
@jonahgraham
Include ecf.filetransfer.httpclient5 in SimRel (#57)
5dffc95 
This is to workaround:

- eclipse-equinox/p2#381

And to enable:

- eclipse-packaging/packages#81

in 2023-12 M3.

The Eclipse Platform is expected to contribute this for RC1
but including this from ECF allows us to start testing this
in 2023-12 M3
@merks
Copy link
Contributor

merks commented Nov 15, 2023

@SebasHein

Thanks so much for your error reporting and testing efforts!

The latest I-Build contains what's potentially a fix for the problem you've reported:

https://download.eclipse.org/eclipse/downloads/

Currently that's this version:

https://download.eclipse.org/eclipse/downloads/drops4/I20231115-0150/

The Eclipse SDK will continue to use the new provider as it's default so you could test whether this latest version works better than before...

@SebasHein
Copy link
Author

SebasHein commented Nov 15, 2023 •
edited
Loading

I downloaded it and run just two short tests:

  1. Just start eclipse sdk, put in proxy host, -port and user credentials and tried to fetch an update site (with install new software and link to https://download.eclipse.org/eclipse/updates/4.30 )
    In this test the proxy authentification failed message is still present.
  2. I added -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava to eclipse.ini and repeated what I did in point 1. This time everything works and I was able to fetch the eclipse project update site.

Therefore it looks like what you did functions as expected at least for the proxy configuration that is present in the network I am logged in.

Thanks again for your fast response to my issue and the extra work you had with it - I highly appreciate it!

@scottslewis scottslewis mentioned this issue Nov 16, 2023
Merged
@scottslewis
Copy link

I downloaded it and run just two short tests:

1. Just start eclipse sdk, put in proxy host, -port and user credentials and tried to fetch an update site (with install new software and link to https://download.eclipse.org/eclipse/updates/4.30 )
   In this test the proxy authentification failed message is still present.

Question: Is the stack trace/message from this fail the same as when you first opened this issue? If you have the stack trace from this failed run (or can run it again), perhaps you could paste it in here so I can verify.

Thanks.

@SebasHein
Copy link
Author

Sure I can.

To summarize: I compared
eclipse-SDK-I20231105-1800-win32-x86_64 (this was the not working version)
with
eclipse-SDK-I20231115-0150-win32-x86_64 (working with -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava in eclipse.ini)
The exceptions are exactly the same. Of course no exception is thrown if I use it.
But a warning appears (4 times) :
org.eclipse.core.runtime.Status[plugin=org.eclipse.ecf.provider.filetransfer;code=2;message=Plugin org.eclipse.ecf.provider.filetransfer.httpclientjava excluded from contributing browse factory;severity2;exception=null;children=[]]
grafik

I added a txt file containing the thrown exceptions for either case:
exceptionsThrown.txt

@jonahgraham
Copy link
Contributor

I don't understand why there is an exception in httpclientjava when it is excluded. I'm glad the install works, but it sounds like something else is going on.

I have added the required -D for the Eclipse EPP packages (in a way that it can be removed once httpclientjava supports this use case) in eclipse-packaging/packages#82

@jonahgraham
Copy link
Contributor

I just would then ask to adjust the N&N accordingly with whatever you think fits to encourage people to try that out in regular EPP / Oomph setups so we get better feedback, because even if we can fix this there might be more to discover:

I added it to the EPP N&N: https://github.com/eclipse-packaging/packages/blob/master/CHANGELOG.md#2023-12

@ZebrasTeam
Copy link

The order seems to be the solution.
In the original eclipse.ini of build.id: 20231201-2043 there is
'-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava' in line 15 and
'-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient4' in line 28.

When you move the httpclientjava-entry from line 15 to the last line everything works fine with the proxy.

The question is, why?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ecf transport
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@scottslewis @merks @jonahgraham @laeubi @sratz @l3-g5 @ZebrasTeam @SebasHein