Launch envoy through capability wrapper #2050

swalberg · 2019-11-21T18:46:13Z

(if capabilities are present)

When capabilities are explicitly dropped

sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-7cb95448b8-mjxhp | grep cap
2019-11-21 17:59:05 AMBASSADOR INFO cap_net_bind_service is not supported, launching Envoy directly

When capabilities are present

sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-bdbc4675d-hmq82 | grep cap
2019-11-21 18:00:08 AMBASSADOR INFO cap_net_bind_service is supported, launching Envoy through a wrapper
2019/11/21 18:00:08 Starting Envoy with CAP_NET_BIND_SERVICE capability
2019/11/21 18:00:08 Succeeded in setting capabilities

Additionally, if the capabilities are present, you can bind Envoy to a
low port, e.g. 80 or 443:

bash-5.0$ netstat -anp | grep 0.0.0.0:80
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
bash-5.0$ ps -ef | grep [w]rapper
  167 8888      0:06 {envoy} wrapper -c /ambassador/bootstrap-ads.json

Related Issues

Fixes #1841

Testing

Manual testing and added CI tests.

Todos

Tests
Documentation

Other

If this is a documentation change, please open the pull request at https://github.com/datawire/ambassador-docs instead.
Code changes should occur against master.

swalberg · 2019-11-21T18:47:12Z

Don't merge this just yet... Going to do a bit more testing. The tests don't run locally and I want to see if they work in CI.

swalberg · 2019-12-16T14:29:02Z

This works fine locally, I'll be darned if I can get tests to pass (on this branch or master) on my Mac or on a new k3s cluster though.

swalberg · 2019-12-17T00:28:30Z

With the latest rebase it's now only my tests that are failing. I'll try it again locally.

kflynn · 2020-03-08T03:23:40Z

Hey, now that we've gotten past the 1.0 transition I'd like to get this landed. Need help from us to get tests sorted out?

swalberg · 2020-03-09T16:45:15Z

Hey, yes, please. Give me some time to get caught up and rebased so I know what the current problem is. Thanks!

setcap(8) doesn't work on symlinks. This was a regression in 1.3.0. Fortunately, because #2050 hasn't landed yet, that didn't actually affect anything.

stale · 2020-05-08T21:55:11Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

swalberg · 2020-05-08T22:55:34Z

A million apologies. Let me get this rebased and over the line.

swalberg · 2020-05-09T00:47:02Z

Is there still an issue running tests on a mac? I tried make test after setting the registry and k8s config and get failure in smoke tests:

Teleproxy exited with 1 error(s):
  K8S: Get https://kubernetes.docker.internal:6443/api?timeout=32s: dial tcp: lookup kubernetes.docker.internal on 127.0.0.11:53: no such host
2020/05/09 00:44:58 poll get: Get http://httptarget: dial tcp: lookup httptarget on 127.0.0.11:53: no such host
--- FAIL: TestSmoke (1.01s)
    teleproxy_test.go:44: exit status 1
    teleproxy_test.go:106: giving up because we have already failed
    teleproxy_test.go:52: os: process already finished

kflynn · 2020-05-19T15:18:03Z

@swalberg I run them on a Mac a lot -- what kind of cluster were you trying to use?

swalberg · 2020-05-20T17:52:42Z

I'm using docker-desktop. Happy to change it to whatever works.

kflynn · 2020-06-02T14:44:59Z

I don't think the whole test suite can fit into Docker on a Mac. 🙁 Internally, we have an old tool called Kubernaut that provisions clusters in EC2, but really any cluster should work, it's just that the tests spin up a lot of Ambassadors, so a beefy cluster is a must.

Alternately, if you merge master yet again, CI should be able to run the tests for you again. You've been running this at your site, yes?

swalberg · 2020-06-04T22:03:45Z

Oh, I can find a bigger test cluster then. I had it in our cluster for testing but since we're so far behind on Ambassador I can't use current versions. This'll change soon 🤞 .

swalberg · 2020-06-23T15:13:39Z

Just a note that we've upgraded things here enough that I can get back to testing.

swalberg · 2020-06-23T15:59:25Z

I'm getting crashes on the capabilities wrapper now. I see there was a minor refactoring in 58c1267 to extract it which seems related. I'm having trouble explaining what I'm seeing, but it looks like the return address of capset is getting overwritten, resulting in a segfault. BUT, if I throw a fmt.Println in that method (anywhere) it works.

I can't explain it yet, but here's a gist to show I'm not nuts: https://gist.github.com/swalberg/c4b50e395c239920863f898551a12c24

swalberg · 2020-06-29T13:11:01Z

Bah is there a way to re-run this? Looks like there was a connection reset in the middle of it when it tries to pull down the gcloud tools.

swalberg · 2020-07-06T14:33:17Z

Is there a way to run just one test locally? I tried make pytest TEST_NAME=CanBindToLowPort but it seems to run everything.

kflynn · 2020-07-06T15:14:20Z

make push pytest-only PYTEST_ARGS="-k CanBindToLowPort" should do it.

…eshu/dstuff Separate dlib

…et/dev/master/pin-pip ambassador/builder: pin pip to 20.2.4

fix doc typo add browser test update changelog add some unit tests add comments to tests

…ite-preview Always run website-preview, even when it'll be a no-op.

…to flynn/dev/from-oss

…-datawire Sync Datawire changes

(cherry picked from commit b602adf)

…adf-master Update gRPC example for insecure host

…/tutorial-master [master] Revise tutorial

(if capabilities are present) When capabilities are explicitly dropped ``` sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-7cb95448b8-mjxhp | grep cap 2019-11-21 17:59:05 AMBASSADOR INFO cap_net_bind_service is not supported, launching Envoy directly ``` When capabilities are present ``` sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-bdbc4675d-hmq82 | grep cap 2019-11-21 18:00:08 AMBASSADOR INFO cap_net_bind_service is supported, launching Envoy through a wrapper 2019/11/21 18:00:08 Starting Envoy with CAP_NET_BIND_SERVICE capability 2019/11/21 18:00:08 Succeeded in setting capabilities ``` Additionally, if the capabilities are present, you can bind Envoy to a low port, e.g. 80 or 443: ``` bash-5.0$ netstat -anp | grep 0.0.0.0:80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - bash-5.0$ ps -ef | grep [w]rapper 167 8888 0:06 {envoy} wrapper -c /ambassador/bootstrap-ads.json ``` Add missing import Try commenting out tests to see if it's me

This reverts commit d2cf331.

(if capabilities are present) When capabilities are explicitly dropped ``` sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-7cb95448b8-mjxhp | grep cap 2019-11-21 17:59:05 AMBASSADOR INFO cap_net_bind_service is not supported, launching Envoy directly ``` When capabilities are present ``` sean~/play/ambassador (use_capabilities_wrapper *%)$ kubectl logs ambassador-bdbc4675d-hmq82 | grep cap 2019-11-21 18:00:08 AMBASSADOR INFO cap_net_bind_service is supported, launching Envoy through a wrapper 2019/11/21 18:00:08 Starting Envoy with CAP_NET_BIND_SERVICE capability 2019/11/21 18:00:08 Succeeded in setting capabilities ``` Additionally, if the capabilities are present, you can bind Envoy to a low port, e.g. 80 or 443: ``` bash-5.0$ netstat -anp | grep 0.0.0.0:80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - bash-5.0$ ps -ef | grep [w]rapper 167 8888 0:06 {envoy} wrapper -c /ambassador/bootstrap-ads.json ``` Add missing import Try commenting out tests to see if it's me

This reverts commit d2cf331.

Yes, it segfaults unless that Println is there. Something about that data.Inheritable smashing the retval on the stack.

The Ambassador container needs both the NET_BIND_SERVICE capability and also securityContext.allowPrivilegeEscalation to be enabled, so updating the init script to check for both. It checks to make sure it's not disabled (love these double negatives) so that if we're on a non docker runtime it won't get tripped up.

Old place didn't seem to get executed first

stale · 2021-02-02T04:41:33Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

swalberg · 2021-06-25T15:42:24Z

Funny enough, PodSecurityPolicies are being removed from K8s, which is the reason I needed this wrapper.

Seems like there are a lot of changes since I looked at this. Is this solution still viable?

LukeShu · 2022-01-27T21:31:28Z

FWIW, I still want to get this in.

akevdmeer · 2023-05-01T16:00:03Z

This feature would still be nice.

swalberg force-pushed the use_capabilities_wrapper branch from 287fac7 to 8d33416 Compare November 21, 2019 21:09

swalberg force-pushed the use_capabilities_wrapper branch from 8d33416 to f9d061c Compare December 5, 2019 16:32

swalberg force-pushed the use_capabilities_wrapper branch from f9d061c to 0d43857 Compare December 16, 2019 13:33

swalberg force-pushed the use_capabilities_wrapper branch from c442dfe to 15c6d4e Compare January 22, 2020 18:47

LukeShu added a commit that referenced this pull request Apr 27, 2020

capabilities_wrapper: Correctly set the filesystem capabilities

d7931d1

setcap(8) doesn't work on symlinks. This was a regression in 1.3.0. Fortunately, because #2050 hasn't landed yet, that didn't actually affect anything.

LukeShu mentioned this pull request Apr 27, 2020

Fixup capabilities_wrapper #2648

Merged

6 tasks

stale bot added the stale Issue is stale and will be closed label May 8, 2020

stale bot removed the stale Issue is stale and will be closed label May 8, 2020

swalberg force-pushed the use_capabilities_wrapper branch from 86f52b8 to dc6922a Compare May 8, 2020 23:15

swalberg force-pushed the use_capabilities_wrapper branch from 6af5080 to 0f9801b Compare June 29, 2020 12:55

swalberg force-pushed the use_capabilities_wrapper branch 2 times, most recently from a20b8a6 to 778c8bd Compare July 6, 2020 12:47

swalberg force-pushed the use_capabilities_wrapper branch from 778c8bd to dd9d184 Compare July 6, 2020 13:22

LukeShu and others added 22 commits November 30, 2020 12:29

(from AES) Merge pull request emissary-ingress#2099 from datawire/luk…

a5edf11

…eshu/dstuff Separate dlib

(from AES) Merge branch 'master' into rhs/scaling-docs

0e30163

(from AES) PR feedback

4026de8

(from AES) Merge pull request emissary-ingress#2104 from datawire/esm…

47baf57

…et/dev/master/pin-pip ambassador/builder: pin pip to 20.2.4

(from AES) Merge branch 'master' into rhs/scaling-docs

4244dab

(from AES) OAuth2: Fix a regression in defaults for scope validation

cfab809

fix doc typo add browser test update changelog add some unit tests add comments to tests

Merge pull request emissary-ingress#3085 from datawire/flynn/dev/webs…

fe83e07

…ite-preview Always run website-preview, even when it'll be a no-op.

(from AES) Merge commit 'fe83e07e64b460942bc59ba262ea4ceccd8c6068' in…

45c8661

…to flynn/dev/from-oss

Merge pull request emissary-ingress#3096 from datawire/flynn/dev/from…

ce9f3f0

…-datawire Sync Datawire changes

Update gRPC example for insecure host

b17dfe5

(cherry picked from commit b602adf)

Drop extraneous space.

bd62939

Merge pull request emissary-ingress#3092 from datawire/flynn/dev/b602…

71e81b7

…adf-master Update gRPC example for insecure host

Merge pull request emissary-ingress#3094 from datawire/mattmcclure-dw…

759065d

…/tutorial-master [master] Revise tutorial

Revert "Try commenting out tests to see if it's me"

5a53b52

This reverts commit d2cf331.

Update to new test harness style

dcd8162

Revert "Try commenting out tests to see if it's me"

8288e0e

This reverts commit d2cf331.

Update to new test harness style

dea25a9

Workaround for segfault

c2a13ab

Yes, it segfaults unless that Println is there. Something about that data.Inheritable smashing the retval on the stack.

Trying out something with tests

b2d3e74

swalberg force-pushed the use_capabilities_wrapper branch from a1a404f to b2d3e74 Compare December 2, 2020 15:55

Moving the xfail to a better spot

2a70da2

Old place didn't seem to get executed first

stale bot added the stale Issue is stale and will be closed label Feb 2, 2021

LukeShu force-pushed the master branch from 1bbb412 to 8ee9cfc Compare May 8, 2021 23:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Launch envoy through capability wrapper #2050

Launch envoy through capability wrapper #2050

swalberg commented Nov 21, 2019 •

edited

Loading

swalberg commented Nov 21, 2019

swalberg commented Dec 16, 2019

swalberg commented Dec 17, 2019

kflynn commented Mar 8, 2020

swalberg commented Mar 9, 2020

stale bot commented May 8, 2020

swalberg commented May 8, 2020

swalberg commented May 9, 2020

kflynn commented May 19, 2020

swalberg commented May 20, 2020

kflynn commented Jun 2, 2020

swalberg commented Jun 4, 2020

swalberg commented Jun 23, 2020

swalberg commented Jun 23, 2020

swalberg commented Jun 29, 2020

swalberg commented Jul 6, 2020

kflynn commented Jul 6, 2020

stale bot commented Feb 2, 2021

swalberg commented Jun 25, 2021

LukeShu commented Jan 27, 2022

akevdmeer commented May 1, 2023

Launch envoy through capability wrapper #2050

Are you sure you want to change the base?

Launch envoy through capability wrapper #2050

Conversation

swalberg commented Nov 21, 2019 • edited Loading

Related Issues

Testing

Todos

Other

swalberg commented Nov 21, 2019

swalberg commented Dec 16, 2019

swalberg commented Dec 17, 2019

kflynn commented Mar 8, 2020

swalberg commented Mar 9, 2020

stale bot commented May 8, 2020

swalberg commented May 8, 2020

swalberg commented May 9, 2020

kflynn commented May 19, 2020

swalberg commented May 20, 2020

kflynn commented Jun 2, 2020

swalberg commented Jun 4, 2020

swalberg commented Jun 23, 2020

swalberg commented Jun 23, 2020

swalberg commented Jun 29, 2020

swalberg commented Jul 6, 2020

kflynn commented Jul 6, 2020

stale bot commented Feb 2, 2021

swalberg commented Jun 25, 2021

LukeShu commented Jan 27, 2022

akevdmeer commented May 1, 2023

swalberg commented Nov 21, 2019 •

edited

Loading