Attempt to make Blockhole blocking on L7 #4
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
henrybear327
force-pushed
the
fix/e2e_blackhole_proxy_review
branch
2 times, most recently
from
4b5895d
to
5290dea
Compare
fuweid
reviewed
Apr 26, 2024
| - map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /version | ||
| - map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /raft/probing | ||
| */ | ||
| if req, err := http.ReadRequest(bufio.NewReader(bytes.NewBuffer(data))); err != nil { |
There was a problem hiding this comment.
quick question: it's going to read data and marshal into request?
There was a problem hiding this comment.
In my current design no, I only look at the header X-PeerURLs
If there are other fields within the request body that’s worth looking into in your mind, please let me know!
Thank you!
Shorten the wait time for the open connection to expire to 5s
henrybear327
force-pushed
the
fix/e2e_blackhole_proxy_review
branch
from
5290dea
to
fab864d
Compare
Based on Fu Wei's idea, we employ blocking on L7 but without using
external tools.
[Main ideas]
The main idea is
1) utilize the `X-PeerURLs` field from the header, as we know all
traffic to peer will contain this header field.
2) As we also know that all nodes will create direct connections with their
peers, so the traffic blocking will have to happen at all nodes'
proxies, contrary to the current design, where only the proxy of the
peer is being blackholed.
Based on the main ideas, we introduce a SSL termination proxy so we can
obtain the `X-PeerURLs` field from the header.
[Issues]
There are 2 known issues with this approach
1) still leaking some traffic. But the leaked traffic (as discussed
later won't affect the blackhole idea that we would like to achieve (as
stream and pipeline traffic between raft nodes are now properly terminated)
2) we would need to employ SSL termination proxy, which might lead to a
small performance hit
For 1), as this way of blocking (by utilizing `X-PeerURLs` from the
header) will miss certain types of traffic to certain endpoints, as
the traffic to some endpoints doesn't have the `X-PeerURLs` field.
Currently, here are the known ones: /members, /version, and /raft/probing.
As you can see from the log, its header doesn't contain the `X-PeerURLs`
field, but only the following fields:
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /members
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /version
- map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] /raft/probing
For 2) in order to read out `X-PeerURLs` from the header, we need to
terminate the SSL connection, as we can't drop cleartext traffic
(ref [1]). Thus, a new option `e2e.WithSSLTerminationProxy(true)`
is introduced, which will change the network flow into
```
A -- B's SSL termination proxy - B's transparent proxy - B
^ newly introduced ^ in the original codebase
```
[Known improvements required before turning RFC into PR]
The prototype needs to be further improved for code review after
fixing the following issues:
- blocking only RX or TX traffic (as currently a call to `blackholeTX`
or `blackholeRX` will black both TX and RX traffic instead of just
the specified one.
- slowness when performing test cleanup (I think this is related to the
SSL timeout setting, but I haven't looked into it yet)
- coding style improvements
References:
[1] etcd-io#15595
henrybear327
force-pushed
the
fix/e2e_blackhole_proxy_review
branch
from
fab864d
to
236d14b
Compare
|
Superseded by etcd-io#17891 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Attempt to make Blockhole blocking on L7
Based on Fu Wei's idea [2], we employ blocking on L7 but without using
external tools.
The main idea is to read out
X-PeerURLsfrom the header, since thisis how peer traffic identifies itself to others. As we also know that
all nodes will create direct connections with their peers, so the traffic
blocking will actually happen at all nodes' proxies (contrary to the
current design, where only the proxy of the peer is being blackholed.
However, this way of blocking (by
X-PeerURLsfrom the header) willmiss certain types of traffic to certain endpoints,
e.g. /members, /version, and /raft/probing.
See below for the log extract,
as its header doesn't contain X-PeerURLs information.
In order to read out
X-PeerURLsfrom the header, we need to terminatethe SSL connection, as we can't drop cleartext traffic (ref [1]). Thus,
a new option
e2e.WithSSLTerminationProxy(true)is introduced, whichwill change the network flow into
This prototype doesn't address
, and the coding convention needs to be improved as it's still a PoC
References:
[1] etcd-io#15595
[2] etcd-io#17737