Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nix: False positives in E2E tests #63

Closed
koxu1996 opened this issue Apr 3, 2024 · 11 comments
Closed

Nix: False positives in E2E tests #63

koxu1996 opened this issue Apr 3, 2024 · 11 comments

Comments

@koxu1996
Copy link
Contributor

koxu1996 commented Apr 3, 2024

Nix's e2e tests relying on curl SHOULD be already broken at least twice - for example the following PRs successfully passed CI checks, even they changed request body in incompatible way:

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 3, 2024

By default, receiving a 500 Internal Server Error would result in an exit code of 0 (indicating no error in the curl command itself). In order to detect server errors, --fail / --fail-with-body should be provided.

@marijanp already submitted fix in #59.

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 4, 2024

CI checks fixed in #59.

@koxu1996 koxu1996 closed this as completed Apr 4, 2024
@marijanp
Copy link
Contributor

marijanp commented Apr 4, 2024

By default, receiving a 500 Internal Server Error would result in an exit code of 0 (indicating no error in the curl command itself). In order to detect server errors, --fail / --fail-with-body should be provided.

@marijanp already submitted fix in #59.

Curl didn't respond with a 500 error code because our server was beghind an nginx server, which returned a html containing a 301 permamently moved error message. This returns a 0 exit code.

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 4, 2024

@marijanp You are wrong 🙄. Here is proof that code 301 is not an error, even if you add --fail-with-body:

$ curl --fail-with-body -v bankier.pl
*   Trying 104.22.72.129:80...
* Connected to bankier.pl (104.22.72.129) port 80
> GET / HTTP/1.1
> Host: bankier.pl
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 04 Apr 2024 12:14:44 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Thu, 04 Apr 2024 13:14:44 GMT
< Location: https://bankier.pl/
< Server: cloudflare
< CF-RAY: 86f1384afb448869-WAW
<
* Connection #0 to host bankier.pl left intact
$ echo $?
0

Also, I didn't say server returned code 500, but only stated that error checking was done incorrectly:

receiving a 500 Internal Server Error WOULD result in an exit code of 0

@marijanp
Copy link
Contributor

marijanp commented Apr 4, 2024

@koxu1996 so what is wrong about my statement? I said it returned 0, because curl fetched a 301. A 0 indicates success.

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 4, 2024

@marijanp Wrong CI checks were fixed by adding --fail-with-body, and code 301 is completely unrelated to this issue.

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 4, 2024

Maybe you are suggesting there is another CI issue?

@marijanp
Copy link
Contributor

marijanp commented Apr 4, 2024

scrot-2024-04-04_2317x294
There were two issues, another one was that I forced SSL in our production config. And I had to override that in the test to get rid of the permament/y moved 301 status code. That's the first reason it didn't error.

@marijanp
Copy link
Contributor

marijanp commented Apr 4, 2024

So even adding --fail-with-body was not the only problem with it:
scrot-2024-04-04_2540x292

@marijanp
Copy link
Contributor

marijanp commented Apr 4, 2024

4e61d68

@koxu1996
Copy link
Contributor Author

koxu1996 commented Apr 4, 2024

Like I said, another CI issue - not related with curl and fixed without context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants