Appsec installation help #30

inthreedee · 2024-12-26T20:17:24Z

I am attempting to configure appsec for my crowdsec install on nginx proxy manager. Manual tests sent with curl from my NPM container to the crowdsec container confirm appsec is working, but requests to the web apps that are protected by crowdsec don't seem to be caught by appsec. For example, accessing my http://{ip-address}/.env does not trigger appsec and the request is passed through to NPM.

I'm sure it's just a configuration problem on my end, but I could use some help sorting out what I've done wrong.

Here's what I have so far:

Using the info in this repo (https://github.com/crowdsecurity/example-docker-compose/tree/main/npm) and the LePresidente fork of NPM, I have crowdsec running. It's connected to the web console and is parsing logs and triggering alerts. So far so good.
Following the instructions in the crowdsec docs (https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty), I have enabled appsec. I'm able to manually test that appsec is working by launching a shell within my NPM container and issuing the curl command shown in the docs to the crowdsec container. It responds with a 403 forbidden as it should. All good here.
I finished the configuration in my docker-compose file and restarted the containers, but attempting to access a .env file from my ip address or the urls of my hosted web apps do not seem to trigger appsec as expected, so it doesn't seem like http requests are being sent to the appsec component. Instead, I see either NPM's 404 page, or a 404 from the web app behind NPM that is being accessed.

This is my docker-compose file.
A note about the ports below in case it matters: I'm using rootless docker with the pasta network driver and iptables rules to forward privileged->unprivileged ports (pasta can't be bound to privileged ports). This allows the source IPs from web traffic to propagate into my logs for manual inspection when necessary (rootless docker hides these with a default configuration).

services:
  app:
    image: 'lepresidente/nginxproxymanager:latest'
    restart: unless-stopped
    ports:
      # These ports are in format <host-port>:<container-port>
      - '8080:80' # Public HTTP Port
      - '4443:443' # Public HTTPS Port
      - '8181:81' # Admin Web Port
    environment:
      PUID: 1000
      GUID: 1000

      CROWDSEC_OPENRESTY_BOUNCER: |
        ENABLED=true
        API_URL=http://crowdsec:8080
        API_KEY=${CROWDSEC_BOUNCER_APIKEY}
        APPSEC_URL=http://crowdsec:7422

    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"
    security_opt:
      - no-new-privileges=true

  crowdsec:
    image: 'docker.io/crowdsecurity/crowdsec:latest'
    restart: unless-stopped
    container_name: crowdsec
    environment:
      - COLLECTIONS=crowdsecurity/nginx-proxy-manager
    volumes:
      - ./crowdsec/data/:/var/lib/crowdsec/data/
      - ./crowdsec/config/:/etc/crowdsec/
      - ./data/logs/:/var/log/npm:ro
    security_opt:
      - no-new-privileges=true

  watchtower:
    image: containrrr/watchtower
    restart: unless-stopped
    volumes:
      - "/run/user/1000/docker.sock:/var/run/docker.sock"
    command: --cleanup

networks:
  default:
    name: npm_proxy
    external: true

This is my acquis.yaml:

filenames:
  - /var/log/npm/*.log
labels:
  type: nginx-proxy-manager
---
appsec_config: crowdsecurity/appsec-default
listen_addr: 0.0.0.0:7422
name: appsec
source: appsec
labels:
  type: appsec

LaurenceJJones · 2024-12-27T12:48:28Z

Following the instructions in the crowdsec docs (https://docs.crowdsec.net/docs/next/appsec/quickstart/nginxopenresty), I have enabled appsec. I'm able to manually test that appsec is working by launching a shell within my NPM container and issuing the curl command shown in the docs to the crowdsec container. It responds with a 403 forbidden as it should. All good here.

There two types of forbidden that can be responded, did you get a JSON response body?

inthreedee · 2024-12-27T14:12:55Z

@LaurenceJJones Thanks for helping out!

Here are the curl requests I'm running from a shell within my NPM container.
The output from the legitimate test request:
curl -X POST http://crowdsec:7422/ -i -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password'

HTTP/1.1 200 OK
Date: Fri, 27 Dec 2024 14:06:25 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

{"action":"allow","http_status":200}

And the malevolent test request:
curl -X POST http://crowdsec:7422/ -i -H 'x-crowdsec-appsec-uri: /.env' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password'

HTTP/1.1 403 Forbidden
Date: Fri, 27 Dec 2024 14:08:03 GMT
Content-Length: 34
Content-Type: text/plain; charset=utf-8

{"action":"ban","http_status":403}

LaurenceJJones · 2024-12-27T14:18:39Z

@LaurenceJJones Thanks for helping out!

Here are the curl requests I'm running from a shell within my NPM container. The output from the legitimate test request: curl -X POST http://crowdsec:7422/ -i -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password'
HTTP/1.1 200 OK
Date: Fri, 27 Dec 2024 14:06:25 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

{"action":"allow","http_status":200}
And the malevolent test request: curl -X POST http://crowdsec:7422/ -i -H 'x-crowdsec-appsec-uri: /.env' -H 'x-crowdsec-appsec-ip: 42.42.42.42' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password'
HTTP/1.1 403 Forbidden
Date: Fri, 27 Dec 2024 14:08:03 GMT
Content-Length: 34
Content-Type: text/plain; charset=utf-8

{"action":"ban","http_status":403}

perfect! I wanted to make sure before we dive into debugging it wasnt something as easy as an invalid api key 😅

Within the error log of npm do you see any crowdsec logs? you might have to route around a little as I know npm likes to split the error logs per host.

inthreedee · 2024-12-27T15:20:51Z

I do see crowdsec actions being taken in some of the npm error logs. Here's everything relevant I can find in all my logs, including logs from the time of my test requests and also the crowdsec actions that I do see so far:

default-host_access.log is the log file for requests directly to my IP since NPM does not have a host connected to it. We can see NPM catching my test request and returning its 404. There's nothing in the associated default-host_error.log.

{ip-addr} - - [27/Dec/2024:09:15:11 -0500] "GET /.env HTTP/1.1" 404 122 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"
{ip-addr} - - [27/Dec/2024:09:15:12 -0500] "GET /favicon.ico HTTP/1.1" 404 122 "http://{ip-addr}/.env" "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"

fallback_error.log has this from crowdsec, but nothing else from the time of the test request:

2024/12/27 04:23:52 [error] 1894#1894: [lua] crowdsec.lua:57: init(): error loading captcha plugin: no recaptcha site key provided, can't use recaptcha
2024/12/27 04:23:52 [alert] 1894#1894: [lua] crowdsec_openresty.conf:5):11: [Crowdsec] Initialisation done

proxy-host-1_access.log is the log for one of the subdomains managed by NPM. We can see the webapp here receiving another test request sent to it and responding with its 404 page.

[27/Dec/2024:09:28:45 -0500] - 404 404 - GET https domain.example.com "/.env" [Client {ip-addr}] [Length 172] [Gzip 1.14] [Sent-to 10.0.0.74] "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0" "-"

There's nothing in the associated proxy-host-1_error.log from the time of that test request but, curiously, I do see a crowdsec ban line from two days ago for an IP that does not show among the alerts in my crowdsec console. Not sure what that's about.

2024/12/25 07:50:44 [alert] 253#253: *12738 [lua] crowdsec.lua:679: Allow(): [Crowdsec] denied '{banned-IP}' with 'ban' (by bouncer), client: {banned-IP}, server: {domain.example.com}, request: "GET / HTTP/1.1", host: "{domain.example.com}", referrer: "http://{domain.example.com}"

Looking through the error logs of other hosts in NPM, I see one has a crowdsec ban for that same IP from a different day which is also not in my console alerts. I don't see any other mentions of crowdsec or ban lines in any NPM error logs for the alerts that do exist in my web console. Not sure if this indicates a configuration problem or if this is normal. 🤷

Here's relevant looking info from the crowdsec container log from both the container starting up and the times of the test requests:

crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Cache duration for auth not set, using default: 1m0s" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading /etc/crowdsec/appsec-configs/appsec-default.yaml" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Loaded 0 outofband rules" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/base-config" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/vpatch-*" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/generic-*" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Loaded 76 inband rules" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Created 1 appsec runners" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="127.0.0.1 - [Thu, 26 Dec 2024 20:19:22 UTC] \"POST /v1/watchers/login HTTP/1.1 200 118.920073ms \"crowdsec/v1.6.4-523164f6-docker\" \""
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Starting processing data"
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="1 appsec runner to start" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="creating TCP server on 0.0.0.0:7422" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Appsec Runner ready to process event" name=appsec runner_uuid={uuid} type=appsec
...
crowdsec      | time="2024-12-27T14:14:49Z" level=info msg="127.0.0.1 - [Fri, 27 Dec 2024 14:14:49 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.916693ms \"crowdsec/v1.6.4-523164f6-docker\" \""
crowdsec      | time="2024-12-27T14:15:14Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:15:14 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 73.269077ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:15:43Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:15:43 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 87.695788ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
...
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 78.772425ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 78.752591ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 79.551812ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:44Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:44 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 73.285854ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:46Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:46 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 74.550761ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:49Z" level=info msg="127.0.0.1 - [Fri, 27 Dec 2024 14:28:49 UTC] \"GET /v1/heartbeat HTTP/1.1 200 24.467334ms \"crowdsec/v1.6.4-523164f6-docker\" \""

LaurenceJJones · 2024-12-27T16:28:59Z

I do see crowdsec actions being taken in some of the npm error logs. Here's everything relevant I can find in all my logs, including logs from the time of my test requests and also the crowdsec actions that I do see so far:

default-host_access.log is the log file for requests directly to my IP since NPM does not have a host connected to it. We can see NPM catching my test request and returning its 404. There's nothing in the associated default-host_error.log.

{ip-addr} - - [27/Dec/2024:09:15:11 -0500] "GET /.env HTTP/1.1" 404 122 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"
{ip-addr} - - [27/Dec/2024:09:15:12 -0500] "GET /favicon.ico HTTP/1.1" 404 122 "http://{ip-addr}/.env" "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"

fallback_error.log has this from crowdsec, but nothing else from the time of the test request:

2024/12/27 04:23:52 [error] 1894#1894: [lua] crowdsec.lua:57: init(): error loading captcha plugin: no recaptcha site key provided, can't use recaptcha
2024/12/27 04:23:52 [alert] 1894#1894: [lua] crowdsec_openresty.conf:5):11: [Crowdsec] Initialisation done

proxy-host-1_access.log is the log for one of the subdomains managed by NPM. We can see the webapp here receiving another test request sent to it and responding with its 404 page.

[27/Dec/2024:09:28:45 -0500] - 404 404 - GET https domain.example.com "/.env" [Client {ip-addr}] [Length 172] [Gzip 1.14] [Sent-to 10.0.0.74] "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0" "-"

There's nothing in the associated proxy-host-1_error.log from the time of that test request but, curiously, I do see a crowdsec ban line from two days ago for an IP that does not show among the alerts in my crowdsec console. Not sure what that's about.

2024/12/25 07:50:44 [alert] 253#253: *12738 [lua] crowdsec.lua:679: Allow(): [Crowdsec] denied '{banned-IP}' with 'ban' (by bouncer), client: {banned-IP}, server: {domain.example.com}, request: "GET / HTTP/1.1", host: "{domain.example.com}", referrer: "http://{domain.example.com}"

Looking through the error logs of other hosts in NPM, I see one has a crowdsec ban for that same IP from a different day which is also not in my console alerts. I don't see any other mentions of crowdsec or ban lines in any NPM error logs for the alerts that do exist in my web console. Not sure if this indicates a configuration problem or if this is normal. 🤷

Here's relevant looking info from the crowdsec container log from both the container starting up and the times of the test requests:

crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Cache duration for auth not set, using default: 1m0s" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading /etc/crowdsec/appsec-configs/appsec-default.yaml" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Loaded 0 outofband rules" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/base-config" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/vpatch-*" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="loading inband rule crowdsecurity/generic-*" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Loaded 76 inband rules" component=appsec_config name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Created 1 appsec runners" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="127.0.0.1 - [Thu, 26 Dec 2024 20:19:22 UTC] \"POST /v1/watchers/login HTTP/1.1 200 118.920073ms \"crowdsec/v1.6.4-523164f6-docker\" \""
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Starting processing data"
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="1 appsec runner to start" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="creating TCP server on 0.0.0.0:7422" name=appsec type=appsec
crowdsec      | time="2024-12-26T20:19:22Z" level=info msg="Appsec Runner ready to process event" name=appsec runner_uuid={uuid} type=appsec
...
crowdsec      | time="2024-12-27T14:14:49Z" level=info msg="127.0.0.1 - [Fri, 27 Dec 2024 14:14:49 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.916693ms \"crowdsec/v1.6.4-523164f6-docker\" \""
crowdsec      | time="2024-12-27T14:15:14Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:15:14 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 73.269077ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:15:43Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:15:43 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 87.695788ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
...
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 78.772425ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 78.752591ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:32Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:32 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 79.551812ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:44Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:44 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 73.285854ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:46Z" level=info msg="172.18.0.6 - [Fri, 27 Dec 2024 14:28:46 UTC] \"GET /v1/decisions?ip={ip-addr} HTTP/1.1 200 74.550761ms \"crowdsec-openresty-bouncer/v1.0.2\" \""
crowdsec      | time="2024-12-27T14:28:49Z" level=info msg="127.0.0.1 - [Fri, 27 Dec 2024 14:28:49 UTC] \"GET /v1/heartbeat HTTP/1.1 200 24.467334ms \"crowdsec/v1.6.4-523164f6-docker\" \""

Just to ask when you added the config to the compose did you recreate the npm container as in stop and down because it seems it hasn't picked up that appsec is now configured.

It may also be that the environment key is only read once and since the config file might already exist it doesn't repopulate it?

LaurenceJJones · 2024-12-27T16:35:04Z

Also adding within the container if you check /config/crowdsec-openresty-bouncer.conf does everything seem to be configure as per your environment var?

inthreedee · 2024-12-27T16:44:50Z

Well, turns out I do not have that particular conf file. Sounds like that could be a problem? Do I need to manually create that?

According to https://docs.crowdsec.net/u/bouncers/openresty, it should be located at /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf, which would map to my docker volume at config/bouncers/.... There's currently no bouncers directory either.

For reference, when setting up crowdsec initially, I followed the steps in this repo's readme here and assumed this was all that was needed if the env vars were being set in the docker compose file.

LaurenceJJones · 2024-12-27T16:56:01Z

Well, turns out I do not have that particular conf file. Sounds like that could be a problem? Do I need to manually create that?

According to https://docs.crowdsec.net/u/bouncers/openresty, it should be located at /etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf, which would map to my docker volume at config/bouncers/.... There's currently no bouncers directory either.

For reference, when setting up crowdsec initially, I followed the steps in this repo's readme here and assumed this was all that was needed if the env vars were being set in the docker compose file.

Yes but npm stores file in different locations, so if you do a find it should be there somewhere. However it won't be under /etc/crowdsec as like said NPM puts files in different locations.

inthreedee · 2024-12-27T16:56:13Z

Just to ask when you added the config to the compose did you recreate the npm container as in stop and down because it seems it hasn't picked up that appsec is now configured.

It may also be that the environment key is only read once and since the config file might already exist it doesn't repopulate it?

Sorry, I missed that reply. I'm not sure if I used a full docker compose down when restarting the container. Just gave that a go and it doesn't seem to have changed any behavior.

inthreedee · 2024-12-27T16:58:08Z

Yes but npm stores file in different locations, so if you do a find it should be there somewhere.

I do not see that or any "*bouncer.conf" files anywhere in the container's filesystem.

LaurenceJJones · 2024-12-27T16:59:12Z

Yes but npm stores file in different locations, so if you do a find it should be there somewhere.

I do not see that or any "*bouncer.conf" files anywhere in the container's filesystem.

You are checking the npm container right?

Checked his repo and it states /config/crowdsec/

inthreedee · 2024-12-27T17:42:43Z

Ah, my bad, I was searching the crowdsec container. I found it in the npm container in /defaults/crowdsec/crowdsec-openresty-bouncer.conf and, you're right, it's missing the APPSEC_URL variable!

Fixing that results in some progress, though now it seems the appsec component is throwing 500 errors with this in the logs:

2024/12/27 12:39:47 [error] 256#256: *84 lua entry thread aborted: runtime error: /etc/nginx/lualib/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /etc/nginx/lualib/crowdsec.lua:410: in function 'get_body'
        /etc/nginx/lualib/crowdsec.lua:523: in function 'AppSecCheck'
        /etc/nginx/lualib/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec_openresty.conf:24):6: in main chunk, client: {ip-addr}, server: example.domain.com, request: "GET /.env HTTP/2.0", host: "example.domain.com"

LaurenceJJones · 2024-12-27T17:48:29Z

Ah, my bad, I was searching the crowdsec container. I found it in the npm container in /defaults/crowdsec/crowdsec-openresty-bouncer.conf and, you're right, it's missing the APPSEC_URL variable!

Fixing that results in some progress, though now it seems the appsec component is throwing 500 errors with this in the logs:
2024/12/27 12:39:47 [error] 256#256: *84 lua entry thread aborted: runtime error: /etc/nginx/lualib/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /etc/nginx/lualib/crowdsec.lua:410: in function 'get_body'
        /etc/nginx/lualib/crowdsec.lua:523: in function 'AppSecCheck'
        /etc/nginx/lualib/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec_openresty.conf:24):6: in main chunk, client: {ip-addr}, server: example.domain.com, request: "GET /.env HTTP/2.0", host: "example.domain.com"

Huh? That should only do this on post requests.

Let me check the source over the weekend and see if we have a logic issue.

inthreedee · 2024-12-27T17:54:16Z

Alright, thanks again for helping out; I really appreciate it. Searching the repo I found a couple other issues related to that error, linking them here for reference:
crowdsecurity/lua-cs-bouncer#83
crowdsecurity/lua-cs-bouncer#44

The first one is reported by LePresidente and seems like my same error.
That second one seems to be related to the captcha remediation component, which I've not configured yet.

LaurenceJJones · 2024-12-27T17:58:25Z

Ah, my bad, I was searching the crowdsec container. I found it in the npm container in /defaults/crowdsec/crowdsec-openresty-bouncer.conf and, you're right, it's missing the APPSEC_URL variable!

Fixing that results in some progress, though now it seems the appsec component is throwing 500 errors with this in the logs:
2024/12/27 12:39:47 [error] 256#256: *84 lua entry thread aborted: runtime error: /etc/nginx/lualib/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /etc/nginx/lualib/crowdsec.lua:410: in function 'get_body'
        /etc/nginx/lualib/crowdsec.lua:523: in function 'AppSecCheck'
        /etc/nginx/lualib/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec_openresty.conf:24):6: in main chunk, client: {ip-addr}, server: example.domain.com, request: "GET /.env HTTP/2.0", host: "example.domain.com"

Yeah so we added a check

https://github.com/crowdsecurity/lua-cs-bouncer/blob/main/lib%2Fcrowdsec.lua#L427

However, this error is happening from the Lua library itself, however, this was removed from the latest version so I don't know if LePresidente needs to bump the version or even NPM.

Don't know if it's useful but I seen a lot more people have success with https://github.com/ZoeyVid/NPMplus

inthreedee · 2024-12-27T18:05:53Z

This is happening for all requests, even legitimate ones so I've disabled the appsec component for now.

I pinged LePresidente over in his issue with your question about the version he's using in his NPM fork. We'll see what info he can provide from that.

inthreedee · 2024-12-27T18:09:01Z

As for my original issue of the variable not being updated, perhaps that behavior should be documented in the readme here along with the location of the config file I found within the NPM container?

inthreedee · 2024-12-28T05:46:39Z

@LaurenceJJones FYI while we wait, version info from cscli version:

version: v1.6.4-523164f6
Codename: alphaga
BuildDate: 2024-11-26_14:18:02
GoVersion: 1.23.3
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.4-523164f6-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_wineventlog

inthreedee · 2025-01-01T16:21:53Z

Ah, my bad, I was searching the crowdsec container. I found it in the npm container in /defaults/crowdsec/crowdsec-openresty-bouncer.conf and, you're right, it's missing the APPSEC_URL variable!

Fixing that results in some progress, though now it seems the appsec component is throwing 500 errors with this in the logs:
2024/12/27 12:39:47 [error] 256#256: *84 lua entry thread aborted: runtime error: /etc/nginx/lualib/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /etc/nginx/lualib/crowdsec.lua:410: in function 'get_body'
        /etc/nginx/lualib/crowdsec.lua:523: in function 'AppSecCheck'
        /etc/nginx/lualib/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec_openresty.conf:24):6: in main chunk, client: {ip-addr}, server: example.domain.com, request: "GET /.env HTTP/2.0", host: "example.domain.com"

Update: NPM updated to v2.12.2 and this error has been resolved for me.

I do have one more configuration question, @LaurenceJJones, if you don't mind; you've been extremely helpful so far. With that error resolved, accessing https://domain.example.com/.env pops the expected CrowdSec access forbidden page. 🎉 When I access the public IP Address of the host machine, however, the requests don't seem to be passed to AppSec. http://{ip-address}/.env is sent to NPM which responds with openresty's 404 page.

In NPM's settings, I can control how NPM responds to these requests in the "Default Site" configuration. The options are Congratulations Page, 404 Page, 444, Redirect, or Custom Page. When set to 404 or 444, AppSec doesn't seem to intercept any requests. When set to the Congratulations Page, AppSec does intercept requests. Is it possible to configure things so AppSec intercepts and evaluates all requests regardless of this setting? I was hoping to have AppSec doing some of its WAF duties before the request gets to NPM in case of some future nginx/NPM exploit.

LaurenceJJones · 2025-01-02T10:32:48Z

Ah, my bad, I was searching the crowdsec container. I found it in the npm container in /defaults/crowdsec/crowdsec-openresty-bouncer.conf and, you're right, it's missing the APPSEC_URL variable!
Fixing that results in some progress, though now it seems the appsec component is throwing 500 errors with this in the logs:
2024/12/27 12:39:47 [error] 256#256: *84 lua entry thread aborted: runtime error: /etc/nginx/lualib/crowdsec.lua:410: http2 requests are not supported without content-length header
stack traceback:
coroutine 0:
        [C]: in function 'read_body'
        /etc/nginx/lualib/crowdsec.lua:410: in function 'get_body'
        /etc/nginx/lualib/crowdsec.lua:523: in function 'AppSecCheck'
        /etc/nginx/lualib/crowdsec.lua:613: in function 'Allow'
        access_by_lua(conf.d/crowdsec_openresty.conf:24):6: in main chunk, client: {ip-addr}, server: example.domain.com, request: "GET /.env HTTP/2.0", host: "example.domain.com"
Update: NPM updated to v2.12.2 and this error has been resolved for me.

I do have one more configuration question, @LaurenceJJones, if you don't mind; you've been extremely helpful so far. With that error resolved, accessing https://domain.example.com/.env pops the expected CrowdSec access forbidden page. 🎉 When I access the public IP Address of the host machine, however, the requests don't seem to be passed to AppSec. http://{ip-address}/.env is sent to NPM which responds with openresty's 404 page.

In NPM's settings, I can control how NPM responds to these requests in the "Default Site" configuration. The options are Congratulations Page, 404 Page, 444, Redirect, or Custom Page. When set to 404 or 444, AppSec doesn't seem to intercept any requests. When set to the Congratulations Page, AppSec does intercept requests. Is it possible to configure things so AppSec intercepts and evaluates all requests regardless of this setting? I was hoping to have AppSec doing some of its WAF duties before the request gets to NPM in case of some future nginx/NPM exploit.

Yes this is a known limitation, so basically if in your fallback you have something like:

location / {
  return 404 "Not found";
}

Since nginx tries to optimize the calls on the stack it will not run these against the access_by_lua hook which means these are never evaluated against the LAPI or AppSec since nginx is like well these will return this code so there is nothing to access such as a file or reverse proxy.

I had a look earlier in 2024 and there no way to force nginx to run these hooks unless you provide the custom page option as then it is "accessing" something.

inthreedee · 2025-01-02T14:41:57Z

Yes this is a known limitation, so basically if in your fallback you have something like:
location / {
  return 404 "Not found";
}
Since nginx tries to optimize the calls on the stack it will not run these against the access_by_lua hook which means these are never evaluated against the LAPI or AppSec since nginx is like well these will return this code so there is nothing to access such as a file or reverse proxy.

I had a look earlier in 2024 and there no way to force nginx to run these hooks unless you provide the custom page option as then it is "accessing" something.

That makes sense, thanks for the explanation. From a security perspective, do you guys currently think it's better to leave the fallback set to 404 or provide a custom page so crowdsec can get involved in the requests? It seems to me there's tradeoffs with either option and I'm not quite sure which would be best practice.

LaurenceJJones · 2025-01-02T14:57:38Z

Yes this is a known limitation, so basically if in your fallback you have something like:
location / {
  return 404 "Not found";
}
Since nginx tries to optimize the calls on the stack it will not run these against the access_by_lua hook which means these are never evaluated against the LAPI or AppSec since nginx is like well these will return this code so there is nothing to access such as a file or reverse proxy.
I had a look earlier in 2024 and there no way to force nginx to run these hooks unless you provide the custom page option as then it is "accessing" something.
That makes sense, thanks for the explanation. From a security perspective, do you guys currently think it's better to leave the fallback set to 404 or provide a custom page so crowdsec can get involved in the requests? It seems to me there's tradeoffs with either option and I'm not quite sure which would be best practice.

99.99% of the time its fine, the fallback page is a static html page that does not allow any interaction. Personally I use the fallback option to redirect back to my site as with anything if are going directly to the IP address then do you want to spend resources showing them a custom page 🤷🏻 most likely not as standard users are going to type in a fqdn.

inthreedee · 2025-01-02T15:02:53Z

Thanks for all your help, I appreciate it! Everything seems to be working on my end now.

inthreedee mentioned this issue Dec 27, 2024

Latest Version of openresty shows error regarding content-length header needs to be defined crowdsecurity/lua-cs-bouncer#83

Open

inthreedee closed this as completed Jan 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Appsec installation help #30

Appsec installation help #30

inthreedee commented Dec 26, 2024 •

edited

Loading

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 •

edited

Loading

inthreedee commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 •

edited

Loading

inthreedee commented Dec 27, 2024 •

edited

Loading

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 •

edited

Loading

inthreedee commented Dec 27, 2024

inthreedee commented Dec 27, 2024

inthreedee commented Dec 28, 2024

inthreedee commented Jan 1, 2025

LaurenceJJones commented Jan 2, 2025

inthreedee commented Jan 2, 2025

LaurenceJJones commented Jan 2, 2025

inthreedee commented Jan 2, 2025

Appsec installation help #30

Appsec installation help #30

Comments

inthreedee commented Dec 26, 2024 • edited Loading

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 • edited Loading

inthreedee commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 • edited Loading

inthreedee commented Dec 27, 2024 • edited Loading

LaurenceJJones commented Dec 27, 2024

inthreedee commented Dec 27, 2024

LaurenceJJones commented Dec 27, 2024 • edited Loading

inthreedee commented Dec 27, 2024

inthreedee commented Dec 27, 2024

inthreedee commented Dec 28, 2024

inthreedee commented Jan 1, 2025

LaurenceJJones commented Jan 2, 2025

inthreedee commented Jan 2, 2025

LaurenceJJones commented Jan 2, 2025

inthreedee commented Jan 2, 2025

inthreedee commented Dec 26, 2024 •

edited

Loading

LaurenceJJones commented Dec 27, 2024 •

edited

Loading

LaurenceJJones commented Dec 27, 2024 •

edited

Loading

inthreedee commented Dec 27, 2024 •

edited

Loading

LaurenceJJones commented Dec 27, 2024 •

edited

Loading