Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Security group not created #322

Open
pierreozoux opened this issue Mar 12, 2024 · 1 comment
Open

[Bug]: Security group not created #322

pierreozoux opened this issue Mar 12, 2024 · 1 comment
Assignees
@outscale-hmi
Labels
bug Something isn't working
Milestone
next Release

Comments

@pierreozoux
Copy link
Contributor

pierreozoux commented Mar 12, 2024 •
edited
Loading

What happened

On a newly created cluster, I can't get logs from a container running on a worker.

Step to reproduce

  • deploy a new cluster with workers
  • deploy cilium CNI
  • deploy a container that creates logs
  • try to get logs from kubectl logs
  • check kube-apiserver and see errors.

Expected to happen

I should see logs.

Add anything

In the cluster-api output logs, and in the oscclusters.infrastructure.cluster.x-k8s.io object, I can see the following securitygroups:

    "description": "Security Group Kw with cluster-api",
    "name": "hedy-lamarr-securitygroup-kw",
    "resourceId": "sg-b1a76fdc",
    "securityGroupRules": [
      {
        "flow": "Inbound",
        "fromPortRange": 179,
        "ipProtocol": "tcp",
        "ipRange": "10.0.0.0/16",
        "name": "simone-veil-securitygrouprule-kw-bgp",
        "toPortRange": 179
      },
      {
        "flow": "Inbound",
        "fromPortRange": 10250,
        "ipProtocol": "tcp",
        "ipRange": "10.0.3.0/24",
        "name": "simone-veil-securitygrouprule-api-kubelet-kw",
        "toPortRange": 10250
      },
      {
        "flow": "Inbound",
        "fromPortRange": 30000,
        "ipProtocol": "tcp",
        "ipRange": "10.0.4.0/24",
        "name": "simone-veil-securitygrouprule-kw-nodeip-kcp",
        "toPortRange": 32767
      },
      {
        "flow": "Inbound",
        "fromPortRange": 10250,
        "ipProtocol": "tcp",
        "ipRange": "10.0.4.0/24",
        "name": "simone-veil-securitygrouprule-api-kubelet-kcp",
        "toPortRange": 10250
      },
      {
        "flow": "Inbound",
        "fromPortRange": 30000,
        "ipProtocol": "tcp",
        "ipRange": "10.0.3.0/24",
        "name": "simone-veil-securitygrouprule-kw-nodeip-kw",
        "toPortRange": 32767
      }
    ]
  },

The issue is that in outscale UI, I can see that these 2 rules are missing:

      {
        "flow": "Inbound",
        "fromPortRange": 10250,
        "ipProtocol": "tcp",
        "ipRange": "10.0.4.0/24",
        "name": "simone-veil-securitygrouprule-api-kubelet-kcp",
        "toPortRange": 10250
      },
      {
        "flow": "Inbound",
        "fromPortRange": 30000,
        "ipProtocol": "tcp",
        "ipRange": "10.0.3.0/24",
        "name": "simone-veil-securitygrouprule-kw-nodeip-kw",
        "toPortRange": 32767
      }

If I add manually this rule:

      {
        "flow": "Inbound",
        "fromPortRange": 10250,
        "ipProtocol": "tcp",
        "ipRange": "10.0.4.0/24",
        "name": "simone-veil-securitygrouprule-api-kubelet-kcp",
        "toPortRange": 10250
      },

Then, I can get logs. Basicaclly, the original issue is that kube-apiserver can't connect to the kubelet of workers.

cluster-api output

{"Tags":[{"Value":"simone-veil-publicip-nat-x","ResourceType":"public-ip","ResourceId":"eipalloc-x","Key":"Name"}],"ResponseContext":{"RequestId":"x"}}
{"level":"Level(-4)","ts":"2024-03-12T09:31:26Z","msg":"Get list of all desired securitygroup in net","controller":"osccluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OscCluster","OscCluster":{"name":"simone-veil","namespace":"simone-veil"},"namespace":"simone-veil","name":"simone-veil","reconcileID":"x","netId":"vpc-x"}
2024/03/12 09:31:26 
POST /api/v1/ReadSecurityGroups HTTP/1.1
Host: api.cloudgouv-eu-west-1.outscale.com
User-Agent: cluster-api-provider-outscale/v0.3.1
Content-Length: 40
Accept: application/json
Authorization: AWS4-HMAC-SHA256 Credential=x/x/eu-west-2/oapi/aws4_request, SignedHeaders=accept;content-type;host;x-amz-date, Signature=x
Content-Type: application/json
X-Amz-Date: x
Accept-Encoding: gzip

{"Filters":{"NetIds":["vpc-x"]}}

{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"received request","webhook":"/mutate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","UID":"x","kind":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=OscMachine","resource":{"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"oscmachines"}}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"default","name":"simone-veil-control-plane-x"}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"wrote response","webhook":"/mutate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","code":200,"reason":"","UID":"x","allowed":true}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"received request","webhook":"/validate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","UID":"x","kind":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=OscMachine","resource":{"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"oscmachines"}}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update","name":"simone-veil-control-plane-x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old vmType","old vmType":"tinav5.c2r4p3"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update vmType","vmType":"tinav5.c2r4p3"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old keypairName","old keypairName":"x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update keyPairName","keypairName":"x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old loadBalancerName","old loadBalancerName":"simone-veil-k8s"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update loadBalancerName","loadBalancerName":"simone-veil-k8s"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old subregionName","old subregionName":"cloudgouv-eu-west-1a"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update subregionName","subregionName":"cloudgouv-eu-west-1a"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old subnetName","old subnetName":"simone-veil-subnet-kcp"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update subnetName","subnetName":"simone-veil-subnet-kcp"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old rootDiskSize","old rootDiskSize":50}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskSize","rootDiskSize":50}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update of old rootDiskIops","old rootDiskIops":500}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskIops","old rootDiskIops":500}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update of old rootDiskTyp","old rootDisktype":"io1"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskType","old rootDiskType":"io1"}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"wrote response","webhook":"/validate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","code":200,"reason":"","UID":"x","allowed":true}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"received request","webhook":"/mutate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","UID":"x","kind":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=OscMachine","resource":{"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"oscmachines"}}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"default","name":"simone-veil-control-plane-x"}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"wrote response","webhook":"/mutate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","code":200,"reason":"","UID":"x","allowed":true}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"received request","webhook":"/validate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","UID":"x","kind":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=OscMachine","resource":{"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"oscmachines"}}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update","name":"simone-veil-control-plane-x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old vmType","old vmType":"tinav5.c2r4p3"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update vmType","vmType":"tinav5.c2r4p3"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old keypairName","old keypairName":"x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update keyPairName","keypairName":"x"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old loadBalancerName","old loadBalancerName":"simone-veil-k8s"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update loadBalancerName","loadBalancerName":"simone-veil-k8s"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old subregionName","old subregionName":"cloudgouv-eu-west-1a"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update subregionName","subregionName":"cloudgouv-eu-west-1a"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old subnetName","old subnetName":"simone-veil-subnet-kcp"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update subnetName","subnetName":"simone-veil-subnet-kcp"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update old rootDiskSize","old rootDiskSize":50}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskSize","rootDiskSize":50}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update of old rootDiskIops","old rootDiskIops":500}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskIops","old rootDiskIops":500}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update of old rootDiskTyp","old rootDisktype":"io1"}
{"level":"info","ts":"2024-03-12T09:31:26Z","logger":"oscmachine-resource","msg":"validate update rootDiskType","old rootDiskType":"io1"}
{"level":"debug","ts":"2024-03-12T09:31:26Z","logger":"controller-runtime.webhook.webhooks","msg":"wrote response","webhook":"/validate-infrastructure-cluster-x-k8s-io-v1beta1-oscmachine","code":200,"reason":"","UID":"x","allowed":true}
2024/03/12 09:31:27 
HTTP/1.1 200 OK
Content-Length: 4063
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Tue, 12 Mar 2024 09:31:27 GMT
Referrer-Policy: same-origin
Server: api-gw/0
Strict-Transport-Security: max-age=31536000; includeSubdomains;
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

{"SecurityGroups":[{"Tags":[],"SecurityGroupName":"default","OutboundRules":[{"FromPortRange":-1,"IpProtocol":"-1","ToPortRange":-1,"IpRanges":["0.0.0.0/0"]}],"SecurityGroupId":"sg-x","AccountId":"x","Description":"default security group","InboundRules":[{"FromPortRange":-1,"IpProtocol":"-1","ToPortRange":-1,"SecurityGroupsMembers":[{"SecurityGroupName":"default","SecurityGroupId":"sg-x","AccountId":"x"}]}],"NetId":"vpc-x"},{"Tags":[{"Value":"owned","Key":"OscK8sClusterID/simone-veil-x"}],"SecurityGroupName":"simone-veil-securitygroup-kcp-x","OutboundRules":[{"FromPortRange":-1,"IpProtocol":"-1","ToPortRange":-1,"IpRanges":["0.0.0.0/0"]}],"SecurityGroupId":"sg-x","AccountId":"x","Description":"Security Group Kcp with cluster-api","InboundRules":[{"FromPortRange":179,"IpProtocol":"tcp","ToPortRange":179,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":6443,"IpProtocol":"tcp","ToPortRange":6443,"SecurityGroupsMembers":[{"SecurityGroupName":"simone-veil-securitygroup-lb-x-","SecurityGroupId":"sg-x","AccountId":"x"}],"IpRanges":["10.0.3.0/24","10.0.4.0/24"]},{"FromPortRange":30000,"IpProtocol":"tcp","ToPortRange":32767,"IpRanges":["10.0.3.0/24"]},{"FromPortRange":10250,"IpProtocol":"tcp","ToPortRange":10252,"IpRanges":["10.0.4.0/24"]},{"FromPortRange":2378,"IpProtocol":"tcp","ToPortRange":2379,"IpRanges":["10.0.4.0/24"]}],"NetId":"vpc-x"},{"Tags":[{"Value":"owned","Key":"OscK8sClusterID/simone-veil-x"}],"SecurityGroupName":"simone-veil-securitygroup-kw-x","OutboundRules":[{"FromPortRange":-1,"IpProtocol":"-1","ToPortRange":-1,"IpRanges":["0.0.0.0/0"]}],"SecurityGroupId":"sg-x","AccountId":"x","Description":"Security Group Kw with cluster-api","InboundRules":[{"FromPortRange":179,"IpProtocol":"tcp","ToPortRange":179,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":10250,"IpProtocol":"tcp","ToPortRange":10250,"IpRanges":["10.0.3.0/24"]},{"FromPortRange":30000,"IpProtocol":"tcp","ToPortRange":32767,"IpRanges":["10.0.4.0/24"]}],"NetId":"vpc-x"},{"Tags":[{"Value":"owned","Key":"OscK8sClusterID/simone-veil-847c0e04-ca2a-43a2-81df-d8c5f91da026"}],"SecurityGroupName":"simone-veil-securitygroup-lb-x","OutboundRules":[{"FromPortRange":6443,"IpProtocol":"tcp","ToPortRange":6443,"SecurityGroupsMembers":[{"SecurityGroupName":"simone-veil-securitygroup-kcp-x","SecurityGroupId":"sg-x","AccountId":"x"}]}],"SecurityGroupId":"sg-x","AccountId":"x","Description":"Security Group Lb with cluster-api","InboundRules":[{"FromPortRange":6443,"IpProtocol":"tcp","ToPortRange":6443,"IpRanges":["0.0.0.0/0"]}],"NetId":"vpc-x"},{"Tags":[{"Value":"owned","Key":"OscK8sClusterID/simone-veil-x"},{"Value":"True","Key":"OscK8sMainSG/simone-veil-x"}],"SecurityGroupName":"simone-veil-securitygroup-node-847c0e04-ca2a-43a2-81df-d8c5f91da026","OutboundRules":[{"FromPortRange":-1,"IpProtocol":"-1","ToPortRange":-1,"IpRanges":["0.0.0.0/0"]}],"SecurityGroupId":"sg-x","AccountId":"x","Description":"Security Group Node with cluster-api","InboundRules":[{"FromPortRange":8472,"IpProtocol":"udp","ToPortRange":8472,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":8285,"IpProtocol":"udp","ToPortRange":8285,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":5473,"IpProtocol":"udp","ToPortRange":5473,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":4789,"IpProtocol":"udp","ToPortRange":4789,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":51821,"IpProtocol":"udp","ToPortRange":51821,"IpRanges":["10.0.0.0/16"]},{"FromPortRange":51820,"IpProtocol":"udp","ToPortRange":51820,"IpRanges":["10.0.0.0/16"]}],"NetId":"vpc-x"}],"ResponseContext":{"RequestId":"x"}}
{"level":"Level(-4)","ts":"2024-03-12T09:31:27Z","msg":"Get securityGroup Id","controller":"osccluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OscCluster","OscCluster":{"name":"simone-veil","namespace":"simone-veil"},"namespace":"simone-veil","name":"simone-veil","reconcileID":"x","securityGroup":["sg-x","sg-x","sg-x","sg-x","sg-x"]}
{"level":"Level(-4)","ts":"2024-03-12T09:31:27Z","msg":"Number of securityGroup","controller":"osccluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OscCluster","OscCluster":{"name":"simone-veil","namespace":"simone-veil"},"namespace":"simone-veil","name":"simone-veil","reconcileID":"x","securityGroupLength":4}
{"level":"Level(-2)","ts":"2024-03-12T09:31:27Z","msg":"Check if the desired securityGroup exist in net","controller":"osccluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"OscCluster","OscCluster":{"name":"simone-veil","namespace":"simone-veil"},"namespace":"simone-veil","name":"simone-veil","reconcileID":"x","securityGroupName":"simone-veil-securitygroup-kw-x"}
2024/03/12 09:31:27

Environment

- Kubernetes version: (use `kubectl version`): 1.27
- OS (e.g. from `/etc/os-release`): ubuntu
- cluster-api-provider-outscale version: v0.3.1
- cluster-api version: v1.6.2
@pierreozoux pierreozoux added the bug Something isn't working label Mar 12, 2024
@pierreozoux
Copy link
Contributor Author

And rules are apparently defined here.

@outscale-hmi outscale-hmi self-assigned this Apr 8, 2024
@outscale-hmi outscale-hmi added this to the next Release milestone Apr 22, 2024
@gvdhart gvdhart mentioned this issue Sep 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@outscale-hmi outscale-hmi

Labels
bug Something isn't working
Milestone
next Release
2 participants
@pierreozoux @outscale-hmi