Could not determine the number of attached_volumes after 5 attempts. Last response was:

[golharam]

Launched an AWS Linux 2023 instance and installed this from master branch using

Using 'sh install.sh -d /dev/nvme2n1 -f lvm.ext4', this seems to work on AWS Linux 2023, where /dev/nvme2n1 is a 200GB gp3 disk

per PR #54

I see this in /var/log/ebs-autoscale.log

[Mon Mar  4 00:00:22 UTC 2024] Stopping EBS Autoscale
[Mon Mar  4 00:00:22 UTC 2024] Starting EBS Autoscale
[Mon Mar  4 00:00:22 UTC 2024] EBS Autoscaling mountpoint: /scratch
[Mon Mar  4 00:00:22 UTC 2024] Region = 
[Mon Mar  4 00:00:22 UTC 2024] Availability Zone = 
[Mon Mar  4 00:00:59 UTC 2024] Could not determine the number of attached_volumes after 5 attempts. Last response was: 
[Mon Mar  4 16:47:55 UTC 2024] Devices  : Size 196G : Used 60G : Available 127G : Used% 32% : Threshold 50%
[Mon Mar  4 16:58:00 UTC 2024] Devices  : Size 196G : Used 81G : Available 106G : Used% 44% : Threshold 50%
[Mon Mar  4 17:05:50 UTC 2024] Could not determine the number of attached_volumes after 5 attempts. Last response was: 
[Mon Mar  4 17:05:50 UTC 2024] LOW DISK (50%): Adding more.
[Mon Mar  4 17:05:50 UTC 2024] Extending logical volume /scratch by 150GB
[Mon Mar  4 17:05:52 UTC 2024] Error creating or attaching EBS volume
[Mon Mar  4 17:05:52 UTC 2024] Devices 1 : Size 196G : Used 92G : Available 95G : Used% 50% : Threshold 50%
[Mon Mar  4 17:06:32 UTC 2024] Could not determine the number of attached_volumes after 5 attempts. Last response was: 

I only noticed this when my /scratch mount filled up.

It looks like ebs-autoscale called initialize() In /usr/local/amazon-ebs-autoscale/shared/utils.sh, which in turn calls get_metadata("placement/availability-zone").

The line in get_metadata,

curl -s $token_wrapper http://$metadata_ip/latest/meta-data/placement/availability-zone

with a valid token_wrapper,

-H "X-aws-ec2-metadata-token: some_token_string"

returns nothing.

Interestingly enough, even doing

curl http://169.254.169.254/

returns nothing.

[golharam]

Looks like this is a change in AWS Linux 2023. A token is required first, which is what get_metadata() does. However the curl command still returns nothing,

curl -s $token_wrapper http://$metadata_ip/latest/meta-data/$key

If I run this in a shell

TOKEN=$(curl --request PUT "http://169.254.169.254/latest/api/token" --header "X-aws-ec2-metadata-token-ttl-seconds: 3600")
INSTANCE_METADATA=$(curl -s http://169.254.169.254/latest/meta-data/ --header "X-aws-ec2-metadata-token: $TOKEN")
echo $INSTANCE_METADATA
ami-id ami-launch-index ami-manifest-path block-device-mapping/ events/ hostname iam/ identity-credentials/ instance-action instance-id instance-life-cycle instance-type local-hostname local-ipv4 mac metrics/ network/ placement/ profile public-keys/ reservation-id security-groups services/ system

I see this does work.

[golharam]

It looks like the offending line is

if [ ! -z "$IMDSV2" ]; then

This evaluates to True hence the token is never fetched.

The instance is set to "IMDSv2 Required" from the console.

[golharam]

Looks like PR #55 added this support. -i needs to be specified when running this to using IMDSv2.

This setting is not propogated to ebs-autoscale.json nor is IMDSV2 referenced anywhere else.

[golharam]

This same issue will happen if ebs-autoscale is a system service and the instance is rebooted.

[golharam]

Referencing #63 as related

[golharam]

As a follow up, there is a secondary problem. In create-ebs-volume, create_and_attach_volume(), the call to get_metadata instance-id returns empty. I think it's failing for the same reason. There needs to be better error handling here.

[golharam]

PR #72 I submitted fixes this.