Auto-Scale Ghost Instances (AWS EC2)
Over the past week I have been testing Amazon’s AWS EC2 Auto-scaling with a few custom AMIs. Early today as I was cleaning up the auto-scaling configuration I encountered a bit of a conundrum – the auto-scaling API was returning a few instances that did not exist and was stopping me from removing an auto-scaling-group. After hitting Google I wasn’t any wiser, but it seemed that I did not dismantle my test setup properly, which was most likely causing the problem.
Here’s the problem I stumbled upon:
#~: as-delete-auto-scaling-group groupname Are you sure you want to delete this AutoScalingGroup? [Ny]y as-delete-auto-scaling-group: Service error: You cannot delete an AutoScalingGroup while there are instances or pending Spot instance request(s) still in the group.
There shouldn’t be any instances in the group, yet running the following command:
Lists the following instances:
INSTANCE i-xxxxxxxx webhive eu-west-1b InService HEALTHY groupname INSTANCE i-xxxxxxxx webhive eu-west-1b InService HEALTHY groupname
According to the EC2 Management Console and API calls, these instances did not exist anywhere!
Remember earlier I mentioned that I did not dismantle the test setup properly? What did I do? I simply suspended the auto-scale policy and manually terminated the instances that had been invoked during the testing.
What should I have done?
- Updated the auto-scaling-group and set the min-size and max-size to 0 (this would have automatically terminated the instances)
- Suspended the polices
- Cleaned up my testing configuration
Here are the commands for the above steps:
#~: as-update-auto-scaling-group groupname --min-size 0 --max-size 0 OK-Updated AutoScalingGroup #~: as-resume-processes groupname OK-Processes Resumed #~: as-describe-auto-scaling-instances Testing No instances found
I know, I should have read the documentation thoroughly! However this is what testing is for, and I hope that this little blog entry helps someone out there like me.
P.S. AWS EC2 Auto-Scaling is fantastic!comments powered by Disqus