I use Terraform to set up ASG for my worker servers running sidekiq. On deploy when AMI image_id changes the instances require long graceful shutdown before terminating to finish job processing (max. 30m).
My idea to accomplish that is to:
Use autoscaling module with an initial lifecycle hook that sets up a transition state to
autoscaling:EC2_INSTANCE_TERMINATING
when removing old ASG and creating new ASG on deploy.Send transition state to SQS queue.
Set up AWS Lambda function that receives state messages from SQS, sends a remote command with SSM to initiate graceful shutdown on the EC2 instance. Graceful shutdown script running on the instance will send SIGTERM to sidekiq, monitor if sidekiq finished and then send
shutdown -h now
command at the end to stop the instance.Lambda exits immediately and does not wait until shutdown script finishes which is expected. It sends SNS notification on success and failure. Now, the problem is that Terraform waits until ASG is successfully removed and it hits timeout which by default is 10m. I would like to force Terraform to continue after ASG is only selected to be removed without actually waiting for it to finish removal.
I have force_delete=true
and wait_for_capacity_timeout=0
options set up.
From docs:
force_delete - (Optional) Allows deleting the autoscaling group without waiting for all instances in the pool to terminate. You can force an autoscaling group to delete even if it's in the process of scaling a resource. Normally, Terraform drains all the instances before deleting the group. This bypasses that behavior and potentially leaves resources dangling.
Terraform exists after 10 minutes with the following error message:
Error: Error deleting autoscaling group: Auto Scaling Group still exists
Why isn't it working? Do you think it is a bug?