Why every time Elastic Beanstalk issues a command to its instance it always timed out?
up vote
34
down vote
favorite
I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:
Deploying new version to instance(s)
it's always followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]
The same thing happens when I try to request snapshot logs. The Beanstalk issues:
requestEnvironmentInfo is starting
then after a few minutes it's again followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].
amazon-web-services elastic-beanstalk
|
show 3 more comments
up vote
34
down vote
favorite
I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:
Deploying new version to instance(s)
it's always followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]
The same thing happens when I try to request snapshot logs. The Beanstalk issues:
requestEnvironmentInfo is starting
then after a few minutes it's again followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].
amazon-web-services elastic-beanstalk
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
1
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
1
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
1
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond togit aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
– Chris Carson
Jun 28 '14 at 0:55
|
show 3 more comments
up vote
34
down vote
favorite
up vote
34
down vote
favorite
I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:
Deploying new version to instance(s)
it's always followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]
The same thing happens when I try to request snapshot logs. The Beanstalk issues:
requestEnvironmentInfo is starting
then after a few minutes it's again followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].
amazon-web-services elastic-beanstalk
I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:
Deploying new version to instance(s)
it's always followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]
The same thing happens when I try to request snapshot logs. The Beanstalk issues:
requestEnvironmentInfo is starting
then after a few minutes it's again followed by:
The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].
amazon-web-services elastic-beanstalk
amazon-web-services elastic-beanstalk
edited Sep 3 '17 at 12:55
Cœur
17k9102140
17k9102140
asked Apr 3 '14 at 8:51
ardfard
173125
173125
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
1
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
1
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
1
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond togit aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
– Chris Carson
Jun 28 '14 at 0:55
|
show 3 more comments
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
1
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
1
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
1
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond togit aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
– Chris Carson
Jun 28 '14 at 0:55
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
1
1
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
1
1
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
1
1
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to
git aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.– Chris Carson
Jun 28 '14 at 0:55
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to
git aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.– Chris Carson
Jun 28 '14 at 0:55
|
show 3 more comments
6 Answers
6
active
oldest
votes
up vote
36
down vote
accepted
I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.
If this is a production environment and you have only 1 instance and you want minimal down time
- configure minimum instances to 2, and Beanstalk will launch another instance for you.
- terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
- configure minimum instance back to 1, Beanstalk will remove one of your two instances.
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).
– Logan Pickup
Mar 7 '16 at 2:25
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
|
show 2 more comments
up vote
5
down vote
By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).
{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}
Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
add a comment |
up vote
1
down vote
Had the same issue here (single t1.micro instance).
Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).
add a comment |
up vote
0
down vote
In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.
If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).
Rebuilding the environment seems to reset things with the new version being used.
Obviously there's the downside with that of a period of downtime.
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
add a comment |
up vote
0
down vote
Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...
If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.
An easier way to fix SQS Client is to restart cfn-hup service:
sudo service cfn-hup restart
add a comment |
up vote
0
down vote
I think is the correct way to deal with this.
I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.
chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.
However, if you do need to increase timeout, see the following:
Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.
Example:
option_settings:
"aws:elasticbeanstalk:command":
Timeout: 2400
*"value" represents the length of time before timeout in seconds.
Reference: https://serverfault.com/a/747800/496353
add a comment |
6 Answers
6
active
oldest
votes
6 Answers
6
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
36
down vote
accepted
I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.
If this is a production environment and you have only 1 instance and you want minimal down time
- configure minimum instances to 2, and Beanstalk will launch another instance for you.
- terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
- configure minimum instance back to 1, Beanstalk will remove one of your two instances.
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).
– Logan Pickup
Mar 7 '16 at 2:25
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
|
show 2 more comments
up vote
36
down vote
accepted
I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.
If this is a production environment and you have only 1 instance and you want minimal down time
- configure minimum instances to 2, and Beanstalk will launch another instance for you.
- terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
- configure minimum instance back to 1, Beanstalk will remove one of your two instances.
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).
– Logan Pickup
Mar 7 '16 at 2:25
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
|
show 2 more comments
up vote
36
down vote
accepted
up vote
36
down vote
accepted
I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.
If this is a production environment and you have only 1 instance and you want minimal down time
- configure minimum instances to 2, and Beanstalk will launch another instance for you.
- terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
- configure minimum instance back to 1, Beanstalk will remove one of your two instances.
I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.
If this is a production environment and you have only 1 instance and you want minimal down time
- configure minimum instances to 2, and Beanstalk will launch another instance for you.
- terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
- configure minimum instance back to 1, Beanstalk will remove one of your two instances.
answered Aug 7 '14 at 7:35
chongzixin
97531642
97531642
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).
– Logan Pickup
Mar 7 '16 at 2:25
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
|
show 2 more comments
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).
– Logan Pickup
Mar 7 '16 at 2:25
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
it was useful for me thanks :)
– Rajaraman
Dec 2 '14 at 13:42
5
5
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
– Marcus Lind
Feb 4 '15 at 8:22
2
2
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
A simple reboot of the instance worked for me once :)
– Babu
May 5 '15 at 6:26
1
1
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:
Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).– Logan Pickup
Mar 7 '16 at 2:25
The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to:
Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping
. The only solution I have found to work is the answer above: reboot (or rebuild).– Logan Pickup
Mar 7 '16 at 2:25
1
1
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
– vivekyad4v
Nov 16 '17 at 11:43
|
show 2 more comments
up vote
5
down vote
By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).
{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}
Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
add a comment |
up vote
5
down vote
By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).
{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}
Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
add a comment |
up vote
5
down vote
up vote
5
down vote
By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).
{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}
Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).
{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}
Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
answered Apr 3 '14 at 9:15
daveoncode
10.6k671120
10.6k671120
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
add a comment |
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
1
1
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
– ardfard
Apr 4 '14 at 3:33
5
5
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
Where do you put this?
– Spoeken
Jun 2 '14 at 9:09
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
@Spoeken - in your myapp.config file within your .ebextensions directory
– Meistro
Aug 8 '14 at 16:18
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
3600 is the new maximum limit.
– ConorB
Jun 13 '16 at 14:49
add a comment |
up vote
1
down vote
Had the same issue here (single t1.micro instance).
Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).
add a comment |
up vote
1
down vote
Had the same issue here (single t1.micro instance).
Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).
add a comment |
up vote
1
down vote
up vote
1
down vote
Had the same issue here (single t1.micro instance).
Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).
Had the same issue here (single t1.micro instance).
Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).
answered Apr 26 at 8:55
mirzik
111
111
add a comment |
add a comment |
up vote
0
down vote
In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.
If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).
Rebuilding the environment seems to reset things with the new version being used.
Obviously there's the downside with that of a period of downtime.
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
add a comment |
up vote
0
down vote
In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.
If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).
Rebuilding the environment seems to reset things with the new version being used.
Obviously there's the downside with that of a period of downtime.
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
add a comment |
up vote
0
down vote
up vote
0
down vote
In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.
If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).
Rebuilding the environment seems to reset things with the new version being used.
Obviously there's the downside with that of a period of downtime.
In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.
If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).
Rebuilding the environment seems to reset things with the new version being used.
Obviously there's the downside with that of a period of downtime.
answered Apr 30 '15 at 14:18
Dan Gravell
4,41442744
4,41442744
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
add a comment |
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
2
2
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
– theBarkman
Feb 18 '16 at 18:51
add a comment |
up vote
0
down vote
Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...
If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.
An easier way to fix SQS Client is to restart cfn-hup service:
sudo service cfn-hup restart
add a comment |
up vote
0
down vote
Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...
If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.
An easier way to fix SQS Client is to restart cfn-hup service:
sudo service cfn-hup restart
add a comment |
up vote
0
down vote
up vote
0
down vote
Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...
If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.
An easier way to fix SQS Client is to restart cfn-hup service:
sudo service cfn-hup restart
Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...
If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.
An easier way to fix SQS Client is to restart cfn-hup service:
sudo service cfn-hup restart
answered May 30 at 17:59
Victor Gavrilenko
11
11
add a comment |
add a comment |
up vote
0
down vote
I think is the correct way to deal with this.
I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.
chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.
However, if you do need to increase timeout, see the following:
Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.
Example:
option_settings:
"aws:elasticbeanstalk:command":
Timeout: 2400
*"value" represents the length of time before timeout in seconds.
Reference: https://serverfault.com/a/747800/496353
add a comment |
up vote
0
down vote
I think is the correct way to deal with this.
I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.
chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.
However, if you do need to increase timeout, see the following:
Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.
Example:
option_settings:
"aws:elasticbeanstalk:command":
Timeout: 2400
*"value" represents the length of time before timeout in seconds.
Reference: https://serverfault.com/a/747800/496353
add a comment |
up vote
0
down vote
up vote
0
down vote
I think is the correct way to deal with this.
I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.
chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.
However, if you do need to increase timeout, see the following:
Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.
Example:
option_settings:
"aws:elasticbeanstalk:command":
Timeout: 2400
*"value" represents the length of time before timeout in seconds.
Reference: https://serverfault.com/a/747800/496353
I think is the correct way to deal with this.
I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.
chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.
However, if you do need to increase timeout, see the following:
Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.
Example:
option_settings:
"aws:elasticbeanstalk:command":
Timeout: 2400
*"value" represents the length of time before timeout in seconds.
Reference: https://serverfault.com/a/747800/496353
edited Nov 15 at 11:59
answered Nov 14 at 20:29
Pranay Majmundar
1655
1655
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f22832447%2fwhy-every-time-elastic-beanstalk-issues-a-command-to-its-instance-it-always-time%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31
I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25
1
@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16
1
@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04
1
I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to
git aws.push
if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.– Chris Carson
Jun 28 '14 at 0:55