Why every time Elastic Beanstalk issues a command to its instance it always timed out?











up vote
34
down vote

favorite
15












I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:




Deploying new version to instance(s)




it's always followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]




The same thing happens when I try to request snapshot logs. The Beanstalk issues:




requestEnvironmentInfo is starting




then after a few minutes it's again followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].











share|improve this question
























  • Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
    – Simon Robb
    Apr 4 '14 at 0:31










  • I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
    – Chris Carson
    Apr 24 '14 at 16:25






  • 1




    @Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
    – Simon Robb
    Apr 26 '14 at 2:16






  • 1




    @SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
    – Chris Carson
    Apr 28 '14 at 17:04






  • 1




    I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
    – Chris Carson
    Jun 28 '14 at 0:55















up vote
34
down vote

favorite
15












I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:




Deploying new version to instance(s)




it's always followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]




The same thing happens when I try to request snapshot logs. The Beanstalk issues:




requestEnvironmentInfo is starting




then after a few minutes it's again followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].











share|improve this question
























  • Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
    – Simon Robb
    Apr 4 '14 at 0:31










  • I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
    – Chris Carson
    Apr 24 '14 at 16:25






  • 1




    @Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
    – Simon Robb
    Apr 26 '14 at 2:16






  • 1




    @SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
    – Chris Carson
    Apr 28 '14 at 17:04






  • 1




    I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
    – Chris Carson
    Jun 28 '14 at 0:55













up vote
34
down vote

favorite
15









up vote
34
down vote

favorite
15






15





I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:




Deploying new version to instance(s)




it's always followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]




The same thing happens when I try to request snapshot logs. The Beanstalk issues:




requestEnvironmentInfo is starting




then after a few minutes it's again followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].











share|improve this question















I have a PHP application deployed to Amazon Elastic Beanstalk. But I notice a problem that every time I push my code changes via git aws.push to the Elastic Beanstalk, the application deployed didn't picked up the changes. I checked the events log on my application Beanstalk environment and notice that every time the Beanstalk issues:




Deploying new version to instance(s)




it's always followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own):
[i-d5xxxxx]




The same thing happens when I try to request snapshot logs. The Beanstalk issues:




requestEnvironmentInfo is starting




then after a few minutes it's again followed by:




The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-d5xxxxx].








amazon-web-services elastic-beanstalk






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Sep 3 '17 at 12:55









Cœur

17k9102140




17k9102140










asked Apr 3 '14 at 8:51









ardfard

173125




173125












  • Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
    – Simon Robb
    Apr 4 '14 at 0:31










  • I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
    – Chris Carson
    Apr 24 '14 at 16:25






  • 1




    @Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
    – Simon Robb
    Apr 26 '14 at 2:16






  • 1




    @SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
    – Chris Carson
    Apr 28 '14 at 17:04






  • 1




    I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
    – Chris Carson
    Jun 28 '14 at 0:55


















  • Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
    – Simon Robb
    Apr 4 '14 at 0:31










  • I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
    – Chris Carson
    Apr 24 '14 at 16:25






  • 1




    @Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
    – Simon Robb
    Apr 26 '14 at 2:16






  • 1




    @SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
    – Chris Carson
    Apr 28 '14 at 17:04






  • 1




    I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
    – Chris Carson
    Jun 28 '14 at 0:55
















Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31




Hi - I'm suddenly experiencing the same problem today, with a minor incremental update on one of my apps. I think this must be a (hopefully transient) issue at Amazon's end.
– Simon Robb
Apr 4 '14 at 0:31












I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25




I'm experiencing the same thing (April 24,) both for environment updates and for logs. I have a load-balanced environment, but I think there's only one instance running. Since it's happening both for updates and logs, I don't think it's a network issue (i.e. composer timing out while fetching repos.) ardford and @Simon Robb - did this problem go away?
– Chris Carson
Apr 24 '14 at 16:25




1




1




@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16




@Chris Carson unfortunately not - I had to rebuild my environment and things have gone smoothly since then.
– Simon Robb
Apr 26 '14 at 2:16




1




1




@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04




@SimonRobb Yeah, I had to do the same. I don't think it's a transient issue -- seems to be happening to lots of folks. Thanks for the reply.
– Chris Carson
Apr 28 '14 at 17:04




1




1




I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
– Chris Carson
Jun 28 '14 at 0:55




I've found, via hard-earned experience and also from conversations with Amazon support, that this has to do with the size of the instances that you're using. t1.micro instances will more often than not fail to respond to git aws.push if they are serving a 'live' website -- i.e. if they're getting any sort of traffic from the outside world. So what worked beautifully when you were in development fails miserably in the critical few days after you've shown it to the client. The only solution I've found so far is to up the size of the instances in the environment and swap the environment URLs.
– Chris Carson
Jun 28 '14 at 0:55












6 Answers
6






active

oldest

votes

















up vote
36
down vote



accepted










I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.



If this is a production environment and you have only 1 instance and you want minimal down time




  1. configure minimum instances to 2, and Beanstalk will launch another instance for you.

  2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2

  3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.






share|improve this answer





















  • it was useful for me thanks :)
    – Rajaraman
    Dec 2 '14 at 13:42






  • 5




    This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
    – Marcus Lind
    Feb 4 '15 at 8:22








  • 2




    A simple reboot of the instance worked for me once :)
    – Babu
    May 5 '15 at 6:26






  • 1




    The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
    – Logan Pickup
    Mar 7 '16 at 2:25






  • 1




    Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
    – vivekyad4v
    Nov 16 '17 at 11:43




















up vote
5
down vote













By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
You can set an higher time up to 30 minutes (1800 seconds).



{
"Namespace": "aws:elasticbeanstalk:command",
"OptionName": "Timeout",
"Value": "1800"
}


Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html






share|improve this answer

















  • 1




    Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
    – ardfard
    Apr 4 '14 at 3:33








  • 5




    Where do you put this?
    – Spoeken
    Jun 2 '14 at 9:09










  • @Spoeken - in your myapp.config file within your .ebextensions directory
    – Meistro
    Aug 8 '14 at 16:18










  • 3600 is the new maximum limit.
    – ConorB
    Jun 13 '16 at 14:49


















up vote
1
down vote













Had the same issue here (single t1.micro instance).



Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).






share|improve this answer




























    up vote
    0
    down vote













    In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.



    If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).



    Rebuilding the environment seems to reset things with the new version being used.



    Obviously there's the downside with that of a period of downtime.






    share|improve this answer

















    • 2




      It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
      – theBarkman
      Feb 18 '16 at 18:51


















    up vote
    0
    down vote













    Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
    2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...



    If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.



    An easier way to fix SQS Client is to restart cfn-hup service:



    sudo service cfn-hup restart





    share|improve this answer




























      up vote
      0
      down vote













      I think is the correct way to deal with this.
      I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.



      chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.



      However, if you do need to increase timeout, see the following:



      Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.



      Example:



      option_settings:
      "aws:elasticbeanstalk:command":
      Timeout: 2400


      *"value" represents the length of time before timeout in seconds.



      Reference: https://serverfault.com/a/747800/496353






      share|improve this answer























        Your Answer






        StackExchange.ifUsing("editor", function () {
        StackExchange.using("externalEditor", function () {
        StackExchange.using("snippets", function () {
        StackExchange.snippets.init();
        });
        });
        }, "code-snippets");

        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "1"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














         

        draft saved


        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f22832447%2fwhy-every-time-elastic-beanstalk-issues-a-command-to-its-instance-it-always-time%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        6 Answers
        6






        active

        oldest

        votes








        6 Answers
        6






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes








        up vote
        36
        down vote



        accepted










        I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.



        If this is a production environment and you have only 1 instance and you want minimal down time




        1. configure minimum instances to 2, and Beanstalk will launch another instance for you.

        2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2

        3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.






        share|improve this answer





















        • it was useful for me thanks :)
          – Rajaraman
          Dec 2 '14 at 13:42






        • 5




          This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
          – Marcus Lind
          Feb 4 '15 at 8:22








        • 2




          A simple reboot of the instance worked for me once :)
          – Babu
          May 5 '15 at 6:26






        • 1




          The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
          – Logan Pickup
          Mar 7 '16 at 2:25






        • 1




          Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
          – vivekyad4v
          Nov 16 '17 at 11:43

















        up vote
        36
        down vote



        accepted










        I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.



        If this is a production environment and you have only 1 instance and you want minimal down time




        1. configure minimum instances to 2, and Beanstalk will launch another instance for you.

        2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2

        3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.






        share|improve this answer





















        • it was useful for me thanks :)
          – Rajaraman
          Dec 2 '14 at 13:42






        • 5




          This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
          – Marcus Lind
          Feb 4 '15 at 8:22








        • 2




          A simple reboot of the instance worked for me once :)
          – Babu
          May 5 '15 at 6:26






        • 1




          The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
          – Logan Pickup
          Mar 7 '16 at 2:25






        • 1




          Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
          – vivekyad4v
          Nov 16 '17 at 11:43















        up vote
        36
        down vote



        accepted







        up vote
        36
        down vote



        accepted






        I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.



        If this is a production environment and you have only 1 instance and you want minimal down time




        1. configure minimum instances to 2, and Beanstalk will launch another instance for you.

        2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2

        3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.






        share|improve this answer












        I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.



        If this is a production environment and you have only 1 instance and you want minimal down time




        1. configure minimum instances to 2, and Beanstalk will launch another instance for you.

        2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2

        3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Aug 7 '14 at 7:35









        chongzixin

        97531642




        97531642












        • it was useful for me thanks :)
          – Rajaraman
          Dec 2 '14 at 13:42






        • 5




          This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
          – Marcus Lind
          Feb 4 '15 at 8:22








        • 2




          A simple reboot of the instance worked for me once :)
          – Babu
          May 5 '15 at 6:26






        • 1




          The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
          – Logan Pickup
          Mar 7 '16 at 2:25






        • 1




          Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
          – vivekyad4v
          Nov 16 '17 at 11:43




















        • it was useful for me thanks :)
          – Rajaraman
          Dec 2 '14 at 13:42






        • 5




          This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
          – Marcus Lind
          Feb 4 '15 at 8:22








        • 2




          A simple reboot of the instance worked for me once :)
          – Babu
          May 5 '15 at 6:26






        • 1




          The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
          – Logan Pickup
          Mar 7 '16 at 2:25






        • 1




          Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
          – vivekyad4v
          Nov 16 '17 at 11:43


















        it was useful for me thanks :)
        – Rajaraman
        Dec 2 '14 at 13:42




        it was useful for me thanks :)
        – Rajaraman
        Dec 2 '14 at 13:42




        5




        5




        This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
        – Marcus Lind
        Feb 4 '15 at 8:22






        This issue is still happening to me and I have no idea why it keeps happening once in a while. Seems like Amazon have not fixed it. I can not find anything in the logs WHY it happens, and it even happens when I deploy files that have been successfully deployed before, so I doubt the files are wrong. Anyone know WHY it happens?
        – Marcus Lind
        Feb 4 '15 at 8:22






        2




        2




        A simple reboot of the instance worked for me once :)
        – Babu
        May 5 '15 at 6:26




        A simple reboot of the instance worked for me once :)
        – Babu
        May 5 '15 at 6:26




        1




        1




        The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
        – Logan Pickup
        Mar 7 '16 at 2:25




        The cause the server's clock drifting, and is no longer being in sync. I do not know what causes this drift (the default Elastic Beanstalk instances all have NTP set up), but once the drift exceeds more than about a minute or two, it will no longer respond to messages from Elastic Beanstalk. If you check /var/log/cfn-hup.log you will see a line similar to: Invocation xxx of command ElasticBeanstalkCommand-AWSEBAutoScalingGroup for stack arn:aws:cloudformation:... expired at 2016-03-07T02:19:37; skipping. The only solution I have found to work is the answer above: reboot (or rebuild).
        – Logan Pickup
        Mar 7 '16 at 2:25




        1




        1




        Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
        – vivekyad4v
        Nov 16 '17 at 11:43






        Did someone got any permanent solution for this ? Rebooting is the only option for me as if now. This still happens quite often & beanstalk never explains the reason for it. I am able to do almost everything with the server, it's just Beanstalk who thinks that server is dead.
        – vivekyad4v
        Nov 16 '17 at 11:43














        up vote
        5
        down vote













        By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
        You can set an higher time up to 30 minutes (1800 seconds).



        {
        "Namespace": "aws:elasticbeanstalk:command",
        "OptionName": "Timeout",
        "Value": "1800"
        }


        Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html






        share|improve this answer

















        • 1




          Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
          – ardfard
          Apr 4 '14 at 3:33








        • 5




          Where do you put this?
          – Spoeken
          Jun 2 '14 at 9:09










        • @Spoeken - in your myapp.config file within your .ebextensions directory
          – Meistro
          Aug 8 '14 at 16:18










        • 3600 is the new maximum limit.
          – ConorB
          Jun 13 '16 at 14:49















        up vote
        5
        down vote













        By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
        You can set an higher time up to 30 minutes (1800 seconds).



        {
        "Namespace": "aws:elasticbeanstalk:command",
        "OptionName": "Timeout",
        "Value": "1800"
        }


        Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html






        share|improve this answer

















        • 1




          Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
          – ardfard
          Apr 4 '14 at 3:33








        • 5




          Where do you put this?
          – Spoeken
          Jun 2 '14 at 9:09










        • @Spoeken - in your myapp.config file within your .ebextensions directory
          – Meistro
          Aug 8 '14 at 16:18










        • 3600 is the new maximum limit.
          – ConorB
          Jun 13 '16 at 14:49













        up vote
        5
        down vote










        up vote
        5
        down vote









        By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
        You can set an higher time up to 30 minutes (1800 seconds).



        {
        "Namespace": "aws:elasticbeanstalk:command",
        "OptionName": "Timeout",
        "Value": "1800"
        }


        Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html






        share|improve this answer












        By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time.
        You can set an higher time up to 30 minutes (1800 seconds).



        {
        "Namespace": "aws:elasticbeanstalk:command",
        "OptionName": "Timeout",
        "Value": "1800"
        }


        Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Apr 3 '14 at 9:15









        daveoncode

        10.6k671120




        10.6k671120








        • 1




          Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
          – ardfard
          Apr 4 '14 at 3:33








        • 5




          Where do you put this?
          – Spoeken
          Jun 2 '14 at 9:09










        • @Spoeken - in your myapp.config file within your .ebextensions directory
          – Meistro
          Aug 8 '14 at 16:18










        • 3600 is the new maximum limit.
          – ConorB
          Jun 13 '16 at 14:49














        • 1




          Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
          – ardfard
          Apr 4 '14 at 3:33








        • 5




          Where do you put this?
          – Spoeken
          Jun 2 '14 at 9:09










        • @Spoeken - in your myapp.config file within your .ebextensions directory
          – Meistro
          Aug 8 '14 at 16:18










        • 3600 is the new maximum limit.
          – ConorB
          Jun 13 '16 at 14:49








        1




        1




        Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
        – ardfard
        Apr 4 '14 at 3:33






        Hi, thanks for your answer. But I don't think the problem lies in how long the commands take their time to complete. Because some changes in requirement I had to reconfigure the Beanstalk application. Because of the changes, the Beanstalk then recreates the EC2 instance. After the application went back to live I try the same things as before but the problem disappears and the command completed under one minute! I don't really know what the real problem is but I suspect it's because in my old configuration, the Beanstalk failed to issues command to it's underlying EC2 instances.
        – ardfard
        Apr 4 '14 at 3:33






        5




        5




        Where do you put this?
        – Spoeken
        Jun 2 '14 at 9:09




        Where do you put this?
        – Spoeken
        Jun 2 '14 at 9:09












        @Spoeken - in your myapp.config file within your .ebextensions directory
        – Meistro
        Aug 8 '14 at 16:18




        @Spoeken - in your myapp.config file within your .ebextensions directory
        – Meistro
        Aug 8 '14 at 16:18












        3600 is the new maximum limit.
        – ConorB
        Jun 13 '16 at 14:49




        3600 is the new maximum limit.
        – ConorB
        Jun 13 '16 at 14:49










        up vote
        1
        down vote













        Had the same issue here (single t1.micro instance).



        Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).






        share|improve this answer

























          up vote
          1
          down vote













          Had the same issue here (single t1.micro instance).



          Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).






          share|improve this answer























            up vote
            1
            down vote










            up vote
            1
            down vote









            Had the same issue here (single t1.micro instance).



            Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).






            share|improve this answer












            Had the same issue here (single t1.micro instance).



            Did solve the problem by rebooting the EC2 instance via the EC2 page on the Management Console (and not from EB page).







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Apr 26 at 8:55









            mirzik

            111




            111






















                up vote
                0
                down vote













                In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.



                If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).



                Rebuilding the environment seems to reset things with the new version being used.



                Obviously there's the downside with that of a period of downtime.






                share|improve this answer

















                • 2




                  It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                  – theBarkman
                  Feb 18 '16 at 18:51















                up vote
                0
                down vote













                In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.



                If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).



                Rebuilding the environment seems to reset things with the new version being used.



                Obviously there's the downside with that of a period of downtime.






                share|improve this answer

















                • 2




                  It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                  – theBarkman
                  Feb 18 '16 at 18:51













                up vote
                0
                down vote










                up vote
                0
                down vote









                In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.



                If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).



                Rebuilding the environment seems to reset things with the new version being used.



                Obviously there's the downside with that of a period of downtime.






                share|improve this answer












                In the case of deployment, an alternative to shutting down the EC2 instances and waiting for Elastic Beanstalk to react, or messing about with minimum and maximum instances, is to simply perform a Rebuild environment on the target environment.



                If a previous deployment failed due to timeout then the new version will still be registered against the environment, but due to the timeout it will not appear to be operational (in my experience the instance appears to still be running the old version).



                Rebuilding the environment seems to reset things with the new version being used.



                Obviously there's the downside with that of a period of downtime.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Apr 30 '15 at 14:18









                Dan Gravell

                4,41442744




                4,41442744








                • 2




                  It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                  – theBarkman
                  Feb 18 '16 at 18:51














                • 2




                  It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                  – theBarkman
                  Feb 18 '16 at 18:51








                2




                2




                It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                – theBarkman
                Feb 18 '16 at 18:51




                It should be noted that rebuilding your environment will nuke any databases that elastic beanstalk may be controlling. So do with caution.
                – theBarkman
                Feb 18 '16 at 18:51










                up vote
                0
                down vote













                Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
                2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...



                If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.



                An easier way to fix SQS Client is to restart cfn-hup service:



                sudo service cfn-hup restart





                share|improve this answer

























                  up vote
                  0
                  down vote













                  Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
                  2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...



                  If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.



                  An easier way to fix SQS Client is to restart cfn-hup service:



                  sudo service cfn-hup restart





                  share|improve this answer























                    up vote
                    0
                    down vote










                    up vote
                    0
                    down vote









                    Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
                    2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...



                    If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.



                    An easier way to fix SQS Client is to restart cfn-hup service:



                    sudo service cfn-hup restart





                    share|improve this answer












                    Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log):
                    2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...



                    If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.



                    An easier way to fix SQS Client is to restart cfn-hup service:



                    sudo service cfn-hup restart






                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered May 30 at 17:59









                    Victor Gavrilenko

                    11




                    11






















                        up vote
                        0
                        down vote













                        I think is the correct way to deal with this.
                        I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.



                        chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.



                        However, if you do need to increase timeout, see the following:



                        Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.



                        Example:



                        option_settings:
                        "aws:elasticbeanstalk:command":
                        Timeout: 2400


                        *"value" represents the length of time before timeout in seconds.



                        Reference: https://serverfault.com/a/747800/496353






                        share|improve this answer



























                          up vote
                          0
                          down vote













                          I think is the correct way to deal with this.
                          I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.



                          chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.



                          However, if you do need to increase timeout, see the following:



                          Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.



                          Example:



                          option_settings:
                          "aws:elasticbeanstalk:command":
                          Timeout: 2400


                          *"value" represents the length of time before timeout in seconds.



                          Reference: https://serverfault.com/a/747800/496353






                          share|improve this answer

























                            up vote
                            0
                            down vote










                            up vote
                            0
                            down vote









                            I think is the correct way to deal with this.
                            I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.



                            chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.



                            However, if you do need to increase timeout, see the following:



                            Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.



                            Example:



                            option_settings:
                            "aws:elasticbeanstalk:command":
                            Timeout: 2400


                            *"value" represents the length of time before timeout in seconds.



                            Reference: https://serverfault.com/a/747800/496353






                            share|improve this answer














                            I think is the correct way to deal with this.
                            I think the correct way to deal with this is to figure out the cause of the timeout by doing what this answer suggests.



                            chongzixin's answer is what needs to be done if you need this fixed ASAP before investigating the reason for a timeout.



                            However, if you do need to increase timeout, see the following:



                            Add configuration files to your source code in a folder named .ebextensions and deploy it in your application source bundle.



                            Example:



                            option_settings:
                            "aws:elasticbeanstalk:command":
                            Timeout: 2400


                            *"value" represents the length of time before timeout in seconds.



                            Reference: https://serverfault.com/a/747800/496353







                            share|improve this answer














                            share|improve this answer



                            share|improve this answer








                            edited Nov 15 at 11:59

























                            answered Nov 14 at 20:29









                            Pranay Majmundar

                            1655




                            1655






























                                 

                                draft saved


                                draft discarded



















































                                 


                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f22832447%2fwhy-every-time-elastic-beanstalk-issues-a-command-to-its-instance-it-always-time%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                Popular posts from this blog

                                Biblatex bibliography style without URLs when DOI exists (in Overleaf with Zotero bibliography)

                                ComboBox Display Member on multiple fields

                                Is it possible to collect Nectar points via Trainline?