close_waits with apache reverse proxy from AMQ and websockets
up vote
0
down vote
favorite
We've been drumming the woodwork for a solution to a problem we're having on a server hosting an apache (it has to be apache) reverse proxy. I'm posting this in hopes something jumps out we haven't thought to check yet.
It's re a Spring boot application using a Zuul gateway in front of microservices, and a legacy Spring MVC app upgraded to run with Spring boot libraries. When a change happens on the legacy app, a route is called on an entity service and AMQ is used to send an mqtt message telling the new app to check the state of the MS sql server database and update a web display driven by Ember. All this runs through 2 apache reverse proxies. The first of the two is where the problems occur.
Client (C) -> Reverse Proxy 1 (RP1) -> Reverse Proxy 2(RP2) -> Service (S)
C: running on Ember.js framework, with ember-paho-mqtt library
It works fine in local dev, but when deployed to windows staging servers the app fails after a few minutes, and then netstat shows a flurry of close_wait states that don't go away, on the server hosting the apache reverse proxy. We believe those follow the failure rather than cause it, but that something might be exhausting some thread pool, maybe that used by apache -- like clients trying to connect unnecessarily? But the close_waits happen when the server has got a message telling it to close a connection, has finished working through any data in process, but hasn't heard back from the client telling it to finish the job, right?
The web UIs fail, but existing connections continue to work when tested more directly and sometimes the system recovers eventually, sometimes not.
The legacy app is deployed to JBoss, the microservices to RedHat tomcat servers.
We also use an azure cloud service as part of a complex oauth2 authorization process.
I'm part of a team and only peripherally involved in the project, mainly re the aforementioned legacy app. We believe the problems occur when the legacy app data is changed and the message goes to the spring boot microservices and that the culprit somehow has to do with websockets. But my understanding of the architecture is vaguer than I'd like, sorry about that.
We do have some existing theories. A couple of years ago I worked on a Netty network application in 'nix, and had a bunch of close_waits I was able to prevent by setting thread limits in Java. AMQ uses Netty under the hood, so one option might be to swap it out and put in RabbitMQ instead. We use Paho for mqtt and have looked hard at how to make sure it closes connections. I've also read websockets.io can cause such problems and that substituting SockJS which is simpler can help. We've tried setting and undoing various timeouts/keepalives without success. We've read there's a thread limitation on Apache the way we've probably got it configured, but that increasing it can make things worse by risking thrashing from memory swapping.
So we're clutching at straws, and would welcome any lifelines thrown our way. I'll do my best to provide more info to interested parties.
apache spring-boot websocket reverse-proxy amq
asked Nov 13 at 22:19
wial
12
add a comment |
up vote
0
down vote
favorite
We've been drumming the woodwork for a solution to a problem we're having on a server hosting an apache (it has to be apache) reverse proxy. I'm posting this in hopes something jumps out we haven't thought to check yet.
It's re a Spring boot application using a Zuul gateway in front of microservices, and a legacy Spring MVC app upgraded to run with Spring boot libraries. When a change happens on the legacy app, a route is called on an entity service and AMQ is used to send an mqtt message telling the new app to check the state of the MS sql server database and update a web display driven by Ember. All this runs through 2 apache reverse proxies. The first of the two is where the problems occur.
Client (C) -> Reverse Proxy 1 (RP1) -> Reverse Proxy 2(RP2) -> Service (S)
C: running on Ember.js framework, with ember-paho-mqtt library
It works fine in local dev, but when deployed to windows staging servers the app fails after a few minutes, and then netstat shows a flurry of close_wait states that don't go away, on the server hosting the apache reverse proxy. We believe those follow the failure rather than cause it, but that something might be exhausting some thread pool, maybe that used by apache -- like clients trying to connect unnecessarily? But the close_waits happen when the server has got a message telling it to close a connection, has finished working through any data in process, but hasn't heard back from the client telling it to finish the job, right?
The web UIs fail, but existing connections continue to work when tested more directly and sometimes the system recovers eventually, sometimes not.
The legacy app is deployed to JBoss, the microservices to RedHat tomcat servers.
We also use an azure cloud service as part of a complex oauth2 authorization process.
I'm part of a team and only peripherally involved in the project, mainly re the aforementioned legacy app. We believe the problems occur when the legacy app data is changed and the message goes to the spring boot microservices and that the culprit somehow has to do with websockets. But my understanding of the architecture is vaguer than I'd like, sorry about that.
We do have some existing theories. A couple of years ago I worked on a Netty network application in 'nix, and had a bunch of close_waits I was able to prevent by setting thread limits in Java. AMQ uses Netty under the hood, so one option might be to swap it out and put in RabbitMQ instead. We use Paho for mqtt and have looked hard at how to make sure it closes connections. I've also read websockets.io can cause such problems and that substituting SockJS which is simpler can help. We've tried setting and undoing various timeouts/keepalives without success. We've read there's a thread limitation on Apache the way we've probably got it configured, but that increasing it can make things worse by risking thrashing from memory swapping.
So we're clutching at straws, and would welcome any lifelines thrown our way. I'll do my best to provide more info to interested parties.
apache spring-boot websocket reverse-proxy amq
asked Nov 13 at 22:19
wial
12
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
We've been drumming the woodwork for a solution to a problem we're having on a server hosting an apache (it has to be apache) reverse proxy. I'm posting this in hopes something jumps out we haven't thought to check yet.
It's re a Spring boot application using a Zuul gateway in front of microservices, and a legacy Spring MVC app upgraded to run with Spring boot libraries. When a change happens on the legacy app, a route is called on an entity service and AMQ is used to send an mqtt message telling the new app to check the state of the MS sql server database and update a web display driven by Ember. All this runs through 2 apache reverse proxies. The first of the two is where the problems occur.
Client (C) -> Reverse Proxy 1 (RP1) -> Reverse Proxy 2(RP2) -> Service (S)
C: running on Ember.js framework, with ember-paho-mqtt library
It works fine in local dev, but when deployed to windows staging servers the app fails after a few minutes, and then netstat shows a flurry of close_wait states that don't go away, on the server hosting the apache reverse proxy. We believe those follow the failure rather than cause it, but that something might be exhausting some thread pool, maybe that used by apache -- like clients trying to connect unnecessarily? But the close_waits happen when the server has got a message telling it to close a connection, has finished working through any data in process, but hasn't heard back from the client telling it to finish the job, right?
The web UIs fail, but existing connections continue to work when tested more directly and sometimes the system recovers eventually, sometimes not.
The legacy app is deployed to JBoss, the microservices to RedHat tomcat servers.
We also use an azure cloud service as part of a complex oauth2 authorization process.
I'm part of a team and only peripherally involved in the project, mainly re the aforementioned legacy app. We believe the problems occur when the legacy app data is changed and the message goes to the spring boot microservices and that the culprit somehow has to do with websockets. But my understanding of the architecture is vaguer than I'd like, sorry about that.
We do have some existing theories. A couple of years ago I worked on a Netty network application in 'nix, and had a bunch of close_waits I was able to prevent by setting thread limits in Java. AMQ uses Netty under the hood, so one option might be to swap it out and put in RabbitMQ instead. We use Paho for mqtt and have looked hard at how to make sure it closes connections. I've also read websockets.io can cause such problems and that substituting SockJS which is simpler can help. We've tried setting and undoing various timeouts/keepalives without success. We've read there's a thread limitation on Apache the way we've probably got it configured, but that increasing it can make things worse by risking thrashing from memory swapping.
So we're clutching at straws, and would welcome any lifelines thrown our way. I'll do my best to provide more info to interested parties.
apache spring-boot websocket reverse-proxy amq
asked Nov 13 at 22:19
wial
12
We've been drumming the woodwork for a solution to a problem we're having on a server hosting an apache (it has to be apache) reverse proxy. I'm posting this in hopes something jumps out we haven't thought to check yet.
It's re a Spring boot application using a Zuul gateway in front of microservices, and a legacy Spring MVC app upgraded to run with Spring boot libraries. When a change happens on the legacy app, a route is called on an entity service and AMQ is used to send an mqtt message telling the new app to check the state of the MS sql server database and update a web display driven by Ember. All this runs through 2 apache reverse proxies. The first of the two is where the problems occur.
Client (C) -> Reverse Proxy 1 (RP1) -> Reverse Proxy 2(RP2) -> Service (S)
C: running on Ember.js framework, with ember-paho-mqtt library
It works fine in local dev, but when deployed to windows staging servers the app fails after a few minutes, and then netstat shows a flurry of close_wait states that don't go away, on the server hosting the apache reverse proxy. We believe those follow the failure rather than cause it, but that something might be exhausting some thread pool, maybe that used by apache -- like clients trying to connect unnecessarily? But the close_waits happen when the server has got a message telling it to close a connection, has finished working through any data in process, but hasn't heard back from the client telling it to finish the job, right?
The web UIs fail, but existing connections continue to work when tested more directly and sometimes the system recovers eventually, sometimes not.
The legacy app is deployed to JBoss, the microservices to RedHat tomcat servers.
We also use an azure cloud service as part of a complex oauth2 authorization process.
I'm part of a team and only peripherally involved in the project, mainly re the aforementioned legacy app. We believe the problems occur when the legacy app data is changed and the message goes to the spring boot microservices and that the culprit somehow has to do with websockets. But my understanding of the architecture is vaguer than I'd like, sorry about that.
We do have some existing theories. A couple of years ago I worked on a Netty network application in 'nix, and had a bunch of close_waits I was able to prevent by setting thread limits in Java. AMQ uses Netty under the hood, so one option might be to swap it out and put in RabbitMQ instead. We use Paho for mqtt and have looked hard at how to make sure it closes connections. I've also read websockets.io can cause such problems and that substituting SockJS which is simpler can help. We've tried setting and undoing various timeouts/keepalives without success. We've read there's a thread limitation on Apache the way we've probably got it configured, but that increasing it can make things worse by risking thrashing from memory swapping.
So we're clutching at straws, and would welcome any lifelines thrown our way. I'll do my best to provide more info to interested parties.
apache spring-boot websocket reverse-proxy amq
apache spring-boot websocket reverse-proxy amq
asked Nov 13 at 22:19
wial
12
asked Nov 13 at 22:19
wial
12
asked Nov 13 at 22:19
wial
12
asked Nov 13 at 22:19
wial
12
asked Nov 13 at 22:19
wial
12
12
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290385%2fclose-waits-with-apache-reverse-proxy-from-amq-and-websockets%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
