apache - Service Temporarily Unavailable under load in OpenShift Enterprise 2.0 -


using openshift enterprise 2.0, have simple jbossews (tomcat7) + mysql 5.1 app uses jsp files connected mysql database. app created non-scaled app (fwiw same issue happens when scaling enabled).

using jmeter driver single concurrent user , no think time, chug along 2 minutes (at 200 req/sec) , start returning "503 service temporarily unavailable" in batches (a few seconds @ time) on , off remainder of test. if change nothing (don't restart app) if wait moment , try again, same thing--first seems fine, start errors.

the gear far fully-utilized (memory/cpu), , log can find shows problem /var/log/httpd/error_log, fills these entries:

[tue mar 25 15:51:13 2014] [error] (99)cannot assign requested address: proxy: http: attempt connect 127.8.162.129:8080 (*) failed 

looking @ 'top' command on node host @ time errors start occur, see several httpd processes surge top on , off.

so looks somehow running out of proxy connections or similar. however, i'm not sure how happening single concurrent user. ideas of how fix this? couldn't find similar posts.

the core problem system running out of ephemeral ports due connections stuck in time_wait. check using:

netstat -pan --tcp | less 

or

netstat -pan --tcp | grep -c ".*time_wait" 

to count number of connections in time wait state.

these connections made node port proxy (httpd) tomcat backend. there several ways change tcp settings in order lessen problem. first attempt enable reuse. append following /etc/sysctl.conf:

# allow reuse of time_wait connections net.ipv4.tcp_tw_reuse=1 

this allow connections in time_wait state reused if there no ephemeral ports available.

however,the problem remains these connections not being pooled. not run issue outside of gear same app+driver--meaning connections pooled , don't have sit in time_wait state @ all. in proxy must interfering connection closure.

looks mod_proxy / mod_rewrite not configured connection pooling/keepalive or not compatible it.


Comments

Popular posts from this blog

java - WrongTypeOfReturnValue exception thrown when unit testing using mockito -

php - Magento - Deleted Base url key -

android - How to disable Button if EditText is empty ? -