Production CMS Down

irifkin's Avatar

irifkin

03 Aug, 2011 08:00 PM

Hello,

As you know, Brandeis upgraded to version 6.8.3.1 yesterday. It was going fine until 5 minutes ago when it was inaccessible. Nothing loading in Web browser.

In the cascade.log I see various errors, such as complaining about the max_allowed_packet size of MySQL and perhaps more concerning "Could not synchronize database state with session".

I tried stopping cascade and got the following error:
Using CATALINA_BASE: /var/applications/cascadeserver/production/tomcat
Using CATALINA_HOME: /var/applications/cascadeserver/production/tomcat
Using CATALINA_TMPDIR: /var/applications/cascadeserver/production/tomcat/temp
Using JRE_HOME: /usr/public/jdk1.6.0_24/
Aug 3, 2011 3:51:22 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused

    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.connect(Socket.java:478)
    at java.net.Socket.<init>(Socket.java:375)
    at java.net.Socket.<init>(Socket.java:189)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:395)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:344)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:435)

I killed the processes, then started the application back up and it seems OK (for now?)

I'm attaching our cascade.log and catalina.out files for your review. Please advise!

Thank you,
Ian

Showing page 2 out of 2. View the first page

  1. 31 Posted by irifkin on 16 Aug, 2011 07:19 PM

    irifkin's Avatar

    Hi,

    It looks like those severe errors are better. Looking at today's catalina.out:

    [irifkin@cms-app-prod logs]$ grep -i severe catalina.out | sort | uniq -c

    1 SEVERE: Context [/._ROOT] startup failed due to previous errors
    1 SEVERE: Error getConfigured
    1 SEVERE: Error in resourceStart()
    1 SEVERE: Error starting static Resources
    1 SEVERE: Exception fixing docBase: {0}

    Looking back at the past few days shows similar results.

    Ian

  2. 32 Posted by irifkin on 16 Aug, 2011 07:21 PM

    irifkin's Avatar

    Is there anything else we should do (e.g. to verify the maxActive number is ideal for my environment)?

    Anything we can do to make improvements is great, but my main goal is to avoid crashes.

    Thanks,
    Ian

  3. 33 Posted by Joel on 16 Aug, 2011 08:40 PM

    Joel's Avatar

    Ian,

    I'm not sure what we coud do to test the MaxActive variable, as it's a setting that you'll have to tune as activity grows within your instance of Cascade Server. I understand the need for preventing crashes, but the most we can do at this point is closely monitor your instance of Cascade Server and tune it accordingly.

    Thanks!

  4. 34 Posted by irifkin on 16 Aug, 2011 08:47 PM

    irifkin's Avatar

    Thanks for the reply. If you think we've done all relevant investigation and tuning relevant to the Cascade Server crash then I think this ticket is all set.

    Good news is that is hasn't crashed again since then yet.

    Thank you for your help.

    Ian

  5. 35 Posted by irifkin on 16 Aug, 2011 09:04 PM

    irifkin's Avatar

    FYI, I found that there is a max_used_connections variable in "show status" which shows the max connections since mysql was rebooted. Our CMS prod DB currently has this at 35.

    Ian

  6. 36 Posted by Joel on 30 Aug, 2011 03:26 PM

    Joel's Avatar

    Good deal Ian, I'd use that to monitor the amount of connections your instance is using and tune it accordingly every so often.

    Thanks!

  7. Joel closed this discussion on 30 Aug, 2011 03:26 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac