Home → General →

Database backup and CMS restarting (DB locking issue)

• Edit

irifkin

18 Jan, 2011 05:04 PM

Hi,

We recently heard a complaint about overnight scheduled publish jobs not happening. I don't see any publish messages, yet I see it's scheduled, so I looked into what's happening at night:

Every night we schedule Cascade to shutdown at 11:55PM.
We start a database backup (mysqldump) at 12:00AM.
We run a script to delete old log files at 12:30AM.
We run a script to rotate logs at 12:35AM.
We schedule Cascade to start at 12:55AM.

The scheduled publish job in question is set to start at 1:30AM.

The mysqldump is taking an increasingly long time, I presume because of the database size. The past two nights it didn't finish until close to 2AM (last night it completed at 1:53AM). Consequently when Cascade starts at 12:55AM it gets stuck waiting for a DB lock until 1:53AM:

2011-01-18 00:55:30,686 INFO [liquibase] Lock Database
2011-01-18 01:53:38,226 INFO [liquibase] Successfully acquired change log lock
2011-01-18 01:53:40,355 INFO [liquibase] Reading from DATABASECHANGELOG
2011-01-18 01:53:41,582 INFO [liquibase] Release Database Lock
2011-01-18 01:53:41,583 INFO [liquibase] Successfully released change log lock

It finishes starting up one minute later:
2011-01-18 01:54:14,151 INFO [StartupTasks] *** Startup tasks finished

I'm not sure why the scheduled publish doesn't start as I thought it would start once Cascade came up. We can start it later and/or shutdown/backup earlier if needed, though that further limits the already tight available overnight publish time. It would be great if the database backup could be quicker, but is there a way to avoid the locking conflict? Then again one of the reasons we shut down Cascade is to make the mysqldump easier without Cascade trying to use the database.

Do you have any suggestions?

Production instance is Cascade 6.4.5 running on Linux.

Thanks,
Ian

1. Support Staff 1 Posted by Tim on 18 Jan, 2011 10:18 PM

   

   Hi Ian,

   You'll probably want to schedule Cascade to start up later in order to give the database more time to back up. If a scheduled publish is supposed to fire while Cascade is not running, I don't believe it will fire the job immediately upon starting Cascade. Instead, it will skip the job and just wait until the next fire time (the next scheduled publish).

   Let me know if you have further questions.

   Thanks

   • Edit

2. 2 Posted by irifkin on 19 Jan, 2011 01:09 AM

   

   Okay, thanks for the reply. I just wanted to make sure I wasn't missing anything and that there isn't a better practice.

   Thanks,
   Ian

   • Edit

3. Support Staff 3 Posted by Tim on 19 Jan, 2011 02:17 PM

   

   No problem Ian!

   • Edit

4. Tim closed this discussion on 19 Jan, 2011 02:17 PM.