tag:help-archives.hannonhill.com,2010-02-09:/discussions/installation/1398-assets-getting-stuck-in-publishing-queueCascade CMS: Discussion 2015-02-16T21:21:52Ztag:help-archives.hannonhill.com,2010-02-09:Comment/279196622013-07-23T18:13:45Z2013-07-23T18:13:45ZAssets getting stuck in publishing queue<div><p>I did a few searches to see if anyone else was having this
problem and I found a couple mentions of it but didn't see any real
solutions outside increasing the memory limits on the server.</p>
<p>What is happening is when I publish something out it gets stuck
in the queue and the only way to get it to publish immediately is
to restart the server. I personally haven't got anything to publish
without restarting the server but I did get a report from someone
in our organization saying that it took about 10 minutes.</p>
<p>Not sure what the problems are but I would like to get this
fixed. It doesn't happen all the time. Very sporadic so maybe it is
a memory issue. Thanks, Bradley</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622013-07-23T18:35:29Z2013-07-23T18:35:29ZAssets getting stuck in publishing queue<div><p>Hi Bradley,</p>
<p>When you have the chance can you please do the following:</p>
<ul>
<li>Tell me which version of the application you are currently
using</li>
<li>Attach a screen shot of the <strong>Administration ->
Dashboard -> Information</strong> tab</li>
<li>Attach your <em>cascade.log</em> file from today (as well as
log files from other days where you know this problem
occurred)</li>
</ul>
<p>Thanks!</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622013-07-24T18:07:49Z2013-07-24T18:07:49ZAssets getting stuck in publishing queue<div><p>I think I fixed it. I went in and figured out how to increase
the memory to the app. We have it set up as a service so fairly
easy. Lets leave this open for a week or so and if I run into any
issues I will report back. Thanks,</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622013-07-24T18:10:24Z2013-07-24T18:10:24ZAssets getting stuck in publishing queue<div><p>OK, sounds good. If you <em>do</em> run into the issue again,
please be sure to report back with the information I mentioned
<a href=
"http://help.hannonhill.com/discussions/installation/1398-assets-getting-stuck-in-publishing-queue#comment_27920058">
here</a>.</p>
<p>Take care.</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-02T18:35:34Z2014-01-02T18:35:34ZAssets getting stuck in publishing queue<div><p>Hey Tim, started running into this issue again. I increased the
memory of the service but wanted to attach the files you originally
asked for so we can see if this is the actual problem. <a href=
"http://help.hannonhill.com/discussions/installation/125-publishing-queue-is-frozen">
Here are the instructions I used to do so</a>. When I look at the
Admin -> Dashboard -> Information Tab it doesn't looks like
the ram is picking up but I could be wrong.</p>
<p>Here is a list of what is attached:</p>
<p>LOGS (These are the most recent days where I experienced an
issue)<br>
-Today | Yesterday | Tuesday</p>
<p>SCREENSHOTS<br>
- AdminInfo.png -> Admin -> Dashboard -> Information Tab -
TomcatBinCascadew.png -> Directory with .exe that handles ram
increase -TomcatBinCascadewProperties - cascadew.exe properties</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-02T18:53:36Z2014-01-02T18:53:36ZAssets getting stuck in publishing queue<div><p>Hi Bradley,</p>
<p>Thanks for attaching those items. I'm not seeing any memory
issues in the log files, but I do see a lot of database deadlock.
Ex.:<br></p>
<pre>
<code>Caused by: java.sql.SQLException: Transaction (Process ID 63) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.</code>
</pre>
These messages tend to appear a lot more often if your database
doesn't happen to be using snapshot isolation. When you have the
chance, can you or your DBA check to see if snapshot isolation is
enabled on the database? You can do so by executing a query like
this:<br>
<pre>
<code>SELECT is_read_committed_snapshot_on FROM sys.databases WHERE name='{databaseName}';</code>
</pre>
(where {databaseName} is the name of the Cascade Server database)
<p>If that returns false (0), you'll want to follow <strong>Step
4</strong> of <a href=
"http://help.hannonhill.com/kb/database/setting-up-the-database-sql-server">
these instructions</a> to enable snapshot isolation.</p>
<p>Let me know what you find.</p>
<p>Also, can you tell me how much total RAM is installed on the
application server?</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-06T22:30:14Z2014-01-06T22:30:14ZAssets getting stuck in publishing queue<div><p>Ok I made the change that you suggested. Seems to be working a
bit smoother. I will keep an eye on it. Thanks for your help.
-Bradley</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-06T22:40:45Z2014-01-06T22:40:45ZAssets getting stuck in publishing queue<div><p>Perfect. Let it run for a few days with this new setting and let
me know if you continue to have problems.</p>
<p>Thanks!</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-20T18:47:47Z2014-01-20T18:47:47ZAssets getting stuck in publishing queue<div><p>Tim, just ran into another issue of the cms not publishing. Like
I said in a previous post I updated the
is_read_committed_snapshot_on to <strong>ON</strong>.</p>
<p>I have attached the log so you can take a look. Thanks.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-20T18:58:08Z2014-01-20T18:58:08ZAssets getting stuck in publishing queue<div><p>Bradley,</p>
<p>Thanks for attaching that log file. This time around I'm not
seeing any deadlock, so I'm curious to hear more about this
problem. When you say that the system is "not publishing", do you
mean that there are jobs in the publish queue that haven't
progressed through the queue in several minutes? Or, are you saying
that when you click to publish an asset you never see it arrive in
the queue?</p>
<p>If jobs are sitting in the queue and don't appear to be moving,
how long have you waited and what type(s) of asset are in there
(pages, files, folders)? Are any of these assets very large?</p>
<p>The one thing I am seeing in your log file several times is the
following message:<br></p>
<pre>
<code>2014-01-20 07:25:23,334 ERROR [JDBCTransaction] JDBC begin failed
java.sql.SQLException: I/O Error: Connection reset</code>
</pre>
We may be able to resolve these by adding some parameters to your
<em>tomcat/conf/context.xml</em> file. Can you attach that here?
Feel free to mask the username/password in that file.
<p>Last but not least, what version of Cascade Server are you
using?</p>
<p>Thanks</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-20T20:15:14Z2014-01-20T20:15:14ZAssets getting stuck in publishing queue<div><p>Yes there are jobs in the publish queue that are getting stuck
and won't publish. This is the same problem that I have been
dealing with all along. The only way I know how to fix it is to
restart the server. Then it works for a while before reverting back
to getting stuck again.</p>
<p>We are using version 7.2.2 but are about to upgrade to 7.8
today. See attached for file you requested:</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-20T20:40:54Z2014-01-20T20:40:54ZAssets getting stuck in publishing queue<div><p>Thanks for attaching that. I may have you try adding a
validation query to your configuration, but first I think it would
be good to get some more information from you and/or your DBA
regarding the database side of things. When you have the chance,
can you have your DBA provide us with logging from SQL Server
around the following times:</p>
<pre>
<code>2014-01-20 07:25:23
2014-01-20 08:59:26
2014-01-20 09:47:31</code>
</pre>
<p>Also - is there a firewall sitting between the application and
database server? If so, any logs from today showing communication
between the app and db server might be helpful (specifically around
the times mentioned above).</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-21T16:58:00Z2014-01-21T16:58:00ZAssets getting stuck in publishing queue<div><p>Here is what my DBA sent me. Let me know if that is what you are
looking for. It is using SQLServer 2012 64bit. The logging mode is
set to simple.</p>
<p>There isn't much in this file so if I need to do something to
get more details let me know.</p>
<p>No firewall between app and DB.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-21T22:22:40Z2014-01-21T22:22:40ZAssets getting stuck in publishing queue<div><p>Bradley,</p>
<p>Thanks for attaching that SQL log. Unfortunately, as you
mentioned, it doesn't look like it will be much help. I do have
some steps for you to follow the next time you run into this
problem which will hopefully help us get a better idea of what's
going on. So, the next time the publisher appears to be 'stuck', do
the following <strong>prior</strong> to restarting the
instance:</p>
<ul>
<li>Click <strong>System Menu -> Utilities ->
Logging</strong></li>
<li>In the <strong>Configure by Category</strong> drop down, select
<strong>Publishing</strong></li>
<li>In the drop down next to that, select
<strong>DEBUG</strong></li>
<li>Click <strong>Add/Update Category</strong></li>
<li>Leave the instance running for approximately 5 minutes</li>
<li>After 5 minutes have passed, immediately attach the most recent
<em>cascade.log</em> file (located in <em>tomcat/logs</em>)</li>
<li>Restart Cascade Server</li>
</ul>
<p>Thanks</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-01-21T22:54:07Z2014-01-21T22:54:07ZAssets getting stuck in publishing queue<div><p>Sounds good. Will do. Thanks!</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-10T19:43:01Z2014-02-10T19:43:01ZAssets getting stuck in publishing queue<div><p>I ran across this problem again today. Here is the log. Can you
take a look when you get a moment? Thanks!</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-10T19:47:14Z2014-02-10T19:47:14ZAssets getting stuck in publishing queue<div><p>Thanks for including that log file. I'll have a look over it
with one of my developers and I'll comment back here once I have
more information for you.</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-20T16:50:31Z2014-02-20T16:50:31ZAssets getting stuck in publishing queue<div><p>Any feedback on this?</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-20T19:14:58Z2014-02-20T19:14:58ZAssets getting stuck in publishing queue<div><p>Hi Bradley,</p>
<p>A couple of my developers have had a chance to look at this but
we don't have any new information yet. I'm hoping to get some more
information and/or next steps soon. Once I do I'll update you
here.</p>
<p>Thanks for your patience!</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-20T21:48:54Z2014-02-20T21:52:28ZAssets getting stuck in publishing queue<div><p>No problem. We currently have an issue with our hyper v server.
Kind of went flaky on us causing the virtual machine that hosts our
Cascade installation to crap out. The server is up and running but
I am having a hard time getting Cascade up and running again.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-21T19:22:42Z2014-02-21T19:22:42ZAssets getting stuck in publishing queue<div><p>Bradley,</p>
<p>The next time this problem occurs, can you please do the
following prior to restarting Cascade Server:</p>
<ul>
<li>Execute this SQL query:
<pre>
<code>select * from cxml_publishrequest</code>
</pre>
and attach the results here.<br></li>
<li>Then, follow <a href=
"http://help.hannonhill.com/discussions/installation/1398-assets-getting-stuck-in-publishing-queue/page/1#comment_31237585">
these steps</a> again and attach that information.</li>
</ul>
<p>Sorry for all of the back and forth. That first query should
tell our developers what state the publish queue is in when the
problem is happening.</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-02-24T22:57:41Z2014-02-24T22:57:41ZAssets getting stuck in publishing queue<div><p>Will do. Thanks again for your help.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-03-10T17:10:37Z2014-03-10T17:10:37ZAssets getting stuck in publishing queue<div><p>Ok here ya go. Followed instructions from last post. Attached is
the data from the query and then the log file. Thanks for your
help.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-03-10T21:47:55Z2014-03-10T21:47:55ZAssets getting stuck in publishing queue<div><p>Hi Bradley,</p>
<p>Thanks so much for this information. Again, I apologize for all
of the back and forth communication on this ticket. Issues like
this can be tough to track down. We are definitely getting closer.
We have a few more questions for you based on what we've seen in
your latest log file (and the state of the database):</p>
<ul>
<li>Do you know if your instance is being restarted occasionally
throughout the week (either manually or on a schedule)? I'm
referring to restarts not related to this problem in any way.</li>
<li>The next time this issue occurs, can you do the following:
<ul>
<li>Stop Cascade Server</li>
<li>Execute the query:
<pre>
<code>select * from QRTZ_TRIGGERS</code>
</pre>
and then provide us with the results.</li>
</ul>
</li>
</ul>
<p>One possibility we're looking into is whether some restarts of
the instance that may be occurring in your environment might be
putting the publish queue in a bad state. What we've noticed is
that all other scheduled jobs in your instance appear to be running
as expected - except for the publish trigger (which fires jobs in
the publish queue). If this is indeed happening, it will cause the
behavior you've been describing where publish jobs simply don't get
picked up in the queue anymore.</p>
<p>The results from that last query I just provided should help us
determine whether this is the case.</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-03-20T20:22:41Z2014-03-20T20:22:41ZAssets getting stuck in publishing queue<div><p>Here is the information you asked for.</p>
<p>Thanks again for all your help. Let me know if you need
anything.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-03-20T20:36:37Z2014-03-20T20:36:37ZAssets getting stuck in publishing queue<div><p>Sorry, forgot to answer your questions. I went into windows task
scheduler and didn't see anything that is set to restart the
computer. If I am looking in the wrong place let me know and I will
take a look.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-03-28T20:31:42Z2014-03-28T20:31:42ZAssets getting stuck in publishing queue<div><p>Hi Bradley,</p>
<p>We appreciate the latest update. Based on what we're seeing,
we'd now like to get a thread dump of the process:</p>
<ul>
<li>Next time the issue occurs, leave Cascade Server running</li>
<li>On the application server:
<ul>
<li>Open a command prompt with the 'Run as Administrator'
option</li>
<li>Navigate to the <em>tomcat/bin</em> directory and run the
command <code>cascadew.exe //MS//</code>. For example:
<pre>
<code>C:\Program Files\Cascade Server\tomcat\bin>cascadew.exe //MS//</code>
</pre></li>
<li>At this point, you should notice a Cascade Server icon in the
system tray (with a green 'play' button over it)</li>
<li>Right-click the icon in the system tray and select
<strong>Thread Dump</strong></li>
<li>Zip up your entire <em>tomcat/logs</em> directory and attach it
here</li>
</ul>
</li>
</ul>
<p>If that file is more than ~10MB, feel free to place it somewhere
for us to download - or - you can upload it to our SFTP server:</p>
<p>host: drop.hannonhill.com<br>
username: transfer<br>
password: f1l3$4HH<br>
directory: files</p>
<p>Just let us know once it's there so we can download it.</p>
<p>Thanks so much again for your patience. This is proving tough to
track down.</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-05-22T22:17:35Z2014-05-22T22:17:35ZAssets getting stuck in publishing queue<div><p>Sorry guys for such a long delay. Up until now I haven't had any
problem. I followed the directions you sent over. Here is a zip
file of all of our logs. I did the Thread Dump on 5/22/14</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-05-29T19:38:54Z2014-05-29T19:38:54ZAssets getting stuck in publishing queue<div><p>Bradley,</p>
<p>When my developer was going through one of the thread dumps, he
noticed that it didn't match up with the source code for our jTDS
driver. I looked back at a screen shot you attached earlier in this
discussion and it appears that the jTDS 1.2.2 driver is in use.
When you have the chance, can you replace it with the 1.2.5 driver
I've attached here?</p>
<p>Steps for replacing are as follows:</p>
<ul>
<li>Stop Cascade Server</li>
<li>Move the file <em>tomcat/lib/jtds-1.2.2.jar</em> to a safe
place (your Desktop, for example, in case we need to have you
restore it)</li>
<li>Download the attached <em>jtds-1.2.5.jar</em> file and place it
in your <em>tomcat/lib</em> directory</li>
<li>Start Cascade Server</li>
</ul>
<p>Once the new .jar file is in place, we'll want to get another
thread dump from you if you do run into the behavior again.</p>
<p>Thanks!</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-06-25T16:52:10Z2014-06-25T16:52:10ZAssets getting stuck in publishing queue<div><p>Ok thanks. I ran into the issue again. I noticed that our
directory had two versions of jtds.jar. The old version and then
the version you attached. I removed them both and replaced it with
the one you provided. We shall see what happens. Thanks again for
your help. I will let you know if we have any more issues.</p></div>bwmyerstag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-06-25T17:43:26Z2014-06-25T17:43:26ZAssets getting stuck in publishing queue<div><p>Perfect. Thank you for the update, Bradley. We're actually
hoping this new driver will fix the issue outright, but if it
doesn't, we'll at least be able to track down what is going on once
we get a look at a new thread dump (with the more recent driver in
place).</p></div>Timtag:help-archives.hannonhill.com,2010-02-09:Comment/279196622014-10-08T18:23:17Z2014-10-08T18:23:17ZAssets getting stuck in publishing queue<div><p>Hey Bradley,</p>
<p>Just checking in. Have you run into this problem at all after
updating the MSSQL driver? If you have not, I'm thinking we can
probably close this discussion and then you can always comment back
(to automatically re-open it) if you do experience this behavior
down the road.</p>
<p>Thanks!</p></div>Tim