I had to move my server (well, I moved into a new house, so the server had to come with me) and ever since restarting it I’ve been getting inconsistent results with SequenceServer. Sometimes things work just fine, sometimes the results never come back (the app “hangs” on the busy spinner). Looking at the active processes shows me that the blastn or blastp is running, and then stops at about the expected timepoint - but the SequenceServer results don’t always come back.
So - how can I track this problem down? It seems consistently to fail with my nt database, but inconsistently fails with nr (sometimes works fine, sometimes doesn’t), and always seems to return with smaller databases. What log files should I be looking at to see what’s going on, and where can I find them?
Thanks in advance for any help you can provide…
I had to move my server (well, I moved into a new house, so the server had
to come with me) and ever since restarting it I've been getting inconsistent
results with SequenceServer. Sometimes things work just fine, sometimes the
results never come back (the app "hangs" on the busy spinner). Looking at
the active processes shows me that the blastn or blastp is running, and then
stops at about the expected timepoint - but the SequenceServer results don't
always come back.
Whenever SS hangs, is it after a fixed interval of time? It is
possible that the server times out after 30s or so. But SS should
hide the spinner in that case.
So - how can I track this problem down? It seems consistently to fail with
my nt database, but inconsistently fails with nr (sometimes works fine,
sometimes doesn't), and always seems to return with smaller databases. What
log files should I be looking at to see what's going on, and where can I
find them?
SS always logs to stderr, which by default is connected to the console
SS was started from. Logging to a file is taken care of by the
deployment setup. As far as I remember, you run SS with the default /
bundled server and have a script to start SS when the machine boots,
right? Can you post the script? I need to check if stderr is being
redirected (2>) to a file.
STDERR is not getting redirected - what do I add to the script to make
that happen?
It seems like BLAST searches with larger query strings seem to fail
consistently; smaller queries go through pretty reliably. Nothing to do
with a specific algorithm (tried blastn, blastp, etc. and they work fine
with small queries). Doesn't seem database specific (although all queries
against nt fail - too large of a database?).
Maybe it's something stupid like loose RAM? Flaky network?
The script I use to start SS up is pretty simple:
#!/bin/bash
cd /var/lib/gems/1.9.1/gems/sequenceserver-0.8.2
bin/sequenceserver
STDERR is not getting redirected - what do I add to the script to make
that happen?
bin/sequenceserver 2> file_name.log
It seems like BLAST searches with larger query strings seem to fail
consistently; smaller queries go through pretty reliably. Nothing to do
with a specific algorithm (tried blastn, blastp, etc. and they work fine
with small queries). Doesn't seem database specific (although all queries
against nt fail - too large of a database?).
Ok. But do you have some idea of the time taken by the biggest query
that doesn't cause SS to hang? A rough estimate would do.
Maybe it's something stupid like loose RAM? Flaky network?
It could be a network issue. It is also possible that somehow how IO
redirection that SS does to get result and errors fail on your system
-- broken pipe or something. I will get one of those nt databases and
try.
I appended that to my script and all that happened was that SS didn't serve web pages.
I'll do some test on the box itself when I get home and see if its a network issue. Does BLAST write logs by any chance?
You will have to give me some time to debug that.
And here’s the update - it was the network. Changing things around so that my wireless router was more reliable made all the problems go away. I didn’t see this problem with any other services - ssh seemed to make it through the glitches, as did basic http etc - maybe there’s a way to make SS handle temporary network interruptions more gracefully? In any event - it’s working for me now, so Happy Wolfgang (and Happy Students)…
Excellent.
In one of the branches I have modified SS to alert users instead of
failing silently if the network connection was dropped. Other than
that, I am not sure SS can really do anything regarding network
issues. Imo, these things should be automatically taken care of by
the server and/or other deployment infrastructure. An option could be
to use Thin web server which is more robust than WEBrick (default).
Just `[sudo] gem install thin` (sudo is options depending on how you
install gems) and SS will automatically pick it up.