Sequenceserver did not pick up BLASTDB in my environment (should it?) and crashed during install

sequenceserver
#1

Sequenceserver did not pick up BLASTDB in my environment (should it?)

and it crashed during install.

This is a fresh install and the first time I’ve used it

Here is my environment:

`
$ env | grep -i blast

BLASTDB=/n/data1/blast/db

BLASTMAT=/n/local/stage/blast/blast-2.2.26/data/

`

and install failure

`
$ /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/bin/sequenceserver

Database dir not set.

SequenceServer needs to know where your FASTA files or BLAST+ databases are.
Please enter the path to the relevant directory (default: current directory).

Press Ctrl+C to quit.

/n/data1/blast/db
/n/data1/blast/db
/n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver/database.rb:24:in initialize': undefined method downcase!’ for nil:NilClass (NoMethodError)
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver/database.rb:106:in new' from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver/database.rb:106:in block in scan_databases_dir’
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver/database.rb:103:in each_line' from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver/database.rb:103:in scan_databases_dir’
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver.rb:133:in init_database' from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/lib/sequenceserver.rb:39:in init’
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/bin/sequenceserver:133:in block (2 levels) in <main>' from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/slop-3.6.0/lib/slop.rb:260:in call’
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/slop-3.6.0/lib/slop.rb:260:in parse!' from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/slop-3.6.0/lib/slop.rb:65:in parse!’
from /n/local/stage/rbenv/rbenv-0.3.0/versions/1.9.3-p125/lib/ruby/gems/1.9.1/gems/sequenceserver-1.0.0.pre.5/bin/sequenceserver:12:in <main>'

#2

Hi,

I learned ‘just enough ruby’ to debug this myself.

It turns out that some of the databases in my BLASTDB were in some regards corrupt, or just old in some regard.

When I executed the same call to blastdbcmd that sequenceserver was executing, except from the shell, and without the -list_outfmt "%f %t %p %n %l %d" options, it completed without error and listed all the blast databases in the directory, like this:


blastdbcmd -recursive -list /n/data1/blast/db.hide/

/n/data1/blast/db/Hox_jul13-11_amemiya_454.02 Nucleotide
/n/data1/blast/db/Hox_jul13-11_amemiya_454.03 Nucleotide
/n/data1/blast/db/Kog Protein
/n/data1/blast/db/P_marinus_supercontigs_WUGSC_5.2 Nucleotide
/n/data1/blast/db/P_marinus_supers_BRI_6-0 Nucleotide

...

However, when I added the -list_outfmt option back in, I got two different kinds of errors:

BLAST Database error: No alias or index file found for nucleotide database [/n/data1/blast/db.hide/P_marinus_5-25-11_jeremiah] in search path [/home/mec/project/wwwblast:/n/data1/blast/db:]

or

BLAST Database error: Error: Not a valid version 4 database.

So, my resolution is to re-create these old indices, and post the following https://github.com/yannickwurm/sequenceserver/issues/145

Working now.

Thanks for sequenceserver!

~Malcolm

#3

Hey Malcom,

thanks for giving things a shot. We just pushed this prerelease version - so thanks for identifying a SNAFU - if you need something right away you may have more luck with ‘gem install sequenceserver’ for which the documentation is still under http://sequenceserver.com/old (we hope you’ll try and persist with the new version which is orders of magnitude more pleasant).

Sequenceserver indeed doesn’t understand the .ncbirc or the traditional BLASTDB/BLASTMAT.
Instead, the first time it runs it asks you for your db dir; and stores that value in ~/.sequenceserver.conf (which can be manually edited).

It should find ‘blastp’ (and other commands if ‘which blastp’ can find them - or alternatively similarly prompts you to enter the path towards downloaded blast binaries (or potentially downloads them).

To figure out why things aren’t behaving as they should, can I ask:

  1. which OS you’re running?
  2. output of ’ blastdbcmd -version’
  3. output of 'blastdbcmd -recursive -list /n/data1/blast/db -list_outfmt “%f %t %p %n %l %d”
  4. to try with ruby 2?

Thanks very much for your patience,

Yannick

#4

Thanks Malcolm - looks like our emails crossed paths. You’re right we should absolutely be detecting these & failing elegantly.

#5

Yes - they did cross paths. Thanks for the excellent product. I will at some point give you more feedback. I am looking for replacement for institutional wwwblast install.

Things I might look for later, or want to contribute myself, if you’re not already on it:

  • Sun Grid Engine integration (or other load balancing scheme, though we already have SGE cluster)

  • Implementation of QBlast URL API (allowing submiting jobs from alternate desktop hosts, such as Vector NTI and./or CLC Genomics which both “speak” this API)

  • ability to somehow “login” and dispatch the BLASTDB dependent upon user/group

  • barring that, ability to post ?BLASTDB=/path/to/blast/db so that users can set up virtual subsets of databases of their own design

  • and the ability to post linkout URI templates to the CGI allowing web user customization of linkback. Or, possibly pick up this aspect of configuration from .config files living in the user specified BLASTDB.

And the like.

I’m strong in perl, and used to strong in lisp, and have a first ruby project ahead of me in a few months after which, watch out! Heh.

In the mean time, I’m going to give access to sequencserver to selected powerusers at my place of work, The Stowers Institute For Medcial Research, in Kansas City, Missouri, USA.

Cheers,

Malcolm Cook (aka mec@stowers.org)

#6

Using sequenceserver a little, and have a few quick thoughts:

  • advanced command-line parameters instead of lots of UI cruft is reasonable. I like!

  • multiple database search is a win! Nice! But consider replacing UI with a better multi-select widget taking less screen. No suggestions.

  • query overview could benefit from summary total number significant hits

  • hovering over the down arrow at the top of each hit suggests it is hot - but nothing happens. What is intention here?

  • reporting number of HSPs for each hit is good. Perhaps include a HSP# with each new HSP.

Great stuff,

Malcolm

#7

Hey Malcolm,

Thanks your inputs. They are of great value.

  • hovering over the down arrow at the top of each hit suggests it is hot - but nothing happens. What is intention here?

If you are talking about the chevron on the left side of each hit, it indicates the collapsible nature of the hit box. Try clicking it.

Thanks,

#8

Thanks Malcolm this is excellent news and these are very relevant ideas (as are the suggestions in your other mail). Your contributions are most welcome!

The way we get BLAST queries from antgenomes.org (very light webserver) executed on a fat machine by ssh with the following:

  • loading our db directory from distant machine in the same path on the web server (so seqserv can find them)
  • rather than providing blast binaries directly on the light webserver, Priyam provides the following type of script (e.g.: for blastp) to execute things on a remote system:

#!/usr/bin/env sh
ssh fatserver /home/fourmidable/archive/www/ncbi-blast-2.2.27+/bin/blastp "$@"

Command-line arguments are passed, and stdout/stderr are where they should be. So sequenceserver doesn’t notice it’s not running things locally. If your queuing system supports a --follow command that outputs all stderr/stdout stuff in realtime it should be easy to make something similar.

Cheers,
Yannick

#9

Yannick,

Thanks for the tip. Sounds easy. I will take this approach for now… and follow progress on related open issues…

After using serquenceserver a little, a few other suggestions come to mind. Hope you don’t mind if I contribute:

Maybe I’ll get a chance to knock off a few …