I have encountered this error when using sequenceserver
If I remember correctly this happens when your sequence ids aren’t unique across different FASTA files. e.g., having >scaffold0001 in two FASTA files. BLAST doesn’t like it and outputs a malformed XML (only in case of TBLASTX - I think).
If not, please look up the google group archive - it has come up before.
Priyam