I have to detect orthologs by using reciprocal BLAST, i.e. blasting the set A against the set B and then blasting the set B against the set A.

If the sets A and B contain redundant entries, how should I deal with the redundancies in both sets (query sets and subject sets)? I was thinking about grouping the genes with the top scores into one. Someone told me that grouping them by scores won't work since it depends on the sequence length. Another idea was to make the non-redundant sets for A and B, which contains the representatives from a cluster of similar sequences within each set. I don't want to go down this road since taking a representatives means a loss of information.

Can you give me an advice on this? If there is any other thing that I should watch out for when doing the reciprocal blast, please let me know. I'd appreciate it very much.

This is more of a Genomics-related question, so I will do my best to answer it. I think you should be able to identify your redundant sequences by plotting a histogram of the blastn percent identity (not score) between ESTs in a self-blast (all ESTs blasted against the same set of ESTs). You should see a bimodal disribution, with redundant ESTs having a percent identity around 98-100% (depending on if they are quality-trimmed etc). You should see a second, broader, peak representing paralogs or gene family members. You should be able to set a threshold for identifying redundant ESTs based on the spread of the first peak. At that point, you would want to choose one EST from each cluster to represent your gene. In the case of UniGene, I believe they use the longest one (which is usually a full-length cDNA sequence).

If you find RBH is problematic in your case, you may want to compare its performance to that of the RSD method (reciprocal smallest distance), which is supposed to be more robust than RBH.

see documentation

- NRDB is a free software product provided by NCBI to eliminate redundancy in sequence sets without losing information.
- SLIM Search (a commercial product) can run the searches for a full genome pair-wise comparison in a few minutes on a typical PC (instead of months on a super computer), so you have more opportunity to find parameters that provide the best biological results.
- There are issues with using Blast percent identities and scores as these relate only to the largest local alignment of the sequence pair. As a result, you can get a short fragment with 100% identity over 100nt out of 1,000 that shows up as a better hit than 80% identity distributed over 1,000nt. This is particularly problematic with transcription factors that show high conservation in short domains with high divergence between.

Also, it would be useful to differentiate between out-paralogs and in-paralogs. Out-paralogs are paralogs that formed before the speciation event, while in-paralogs are those that formed after the speciation event. In-paralogs are true orthologs.

Inparanoid is a BLAST based tool that attempts to differentiate between the two categories:

Remm M, Storm CE, Sonnhammer EL. Automatic clustering of orthologs and in-paralogs from pairwise species comparisons. 2001. J Mol Biol 314:1041-1052.

Online tool located here: http://inparanoid.sbc.su.se/cgi-bin/index.cgi