org.tgels.search.rankingalgorithm
Class RankingQuery

java.lang.Object
  org.tgels.search.rankingalgorithm.RankingQuery

public class RankingQuery
extends java.lang.Object

The RankingAlgorithm implementation, uses Apache Lucene to get documents from the index but scores and ranks using the RankingAlgorithm

Two Algorithm are available SIMPLE and COMPLEX. SIMPLE is a very fast algorithm and can return queries in <100ms on a 10m wikipedia index (complete index). It can also scale to 100m docs or maybe more. COMPLEX is a more complex algorithm so is a little slower compared to the SIMPLE, but can also still return queries in < 100ms on a 10m wikipedia index (complete index). COMPLEX is more accurate and should be able to give you the best rankings as compared to SIMPLE.

RankingAlgorithm can be used in two modes, Document mode (default) and Product mode. The scoring changes with the mode. In Document mode, documents are matched for relevancy while in Product mode, documents are matched for term occurence. Document mode is useful for matching text, html, rich text pdf/word, books, faq, forums discussions, etc. Product mode is useful for small text as in Retail/ecommerce product matches, etc.

 
 Programmtic:
  rq.setMode(RankingQuery.MODE_DOCUMENT);
  
  Property:
  To change MODE, start application with -Dmode=document, 
  for product, -Dmode=document
 

You can also set an attribute, scan to fast/medium/full scan. Fast is the default and the fastest while full scan is the most accurate but also slow, and takes lots of memory.

SIMPLE is also very good and may be well suited than COMPLEX for some type of queries.

 Programmtic:
  rq.setAlgorithm(RankingQuery.ALGORITHM_COMPLEX);
   
  Property: 
  To enable SIMPLE, start application with -Dalgorithm=SIMPLE, 
  for COMPLEX, -Dalgorithm=COMPLEX	 
 
 

You will need to have the Apache Lucene 3.x in the class path. At RankingQuery instantiation a Lucene IndexSearcher or IndexReader object is needed as RankingQuery uses the IndexReader to read the documents from the Index. See examples below ...

(Note: Lucene is a trademark of Apache Software Foundation)

 Example 1:
 		RankingQuery rq = new RankingQuery(); 
 		IndexSearcher is = new IndexSearcher(index);
 		StandardAnalyzer analyzer = new StandardAnalyzer();
		QueryParser parser = new QueryParser(field, analyzer);
		Query query = parser.parse(searchterms);
 		RankingHits rh = rq.search(query, is); //is = Lucene IndexSearcher object
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
 
 Example 2: [ Much faster and uses very little memory, scales upto 100m docs ]
 		IndexReader reader = IndexReader.open(FSDirectory.open(new File(index))); 
		RankingQuery rq = new RankingQuery();			
		StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
		QueryParser parser = new QueryParser(Version.LUCENE_30, field, analyzer);
		Query query = parser.parse(searchterms);
		TopScoreDocCollector tdc = TopScoreDocCollector.create(1000, true);
		rq.search(query, null, reader, tdc); //is = Lucene IndexSearcher object
		int hits = tdc.getTotalHits();
		ScoreDoc sda[] = null;
		if (hits > 0) {
			sda = tdc.topDocs().scoreDocs;
		}
		System.out.println("num hits=" + hits + "--no docs=" + reader.maxDoc()); 
		for (int i=0; i<hits && i<10; i++) {
			ScoreDoc sd = sda[i];
			System.out.println("i=" + i + "--" + sd.score + "--docid=" + sd.doc + "--doc=" + reader.document(sd.doc).get(title) );
		}
		reader.close();
  
 

Author:

Nagendra Nagarajayya

See Also:

RankingHits, RankingScore, TopScoreDocCollector

Field Summary

static int	ALGORITHM_COMPLEX
static int	ALGORITHM_SIMPLE
static int	AND
static int	AND_OR
static int	MODE_DOCUMENT
static int	MODE_PRODUCT
static int	OR
static int	SCAN_FAST
static int	SCAN_FULL
static int	SCAN_MEDIUM

Constructor Summary

RankingQuery()

RankingQuery(org.apache.lucene.index.IndexReader reader)
Constructor to create a RankingQuery object.

RankingQuery(org.apache.lucene.search.IndexSearcher is)
Constructor to create a RankingQuery object.

RankingQuery(java.lang.String indexPath)
Constructor to create a RankingQuery object.

Method Summary

void	addToLowerBoostSet(java.lang.String keywords) Experimental, can change
void	close() Closes the IndexReader objects opened.
org.apache.lucene.document.Document	doc(int docid) Similar to IndexSearcher doc(id), returns a Lucene Document object
int	getAlgorithm()
int	getAndOr()
int	getMode()
int	getScan()
RankingHits	search(org.apache.lucene.search.Query query) Search a Lucene index for terms in the query.
RankingHits	search(org.apache.lucene.search.Query query, org.apache.lucene.search.Filter filter, org.apache.lucene.index.IndexReader ir, org.apache.lucene.search.Collector collector) Search a Lucene index for terms in the query.
RankingHits	search(org.apache.lucene.search.Query query, org.apache.lucene.index.IndexReader r) Search a Lucene index for terms in the query.
RankingHits	search(org.apache.lucene.search.Query query, org.apache.lucene.search.IndexSearcher is) Search a Lucene index for terms in the query.
RankingHits	search(java.lang.String field, java.lang.String searchTerms) Search a Lucene index for terms in the query.
RankingHits	search(org.apache.lucene.search.Weight weight, org.apache.lucene.search.Filter filter, org.apache.lucene.index.IndexReader ir, org.apache.lucene.search.Collector collector) Similar to Lucene search.
void	setAlgorithm(int type) Set algorithm, SIMPLE or COMPLEX.
void	setAndOr(int type) Set And Or or AndOr combinations to get at the results.
void	setMode(int type) Set mode, Document or Product mode.
void	setScan(int scan) Used along with mode on how to scan a document.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ALGORITHM_COMPLEX

public static final int ALGORITHM_COMPLEX

See Also:

Constant Field Values

ALGORITHM_SIMPLE

public static final int ALGORITHM_SIMPLE

See Also:

Constant Field Values

MODE_PRODUCT

public static final int MODE_PRODUCT

See Also:

Constant Field Values

MODE_DOCUMENT

public static final int MODE_DOCUMENT

See Also:

Constant Field Values

SCAN_FAST

public static final int SCAN_FAST

See Also:

Constant Field Values

SCAN_MEDIUM

public static final int SCAN_MEDIUM

See Also:

Constant Field Values

SCAN_FULL

public static final int SCAN_FULL

See Also:

Constant Field Values

AND_OR

public static final int AND_OR

See Also:

Constant Field Values

AND

public static final int AND

See Also:

Constant Field Values

OR

public static final int OR

See Also:

Constant Field Values

Constructor Detail

RankingQuery

public RankingQuery(org.apache.lucene.index.IndexReader reader)

Constructor to create a RankingQuery object.

Parameters:

reader - Lucene InndexReader object.

RankingQuery

public RankingQuery(org.apache.lucene.search.IndexSearcher is)

Constructor to create a RankingQuery object.

Parameters:

is - Lucene IndexSearcher object.

RankingQuery

public RankingQuery(java.lang.String indexPath)
             throws java.lang.Throwable

Constructor to create a RankingQuery object. An Lucene IndexReader object is created to read the index in the indexPath.

Parameters:

indexPath - to a Lucene index.

Throws:

java.lang.Throwable

RankingQuery

public RankingQuery()

Method Detail

setScan

public void setScan(int scan)

Used along with mode on how to scan a document. Scan speed can be, FAST/MEDIUM/FULL. Default is FAST

Fast is the default and the fastest while a full scan is the most accurate but also slow, and takes lots of memory. Medium is in between.

 
 Programmtic:
  rq.setScan(RankingQuery.SCAN_FAST);
  
  Property:
  To change SCAN, start application with -Dscan=fast, 
    for product, -Dscan=product
 

Parameters:

scan - Valid values are RankingQuery.SCAN_FAST, RankingQuery.SCAN_MEDIUM, RankingQuery.SCAN_FULL

getScan

public int getScan()

Returns:

RankingQuery.SCAN_FAST, RankingQuery.SCAN_MEDIUM, RankingQuery.SCAN_FULL

setMode

public void setMode(int type)

Set mode, Document or Product mode. Default is Document mode

RankingAlgorithm can be used in two modes, Document mode (default) and Product mode. The scoring changes with the mode. In Document mode, documents are matched for relevancy while in Product mode, documents are matched for term occurence. Document mode is useful for matching text, html, rich text pdf/word, books, faq, forums discussions, etc. Product mode is useful for small text as in Retail/ecommerce product matches, etc.

 
 Programmtic:
  rq.setMode(RankingQuery.MODE_DOCUMENT);
  
  Property:
  To change MODE, start application with -Dmode=document, 
    for product, -Dmode=document
 

Parameters:

type - Valid values are RankingQuery.MODE_DOCUMENT or RankingQuery.MODE_PRODUCT

getMode

public int getMode()

Returns:

RankingQuery.MODE_DOCUMENT or RankingQuery.MODE_PRODUCT

setAlgorithm

public void setAlgorithm(int type)

Set algorithm, SIMPLE or COMPLEX. Default is COMPLEX mode

Two Algorithm are available SIMPLE and COMPLEX. SIMPLE is a very fast algorithm and returns queries in <100ms on a 10m wikipedia index (complete index). It can also scale to 100m docs or maybe more. COMPLEX is a more complex algorithm so is a little slower compared to the SIMPLE, but can still return queries in < 200ms on a 10m wikipedia index (complete index). COMPLEX is more accurate and should be able to give you the best rankings as compared to SIMPLE.

You can also set an attribute, scan to fast/medium/full scan. Fast is the default and the fastest while full scan is the most accurate but also slow, and takes lots of memory.

SIMPLE is also very good and may be well suited than COMPLEX for some type of queries.

 Programmtic:
  rq.setAlgorithm(RankingQuery.ALGORITHM_COMPLEX);
   
  Property: 
  To enable SIMPLE, start application with -Dalgorithm=SIMPLE, 
  for COMPLEX, -Dalgorithm=COMPLEX	 
 
 

Parameters:

type - Valid values are RankingQuery.ALGORITHM_SIMPLE or RankingQuery.ALGORITHM_COMPLEX

getAlgorithm

public int getAlgorithm()

Returns:

RankingQuery.ALGORITHM_SIMPLE or RankingQuery.ALGORITHM_COMPLEX

setAndOr

public void setAndOr(int type)

Set And Or or AndOr combinations to get at the results. AND is 100%, OR is 0%, AND_OR is 50% relevancy. Similar to mm parameter in Solr Default is Or

Parameters:

type - Valid values are RankingQuery.AND or RankingQuery.AND_OR or RankingQuery.OR. One can also set this to any value between 0 and 100 as needed.

getAndOr

public int getAndOr()

Returns:

a value betewen 1 and 100. 0% = RankingQuery.OR, 50% = RankingQuery.AND or RankingQuery.AND_OR or RankingQuery.AND

close

public void close()
           throws java.lang.Throwable

Closes the IndexReader objects opened.

Note: 1. IndexReader object is closed only if the constructor RankingQuery(indexpath) is used. If the IndexSearcher or IndexReader is passed explicitly as in RankingQuery(IndexSearcher is) or RankingQuery(IndexReader ir) it is not closed. 2. Make sure close is called the last since RankingHits has a reference to an IndexReader that will also get closed in IndexReader is closed. See SimpleExample.java for usage.

Throws:

java.lang.Throwable

See Also:

RankingQuery(String)

doc

public org.apache.lucene.document.Document doc(int docid)
                                        throws java.lang.Throwable

Similar to IndexSearcher doc(id), returns a Lucene Document object

Parameters:

docid - Lucene document id

Returns:

Document object

Throws:

java.lang.Throwable

search

public RankingHits search(org.apache.lucene.search.Query query)
                   throws java.lang.Throwable

Search a Lucene index for terms in the query. RankingQuery needs to have been instantiated with Lucene IndexReader or IndexSearcher objects.

Parameters:

query - A Lucene query object

Returns:

A list of hits matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits

search

public RankingHits search(java.lang.String field,
                          java.lang.String searchTerms)
                   throws java.lang.Throwable

Search a Lucene index for terms in the query. RankingQuery needs to have been instantiated with path to a Lucene index.

 Example:
 		RankingQuery rq = new RankingQuery("/lucene/index/perl"); 
 		RankingHits rh = rq.search("search_field", "text"); 
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
  
 

Parameters:

field - to search

searchTerms - search terms

Returns:

RankingHits object, a list of documents matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits

search

public RankingHits search(org.apache.lucene.search.Query query,
                          org.apache.lucene.search.IndexSearcher is)
                   throws java.lang.Throwable

Search a Lucene index for terms in the query. Needs a Lucene Query and IndexSearcher objects to access the index

 Example 1:
 		RankingQuery rq = new RankingQuery(); 
 		RankingHits rh = rq.search(query, is); //is = Lucene IndexSearcher object
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
 
 Example 2: [ Much faster and uses very little memory, scales upto 100m docs ]
 		RankingQuery rq = new RankingQuery();  * 		
 		StandardAnalyzer analyzer = new StandardAnalyzer();
		QueryParser parser = new QueryParser(field, analyzer);
		Query query = parser.parse(searchterms);
		TopScoreDocCollector tdc = new TopScoreDocCollector();
 		rq.search(query, null, indexreader, tdc); //is = Lucene IndexSearcher object
 		int hits = tdc.getTotalHits();
		System.out.println("num hits=" + hits + "--no docs=" + indexreader.maxDoc()); 
		for (int i=0; i<hits && i<10; i++) {
			ScoreDoc sd = tdc.topDocs().scoreDocs[i]
			System.out.println("i=" + i + "--" + sd.score(i) + "--docid=" + sd.doc + "--doc=" + indexreader.document(sd.doc).get(title) );
		}
    
 

Parameters:

query - Lucene query object

is - is a Lucene IndexSearcher object.

Returns:

RankingHits object, a list of documents matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits, RankingScore

search

public RankingHits search(org.apache.lucene.search.Query query,
                          org.apache.lucene.index.IndexReader r)
                   throws java.lang.Throwable

Search a Lucene index for terms in the query. Needs a Query object and an IndexReader to access the index

 Example:
 		RankingQuery rq = new RankingQuery(); 
 		RankingHits rh = rq.search(query, is); //is = Lucene IndexSearcher object
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
  
 

Parameters:

query - Lucene query object

r - Lucene IndexSearcher object

Returns:

RankingHits object, a list of documents matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits

search

public RankingHits search(org.apache.lucene.search.Query query,
                          org.apache.lucene.search.Filter filter,
                          org.apache.lucene.index.IndexReader ir,
                          org.apache.lucene.search.Collector collector)
                   throws java.lang.Throwable

Search a Lucene index for terms in the query. Uses the Query object and IndexReader to access the index

A Lucene Filter can be used to limit returned results, while a Lucene Collector (like TopScoreDocCollector) can be used to collect the relevant docs.

 Example:
 		RankingQuery rq = new RankingQuery(); 
 		RankingHits rh = rq.search(query, filter, ir, collector); //is = Lucene IndexSearcher object
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
 
 Example 2: [ Much faster and uses very little memory, scales upto 100m docs ]
 		IndexReader reader = IndexReader.open(FSDirectory.open(new File(index))); 
		RankingQuery rq = new RankingQuery();			
		StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
		QueryParser parser = new QueryParser(Version.LUCENE_30, field, analyzer);
		Query query = parser.parse(searchterms);
		TopScoreDocCollector tdc = TopScoreDocCollector.create(1000, true);
		rq.search(query, null, reader, tdc); //is = Lucene IndexSearcher object
		int hits = tdc.getTotalHits();
		ScoreDoc sda[] = null;
		if (hits > 0) {
			sda = tdc.topDocs().scoreDocs;
		}
		System.out.println("num hits=" + hits + "--no docs=" + reader.maxDoc()); 
		for (int i=0; i<hits && i<10; i++) {
			ScoreDoc sd = sda[i];
			System.out.println("i=" + i + "--" + sd.score + "--docid=" + sd.doc + "--doc=" + reader.document(sd.doc).get(title) );
		}
		reader.close();
    
 

Parameters:

query - Lucene query object

filter - is a Lucene filter object

ir - is a Lucene IndexReader object.

collector - to collect returned results

Returns:

RankingHits object, a list of documents matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits, RankingScore

search

public RankingHits search(org.apache.lucene.search.Weight weight,
                          org.apache.lucene.search.Filter filter,
                          org.apache.lucene.index.IndexReader ir,
                          org.apache.lucene.search.Collector collector)
                   throws java.lang.Throwable

Similar to Lucene search. Uses the Query object and IndexReader to access the index

A Lucene Filter can be used to limit returned results, while a Lucene Collector (like TopScoreDocCollector) can be used to collect the relevant docs.

 Example:
 		RankingQuery rq = new RankingQuery(); 
 		RankingHits rh = rq.search(weight, filter, ir, collector); //is = Lucene IndexSearcher object
		System.out.println("num hits=" + rh.getNumHits() + "--no docs=" + is.maxDoc()); 
		for (int i=0; i<rh.getNumHits() && i<10; i++) {
			System.out.println("i=" + i + "--" + rh.score(i) + "--docid=" + rh.docid(i) + "--doc=" + rh.doc(i).get(title) );
		}
 
 
 Example 2: [ Much faster and uses very little memory, scales upto 100m docs ]
 		IndexReader reader = IndexReader.open(FSDirectory.open(new File(index))); 
		RankingQuery rq = new RankingQuery();			
		StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
		QueryParser parser = new QueryParser(Version.LUCENE_30, field, analyzer);
		Query query = parser.parse(searchterms);
		TopScoreDocCollector tdc = TopScoreDocCollector.create(1000, true);
		rq.search(query, null, reader, tdc); //is = Lucene IndexSearcher object
		int hits = tdc.getTotalHits();
		ScoreDoc sda[] = null;
		if (hits > 0) {
			sda = tdc.topDocs().scoreDocs;
		}
		System.out.println("num hits=" + hits + "--no docs=" + reader.maxDoc()); 
		for (int i=0; i<hits && i<10; i++) {
			ScoreDoc sd = sda[i];
			System.out.println("i=" + i + "--" + sd.score + "--docid=" + sd.doc + "--doc=" + reader.document(sd.doc).get(title) );
		}
		reader.close();
    
 

Parameters:

weight - Lucene weight object

filter - is a Lucene filter object

ir - is a Lucene IndexReader object.

collector - to collect returned results

Returns:

RankingHits object, a list of documents matching the search terms

Throws:

java.lang.Throwable

See Also:

RankingHits, RankingScore

addToLowerBoostSet

public void addToLowerBoostSet(java.lang.String keywords)

Experimental, can change