Interesting Post About Query Parser

• Paper:

The Magic and Wonder of Query Parsing – Search Technologies

More info:Sometimes it is necessary to apply a custom filter to an existing SQL query (to perform search … query, you should first create … parser will work faster if we …

URL: http://www.searchtechnologies.com/search-query-parsing

• Tool:

OpenNLP

The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text.
It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution. These tasks are usually required to build more advanced text processing services. OpenNLP also includes maximum entropy and perceptron based machine learning.

• My understandings:

1. Different users speak different query languages; different kinds of search engines should have different documents or regulations.

2. Not only a string of simple tokens, how can we understand some operations or special signs, such as "+/-", "NOT", "OR", and so on?

3. How can you set up the relationships between "chair" and "chairs", between "mouse" and "mice", and so on?

4. Spell checker should put before query parser, which can remove some dirty data.

5. Relevancy ranking: Documents which contain search terms in the title or abstract may be considered to be more relevant; Query parsers can boost documents which contain all of the terms close together (proximity weighting) or boost documents from friendly web sites while reducing documents from un-friendly sites.

Jazzy-based Spell Checker

Jazzy is a set of APIs that allow you to add spell checking functionality to Java Applications easily.

For a misspelled word, Jazzy can provide with some corrected words for selection.

Additional task is to judge which word is better. We have two schemes to do that: 1. Edit distance; 2. Input frequency or context. Context information is based on Jazzy dictionary. However, for some specific search engine, common dictionary is not enough. For example, we should add some merchant brand names into the dictionary for Amazon's search engine.

There are still two problems here:

1. How do we customize our own dictionary? How do we confirm the validity of the data source for building the dictionary?
2. How do you balance the results of edit distance and context?