Morphological Analysis

Our morphological analyzer generates root forms (lemmas) of word tokens. It is a rule-based analyzer inspired by the WordNet morphy although it uses a larger dictionary gathered from various sources and more advanced heuristics. It also normalizes numbers, redundant punctuation, hyperlinks, etc (see the examples below), which is found to be useful for several NLP tasks (see toSimplifiedForm).

Normalization

Ordinals and cardinals are normalized to #ord# and #crd#, respectively:

1st         CD    #ord#
12nd        CD    #ord#
23rd        CD    #ord#
34th        CD    #ord#
first       CD    #ord#
third       CD    #ord#
fourth      CD    #ord#
zero        CD    #crd#
ten         CD    #crd#
eleven      CD    #crd#
fourteen    CD    #crd#

Numeric expressions are normalized to 0:

10%                    XX    0
$10                    XX    0
A.01                   XX    a.0
A:01                   XX    a:0
A/01                   XX    a/0
.01                    XX    0
12.34                  XX    0
12,34,56               XX    0
12-34-56               XX    0
12/34/46               XX    0
$10.23,45:67-89/10%    XX    0

Redundant punctuation characters are collapsed:

.!?-*=~,                                                    XX    .!?-*=~,
..!!??--**==~~,,                                            XX    ..!!??--**==~~,,
...!!!???---***===[[user:jdchoi77]],,,                      XX    ..!!??--**==~~,,
....!!!!????----****====[[user:jdchoi77|1384441338]],,,,    XX    ..!!??--**==~~,,

Hyperlinks are normalized to #hlink#:

http://www.google.com         XX    #hlink#
www.google.com                XX    #hlink#
mailto:somebody@google.com    XX    #hlink#
some-body@google+.com         XX    #hlink#

API

EnglishLemmatizerTest shows how the anlayzer can be used in APIs.