This is an abstract base class of Analyzers.
An Analyzer builds TokenStreams, which analyze text. It thus represents
a policy for extracting index terms from text.
Typical implementations first build a Tokenizer, which breaks the stream
of characters from the Reader into raw Tokens. One or more TokenFilters
may then be applied to the output of the Tokenizer.