edu.northwestern.at.utils.corpuslinguistics
Class DefaultPretokenizer

java.lang.Object
  edu.northwestern.at.utils.corpuslinguistics.DefaultPretokenizer

All Implemented Interfaces:

Pretokenizer

public class DefaultPretokenizer
extends java.lang.Object
implements Pretokenizer

Prepare a string for tokenization.

Field Summary

static java.lang.String

PUNCTUATION

Constructor Summary

DefaultPretokenizer()

Method Summary

java.lang.String

pretokenize(java.lang.String line) Prepares a string for tokenization.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PUNCTUATION

public static final java.lang.String PUNCTUATION

See Also:

Constant Field Values

Constructor Detail

DefaultPretokenizer

public DefaultPretokenizer()

Method Detail

pretokenize

public java.lang.String pretokenize(java.lang.String line)

Description copied from interface: Pretokenizer

Prepares a string for tokenization.

Specified by:

pretokenize in interface Pretokenizer

Parameters:

line - The string to prepare for tokenization.

Returns:

The modified string ready for tokenization.