com.saelist.stx.parser
Class Tokenizer

java.lang.Object
  com.saelist.stx.parser.Tokenizer

public class Tokenizer

extends java.lang.Object

Tokenizes free form structured text into @{link Token}s.

Field Summary

static Logger

logger

Constructor Summary

Tokenizer()

Method Summary

static java.util.List	assemble(java.util.List tokens) Reconstructs quoted text and removes unquoted whitespace.
static java.util.List	tokenize(java.lang.String text)
static java.lang.String	toString(java.util.List tokens)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

public static Logger logger

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

tokenize

public static java.util.List tokenize(java.lang.String text)

assemble

public static java.util.List assemble(java.util.List tokens)

Reconstructs quoted text and removes unquoted whitespace. E.g the three tokens "'", "def", "'" are joined in "'abc'"

toString

public static java.lang.String toString(java.util.List tokens)