Description
Text_Tokenizer defines the interface for a tokenizer class and the data model for a single token, usable when parsing text.
Download
Download at http://pear.sergiocarvalho.com/repository/Text_Tokenizer-1.0.0.tgz
Documentation at http://pear.sergiocarvalho.com/docs/Text_Tokenizer-1.0.0/
Browse source code at http://pear.sergiocarvalho.com/browse/Text_Tokenizer-1.0.0
Install with:
pear install sergiosgc/Text_Tokenizer