suou.lex

suou.lex(text: str, table: Iterable[TokenSym], *, whitespace=False)

Return a text as a list of tokens, given a token table (iterable of TokenSym).

ilex() returns a generator; lex() returns a list.

table must be a result from symbol_table().