About

In Natural Language processing, Tokens can be things like:

• word,
• numbers,
• acronyms,
• word-roots
• or fixed-length character strings.

A token is the result of parsing (tokenization) the document down to the atomic elements generally of a language.

The token are then searchable.

See:

• Part
• Term

Articles Related