Tokenization
The process of breaking text into its smallest meaningful units, words, fragments, or characters, so a search engine can analyze and match it.
Definition
Definition
The engine breaks down the query and the indexed content into tokens. A token usually matches a word, but it can be a syllable, prefix, suffix or character depending on the strategy. The process affects both the search query and the indexed content. It's usually accompanied by: conversion to lowercase, removal of stopwords, and stemming/lemmatization (reducing verb forms to a common root).
What it's used for
What it's used for
Tokenization determines whether two strings are treated as the same or different by the search engine. "Running shoes" and "RUNNING SHOES" should be the same (handled by lowercasing during tokenization). "Women's" and "womens" should also match (handled by stripping possessives).
How a system tokenizes input directly affects whether typos, compound words, hyphenated terms, and special characters produce matches or misses.
Doofinder
Doofinder applies tokenization during both indexing and query processing, using morphological analysis appropriate to the configured language of the search engine.
The tokenization strategy varies by language setting, a search engine configured for English tokenizes differently from one configured for German, where compound words are common and need special handling.
Example
Case study
A user types "men's leather-soled oxford" into a shoe store's search. Tokenization breaks this into ["mens", "leather", "soled", "oxford"], removing the apostrophe and hyphen as token separators.
The same tokenization was applied to the catalog during indexing. Products with "oxford" in the title and "leather" in the material field match, even though the original query used forms that don't appear literally in any product description.