Table of Contents
Search Analyzers
Table of Contents
Search Analyzers define how text is processed or manipulated. These analyzers give you control over how your text data is used in the search. The goal is to standardize text, for example, lowercasing or converting special characters (diacritics) or handling of singular/plural in translations (e.g., men, man). Search Analyzers are available for string and translated string fields.
Simple search analyzer
access in search queries: simple
The simple search analyzer is a custom Fotoware Alto implementation that does not use Elasticsearch defaults. The custom analyzer uses a regex:
- Regex
*/"(\[^\\p\{L\}\\d\]+)|(?<=\\D)(?=\\d)|(?<=\\d)(?=\\D)|(?<=\[\\p\{L\}&&\[^\\p\{Lu\}\]\])(?=\\p\{Lu\})|(?<=\\p\{Lu\})(?=\\p\{Lu\}\[\\p\{L\}&&\[^\\p\{Lu\}\]\])"/* - Outcome:
- Lowercase / Uppercase
- Digit / non-digit
- Stemming
- HTML Strip
- Examples
- Fotoware Alto = Fotoware Alto, fotoware alto
- Case Study = Case, Study, case, study
If you want to test the simple search analyzer, you can check your terms in a regex tester to see the outcome.
- Open a regex checker.
- Add your term as a test string.
- Check the outcome.
Path hierarchy analyzer
access in search queries: pathHierarchy
- The path hierarchy analyzer will:
- Take a path found in a field (fotowarealto\platform\manual) and delimit the individual terms
- Example
- fotoarealto\platform\manual = fotowarealto\platform\manual, fotoarealto\platform, manual
Products/Family/Industry = Products/Family, Products, Products/Family/Industry
You should only configure this analyzer if it is used via the API. The simple search in Fotoware Alto escapes Special Characters, and therefore, you won't find assets when searching for some of the tokens generated by this analyzer.
An example can be found in Elastic Search documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html
Edge Ngram analyzer
access in search queries: edgeNGram
This tokenizer is very similar to nGram but only keeps n-grams that start at the beginning of a token. Settings allow us to define min and max grams created on indexing and token_chars, which are character classes to keep in the tokens. Elasticsearch splits on characters that don't belong to any of these classes.
Examples are in Elastic Search documentation:
Edge n-gram tokenizer | Elasticsearch Guide [8.17] | Elastic
Ngram analyzer
access in search queries: ngram
The starting point for exact substring matches was n-gram tokenization, which indexes all substrings of length n. The drawback of n-gram tokenization is the large amount of disk space it consumes.
Best practice:
- Use ngram only if required - use carefully and not for every string
Settings allow us to define min and max grams created on indexing and token_chars, which are character classes to keep in the tokens. Elasticsearch splits on characters that don't belong to any of these classes.
Example: Search "Raven"
- NGrams (splits term into tokens with one character):
- Rav
- Rave
- Raven
- ave
- aven
- Ven
- ...
Example: Search "Pegasus"
- NGrams (splits term into tokens with one character):
- Pegasus
- Degas
Examples are in Elastic Search documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html
No diacritics analyzer
access in search queries: no-diacritics
The no diacritics analyzer:
- only works for text fields
- strip diacritic characters, so when the text value is: Kovačić Mateo, you can search for “Kovačić Mateo” or “Kovacic Mateo”.
An example can be found in Elastic Search documentation: ASCII folding token filter | Elasticsearch Guide [8.17] | Elastic
Keyword lowercase filter analyzer
no access in search queries, only available in filters
The filter analyzer converts multiple keywords with various spellings to lowercase but does not tokenize words; e.g., jpg and JPG will be converted into a single keyword: "jpg" for admins to use in filters and search.
This filter only allows exact matches from non-translated values, ignoring letter casing, and does not tokenize (split) any words.
Language analyzer
access in search queries: language
There are several language analyzers available for Elastic Search. Language analyzers prevent stemming from language-specific values and language-specific stopwords.
Language analyzers | Elasticsearch Guide [8.17] | Elastic
The current implementation uses the default Elasticsearch language analyzers listed in the link. We are using the default stop words and stemming rules, without any customization.
Useful ElasticSearch documentation
Simple Analyzer: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simple-analyzer.html
No Diacritics: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-asciifolding-tokenfilter.html
Path Hierarchy: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html
Language: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html
NGram: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html
EdgeNgram: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/analysis-edgengram-tokenizer.html
The query can be adjusted to support advanced search queries on analyzed fields, taking the analyzer into account.
Search: Find Items by Product Codes (Solutions)
Search: Find Items with specific values in text fields (Solutions)
Search: Find Items with values in translated fields. (Solutions)