WorldCat Discovery release notes, Thai language search and sort

Normalization

We do not apply normalization to any Thai indexes. We always treat all Thai vowel symbols and diacritics as significant and never ignore them.

Exception: Characters with tone marks: composed/decomposed

We index Thai characters that have tone marks in both composed and decomposed forms.

Example:

When entered in composed form in a search term, the character will match both composed and decomposed forms in the index and vice versa: if searched in decomposed form it will match both the decomposed and composed form.

Tokenization

Because Thai script phrases are written without spaces between words, to recognize and index individual Thai words, we apply tokenization to build all Thai word indexes and to parse word index queries. We do not apply tokenization for phrase indexes or phrase index queries.

Indexing of individual Thai words enables word index searching whereby records containing the query terms anywhere in the appropriate indexed fields are retrieved.

Example

Common words

Referring to the list of common Standard Thai words below, rather than treating them as stop words whereby we would ignore them for indexing and matching, we combine them with adjacent words when we build word indexes and parse word index queries.

When a common Thai word is ignored (treated as a stop word), an adjacent word that remains can have a different meaning than when combined with/adjacent to a common word. This different meaning can lead to the retrieval of irrelevant records. In cases where the meaning of a word would have changed had we removed the adjacent common word, combining it with the adjacent common word helps to disambiguate its meaning, providing greater search precision by reducing retrieval of irrelevant records.

We apply the above processing of common words when building and searching the following indexes:

• se: Series
• ti: Title
• kw: Keyword

Example

Common word treated as a stop word

• Query ti:มาตราการ (measures/procedure)
• Tokenized into มาตรา and การ
• The common word การ is removed as a stop word leaving only มาตรา
• มาตรา has a different meaning (section or clause of law) from มาตราการ (measures/procedure) and therefore retrieves records that are not relevant to the query ti:มาตราการ

Common word combined with an adjacent word

• Query ti:มาตราการ (measures/procedure)
• Tokenized into มาตรา_การ (because การ is defined as a common word)
• Records with title fields containing มาตราการ
• Titles are tokenized into มาตรา มาตรา_การ การ
• Only records containing มาตราการ are retrieved.

Thai common word list