![NLP and Tokenization
¢ How does computer represent text?
¢ String (or Char[ ] or Byte[ ] )
＊ "Abc"
＊ "Hello World"](https://image.slidesharecdn.com/introductiontojapanesetokenizer-201206054416/85/Introduction-to-japanese-tokenizer-6-320.jpg)
![NLP and Tokenization
¢ Tokenization / Segmentation
¢ The ?rst step to solve NLP problems is usually
identifying words from the string
＄ Input: string, char[ ] (or byte[ ])
＄ Output: a list of meaningful words (or tokens)](https://image.slidesharecdn.com/introductiontojapanesetokenizer-201206054416/85/Introduction-to-japanese-tokenizer-10-320.jpg)
![NLP and Tokenization
"Biden is projected winner in Michigan, Wisconsin as
tense nation watch final tally".split(/W+/)
> ["Biden", "is", "projected", "winner", "in", ...]](https://image.slidesharecdn.com/introductiontojapanesetokenizer-201206054416/85/Introduction-to-japanese-tokenizer-11-320.jpg)
![Japanese Tokenization
¢ Use prior Japanese knowledge (Dictionary)
＄ が, に, ´, 箆, 巒, ´, バイデン
¢ Consider the context and combination of characters
¢ Consider the likelihood
＄ e.g. |奨脅 => [|奨, 脅], or [|, 奨脅]](https://image.slidesharecdn.com/introductiontojapanesetokenizer-201206054416/85/Introduction-to-japanese-tokenizer-15-320.jpg)
More Related Content
What's hot (20)
Similar to Introduction to japanese tokenizer (8)
More from Fangda Wang (11)
![[WWCode] How aware are you of your deciding model?](https://cdn.slidesharecdn.com/ss_thumbnails/wwcodehowawareareyouofyourdecidingmodelaudienceinputincluded-230828022746-2d6216d5-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
Introduction to japanese tokenizer
- 1. Introduction to Japanese tokenizers WebHack 2020-11-10 by Wanasit T.
- 2. About Me ¢ Github: @wanasit ＄ Text / NLP projects ¢ Manager, Software Engineer @ Indeed ＄ Search Quality (Metadata) team ＄ Work on NLP problems for Jobs / Resumes
- 3. Disclaimer 1. This talk NOT related to any of Indeed¨s technology 2. I¨m not a Japanese (or a native-speaker) ＄ But I built a Japanese tokenizer on my free time
- 4. Today Topics ¢ NLP and Tokenization (for Japanese) ¢ Lattice-based Tokenizers (MeCab -style tokenizers) ¢ How it works ＄ Dictionary ＄ Tokenization
- 6. NLP and Tokenization ¢ How does computer represent text? ¢ String (or Char[ ] or Byte[ ] ) ＊ "Abc" ＊ "Hello World"
- 7. NLP and Tokenization "Biden is projected winner in Michigan, Wisconsin as tense nation watch final tally" Source: NBC News
- 8. NLP and Tokenization "Biden is projected winner in Michigan, Wisconsin as tense nation watch final tally" ¢ What¨s the topic? ¢ Who is winning? where? Source: NBC News
- 9. NLP and Tokenization "Biden is projected winner in Michigan, Wisconsin as tense nation watch final tally" ¢ What¨s the topic? ¢ Who is winning? where? Source: NBC News
- 10. NLP and Tokenization ¢ Tokenization / Segmentation ¢ The ?rst step to solve NLP problems is usually identifying words from the string ＄ Input: string, char[ ] (or byte[ ]) ＄ Output: a list of meaningful words (or tokens)
- 11. NLP and Tokenization "Biden is projected winner in Michigan, Wisconsin as tense nation watch final tally".split(/W+/) > ["Biden", "is", "projected", "winner", "in", ...]
- 14. Japanese Tokenization "バイデン箆がミシガン巒拈、寄yIにむけ＾藍返" ¢ No punctuations ¢ Q: How do you split this into words? Source: TBS News
- 15. Japanese Tokenization ¢ Use prior Japanese knowledge (Dictionary) ＄ が, に, ´, 箆, 巒, ´, バイデン ¢ Consider the context and combination of characters ¢ Consider the likelihood ＄ e.g. |奨脅 => [|奨, 脅], or [|, 奨脅]
- 17. Lattice-based Tokenizers ¢ aka. MeCab -based tokenizer (or Viterbi tokenizer) ¢ How: ＄ From a Dictionary (required) ＄ Build a Lattice (or a graph) from surface dictionary terms ＄ Run Viterbi algorithm to ?nd the best connected path
- 18. Lattice-Based Tokenizers ¢ Most tokenizers are MeCab (C/C++)¨s re-implementation on different platforms: ＄ Kuromoji, Sudachi (Java), Kotori (Kotlin) ＄ Janome, SudachiPy (Python) ＄ Kagome (Go) ＄ ...
- 19. Non- Lattice-Based Tokenizers ¢ Is Lattice-based the only approach? ¢ Mostly yes, but there are also: ＄ Juman++, Nagisa (RNN) ＄ SentencePiece (Unsupervised, used in BERT) ¢ Out-of-scope of this presentation
- 20. How it works > Dictionary
- 21. Dictionary ¢ Lattice-based tokenizers need dictionary ＄ To recognize prede?ned terms and grammar ¢ Dictionaries are often can be downloaded as Plugins e.g. ＄ $ brew install mecab ＄ $ brew install mecab-ipadic
- 22. Dictionary ¢ Recommended beginner dictionary is MeCab¨s IPADIC ¢ Available from this website
- 23. Dictionary - Term Table / Lexicon / CSV ?les Surface Form Context ID (left) Context ID (right) Cost Type Form Spelling ... |奨 1293 1293 3003 兆~ (place) - トウキョウ ... 奨脅 1293 1293 2135 兆~ (place) - キョウト ... |奨V 1293 1293 8676 兆~ (place) - ヒガシキョウ ヅカ ... 佩く 992 992 8852 嘖~ (v) 児云侘 イク ... 佩か 1002 1002 7754 嘖~ (v) 隆隼侘 イカ ... いく 992 992 9672 嘖~ (v) 児云侘 イク ...
- 24. Dictionary - Term Table ¢ Surface Form: How the term should appear in the string ¢ Context ID (left/right): ID used for connecting terms together (see. later) ¢ Cost: How commonly used the term ＄ The more the cost, the less common or less likely
- 25. Dictionary - Connection Table / Connection Cost Context ID (from) Context ID (to) Cost ... ... 992 992 3003 992 993 2135 ... ... 992 1293 -1000 992 1294 -1000 ... ... ¢ Connection cost between type of terms. ¢ The lower, the more likely ¢ e.g. ¢ 992 (v-ru) then 992 (v-ru) ＄ Cost = 3000 (unlikely) ¢ 992 (v-ru) then 1294 (noun) ＄ Cost = -1000 (likely)
- 26. Dictionary - Term Table Term table size: ¢ Kotori (default) ~380,000 terms (3.7 MB) ¢ MeCab-IPADict ~400,000 terms (12.2 MB) ¢ Sudachi - Small ~750,000 terms (39.8 MB) ¢ Sudachi - Full ~2,800,000 terms (121 MB)
- 27. Dictionary - Term Table Term table size: ¢ Kotori (default) ~380,000 terms (3.7 MB) ¢ MeCab-IPADict ~400,000 terms (12.2 MB) ¢ Sudachi - Small ~750,000 terms (39.8 MB) ¢ Sudachi - Full ~2,800,000 terms (121 MB) ＄ Include term like: "c(```)ノ"
- 28. Dictionary - Term Table ¢ What about words not in the table? ＄ e.g. "ワナシット タナキットルンアン" ＄ ＾Unknown-Term Extraction￣ Problem ＄ Typically, some heuristic rules ＊ e.g. if there are consecutive katana, it¨s a Noun. ¢ Out-of-scope of this presentation
- 29. How it works > Tokenization
- 30. Lattice-Based Tokenization Given: ¢ The Dictionary ¢ Input:"|奨脅に廖む" Tokenizer: 1. Find all terms in the input and build a lattice 2. Find the minimum cost path through the lattice
- 31. Step 1: Finding all terms
- 32. Step 1: Finding all terms ¢ For each index i-th ＄ ?nd all terms in dictionary starting at i-th location ¢ String / Pattern Matching problem ＄ Require e?cient lookup data structure for the dictionary ＄ e.g. Trie, Finite-State-Transidual
- 33. Step 2: Finding minimum cost ¢ Viterbi Algorithm (Dynamic Programing) ¢ For each node from the left to right ＄ Find the minimum cost path leading to that node ＄ Reuse the selected path when consider the following nodes
- 34. Summary
- 35. Introduction to Japanese Tokenizers ¢ Introduction to NLP and Tokenization ¢ Lattice-based tokenizers (MeCab and others) ＄ Dictionary ＊ Term table, Connection Cost, ... ＄ Tokenization Algorithms ＊ Pattern Matching, Viterbi Algorithm, ...
- 36. Learn more: ¢ Kotori (on Github), A Japanese tokenizer written in Kotlin ＄ Small and performant (fastest among JVM-based) ＄ Support multiple dictionary formats ¢ Article: How Japanese Tokenizers Work (by Wanasit) ¢ Article: 晩云Z侘B殆盾裂のY箸鰔く (by Cookpad Developer) ¢ Book: 徭隼冱ZI尖の児A (by Manabu Okumura)