際際滷

際際滷Share a Scribd company logo

TechTalk - POI Tokenizing

Download as PPTX, PDF
Daesung Park
Daesung Park

Techinal Talk 螳 覦 伎. Python SOYNLP 襯 螳. 2022. 05.

1 of 21
Download to read offline
POI 覈豺 Tokenizing 郁規 螳覦Lab 覦
伎 (Tokenizing) 一伎襴 ろ 一危一 覲企ゼ 覲襦 蟆. 企語 豢豌 : https://magazine.hankyung.com/business/article/202010133532b
蟆 讌 襷 (Inverted Index) 企語 豢豌 : https://www.skyer9.pe.kr/wordpress/?p=1002
伎 譬襯 伎郁鍵襦 蟲覿 Time is money time is mone y 襯 螻ろ讌 覓語ル 狩 蟆郁骸襦 豬蠍 覓語螳 = BoW (Bag of Words) Money is time time is mone y
伎 譬襯 n-gram uni-gram(1-gram), bi-gram(2-gram), tri-gram(3-gram), 4-gram Time is money ti im me _i is s_ _m mo on e_ ne ey
伎 譬襯 n-gram uni-gram(1-gram), bi-gram(2-gram), tri-gram(3-gram), 4-gram Time is money ti im me _i is s_ _m mo on e_ ne ey mo on ne y_ _i is s_ _t ti ey im me Money is time
伎 譬襯 覿 (蠏 蟆讌 ) 瑚概 蟆暑 語 覈 (NNLM) " 覦郁 蟇語伎 讌 螳給" /覈 /覲伎＾ 覦/覈 +螳/蟆譟一 / +/郁屋企 蟇/ +伎/郁屋企 讌/覈 +/蟆譟一 螳/ +/企企 +給/企企
覿 覈語 伎郁鍵 る 豬 譟一, ろ, 蟲伎牡 豬 覯讌螳覦れ願 企蠏語 旧伎る 郁鍵螳 覲伎 ! 覓伎覦 譯殊瑚概 覓 豐る純
企觜 POI 覈豺 轟 伎郁鍵螳 . ( 覿 豬) 覓 讌щ. (n-gram 覿 豬) 觜訣螻 覦覲給 企れ 譟壱 殊煙(2語) 覦燕(覓伎狩一) 蟲螻牛 殊煙讌 殊煙朱
POI 覈豺 覿 蟆郁骸 Open Korean Text 覿蠍 蟆郁骸 殊煙(2語) 覦燕(覓伎狩一) 貊ろ語語 企誤碁企 殊煙朱 殊煙, , 2, 語 , 覦燕, 覓伎, 狩, 貊ろ語, , 企, , 企 殊煙, 觜 襦 覿襴讌 覈詩
企至 伎蟾 螳螻旧 : 讌覈 螳 : 讌覈 螻旧
POI 覈豺 覦覲給 願 觜覯蟆 煙 螳襴蟲焔覲 蠍誤 螻燕蟲 煙螻旧 企Π願概 螻狩豌危螻旧 觜覯蟆 煙ロ 企 蠍一朱 襷 讌 蟾..?
糾 蠍磯 譟一伎 螳 覩碁焔 覓語 (Out of vocabulry, OOV) 襯 企至 願屋讌? 語伎 企 覦
糾 蠍磯 伎 蠍一 覦覲 煙ロ れ襯 企 語 覓語襯 願屋
SOYNLP 讌 襯螻 觚豺 碁朱ゼ 伎 殊企襴 https://github.com/lovit/soynlp
SOYNLP 讌 襯 (cohesion probability, CP) 譯殊伎 覓語伎 蠍一朱 郁屋 蟷 譯 襯 觚豺 碁 (branching entropy, BE) 譯殊伎 覓語 る 煙ロ 蠍 覿れ 讌
ろ
ろ
ろ
ろ 蟆郁骸 ろ れ OKT 覿 蟆郁骸 SOYNLP SOYNLP 企誤碁企 企, , 企 企誤碁企 企語, 企瑚蟲 企, 碁企 碁企 企誤碁企 殊煙朱 殊煙, , 觜 殊煙碁, 殊煙螻糾骸 蟲 殊, 朱 殊煙碁, 殊煙螻糾骸 蟲, 殊煙 殊, 朱 殊煙碁, 殊煙, 殊 蟲, 殊 殊, 朱 殊煙 殊煙, 朱
谿瑚襭 Branching Entropy - https://ratsgo.github.io/from%20frequency%20to%20semantics/2017/05/06/BranchingEntropy/ SOYNLP tokenizer 讌襯 - https://sykflyinginthesky.tistory.com/59 ル 伎 一 豌襴 覓 - https://wikidocs.net/92961 蟲 豌襴 - https://taepseon.tistory.com/156 https://gitlab.com/hiongun/be-cp https://github.com/lovit/soynlp

More Related Content

TechTalk - POI Tokenizing

  • 1. POI 覈豺 Tokenizing 郁規 螳覦Lab 覦
  • 2. 伎 (Tokenizing) 一伎襴 ろ 一危一 覲企ゼ 覲襦 蟆. 企語 豢豌 : https://magazine.hankyung.com/business/article/202010133532b
  • 3. 蟆 讌 襷 (Inverted Index) 企語 豢豌 : https://www.skyer9.pe.kr/wordpress/?p=1002
  • 4. 伎 譬襯 伎郁鍵襦 蟲覿 Time is money time is mone y 襯 螻ろ讌 覓語ル 狩 蟆郁骸襦 豬蠍 覓語螳 = BoW (Bag of Words) Money is time time is mone y
  • 5. 伎 譬襯 n-gram uni-gram(1-gram), bi-gram(2-gram), tri-gram(3-gram), 4-gram Time is money ti im me _i is s_ _m mo on e_ ne ey
  • 6. 伎 譬襯 n-gram uni-gram(1-gram), bi-gram(2-gram), tri-gram(3-gram), 4-gram Time is money ti im me _i is s_ _m mo on e_ ne ey mo on ne y_ _i is s_ _t ti ey im me Money is time
  • 7. 伎 譬襯 覿 (蠏 蟆讌 ) 瑚概 蟆暑 語 覈 (NNLM) " 覦郁 蟇語伎 讌 螳給" /覈 /覲伎＾ 覦/覈 +螳/蟆譟一 / +/郁屋企 蟇/ +伎/郁屋企 讌/覈 +/蟆譟一 螳/ +/企企 +給/企企
  • 8. 覿 覈語 伎郁鍵 る 豬 譟一, ろ, 蟲伎牡 豬 覯讌螳覦れ願 企蠏語 旧伎る 郁鍵螳 覲伎 ! 覓伎覦 譯殊瑚概 覓 豐る純
  • 9. 企觜 POI 覈豺 轟 伎郁鍵螳 . ( 覿 豬) 覓 讌щ. (n-gram 覿 豬) 觜訣螻 覦覲給 企れ 譟壱 殊煙(2語) 覦燕(覓伎狩一) 蟲螻牛 殊煙讌 殊煙朱
  • 10. POI 覈豺 覿 蟆郁骸 Open Korean Text 覿蠍 蟆郁骸 殊煙(2語) 覦燕(覓伎狩一) 貊ろ語語 企誤碁企 殊煙朱 殊煙, , 2, 語 , 覦燕, 覓伎, 狩, 貊ろ語, , 企, , 企 殊煙, 觜 襦 覿襴讌 覈詩
  • 11. 企至 伎蟾 螳螻旧 : 讌覈 螳 : 讌覈 螻旧
  • 12. POI 覈豺 覦覲給 願 觜覯蟆 煙 螳襴蟲焔覲 蠍誤 螻燕蟲 煙螻旧 企Π願概 螻狩豌危螻旧 觜覯蟆 煙ロ 企 蠍一朱 襷 讌 蟾..?
  • 13. 糾 蠍磯 譟一伎 螳 覩碁焔 覓語 (Out of vocabulry, OOV) 襯 企至 願屋讌? 語伎 企 覦
  • 14. 糾 蠍磯 伎 蠍一 覦覲 煙ロ れ襯 企 語 覓語襯 願屋
  • 15. SOYNLP 讌 襯螻 觚豺 碁朱ゼ 伎 殊企襴 https://github.com/lovit/soynlp
  • 16. SOYNLP 讌 襯 (cohesion probability, CP) 譯殊伎 覓語伎 蠍一朱 郁屋 蟷 譯 襯 觚豺 碁 (branching entropy, BE) 譯殊伎 覓語 る 煙ロ 蠍 覿れ 讌
  • 17. ろ
  • 18. ろ
  • 19. ろ
  • 20. ろ 蟆郁骸 ろ れ OKT 覿 蟆郁骸 SOYNLP SOYNLP 企誤碁企 企, , 企 企誤碁企 企語, 企瑚蟲 企, 碁企 碁企 企誤碁企 殊煙朱 殊煙, , 觜 殊煙碁, 殊煙螻糾骸 蟲 殊, 朱 殊煙碁, 殊煙螻糾骸 蟲, 殊煙 殊, 朱 殊煙碁, 殊煙, 殊 蟲, 殊 殊, 朱 殊煙 殊煙, 朱
  • 21. 谿瑚襭 Branching Entropy - https://ratsgo.github.io/from%20frequency%20to%20semantics/2017/05/06/BranchingEntropy/ SOYNLP tokenizer 讌襯 - https://sykflyinginthesky.tistory.com/59 ル 伎 一 豌襴 覓 - https://wikidocs.net/92961 蟲 豌襴 - https://taepseon.tistory.com/156 https://gitlab.com/hiongun/be-cp https://github.com/lovit/soynlp