How Unidecoder Transliterates UTF-8 to ASCII
?
1 like?787 views
ºÝºÝߣs of my talk at Paris.rb on 2014-11-07. How does UTF-8 work? How to leverage it to convert chinese, russian or any non-ASCII character to ASCII? Here is what the Unidecoder gem does.
![Unpack it
¡°§±¡±.unpack(¡°U¡±)
[1055]
00000100 00011111
4 31](https://image.slidesharecdn.com/unidecoder-141029060556-conversion-gate01/85/How-Unidecoder-Transliterates-UTF-8-to-ASCII-25-320.jpg)
How Unidecoder Transliterates UTF-8 to ASCII
- 1. Unidecoder Simon Courtois - @happynoff
- 3. Ni Hao
- 5. How does it work?
- 6. At the beginning there was ASCII
- 7. A 65 B 66 C 67 a 97 b 98 c 99
- 8. A 65 10 00001 64 32 16 8 4 2 1 a 97 11 00001 64 32 16 8 4 2 1
- 9. B 66 10 00010 64 32 16 8 4 2 1 b 98 11 00010 64 32 16 8 4 2 1
- 11. So every country had its own encoding(s)!
- 12. All was fine ³Ü²Ô³Ù¾±±ô¡
- 13. The World Wide Web
- 14. UTF-8 to the rescue!
- 15. Everything on 32 bits?
- 16. Bad idea c a f ¨¦
- 17. Bad idea c a f ¨¦
- 18. Bad idea 0
- 19. A better idea A 65 010 00001 110 XXXXX 10 XXXXXX 1110 XXXX 10 XXXXXX 10 XXXXXX
- 20. A better idea 110 XXXXX 10 XXXXXX 110 10000 10 011111
- 21. A better idea 10000011111 1055 §±
- 22. So, how does unidecoder work?
- 23. How do we go from §± to P ?
- 24. Start from a string like ¡°§±¡±
- 25. Unpack it ¡°§±¡±.unpack(¡°U¡±) [1055] 00000100 00011111 4 31
- 26. 4 x04 x04.yml Ie Io Dj ¡ P 0 1 2 31
- 27. How to obtain 4 and 31 ?
- 28. unpacked = 1055 0000010000011111 unpacked 8 0000010000011111 4
- 29. How to obtain 4 and 3 1 ?
- 30. 31 unpacked = 1055 0000010000011111 unpacked 255 0000010000011111 0000000011111111 0000010000011111
- 31. Brain fried yet? advertising time!
- 32. www.tinci.fr Web Development Software Development Consulting Support @tincihq
- 33. Resources Characters, Symbols and the Unicode Miracle: bit.ly/why-utf8 Unidecoder: github.com/norman/unidecoder ºÝºÝߣs: bit.ly/unidecoder
- 34. Thank you! Simon Courtois - @happynoff