ݺߣ

ݺߣShare a Scribd company logo

?????? ?? ?? ??? ????(AI for Visual Storytelling)

?
?
B
Byoung-Hee Kim

?12? ????? ??????? ??? ???? ???? ?? (????? ?? ????? ?? ??? ?? ??) [??] ?????? ??? ??? ???? ???? ??? ??? ???? ???? ????. ????(AI) ??? ??? ??? ?? ???? ?, ??, ??, ?? ?? ??? ???? ???? ?????? ?? ? ? ?? ??? ??? ???????. ? ?????? ?? ?? ???? ?? ????? ??? ??? ??? ??? ?? ????, ? ??? ?? ??? ???? ???? ??? ???? ??? ?????. ????? ?? ??? ?????? ?? ???? ???(Thinking Aids)? ????? ???? ??? ???? ? ?? ????.

1 of 37
Downloaded 70 times
?????? ?? ?? ??? ???? AI for Visual Storytelling 2019-02-22 ? ? ? CTO ????? ???? ?12? ????? ??????? ??? ???? ????
? ? . (AI) , , , . , . (Thinking Aids) . ? 2017-2018, Surromind Robotics, Inc. 2
????? ? ???? ? 2017-2018, Surromind Robotics, Inc. 3https://m.nocutnews.co.kr/news/5104825#_enliple ??? ?? ???? ?? ??? ??? ????? ???? ???? ?? ???? ??
????? ??? ?????? ??? ??? ? ?? ? ? ???? ?? ??, ??, ?? ??? ?? ???? ?? ??, ??, ??, ??? ??? ??, ??? ?? ??? ???? ?? ???? ?????? ?? ????? ???? ? ? 2017-2018, Surromind Robotics, Inc. 4
????? ????? ?????? ?? ???? ??? ?? ???? ??(??, ??, ??, ??, ?? ?) ??? ??, ?? ??, ?? ?? ? ???? ??? ??? ???? ??? ?? ?? ??? ???? ???? ?? ?? ??????, ??? ?? ??? ?? ?????? ????, ?? ??? ???? ???, ??? ??? ???? ????? ?? ??? ??? ???? ?? ???? ????, ????, ???? ??? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 5 https://en.wikipedia.org/wiki/Storytelling - Contemporary storytelling ???? ????? ??!
?????? ???? ????(AI) ??? ?????? ??? ? ???? ??? ??? ?? ???? ?? ? ?? ??? ?? ?????? ???? ??? ???? ??? ???? ??? ??? ??? ???? ??? ? ?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 6
AI ????? ????? ?? ?? - thefatheroftheoreticalcomputerscienceandartificialintelligence ? ???? ??? ?? ??(Turing, 1947) (Turing, 1950) ? ???? ??? ???? ???? ???? ??? ??? ?? ?? ??? ?? C the father of information theory ? ??? ??? ?? ??? ?? ??? ??. ?? ??? ????? ?? ??? ??? ???? ??? ??(Shannon, 1951) ? ?? ??? ???? ?? ?? ??(language model) ??? ? ?? ?? ??? ????? ??? ??. ?? ??? ??? ? ????? ?? ??? ?? ?? ?? ??? C AI Pioneer ? ???? ??? ?? ??? ??? ???(Minsky, 1961), ???? ??? ?? thinking aids? ??? ??? ?? ? ???? ??? MIT? ??? ???(Patrick Winston) ?? ? ????? ??? AI? ???? ?? ??? ? ??? ??? ????? ???, ??? ??? ??? ????? ??, ???? ???? ??, ??? ??? ???? ???? ??? ???? ??? ??? ????? ?? ? 2017-2018, Surromind Robotics, Inc. 7 Alan Turing (1912-1954) Claude Shannon (1916-2001) Marvin Minsky (1927-2016) ????? ????? ????? ??? ??????.
6-elements-of-every- complete-narrative 1. Setting The setting is the time and location in which your story takes place. 2. Characters Central Characters, Protagonist, Antagonist 3. Plot The plot is the sequence of events that connect the audience to the protagonist and their ultimate goal. 4. Conflict The conflict is what drives the story. Its what creates tension and builds suspense, which are the elements that make a story interesting. 5. Theme The theme is what the story is really about. Its the main idea or underlying meaning. 6. Narrative Arc Setup C rising tension C climax C resolution Topics of AI for Storytelling Papers ? 2017-2018, Surromind Robotics, Inc. 8 ?????? ?? http://www.visionandlanguage.net/workshop2018/index.htmlstorytelling-101-the-6-elements-of-every-complete-narrative by pond5.com ?????? ??? ??? ??, ??, ??? ? ?? ??? ???? ?? ???? ?? ??? ???? ????
?????? ?? ???? ???(Thinking Aids) Accessible and assistive storytelling Interactive work Augmenting human storytelling Collaborative storytelling ?? ???? ?? ??? ??? ???? ??? ?? ??? ?? ??, ?? ? 2017-2018, Surromind Robotics, Inc. 9
AI for Storytelling ??? ???? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 10 ?(text) ??(music) ??(image) ???(voice) ???(video) Generative Artificial Intelligence
???? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 11 ???? ?? ?? ?? ???? ??? ?? ?? ??? ??? ?(template)???(rule)? ??? ??? ??? ?? ???(deep learning)?? ???? ?? AI ?? ?????? ?? ?? ?? ??? ??? ?? ?? ???? ???? ??? ?? ???? ?? ?? ?? ?? ??? ???? ?? ???? ??? ?? ???? ????. (https://cs.stanford.edu/~zxie/textgen.pdf Fig. 1)
????? ? ???? ? 2017-2018, Surromind Robotics, Inc. 12https://m.nocutnews.co.kr/news/5104825#_enliple ??? ?? ???? ?? ??? ??? ????? ???? ???? ?? ???? ??
???? AI? ???? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 13 ?? ??? ??? ??? ?? ???? ?? ?? ??? ??? ?? ?? ? ???? ??? ?? ??? ??? ? ?? ??? ?? ?? ?? ? ??? ?? ?? ???? ??? ?? ??, ?? ?? ?? ??(?? ???) (SW ???) [AP, Yahoo!Sports, ] (structured data)
?? ?? ?? AI? ? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 14 ~8?? ?? ???? (~40GB) Language Model ?? ?? ?? ?? ?? ?? ?? ??(??) ?? ??? ??? ???? ??? ?? ?? (~30 sec) (??? ?? ?? ???? ??) ??? ?? Massive Computing Power Massive Data (~80 GPUs * ~30 days)(GPT-2 by OpenAI) (AI ???) [??, ??, ??]
? 2017-2018, Surromind Robotics, Inc. 15 ???? ?????, ????? ??? ? ???? ??? ?? ?? ?? ??? ?? ??? ???? ???? ?? ?? AI ??? ?? ?? AI ??? ??? ? ????
AI?? ? ?? ??? ?? ?????? ????? ? 2017-2018, Surromind Robotics, Inc. 16
?? ? ? 2017-2018, Surromind Robotics, Inc. 17 http://max-image-caption-generator-web-app.mybluemix.net/ https://pix2story.azurewebsites.net/ https://app.photerloo.com/InstagramHashtagKeywordingApp/
Album Storyteller ? 2017-2018, Surromind Robotics, Inc. 18 http://bit.ly/albumstoryteller (closed-beta version)
Album Storyteller ? 2017-2018, Surromind Robotics, Inc. 19 ?? ??? ?? ?????
?????? ?? ???? ??? ?? ??? ?? ?????? ??? ??? ??? ??? ??, ??, ?? ??? ?????. AI ????? ??? ??? ??? ????? ???? ? ?? ??? ??? ??????? ?? ??? ?? ????? ??? ?? ?? ??? ??? ?? ?? ??? ? ?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 20
?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 21 Unsupervised Text Style Transfer using Language Models as Discriminators (NIPS 2018) (??? ?? ??? ??) (??? ?? ??? ??) ?? ??? ??(sentiment)? ???? ???? ??
?? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 22 Toward Unsupervised Text Content Manipulation (arXiv, 2019.01) Rule/Template-based ?? ??? Neural ???? ?? ? ??? ?? ?? ?? (NBA ?? ???? ??? ????)
????? ??? ??, ?? ? 2017-2018, Surromind Robotics, Inc. 23 FaceApp by Facebook StyleGAN by NVIDIA Synthesizing Obama: Learning Lip Sync from Audio (SIGGRAPH 2017) Deep image analogy
More Visual Modifications ? 2017-2018, Surromind Robotics, Inc. 24 ݺߣ from /mlreview/tutorial-on-theory-and-application-of-generative-adversarial-networks ??? Image-to-image Translation ??
More Visual Modifications ? 2017-2018, Surromind Robotics, Inc. 25 ??? ????? ???? ????
??? ??? ?? ? ?? ??: classification, detection, segmentation ? ??? ??: ??, ?? ??, ?? ??, ? ?? ??? ??? grounding ? ?? ?? ?? ? ?? ? ??? ?? ?? ? ? 2017-2018, Surromind Robotics, Inc. 26 Bilinear Attention Networks (NIPS 2018) (????? ???) ????? ?? ??? ???? ?? ?? ?? ??? ?? ???? ????.
? ?? ? 2017-2018, Surromind Robotics, Inc. 27 Image form the StackGAN paper : https://arxiv.org/pdf/1612.03242.pdf
AI for Visual Storytelling ??? ???? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 28 ?? ?? ? ?? ?? ?? ??? ?? ??? ?? ??, ??? ?? ?? ??? ?? ???? ?? ?? ? ?? ???? ???, ?? ?? ?? ?? ?? ?? ??? ?? ? ??
AI for Visual Storytelling ???? ?? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 29 # of paper on GAN (2014~2018.02) 500 2014 2018
AI? ?? ?????? ??? ? 2017-2018, Surromind Robotics, Inc. 30 smart machines ????? ??? ????? 4? ?? ?? (David Wood, Anticipating the future of software, 2015? ??? ???? ?)
??? ? 2017-2018, Surromind Robotics, Inc. 31 ?? ???? ???? ?? ??? ???? ? ?? ?? ?? ????? ??? ???? ??? ???? Album Storyteller ?? ??AI? ??AI ??? ???? ??? ? ???? ? ????
Take-Home Message ? 2017-2018, Surromind Robotics, Inc. 32 And Dont Be Evil in Using AI
? 2017-2018, Surromind Robotics, Inc. 33
?? ?? ? A. M. Turing (1947) Lecture to the London Mathematical Society. http://www.vordenker.de/downloads/turing- vorlesung.pdf (February 20, 1947) ? A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49:433-460. (Ch.7 Learning Machines) ? C. E. Shannon (1951) Prediction and Entropy of Printed English. Bell Labs Technical Journal 30(1):50-64. ? M. Minsky (1961) Steps Toward Artificial Intelligence. Proceedings of the IRE 49(1):8-30. ? 2017-2018, Surromind Robotics, Inc. 34
SurroMind Robotics Confidential Album Storyteller Quick Start Guide http://bit.ly/albumstoryteller (closed-beta version)
? Web Browser: ??? ?? ??(Chrome)? ????? ? Google Photos ?? C ???? ??? Google Photos? ???? ?? ???? ??? ?? ???? ??? ? ????. C ?? ??? ??? https://photos.google.com/?? ?? ??? ??? ? ????. SurroMind Robotics Confidential ?? ?? C ?? ?? ??????
? ?? ?????? URL ?? http://bit.ly/albumstoryteller C ?? ??? ?? ?? ???? ???. ?? ???? ???? ?? ??? ??? ???? ??????. ? ????? ???? ???? ?????? ? ????? ? ? ????? ?????? ? ??? ?? C ? ????? ??? ???? ????. ? ??? ??? ???? ????? C ??? ???? ?? ???? SNS ?? ??? ?? ???? ?????? ? ???? ??? ?????? ?? ??? ??????. SurroMind Robotics Confidential Album Storyteller ??

More Related Content

?????? ?? ?? ??? ????(AI for Visual Storytelling)

  • 1. ?????? ?? ?? ??? ???? AI for Visual Storytelling 2019-02-22 ? ? ? CTO ????? ???? ?12? ????? ??????? ??? ???? ????
  • 2. ? ? . (AI) , , , . , . (Thinking Aids) . ? 2017-2018, Surromind Robotics, Inc. 2
  • 3. ????? ? ???? ? 2017-2018, Surromind Robotics, Inc. 3https://m.nocutnews.co.kr/news/5104825#_enliple ??? ?? ???? ?? ??? ??? ????? ???? ???? ?? ???? ??
  • 4. ????? ??? ?????? ??? ??? ? ?? ? ? ???? ?? ??, ??, ?? ??? ?? ???? ?? ??, ??, ??, ??? ??? ??, ??? ?? ??? ???? ?? ???? ?????? ?? ????? ???? ? ? 2017-2018, Surromind Robotics, Inc. 4
  • 5. ????? ????? ?????? ?? ???? ??? ?? ???? ??(??, ??, ??, ??, ?? ?) ??? ??, ?? ??, ?? ?? ? ???? ??? ??? ???? ??? ?? ?? ??? ???? ???? ?? ?? ??????, ??? ?? ??? ?? ?????? ????, ?? ??? ???? ???, ??? ??? ???? ????? ?? ??? ??? ???? ?? ???? ????, ????, ???? ??? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 5 https://en.wikipedia.org/wiki/Storytelling - Contemporary storytelling ???? ????? ??!
  • 6. ?????? ???? ????(AI) ??? ?????? ??? ? ???? ??? ??? ?? ???? ?? ? ?? ??? ?? ?????? ???? ??? ???? ??? ???? ??? ??? ??? ???? ??? ? ?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 6
  • 7. AI ????? ????? ?? ?? - thefatheroftheoreticalcomputerscienceandartificialintelligence ? ???? ??? ?? ??(Turing, 1947) (Turing, 1950) ? ???? ??? ???? ???? ???? ??? ??? ?? ?? ??? ?? C the father of information theory ? ??? ??? ?? ??? ?? ??? ??. ?? ??? ????? ?? ??? ??? ???? ??? ??(Shannon, 1951) ? ?? ??? ???? ?? ?? ??(language model) ??? ? ?? ?? ??? ????? ??? ??. ?? ??? ??? ? ????? ?? ??? ?? ?? ?? ??? C AI Pioneer ? ???? ??? ?? ??? ??? ???(Minsky, 1961), ???? ??? ?? thinking aids? ??? ??? ?? ? ???? ??? MIT? ??? ???(Patrick Winston) ?? ? ????? ??? AI? ???? ?? ??? ? ??? ??? ????? ???, ??? ??? ??? ????? ??, ???? ???? ??, ??? ??? ???? ???? ??? ???? ??? ??? ????? ?? ? 2017-2018, Surromind Robotics, Inc. 7 Alan Turing (1912-1954) Claude Shannon (1916-2001) Marvin Minsky (1927-2016) ????? ????? ????? ??? ??????.
  • 8. 6-elements-of-every- complete-narrative 1. Setting The setting is the time and location in which your story takes place. 2. Characters Central Characters, Protagonist, Antagonist 3. Plot The plot is the sequence of events that connect the audience to the protagonist and their ultimate goal. 4. Conflict The conflict is what drives the story. Its what creates tension and builds suspense, which are the elements that make a story interesting. 5. Theme The theme is what the story is really about. Its the main idea or underlying meaning. 6. Narrative Arc Setup C rising tension C climax C resolution Topics of AI for Storytelling Papers ? 2017-2018, Surromind Robotics, Inc. 8 ?????? ?? http://www.visionandlanguage.net/workshop2018/index.htmlstorytelling-101-the-6-elements-of-every-complete-narrative by pond5.com ?????? ??? ??? ??, ??, ??? ? ?? ??? ???? ?? ???? ?? ??? ???? ????
  • 9. ?????? ?? ???? ???(Thinking Aids) Accessible and assistive storytelling Interactive work Augmenting human storytelling Collaborative storytelling ?? ???? ?? ??? ??? ???? ??? ?? ??? ?? ??, ?? ? 2017-2018, Surromind Robotics, Inc. 9
  • 10. AI for Storytelling ??? ???? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 10 ?(text) ??(music) ??(image) ???(voice) ???(video) Generative Artificial Intelligence
  • 11. ???? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 11 ???? ?? ?? ?? ???? ??? ?? ?? ??? ??? ?(template)???(rule)? ??? ??? ??? ?? ???(deep learning)?? ???? ?? AI ?? ?????? ?? ?? ?? ??? ??? ?? ?? ???? ???? ??? ?? ???? ?? ?? ?? ?? ??? ???? ?? ???? ??? ?? ???? ????. (https://cs.stanford.edu/~zxie/textgen.pdf Fig. 1)
  • 12. ????? ? ???? ? 2017-2018, Surromind Robotics, Inc. 12https://m.nocutnews.co.kr/news/5104825#_enliple ??? ?? ???? ?? ??? ??? ????? ???? ???? ?? ???? ??
  • 13. ???? AI? ???? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 13 ?? ??? ??? ??? ?? ???? ?? ?? ??? ??? ?? ?? ? ???? ??? ?? ??? ??? ? ?? ??? ?? ?? ?? ? ??? ?? ?? ???? ??? ?? ??, ?? ?? ?? ??(?? ???) (SW ???) [AP, Yahoo!Sports, ] (structured data)
  • 14. ?? ?? ?? AI? ? ?? ?? ? 2017-2018, Surromind Robotics, Inc. 14 ~8?? ?? ???? (~40GB) Language Model ?? ?? ?? ?? ?? ?? ?? ??(??) ?? ??? ??? ???? ??? ?? ?? (~30 sec) (??? ?? ?? ???? ??) ??? ?? Massive Computing Power Massive Data (~80 GPUs * ~30 days)(GPT-2 by OpenAI) (AI ???) [??, ??, ??]
  • 15. ? 2017-2018, Surromind Robotics, Inc. 15 ???? ?????, ????? ??? ? ???? ??? ?? ?? ?? ??? ?? ??? ???? ???? ?? ?? AI ??? ?? ?? AI ??? ??? ? ????
  • 16. AI?? ? ?? ??? ?? ?????? ????? ? 2017-2018, Surromind Robotics, Inc. 16
  • 17. ?? ? ? 2017-2018, Surromind Robotics, Inc. 17 http://max-image-caption-generator-web-app.mybluemix.net/ https://pix2story.azurewebsites.net/ https://app.photerloo.com/InstagramHashtagKeywordingApp/
  • 18. Album Storyteller ? 2017-2018, Surromind Robotics, Inc. 18 http://bit.ly/albumstoryteller (closed-beta version)
  • 19. Album Storyteller ? 2017-2018, Surromind Robotics, Inc. 19 ?? ??? ?? ?????
  • 20. ?????? ?? ???? ??? ?? ??? ?? ?????? ??? ??? ??? ??? ??, ??, ?? ??? ?????. AI ????? ??? ??? ??? ????? ???? ? ?? ??? ??? ??????? ?? ??? ?? ????? ??? ?? ?? ??? ??? ?? ?? ??? ? ?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 20
  • 21. ?? ??? ?? ? 2017-2018, Surromind Robotics, Inc. 21 Unsupervised Text Style Transfer using Language Models as Discriminators (NIPS 2018) (??? ?? ??? ??) (??? ?? ??? ??) ?? ??? ??(sentiment)? ???? ???? ??
  • 22. ?? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 22 Toward Unsupervised Text Content Manipulation (arXiv, 2019.01) Rule/Template-based ?? ??? Neural ???? ?? ? ??? ?? ?? ?? (NBA ?? ???? ??? ????)
  • 23. ????? ??? ??, ?? ? 2017-2018, Surromind Robotics, Inc. 23 FaceApp by Facebook StyleGAN by NVIDIA Synthesizing Obama: Learning Lip Sync from Audio (SIGGRAPH 2017) Deep image analogy
  • 24. More Visual Modifications ? 2017-2018, Surromind Robotics, Inc. 24 ݺߣ from /mlreview/tutorial-on-theory-and-application-of-generative-adversarial-networks ??? Image-to-image Translation ??
  • 25. More Visual Modifications ? 2017-2018, Surromind Robotics, Inc. 25 ??? ????? ???? ????
  • 26. ??? ??? ?? ? ?? ??: classification, detection, segmentation ? ??? ??: ??, ?? ??, ?? ??, ? ?? ??? ??? grounding ? ?? ?? ?? ? ?? ? ??? ?? ?? ? ? 2017-2018, Surromind Robotics, Inc. 26 Bilinear Attention Networks (NIPS 2018) (????? ???) ????? ?? ??? ???? ?? ?? ?? ??? ?? ???? ????.
  • 27. ? ?? ? 2017-2018, Surromind Robotics, Inc. 27 Image form the StackGAN paper : https://arxiv.org/pdf/1612.03242.pdf
  • 28. AI for Visual Storytelling ??? ???? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 28 ?? ?? ? ?? ?? ?? ??? ?? ??? ?? ??, ??? ?? ?? ??? ?? ???? ?? ?? ? ?? ???? ???, ?? ?? ?? ?? ?? ?? ??? ?? ? ??
  • 29. AI for Visual Storytelling ???? ?? ??? ???? ? 2017-2018, Surromind Robotics, Inc. 29 # of paper on GAN (2014~2018.02) 500 2014 2018
  • 30. AI? ?? ?????? ??? ? 2017-2018, Surromind Robotics, Inc. 30 smart machines ????? ??? ????? 4? ?? ?? (David Wood, Anticipating the future of software, 2015? ??? ???? ?)
  • 31. ??? ? 2017-2018, Surromind Robotics, Inc. 31 ?? ???? ???? ?? ??? ???? ? ?? ?? ?? ????? ??? ???? ??? ???? Album Storyteller ?? ??AI? ??AI ??? ???? ??? ? ???? ? ????
  • 32. Take-Home Message ? 2017-2018, Surromind Robotics, Inc. 32 And Dont Be Evil in Using AI
  • 33. ? 2017-2018, Surromind Robotics, Inc. 33
  • 34. ?? ?? ? A. M. Turing (1947) Lecture to the London Mathematical Society. http://www.vordenker.de/downloads/turing- vorlesung.pdf (February 20, 1947) ? A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49:433-460. (Ch.7 Learning Machines) ? C. E. Shannon (1951) Prediction and Entropy of Printed English. Bell Labs Technical Journal 30(1):50-64. ? M. Minsky (1961) Steps Toward Artificial Intelligence. Proceedings of the IRE 49(1):8-30. ? 2017-2018, Surromind Robotics, Inc. 34
  • 35. SurroMind Robotics Confidential Album Storyteller Quick Start Guide http://bit.ly/albumstoryteller (closed-beta version)
  • 36. ? Web Browser: ??? ?? ??(Chrome)? ????? ? Google Photos ?? C ???? ??? Google Photos? ???? ?? ???? ??? ?? ???? ??? ? ????. C ?? ??? ??? https://photos.google.com/?? ?? ??? ??? ? ????. SurroMind Robotics Confidential ?? ?? C ?? ?? ??????
  • 37. ? ?? ?????? URL ?? http://bit.ly/albumstoryteller C ?? ??? ?? ?? ???? ???. ?? ???? ???? ?? ??? ??? ???? ??????. ? ????? ???? ???? ?????? ? ????? ? ? ????? ?????? ? ??? ?? C ? ????? ??? ???? ????. ? ??? ??? ???? ????? C ??? ???? ?? ???? SNS ?? ??? ?? ???? ?????? ? ???? ??? ?????? ?? ??? ??????. SurroMind Robotics Confidential Album Storyteller ??