COCO's Memory Palace: A Strange Fantasia
1 like238 views
This document describes COCO's Memory Palace, a project that uses AI image recognition APIs like Google Vision and Amazon Rekognition to analyze images and generate poems. COCO is trained on the MS COCO dataset of 80 common objects. The project combines images, poetry, and AI outputs to strange fantastical effect, mixing dreamlike memories and imaginings. It explores ideas from classical arts of memory involving the association of images to locations for remembering information.
![MS COCO Dataset
Common Objects in Context image training dataset:
Our dataset contains photos of [80 objects]* types that would
be easily recognizable by a 4 year old.
http://cocodataset.org/#home *2014 was 80, not 91](https://image.slidesharecdn.com/kikktalk-191223101824/85/COCO-s-Memory-Palace-A-Strange-Fantasia-33-320.jpg)
COCO's Memory Palace: A Strange Fantasia
- 1. COCOs Memory Palace: A Strange Fantasia Lynn Cherny @arnicas
- 3. Images
- 6. Project 1: Color Poems Color analysis & Google Vision API, Unsplash API
- 7. Martin Krzywinskis color summarizer API http://mkweb.bcgsc.ca/colorsummarizer/
- 8. Martin Krzywinskis color summarizer API http://mkweb.bcgsc.ca/colorsummarizer/
- 9. My list uses the following sources: bang-v2, beer SRM chart, colorhexa, crayola, n3, ntc, pantone, raveling, resene, rgb.txt (X11), steve.hollasch, wikipedia, and xkcd. 9284 Named Colors http://mkweb.bcgsc.ca/colornames/
- 15. Some More Design Details BigText.js for text layout (space-鍖lling width/size) Complicated drop shadows to make text stand out against the image background CSS image effects to grayscale, blur, opacity (Flexbox saved my ass)
- 18. John Berger, Understanding a Photograph I have decided that seeing this is worth recording. Photography is the process of rendering observation self-conscious.
- 19. https://www.nyu.edu/classes/bkg/methods/harper.pdf Talking about pictures: a case for photo elicitation (Harper, 2002)
- 22. Frances Yates, The Art of Memory There are two kinds of memory one natural, the other arti鍖cial. The natural memory is that which is engrafted in our minds, born simultaneously with thought. The arti鍖cial memory is a memory strengthened or con鍖rmed by training.
- 25. Frances Yates, The Art of Memory The arti鍖cial memory is established from places and images. A locus is a place easily grasped by the memory, such as a house, an intercolumnar space, a corner, an arch, or the like. Images are forms, marks or simulacra.. of what we wish to remember. For instance if we wish to recall the genus of a horse, or a lion, of an eagle, we must place their images on de鍖nite loci.
- 26. @TheGlenlivet
- 29. Recognition works on either an S3 bucket url, or an image blob.
- 30. Using Google Vision API is similar but can take any url
- 31. Project 2: COCOs poems, & 3 voices
- 32. Fourteen Mary Doria Russell All bone, no meat Growing, not grown. Desire, dreams, Nothing to show. Diving, but still dry. Photo courtesy of Accidental Mysteries, collection of John Foster. Poems About Photography
- 33. MS COCO Dataset Common Objects in Context image training dataset: Our dataset contains photos of [80 objects]* types that would be easily recognizable by a 4 year old. http://cocodataset.org/#home *2014 was 80, not 91
- 34. Useful articles How to Freeze a Model and Serve It With A Python API Dat Trans post: How to Train Your Own Object Detector With Tensor鍖ow
- 35. Useful articles How to Freeze a Model and Serve It With A Python API Dat Trans post: How to Train Your Own Object Detector With Tensor鍖ow
- 37. Undergraduates said. people, 艶厩艶稼岳
- 39. What is the model trained on ==
- 40. What is the model trained on == what can it talk about?
- 41. COCOs world of 80 things. "person" "bicycle" "car" "motorcycle" "airplane" "bus" "train" "truck" "boat" "traf鍖c light" "鍖re hydrant" "stop sign" "parking meter" "bench"
- 42. COCOs world of 80 things. "person" "bicycle" "car" "motorcycle" "airplane" "bus" "train" "truck" "boat" "traf鍖c light" "鍖re hydrant" "stop sign" "parking meter" "bench" "bird" "cat" "dog" "horse" "sheep" "cow" "elephant" "bear" "zebra" "giraffe"
- 43. "backpack" "umbrella" "handbag" "tie" "suitcase" "frisbee" "skis" "snowboard" "sports ball" "kite" "baseball bat" "baseball glove" "skateboard" "surfboard" "tennis racket"
- 44. "backpack" "umbrella" "handbag" "tie" "suitcase" "frisbee" "skis" "snowboard" "sports ball" "kite" "baseball bat" "baseball glove" "skateboard" "surfboard" "tennis racket" "bottle" "wine glass" "cup" "fork" "knife" "spoon" "bowl" "banana" apple "orange" "broccoli" "carrot" "hot dog" "pizza" "donut" cake" "sandwich"
- 45. "chair" "couch" "potted plant" "bed" "dining table" "toilet" "tv" "laptop" "mouse" "remote" "keyboard" "cell phone" "microwave" oven" "toaster" "sink" "refrigerator"
- 46. "chair" "couch" "potted plant" "bed" "dining table" "toilet" "tv" "laptop" "mouse" "remote" "keyboard" "cell phone" "microwave" oven" "toaster" "sink" "refrigerator" "book" "clock" "vase" "scissors" "teddy bear" "hair drier" "toothbrush"
- 47. potted plant but not tree.
- 48. potted plant but not tree. COCO knows about computers, but there is no sky, no grass, no mountains.
- 51. There is a Jupyter notebook there that will run the tensor鍖ow demo for you.
- 52. How poetic is COCO?
- 54. eh, whoops. Not legit: traf鍖c light not light, 鍖re hydrant not 鍖re.
- 56. couch > cat toilet > umbrella traf鍖c > bus spoon == fork but knife beats both
- 57. > grep sandwich poetry-json
- 58. in microwave fashion donut-sized scales even if ice-cream and pizza attach dead-pan humor > grep sandwich poetry-json
- 59. COCOs grammar* Includes words related to her degree of uncertainty at the object recognition nouns mixed in that are NOT in her vocabulary, from a word2vec lookup of related terms in the Gigaword corpus served by API (part-of-speech tagged) Language style re鍖ecting naivety and youth *tracery.js from Kate Compton
- 62. Part-of-speech tagged Gigaword corpus word2vec model Words similar to castle_NOUN:
- 63. John Berger A photograph, whilst recording what has been seen, always and by its nature refers to what is not seen. the quantum of truth in an 'impersonal' photograph must still depend upon the general categories already in the spectator's mind.
- 68. Adding more voices Google Vision API more objective and scienti鍖c language Amazon Rekognition more colorful, more poetic
- 70. Google Vision API & Amazon Rekognition (the nouns)
- 74. Using COCO, Amazon, and Google APIs all together
- 77. WH Auden on The Fall of Icarus by Peter Brueghel
- 80. (Google said FU to the URL)
- 81. the old guitarist, Picasso The Man with the Blue Guitar by Wallace Stevens
- 82. The man bent over his guitar, A shearsman of sorts. The day was green. They said, You have a blue guitar, You do not play things as they are. The man replied, Things as they are Are changed upon the blue guitar.
- 85. COCO says, man with a teddy bear. https://www.amazon.com/PERSONAL-Recordable-Plush-Talking-Teddy/dp/B001RZE0E4
- 86. COCO says, man with a teddy bear. https://www.amazon.com/PERSONAL-Recordable-Plush-Talking-Teddy/dp/B001RZE0E4
- 88. Wallace Stevens, Man With the Blue Guitar We shall forget by day, except The moments when we choose to play The imagined pine, the imagined jay.
- 89. Yates, the Art of Memory an arti鍖cial memory gone out of control into wild imaginative indulgence
- 90. Yates, the Art of Memory an arti鍖cial memory gone out of control into wild imaginative indulgence the dream archaeology of a humanist mingles with dream memory systems to form the strange fantasia.
- 91. Thank you. -@arnicas 際際滷s will go on slideshare and a blog post.