JanusCon 2024: Mom there are robots in my meeting
?
0 likes?11 views
ݺߣs from my presentation at JanusCon 2024
JanusCon 2024: Mom there are robots in my meeting
- 1. Mon, there are robots in my meeting! Sal Ibarra Corretg
- 2. Once upon a پ
- 7. Set of Open Source projects Easily deploy scalable and secure video conferencing solutions. APIs and mobile SDKs Integrate Jitsi into existing products to add video conferencing capabilities. Community Open Source enthusiasts from all around the world contribute to Jitsi.
- 9. JaaS Hosted Jitsi, by 8x8 jaas.8x8.vc
- 11. 2 API levels ? High level: iframe based, very versatile and customisable ? Low level: your own UI
- 12. Product + Platform ? Work is spread across both ? Improvements in the in- meeting experience ? Improvements in APIs for embedders
- 13. The AI gold rush AI + meetings = ?
- 14. Alright, lets build a chatbot! ? It needs to join the meeting as a participant ? Interact via text (built-in chat) ? Private messages for questions ? Hot word detection? ? Audio output? ? Video output? How hard can it be?
- 15. Demos Recorded, no sweat ?
- 17. What was that? ? Node app running Chrome via Playwright ? Bot written with lib-jitsi-meet, our low level API ? Google for transcriptions ? ChatGPT for AI ? Zapier for emails
- 19. What was that? ? Node app running Chrome via Playwright ? Bot written with lib-jitsi-meet, our low level API ? Google for transcriptions ? ChatGPT for AI ? PlayHT for playing back audio ? WebGL for the moving robot
- 21. But we had other robots before ? Our recorder is based on a Chrome instance running on a container ? PSTN access is accomplished with a server side WebRTC client ? Our robots are invisible though ? So we used Chrome because YOLO
- 22. Ian Malcolm, Jurassic Park Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should.
- 23. How do we interact with chatbots?
- 25. AI is personal
- 26. Lets focus on the building blocks ? GenAI has text as its lingua franca ? Whatever the input it gets converted to text ? The some LLM operates on that text ? Last, the result is converted to the desired output
- 27. Text is King
- 28. Start by adding core primitives Transcriptions + Summaries
- 30. Real Time Usable as subtitles
- 32. A10
- 36. How does it all f it together?
- 37. Getting to users How would they know to turn transcriptions on?
- 42. Brave Talk What are our customers doing?
- 45. How can you built this? ? Use transcription web hooks to keep a full transcript ? Use transcription iframe events to locally display the transcript + the full transcript to catch up ? Custom LLM (Brave doesnt use Skynet) for prompting
- 47. What we leaned ? Every time we tried to solve a problem with a robot there was a better way to solve it without it ? In the context of a meeting ? Understanding transcriptions are a type of recording makes sense ? The f irst demo was poisonous, complexity wise; start simple! Hopefully this helps others
- 48. Try it out! ? JaaS: https://jaas.8x8.vc Get started for free ? meet.jit.si Free Jitsi Meet instance ? github.com/jitsi host it yourself
- 49. Future with AI ? Current generation of AI is very powerful but still ways to go ? A good future would involve small models for dedicated tasks ? Skynet ? Keep iterating ? Image summaries with LLaVA? ? Will we do bots?
- 50. Thanks Tudor Avram R?zvan Purdel
- 51. Thats all folks! Questions? github.com/jitsi ? @jitsinews ? @saghul ? s@social.saghul.net