Asynchronicity
0 likes722 views
The document outlines the activities of Asynchronicity, co-founded by Takahiro Yoshimura and Shinya Kuroda, focusing on security research and development, particularly in mobile and web applications, alongside penetration testing and forensic analysis. It highlights innovative techniques for mass scraping web pages, including the use of asynchronous programming for enhanced efficiency in scraping operations. Additionally, it discusses various tools and methodologies employed to tackle challenges in web scraping and network interactions.
![ASYNCHRONICITY
WHAT WE DO
? Security research and development
? iOS/Android Apps?
★Financial, Games, IoT related, etc. (>200)?
★trueseeing: Non-decompiling Android Application
Vulnerability Scanner [2017]
? Windows/Mac/Web/HTML5 Apps?
★POS, RAD tools etc.
? Network/Web penetration testing?
★PCI-DSS etc.
? Search engine reconnaissance?
(aka. Google Hacking)
? Whitebox testing
? Forensic analysis
? Research?
★Clairvoyance: concurrent lip reader [2019]](https://image.slidesharecdn.com/hackersparty19-190722012351/85/Asynchronicity-3-320.jpg)
Asynchronicity
- 1. ASYNCHRONICITY 19.7.2019 HACKERS¨ PARTY 2019 thread by adkeno06 on flickr, CC-BY-NC 2.0
- 2. ASYNCHRONICITY WHO WE ARE ? Takahiro Yoshimura (@alterakey)? Monolith Works Inc.? Co-founder ? Shinya Kuroda (@ameOtoko)? Monolith Works Inc.? Security Analyst
- 3. ASYNCHRONICITY WHAT WE DO ? Security research and development ? iOS/Android Apps? ★Financial, Games, IoT related, etc. (>200)? ★trueseeing: Non-decompiling Android Application Vulnerability Scanner [2017] ? Windows/Mac/Web/HTML5 Apps? ★POS, RAD tools etc. ? Network/Web penetration testing? ★PCI-DSS etc. ? Search engine reconnaissance? (aka. Google Hacking) ? Whitebox testing ? Forensic analysis ? Research? ★Clairvoyance: concurrent lip reader [2019]
- 4. ASYNCHRONICITY WHAT WE DO ? CTF ? Enemy10, Sutegoma2 ? METI CTFCJ 2012 Qual.: 1st ? METI CTFCJ 2012: 3rd ? DEF CON 21 CTF: 6th ? DEF CON 22 OpenCTF: 4th ? Talks:? DEF CON 25 Demo Labs? CODE BLUE 2017? DEF CON 27 AI Village etc. DEFCON 2016 by Wiyre Media on flickr, CC-BY 2.0
- 5. GRABBING CAPTCHA ACT. 1 screened by new 1llminati on flickr, CC-BY 2.0
- 6. ASYNCHRONICITY ORDINAL GRAB.. ? Grabbing CAPTCHAs ? in order to destroy them ? Let¨s review Dream Market case? ！ seems that they are ＾Nightmare￣
- 7. ASYNCHRONICITY ORDINAL GRAB.. NOT! ? Simple serial scraping strategy ? Loop and scrape
- 8. ASYNCHRONICITY ORDINAL GRAB.. NOT! ? Horrid RTTs ? Normal strategy would take ages! ? Fail FAIL
- 9. ASYNCHRONICITY BRANCHING INTO BACKGROUND ? Forking into thread pool, with venerable concurrent.futures.TPE? (ThreadPoolExecutor) ? You have 2 executors: ? ThreadPoolExecutor ? ProcessPoolExecutor
- 10. ASYNCHRONICITY BRANCHING INTO BACKGROUND ? GIL ！ Global Interpreter Lock ? OS threads do not scale well ? OS processes can scale, but they often have huge overhead? (and require tasks to be pickled)? ！ hence, we have used TPEs ? OS threads have cost? - memory and context switches ? Suboptimal ！ Needs more work..
- 11. ASYNCHRONICITY DESYNCHRONIZED BEAUTY ? Pack as coroutines and stack up with asyncio.gather et al. ? async / await (Python 3.5+) ? async def marks coroutines ? await runs and wait coroutines? ！ hence, ＾voluntary handoff￣ ? Cooperative threading
- 12. ASYNCHRONICITY DESYNCHRONIZED BEAUTY ? Asynchronous, event based run ? Uses only one OS thread? ！ just wait and handle events? ！ uvloop gives us even more boost ? Negligible cost? ！ A single fd in this case:? only your limit is max open ?les? ！ Async super parallel grabber! ? Optimal ！ 12% better in this case
- 13. ASYNCHRONICITY DESYNCHRONIZED BEAUTY ? Stubborn synchronous operations? ? Run them in TPEs through BEL.run_in_executor! ? You can even combine PPEs to sidestep GIL as needed
- 14. ASYNCHRONICITY AN EXCELLENT NETWORK STACK, UVLOOP ? The heart of node.js
- 15. ASYNCHRONICITY AN EXCELLENT NETWORK STACK, UVLOOP ? The heart of node.js ? Can go faster than node.js itself
- 16. ASYNCHRONICITY AIOHTTP: ASYNCHRONOUS HTTP STACK ? Asynchronous HTTP stack ? Parser is not so fast (nor async.)? ！ thanks for GIL ? httptools is more faster? (and a bit low level)
- 17. ASYNCHRONICITY LIMITING CONCURRENCY ? Tor DoS protection kicks in around 100 concurrent connections ? Enforcing concurrency? - asyncio.Semaphore in coroutines? - limit in connector (aiohttp)
- 18. MASS SCRAPING ACT. 2 scrape by Katie McMahon on flickr, CC-BY-NC 2.0
- 19. ASYNCHRONICITY CHASING FOR THE WIDTH ? Can we mass-scrape Web pages?? Say, 100 domains, from tor? ? Of course we can ;) walker getting scraped off by Caffeinatrix on flickr, CC-BY-NC-ND 2.0
- 20. ASYNCHRONICITY BASIC STRATEGY ? WebDriver (headless Firefox) ? Scenario:? Save screenshot of the front page? (from the dark web of course) ? You may feel like limit concurrency? ！ WD will easily swamp you! Wolf Pack by Neil McIntosh on flickr, CC-BY 2.0
- 21. ASYNCHRONICITY LURKING IN THE DARKNESS ? Yes, WD can use tor ? Set up a custom pro?le ? network.proxy.type=1 network.proxy.socks=127.0.0.1? network.proxy.socks_port=9150? network.proxy.socks_version=5? network.proxy.socks_remote_dns=t? network.h.m-p-c-p-p=256? network.h.p.v=1.1
- 22. ASYNCHRONICITY SCRAPE THE DARKNESS ? Corpse: OnionDir ? 198 services ? RTT is horrid as always
- 23. ASYNCHRONICITY SCRAPABLE! ? Normal scraping with WebDriver? ~2.5h @ 198 services ? Desynchronized WebDrivers? ~13m @ 198 services ? Feasible: 1000%~ better
- 25. ASYNCHRONICITY SCRAPABLE? - CLEARWEB CASE ? Corpse: ? Some of SANS Suspicious Domains? (Medium Sensitivity Level) ? Colorful 150 domains Hazards by El Bingle on flickr, CC-BY-NC 2.0
- 26. ASYNCHRONICITY FACTS ? Normal scraping? ~30m @ 150 domains ? Desynchronized? ~8m @ 150 domains
- 28. Q?
- 29. FIN. 19.7.2019 MONOLITH WORKS INC.