際際滷

際際滷Share a Scribd company logo
Robot Attack!
Repelling Bots, DDOS, and other Fiends
Stanford Drupal Camp 2015
Suzanne Aldrich
Senior Customer Success Engineer - Pantheon
Martijn Gonlag
Technical Support Engineer - CloudFlare
MEET YOUR GUIDES
Surveying Robots
Detecting Attacks
Evading Spam
Withstanding High Traffic
Questions
AGENDA
Internet bot:
 Robot, WWW bot, bot, botnet, zombies
 Automated scanning of website
resources at high rate
 Good bots: Web spiders
 Googlebot
 MSNBot/Bingbot
 Baidu
 Yandex
 Pingdom
Drupals robots.txt
https://api.drupal.org/api/drupal/robots.
txt/7
User-agent: *
Crawl-delay: 10
Disallow: /includes/
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /install.php
Disallow: /update.php
Disallow: /xmlrpc.php
HISTORY OF THE ROBOT
Bad bots:
 Spambots - advertising links
 Email harvesters
 Downloaders & scrapers
 Referral & click fraud
 Rogue spiders
 MegaIndex:
Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.
megaindex.ru/?tab=linkAnalyze)
 Infectious agents
 Botnets & zombies
BAD BOTS
Id Date Severit Type
Message
3161818 16/Jun 16:45 notice spambot
Blocked registration: email=supplyweqz@gmail.
com,ip=120.43.21.95
3161817 16/Jun 16:45 notice user
Login attempt failed for JulianHut.
3161794 16/Jun 16:44 notice user
Login attempt failed for Julianml.
DETECTING ATTACKS
Common SPAM Defense Methods:
 CAPTCHA - Completely Automated Public
Turing test to tell Computers and
Humans Apart
 Timegate (Time Difference)
 Honeypot
 Content analysis
 Visitor reputation
Popular Drupal Modules:
CAPTCHA/reCAPTCHA - https://www.drupal.
org/project/captcha
https://www.drupal.org/project/recaptcha
Mollom - https://www.drupal.org/project/mollom
Honeypot - https://www.drupal.org/project/honeypot
Antispam - https://www.drupal.org/project/antispam
Spambot - https://www.drupal.org/project/spambot
CloudFlare - https://www.drupal.org/project/cloudflare
Spam prevention - https://groups.drupal.org/node/77093
EVADING SPAM
Problems with CAPTCHA:
 Cookies prevent anonymous caching
 High traffic sites require edge cache
 Usability
 Inconvenient
 Barrier
 Accessibility
 Visual impairment
ANTI-SPAM STRATEGIC PITFALLS
Problems with External APIs:
 3rd party dependency
 Availability & rate limiting
 CAPTCHA fallback
 Cost of service
 User Privacy
 Poor performance + bots = downtime
 Server and log monitoring
 Fix site errors in module code and theme
templates
 Anonymous page caching
 Views query and rendered results caching
 Dedicated cacheserver - Redis
 Disable comments/cookies/statistics
 Setup CDN for serving assets
 Block IPs at firewall
 Withstand many Layer 7 attacks
WITHSTANDING HIGH TRAFFIC
$ curl -Ik http://www.example.
com/comment/reply/12345
...
X-Varnish: 3649165893
Age: 0
Via: 1.1 varnish
Connection: keep-alive
Vary: Cookie, Cookie
 Cloud-based SaaS
 Reverse Proxy
 Security
 Performance
 Optimization
 CDN
 DNS
CLOUDFLARE SECURITY
D0000 - Block Large Requests to xmlrpc.php for Drupal CMS
D0002 - Block requests with odd array arguments
D0001 - Block Requests to xmlrpc.php for Drupal CMS
URIs:
/xmlrpc.php -- most common
/?q=node&destination=node
/blog/xmlrpc.php
/user/login/
HTTP Method:
POST -- most common
GET
10.223.224.238 - - [05/Feb/2015:23:34:47 +0000] "POST /xmlrpc.
php HTTP/1.1" 404 5377 "-" "Mozilla/4.0 (compatible: MSIE 7.0;
Windows NT 6.0)" 0.251 "5.189.129.224, 108.162.254.28,
10.183.251.3"
10.223.224.238 - - [05/Feb/2015:23:34:47 +0000] "GET /feed/
HTTP/1.1" 200 6354 "http://example.com/feed/" "SimplePie/1.3.1
(Feed Parser; http://simplepie.org; Allow like Gecko)
Build/20140407093003" 0.201 "54.216.178.194, 141.101.98.27,
10.183.251.3"
10.223.193.24 - - [05/Feb/2015:23:34:47 +0000] "POST /xmlrpc.
php HTTP/1.1" 404 5377 "-" "Mozilla/4.0 (compatible: MSIE 7.0;
Windows NT 6.0)" 0.233 "5.189.129.224, 108.162.254.28,
10.183.251.3"
CLOUDFLARE DRUPAL WAF RULES
Frequency of WAF Triggers Over 30 Days Percentage of Triggers by WAF Rule
CLOUDFLARE DRUPAL WAF TRIGGERS
CONNECT WITH US!
https://twitter.com/SuzanneAldrich https://twitter.com/MartijnGonlag

More Related Content

Stanford Drupal Camp 2015 - Repelling Bots, DDOS, and other Fiends

  • 1. Robot Attack! Repelling Bots, DDOS, and other Fiends Stanford Drupal Camp 2015
  • 2. Suzanne Aldrich Senior Customer Success Engineer - Pantheon Martijn Gonlag Technical Support Engineer - CloudFlare MEET YOUR GUIDES
  • 3. Surveying Robots Detecting Attacks Evading Spam Withstanding High Traffic Questions AGENDA
  • 4. Internet bot: Robot, WWW bot, bot, botnet, zombies Automated scanning of website resources at high rate Good bots: Web spiders Googlebot MSNBot/Bingbot Baidu Yandex Pingdom Drupals robots.txt https://api.drupal.org/api/drupal/robots. txt/7 User-agent: * Crawl-delay: 10 Disallow: /includes/ Disallow: /CHANGELOG.txt Disallow: /cron.php Disallow: /install.php Disallow: /update.php Disallow: /xmlrpc.php HISTORY OF THE ROBOT
  • 5. Bad bots: Spambots - advertising links Email harvesters Downloaders & scrapers Referral & click fraud Rogue spiders MegaIndex: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www. megaindex.ru/?tab=linkAnalyze) Infectious agents Botnets & zombies BAD BOTS
  • 6. Id Date Severit Type Message 3161818 16/Jun 16:45 notice spambot Blocked registration: email=supplyweqz@gmail. com,ip=120.43.21.95 3161817 16/Jun 16:45 notice user Login attempt failed for JulianHut. 3161794 16/Jun 16:44 notice user Login attempt failed for Julianml. DETECTING ATTACKS
  • 7. Common SPAM Defense Methods: CAPTCHA - Completely Automated Public Turing test to tell Computers and Humans Apart Timegate (Time Difference) Honeypot Content analysis Visitor reputation Popular Drupal Modules: CAPTCHA/reCAPTCHA - https://www.drupal. org/project/captcha https://www.drupal.org/project/recaptcha Mollom - https://www.drupal.org/project/mollom Honeypot - https://www.drupal.org/project/honeypot Antispam - https://www.drupal.org/project/antispam Spambot - https://www.drupal.org/project/spambot CloudFlare - https://www.drupal.org/project/cloudflare Spam prevention - https://groups.drupal.org/node/77093 EVADING SPAM
  • 8. Problems with CAPTCHA: Cookies prevent anonymous caching High traffic sites require edge cache Usability Inconvenient Barrier Accessibility Visual impairment ANTI-SPAM STRATEGIC PITFALLS Problems with External APIs: 3rd party dependency Availability & rate limiting CAPTCHA fallback Cost of service User Privacy
  • 9. Poor performance + bots = downtime Server and log monitoring Fix site errors in module code and theme templates Anonymous page caching Views query and rendered results caching Dedicated cacheserver - Redis Disable comments/cookies/statistics Setup CDN for serving assets Block IPs at firewall Withstand many Layer 7 attacks WITHSTANDING HIGH TRAFFIC $ curl -Ik http://www.example. com/comment/reply/12345 ... X-Varnish: 3649165893 Age: 0 Via: 1.1 varnish Connection: keep-alive Vary: Cookie, Cookie
  • 10. Cloud-based SaaS Reverse Proxy Security Performance Optimization CDN DNS CLOUDFLARE SECURITY
  • 11. D0000 - Block Large Requests to xmlrpc.php for Drupal CMS D0002 - Block requests with odd array arguments D0001 - Block Requests to xmlrpc.php for Drupal CMS URIs: /xmlrpc.php -- most common /?q=node&destination=node /blog/xmlrpc.php /user/login/ HTTP Method: POST -- most common GET 10.223.224.238 - - [05/Feb/2015:23:34:47 +0000] "POST /xmlrpc. php HTTP/1.1" 404 5377 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 0.251 "5.189.129.224, 108.162.254.28, 10.183.251.3" 10.223.224.238 - - [05/Feb/2015:23:34:47 +0000] "GET /feed/ HTTP/1.1" 200 6354 "http://example.com/feed/" "SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20140407093003" 0.201 "54.216.178.194, 141.101.98.27, 10.183.251.3" 10.223.193.24 - - [05/Feb/2015:23:34:47 +0000] "POST /xmlrpc. php HTTP/1.1" 404 5377 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" 0.233 "5.189.129.224, 108.162.254.28, 10.183.251.3" CLOUDFLARE DRUPAL WAF RULES
  • 12. Frequency of WAF Triggers Over 30 Days Percentage of Triggers by WAF Rule CLOUDFLARE DRUPAL WAF TRIGGERS
  • 13. CONNECT WITH US! https://twitter.com/SuzanneAldrich https://twitter.com/MartijnGonlag