Our My first DDoS attack - O'Reilly First DDoS attack Presentation... · Our My first DDoS attack ...…

  • Published on

  • View

  • Download


Our My first DDoS attackVelocity Europe 2011 BerlinCosimo StrepponeOperations Leadmy.opera.com/Ao-Trang-Oi/blog/nginx secret sauces?# Pavel's secret gzip tuning saucegzip on;gzip_disable msie6;gzip_min_length 1100;gzip_buffers 16 8k;gzip_comp_level 3;gzip_types text/plain application/xml application/x-javascript text/css;nginx secret sauces?# Michael's secret file cache sauceopen_file_cache max=1000 inactive=20s;open_file_cache_valid 30s;open_file_cache_min_uses 2;open_file_cache_errors on;nginx antidos.conf# More on https://calomel.org/nginx.htmlclient_header_timeout 5;client_body_timeout 10; ignore_invalid_headers on; send_timeout 10;# To limit slowloris-like attacksclient_header_buffer_size 4k; large_client_header_buffers 4 4k;# Cut abusive established connections,# forcing clients to reconnectlocation ~ ^/Ao-Trang-Oi/blog/ { return 444;}nginx drop client connectionsnginxbackendsvarnishnginx varnish cachingiptrafGET /Ao-Trang-Oi/blog/show.dml/14715682 HTTP/1.1 User-Agent: 1.{RND 10}.{RND 10} Referrer: http://my.opera.com/Ao-Trang-Oi/ Cache-Control: no-cache Cookie: __utma=218314117.745395330 [] __utmz=218314117.1286774593. [] utmcsr=google|utmccn= [] utmctr=cach%20de%20hoc%20mon []tcpdump of anomalous trafficGET /Ao-Trang-Oi/blog/?startidx=1295 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;) Gecko/20030624 Netscape/7.1 (ax) Accept: Accept=text/html,application/xhtml+xml,... Accept-Language: Accept-Language=en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: Accept-Charset=ISO-8859-1,... Referer: http://my.opera.com/Ao-Trang-Oi/blog/ Pragma: no-cache Keep-Alive: 300 ua-cpu: x86 Connection: closetcpdump of anomalous trafficcosimo: we're seeing a pretty "interesting" problem within our nginx frontscosimo: there's a few hosts sending a legitimate HTTP GET requestcosimo: followed by a binary stream of random bytes that never endscosimo: this is just 1 request going on and oncosimo: is there some way to alter the nginx config to shut down these client connections?cosimo: the client is sending something like:cosimo: GET /blah HTTP/1.1cosimo: Host: ...cosimo: Etc: etc...cosimo: and then random bullshit vr: :) vr: this is nkiller2 vr: haproxy can fight this vr: you can set a timeout http-request vr: don't know if nginx can do thiscosimo: coolOMGWTFBBQ!!!!11111this is nkiller2#nginx, 14th October 2010BLAH BLAH BLAH BLAH BLAH BLBLAH BLAH BLAHPHRACK#66tcp window zero?iptables -A -m u32 --u32 6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C () 12&0xFFFF=0x0000 -j ZERO_WINDOW_RECENTu32 zero window filter6 &0xFF =0x6 4 &0x1FFF =0x0 u32 zero window filteru32 zero window filter0>>22 &0x3C ()12 &0xFFFF =0x00>>22 &0x3C ()12 &0xFFFF =0x0??u32 zero window filter0>>22&0...@12&0xFFFF=0x00000>>22&0x3C@12&0xFFFF=0x00000>>22& [EMAIL PROTECTED] &0xFFFF=0x00000>>22&0x3C@12&0xFFFF=0x00000>>22 &0x3C @12 &0xFFFF =0x0u32 zero window filteriptables rules - logging$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent --set --name ZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent --update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j LOG --log-level info --log-prefix "ZeroWindow"~18k distinct IPsiptables rules - blocking$ipt -N ZERO_WINDOW_RECENT$ipt -A INPUT -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@12&0xFFFF=0x0000" -j ZERO_WINDOW_RECENT$ipt -A ZERO_WINDOW_RECENT -m recent set --name ZERO_WINDOW$ipt -A ZERO_WINDOW_RECENT -m recent update --seconds 60 --hitcount 20 --name ZERO_WINDOW -j DROPnginxbackendsvarnishshields-up.vclnon-cacheable contentcacheable contentnginxbackendsvarnishHTTPS-only trafficshields-up.vclall HTTP contentnginx feels better10s20s0sPingdom response timeEnd 29-Oct-2010Start 13-Oct-2010 End 29-Oct-2010Packets/s seen by firewallQuestions?What can we, as Ops, do better? Embrace failures and learn from them Be fast (no panic/blame, think Mr. Wolf) Coordinate (#ops, war rooms, ...) Take notes Learn TCP/IP Know your tools (tcpdump, tcpflow, strace, nc, iptraf, )my base_packages puppet moduleclass base_packages { $packagelist = [ "ack-grep", "colordiff", "curl", "facter", "git-core", "htop", "iftop", "iptraf", "jed", "joe", "libwww-perl", "logrotate", "lsof", "make", "mc", "oprofile", "psmisc", "rsync", "screen", "svn", "sysstat", "tcpdump", "tcpflow", "telnet", "unzip", "vim", "zip" ] package { $packagelist: ensure => "installed", }}Thanks to... ithilgore (sock-raw.org) for writing nkiller2 @vr in #nginx for pointing us at nkiller2 David Falloon for his great untested idea marc.info for correctly handling @ in ml SANS Institute for the TCP/IP references My team at OperaDanke!Our first DDoS attackMr. WolfSynoptic - 1Synoptic - 2Synoptic - 4Synoptic - 3AoTrangOi blog screenshotnginx secret sauce? / 1nginx secret sauce? / 2nginx, antidos configurationnginx, return 444; trickVarnish caching behind nginx frontendsiptraf screenshottcpdump of anomalous traffictcpdump of anomalous traffic / 2A random day on IRC and #nginxPhrack #66, exploit of TCP persist timertcp window == zerodailydave messageu32 - 6&0xFF=0x6u32 - 4 & 0x1FFF = 0u32 - 0>>22&0x3C () 12&0xFFFF=0x0000u32 - TCP packet headerdailydave message at mail-archivedailydave message at neohapsis.comdailydave message at lists.virus.orgdailydave message at marc.infou32, recv window=0iptables - logging zero window packetsDDoS geographic mapiptables - blocking zero window packetsshields up, infrastructure changeshields up, infrastructure change 2Slide 34Pingdom response time10 days more and the attack stopsfinal standingsVecebotVecebot - Analysis by Dell Secureworksquestions?Take aways!puppet base packagesmy thanksThanks!