Monthly Archives: May 2015

Tons of Empty HTTP POST Requests to Your Web Server? It’s the Pushdo Botnet.

You’re checking your Apache logs one morning and you’re finding WebLogExpert taking a bit longer than usual to create your activity report.

That’s strange.  Maybe the log file was corrupted?

Checking your day’s log, you find the file is over a dozen times larger than usual.  Multiple megabytes over the usual size.  After opening the file in your text editor, you find it full of nonsensical blank entries.  Tens of thousands of lines of this:

Uh, oh.  For some reason, out of nowhere, a huge amount of garbage traffic has been arriving at your web site.  Each an empty POST request, with the requests arriving from hundreds of different IP addresses.

Checking your web server’s status through your control panel, you see a spike in CPU activity showed up along with those empty POST requests, making your report look something like this:

pushdo_cpu_activity

A nice, sleepy web server, until one day…

And your web server’s disk i/o has also spiked:

pushdo_disk_io_activity

And sudden disk activity, too

It’s not noticeably slowing your web site down yet, but it looks like that spike in activity is going to continue growing and it will soon affect your web site’s performance.  One way or another, seeing your web server spike in activity for absolutely no reason can’t be good.

Is someone attacking?  Is this the start of a denial of service?  A prelude to some kind of cyber-extortion?

Congratulations.  You’ve just found yourself the latest ‘winner’ of the Pushdo lottery.

Huh?  Pushdo?

Before we start explaining how to fix things, let’s take a look at the problem’s cause.

You’ve probably heard of botnets:  Networks of ‘zombie’ computers taken control of by viruses and joined together to do a variety of nasty things, from sending spam email to attacking web sites and taking them offline with huge bursts of traffic.  The traffic you’re seeing on your web site now is coming from one of those botnets.  This one’s referred to as Pushdo.

As explained through that link, the Pushdo botnet is an infrastructure of sorts to handle various computer crime tasks and, to hide its activity, it constantly blasts out traffic to make it difficult to spot the center of command for its work.  Your web server is now receiving one of those bursts of network traffic.

For one analogy, you can think of Pushdo acting like a thief who has zombified the population of a small town to help hide his crimes.  Using his zombie powers, the thief sets up thousands of people to constantly walk a city’s streets and stumble into its buildings and houses.  The zombies continue to stumble along, day and night, so that whenever the thief wants to rob a bank, he can just stumble off and quickly blend in with the crowd.  Sure, the city streets would grind to a halt as those thousands of zombies milled about 24 hours a day, but it would be reasonably easy for the thief to perform whatever crime he’d like and then slip away in the crowds, stumbling along with the rest of the zombies.

That ‘needle in a haystack’ situation is similar to what you’re experiencing with Pushdo.  Your web server has become a house filled with zombies, with Pushdo stuffing it full of web site traffic arriving from thousands of computers to hide its activity from anyone who might try to figure out which computer was sending meaningful criminal information to the other.

Pushdo isn’t attacking your web site directly, and your web server (probably) isn’t infected with Pushdo itself.  Your site is just an innocent bystander being used to hide activity.

So, why was your site chosen?  Simple bad luck, really.  Your site was one of hundreds picked at random from the world’s DNS records, and now you’re the target of traffic from thousands of botnet computers.

How Do I Stop This Surge in Zombie Traffic?

The bad news is, you can’t stop the surge itself and it’s probably going to continue hammering your web site for the next few months – possibly the next few years.  Pushdo will continue to hide its activity by using your web site as a random target and it will only stop once your site rotates off the list of zombie destinations used for camouflage.

While you can’t stop the traffic at its source, you can configure your web server to ignore the traffic as it arrives

However, while you can’t stop the traffic at its source, you can configure your web server to ignore the traffic as it arrives.  The tens of thousands of zombie messages will still come knocking at your web site every day, but you can set your computer to block those messages the moment they arrive on your network and keep them from reaching your web server.

To do this, you’ll need to set up a firewall rule to block all traffic which fits the Pushdo message signature.

Using iptables to Block Pushdo

We’re assuming here that you’re running your own web server on Linux, either as a stand-alone computer or through a hosted VPS package with Linode, Slicehost or another provider.  If your web site is instead being managed by a full-service hosting provider, your best action would be to contact your server support team and have them apply the appropriate traffic-blocking actions for you.

Continuing with the assumption that you’re running your own web server on Linux and have full control over that computer, here’s an overview of how to fix your Pushdo problem:

To block the zombie network traffic bombarding your web server, you’re going to use iptables.  The iptables utility is a network packet filter which you can use to regulate internet traffic.  Linode provides an excellent overview of the package in their documentation, and the raw help text for the tool can also be reviewed direct through the “man iptables” command on your server computer.

Watch out!  Since a mis-configuration of iptables can also block legitimate traffic (even your own login), you will want to review the documents linked above to gain a better understanding of exactly how the utility works before applying the examples given here.  It’s really easy to accidentally set iptables to block all network traffic arriving on your web server and, since such a mistake would block any attempts you might make to fix things, that could lead to irreparable disaster.  Trust me:  It’s worth the extra bit of effort to look over the documentation and understand at least the basics before proceeding.

Have you read those guides over?  Good.  Let’s put them to work.

In reviewing those documents, we can see that it’s very easy to configure iptables to block individual IP addresses.  That’s really helpful if there’s a particular nuisance source of traffic you’d like to block but, in this case, the Pushdo traffic is arriving from many thousands of different IP addresses and it would be extremely tedious to enter a blocking rule for each of them every time a new address showed up.

What we’re going to do instead is block the arriving traffic based upon its contents.  As you might have noticed when examining your Apache logs, all of the Pushdo traffic has the same general signature of:

For all of the messages received, there’s a consistent POST command of either:

‘POST / HTTP/1.1’
or
‘POST / HTTP/1.0’

So, if we can detect that particular bit of text in a received packet, and that text is not used in any legitimate traffic accepted by your web server, we can safely set up a rule to block anyone sending a message which contains that particular blurb of text.

Ah!  But your own web site might have legitimate traffic with that same signature!  Before implementing this particular fix (which worked for us, as there were no legitimate POST requests on our own web server which were blank), you will want to search through your past Apache logs to confirm the text being scanned for is not used by legitimate visitors to your site.  If it is, you would best select additional criteria to examine for blocking the message (for example, adding in the ‘MSIE 6.0; Windows NT 5.1; SV1’ bit, since blocking users of IE 6 who send along POST requests is probably not going to nab legitimate traffic).

The iptables Commands to Use For Blocking Pushdo

To block the two snippets of text identified above, you will want to apply the following iptables rules as root (preferably, using sudo):

Where [your web server IP address] is the IP address of the web server computer on which iptables is running.

Taking a look at the contents of that command, this is what it is doing:

iptables : running iptables
-I : inserting a rule
INPUT : for packets going to local sockets
-d [your web server IP address] : heading to the computer with this IP address
-p : now, here comes the rule:
tcp –dport 80 : for tcp traffic arriving on port 80
-m string : use the string module
–string ‘POST / HTTP/1.1’ : so that any traffic matching the string ‘POST / HTTP/1.1’
–algo bm : which we’re identifying using the Boyer-Moore (bm) string-matching algorithm
-j DROP : will immediately jump to the DROP rule the moment it’s spotted.

After applying those rules to iptables, any network activity spotted which contains the text ‘POST / HTTP/1.1’ will be dropped immediately, closing your web site’s door to those zombies.

So, What Next?

With the iptables rules applied, you’re done.  Pushdo traffic will still come knocking at your web server with zombie traffic, but iptables will prevent that traffic from entering your web site.  You can confirm that by checking your web server’s log files a few minutes after applying those rules:  You should see the blank POST entries suddenly stop.

Your computer will still show higher CPU and disk I/O usage than usual, it just won’t be as bad as it could have been and (with luck) won’t noticeably affect the performance of your web site.  For one example, the charts shown in the earlier part of this article display the result of applying those iptables rules the day Pushdo traffic first arrived on our site.  In our case, there was an initial dip in CPU usage as the first round of Pushdo traffic was handled, followed by a surge as the real torrent of zombies arrived.  Our iptables utility was kept pretty busy that month, but website performance was not affected.

Remember:  Your iptables rules will be cleared out after every reboot of your web server, so you must be sure to apply those rules after every reboot.  If you don’t yet have a checklist of steps to follow whenever rebooting your web server, now would be a good time to create one.

Your iptables rules will need to remain in place until Pushdo moves on from your web site, and that will probably be quite some time.  I’d recommend adding a note to your server reboot checklist reminding you to leave out your Pushdo iptables rules every three months to see if that unwanted traffic has disappeared.  After rebooting your web server with those rules removed, you can then check your Apache logs after an hour to see if those telltale blank POST messages are still appearing.

You should find that the Pushdo traffic drops off in intensity after about six months but still remains in place for a bit over a year after its first appearance.  However, one day it will disappear completely and, when that day comes, you can stop applying those iptables rules to your web server.

Be patient.  Being a target of Pushdo is a pain, but there’s nothing that can be done about it but to batten down your web server’s ports and leave iptables to do its filtering job.

Oculus Rift Virtual Reality Initial Impressions

Here are my impressions after having the Oculus Development Kit (DK2) in-hand for a bit over two weeks now:

The Technology Behind Oculus

The hardware itself will be familiar to anyone who has developed software for a touch screen phone. Described generally, the head-mounted display is a cell phone screen placed in a headset mount. A gyroscope, magnetometer and accelerometer provide instrumentation, with an accompanying camera used to identify the position of the wearer’s head. In all, the DK2 seems a perfect example of innovation, where the device introduces no new technologies but is instead a recombination of existing parts to create something wholly new.

dk2_image

A remarkable innovation of existing technology

That doesn’t in any way diminish how much of a marvel the device is, but it does suggest that, as with cell phones, VR headsets are likely to drop in price when created in volume and may be expected to improve in quality at a rapid pace.

Oculus Immersion

The screen’s low resolution does produce a bit of a ‘screen door’ effect, with the LCD pixels very easy to see.  The sensation is a bit like sitting too close to an old CRT television set.  Since the DK2 is a prototype development kit (and not the final product) that lower image quality wasn’t surprising, and I found the immersive experience of the headset allowed me to very quickly dismiss quibbles about resolution.

The sense of presence is very effective. Even with the ‘seated at a desk’ test display, I found myself reaching out to confirm that what I saw on the desk before me wasn’t actually there . It’s clear that the environment one is looking at is a set of video game graphics, but one’s mind is easily convinced that what is being seen is actually real.

ocean_rift

Feel like you’re swimming in the ocean while seated at your desk

Along with that immersion, an awareness of my actual surroundings disappeared almost entirely at times, and I sometimes found myself startled to feel my hands touch a desk or chair when my eyes saw open air around me. I’m not sure if I would trust myself walking with a VR headset, and it will be interesting to see how Steam VR overcomes that sense of displacement from reality with their own plans for a headset which encourages standing, mobile user operation.

In the virtual environment, I felt a sense of apprehension whenever deliberately putting myself in danger.  For one example, when swimming about in the Ocean Rift demo, I was reluctant to descend into the depths where the giant shark (which you’re warned about) was waiting to attack. Even though I was obviously not really swimming beneath the waves and diving deep into a dark ocean trench, it was a spookily genuine experience all the same.  Given my uneasiness with something so tame and G-rated, there is no way in hell I’d play a horror game like Alien Isolation in VR.

It will be interesting to see if that sense of dread diminishes once my brain becomes accustomed to the VR environment and learns about a new thing it can safely ignore. I remember telling my friends about the sense of total immersion I felt playing a pixelated 3D Quake, 640×480, on a 14 inch monitor, and I can’t help but wonder if my initial Oculus experiences will sound just as exaggeratedly real when looking back on this technology in the future.

quake_500x375

So real! It’s like you’re there!  (It felt like that in 1995.  Honest!)

Surprisingly, the various roller coaster demos (which famously freak people out) did very little for me. I’m the type of heights-dreading person who feels apprehension just watching a parkour video and ends up with sweaty palms after playing a game like Mirror’s Edge on a regular monitor, but I found the Oculus roller coasters to be a bit underwhelming.  I suspect that will change once issues of judder and resolution are resolved.

Oculus Nausea

The nausea experienced when first using the DK2 was surprisingly intense.  Just a few minutes of wandering around the low-intensity Tuscany Villa demo had me feeling queasy, and about 10 seconds flying in the AirDrift demo almost made me vomit. Regardless of the settings or passiveness of the environment, for the first few days it was not possible for me to use the DK2 for more than a few minutes at a time.  An extended period of use would see me wearing the device for no more than 10 minutes, and only with tremendous effort.

oculus_tuscany_700x379

Quiet, peaceful, and more nausea-inducing than an economy-class flight through a thunderstorm

The nausea was of a surprisingly intense and persistent sort. Whenever I’ve experienced motion sickness when playing games, the sensation tended to grow gradually and quickly diminished once I’d left the computer screen. However, VR sickness seemed to onset immediately and persisted for a bit over two hours after I’d left the device behind. A sense of dizziness and unsteadiness remained after taking off the DK2, and I’d recommend anyone looking to try a VR headset ensure they’re not needing to operate machinery or a vehicle for a few hours after their first attempt — I’d never have trusted myself in a car or on a bike at that point.

The nausea and dizziness was so intense at first that I was concerned I would not be able to use VR at all. Fortunately, this has turned out not to be the case.

One quickly learns not to change position too quickly while wearing the VR headset. The sensation when swiftly turning my virtual body’s view about was as if blood were being pressed to the side of my brain to cushion it from a sudden shock. Whether that was an illusion or an actual reaction from my nervous system is something I’ll leave to the biologists and physicians, but the unpleasantness of the experience was very real.

I gained my ‘VR legs’ rather quickly and the discomfort was considerably reduced after a week of occasional use. Nausea diminished with each session using the DK2, and use of it for extended periods (30+ minutes) after about 10 days barely registered any sense of discomfort at all. It then became possible to skip through the various Oculus demonstration apps with ease and play games for an hour or so at a time without any difficulty. Flying in AirDrift became a breeze. Use of the device soon began to feel so natural that I would have likely misremembered my earlier nausea as being considerably less intense if I’d not made notes about it at the time.

airdrift

Give it a few days, and even flying through the air starts to feel natural

It’s unclear to me whether my diminishing nausea was due to my senses growing accustomed to the VR environment, or my having optimized my computer’s drivers and configuration to ensure the DK2 was being used in the best possible fashion. Others who have tried the DK2 device on my computer after I had finished its configuration and video driver optimization experienced no nausea issue whatsoever. So, either I should have spent more time configuring my system’s drivers and DK2 configuration before starting to use the device, or I was just an unlucky outlier who was particularly susceptible to VR motion sickness.

Gaming With Oculus

The Direct to Rift mode for the DK2 worked flawlessly, with all games and tech demos suited for that mode running without issue. However, setting up Extend Desktop mode on my own computer to use Steam was a difficult experience which reminded me of the ‘DLL hell’ presented when installing a video card on Windows 3.1. That difficulty is no fault of Oculus or any other hardware/software vendor: The DK2 is beta development hardware, and my difficulties could have come from a variety of sources, from video card drivers to my monitor setup under Windows. I only mention these issues as a reminder that the DK2 is not consumer hardware:  Everything might work for you right out of the box at this point, but anyone who purchases a DK2 should be prepared to spend a good amount of time configuring their system to ensure it works correctly.

hl2_headcrab

It turns out these guys are actually kinda big

Once Extend Desktop mode was enabled for my DK2, I tried out Half-Life 2 using Steam’s beta VR client and found the game a new experience entirely. The screen door effect is still there, with pixels visible when you’re looking for them, but the sensation of actually feeling ‘inside’ the game allowed me to quickly set that roughness aside. Even though the game was familiar to me, the VR immersion allowed me to pick out scenery details I’d not seen before, and I gained a sense of scale of my surroundings which had previously been missing — I had no idea head crabs were so big!

Video With Oculus

Virtual movie theaters and home entertainment setups seem to be one of the most popular initial applications of VR displays, and I believe video will end up becoming the most popular use of the Oculus and similar devices in the future. Even early in my use of the device, I did not feel nausea when using the DK2 to passively view video and that use seems likely to be the gateway through which wider adoption of VR headsets will occur.

A personal, isolated virtual movie theatre

The ability to set whatever environment one would like for movie viewing (whether in a living room, your choice of theater and row, or just a floating screen), combined with the full sense of immersion and lack of interruption from the outside world delivered by the headset provides the possibility of a perfect movie experience. A VR display delivers the option of viewing regular two-dimensional or fully 3D video in whatever format one would like, whether an IMAX-sized screen, a fully surrounding 3D environment, or a small corner television set, all while remaining seated or reclined in a favourite chair, couch or bed. The low resolution of the DK2 still delivers the feeling of sitting a bit too close to an old CRT television, but the higher resolution to be expected from the consumer Oculus device will likely resolve that distraction entirely in time.

There is also the possibility of interactive movies, or fully 360’ degree movie experiences which put you in the role of a character of your choice. The Birdy King Land Oculus demo gives a really good example of that – you’re a character IN the movie. However, as fun as interactivity and gaming is with the Oculus, easily delivered passive entertainment seems likely to become the ‘killer app’ for head-mounted VR displays. Often, people just want to lie back and relax, and a personal, isolated virtual movie theatre has the potential to perfectly deliver that experience on-demand.

Oculus Tips From Experience

If I could go back in time to the moment I started using the DK2, I’d give myself these bits of advice:

You should use the Rift configuration tools to set your Inter-Pupillary Distance (IPD) rather than using the value measured for your glasses. IPD for the Rift (and presumably other VR headsets) should match the value used when your eyes are focused to infinity (informally confirmed here by Oculus Founder Palmer Luckey), which is what the configuration tool provides. Using the IPD value set for your glasses will probably be incorrect.

Ensure your video drivers are up to date. If using an NVidia card, install and use the NVidia ‘GeForce Experience’ tool to simplify the upgrade of your video drivers. It saves a lot of hassle when identifying and applying the correct driver version.

Contact lenses are helpful but not necessary if you can see well without glasses. With natural eyesight, I would be incapable of reading a cell-phone held at Oculus distance from my eyes, but the corrective optics of the device’s lenses extend that distance to remove any difficulty with viewing. VR resolution isn’t sharp enough yet to justify concerns with fine visual detail, but that will probably change with consumer versions of the device.

Be sure to move slowly in the VR environment. If you’re used to flicking the mouse around for a quick change in orientation in first-person shooter games, using the same gesture to move about in VR will probably feel uncomfortable at first and lead to nausea.

If you’re taller than average, set the Player Height value in the Oculus settings to be lower. The people creating Oculus games and experience demos are likely to be of average height, and the environments will often feel uncomfortably cramped (as if you’re about to hit your head on the ceiling) if they are set to take into account player height. Just set your configuration height at 5’5” and you’re good.

Should I Buy An Oculus Rift DK2?

If you’re looking to develop VR software right now then, yes, you should buy an Oculus DK2. You should be prepared to fiddle with system settings, drivers, incomplete documentation and the other hassles which come with being on the ‘bleeding edge’ of a technology, but the device is good to go for development.

If you’re considering the development of VR software, are curious about the experience or just want to play games with the system, you should wait. The consumer versions of the Oculus Rift are going to be superior to the development kits in pretty much every way, and might even be less expensive at first.  It will be worth the anticipation to wait a few more months for those to arrive since, once consumer versions of VR headsets are available, the early implementations now available with the Oculus DK1 and DK2 are probably going to be set aside to gather dust.