This site had been running for a years with no big issues. I had performance and system saturation issues about 3 years ago and then move the site to DigitalOcean.com.
DigitalOcean.com is inexpensive and the performance is awesome.
Then last Monday and every day since the site had been going down. Simplest “solution” for me was just to get on and bounce the machine.
That cleared it up.
After this went on for a few days I contacted DigitalOcean saying I didn’t see how it could be an issue with them, but I’d asked anyway.
Sure enough they had identified the issue, which had nothing to do with them, and gave me the solution.
Basically the problem is a DoS attack using xmlrpc.php from WordPress.
To verify this I looked into the logs and sure enough there is rapid access to xmlrpc.php
cd /var/log/apache2
grep xmlrpc *
other_vhosts_access.log.1:104.131.152.183:80 37.1.214.203 – – [14/Aug/2016:10:09:25 -0400] “POST /xmlrpc.php HTTP/1.1″ 500 569 “-” “-”
other_vhosts_access.log.1:104.131.152.183:80 37.1.214.203 – – [14/Aug/2016:10:09:25 -0400] “POST /xmlrpc.php HTTP/1.1″ 500 0 “-” “-”
other_vhosts_access.log.1:104.131.152.183:80 37.1.214.203 – – [14/Aug/2016:10:09:26 -0400] “POST /xmlrpc.php HTTP/1.1″ 500 569 “-” “-“
There are number of solutions, but the easiest for me was to use the Jetpack plugging which comes with a “protect” option. After activating the protect options sure enough the xmlrpc.php access stop. In the following “grep” we see xmlrpc.php rapid access just before the Jetpack option is turned on then they stop. Yay.
root@datavirtualizer:/var/log/apache2# date
Mon Aug 22 12:58:07 EDT 2016
root@datavirtualizer:/var/log/apache2# grep rpc other_vhosts_access.log
104.131.152.183:80 154.16.199.74 – – [22/Aug/2016:07:33:28 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.19.7″
104.131.152.183:80 64.137.253.68 – – [22/Aug/2016:09:01:24 -0400] “POST /xmlrpc.php HTTP/1.1″ 500 569 “https://google.com/” “Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36″
104.131.152.183:80 114.44.230.12 – – [22/Aug/2016:09:01:52 -0400] “POST /xmlrpc.php HTTP/1.1″ 500 569 “https://google.com/” “Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36″
104.131.152.183:80 163.172.177.30 – – [22/Aug/2016:09:17:56 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.19.5 libcurl/7.38.0 GnuTLS/3.3.8 zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3″
104.131.152.183:80 163.172.174.255 – – [22/Aug/2016:09:17:59 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.24.0″
104.131.152.183:80 163.172.179.147 – – [22/Aug/2016:09:18:00 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.19.5 libcurl/7.38.0 GnuTLS/3.3.8 zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3″
104.131.152.183:80 163.172.175.207 – – [22/Aug/2016:09:18:03 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.24.0″
104.131.152.183:80 154.16.199.74 – – [22/Aug/2016:09:40:36 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.19.7″
104.131.152.183:80 154.16.199.74 – – [22/Aug/2016:12:11:17 -0400] “GET /xmlrpc.php HTTP/1.1″ 500 569 “-” “PycURL/7.19.7″
104.131.152.183:80 195.212.29.168 – – [22/Aug/2016:12:54:25 -0400] “GET /xmlrpc.php?rsd HTTP/1.1″ 200 995 “-” “Mozilla/4.0 (compatible;)”
104.131.152.183:80 195.212.29.168 – – [22/Aug/2016:12:54:26 -0400] “GET /xmlrpc.php HTTP/1.1″ 405 281 “-” “Mozilla/4.0 (compatible;)”
104.131.152.183:80 216.81.94.75 – – [22/Aug/2016:12:55:56 -0400] “GET /xmlrpc.php?rsd HTTP/1.1″ 200 995 “-” “Mozilla/4.0 (compatible;)”
104.131.152.183:80 216.81.94.75 – – [22/Aug/2016:12:55:56 -0400] “GET /xmlrpc.php HTTP/1.1″ 405 281 “-” “Mozilla/4.0 (compatible;)”
Let’s see if the site stays up now.