MySQL Reverse DNS took us all day to debug and before reinstalling MySQL on another server guy who had this problem told us the solution: skip-name-resolve
After searching the net I’ve found that by default MySQL grants authentication on users based on IP address or hostnames. When there is a connection from a particular host, MySQL does a reverse DNS lookup and compares the hostname and IP address.
In our case there was some problem with the DNS server and the high traffic website with more then 70,000 visitors a day was down. We started looking into the memcache server, after that the nginx and the 2 Apache servers and at the end we have located the problem in the MySQL database.
Because the website was down for a long time we have decided to attach the database on one of the other servers and later find out what was the problem. But as the backup of the database was finishing a guy walks in and with a little smile on his face asked us: Do you have MySQL reverse DNS turned off? And there was the solution!
Here is the simple fix to this problem; just add the following line into your MySQL configuration file and restart the server.
[mysqld] # Skip reverse DNS lookup of clients skip-name-resolve
This saved us a lot of time and after this I’ve reconfigured all my MySQL servers with skip-name-resolve parameter.
For more informations see the MySQL reference manual for more details.