Yesterday my web sites were down for a short time. I logged on my server and my first reaction was restart the apache web server. After that everything was working fine. So I start to check the ganglia metrics to see what happened. It was clear that one minute before I restarted apache, the number of requests to the web server was very high, surpassing the limits of Apache and blocking other requests.
I checked manually the apache logs, filtering the traffic minutes before and after the restart. There were no signs of something wrong. I also analyze the logs with some tools (awstats, bots script, etc) with similar results. I do the same with the error logs, checking carefully for some strange behaviour. No success.
So I'm pretty sure the problem was a sudden increment of the requests to the apache web server. But I don't know how this happened, if this was an attack, some nasty bug, a problem in the application, or something else I don't know. What will you do if something similar happens in your web server? What other tools you use? What other logs you check? Was it wrong restart the web server as the first measure to solve the problem?