2009/09/21 02:34 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 02:42 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.898 second response time 2009/09/21 02:54 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 03:02 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.691 second response time 2009/09/21 04:20 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 04:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.695 second response time 2009/09/21 05:08 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 05:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 6.827 second response time 2009/09/21 05:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 06:00 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.209 second response time 2009/09/21 06:05 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 07:23 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.818 second response time .. 2009/09/21 08:10 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 08:38 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.633 second response time 2009/09/21 08:41 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds 2009/09/21 08:41 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds 2009/09/21 08:43 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds hi hey checking on dev, yea CPU is under load tks 2009/09/21 09:09 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 0.156 second response time 2009/09/21 09:09 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 0.675 second response time 2009/09/21 09:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.387 second response time 2009/09/21 09:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 09:29 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.777 second response time 2009/09/21 09:55 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 10:03 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.257 second response time 2009/09/21 11:09 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 11:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.156 second response time 2009/09/21 12:03 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 12:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.380 second response time 2009/09/21 12:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds / 2009/09/21 13:09 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.181 second response time 2009/09/21 13:21 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 13:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.014 second response time 2009/09/21 14:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 14:22 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.846 second response time 2009/09/21 14:33 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 15:01 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.370 second response time 2009/09/21 15:46 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 16:44 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.319 second response time 2009/09/21 17:00 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 17:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.930 second response time 2009/09/21 17:58 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 18:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.320 second response time 2009/09/21 18:36 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds .. 2009/09/21 18:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.993 second response time 2009/09/21 19:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 19:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.725 second response time 2009/09/21 19:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 19:50 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.781 second response time 2009/09/21 20:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/21 21:34 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds 2009/09/21 21:34 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds amette http critical on dev.tikiwiki.org the site does not load what did you do? we restart the apache] its ok now] 2009/09/21 22:02 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 3.696 second response time in extreme cases wait for the load to go down before starting apache again ok will do doc.tikiwiki.org site loads but there is http-features critical still 2009/09/21 22:07 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 5.738 second response time yeah, the setup is not very well performing ok apache was not coming down cleanly on web1, I killed it with -9, now waiting for the load to come down ok uuuuuhm.... now that looks bad... ... apache doesn't start! :( should we try to restart again? 2009/09/21 22:12 CRIT web1 HTTP Connection refused 2009/09/21 22:12 CRIT web1 TikiWiki Connection refused try finding, why it doesn't start ok il try 2009/09/21 22:35 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.021 second response time I shut the machine down - checking file-system offline ok 2009/09/21 22:37 CRIT web1 CRITICAL - Host Unreachable (10.100.100.11) 2009/09/21 22:36 ?? web1 Disk Error reading table : Timeout 2009/09/21 22:37 CRIT web1 PING CRITICAL - Host Unreachable (10.100.100.11) 2009/09/21 22:37 ?? web1 Swap Error reading table : Timeout 2009/09/21 22:38 CRIT web1 SSH No route to host 2009/09/21 22:42 OK web1 PING OK - Packet loss = 0%, RTA = 0.23 ms 2009/09/21 22:42 OK web1 PING PING OK - Packet loss = 0%, RTA = 2.46 ms 2009/09/21 22:42 OK web1 Swap OK : Swap space: 0%used(0MB/1024MB) : < 80 % 2009/09/21 22:40 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds /etc/init.d/apache2 was an empty file I copied the one from web2 over and now apache starts again 2009/09/21 22:43 OK web1 SSH SSH OK - OpenSSH_5.2 (protocol 2.0) ok cool file system check failed with "cannot determine device size" *shrug* I'm happy it wasn't a bigger emergency, but I'm scared of how big it was.... ;) 2009/09/21 22:46 OK web1 Disk OK : /: 74%used(5296MB/7158MB) : < 80 % 2009/09/21 22:12 CRIT web1 HTTP Connection refused 2009/09/21 22:12 CRIT web1 TikiWiki Connection refused a message from the past?!?!? ?? look at nagios times there is all the time this one query hanging on db0.tw.o :( it gets worse... ya 2009/09/21 23:33 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.159 second response time 2009/09/22 01:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds 2009/09/22 01:20 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.529 second response time