***: dszetu has left nagios: 2009/09/21 02:34 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 02:42 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.898 second response time ***: deepaks has left
deepaks has joined #tikiwiki-monitor
dszet1 has joined #tikiwiki-monitor
dszet1 has left
dszet1 has joined #tikiwiki-monitor nagios: 2009/09/21 02:54 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 03:02 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.691 second response time ***: rupeni has joined #tikiwiki-monitor
franck has quit IRC (Read error: 113 (No route to host)) nagios: 2009/09/21 04:20 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 04:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.695 second response time
2009/09/21 05:08 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 05:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 6.827 second response time
2009/09/21 05:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 06:00 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.209 second response time
2009/09/21 06:05 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: marclaporte has quit IRC (Read error: 60 (Operation timed out)) nagios: 2009/09/21 07:23 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.818 second response time ***: dszet1 has left
deepaks has quit IRC (Remote closed the connection)
melaia has joined #tikiwiki-monitor
franck has joined #tikiwiki-monitor
ChanServ sets mode: +o franck melaia: .. ***: melai1 has joined #tikiwiki-monitor nagios: 2009/09/21 08:10 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: rupeni has left
rupeni has joined #tikiwiki-monitor
srishti has joined #tikiwiki-monitor nagios: 2009/09/21 08:38 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.633 second response time
2009/09/21 08:41 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds
2009/09/21 08:41 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds
2009/09/21 08:43 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: marclaporte has joined #tikiwiki-monitor marclaporte: hi srishti: hey
checking on dev, yea
CPU is under load marclaporte: tks nagios: 2009/09/21 09:09 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 0.156 second response time
2009/09/21 09:09 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 0.675 second response time
2009/09/21 09:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.387 second response time
2009/09/21 09:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: franck has quit IRC () nagios: 2009/09/21 09:29 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.777 second response time ***: srishti has quit IRC ("Leaving.")
srishti has joined #tikiwiki-monitor nagios: 2009/09/21 09:55 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 10:03 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.257 second response time ***: franck has joined #tikiwiki-monitor
ChanServ sets mode: +o franck nagios: 2009/09/21 11:09 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 11:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.156 second response time
2009/09/21 12:03 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 12:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.380 second response time
2009/09/21 12:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds melai1: / ***: marclaporte has quit IRC (Read error: 113 (No route to host)) nagios: 2009/09/21 13:09 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.181 second response time
2009/09/21 13:21 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 13:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.014 second response time
2009/09/21 14:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 14:22 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.846 second response time
2009/09/21 14:33 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: epeli_e has joined #tikiwiki-monitor nagios: 2009/09/21 15:01 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.370 second response time
2009/09/21 15:46 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: epeli_e has quit IRC (Remote closed the connection)
epeli_e has joined #tikiwiki-monitor nagios: 2009/09/21 16:44 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.319 second response time
2009/09/21 17:00 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 17:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.930 second response time
2009/09/21 17:58 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 18:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.320 second response time
2009/09/21 18:36 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: melaia has left
franck_ has joined #tikiwiki-monitor
ChanServ sets mode: +o franck_ melai1: .. ***: franck has quit IRC (Read error: 113 (No route to host))
franck_ is now known as franck nagios: 2009/09/21 18:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.993 second response time
2009/09/21 19:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds
2009/09/21 19:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.725 second response time
2009/09/21 19:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: srishti has left nagios: 2009/09/21 19:50 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.781 second response time
2009/09/21 20:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: melai1 has left
jasprit has joined #tikiwiki-monitor
jasprit has quit IRC (Remote closed the connection)
jasprit has joined #tikiwiki-monitor nagios: 2009/09/21 21:34 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds
2009/09/21 21:34 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds jasprit: amette
http critical on dev.tikiwiki.org
the site does not load amette: what did you do? epeli_e: we restart the apache]
its ok now] nagios: 2009/09/21 22:02 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 3.696 second response time amette: in extreme cases wait for the load to go down before starting apache again jasprit: ok will do
doc.tikiwiki.org site loads but there is http-features critical still nagios: 2009/09/21 22:07 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 5.738 second response time amette: yeah, the setup is not very well performing jasprit: ok amette: apache was not coming down cleanly on web1, I killed it with -9, now waiting for the load to come down jasprit: ok amette: uuuuuhm.... now that looks bad...
... apache doesn't start! :( epeli_e: should we try to restart again? nagios: 2009/09/21 22:12 CRIT web1 HTTP Connection refused
2009/09/21 22:12 CRIT web1 TikiWiki Connection refused amette: try finding, why it doesn't start epeli_e: ok il try ***: franck has quit IRC (Read error: 145 (Connection timed out))
franck has joined #tikiwiki-monitor
ChanServ sets mode: +o franck nagios: 2009/09/21 22:35 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.021 second response time amette: I shut the machine down - checking file-system offline jasprit: ok nagios: 2009/09/21 22:37 CRIT web1 CRITICAL - Host Unreachable (10.100.100.11)
2009/09/21 22:36 ?? web1 Disk Error reading table : Timeout
2009/09/21 22:37 CRIT web1 PING CRITICAL - Host Unreachable (10.100.100.11)
2009/09/21 22:37 ?? web1 Swap Error reading table : Timeout
2009/09/21 22:38 CRIT web1 SSH No route to host ***: marclaporte has joined #tikiwiki-monitor nagios: 2009/09/21 22:42 OK web1 PING OK - Packet loss = 0%, RTA = 0.23 ms
2009/09/21 22:42 OK web1 PING PING OK - Packet loss = 0%, RTA = 2.46 ms
2009/09/21 22:42 OK web1 Swap OK : Swap space: 0%used(0MB/1024MB) : < 80 %
2009/09/21 22:40 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds amette: /etc/init.d/apache2 was an empty file
I copied the one from web2 over and now apache starts again nagios: 2009/09/21 22:43 OK web1 SSH SSH OK - OpenSSH_5.2 (protocol 2.0) jasprit: ok cool ***: franck has quit IRC (Read error: 145 (Connection timed out)) amette: file system check failed with "cannot determine device size" *shrug*
I'm happy it wasn't a bigger emergency, but I'm scared of how big it was.... jasprit: ;) nagios: 2009/09/21 22:46 OK web1 Disk OK : /: 74%used(5296MB/7158MB) : < 80 %
2009/09/21 22:12 CRIT web1 HTTP Connection refused
2009/09/21 22:12 CRIT web1 TikiWiki Connection refused amette: a message from the past?!?!? epeli_e: ?? amette: look at nagios times ***: franck has joined #tikiwiki-monitor
ChanServ sets mode: +o franck amette: there is all the time this one query hanging on db0.tw.o :(
it gets worse... jasprit: ya nagios: 2009/09/21 23:33 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.159 second response time ***: marclaporte has quit IRC (Read error: 113 (No route to host))
epeli_e has quit IRC (Remote closed the connection)
tverma has joined #tikiwiki-monitor
tverma1 has joined #tikiwiki-monitor nagios: 2009/09/22 01:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds ***: jasprit has left
jasprit has joined #tikiwiki-monitor
timothyv has joined #tikiwiki-monitor nagios: 2009/09/22 01:20 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.529 second response time ***: tverma has quit IRC (Read error: 110 (Connection timed out))
tverma1 has quit IRC (Read error: 113 (No route to host))