[00:16] *** Ofa has quit IRC ("Leaving.") [00:28] *** dszetu has left [00:38] 2009/09/21 02:34 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [00:43] 2009/09/21 02:42 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.898 second response time [00:44] *** deepaks has left [00:45] *** deepaks has joined #tikiwiki-monitor [00:47] *** dszet1 has joined #tikiwiki-monitor [00:48] *** dszet1 has left [00:51] *** dszet1 has joined #tikiwiki-monitor [00:58] 2009/09/21 02:54 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [01:03] 2009/09/21 03:02 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.691 second response time [01:27] *** rupeni has joined #tikiwiki-monitor [01:34] *** franck has quit IRC (Read error: 113 (No route to host)) [02:24] 2009/09/21 04:20 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [02:54] 2009/09/21 04:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.695 second response time [03:12] 2009/09/21 05:08 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [03:32] 2009/09/21 05:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 6.827 second response time [03:46] 2009/09/21 05:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [04:01] 2009/09/21 06:00 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.209 second response time [04:09] 2009/09/21 06:05 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [05:05] *** marclaporte has quit IRC (Read error: 60 (Operation timed out)) [05:23] 2009/09/21 07:23 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.818 second response time [05:31] *** dszet1 has left [05:32] *** deepaks has quit IRC (Remote closed the connection) [05:36] *** melaia has joined #tikiwiki-monitor [05:37] *** franck has joined #tikiwiki-monitor [05:37] *** ChanServ sets mode: +o franck [05:37] .. [05:42] *** melai1 has joined #tikiwiki-monitor [06:13] 2009/09/21 08:10 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [06:13] *** rupeni has left [06:14] *** rupeni has joined #tikiwiki-monitor [06:36] *** srishti has joined #tikiwiki-monitor [06:38] 2009/09/21 08:38 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.633 second response time [06:44] 2009/09/21 08:41 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds [06:45] 2009/09/21 08:41 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds [06:46] 2009/09/21 08:43 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [06:50] *** marclaporte has joined #tikiwiki-monitor [06:52] hi [06:52] hey [06:52] checking on dev, yea [06:54] CPU is under load [07:00] tks [07:09] 2009/09/21 09:09 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 0.156 second response time [07:09] 2009/09/21 09:09 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 0.675 second response time [07:11] 2009/09/21 09:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.387 second response time [07:19] 2009/09/21 09:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [07:27] *** franck has quit IRC () [07:29] 2009/09/21 09:29 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.777 second response time [07:54] *** srishti has quit IRC ("Leaving.") [07:54] *** srishti has joined #tikiwiki-monitor [07:58] 2009/09/21 09:55 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [08:03] 2009/09/21 10:03 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.257 second response time [08:54] *** franck has joined #tikiwiki-monitor [08:54] *** ChanServ sets mode: +o franck [09:12] 2009/09/21 11:09 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [09:28] 2009/09/21 11:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.156 second response time [10:07] 2009/09/21 12:03 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [10:12] 2009/09/21 12:11 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.380 second response time [10:20] 2009/09/21 12:16 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [10:43] / [10:50] *** marclaporte has quit IRC (Read error: 113 (No route to host)) [11:10] 2009/09/21 13:09 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.181 second response time [11:25] 2009/09/21 13:21 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [12:00] 2009/09/21 13:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.014 second response time [12:08] 2009/09/21 14:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [12:23] 2009/09/21 14:22 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.846 second response time [12:37] 2009/09/21 14:33 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [12:40] *** epeli_e has joined #tikiwiki-monitor [13:02] 2009/09/21 15:01 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.370 second response time [13:50] 2009/09/21 15:46 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [14:08] *** epeli_e has quit IRC (Remote closed the connection) [14:09] *** epeli_e has joined #tikiwiki-monitor [14:44] 2009/09/21 16:44 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.319 second response time [15:03] 2009/09/21 17:00 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [15:53] 2009/09/21 17:53 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.930 second response time [16:01] 2009/09/21 17:58 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [16:31] 2009/09/21 18:31 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.320 second response time [16:39] 2009/09/21 18:36 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [16:44] *** melaia has left [16:46] *** franck_ has joined #tikiwiki-monitor [16:46] *** ChanServ sets mode: +o franck_ [16:55] .. [16:57] *** franck has quit IRC (Read error: 113 (No route to host)) [16:57] *** franck_ is now known as franck [16:59] 2009/09/21 18:59 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.993 second response time [17:07] 2009/09/21 19:04 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [17:27] 2009/09/21 19:27 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.725 second response time [17:45] 2009/09/21 19:42 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [17:49] *** srishti has left [17:50] 2009/09/21 19:50 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 9.781 second response time [18:10] 2009/09/21 20:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [19:01] *** melai1 has left [19:05] *** jasprit has joined #tikiwiki-monitor [19:08] *** jasprit has quit IRC (Remote closed the connection) [19:12] *** jasprit has joined #tikiwiki-monitor [19:37] 2009/09/21 21:34 CRIT web1 HTTP CRITICAL - Socket timeout after 10 seconds [19:37] 2009/09/21 21:34 CRIT web1 TikiWiki CRITICAL - Socket timeout after 10 seconds [20:01] amette [20:01] http critical on dev.tikiwiki.org [20:01] the site does not load [20:01] what did you do? [20:02] we restart the apache] [20:02] its ok now] [20:02] 2009/09/21 22:02 OK web1 HTTP HTTP OK - HTTP/1.1 302 Found - 3.696 second response time [20:06] in extreme cases wait for the load to go down before starting apache again [20:06] ok will do [20:07] doc.tikiwiki.org site loads but there is http-features critical still [20:07] 2009/09/21 22:07 OK web1 TikiWiki HTTP OK HTTP/1.1 200 OK - 5.738 second response time [20:08] yeah, the setup is not very well performing [20:08] ok [20:09] apache was not coming down cleanly on web1, I killed it with -9, now waiting for the load to come down [20:12] ok [20:14] uuuuuhm.... now that looks bad... [20:14] ... apache doesn't start! :( [20:15] should we try to restart again? [20:15] 2009/09/21 22:12 CRIT web1 HTTP Connection refused [20:15] 2009/09/21 22:12 CRIT web1 TikiWiki Connection refused [20:16] try finding, why it doesn't start [20:17] ok il try [20:32] *** franck has quit IRC (Read error: 145 (Connection timed out)) [20:35] *** franck has joined #tikiwiki-monitor [20:35] *** ChanServ sets mode: +o franck [20:35] 2009/09/21 22:35 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.021 second response time [20:36] I shut the machine down - checking file-system offline [20:36] ok [20:37] 2009/09/21 22:37 CRIT web1 CRITICAL - Host Unreachable (10.100.100.11) [20:37] 2009/09/21 22:36 ?? web1 Disk Error reading table : Timeout [20:37] 2009/09/21 22:37 CRIT web1 PING CRITICAL - Host Unreachable (10.100.100.11) [20:38] 2009/09/21 22:37 ?? web1 Swap Error reading table : Timeout [20:39] 2009/09/21 22:38 CRIT web1 SSH No route to host [20:40] *** marclaporte has joined #tikiwiki-monitor [20:42] 2009/09/21 22:42 OK web1 PING OK - Packet loss = 0%, RTA = 0.23 ms [20:42] 2009/09/21 22:42 OK web1 PING PING OK - Packet loss = 0%, RTA = 2.46 ms [20:43] 2009/09/21 22:42 OK web1 Swap OK : Swap space: 0%used(0MB/1024MB) : < 80 % [20:43] 2009/09/21 22:40 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [20:44] /etc/init.d/apache2 was an empty file [20:44] I copied the one from web2 over and now apache starts again [20:44] 2009/09/21 22:43 OK web1 SSH SSH OK - OpenSSH_5.2 (protocol 2.0) [20:44] ok cool [20:45] *** franck has quit IRC (Read error: 145 (Connection timed out)) [20:45] file system check failed with "cannot determine device size" *shrug* [20:45] I'm happy it wasn't a bigger emergency, but I'm scared of how big it was.... [20:46] ;) [20:46] 2009/09/21 22:46 OK web1 Disk OK : /: 74%used(5296MB/7158MB) : < 80 % [20:48] 2009/09/21 22:12 CRIT web1 HTTP Connection refused [20:48] 2009/09/21 22:12 CRIT web1 TikiWiki Connection refused [20:49] a message from the past?!?!? [20:50] ?? [20:52] look at nagios times [20:53] *** franck has joined #tikiwiki-monitor [20:53] *** ChanServ sets mode: +o franck [21:30] there is all the time this one query hanging on db0.tw.o :( [21:31] it gets worse... [21:32] ya [21:33] 2009/09/21 23:33 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 8.159 second response time [21:58] *** marclaporte has quit IRC (Read error: 113 (No route to host)) [22:26] *** epeli_e has quit IRC (Remote closed the connection) [22:39] *** tverma has joined #tikiwiki-monitor [23:05] *** tverma1 has joined #tikiwiki-monitor [23:10] 2009/09/22 01:07 CRIT web2 Features Page CRITICAL - Socket timeout after 10 seconds [23:12] *** jasprit has left [23:12] *** jasprit has joined #tikiwiki-monitor [23:16] *** timothyv has joined #tikiwiki-monitor [23:20] 2009/09/22 01:20 OK web2 Features Page HTTP OK HTTP/1.1 200 OK - 7.529 second response time [23:24] *** tverma has quit IRC (Read error: 110 (Connection timed out)) [23:30] *** tverma1 has quit IRC (Read error: 113 (No route to host))