[LRUG] Finding out why server was unresponsive

[Andrew Stewart]

On 16 Apr 2013, at 12:23, Richard Taylor wrote:
> Another sneaky one is running out of inodes.  I've had this a couple of times when I haven't been cleaning up rails releases often enough.  It behaves the same way as running out of disk space but your disk space indicator shows there's nothing wrong.
> 
> df -ih is your friend here:
> 
> ~$ df -ih
> Filesystem     Inodes IUsed IFree IUse% Mounted on
> /dev/xvda        5.8M  102K  5.7M    2% /

That is sneaky!  I think that was ok though:

$ df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/md2          64M  583K   64M    1% /
udev             3.9M   524  3.9M    1% /dev
tmpfs            3.9M   435  3.9M    1% /run
none             3.9M     1  3.9M    1% /run/lock
none             3.9M     1  3.9M    1% /run/shm
cgroup           3.9M     9  3.9M    1% /sys/fs/cgroup
/dev/md3         110M   22K  110M    1% /home
/dev/md1         128K   232  128K    1% /boot

I wasn't able to ping the server.  Here's the mtr traceroute I took at the time:

                                       My traceroute  [v0.82]
calafia.local (0.0.0.0)                                                    Mon Apr 15 16:22:58 2013
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                           Packets               Pings
 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.88.1                                          0.0%  1967    3.9   4.2   3.3  22.6   1.3
 2. 217.89.66.41                                          1.0%  1967   10.9   9.3   5.7 590.8  15.4
 3. kn-ea1.kn.de.net.dtag.de                              1.1%  1967   16.9  14.6   9.7 274.6  10.4
 4. f-ed4-i.f.de.net.dtag.de                              1.2%  1967   17.6  20.7  14.1 211.6  13.9
 5. 62.157.251.34                                         1.3%  1967   16.0  17.9  14.3 389.5  12.2
 6. hos-bb2.juniper2.rz19.hetzner.de                      1.9%  1967   21.9  23.7  19.6 189.0   9.3
 7. hos-tr4.ex3k1.rz19.hetzner.de                         1.4%  1967   23.1  26.1  21.1 341.6  11.0
 8. ???

If I run mtr now, the packets pass through 1-7 as above and then reach my server in hop 8.

I haven't found a log yet with any activity during the downtime.

Cheers,
Andy