サーバが落ちたメモ
とりあえず備忘録として。
uptimeが半年を超えたサーバのウェブサイトが落ちた。pingは通るがsshで入れない。さくらVPSのコントロールパネルではCPUが2.0Kに振り切ったまま。5時間落ちてた。コンパネから再起動で復帰。
Linux version 2.6.18-348.12.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Wed Jul 10 05:28:41 EDT 2013
/var/log/messages
Feb 8 17:28:27 www*****u kernel: Mem-info: Feb 8 17:30:21 www*****u kernel: Node 0 DMA per-cpu: Feb 8 17:30:22 www*****u kernel: cpu 0 hot: high 0, batch 1 used:0 Feb 8 17:30:25 www*****u kernel: cpu 0 cold: high 0, batch 1 used:0 Feb 8 17:30:26 www*****u kernel: cpu 1 hot: high 0, batch 1 used:0 Feb 8 17:30:30 www*****u kernel: cpu 1 cold: high 0, batch 1 used:0 Feb 8 17:30:49 www*****u kernel: Node 0 DMA32 per-cpu: Feb 8 17:30:50 www*****u kernel: cpu 0 hot: high 186, batch 31 used:33 Feb 8 17:30:50 www*****u kernel: cpu 0 cold: high 62, batch 15 used:15 Feb 8 17:30:51 www*****u kernel: cpu 1 hot: high 186, batch 31 used:30 Feb 8 17:30:53 www*****u kernel: cpu 1 cold: high 62, batch 15 used:48 Feb 8 17:30:55 www*****u kernel: Node 0 Normal per-cpu: empty Feb 8 17:31:48 www*****u kernel: Node 0 HighMem per-cpu: empty Feb 8 17:33:40 www*****u kernel: Free pages: 6956kB (0kB HighMem) Feb 8 17:39:58 www*****u kernel: Active:127037 inactive:102689 dirty:0 writeback:0 unstable:0 free:1739 slab:6080 mapped-file:1155 map ped-anon:229203 pagetables:13134 Feb 8 17:41:48 www*****u kernel: Node 0 DMA free:2948kB min:36kB low:44kB high:52kB active:0kB inactive:0kB present:9684kB pages_scann ed:0 all_unreclaimable? yes Feb 8 17:42:01 www*****u kernel: lowmem_reserve[]: 0 994 994 994 Feb 8 17:42:02 www*****u kernel: Node 0 DMA32 free:4008kB min:4012kB low:5012kB high:6016kB active:507904kB inactive:410872kB present: 1018072kB pages_scanned:3244940 all_unreclaimable? yes Feb 8 17:42:02 www*****u kernel: lowmem_reserve[]: 0 0 0 0 Feb 8 17:42:02 www*****u kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 a ll_unreclaimable? no Feb 8 17:42:02 www*****u kernel: lowmem_reserve[]: 0 0 0 0 Feb 8 17:42:02 www*****u kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scan ned:0 all_unreclaimable? no Feb 8 17:42:02 www*****u kernel: lowmem_reserve[]: 0 0 0 0 Feb 8 17:42:02 www*****u kernel: Node 0 DMA: 5*4kB 4*8kB 5*16kB 4*32kB 2*64kB 2*128kB 1*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 2948kB Feb 8 17:42:02 www*****u kernel: Node 0 DMA32: 104*4kB 7*8kB 1*16kB 16*32kB 15*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4008kB Feb 8 17:42:02 www*****u kernel: Node 0 Normal: empty Feb 8 17:42:02 www*****u kernel: Node 0 HighMem: empty Feb 8 17:42:02 www*****u kernel: 6961 pagecache pages Feb 8 17:42:02 www*****u kernel: Swap cache: add 9457575874, delete 9457570200, find 2144320127/3685532428, race 1898+6301 Feb 8 17:42:02 www*****u kernel: Free swap = 0kB Feb 8 17:42:02 www*****u kernel: Total swap = 2048276kB Feb 8 17:42:02 www*****u kernel: Free swap: 0kB Feb 8 17:42:02 www*****u kernel: 262141 pages of RAM Feb 8 17:42:02 www*****u kernel: 5628 reserved pages Feb 8 17:42:02 www*****u kernel: 25981 pages shared Feb 8 17:42:02 www*****u kernel: 5768 pages swap cached Feb 8 17:42:02 www*****u kernel: Out of memory: Killed process 9706, UID 48, (httpd). Feb 8 17:42:02 www*****u kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Feb 8 17:42:02 www*****u kernel: Feb 8 17:42:02 www*****u kernel: Call Trace: Feb 8 17:42:02 www*****u kernel: [<ffffffff800cb2ee>] out_of_memory+0x8e/0x2f3 Feb 8 17:42:02 www*****u kernel: [<ffffffff8002e4c7>] __wake_up+0x38/0x4f Feb 8 17:42:02 www*****u kernel: [<ffffffff8000f691>] __alloc_pages+0x27f/0x308 Feb 8 17:42:02 www*****u kernel: [<ffffffff80013081>] __do_page_cache_readahead+0x96/0x17b Feb 8 17:42:02 www*****u kernel: [<ffffffff800139be>] filemap_nopage+0x14c/0x360 Feb 8 17:42:02 www*****u kernel: [<ffffffff80008972>] __handle_mm_fault+0x1fd/0x103b Feb 8 17:42:02 www*****u kernel: [<ffffffff800183d7>] do_sync_write+0xc7/0x104 Feb 8 17:42:02 www*****u kernel: [<ffffffff8006719c>] do_page_fault+0x499/0x842 Feb 8 17:42:02 www*****u kernel: [<ffffffff800a3c28>] autoremove_wake_function+0x0/0x2e Feb 8 17:42:02
