Received: with ECARTIS (v1.0.0; list linux-xfs); Fri, 12 Dec 2003 02:11:53 -0800 (PST) Received: from pophost.wldelft.nl (sunray.wldelft.nl [145.9.132.100]) by oss.sgi.com (8.12.10/8.12.9) with SMTP id hBCABMTa024607 for ; Fri, 12 Dec 2003 02:11:23 -0800 Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id LAA04205 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 11:11:22 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id LAA28658 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 11:01:17 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id KAA17442 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 10:51:12 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id KAA11503 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 10:41:05 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id KAA11294 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 10:30:57 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id KAA06977 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 10:21:20 +0100 (MET) Received: (from root@localhost) by pophost.wldelft.nl (8.9.3/8.9.3vc) id KAA16374 for linux-xfs@oss.sgi.com; Fri, 12 Dec 2003 10:06:34 +0100 (MET) Received: from WLdelft.nl (wl10996 [145.9.222.37]) by pophost.wldelft.nl (8.9.3/8.9.3) with ESMTP id KAA15914 for ; Fri, 12 Dec 2003 10:06:23 +0100 (MET) Message-ID: <3FD9850F.90702@WLdelft.nl> Date: Fri, 12 Dec 2003 10:06:23 +0100 From: Leroy van Logchem Organization: WL delft hydraulics User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 X-Accept-Language: en-us, en MIME-Version: 1.0 To: linux-xfs@oss.sgi.com Subject: Kernel 2.4.22 xfs_force_shutdown Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-archive-position: 1391 X-ecartis-version: Ecartis v1.0.0 Sender: linux-xfs-bounce@oss.sgi.com Errors-to: linux-xfs-bounce@oss.sgi.com X-original-sender: Leroy.vanLogchem@wldelft.nl Precedence: bulk X-list: linux-xfs Content-Length: 6093 Lines: 115 Best XFS mailinglist, We have been using XFS on semi-large servers the last 2.5 years [ 2 x 600 GB ] With that in mind we have bought two large servers with general specs: - 24 x 250 GB disks (leaving 4.5 TB netto when using RAID5+2hotspare) - 2 x 3Ware hardware RAID controllers - 6 GB ram - Dual CPU on TYAN mainboard - Debian 3 using 2.4.22 with XFS patches Main purpose is serving NFS v3 and some Samba v3 to >300 users. The total amount of files exceeds 5 milion, most of them are below 1 MB. Loadavg on a normal working day varies between 0.5 <-> 2, which seems quite good & low to me. The servers are running for 2 weeks now but last night one 1.1 TB partition went down -xfs_force_shutdown(sd(8,48),0x8)-. I couldnt umount it so I had to reboot the server. Not a single sign of hardware error messages from the raid drivers. The partitions are mkfs'ed using: mkfs -V -t xfs -f -d su=65536,sw=8 /dev/sdd Logfiles and various info follows: /proc/partitions major minor #blocks name rio rmerge rsect ruse wio wmerge wsect wuse running use aveq 8 0 976790016 sda 3097 15187 146001 155810 6292 94489 804001 265190 0 66830 421000 8 16 1220987520 sdb 33 76 649 180 3 62 513 70 0 250 250 8 32 976790016 sdc 274 76 2521 2450 4 63 529 40 0 2490 2490 8 48 1220987520 sdd 9456 174151 1468331 1372750 4402 42143 367990 505730 0 232020 1878480 /proc/interrupts 26: 357 13379 IO-APIC-level 3ware Storage Controller 27: 132 9000 IO-APIC-level 3ware Storage Controller Elevators optimized for writing are: /dev/sd[a-d] elevator ID 8 read_latency: 32 write_latency: 8192 /proc/version Linux version 2.4.22-xfs (root@kingfish) (gcc version 3.3.2 (Debian)) #12 SMP Thu Nov 27 17:34:34 CET 2003 /var/log/messages Dec 12 01:54:58 kingfish -- MARK -- Dec 12 01:56:22 kingfish kernel: f6bd7c68 f6bd7cc4 c01bbb60 c03259af 00000001 00000000 c03259a3 00000662 Dec 12 01:56:22 kingfish kernel: c01bcbfa 00000000 00000001 f707cc00 f4aaeeac 000a8d43 0000000d 00000001 Dec 12 01:56:22 kingfish kernel: 000a8ce0 00000030 00000000 00000001 00000000 ee73dd10 00000000 f6bd7d3c Dec 12 01:56:22 kingfish kernel: Call Trace: Dec 12 01:56:22 kingfish kernel: [xfs_free_ag_extent+1056/1808] xfs_free_ag_extent+0x420/0x710 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_free_extent+186/224] xfs_free_extent+0xba/0xe0 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_free_extent+186/224] xfs_free_extent+0xba/0xe0 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_efd_init+124/126] xfs_efd_init+0x7c/0x7e [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_trans_get_efd+54/80] xfs_trans_get_efd+0x36/0x50 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_bmap_finish+306/448] xfs_bmap_finish+0x132/0x1c0 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_itruncate_finish+513/1008] xfs_itruncate_finish+0x201/0x3f0 [kernel] Dec 12 01:56:22 kingfish kernel: [xfs_inactive+1270/1360] xfs_inactive+0x4f6/0x550 [kernel] Dec 12 01:56:22 kingfish kernel: [vn_rele+167/176] vn_rele+0xa7/0xb0 [kernel] Dec 12 01:56:22 kingfish kernel: [linvfs_clear_inode+25/48] linvfs_clear_inode+0x19/0x30 [kernel] Dec 12 01:56:22 kingfish kernel: [clear_inode+261/320] clear_inode+0x105/0x140 [kernel] Dec 12 01:56:22 kingfish kernel: [iput+214/736] iput+0xd6/0x2e0 [kernel] Dec 12 01:56:22 kingfish kernel: [vfs_unlink+395/640] vfs_unlink+0x18b/0x280 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd_unlink+285/576] nfsd_unlink+0x11d/0x240 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd3_proc_remove+132/272] nfsd3_proc_remove+0x84/0x110 [kernel] Dec 12 01:56:22 kingfish kernel: [nfs3svc_decode_diropargs+134/240] nfs3svc_decode_diropargs+0x86/0xf0 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd_dispatch+204/485] nfsd_dispatch+0xcc/0x1e5 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd_dispatch+0/485] nfsd_dispatch+0x0/0x1e5 [kernel] Dec 12 01:56:22 kingfish kernel: [svc_process+853/1360] svc_process+0x355/0x600 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd+495/832] nfsd+0x1ef/0x340 [kernel] Dec 12 01:56:22 kingfish kernel: [arch_kernel_thread+46/64] arch_kernel_thread+0x2e/0x40 [kernel] Dec 12 01:56:22 kingfish kernel: [nfsd+0/832] nfsd+0x0/0x340 [kernel] Dec 12 01:56:22 kingfish kernel: xfs_force_shutdown(sd(8,48),0x8) called from line 4049 of file xfs_bmap.c. Return address = 0xc0223ad8 The recovery upon reboot: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1634 of file xfs_alloc.c. Caller 0xc01bcbfa f6af1bac f6af1c08 c01bbb60 c03259af 00000001 00000000 c03259a3 00000662 c01bcbfa 00000000 00000001 f62b1c00 f689fe8c 000a8d43 0000000d 00000001 000a8ce0 00000030 00000000 00000001 00000000 f68abc50 f68b4e98 f6af1c80 Call Trace: [] xfs_free_ag_extent+0x420/0x710 [kernel] [] xfs_free_extent+0xba/0xe0 [kernel] [] xfs_free_extent+0xba/0xe0 [kernel] [] xfs_efd_init+0x7c/0x7e [kernel] [] xfs_trans_get_efd+0x36/0x50 [kernel] [] xlog_recover_process_efi+0x186/0x1f0 [kernel] [] xlog_recover_process_efis+0x77/0x90 [kernel] [] xfs_iget+0x143/0x1a0 [kernel] [] xlog_recover_finish+0x1d/0x26a [kernel] [] xfs_log_mount_finish+0x2d/0x30 [kernel] [] xfs_mountfs+0x6ac/0xd20 [kernel] [] xfs_ioinit+0x20/0x40 [kernel] [] xfs_mount+0x27d/0x420 [kernel] [] vfs_mount+0x41/0x50 [kernel] [] linvfs_read_super+0x8f/0x270 [kernel] [] alloc_super+0x3d/0x1b0 [kernel] [] get_sb_bdev+0x18e/0x290 [kernel] [] alloc_vfsmnt+0x98/0xe0 [kernel] [] do_kern_mount+0x11f/0x130 [kernel] [] do_add_mount+0x74/0x160 [kernel] [] path_lookup+0x3a/0x40 [kernel] [] do_mount+0x130/0x180 [kernel] [] copy_mount_options+0x5d/0xb0 [kernel] [] sys_mount+0xb6/0x120 [kernel] [] system_call+0x33/0x38 [kernel] Ending XFS recovery on filesystem: sd(8,48) (dev: 8/48) Best regards, Leroy van Logchem The Netherlands WL | delft hydraulics