Date: Fri, 31 Oct 2003 21:44:25 -0500
From: Andrew Klaassen <ak@dkp.com>
To: Simon Matter <simon.matter@ch.sauter-bc.com>
Cc: linux-xfs@oss.sgi.com
Subject: Re: XFS + SMP + Software RAID 5 slow?
Message-ID: <20031101024425.GA28503@dkp.com>
Mail-Followup-To: Simon Matter <simon.matter@ch.sauter-bc.com>,
	linux-xfs@oss.sgi.com
References: <20031031215013.GB25822@dkp.com> <20031031223220.GC25822@dkp.com> <38213.213.173.165.140.1067642085.squirrel@imap01.ch.sauter-bc.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <38213.213.173.165.140.1067642085.squirrel@imap01.ch.sauter-bc.com>
User-Agent: Mutt/1.5.4i
Sender: linux-xfs-bounce@oss.sgi.com
Errors-to: linux-xfs-bounce@oss.sgi.com
Precedence: bulk
Content-Length: 1868
Lines: 53

On Sat, Nov 01, 2003 at 12:14:45AM +0100,
Simon Matter wrote:

> > On Fri, Oct 31, 2003 at 04:50:13PM -0500,
> > Andrew Klaassen wrote:
> >
> >> I've been experiencing *very* slow sustained write ( <
> >> .5MB/sec) using 2.4.20-20.9.XFS1.3.1smp on a dual-Xeon box
> >> with a 3ware 7506 card in JBOD mode.  Software RAID 5.  No
> >> such problems with Ext3 ( > 20MB/sec sustained write) with
> >> the same kernel on the same software RAID 5 device. 
> >> Obviously there's something very wrong here.
> >>
> >> Known issue?  Any idea what might be causing this?
> >
> > By the way: Using hardware RAID 5 on exactly the same setup,
> > the XFS and EXT3 numbers are virtually identical (~5MB/sec
> > sustained write).  Only software RAID 5 seems to cause this
> > great divergence.
> 
> Did you try your setup with an external log? IIRC the problem
> with bad write performance on software RAID 5 should have been
> adressed and mostly fixed with the XFS1.3.x release. I
> recommend to try with external log anyway. I never had good
> results on my software RAID 5 configurations until I used
> external log. However, I never did my tests again in the
> XFS1.3.x days.

*Wow*, that made huge difference.  From < .5MB/sec to >
10MB/sec.  Still not as good as Ext3, but getting better.

I've noticed the variance in amount of data going out to disk
with XFS was much higher than with Ext3, too.  Using the
external log on XFS, as you suggested, and a default Ext3 (these
both on software RAID 5, of course):

Ext3:
Mean: 17541.5MB/s
Std. Dev: 2375.89MB/s

XFS:
Mean: 12960.4MB/s
Std. Dev: 6535.16MB/s

That's computed with a 5-second timeslices.  In English, it
means that XFS write rates are jumping all over the place, from
.5MB/s to 20MB/s.  Any reason why that would be?

Again, this is with fs-bench, which fills up the filesystem with
~100KB files.

Andrew Klaassen