DomainKey-Signature: a=rsa-sha1; c=nofws; d=dreamhost.com; h=message-id
	:subject:from:reply-to:to:cc:date:in-reply-to:references
	:content-type:content-transfer-encoding:mime-version; q=dns; s=
	dreamhost.com; b=RJBsKxo9ciPaYy2UW3BcDo9Sm/gWZx8j0jc8QD37KaCN1W4
	v++EzBF/HVgt4HH2C/B4Jafzo7RZpNUiID4xwuH2djg1ABlKIB/+xtfnmqdYZHks
	C4xmzgVIGa0PHNlTDrnoy4WoZejfjj7nnnR0cOiRKwd6ke3B7h74qVX4S2bU=
Message-ID: <1325697491.3346.18.camel@doink>
Subject: Re: [LSF/MM TOPIC] [ATTEND] xfstests: what do we need to do to make
 it better?
From: Alex Elder <elder@dreamhost.com>
Reply-To: elder@dreamhost.com
To: Dave Chinner <david@fromorbit.com>
Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org,
        xfs@oss.sgi.com
Date: Wed, 04 Jan 2012 11:18:11 -0600
In-Reply-To: <20120103234455.GU23662@dastard>
References: <20120103234455.GU23662@dastard>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0

On Wed, 2012-01-04 at 10:44 +1100, Dave Chinner wrote:
> Given that more people are using xfstests and developing tests, we
> need to consider how to make it friendlier to hack on. The current
> structure of the tree is difficult to work with, the way tests are
> organised and numbered make it difficult to co-ordinate new tests
> and results in patch conflicts, etc.

Coordination of numbers is not a big deal, the test names/numbers
can be easily fixed up at commit time.  I also thought that the
numbers--though meaningless on their own--also avoided having to
decide where a particular test belongs.  I.e., a test that exercises
several categories of things (maybe preallocation, quota, and ENOSPC)
won't be hidden in any sort of "enospc" test directory.

I do think the growing number of tests is making it a bit unwieldy
though, so I think some sort of reorganization is a good plan.

> We also see problems arising from people not really understanding how
> the xfstests harness is designed and how it really is supposed to
> work, so an overview of the underlying principles of operation would
> probably be helpful to a lot of people. It will also save
> review and rework time if we can avoid having people make the same
> mistakes the first time they submit tests....

This is very important.  And the gist of it ought to be
captured somewhere if it is not already.

> I'd also like to discuss some potential infrastructure changes to
> make it easier to add new tests without conflicts with others
> developing new tests. Some of the ideas Christoph and I have
> previously tossed around include:
> 
> 	- break tests up into groups in their own subdirectories.
> 	  e.g. generic tests, xfs/ext4/btrfs specific tests, stress
> 	  tests, performance tests, large FS tests, etc
> 	- change the way we define groups of tests so we don't have
> 	  a single registry of tests and their groups
> 	- allow different naming of tests, such as desciptive text
> 	  names rather than just plain numbers
> 	- allow duplicate test names in different groups

Despite what I said above, I don't disagree with any of this.
Perhaps the tests can be buried in one or more subdirectories,
but each FSTYP defines its own groups file to drive testing.

> I'm sure that other users of xfstests will have some ideas on how to
> improve it for the way they run it, so I'd like to gather and
> incorporate these ideas into any structural change we make to
> xfstests.

Should be a good discussion.  It might be useful to have a
proposal or two to work with as a starting point, or maybe
an outline of the types of changes (naming, directory
structure, etc.), to help keep things focused.

					-Alex

> Cheers,
> 
> Dave.