xfs
[Top] [All Lists]

getting slower and sslloowweerr...

To: linux-xfs@xxxxxxxxxxx
Subject: getting slower and sslloowweerr...
From: Jan De Landtsheer <Jan.DeLandtsheer@xxxxxxxxxx>
Date: Tue, 30 Sep 2003 23:15:17 +0200
Sender: linux-xfs-bounce@xxxxxxxxxxx
Ok, I 've been searching and testing and setting up a million times, but
I still keep having the same issue...

I want to have a solution where I can store ALOT of files:
With Alot, I mean A LOT. Right now I have 8 1.2TB JBOD's (SCSI to IDE,
and I know they are not Super HotStuff, but Hey! what can I say?
budgets).

Here's the catch... a probe (actually plenty of them) send me a bunch of
little files. A whola lotta files, I can tell you. Somethig to hose even
the most performant SCSI setup. But then again... budgets.
Files are of the size of 64 or 128K, that need to be stored at a pace of
50-60 MegaBYTE/sec (Gigabit NIC's). As the probes just keep a list with
files they sent to the central server, they can come back later to ask
for them, and that can even be at the same pace of writing them on the
network to the server. 

So far so good, I set up a test environment with a 1TB XFS filesystem,
and start to populate the thing with 64K files. Organisation is done
like ccache (ie. am md5 is calculated form this random data, and sent
over as file with md5 as name) and stored almost the same wa (we keep
the complete md5 name in the directory tree). How we're going to do tha
is still an open issue. Right now we want to test feasibility.

So I wrote a little script to generate random data, calculate the key
from it and store these files onto the JBOD.

dirtree :
/mnt/0/0/0/000acf123457dfdfdfdf...md5 of 64or 128 K

my little script generates on a 2.8GigaHZ P IV almost 100K files in 44
seconds to /dev/null, so it would be nice to get that stashed away on
the filesystem... Yeahyeah, I'm dreaming. 

So let's not dream and try to get something clean.
Creating a 1Gig file (dd if=/dev/zero of=/mnt/bigfile bs=1M count=1k)
gets done in about 12 secs, after a sync, so this JBOD tells me that it
can stash away rapidly.

starting the little script ( random data, totally unclean buffers, cache
is just there as a buffer ) gives me roughly 3-4 seconds for every 500
files of 64K, 5-6 seconds for 500 files of 128K... 

BUT THIS IS ONLY AT THE BEGINNING! 

After a while, say, 700K files, things get slower and slower. I read the
XFS manual inside out, know it by heart, and my wife tells me I mumble
xfs options in my sleep, but still... every 500 files get written at
10-12 files/second, and that is hard to come by. Remember, I have 8
JBODS and thus 8 TB of 64-128K files to store, retrieve or look for
their existence, at a pace of 400-500 /sec.

I can tell you that Reiserfs or XFS or JFS behave exactly the same way,
and I cannot imagine that nntp servers send and receive and store these
files at a pace of 10-12 per second.

Can anybody help me out please ?
Thanks,

Jan


<Prev in Thread] Current Thread [Next in Thread>