Perhaps this is not the appropriate place to ask this... If not, please
forgive me, but this looks like just about the best place to get the feedback
I am looking for.
I am in the early design stages of some backup/HSM software. At this point I
am just researching my options and I was hoping someone could help me out
with some questions I have about DMAPI in general (since XFS seems to be the
only place where anyone is working in this area under linux - at least since
the OpenXDSM project died) and about the XFS implementation of DMAPI.
One of our requirements is near-real-time backup. Basically what we need to
be able to do is duplicate changes to secondary storage as soon as possible,
and then after a file has aged we will go back and poke holes as necessary.
When a file is accessed we would fill in the holes again. Unfortunately our
secondary storage is SIGNIFICANTLY slower than a hard drive (writes at
between 1.0 and 1.5Mb/s). Rather than trying to duplicate every write, we
were thinking about just adding files to a backup queue when the file is
closed. Unfortunately, the easist way that I can see for handling this is to
handle the debut event (initialize the event list for a file) and the close
event(schedule the file for backup). We understand that this method will not
ensure EVERY change to a file is backed up, but for our application, files
will not change often enough to make this a big concern.
Of course, this will only work if XFS adds support for the debut and close
events. Are there any plans to implement these events in XFS? If so, is
there an estimate on when this will be available? If not, is there any other
way of handling this that I have not considered yet.
I have also considered creating a separate write queue. In this
implementation I would basically try to consolidate writes (a non-trivial
operation with questionable advantages - may just forget about trying that)
and then after no write events have appeared after a pre-determined amount of
time, flush all writes to the secondary storage. As I stated earlier I am
still in the VERY early stages of design and I have probably overlooked some
issues (locking and file consistency... ugh... LOTS of work to do there...)
with this approach too. Anyhow, any input would be appreciated.
Maybe I would be better off just going with the standard NFS-server solution?