[PLUG] copying large numbers of files

RParr RParr at TemporalArts.COM
Fri Mar 31 16:39:19 UTC 2006


Brent Rieck wrote:

>Hello,
>  I'm trying to make a copy of a directory tree containing a large
>number of very small files, and it's taking forever, as in 2 hours and
>counting.  Is there a "fast" way to do this?
>
>  I estimate that there is about 6 million files (I'm still waiting for
>the file count to come back), most files are < 1KB, some directories
>contain maybe 60K files.  I need to make a copy of it so I can "thin"
>the files by several orders of magnitude, and I want to make sure I
>don't "over thin" the original.
>
>  
>

I typically use cpio pass-through

$ mkdir destdir
$ cd srcdir
$ find * | cpio -pvdmu srcdir

If you are going to repeatedly make this same copy to archive or mirror 
the original source,
I would strongly recommend using rsync.  I'd use cpio for the initial 
copy, then update the destdir
from the srcdir thereafter using rsync

R.Parr, RHCE, Temporal Arts






More information about the PLUG mailing list