[PLUG] copying large numbers of files
ianburrell at gmail.com
Thu Mar 30 16:28:07 PST 2006
On 3/30/06, Brent Rieck <bsr at spek.org> wrote:
> Russell Senior wrote:
> > Out of curiosity, which file system(s) are these going from/to?
> >From a NFS mounted ext3 partition to a local reiserfs partition - the
> network appears to be saturated too. But yay! it just finished copying.
You might want to see if you can get on the file server and transfer
directly from there. My guess is that the slowness is caused by
latency in requesting each file. I am pretty sure NFS requires a
round-trip message for each file; this adds up for a million files.
Reading lots of small files is faster on a local disk because Linux
does caching of file metadata and blocks.
More information about the PLUG