[PLUG] When tmpfs==good and ext3==bad :-)

[Paul Heinlein]

I've been tasked with scraping data from some off-site directories
and consolidating them in a locally accessible LDAP directory.

Lots of bash and perl later, I've got a 4.5MB LDIF file containing 
about 16,000 records that's ready for slurping.

Using ldapadd is painfully slow for a record set of that size, so I 
decided to use slapadd instead.

All my local filesystems are ext3 partitions. What became apparent
really quickly was that ext3 is ill-equipped to deal with the sort of
quick transactional writes done by the db libraries. It was almost
painful to watch.

tmpfs to the rescue!

  mount -t tmpfs -o defaults tmpfs /mnt/tmp

I used slapadd to transform the LDIF file into what became about 30MB 
of db data and indices.

  System 1: Red Hat 9, P3 550MHz, 128 MB RAM, IDE HD
  System 2: Red Hat 8, dual P3 1.2GHz, 512MB RAM, UltraSCSI 160 HD

Writing to a tmpfs partition:

  System 1           System 2
  -----------------  -----------------
  real    2m52.418s  real    0m50.038s
  user    1m50.330s  user    0m24.914s
  sys     1m1.960s   sys     0m25.119s

Writing to an ext3 partition:

  System 1            System 2
  ------------------  -----------------
  real    21m28.784s  real    2m33.932s
  user    1m55.440s   user    0m26.014s
  sys     1m26.860s   sys     0m32.426s

Obviously, a filesystem living in RAM is going to outperform one on 
disk, but I don't think that explains a 5x to 7x speed difference.

It seems to me that the journaling overhead becomes unbearable on all
those little atomic db updates.

--Paul Heinlein <heinlein at cse.ogi.edu>