I ran across these notes I took while tuning DRBD on a pair of NFS servers.

[root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    50   0  1070   9   107   1    63   1  1296  10   106   1
Latency               283ms     219ms     226ms     331ms     205ms     212ms
1.96,1.96,smtp10,1,1322542347,,,,,,,,,,,,,,16,,,,,50,0,1070,9,107,1,63,1,1296,10,106,1,,,,,,,283ms,219ms,226ms,331ms,205ms,212ms

Change drbd max-buffers and max-epoch-size to 8000 on nfs08 and nfs09

[root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    66   1  1235   9   109   1    65   1  1268   9   107   1
Latency               215ms     219ms     226ms     343ms     204ms     122ms
1.96,1.96,smtp10,1,1322540643,,,,,,,,,,,,,,16,,,,,66,1,1235,9,109,1,65,1,1268,9,107,1,,,,,,,215ms,219ms,226ms,343ms,204ms,122ms

change drbd sndbuf-size from default 128k to 512k.

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 18:34:12 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    65   1  1050   8   107   1    67   1  1070   8   107   1
Latency               331ms     220ms     224ms     218ms     205ms     185ms
1.96,1.96,smtp10,1,1322537896,,,,,,,,,,,,,,16,,,,,65,1,1050,8,107,1,67,1,1070,8,107,1,,,,,,,331ms,220ms,224ms,218ms,205ms,185ms
Mon Nov 28 18:47:59 CST 2011

Numbers got worse. Re-run test.

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 18:48:50 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    66   1  1302  11   106   1    66   1  1351  11   106   1
Latency               223ms     219ms     214ms     294ms     204ms     129ms
1.96,1.96,smtp10,1,1322537091,,,,,,,,,,,,,,16,,,,,66,1,1302,11,106,1,66,1,1351,11,106,1,,,,,,,223ms,219ms,214ms,294ms,204ms,129ms
Mon Nov 28 19:02:36 CST 2011

better, test again

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 19:09:59 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    64   1  1154   9   105   1    65   1  1377  10   107   1
Latency               253ms     204ms     214ms     270ms     205ms     213ms
1.96,1.96,smtp10,1,1322535775,,,,,,,,,,,,,,16,,,,,64,1,1154,9,105,1,65,1,1377,10,107,1,,,,,,,253ms,204ms,214ms,270ms,205ms,213ms
Mon Nov 28 19:23:55 CST 2011

TCP tuning between nfs08 and nfs09

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 20:23:36 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    65   1  1216   9   105   1    67   1  1257  10   105   1
Latency               403ms     217ms     216ms   76807us     205ms     213ms
1.96,1.96,smtp10,1,1322532511,,,,,,,,,,,,,,16,,,,,65,1,1216,9,105,1,67,1,1257,10,105,1,,,,,,,403ms,217ms,216ms,76807us,205ms,213ms
Mon Nov 28 20:37:27 CST 2011

# drbd increase al-extents from 127 to 997

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 20:48:06 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    66   1  1234  10   108   1    65   1  1277  10   106   1
Latency             73106us     214ms     214ms     293ms     204ms     212ms
1.96,1.96,smtp10,1,1322531581,,,,,,,,,,,,,,16,,,,,66,1,1234,10,108,1,65,1,1277,10,106,1,,,,,,,73106us,214ms,214ms,293ms,204ms,212ms
Mon Nov 28 21:01:54 CST 2011

# adjust drbd syncer rate from default of 12M to 100M

[root@smtp10 ~]# rm -fr /mailstore/benchmark/Bonnie.32558/

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 23:19:16 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    66   1  1030   7   106   1    67   1  1263   9   107   1
Latency               264ms     229ms     226ms     257ms     204ms     209ms
1.96,1.96,smtp10,1,1322521623,,,,,,,,,,,,,,16,,,,,66,1,1030,7,106,1,67,1,1263,9,107,1,,,,,,,264ms,229ms,226ms,257ms,204ms,209ms
Mon Nov 28 23:33:00 CST 2011

start NFS tuning (http://nfs.sourceforge.net/nfs-howto/ar01s05.html)

Change /mailstore mount point to rsize=32768,wsize=32768 (read and write block size of 32K instead of 8k)

[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Tue Nov 29 00:26:05 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Create------ --------Random Create--------
smtp10              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    65   1  1044   8   107   1    66   1  1401  11   106   1
Latency               296ms     219ms     226ms     219ms    4952us     523ms
1.96,1.96,smtp10,1,1322548134,,,,,,,,,,,,,,16,,,,,65,1,1044,8,107,1,66,1,1401,11,106,1,,,,,,,296ms,219ms,226ms,219ms,4952us,523ms
Tue Nov 29 00:39:52 CST 2011

This configuration was never used in production, so I'm unsure how helpful it may or may not be.