I ran across these notes I took while tuning DRBD on a pair of NFS servers.
[root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 50 0 1070 9 107 1 63 1 1296 10 106 1
Latency 283ms 219ms 226ms 331ms 205ms 212ms
1.96,1.96,smtp10,1,1322542347,,,,,,,,,,,,,,16,,,,,50,0,1070,9,107,1,63,1,1296,10,106,1,,,,,,,283ms,219ms,226ms,331ms,205ms,212ms
Change drbd max-buffers and max-epoch-size to 8000 on nfs08 and nfs09
[root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1235 9 109 1 65 1 1268 9 107 1
Latency 215ms 219ms 226ms 343ms 204ms 122ms
1.96,1.96,smtp10,1,1322540643,,,,,,,,,,,,,,16,,,,,66,1,1235,9,109,1,65,1,1268,9,107,1,,,,,,,215ms,219ms,226ms,343ms,204ms,122ms
Change drbd sndbuf-size from default 128k to 512k.
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 18:34:12 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 65 1 1050 8 107 1 67 1 1070 8 107 1
Latency 331ms 220ms 224ms 218ms 205ms 185ms
1.96,1.96,smtp10,1,1322537896,,,,,,,,,,,,,,16,,,,,65,1,1050,8,107,1,67,1,1070,8,107,1,,,,,,,331ms,220ms,224ms,218ms,205ms,185ms
Mon Nov 28 18:47:59 CST 2011
Numbers got worse. Re-run test.
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 18:48:50 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1302 11 106 1 66 1 1351 11 106 1
Latency 223ms 219ms 214ms 294ms 204ms 129ms
1.96,1.96,smtp10,1,1322537091,,,,,,,,,,,,,,16,,,,,66,1,1302,11,106,1,66,1,1351,11,106,1,,,,,,,223ms,219ms,214ms,294ms,204ms,129ms
Mon Nov 28 19:02:36 CST 2011
better, test again
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 19:09:59 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 64 1 1154 9 105 1 65 1 1377 10 107 1
Latency 253ms 204ms 214ms 270ms 205ms 213ms
1.96,1.96,smtp10,1,1322535775,,,,,,,,,,,,,,16,,,,,64,1,1154,9,105,1,65,1,1377,10,107,1,,,,,,,253ms,204ms,214ms,270ms,205ms,213ms
Mon Nov 28 19:23:55 CST 2011
TCP tuning between nfs08 and nfs09
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 20:23:36 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 65 1 1216 9 105 1 67 1 1257 10 105 1
Latency 403ms 217ms 216ms 76807us 205ms 213ms
1.96,1.96,smtp10,1,1322532511,,,,,,,,,,,,,,16,,,,,65,1,1216,9,105,1,67,1,1257,10,105,1,,,,,,,403ms,217ms,216ms,76807us,205ms,213ms
Mon Nov 28 20:37:27 CST 2011
# drbd increase al-extents from 127 to 997
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 20:48:06 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1234 10 108 1 65 1 1277 10 106 1
Latency 73106us 214ms 214ms 293ms 204ms 212ms
1.96,1.96,smtp10,1,1322531581,,,,,,,,,,,,,,16,,,,,66,1,1234,10,108,1,65,1,1277,10,106,1,,,,,,,73106us,214ms,214ms,293ms,204ms,212ms
Mon Nov 28 21:01:54 CST 2011
# adjust drbd syncer rate from default of 12M to 100M
[root@smtp10 ~]# rm -fr /mailstore/benchmark/Bonnie.32558/
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Mon Nov 28 23:19:16 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1030 7 106 1 67 1 1263 9 107 1
Latency 264ms 229ms 226ms 257ms 204ms 209ms
1.96,1.96,smtp10,1,1322521623,,,,,,,,,,,,,,16,,,,,66,1,1030,7,106,1,67,1,1263,9,107,1,,,,,,,264ms,229ms,226ms,257ms,204ms,209ms
Mon Nov 28 23:33:00 CST 2011
start NFS tuning (http://nfs.sourceforge.net/nfs-howto/ar01s05.html)
Change /mailstore mount point to rsize=32768,wsize=32768 (read and write block size of 32K instead of 8k)
[root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
Tue Nov 29 00:26:05 CST 2011
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 65 1 1044 8 107 1 66 1 1401 11 106 1
Latency 296ms 219ms 226ms 219ms 4952us 523ms
1.96,1.96,smtp10,1,1322548134,,,,,,,,,,,,,,16,,,,,65,1,1044,8,107,1,66,1,1401,11,106,1,,,,,,,296ms,219ms,226ms,219ms,4952us,523ms
Tue Nov 29 00:39:52 CST 2011
This configuration was never used in production, so I'm unsure how helpful it may or may not be.