This guide is based on practical notes taken while tuning DRBD (Distributed Replicated Block Device) on a Linux system. DRBD is a software-based, shared-nothing, replicated storage solution for mirroring the content of block devices (hard disks, partitions, logical volumes, etc.) between servers. Proper tuning of DRBD can significantly improve performance and reliability. This guide will help engineers fine-tune DRBD settings to achieve optimal performance.

Benchmarking with Bonnie++

Before making any changes to the DRBD configuration, it's essential to benchmark the current performance. The bonnie++ tool is used for this purpose. Below are the initial benchmarks:

[root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
Using uid:666, gid:666.
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 50 0 1070 9 107 1 63 1 1296 10 106 1
Latency 283ms 219ms 226ms 331ms 205ms 212ms
1.96,1.96,smtp10,1,1322542347,,,,,,,,,,,,,,16,,,,,50,0,1070,9,107,1,63,1,1296,10,106,1,,,,,,,283ms,219ms,226ms,331ms,205ms,212ms

Adjusting DRBD Parameters

Increasing max-buffers and max-epoch-size

Changing max-buffers and max-epoch-size to 8000 on both nodes (nfs08 and nfs09):

1. Edit the DRBD configuration file:

   nano /etc/drbd.conf
   

2. Add or modify the following settings:

   disk {
       max-buffers 8000;
       max-epoch-size 8000;
   }
   

3. Restart DRBD:

   systemctl restart drbd
   

4. Re-run benchmarks:

   [root@smtp10 ~]# bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/
   

Results

Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1235 9 109 1 65 1 1268 9 107 1
Latency 215ms 219ms 226ms 343ms 204ms 122ms
1.96,1.96,smtp10,1,1322540643,,,,,,,,,,,,,,16,,,,,66,1,1235,9,109,1,65,1,1268,9,107,1,,,,,,,215ms,219ms,226ms,343ms,204ms,122ms

Increasing sndbuf-size

Change drbd sndbuf-size from the default 128k to 512k:

1. Edit the DRBD configuration file:

   nano /etc/drbd.conf
   

2. Add or modify the following setting:

   net {
       sndbuf-size 512k;
   }
   

3. Restart DRBD:

   systemctl restart drbd
   

4. Re-run benchmarks:

   [root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
   

Results

Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 65 1 1050 8 107 1 67 1 1070 8 107 1
Latency 331ms 220ms 224ms 218ms 205ms 185ms
1.96,1.96,smtp10,1,1322537896,,,,,,,,,,,,,,16,,,,,65,1,1050,8,107,1,67,1,1070,8,107,1,,,,,,,331ms,220ms,224ms,218ms,205ms,185ms

The numbers initially worsened, so further testing and adjustments were necessary.

Increasing al-extents

Increase al-extents from 127 to 997:

1. Edit the DRBD configuration file:

   nano /etc/drbd.conf
   

2. Add or modify the following setting:

   disk {
       al-extents 997;
   }
   

3. Restart DRBD:

   systemctl restart drbd
   

4. Re-run benchmarks:

   [root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
   

Results

Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1234 10 108 1 65 1 1277 10 106 1
Latency 73106us 214ms 214ms 293ms 204ms 212ms
1.96,1.96,smtp10,1,1322531581,,,,,,,,,,,,,,16,,,,,66,1,1234,10,108,1,65,1,1277,10,106,1,,,,,,,73106us,214ms,214ms,293ms,204ms,212ms

Adjusting syncer-rate

Adjust syncer-rate from the default of 12M to 100M:

1. Edit the DRBD configuration file:

   nano /etc/drbd.conf
   

2. Add or modify the following setting:

   syncer {
       rate 100M;
   }
   

3. Restart DRBD:

   systemctl restart drbd
   

4. Re-run benchmarks:

   [root@smtp10 ~]# rm -fr /mailstore/benchmark/Bonnie.32558/
   [root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
   

Results

Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 66 1 1030 7 106 1 67 1 1263 9 107 1
Latency 264ms 229ms 226ms 257ms 204ms 209ms
1.96,1.96,smtp10,1,1322521623,,,,,,,,,,,,,,16,,,,,66,1,1030,7,106,1,67,1,1263,9,107,1,,,,,,,264ms,229ms,226ms,257ms,204ms,209ms

Tuning NFS Settings

Begin NFS tuning based on the NFS-HOWTO:

1. Change the /mailstore mount point to use rsize=32768,wsize=32768:

   nano /etc/fstab
   

2. Modify the mount options:

   server:/exported/path /mailstore nfs rsize=32768,wsize=32768 0 0
   

3. Remount the file system:

   mount -o remount /mailstore
   

4. Re-run benchmarks:

   [root@smtp10 ~]# date; bonnie++ -s0 -r 1000 -u 666 -d /mailstore/benchmark/; date
   

Results

Version 1.96 ------Sequential Create------ --------Random Create--------
smtp10 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 65 1

 1044 8 107 1 66 1 1401 11 106 1
Latency 296ms 219ms 226ms 219ms 4952us 523ms
1.96,1.96,smtp10,1,1322548134,,,,,,,,,,,,,,16,,,,,65,1,1044,8,107,1,66,1,1401,11,106,1,,,,,,,296ms,219ms,226ms,219ms,4952us,523ms

Conclusion

This guide provided step-by-step instructions on how to tune DRBD for better performance on a Linux system. By carefully adjusting parameters such as max-buffers, max-epoch-size, sndbuf-size, al-extents, and syncer-rate, and by tuning NFS settings, significant performance improvements can be achieved. These adjustments, tested through benchmarks with bonnie++, show the impact of each change and provide a clear path for engineers looking to optimize their DRBD configurations. While these configurations were not used in production, they offer valuable insights into the tuning process.