Archive | DRBD RSS feed for this section

Monitoring DRBD using Nagios and SNMP

9 Oct

Wrote a Naguis script to monitor DRBD using SNMP (I don’t really understand why Nagios made up its own plugin system when you can just use SNMP??), Anyway, to make this work you’ll need to:

1. Add a new ‘Check Command’ into Nagios’ checkcommands.cfg, something like: (the check_smpd_drbd.pl script is below)


define command{
command_name check_snmp_drbd
command_line $USER1$/check_snmp_drbd.pl -h $HOSTADDRESS$
}

2. Add a new ‘service’ definition to your services/**.cfg, something like:


define service{
use generic-service
host_name host_to_monitor
service_description DISK STATUS
is_volatile 0
check_period workhours
max_check_attempts 10
normal_check_interval 5
retry_check_interval 2
contact_groups infoservices-admins
notification_interval 120
notification_period 24x7
notification_options c,r
check_command check_snmp_drbd
}

3. On the target machine, you’ll need to make sure your snmpd daemon is sending you what you want, for Net-SNMP i just changed /etc/snmpd/snmpd.conf appending:


exec drbd_data /sbin/drbdadm state data
exec drbd_home /sbin/drbdadm state home
exec drbd_share /sbin/drbdadm state share

4. Make sure your firewall on the target machine allows snmpd through, you might want to put snmpd in your startup scripts to

5. The script to actually do the monitoring from the Nagios machine is:


#!/usr/bin/perl -w
use strict;
use Getopt::Long;

my %ERRORS=('OK'=>0,'WARNING'=>1,'CRITICAL'=>2,'UNKNOWN'=>3,'DEPENDENT'=>4);
my ($hostname, $snmp_resources, $snmp_values);
my (@resources, @values);
my (@tmp, @tmp2, $tmp3);
my ($key, $value);
my %status = ();
my $i;
my $x;
my $error;

# Get the command line options
# only "-h Hostname"
Getopt::Long::Configure ("bundling");
GetOptions(
'h=s' => \$hostname);

# Grab the snmp details - note this should
# probably use Net::SNMP
$snmp_resources=`/usr/bin/snmpwalk -v 1 -c snmponly $hostname 1.3.6.1.4.1.2021.8.1.2 2>/dev/null`;
$snmp_values=`/usr/bin/snmpwalk -v 1 -c snmponly $hostname 1.3.6.1.4.1.2021.8.1.101 2>/dev/null`;

# Didn't get any output - error
if ($snmp_resources eq "" )
{
print "Unknown host: $hostname\n";
exit $ERRORS{"CRITICAL"};
}

@resources = split(/\n/,$snmp_resources);
@values = split(/\n/,$snmp_values);
for ($i=0;$i< $#resources+1;$i++)
{
@tmp = split(/\s+/,$resources[$i]);
@tmp2 = split(/\s+/,$values[$i]);
$tmp3 = $values[$i];
$tmp3 =~ s/UCD-SNMP-MIB::extOutput..* = STRING: //g;
$status{$tmp[3]} = $tmp3;
}

# Check for "Primary/Secondary" or "Secondary/Primary"
while(($key, $value) = each(%status))
{
if (!($value eq "Primary/Secondary") && !($value eq "Secondary/Primary"))
{
print "ERROR: $key says: $value\n";
$error = 1;
}
}

# Send out status
if ($error)
{
exit $ERRORS{"CRITICAL"};
} else {
print "DRBD OK: ";
foreach $key (keys %status)
{
print $key . ",";
}
print "\n";
exit $ERRORS{"OK"};
}

Advertisements

DRBD

31 Mar

If a machine is in a ‘StandAlone’ state and the other is in ‘WFConnection’, you probably need to take the ‘StandAlone’ one down:

1: cs:StandAlone st:Primary/Unknown ld:Consistent

$> /sbin/drbdadm down testdrbd

then put it ‘up’ again:

$> /sbin/drbdadm up testdrbd

and it should resync

Recovery

29 Nov

DRBD – What to do when something goes wrong

iisjsoo 6 Jan 04
#########################################
This document details the steps required when ONE machine goes down in a DRBD-Heartbeat failover
configuration.

1. A machine (either the Primary or Secondary) has died.

This situation requires immediate action.

If the Primary has died:
In a Heartbeat-failover situation, the Secondary Machine *should have* taken over the Primary Machine’s
services, in this scenario DRBD on the Secondary Machine should still be running and the shared partition
mounted on the Secondary Machine.

You should log into the Secondary Machine and manually check to see that all required services are running.
The services that should be running are documented in /etc/ha.d/haresources
– it is likely that these include (but check the haresources file to be sure):
1. httpd : Apache webserver
2. pks : Public Key Server – required by the EDI
3. named : Bind – the nameserver
4. cvsd : CVS server for external use
5. squid : Squid working as a Reverse Proxy Accelerator (for tunneling IMVS Webmail)
[NOTE: the mailserver, Qmail, runs regardless of Heartbeat, i.e. it is running on the Primary
and Secondary all the time]

It is important to realise that the shared partition (which these services require) could be mounted
*with or without* DRBD being loaded. If it has been mounted *with* DRBD then all data written to this
partition will also be sent to DRBD, hence DRBD knows that there have been changes to the data. However
if the partition was mounted *without* DRBD then DRBD knows nothing about any data written on the
parition by the Secondary Machine.

The real difference between mounting the parition with or without DRBD comes when the dead machine comes
back online. By mounting the partition without DRBD, you are effectively circumventing DRBD, this means
that DRBD’s metadata might be incorrect and indicate that the wrong machine has the good data. In this
event, the reconnection of the Primary Machine may lead to a sync *in the wrong direction* – destroying
your good data!!!

Now the good news – this is easy to avoid as long as you don’t mind a total resync of data (10gig takes
about 30 mins on a 100Mbps network). IN FACT, BY DOING A TOTAL RESYNC YOU CIRCUMVENT ALL PROBLEMS
SPECIFIED ABOVE (ALTHOUGH **IT IS IMPORTANT TO KNOW WHAT IS GOING ON**). Please see step (2).

If the Secondary has died:
No changes will need to be made – the Primary Machine should continue operating okay and will only need
to be stopped in order to begin the resync of data. Please see step (2).

2. Resyncing data
It is recommended that a full resync is done whenever a machine goes down for any length of time or for
hardware maintenance.

Partial resyncs *are* possible, however, a ‘clean slate’ approach is recommended.

To do this, you must delete ALL the metadata on the recovered machine BEFORE it comes back online
– this will force it to resync against the other machine. Note do this ONLY on the machine with the
bad data.

$> rm /var/lib/drbd/drbd[0-9]*

These files will be recreated with zerod contents, thus loosing election as a sync source but they will
become full sync targets (i.e. resyncs in the wrong direction are now impossible!).

Now you will need to restart DRBD on both machines in order to start the sync. Follow the directions
below. The name of the boxen are Good and Bad after the data they contain.

Good> # stop services
Good> umount /data
Good> drbd start
< # IFF you don’t have STONITH, you may start heartbeat here.
It cannot see the Bad node, and will do the following two steps on your behalf.
Good> datadisk start
Good> # start services if you want
# It probably won’t hurt to confirm that /proc/drbd says WFConnect (wait for connection)

Bad> # remove metadata, effectively setting all counters to zero
# This makes sure that it will lose any and all elections about
# “who has the good data”
Bad> rm /var/lib/drbd[0-9]*
Bad> drbd start

This now should connect and start to sync, and show you every now and then the progress of the sync.
(You can also: ‘cat /proc/drbd’ to see what is happening).

If a replication is NOT happening, then you can trigger a full-resync manually:
Bad> drbdsetup replicate

3. Useful tools

3a.
To find out what is going with DRBD:
$> /etc/init.d/drbd status

This will return either:

1. “drbd: /proc/drbd not found. Is DRBD in the kernel?”
DRBD is not loaded into the kernel.
You should not be running a machine in this state. If you have a partition mounted that is
meant to be under the control of DRBD this means that you have circumvented DRBD and mounted
the partition directly. You should unmount the partition, load DRBD into the kernel and then
remount the partition using the datadisk script:

$> unmount /data
$> /etc/init.d/drbd start
$> /etc/ha.d/resource.d/datadisk start

(Note the partitions should have been properly setup in /etc/drbd.conf)

2. “drbd0: stopped”
DRBD is loaded into the kernel but this machine is currently NOT connected to another machine.
This is the correct state a machine should be in when one machine is down.

3. “drbd0: running”
DRBD is loaded and the machine is currently in sync with another.
This is the correct state a machine should be in when both machines are functioning correctly.

3b.
To find out what is going with datadisk:
$> /etc/ha.d/resource.d/datadisk status

This will return either:

1. “datadisk: /proc/drbd not found. Try drbd start first”
This means that drbd is not loaded into the kernel. There is no way datadisk can do anything.

2. “drbd0: stopped”;
NOTE: this is confusing because it says ‘drbd0’ but is really referring to datadisk!
This means that datadisk is not running – hence your partition has not been mounted through
datadisk (if your partition is visible it means that you mounted it manually and circumvented
DRBD/datadisk – which is bad!)

3. “drbd0: running”
NOTE: this is confusing because it says ‘drbd0’ but is really referring to datadisk!
This means that DRBD is running and datadisk has successfully mounted a partition.

Install Notes

25 Jun

DRBD Notes
Compilation
System:
0.6.1 DRBD
7.3 Red Hat
2.4.18-3 Kernel

1. Make sure the kernel source is in /usr/src
– Use the kernel-source rpm for your kernel to install the source if necessary
2. Try to ‘make’ DRBD
• If there are errors like:
drbd_syncer.c: In function ‘drbd_syncer’:
drbd_syncer.c:394 Structure has no member named ‘nice’
Then go to drbd/drbd_syncer.c:
At about line 325,
Comment out: ‘define drbd_set_user_nice(current,x) (current-> nice=(x))’
Put in: ‘define drbd_set_user_nice(current,x) set_user_nice(current,(x))’

Configuration
1. drbd.conf: fsck-cmd should really be fsckcmd
For a journalled file system set this to: fsckcmd=/bin/true
This command will be exec’d when a disks goes down (i.e. useful in ext2)
2. make sure the directory /var/lib/drbd exists
3. make sure drbd.conf is the same on both machines
4. make sure that /data does not get mounted on boot (you are going to let heartbeat control this). So comment out the /data mount in /etc/fstab
Put in a line in /etc/fstab so that /etc/init.d/datadisk will be able to mount the drbd partition:
/dev/nb0 /data ext3 noauto 0 0
5. use /etc/init.d/drbd start to start drbd. Make sure to start the primary node first – when you start the first node it will ask “do you want this machine to be a primary?”, don’t answer but start the second node, drbd will sort out the rest
6. when drbd is up, you should be able to cat /proc/drbd on both machines to see the disks syncing up. The values will be changing, but a big resync (>gigs) may take a long time
7. you can only mount /dev/nb0 on one machine at a time AND only the primary can mount it – make sure you are the primary by checking cat /proc/drbd. To become the primary issue the command drbdsetup /dev/nb0 primary
8. You can format a disk whilst it is syncing, or after. Otherwise you can force the sync to stop using: drbdsetup /dev/nbX disconnect

Issues
1. After doing a mkfs -t ext3 /dev/nb0, the disks will have to re-sync, so be careful. A 12 gig disk do a full re-sync on a 10T network takes about 4 hours (800K/sec), on a 100T network it takes about 30 minutes (7Meg/sec).

Reference
Protocol Description
A A write operation is complete as soon as the data is written to disk and sent to the network.
B A write operation is complete as soon as a reception acknowledgement arrives.
C A write operation is complete as soon as a write acknowledgement arrives.

Heartbeat
Gotchas
Note that you MUST have the drbd service running before heartbeat. (i.e. cat /proc/drbd HAS to give output first, start drbd with /etc/init.d/drbd start). Only then can you start heartbeat. If you start heartbeat before doing this, you’ll end up serving /data on the web which isn’t actually your mounted LAN RAID 1 but just a directory.

If you start heartbeat before drbd, you must STOP heartbeat, then start drbd then re-start heartbeat for the drbd directories to be mounted correctly.

If you stop both DRBD servers, it will have to do a complete ‘SyncingAll’ when you restart – this seems like strange behaviour.

Files on a DRBD are associated with UID’s not usernames or group names. This means that usernames/groups may change when the Primary node changes and the drbd share re-mounted. To get around this problem, make sure that the /etc/passwd files are synchronised for users who are going to be writing to the drbd.
– use usermod command to alter user UID’s

Heartbeat + Drbd Startup
1. Start DRBD on Primary: service drbd start
2. Start DRBD on Secondary: service drbd start
3. Make sure that the Primary is actually primary – issue the drbdsetup /dev/nb0 primary command on the primary node to make sure it is the primary. Check the output of cat /proc/drbd to see which node is the primary.
4.
5. Start Heartbeat on the Primary service heartbeat start
6. Start Heartbeat on the Secondary service heartbeat start

/data should be automatically mounted on the primary
httpd should automatically be started on the primary
Heartbeat + Drbd Shutdown
1. Stop heartbeat on the secondary: service drbd stop
2. Stop heartbeat on the primary: service drbd stop
[Note: this will umount /data for you and stop all services heartbeat is providing]
3. Before stopping drbd, make sure that both nodes are in the Secondary state. I.e. the output of cat /proc/drbd gives Secondary/Secondary
If either note is not in this state, make it so by issuing the command: drbdsetup /dev/nb0 secondary on the node
4. Stop drbd on both nodes: service drbd stop
[Note if you do not shut down drbd in this manner, the disks will do a ‘SyncingAll’ when they are next brought up..]