I’m duplicating this post here just in case it ever disappears:

This tutorial will assist a home server administrator to configure mdadm e-mail notifications without a full fledged MTA (such as postfix or exim).

Estimated Time: 1 hour

My Goals:

* If a drive fails in any of my RAID arrays, I’d like to be notified by e-mail. The server should tell me, I shouldn’t have to go looking for failures.

* E-mail configuring should be as simple as possible, and should authenticate against my ISP’s SMTP server. I don’t want to have to configure a full Mail Transfer Agent (MTA).

Disclaimer:

* I take no responsibility if, after following these guidelines, your server fails to notify you by e-mail of a failed drive. This tutorial is accurate to the best of my knowledge, and has been tested to work, but use at your own risk. If you have important or critical data, and you’re not completely comfortable with linux server administration, please, get a professional to setup your RAID.

Caveats:

* I’m obviously running a particular setup and distro, but hopefully one could take this instructions and apply them to their own configuration as needed.

System Specs:

* Ubuntu 8.04 Hardy LTS Server (2.6.27-7-generic)
* mdadm v2.6.7.1
* msmtp 1.4.13
* 3x RAID Arrays : /dev/md250 (3x250GB), /dev/md500 (5x500GB), /dev/md1000 (4x1TB)

Preconditions & Assumptions:

I assume here that you have a working server, with a working RAID device already configured. This tutorial does not cover how to setup or configure RAID, but instead how to configure e-mail notifications for an already working mdadm RAID array.

Background Information:

* Firstly. How does e-mail work? View the operations and diagram from wikipedia here.

* Secondly, here’s what our pieces of the puzzle are for e-mail:

Quote:

MUA (the client): msmtp (a sendmail compatible interface)
MTA (the mail server): your isp or gmail
MSA (smtp middle man): msmtp (a simple MTA that gets mail from your local MTA to your real MTA or mailhub)

* As you can see, to simplify things, we’ll be using msmtp as both the MUA and the MSA.

* mdadm e-mail notification by default looks for /usr/sbin/sendmail.

Process the First – Installing and Configuring the MSA (msmtp)

So firstly, we need to install msmtp. On ubuntu, this is trivial:

Code:

sudo apt-get install msmtp

Now it’s time to configure. There are two methods for msmtp configuration; system wide, and user base. We’re interested in a system wide configuration which will apply to all users, unless otherwise overrided by a user’s own msmtp configuration file.

Code:

sudo nano /etc/msmtprc

Quote:

# ------------------------------------------------------------------------------
#msmtp System Wide Configuration file
# ------------------------------------------------------------------------------
# A system wide configuration is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.
# ------------------------------------------------------------------------------
# Accounts
# ------------------------------------------------------------------------------
# Sympatico Account
account sympatico
host smtphm.sympatico.ca
from noreply@yourserver.name
auth on
user valid_email@bell.net
password ********
# Rogers Account
account rogers
host smtp.broadband.rogers.com
port 587
from noreply@yourserver.name
auth login
user valid_email@rogers.com
password ********


# gmail account
# Configuring for gmail is beyond the scope of this tutorial
# Googling for "gmail msmtp" should help


# Other ISP Account
# Configuring for other ISPs is beyond the scope of this tutorial
# Googling for "myisp outlook smtp" should help


# ------------------------------------------------------------------------------
# Configurations
# ------------------------------------------------------------------------------


# Construct envelope-from addresses of the form "user@oursite.example".
#auto_from on
#maildomain fermmy.server


# Use TLS.
#tls on
#tls_trust_file /etc/ssl/certs/ca-certificates.crt


# Syslog logging with facility LOG_MAIL instead of the default LOG_USER.
syslog LOG_MAIL


# Set a default account
account default : rogers


# ------------------------------------------------------------------------------

Save and close this file.

See http://msmtp.sourceforge.net/doc/msm…guration-files for additional details.

Now it’s time to test that e-mail works! First, we’ll run a “pretend test” to make sure the settings we’ve just configured have taken affect.

Code:

msmtp --pretend

Verify the output is correct as best as you know it to be.

Now, we’ll try an actual test.

Code:

echo "This is a test e-mail from my server using msmtp!" | msmtp youremail@domain.com

If all went well, you should receive an e-mail very shortly!

If things didn’t go well, you can check the mail log files (/var/log/mail.log) for hints as to what went wrong, or perhaps even the msmtp executable will spit out some useful errors.

Step the Second – Installing and Configuring our MUA (“sendmail”)

Now, as per the homepage of msmtp, msmtp is a “Sendmail compatible interface (command line options and exit codes).”

This is exciting because it means that if msmtp is compatible with sendmail commands and exit codes, we can simply use msmtp as our e-mail command line executable.

Create a symlink for sendmail to msmtp:

Code:

sudo ln -s /usr/bin/msmtp /usr/sbin/sendmail

Caveat: I’m not sure if this is “kosher” in general with the Linux community. This method worked for me, but if someone has a “proper” solution, I’d be more than happy to hear it.

Step the Third – Configuring MDADM for e-mail notifications

Edit the mdadm configuration file:

Code:

sudo nano /etc/mdadm/mdadm.conf

NOTE: Most distributions expect the mdadm.conf file in /etc/, not /etc/mdadm. I believe this is a “ubuntu-ism” to have it as /etc/mdadm/mdadm.conf.

I’ll avoid details about the DEVICE declarations and such that might already exist in this configuration file, and instead we’ll focus on the MAIL options. If you’re interested in more details on the mdadm.conf, check the man page here: http://man-wiki.net/index.php/5:mdadm.conf.

Quote:

# instruct the monitoring daemon where to send mail alerts

MAILADDR youremail@domain.com

MAILFROM my-server-name – mdadm

Save and close this file.

That’s it!

Let’s run a simple test from mdadm to ensure e-mail notifications are working. If you have DEVICE declarations in your mdadm.conf file, use the following command:

Code:

sudo mdadm --monitor --scan --test --oneshot

This should deliver an e-mail, scanning all devices found in mdadm.conf.

If you rely on mdadm scanning your partitions for RAID devices, you’ll want to specify the devices (usually found as /dev/md###):

Code:

sudo mdadm --monitor --scan --test --oneshot /dev/md[[:digit:]]*

This should deliver an e-mail, with the results from /proc/mdstat. The “/dev/md[[:digit:]]*” portion of the command should match all RAID devices starting with “md” for you automatically.

I then received the following e-mail (actually x3, one for each of my arrays because I have 3):

Quote:

——– Original Message ——–

From: my-server-name – mdadm

Sent: Thu 11 Jun 2009 10:31:37 PM EDT

To: myemail@domain.com

Cc: Subject: TestMessage event on /dev/md500:my-server-name

This is an automatically generated mail message from mdadm

running on my-server-name

A TestMessage event had been detected on md device /dev/md500.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid6] [raid5] [raid4] md1000 : active raid5 sdb1[0]

sda1[3] sdd1[2] sdc1[1]

2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md250 : active raid5 sdh2[0] sdj1[2] sdi1[1]

488375808 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md500 : active raid5 sdk1[0] sdg1[4] sdf1[3] sde1[2] sdh1[1]

1953535744 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

unused devices:

Step the Fourth – Configuring MDADM for Monitoring

At this stage of the game, you’re (hopefully) almost done! The only part left is to ensure that your linux distribution is already monitoring your RAID array, and if it’s not, configure it yourself.

In short, a “monitor” mdadm process must always be running which periodically checks the status of your array(s). This is typical of any RAID system. It’s my understanding that the mdadm monitor will send out e-mail alerts on the following events:

Quote:

Fail, FailSpare, DegradedArray, and TestMessage

These events really are the only things we would typically care to receive an e-mail about. Good.

Now, a typical mdadm monitoring mode process command might like like:

Code:

mdadm --monitor --scan --daemonize --test --syslog (/dev/md[[:digit:]]*)

Let’s break down the above command:

* –monitor –> Sets the mdadm process to “Monitor Mode”.

* –scan –> Scans the mdadm.conf file for RAID DEVICES.

* –daemonize –> This process should be treated as a daemon.

* –test –> On system startup, send out test e-mail(s).

* –syslog –> Log to the system log (/var/logs/syslog)

* The last portion is optional(?) for those who don’t have devices defined in the RAID configuration file.

For addition details on mdadm monitor mode, try mdadm –monitor –help or review the man file for monitor mode at http://man-wiki.net/index.php/8:mdadm#For_Monitor_mode:.

For my case in particular, Ubuntu 8.04, the monitoring of all RAID arrays had already been implemented, and started on system startup. This was realized by the file /etc/init.d/mdadm, and it’s debian configuration options defined in /etc/default/mdadm.

I prefer to have a test e-mail on startup, so in the /etc/default/mdadm option file, I changed

Quote:

DAEMON_OPTIONS=”–syslog”

to

Quote:

DAEMON_OPTIONS=”–syslog –test”

or (OPTIONAL)

Quote:

DAEMON_OPTIONS=”–syslog –test (/dev/md[[:digit:]]*)”

NOTE: As per the beginning of “Step the Fourth”, if you don’t have any devices defined in your mdadm.conf configuration file, you need to suffix the DAEMON_OPTIONS with “(/dev/md[[:digit:]]*)”

To verify that the mdadm monitor process is actually running, try:

Code:

ps aux | grep mdadm

Result in my case, showing the monitor process is already running:

Quote:

root 6230 0.0 0.0 2160 600 ? Ss May29 0:04 /sbin/mdadm –monitor –pid-file /var/run/mdadm/monitor.pid –daemonise –scan –syslog –test

or (OPTIONAL)

Quote:

root 21021 0.0 0.0 2160 584 ? Ss 20:12 0:00 /sbin/mdadm –monitor –pid-file /var/run/mdadm/monitor.pid –daemonise –scan –syslog –test /dev/md1000 /dev/md250 /dev/md500

If your linux distribution doesn’t already have the mdadm monitoring daemon, you’ll have to configure this yourself, but I’m afraid this is beyond the scope of this tutorial. I suggest Googling “mdadm monitor daemon” for help, and in particular, this is useful: http://linux-raid.osdl.org/index.php…ng_RAID_arrays.

Good luck! Once you know a mdadm monitor daemon is running, you can simulate a failure to verify that the system will indeed e-mail you, which is what we’re after right?

* I found this website very useful: http://linux-raid.osdl.org/index.php…_drive_failure

By this point, you should find comfort in the knowledge that your server will tell YOU when a drive fails!

Pasted from <http://ubuntuforums.org/showthread.php?t=1185134>

And here is the way to fail a drive in a software raid array through commandline:

Force-fail by software

You can just simulate a drive failure without unplugging things. Just running the command

mdadm --manage --set-faulty /dev/md1 /dev/sdc2

should be enough to fail the disk /dev/sdc2 of the array /dev/md1.

Now things move up and fun appears. First, you should see something like the first line of this on your system’s log. Something like the second line will appear if you have spare disks configured.

kernel: raid1: Disk failure on sdc2, disabling device.
kernel: md1: resyncing spare disk sdb7 to replace failed disk

Checking /proc/mdstat out will show the degraded array. If there was a spare disk available, reconstruction should have started.

Another useful command at this point is:

mdadm --detail /dev/md1

Enjoy the view.

Now you’ve seen how it goes when a device fails. Let’s fix things up.

First, we will remove the failed disk from the array. Run the command

mdadm /dev/md1 -r /dev/sdc2

Note that mdadm cannot pull a disk out of a running array. For obvious reasons, only faulty disks can be hot-removed from an array (even stopping and unmounting the device won’t help – if you ever want to remove a ‘good’ disk, you have to tell the array to put it into the ‘failed’ state as above).

Now we have a /dev/md1 which has just lost a device. This could be a degraded RAID or perhaps a system in the middle of a reconstruction process. We wait until recovery ends before setting things back to normal.

So the trip ends when we send /dev/sdc2 back home.

mdadm /dev/md1 -a /dev/sdc2

As the prodigal son returns to the array, we’ll see it becoming an active member of /dev/md1 if necessary. If not, it will be marked as a spare disk. That’s management made easy

Pasted from <http://linux-raid.osdl.org/index.php/Detecting,_querying_and_testing#Monitoring_RAID_arrays>

Tags: email, linux, mdadm, raid