[PLUG] Unable to reactivate a RAID10 mdadm device

Arun Khan knura9 at gmail.com
Tue Feb 12 00:23:12 PST 2013


Recovery OS -- System Rescue CD v 2.8.0

Production OS - Debian Squeeze (6) 2.6.32 stock kernel, using mdadm raid

/dev/md0 in raid level RAID10 - members /dev/sdb1, /dev/sdc1,
/dev/sdd1, /dev/sde1 all with partion id=fd

HDD /dev/sdb went bad, replaced it with another disk with same size
partion (id=fd)
using System Rescue CD v2.8.0

1. System Rescue CD recognized the md devices but it comes up as 'inactive'

Searched for possible solutions and I have tried several things including
zeroing the super block and adding them back to the array.

Still unable to bring back /dev/md0 with all 4 partions in active mode.

I have included below, the entire transcript of the commands I have
tried to recover /dev/md0

I have data on /dev/md0 that I need. I do have back ups of critical
files (but not all).

I prefer solving the problem v/s recreating the /dev/md0 from scratch.

Any help in solving this problem would be highly appreciated.

TIA,
-- Arun Khan

---- transcript of mdadm activity  with System Rescue CD v2.8.0  ----

# mdadm -V
mdadm - v3.1.4 - 31st August 2010

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sdd1[2] sde1[3]
      312574512 blocks super 1.0

# mdadm -S /dev/md0
mdadm: stopped /dev/md0

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
unused devices: <none>


# mdadm -v -v -A /dev/md0 -R /dev/sd[bcde]1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: added /dev/sdc1 to /dev/md0 as 1
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sdd1[2] sde1[3]
      312574512 blocks super 1.0

unused devices: <none>

from /var/log/messages
Feb 12 06:13:59 sysresccd kernel: [ 7593.339015] md: md0 stopped.
Feb 12 06:13:59 sysresccd kernel: [ 7593.374016] md: bind<sdb1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374417] md: bind<sdc1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374604] md: bind<sde1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374869] md: bind<sdd1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.374899] md: kicking non-fresh
sdc1 from array!
Feb 12 06:13:59 sysresccd kernel: [ 7593.374903] md: unbind<sdc1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.379016] md: export_rdev(sdc1)
Feb 12 06:13:59 sysresccd kernel: [ 7593.379041] md: kicking non-fresh
sdb1 from array!
Feb 12 06:13:59 sysresccd kernel: [ 7593.379044] md: unbind<sdb1>
Feb 12 06:13:59 sysresccd kernel: [ 7593.386010] md: export_rdev(sdb1)
Feb 12 06:13:59 sysresccd kernel: [ 7593.387382] md/raid10:md0: not
enough operational mirrors.
Feb 12 06:13:59 sysresccd kernel: [ 7593.387410] md: pers->run() failed ...

# mdadm -D /dev/md0
/dev/md0:
        Version : 1.0
  Creation Time : Fri Apr 29 04:27:04 2011
     Raid Level : raid10
  Used Dev Size : 156287232 (149.05 GiB 160.04 GB)
   Raid Devices : 4
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Feb 11 13:43:52 2013
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 32K

           Name : brahmaputra:0
           UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
         Events : 783527

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       0        0        1      removed
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1


# mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
           Name : brahmaputra:0
  Creation Time : Fri Apr 29 04:27:04 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
     Array Size : 625148928 (298.09 GiB 320.08 GB)
  Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
   Super Offset : 312574640 sectors
          State : clean
    Device UUID : 2fbc103e:ca40e0c2:b8e4d64f:0fbc7b94

Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Feb 11 13:43:52 2013
       Checksum : 2e8e9fad - correct
         Events : 0

         Layout : near=2
     Chunk Size : 32K

   Device Role : spare
   Array State : ..AA ('A' == active, '.' == missing)


#  mdadm -E /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
           Name : brahmaputra:0
  Creation Time : Fri Apr 29 04:27:04 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
     Array Size : 625148928 (298.09 GiB 320.08 GB)
  Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
   Super Offset : 312574640 sectors
          State : clean
    Device UUID : e27c187b:9004cb93:5bb05639:164822cd

Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Feb 11 13:43:52 2013
       Checksum : 5ea77bd0 - correct
         Events : 0

         Layout : near=2
     Chunk Size : 32K

   Device Role : spare
   Array State : ..AA ('A' == active, '.' == missing)

#  mdadm -E /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
           Name : brahmaputra:0
  Creation Time : Fri Apr 29 04:27:04 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
     Array Size : 625148928 (298.09 GiB 320.08 GB)
  Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
   Super Offset : 312574640 sectors
          State : clean
    Device UUID : 78fc82bc:2eb18f07:56c98922:7639269e

Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Feb 11 13:43:52 2013
       Checksum : ce19a703 - correct
         Events : 783527

         Layout : near=2
     Chunk Size : 32K

   Device Role : Active device 3
   Array State : ..AA ('A' == active, '.' == missing)

#  mdadm -E /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7
           Name : brahmaputra:0
  Creation Time : Fri Apr 29 04:27:04 2011
     Raid Level : raid10
   Raid Devices : 4

 Avail Dev Size : 312574512 (149.05 GiB 160.04 GB)
     Array Size : 625148928 (298.09 GiB 320.08 GB)
  Used Dev Size : 312574464 (149.05 GiB 160.04 GB)
   Super Offset : 312574640 sectors
          State : clean
    Device UUID : 1dce2f63:fef488cb:1a362c57:7ed908ac

Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Feb 11 13:43:52 2013
       Checksum : 7b44c944 - correct
         Events : 783527

         Layout : near=2
     Chunk Size : 32K

   Device Role : Active device 2
   Array State : ..AA ('A' == active, '.' == missing)

# mdadm --zero-superblock /dev/sdb1

# mdadm -E /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.

# mdadm /dev/md0 --add /dev/sdb1
mdadm: add new device failed for /dev/sdb1 as 4: Invalid argument

from /var/log/messages
Feb 12 06:52:26 sysresccd kernel: [ 9900.298880] md0: ADD_NEW_DISK not supported



More information about the plug-mail mailing list