Dear Maintainer,
*** Please consider answering these questions, where appropriate ***

I attached 4 disks to my computer (already booted) and used mdadm to
create a new raid5 volume, created a partition, formatted it, wrote data
on it.

   mdadm /dev/md/doom --create -e 1.0 -z 3900390625 -v -n4 -c 256 -l 5 -p ddf-zero-restart --assume-clean \
      ata-HGST_HDS724040ALE640_1 \
      ata-HGST_HDS724040ALE640_2 \
      ata-HGST_HDS724040ALE640_3 \
      ata-HGST_HDS724040ALE640_4

Later, i deattached it, and on the next day, reattached it again, used mdadm to assemble the array, everything worked fine:

   mdadm -A /dev/md/doom \
      ata-HGST_HDS724040ALE640_1 \
      ata-HGST_HDS724040ALE640_2 \
      ata-HGST_HDS724040ALE640_3 \
      ata-HGST_HDS724040ALE640_4

Then I rebooted. After the reboot, I again tried to assemble the array, to no avail:

   mdadm: no RAID superblock on ata-HGST_HDS724040ALE640_2
   mdadm: ata-HGST_HDS724040ALE640_2 has no superblock - assembly aborted

Now, when I try to recreate the array (first mdadm command), I get:

   mdadm: ata-HGST_HDS724040ALE640_1 appears to be part of a raid array:
          level=raid5 devices=4 ctime=Thu Oct 31 15:13:13 2013
   mdadm: ata-HGST_HDS724040ALE640_2 appears to be part of a raid array:
          level=raid0 devices=0 ctime=Thu Jan  1 01:00:00 1970
   mdadm: partition table exists on ata-HGST_HDS724040ALE640_2 but will be lost or
          meaningless after creating array
   mdadm: ata-HGST_HDS724040ALE640_3 appears to be part of a raid array:
          level=raid5 devices=4 ctime=Thu Oct 31 15:13:13 2013
   mdadm: ata-HGST_HDS724040ALE640_4 appears to be part of a raid array:
          level=raid5 devices=4 ctime=Thu Oct 31 15:13:13 2013
   mdadm: automatically enabling write-intent bitmap on large array
   Continue creating array?

At that point I looked at /dev/mdstat, and saw that the array was already
assembled - without the disk #2.

   Personalities : [raid6] [raid5] [raid4]
   md127 : active (auto-read-only) raid5 sdb[0] sdf[3] sde[2]
         11701171200 blocks super 1.0 level 5, 256k chunk, algorithm 1 [4/3] [U_UU]

         unused devices: <none>

There are two pairs of disks, and each is on a different on-board
controller, so I cannot explain why, apperently, one disk was kicked out
on reboot (there are no I/O errors in the kernel log).

In any case, the message that mdadm prints (... has no superblock) is
simply wrong.

Formatting the drive with --assume-clean made the disks reappear - I know
they must be clean because the device was unmounted and not in use before
and after the reboot, so nothing should have written to it.

*** End of the template - remove these lines ***
--- /etc/default/mdadm
INITRDSTART='none'
AUTOSTART=true
AUTOCHECK=true
START_DAEMON=true
DAEMON_OPTIONS="--syslog"
VERBOSE=false

--- /proc/partitions:
major minor  #blocks  name

   8       16 3907018584 sdb
   8        0 3907018584 sda
   8       32  117220824 sdc
   8       33  117220760 sdc1
   8       48  732574584 sdd
   8       49   40000000 sdd1
   8       50  105500001 sdd2
   8       51  587074551 sdd3
   8       64 3907018584 sde
   8       80 3907018584 sdf
   8       96 9765570560 sdg
   8       97 9765570479 sdg1
   8      112 20507734016 sdh
   8      113 20507732975 sdh1
 253        0   68157440 dm-0
 253        1   19714048 dm-1
 253        2      65536 dm-2
 253        3  488267776 dm-3
 253        4      65536 dm-4
 253        5 19971600384 dm-5
 253        6     131072 dm-6
 253        7 9277300736 dm-7
 253        8    8388608 dm-8
 253        9   14544896 dm-9
 253       10   14540800 dm-10
 253       11     131072 dm-11

--- LVM physical volumes:
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sdc1  vg_cerebro lvm2 a--  111.79g      0
  /dev/sdg1  vg_cerebro lvm2 a--    9.09t      0
  /dev/sdh1  vg_cerebro lvm2 a--   19.10t 503.17g
--- mount output
/dev/mapper/vg_cerebro-root on / type ext4 (rw,noatime)
none on /run type tmpfs (rw)
none on /run/shm type tmpfs (rw)
/dev/sdd2 on /oldroot type btrfs (rw,noatime,nodatacow,autodefrag,compress=lzo)
/oldroot/tmp on /tmp type none (rw,bind)
/dev/mapper/vg_cerebro-localvol on /localvol type xfs (rw,noatime,nobarrier,inode64,largeio,logdev=/dev/vg_cerebro/log_localvol)
/run/shm on /dev/shm type none (rw,bind)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

--- initrd.img-3.10-3-amd64:
82419 blocks
f80f3cb601df7d0849bbd0f715c9d349  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-crypt.ko
84a4ccc5f118ab6c9f0836978c99c21c  ./lib/modules/3.10-3-amd64/kernel/drivers/md/raid10.ko
89ba57921e3d5f69e27144bfdb0fe82f  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-log.ko
ce81f445a5572c19734ae89e7161558b  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-snapshot.ko
a794165ff31bec5ae51f97fb9c050ccb  ./lib/modules/3.10-3-amd64/kernel/drivers/md/md-mod.ko
eea1ec841daa8e7c08e49dff55f19b1f  ./lib/modules/3.10-3-amd64/kernel/drivers/md/raid456.ko
4e6dd9f979f93932a4f9be8f0d162dcc  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-region-hash.ko
351beabdd5cf0d23c2d7b816d290e8ee  ./lib/modules/3.10-3-amd64/kernel/drivers/md/linear.ko
97d3a620d6c39d5521bcaee8a191f440  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-mod.ko
fff39544c2a811e44dd3f88d07a4870f  ./lib/modules/3.10-3-amd64/kernel/drivers/md/dm-mirror.ko
9d3a5739c24beea024604bef5b3f6f75  ./lib/modules/3.10-3-amd64/kernel/drivers/md/multipath.ko
3ef426a3f3c2108b5ebd0019290da783  ./lib/modules/3.10-3-amd64/kernel/drivers/md/raid1.ko
4fead3e821754a804ff9940c25552559  ./lib/modules/3.10-3-amd64/kernel/drivers/md/raid0.ko
f338efe0b2ecaa2938c26a4331667b0b  ./conf/mdadm
82cb072fb935cfb02e94dbaf556de7c5  ./scripts/local-top/mdadm
c97a8b860757795b8a4d692d0c4c586b  ./etc/mdadm/mdadm.conf
18e4e8f135699c0d908f0dec8e314f63  ./sbin/mdadm

--- initrd's /conf/conf.d/md:
no conf/md file.

--- /proc/modules:
dm_crypt 18457 0 - Live 0xffffffffa077b000
dm_mod 64008 32 dm_crypt, Live 0xffffffffa034f000
raid456 57159 0 - Live 0xffffffffa033b000
async_raid6_recov 12574 1 raid456, Live 0xffffffffa0336000
async_memcpy 12426 2 raid456,async_raid6_recov, Live 0xffffffffa022e000
async_pq 12547 2 raid456,async_raid6_recov, Live 0xffffffffa0229000
async_xor 12422 3 raid456,async_raid6_recov,async_pq, Live 0xffffffffa0224000
async_tx 12566 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor, Live 0xffffffffa0171000
raid6_pq 95431 3 btrfs,async_raid6_recov,async_pq, Live 0xffffffffa031d000
md_mod 92559 1 raid456, Live 0xffffffffa0305000

--- /var/log/syslog:

--- volume detail:
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
--
/dev/sdb:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 8ca7705b:520cc03d:2c5d8c1d:16161f48
           Name : cerebro:doom  (local to host cerebro)
  Creation Time : Thu Oct 31 15:13:13 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7814036896 (3726.02 GiB 4000.79 GB)
     Array Size : 11701171200 (11159.11 GiB 11982.00 GB)
  Used Dev Size : 7800780800 (3719.70 GiB 3994.00 GB)
   Super Offset : 7814037152 sectors
   Unused Space : before=0 sectors, after=13256352 sectors
          State : clean
    Device UUID : 6ed5a266:877ac5c8:bffad201:8b18733e

    Update Time : Mon Nov  4 00:13:23 2013
       Checksum : 3e2e55e5 - correct
         Events : 188

         Layout : right-asymmetric
     Chunk Size : 256K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
--
/dev/sdc:
   MBR Magic : aa55
Partition[0] :    234441520 sectors at          128 (type 8e)
--
/dev/sdc1 is not recognised by mdadm.
/dev/sdd:
   MBR Magic : aa55
Partition[0] :     80000000 sectors at           63 (type 07)
Partition[1] :    211000003 sectors at     80000063 (type 83)
Partition[2] :   1174149102 sectors at    291000066 (type 07)
--
/dev/sdd1:
   MBR Magic : aa55
Partition[0] :   1836016416 sectors at   1936269394 (type 4f)
Partition[1] :    544437093 sectors at   1917848077 (type 73)
Partition[2] :    544175136 sectors at   1818575915 (type 2b)
Partition[3] :        54974 sectors at   2844524554 (type 61)
--
/dev/sdd2 is not recognised by mdadm.
/dev/sdd3:
   MBR Magic : aa55
Partition[0] :   1836016416 sectors at   1936269394 (type 4f)
Partition[1] :    544437093 sectors at   1917848077 (type 73)
Partition[2] :    544175136 sectors at   1818575915 (type 2b)
Partition[3] :        54974 sectors at   2844524554 (type 61)
--
/dev/sde:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 8ca7705b:520cc03d:2c5d8c1d:16161f48
           Name : cerebro:doom  (local to host cerebro)
  Creation Time : Thu Oct 31 15:13:13 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7814036896 (3726.02 GiB 4000.79 GB)
     Array Size : 11701171200 (11159.11 GiB 11982.00 GB)
  Used Dev Size : 7800780800 (3719.70 GiB 3994.00 GB)
   Super Offset : 7814037152 sectors
   Unused Space : before=0 sectors, after=13256352 sectors
          State : clean
    Device UUID : 24c1a2c9:a8b451a1:8357fd85:9761e2c2

    Update Time : Mon Nov  4 00:13:23 2013
       Checksum : 8254218f - correct
         Events : 188

         Layout : right-asymmetric
     Chunk Size : 256K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
--
/dev/sdf:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 8ca7705b:520cc03d:2c5d8c1d:16161f48
           Name : cerebro:doom  (local to host cerebro)
  Creation Time : Thu Oct 31 15:13:13 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7814036896 (3726.02 GiB 4000.79 GB)
     Array Size : 11701171200 (11159.11 GiB 11982.00 GB)
  Used Dev Size : 7800780800 (3719.70 GiB 3994.00 GB)
   Super Offset : 7814037152 sectors
   Unused Space : before=0 sectors, after=13256352 sectors
          State : clean
    Device UUID : 4bfc314a:505aa204:524f6cd2:ebe634a9

    Update Time : Mon Nov  4 00:13:23 2013
       Checksum : 98f57f81 - correct
         Events : 188

         Layout : right-asymmetric
     Chunk Size : 256K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
--
/dev/sdg:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
--
/dev/sdg1 is not recognised by mdadm.
/dev/sdh:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
--
/dev/sdh1 is not recognised by mdadm.
/dev/sdi is not recognised by mdadm.
/dev/sdj is not recognised by mdadm.

--- /proc/cmdline
auto BOOT_IMAGE=3.10.3 ro root=/dev/vg_cerebro/root rootfstype=ext4 rootdelay=5 relatime ide_core.trim=1 elevator=cfq transparent_hugepage=always kernelcore=6G ip=10.0.0.1::10.0.0.5:255.255.255.240:cerebro-preboot:eth0:off pcie_aspm=force intel_pstate=1

--- grub2:
set root='(vg_cerebro-root)'
  set root='(vg_cerebro-root)'
set root='(vg_cerebro-root)'
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg_cerebro-root ro  quiet
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.10-1-amd64 root=/dev/mapper/vg_cerebro-root ro  quiet
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.10-1-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.9-0.bpo.1-amd64 root=/dev/mapper/vg_cerebro-root ro  quiet
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-3.9-0.bpo.1-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-2.6.39-2-amd64 root=/dev/mapper/vg_cerebro-root ro  quiet
	set root='(vg_cerebro-root)'
	linux	/boot/vmlinuz-2.6.39-2-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(vg_cerebro-root)'
	set root='(vg_cerebro-root)'
	set root='(vg_cerebro-root)'
	set root='(vg_cerebro-root)'
	set root='(vg_cerebro-root)'
	set root='(hd3,msdos1)'
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-3.10-1-amd64 root=/dev/mapper/vg_cerebro-root ro quiet
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-3.10-1-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-3.9-0.bpo.1-amd64 root=/dev/mapper/vg_cerebro-root ro quiet
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-3.9-0.bpo.1-amd64 root=/dev/mapper/vg_cerebro-root ro single
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-2.6.39-2-amd64 root=/dev/mapper/vg_cerebro-root ro quiet
	set root='(hd3,msdos2)'
	linux /boot/vmlinuz-2.6.39-2-amd64 root=/dev/mapper/vg_cerebro-root ro single

--- lilo:
append = "root=/dev/vg_cerebro/root rootfstype=ext4 rootdelay=5 relatime ide_core.trim=1 elevator=cfq transparent_hugepage=always kernelcore=6G ip=10.0.0.1::10.0.0.5:255.255.255.240:cerebro-preboot:eth0:off pcie_aspm=force intel_pstate=1"

--- udev:
ii  udev           175-7.2      amd64        /dev/ and hotplug management daem
31f02091e9d9fc06396df81f4f9e7668  /lib/udev/rules.d/63-md-raid-arrays.rules
ec25c304457feb2f2951160ba551fb6f  /lib/udev/rules.d/64-md-raid-assembly.rules

--- /dev:
brw-rw---T 1 root disk 9, 127 Nov  4 03:15 /dev/md127

/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root  9 Nov  4 03:12 ata-HGST_HDS724040ALE640_PK1331PAK8KJSS -> ../../sdb
lrwxrwxrwx 1 root root  9 Nov  4 03:04 ata-HGST_HDS724040ALE640_PK1331PAKA7ENS -> ../../sda
lrwxrwxrwx 1 root root  9 Nov  4 03:12 ata-HGST_HDS724040ALE640_PK1331PAKABUYS -> ../../sde
lrwxrwxrwx 1 root root  9 Nov  4 03:12 ata-HGST_HDS724040ALE640_PK1331PAKAKXAS -> ../../sdf
lrwxrwxrwx 1 root root  9 Nov  4 03:12 ata-OCZ-VERTEX2_OCZ-49X05Y4CIV859B13 -> ../../sdc
lrwxrwxrwx 1 root root 10 Nov  4 03:04 ata-OCZ-VERTEX2_OCZ-49X05Y4CIV859B13-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 ata-SAMSUNG_HD753LJ_S13UJ1MQ107524 -> ../../sdd
lrwxrwxrwx 1 root root 10 Nov  4 03:04 ata-SAMSUNG_HD753LJ_S13UJ1MQ107524-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 ata-SAMSUNG_HD753LJ_S13UJ1MQ107524-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 ata-SAMSUNG_HD753LJ_S13UJ1MQ107524-part3 -> ../../sdd3
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-bp -> ../../dm-7
lrwxrwxrwx 1 root root 11 Nov  4 03:04 dm-name-vg_cerebro-cache_bp -> ../../dm-10
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-cache_wd -> ../../dm-9
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-db -> ../../dm-1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-localvol -> ../../dm-3
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-log_bp -> ../../dm-2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-log_localvol -> ../../dm-4
lrwxrwxrwx 1 root root 11 Nov  4 03:04 dm-name-vg_cerebro-log_lv5 -> ../../dm-11
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-log_wd -> ../../dm-6
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-root -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-swap -> ../../dm-8
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-name-vg_cerebro-wd -> ../../dm-5
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9b34EqkTm9QnBnEcFVyVl7ZqY3wVIhZdkt -> ../../dm-9
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9b9XLvz3DQyUo3s6Sb290l5B7TYK7fVtkZ -> ../../dm-7
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bCtfK7aQ7tpVbOhg5kj2jmVU3gNeUe3FJ -> ../../dm-3
lrwxrwxrwx 1 root root 11 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bJ38bYbdiwgCA7RNdA5OTzGo1hraZKQPc -> ../../dm-10
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bO2pO3ORf0mw4sg3Dz0hTnZaJVcnF7FL1 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bRh1Jp1w7KcYBBcb6RlOQIIsuc6wdESMX -> ../../dm-8
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bVqNskLTE9yYCH7Bf8ijBQeWiX9hvkY7r -> ../../dm-6
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bXG8NLgOeUPwlXeeZznxDk36lb0RtGQl5 -> ../../dm-2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bfyBz5mPzrUB7wlPXZHs3Ii6PS3VR1x3D -> ../../dm-4
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bhMCYkW3Lm93XrVd1WzvDqvuc2UONp9BT -> ../../dm-1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9bmFc31MW2VdpExbHpBrk58JgXBgRXby1H -> ../../dm-5
lrwxrwxrwx 1 root root 11 Nov  4 03:04 dm-uuid-LVM-OEnuTsrJ9wb7lF2KUJHUaboykoIuwv9btOZaUJJDO23Q63AxAIND2uQFR1eAevGQ -> ../../dm-11
lrwxrwxrwx 1 root root  9 Nov  4 03:04 scsi-3600050e000064a009435000026b50000 -> ../../sdh
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-3600050e000064a009435000026b50000-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 scsi-3600050e09c553a005c8d0000acb70000 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-3600050e09c553a005c8d0000acb70000-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 Nov  4 03:12 scsi-SATA_HGST_HDS724040A_PK1331PAK8KJSS -> ../../sdb
lrwxrwxrwx 1 root root  9 Nov  4 03:04 scsi-SATA_HGST_HDS724040A_PK1331PAKA7ENS -> ../../sda
lrwxrwxrwx 1 root root  9 Nov  4 03:12 scsi-SATA_HGST_HDS724040A_PK1331PAKABUYS -> ../../sde
lrwxrwxrwx 1 root root  9 Nov  4 03:12 scsi-SATA_HGST_HDS724040A_PK1331PAKAKXAS -> ../../sdf
lrwxrwxrwx 1 root root  9 Nov  4 03:12 scsi-SATA_OCZ-VERTEX2_OCZ-49X05Y4CIV859B13 -> ../../sdc
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-SATA_OCZ-VERTEX2_OCZ-49X05Y4CIV859B13-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 scsi-SATA_SAMSUNG_HD753LJS13UJ1MQ107524 -> ../../sdd
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-SATA_SAMSUNG_HD753LJS13UJ1MQ107524-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-SATA_SAMSUNG_HD753LJS13UJ1MQ107524-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 scsi-SATA_SAMSUNG_HD753LJS13UJ1MQ107524-part3 -> ../../sdd3
lrwxrwxrwx 1 root root  9 Nov  4 03:04 usb-Generic_Ultra_HS-SD_MMC_000000264001-0:0 -> ../../sdi
lrwxrwxrwx 1 root root  9 Nov  4 03:04 wwn-0x50000f000b015742 -> ../../sdd
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x50000f000b015742-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x50000f000b015742-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x50000f000b015742-part3 -> ../../sdd3
lrwxrwxrwx 1 root root  9 Nov  4 03:12 wwn-0x5000cca22bee2b53 -> ../../sdb
lrwxrwxrwx 1 root root  9 Nov  4 03:04 wwn-0x5000cca22beeee27 -> ../../sda
lrwxrwxrwx 1 root root  9 Nov  4 03:12 wwn-0x5000cca22beefea8 -> ../../sde
lrwxrwxrwx 1 root root  9 Nov  4 03:12 wwn-0x5000cca22bef1578 -> ../../sdf
lrwxrwxrwx 1 root root  9 Nov  4 03:12 wwn-0x5e83a97fa08d4ce7 -> ../../sdc
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x5e83a97fa08d4ce7-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 wwn-0x600050e000064a009435000026b50000 -> ../../sdh
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x600050e000064a009435000026b50000-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 wwn-0x600050e09c553a005c8d0000acb70000 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov  4 03:04 wwn-0x600050e09c553a005c8d0000acb70000-part1 -> ../../sdg1

/dev/disk/by-label:
total 0
lrwxrwxrwx 1 root root 12 Nov  4 03:05 DB2 -> ../../loop19
lrwxrwxrwx 1 root root 10 Nov  4 03:04 OLDROOT -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 ROOT -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov  4 03:04 WIN_C -> ../../sdd1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 WIN_X -> ../../sdd3

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root  9 Nov  4 03:04 pci-0000:00:1a.0-usb-0:1.1.1.1:1.0-scsi-0:0:0:0 -> ../../sdi
lrwxrwxrwx 1 root root  9 Nov  4 03:04 pci-0000:00:1a.0-usb-0:1.2.1.1:1.0-scsi-0:0:0:0 -> ../../sdj
lrwxrwxrwx 1 root root  9 Nov  4 03:12 pci-0000:00:1f.2-scsi-0:0:0:0 -> ../../sdc
lrwxrwxrwx 1 root root 10 Nov  4 03:04 pci-0000:00:1f.2-scsi-0:0:0:0-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Nov  4 03:04 pci-0000:00:1f.2-scsi-0:0:0:0-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 pci-0000:00:1f.2-scsi-0:0:0:0-part3 -> ../../sdd3
lrwxrwxrwx 1 root root  9 Nov  4 03:04 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov  4 03:04 pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 Nov  4 03:04 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdh
lrwxrwxrwx 1 root root 10 Nov  4 03:04 pci-0000:03:00.0-scsi-0:0:1:0-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Nov  4 03:12 pci-0000:06:00.0-scsi-0:0:0:0 -> ../../sde

/dev/disk/by-uuid:
total 0
lrwxrwxrwx 1 root root 10 Nov  4 03:04 1ED6E0EDD6E0C5DD -> ../../sdd3
lrwxrwxrwx 1 root root 10 Nov  4 03:04 3352bc4d-2085-49db-b598-f51262b0cfc8 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Nov  4 03:04 345b10aa-73e4-41c3-878e-08a18383c71e -> ../../sdd2
lrwxrwxrwx 1 root root 10 Nov  4 03:04 36088a56-487f-4b2e-9ffb-df992fdeeecd -> ../../dm-3
lrwxrwxrwx 1 root root 10 Nov  4 03:04 A21C45C71C4596E7 -> ../../sdd1
lrwxrwxrwx 1 root root 12 Nov  4 03:05 ee93ab3c-4032-445a-9c8c-e54df323ec2b -> ../../loop19

/dev/md:
total 0

Auto-generated on Mon, 04 Nov 2013 03:16:55 +0100
by mdadm bugscript 3.3-2

[Replying to a relatively old bugreport]

04.11.2013 06:27, Marc Lehmann wrote:

I'm not sure what to make out of this bugreport really.  Tagging with
`moreinfo' for now.

I don't know what happened, and how superblock temporarily disappeared
from the device ata-HGST_HDS724040ALE640_2.  But what mdadm tells you
is what it finds.

The only way to verify what's going on is to see what mdadm does when
you asked it to assemble the array when it complained about missing
superblock.  Like, running it under strace to see what it is doing and
what system returns to its requests.

What I notice however is that ata-HGST_HDS724040ALE640_2 contains a
partition table, and mdadm correctly detects this.  So this is a first
data device in the array, and actual data starts there.  With superblock
format v. 1.0, this is also the start of the drive, ie, the same partition
table exists on this drive and on your array.  This is a sure way of
making problems, since the system first see your ata drive and tries to
parse partition table in there, and might even try to mount some partitions,
making the drive busy, so mdadm is unable to open it for assembly.

When you create raid5 array, use superblock version 1.1 or 1.2 (1.2 is
the default), so no other tools will try to access your component devices
outside the array.

With this context I'd say it is an operator error.

Thanks,

/mjt

Not sure what more info I could deliver, it seems to be clear from the
bugreport as-is. However, the volume is long gone, so I now have the same
information as you.

Nothing in my system will randomly mount partitions it hasn't explicitly
been asked to mount.

In any case, if this were true,. it's simply a bug in mdadm - mdadm should
not say "there is no superblock" when there really is a superblock but
something simply kept it from opening the device.
to report an actual bug in the tool, I didn't try to abuse the bug tracker to
get support. I don't have an actual problem with mdadm, I simply reported
that mdadm gives a erong error message.

Again, I think you misunderstood - I was reporting a bug, not asking for
support. It might or might not be true that, what I did is not the optimal
way, or the way it should be done, or the state-of-the-art-technique to do
things, but that is completely irrelevant, as this report is about a wrong
message in mdadm.

06.12.2014 00:06, Marc Lehmann wrote:

Did it happen only once, or was it a persisten prob happening on each
boot?  I mean the unability to assemble the raid device.

You don't use systemd, do you?  (Not trying to open a can of worms)

Yes, I thought about that.  I don't see such error in the code, and I
sure remember mdadm reporting correct errors when the device it was
asked to use was busy or there was an I/O error in it.  The result,
however, is the same: it can't assemble the array this way, given an
explicit list of drives to assemble it from.  So if anything it is a
cosmetic error, that is, wrong error is being reported, the behavour
is the same.

Hence the 'moreinfo'.  I wasn't confused, instead, I just didn't know ;)

Now it looks like it is you who's confused.  What I mentioned above
about raid5 and having metadata at the end of the component devices
is true regardless whenever you're asking for help or not (the same
applies to other raid levels as well, except raid1 aka mirror, because
raid1 is the only level where one can use a component device directly
without assembling the array).  And for some reason you created such
an array, so I merely pointed out that it's a way for possible issues.
It's not in any way a "solution" to this bugreport.

But anyway, I still don't know what to do about this bug.  If it were
possible to reproduce it, but heck... ;)  I tried to re-create this
situation yesterday in qemu/kvm, but it all just works as expected.

One more wild guess.  Is it possible this has something to do with HPA
(host-protected area) in the drive(s), when the drive can be made larger
or smaller by issuing a special command to it?  That'd explain things,
because 1.0 metadata placement is tied to the end of the device, and if
device changes size this metadata is impossible to find.  But it is a
wild guess indeed, because in your case it only happened to one drive
not all of them, and this can't be just a bad luck that this is also
the first drive in the array and contains partition table.  I still
think it has something to do with the partition table and mdadm mis-
reporting something like "busy" this way.

Maybe by changing wording of this message to a more neutral, like,
"unable to FIND superblock" instead of "has no superblock", will fix
this :)

Thanks,

/mjt