Programs run under mpirun(1) freeze on first MPI operation when
	multiple addresses per interface are involved.

	The configuration was roughly as follows:

$ ip addr
…
2: eth0: …
    …
    inet6 2001:db8::2XX:XXXX:XXXX:XXXX/64 scope global dynamic
       …
    inet6 2001:db8::17a:170:1:4/64 scope global
       valid_lft forever preferred_lft forever
…
$

	(I. e., one address was configured in interfaces(5), while the
	other was thanks to the stateless IPv6 autoconfiguration.)

	The mpirun(1) was invoked like:

$ mpirun -nperboard 4 -H node1…,node2…,node3… hpcc… < /dev/null &

	which resulted in orted(1) being spawned on the nodes, with both
	of the IPv6 addresses in the tcp6:// URI's.  The payload
	processes were consuming 100% CPU each, but no progress was
	made.  (While we've tried to diagnose the problem, it was
	observed that the nodes have actually formed two disjoint sets,
	with the nodes of a single set being able to participate in a
	parallel computation spawned at any node, but all the attempts
	to spawn a task using the nodes from different sets have
	resulted in the behavior described above; apparently, the sets
	were formed with some dependence on the MAC address.)

	The behavior was 100%-reproducible.

	Switching the stateless configuration off on the nodes with
	sysctl(8) and removing the extra IP have fixed the problem.

# sysctl -w net.ipv6.conf.eth0.accept_ra=0
# ip addr del 2001:db8::2XX:XXXX:XXXX:XXXX/64 dev eth0
#

tag 584699 + help
thanks

Hi Ivan,

thanks for  the report! I also took this upstream, but unfortunately
neither upstream nor I can reproduce the bug since we do not have
multi-homed IPv6 hosts available for testing. If you could help us
testing and/or provide us with further information this would be really
great! I CC'ed Jeff (upstream) since I'm not exactly sure what he needs.

Jeff, you can subscribe to individual bugs by sending an empty message
to nnn-subscribe@bugs.debian.org, where "nnn" is a bug number[1]. I
think this might be the easiest way to not miss anything, as I expect
that a few messages might get exchanged. I'm not sure if Ivan is
subscribed to the bug. In case he isn't, you can always send a copy to
nnn-submitter@bugs.debian.org to reach the submitter of a bug[2].

Thanks to both of you for working on the issue!

Best regards,
Manuel

[1] http://www.debian.org/Bugs/Developer#subscribe
[2] http://www.debian.org/Bugs/Developer#followup

tag 584699 + help
thanks

Hi Ivan,

thanks for  the report! I also took this upstream, but unfortunately
neither upstream nor I can reproduce the bug since we do not have
multi-homed IPv6 hosts available for testing. If you could help us
testing and/or provide us with further information this would be really
great! I CC'ed Jeff (upstream) since I'm not exactly sure what he needs.

Jeff, you can subscribe to individual bugs by sending an empty message
to nnn-subscribe@bugs.debian.org, where "nnn" is a bug number[1]. I
think this might be the easiest way to not miss anything, as I expect
that a few messages might get exchanged. I'm not sure if Ivan is
subscribed to the bug. In case he isn't, you can always send a copy to
nnn-submitter@bugs.debian.org to reach the submitter of a bug[2].

Thanks to both of you for working on the issue!

Best regards,
Manuel

[1] http://www.debian.org/Bugs/Developer#subscribe
[2] http://www.debian.org/Bugs/Developer#followup

 > thanks for the report! I also took this upstream, but unfortunately
 > neither upstream nor I can reproduce the bug since we do not have
 > multi-homed IPv6 hosts available for testing. If you could help us
 > testing and/or provide us with further information this would be
 > really great! I CC'ed Jeff (upstream) since I'm not exactly sure what
 > he needs.

	When working with the package recently, I've stumbled upon what
	may be another manifestation of the same bug.  Now, I suspect
	that it's not IPv6 addresses by itself that trigger the problem,
	but rather the length of their respective string
	representations.  Actually, I guess that there's some static
	buffer involved, which is long enough to hold something like
	tcp://192.168.144.120:54321 (and tcp6://2002:bc87::1:54321), but
	truncates tcp6://2002:bc87:e7e5:2444:272:eff:fe0f:e02f:43210 at
	the 48'th character or so.

	Unfortunately, I was unable to find the relevant part of the
	code to check it myself.

	(Hopefully, I'd be able to gather some debugging output
	tomorrow.)

[…]

 >> thanks for the report! I also took this upstream, but unfortunately
 >> neither upstream nor I can reproduce the bug since we do not have
 >> multi-homed IPv6 hosts available for testing. If you could help us
 >> testing and/or provide us with further information this would be
 >> really great! I CC'ed Jeff (upstream) since I'm not exactly sure
 >> what he needs.

 > When working with the package recently, I've stumbled upon what may
 > be another manifestation of the same bug.  Now, I suspect that it's
 > not IPv6 addresses by itself that trigger the problem, but rather the
 > length of their respective string representations.

[…]

	Apparently, I was confused over the output like (as enabled by
	the ‘--mca btl_base_verbose 30 option’):

[n1-4.ncu.am-1.org:12674] btl: tcp: attempting to connect() to address 192.168.57.85 on port 516

	Note that port 516 (be it TCP or UDP) is privileged and
	obviously cannot be used by a program run unprivileged.  Now, I
	see that it's not truncation, but a byte order problem — it's
	the port 1026 (#x402) that's meant here, not 516 (#x204.)  Which
	is most probably a consequence of a missing (or wrongly placed)
	ntohs () or htons () call.  Which, however, I assume to be a
	completely different issue.

 > thanks for the report! I also took this upstream, but unfortunately
 > neither upstream nor I can reproduce the bug since we do not have
 > multi-homed IPv6 hosts available for testing.

	“Fortunately,” it appears that you don't need one, as the
	problem apparently arises on multi-IPv4-homed hosts as well.

	Trying to work-around the problem, I've tried both the

--- openmpi-1.4.2/debian/rules
+++ openmpi-1.4.2/debian/rules
@@ -57,6 +57,7 @@
 			--includedir=\$${prefix}/lib/openmpi/include	\
 			--with-devel-headers \
 			--enable-heterogeneous \
+			--disable-ipv6 \
 			$(TORQUE)

 # Thread support disabled because it's broken, see bug #435581

	To my surprise, it didn't help!

	Then, however, I observed that the system is IPv4-multihomed
	just as well:

$ ip -4
…
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    inet 192.168.57.XX/24 scope global eth0
    inet 192.168.57.ZZ/24 scope global eth0
…
$

	As soon as I have removed one of the addresses (with
	# ip addr del), the problem was gone.  (As long as IPv6 is
	turned off, — I cannot drop the extra IPv6 addresses on that
	host without running into issues.)

	To reproduce the problem, one can try, e. g. (assuming A.B.C.D
	is an unused address in the network, MASK is the netmask, and
	ethN is the network interface):

root# ip addr add A.B.C.D/MASK dev ethN
root#

$ mkdir -- test
$ cd test/
$ cp -- /usr/share/doc/hpcc/examples/_hpccinf.txt hpccinf.txt
$ rm -f -- hpccoutf.txt
$ mpirun.openmpi \
      --mca btl_base_verbose 30 \
      --mca oob_tcp_debug 1 \
      --mca oob_tcp_disable_family 6 \
      --mca btl_tcp_disable_family 6 \
      hpcc \
      < /dev/null

	While normally this would create ‘hpccoutf.txt’ almost
	immediately, the problem being discussed will make ‘hpcc’ stuck
	before it'll try to open (create) the file.

	Removing the extra IP addresses should eliminate the problem.

[…]