Hi Sebastian, I actually never understood this whole thing, and I also don't use linux bridge, but openvswitch instead for many years now, so these rules don't do anything at all for me. This also requires putting IP addresses in the guest config file, which I never do. So, I end up having these kinds of rules in the dom0 iptables: commit 67e0d840e74f7a40087a0a4436ce6ecdd3811044 Author: mjw@wray-m-3.hpl.hp.com <mjw@wray-m-3.hpl.hp.com> Date: Fri Jul 9 10:38:33 2004 +0000 bitkeeper revision 1.1053.1.1 (40ee75a9YghVZFFolzFjyJngpxAbKg) Move to new model of network and vif control using shell scripts. ... And then it was moved around several times. Just commenting the handle_iptable here and there will disable it. I think we should do that in our packaging. It's just annoying. Yes, that's true. There is no proper working thing out of the box. Perfect... Maybe for very simple configurations, where you also specify some mac address and ip address in the guest config file. It's not perfect. E.g. the domU might be a router... how are you going to deal with specifying what IP addresses it can send traffic for? (only doing layer 2 in the dom0). Anyway, implementing an actual working better alternative is something that has to happen upstream. Which still does mean we can help, but fixing this will not be a debian add-on in the packaging. I must admit I didn't spend a lot of time to understand the physdev rules, but it looks like you found a combination of rules that actually does what you think it has to do. If that's the case we could prepare a patch for upstream to fix it. And, remove the handle_iptable from vif-openvswitch, doh. Thanks, Hans
Hi Hans,
I actually solved this issue by:
1) I disabled all the original iptables creation rules in the xen scripts.
2) I configure all IP-addresses and vif-names in the domU config files
(I want control about the IP address I assign to a domU, if somebody
changed the IP address in the domU the communication is broken then)
3) The iptables rules for the my virtualized firewall are loaded directly by iptables.
4) All iptables rules for the domU are created by the xen scripts - I
created new iptables creation rules in the xen scripts.
The advantage of this approach is that with every new domU the right
iptables rules get created and I don't have to mess around with
iptables. The drawback is that in case I reload iptables the
antispoofing rules are gone - but without them all communication to the
domUs is blocked (just not to the firewall), so I know immediately I
need to insert these rules again.
I had to adjust the iptables rules as the one I originally suggested
only secure the first domU. With the second domU the rules get
sidelined. To achieve antispoofing there are drop and accept rules
needed per domU based on its IP address from the domU config. All drop
rules for all domUs have to be evaluated first. The accept rules need to
be evaluated after evaluation of the drop rules. It is a bit difficult
to achieve that automatically in iptables without inspecting the
existing rule base first. I achieved that by a trick - I created a new
chain called domU and inserted that into the FORWARD chain (done by
iptables rules config). In the new domU chain with the start of every
single domU the drop rules get inserted (iptables -I) in the beginning
of that chain, and the accept rules get appended (iptables -A). So I get
the right order and don't need to inspect the iptables rule set.
To get these rules evaluated by iptables a sysctl setting needs to be done:
sysctl net.bridge.bridge-nf-call-iptables=1
With these rules should somebody change the IP address of a running domU
the antispoofing rules would block any traffic. Only by changing the
iptables rules or the IP address in the domU config file a communication
would be possible. To do this dom0 access is needed. That is exactly
what I want as I expose some domUs to the Internet.
Patch for the xen script vim-common.sh:
-----------8<------------
--- vif-common.sh 2018-11-02 17:13:30.813238234 +0100
+++ vif-common.sh.new 2018-11-06 12:22:46.179073064 +0100
@@ -124,15 +124,21 @@
{
if [ "$command" == "online" -o "$command" == "add" ]
then
- local c="-I"
+ local ci="-I"
+ local ca="-A"
else
- local c="-D"
+ local ci="-D"
+ local ca="-D"
fi
- iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \
- "$@" -j ACCEPT 2>/dev/null &&
- iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \
- -j ACCEPT 2>/dev/null
+ iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \
+ -p all -d "$2" -j ACCEPT 2>/dev/null &&
+ iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \
+ -p all -s "$2" -j ACCEPT 2>/dev/null &&
+ iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \
+ -p all ! -d "$2" -j DROP 2>/dev/null &&
+ iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \
+ -p all ! -s "$2" -j DROP 2>/dev/null
if [ \( "$command" == "online" -o "$command" == "add" \) -a $? -ne 0 ]
then
@@ -169,11 +175,11 @@
frob_iptable -s "$addr"
done
- # Always allow the domain to talk to a DHCP server.
- frob_iptable -p udp --sport 68 --dport 67
else
# No IP addresses have been specified, so allow anything.
- frob_iptable
+ # needs revision, currently not covered by iptables rules in frob_iptable,
+ # therefore commented out
+ # frob_iptable
fi
release_lock "iptables"
-----------8<------------
And these are the iptable rules look like after some domUs got
started (as it is rather long I copy the relevant FORWARD section with
the new chain domU - domf is one domU, jump another and fw is the
firewall, the interfaces are named vif-domU-bridge):
Chain FORWARD (policy ACCEPT)
target prot opt source destination
domu all -- anywhere anywhere
ACCEPT all -- anywhere anywhere PHYSDEV
match --physdev-in vif-fw-OAM --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV
match --physdev-out vif-fw-OAM --physdev-is-bridged
...
REJECT all -- anywhere anywhere
reject-with icmp-port-unreachable
...
Chain domu (1 references)
target prot opt source destination
DROP all -- !192.168.249.3 0.0.0.0/0 PHYSDEV
match --physdev-in vif-jump-FE --physdev-is-bridged
DROP all -- 0.0.0.0/0 !192.168.249.3 PHYSDEV
match --physdev-out vif-jump-FE --physdev-is-bridged
DROP all -- !192.168.240.6 0.0.0.0/0 PHYSDEV
match --physdev-in vif-domf-INT --physdev-is-bridged
DROP all -- 0.0.0.0/0 !192.168.240.6 PHYSDEV
match --physdev-out vif-domf-INT --physdev-is-bridged
DROP all -- !192.168.250.6 0.0.0.0/0 PHYSDEV
match --physdev-in vif-domf-LAN --physdev-is-bridged
DROP all -- 0.0.0.0/0 !192.168.250.6 PHYSDEV
match --physdev-out vif-domf-LAN --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 192.168.250.6 PHYSDEV
match --physdev-out vif-domf-LAN --physdev-is-bridged
ACCEPT all -- 192.168.250.6 0.0.0.0/0 PHYSDEV
match --physdev-in vif-domf-LAN --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 192.168.240.6 PHYSDEV
match --physdev-out vif-domf-INT --physdev-is-bridged
ACCEPT all -- 192.168.240.6 0.0.0.0/0 PHYSDEV
match --physdev-in vif-domf-INT --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 192.168.249.3 PHYSDEV
match --physdev-out vif-jump-FE --physdev-is-bridged
ACCEPT all -- 192.168.249.3 0.0.0.0/0 PHYSDEV
match --physdev-in vif-jump-FE --physdev-is-bridged
Cheers,
Sebastian
The patch file has a little bug as I commented out the else branch in the last if section. Now it works again...--- vif-common.sh 2018-11-02 17:13:30.813238234 +0100 +++ vif-common.sh.new 2018-11-06 22:24:45.331845798 +0100 @@ -124,15 +124,30 @@ { if [ "$command" == "online" -o "$command" == "add" ] then - local c="-I" +### local c="-I" + local ci="-I" + local ca="-A" else - local c="-D" +### local c="-D" + local ci="-D" + local ca="-D" fi - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \ - "$@" -j ACCEPT 2>/dev/null && - iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \ - -j ACCEPT 2>/dev/null +### entfernt, da Antispoofing-Regel nicht funktioniert +### iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-in "$dev" \ +### "$@" -j ACCEPT 2>/dev/null && +### entfernt, da Antispoofing-Regel nicht funktioniert +### iptables "$c" FORWARD -m physdev --physdev-is-bridged --physdev-out "$dev" \ +### -j ACCEPT 2>/dev/null +### neue Antispoofing-Regeln + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all -d "$2" -j ACCEPT 2>/dev/null && + iptables "$ca" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all -s "$2" -j ACCEPT 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-out "$dev" \ + -p all ! -d "$2" -j DROP 2>/dev/null && + iptables "$ci" domu -m physdev --physdev-is-bridged --physdev-in "$dev" \ + -p all ! -s "$2" -j DROP 2>/dev/null if [ \( "$command" == "online" -o "$command" == "add" \) -a $? -ne 0 ] then @@ -170,10 +185,14 @@ done # Always allow the domain to talk to a DHCP server. - frob_iptable -p udp --sport 68 --dport 67 +### entfernt, da DomUs kein DHCP benötigen, IP@ fix +### frob_iptable -p udp --sport 68 --dport 67 else # No IP addresses have been specified, so allow anything. - frob_iptable + # needs revision, currently not covered by iptables rules in frob_iptable, + # therefore commented out + # frob_iptable + true fi release_lock "iptables"
Hi, I think we should move this discussion to (or extend to) upstream Xen development. If you want, please do so by starting a topic on the xen-users mailing list, summarize things a bit and refer to the url of the Debian bug. What I'm going to do now is patch out the handle_iptable line in the scripts, because we already know the current behaviour is incorrect and useless. This Debian bug will remain open, and there will most likely not be a proper working replacement before the Buster release. Anyway, thanks for already sharing your thoughts and examples. Hans
Ahem, I see that already happened, and that is how the story started. :) So, taking it to the next level... Getting this forward is more likely to happen with a proposed working alternative implementation as patch-set to the xen-devel list. There seems not to be much interest in this by default, and only complaining that things don't work often doesn't improve that situation. I still think proper anti-spoofing behaviour out of the box would be a really nice thing. But, as we've seen already, it's not trivial because the dom0 needs to have all kinds of information about what's happening inside the domU it starts. Hans
tags 894013 + upstream severity 894013 wishlist thanks I'm moving this one to the wishlist department of the bug list, and tagging upstream, because this needs to happen upstream. This doesn't mean I'm trying to sweep it under the carpet. This is one of the more interesting issues, but it is not in a state where we can just "do something" and fix it in the next package upload. Hans
Hi, Some water has passed under the bridge, and I feel the whole thing should be rewritten as nftables rules instead of iptables rules now. As a bonus, we could combine filters for mac spoofing and ip spoofing in one neat package.