In the last blog post I setup a basic VXLAN config just allowing 2 hosts on separate leaf switches to communicate within their common subnet. In that configuration all of the hosts were learned by the VTEPs from BUM traffic flooded using multicast. Using BGP EVPN, hosts are learned by their own leaf switch and then advertised to the BGP speaking spine switches which reflect them back down to all other leafs. As soon as a leaf switch learns of a host (most likely from that host sending a GARP or some ARP request) it will advertise that host’s information to everyone else. Even when using BGP EVPN arp requests are still sent to all VTEPs using multicast even though the VTEPs have probably already learned about the remote host via BGP. That is where arp suppression comes into play. Arp suppression allows the VTEP to respond to arp requests if the VTEP already knows the remote hosts mac address. There is a bit of upfront work to get this setup though because some switches require you carve up the TCAM to make room for ether-arp.
LEAF-1(config)# hardware access-list tcam reg arp-ether 256 double-wide ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM slices. Please re-configure. ! LEAF-1# sh hardware access-list tcam region ..... Find one that you can live without. In my case, I set RARP to 512 (wouldn't work at 1024 either, so i droped it to 512) LEAF-1(config)# hardware access-list tcam region racl 512 Warning: Please save config and reload the system for the configuration to take effect .... .... Reload .... LEAF-1# hardware access-list tcam reg arp-ether 256 double-wide .... .... Reload ....
Diagram I’ll be working with:
SPINE Configuration
# SPINES-1 and 2 feature vn-segment-vlan-based feature nv overlay feature bgp nv overlay evpn
Next we will configure the EVPN address family for BGP. I’ll be using iBGP and since we don’t have a full mesh, i’m going to configure the spines as route reflectors.
router bgp 100
neighbor 192.168.255.3
remote-as 100
update-source lo0
address-family l2vpn evpn
send-community extended
route-reflector-client
neighbor 192.168.255.4
remote-as 100
update-source lo0
address-family l2vpn evpn
send-community extended
route-reflector-client
That’s it for the spines. Now onto the leafs
LEAF Configuration
We already have most of the features enabled, so I’ll just be adding in the ones needed for BGP EVPN and enabling SVI routing
# LEAF-1 and 2 feature interface-vlan feature bgp nv overlay evpn
Now we need a new vlan/VNI and VRF to use for VXLAN Routing. It also requires an SVI for the vlan without an IP address
vlan 1000 vn-segment 1000 ! ## DO NOT forget the "evpn" portion or else nothing will work vrf context HQ vni 1000 rd auto add ipv4 unicast route-target both auto evpn ! int vlan 1000 vrf member HQ no shut ip forward
I’ll now add the VNI to the NVE and tell the NVE to advertise hosts via BGP.
int nve1 host-reachability protocol bgp member vni 1000 associate-vrf member vni 100 mcast-group 239.1.1.1 suppress-arp
BGP can now be configured to peer with the route reflectors for address-family l2vpn evpn
router bgp 100 neighbor 192.168.255.1 remote-as 100 update-source loopback0 address-family l2vpn evpn send-community extended neighbor 192.168.255.2 remote-as 100 update-source loopback0 address-family l2vpn evpn send-community extended
The VNIs currently in use (not the one associated with the VRF) can now be advertised into BGP. As you build more you simply add them to this evpn config
evpn vni 100 l2 rd auto route-target both auto
Last thing i’m going to do is enable the anycast gateway feature. All of the VTEPs will share the same IP and mac address for all L3 VXLAN Vlans. You should probably make sure you don’t configure a mac address that’s already in use somewhere, so finding some reserved or globally unused OUI space is recommended.
# Configure the anycast mac address which is a global config fabric forwarding anycast-gateway-mac 0007.fa00.0001 ! int vlan 100 vrf member HQ ip add 192.168.10.1/30 fabric forwarding mode anycast-gateway no shut
Let’s add another VNI, move HOST-2 into the vlan and verify routing works
vlan 200 vn-segment 200 int vlan 200 vrf member HQ ip add 192.168.20.1/24 fabric forwarding mode anycast-gateway no shut int nve1 member vni 200 mcast-group 239.1.1.1 suppress-arp evpn vni 200 l2 rd auto route-target both auto
LEAF-1# sh bgp l2vpn evpn summary BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 192.168.255.3, local AS number 100 BGP table version is 39, L2VPN EVPN config peers 2, capable peers 2 1 network entries and 1 paths using 216 bytes of memory BGP attribute entries [1/156], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.255.1 4 100 61 64 39 0 0 00:50:08 0 192.168.255.2 4 100 61 66 39 0 0 00:50:06 0 ! ! LEAF-2# sh bgp l2vpn evpn summary BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 192.168.255.4, local AS number 100 BGP table version is 47, L2VPN EVPN config peers 2, capable peers 2 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.255.1 4 100 64 63 47 0 0 00:49:36 0 192.168.255.2 4 100 64 61 47 0 0 00:49:41 0 You can already see that no advertisements have been received by either side.
I generated traffic on the hosts by pinging each other
LEAF-1# sh bgp l2vpn evpn sum
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 192.168.255.3, local AS number 100
BGP table version is 84, L2VPN EVPN config peers 2, capable peers 2
9 network entries and 11 paths using 1816 bytes of memory
BGP attribute entries [8/1248], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.255.1 4 100 51 62 84 0 0 00:27:45 2
192.168.255.2 4 100 49 60 84 0 0 00:27:50 2
!
!
LEAF-1# sh bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 84, local router ID is 192.168.255.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.254.3:32867 (L2VNI 100)
*>l[2]:[0]:[0]:[48]:[aabb.cc00.0300]:[0]:[0.0.0.0]/216
192.168.254.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[aabb.cc00.0400]:[0]:[0.0.0.0]/216
192.168.254.4 100 0 i
*>l[2]:[0]:[0]:[48]:[aabb.cc00.0300]:[32]:[192.168.10.10]/272
192.168.254.3 100 32768 i
*>i[2]:[0]:[0]:[48]:[aabb.cc00.0400]:[32]:[192.168.10.30]/272
192.168.254.4 100 0 i
Route Distinguisher: 192.168.254.3:32967 (L2VNI 200)
*>l[2]:[0]:[0]:[48]:[aabb.cc00.0200]:[0]:[0.0.0.0]/216
192.168.254.3 100 32768 i
*>l[2]:[0]:[0]:[48]:[aabb.cc00.0200]:[32]:[192.168.20.20]/272
192.168.254.3 100 32768 i
Route Distinguisher: 192.168.254.4:32867
* i[2]:[0]:[0]:[48]:[aabb.cc00.0400]:[0]:[0.0.0.0]/216
192.168.254.4 100 0 i
*>i 192.168.254.4 100 0 i
* i[2]:[0]:[0]:[48]:[aabb.cc00.0400]:[32]:[192.168.10.30]/272
192.168.254.4 100 0 i
*>i 192.168.254.4 100 0 i
Route Distinguisher: 192.168.255.3:4 (L3VNI 1000)
*>i[2]:[0]:[0]:[48]:[aabb.cc00.0400]:[32]:[192.168.10.30]/272
192.168.254.4 100 0 i
I sent a set of pings from HOST-3 to HOST-2 (different VTEPs and Subnets) and grabbed the packets off the wire to look. They are using the VXLAN routing VNI 1000 when being routed to another VTEP
Do you know how to configure BGP null route on Cisco EVPN/VxLAN fabric for DDoS protection?