In part 3, I’ll be setting up the multisite between site 1 and 2 (Site 2 has been pre-configured) using a route server (9K-9) seen in the diagram below.
Spine needs to have the vlans configured with VNIs to it knows which to send to internal peers (Route targets)
9K-1(config)# feature vn-segment-vlan-based 9K-1(config)# vlan 100 9K-1(config-vlan)# vn 100 9K-1(config-vlan)# vlan 200 9K-1(config-vlan)# vn 200
EVPN to generate RD/RTs for Importing/Exporting into BGP
9K-1(config-vlan)# evpn 9K-1(config-evpn)# vni 100 l2 9K-1(config-evpn-evi)# rd auto 9K-1(config-evpn-evi)# route-target both auto 9K-1(config-evpn-evi)# vni 200 l2 9K-1(config-evpn-evi)# rd auto 9K-1(config-evpn-evi)# route-target both auto
L3 VNI to create the VRF Routing Table for Tenant1
9K-1(config-evpn-evi)# vlan 1000 9K-1(config-vlan)# vn 1000 ! 9K-1(config-vlan)# vrf context TENANT1 9K-1(config-vrf)# vn 1000 9K-1(config-vrf)# rd auto 9K-1(config-vrf)# add ipv4 un 9K-1(config-vrf-af-ipv4)# route-target both auto evpn 9K-1(config-vrf-af-ipv4)# ! 9K-1(config-vrf-af-ipv4)# feature interface-vlan 9K-1(config)# int vl1000 9K-1(config-if)# vrf mem TENANT1 Warning: Deleted all L3 config on interface Vlan1000 9K-1(config-if)# ip for 9K-1(config-if)# no shut
Next, configuring the multisite site number, which is 1 in this case. This site number is shared with any other BGWs at the site. Only a single node in my case.
9K-1(config-if-nve)# evpn multisite border-gateway 1
This will not work without specifying interfaces to be used for fabric-tracking (internal facing) and dci-tracking (external facing). These should be used to track the interfaces and bring down the BGW if it loses internal/external connectivity. The 9000v i’m using doesn’t allow these commands on sub-interfaces that i’m using. I had to apply them on other interfaces i’m not using which doesn’t seem to matter as long as they are routed interfaces (don’t need IPs configured) and up/up.
9K-1(config-evpn-msite-bgw)# int eth1/4 9K-1(config-if)# no switchport 9K-1(config-if)# evpn multisite fabric-tracking 9K-1(config-if)# no shut 9K-1(config-if)# int eth1/5 9K-1(config-if)# no switchport 9K-1(config-if)# evpn multisite dci-tracking 9K-1(config-if)# no shut
This node will act as an NVE peer for inter-site traffic, and external connectivity
9K-1(config-if)# int nve1 9K-1(config-if-nve)# host-reachability protocol bgp 9K-1(config-if-nve)# source-interface lo1 9K-1(config-if-nve)# multisite border-gateway interface lo2 9K-1(config-if-nve)# mem vni 1000 associate-vrf 9K-1(config-if-nve)# mem vni 100 9K-1(config-if-nve-vni)# mcast 239.1.1.1 9K-1(config-if-nve-vni)# multisite ingress-replication 9K-1(config-if-nve-vni)# suppress-arp 9K-1(config-if-nve-vni)# mem vni 200 9K-1(config-if-nve-vni)# mcast 239.1.1.1 9K-1(config-if-nve-vni)# multisite ingress-replication 9K-1(config-if-nve-vni)# suppress-arp
Next, eBGP is neighbored with R1 (10.0.101.1) to advertise Loopbacks 0 – 2. Loopbacks were already configured within BGP in part 1.
9K-1(config-if-nve-vni)# router bgp 100 9K-1(config-router)# nei 10.0.101.1 9K-1(config-router-neighbor)# remote-as 65000 9K-1(config-router-neighbor)# ebgp-multihop 5 9K-1(config-router-neighbor)# add ipv4 un 9K-1(config-router-neighbor-af)# exit
The route server located is configured as 10.0.106.2
9K-1(config-router)# nei 10.0.106.2 9K-1(config-router-neighbor)# remote-as 1001 9K-1(config-router-neighbor)# update-so lo0 9K-1(config-router-neighbor)# ebgp-multihop 5 9K-1(config-router-neighbor)# peer-type fabric-external 9K-1(config-router-neighbor)# add l2 ev 9K-1(config-router-neighbor-af)# send-community both 9K-1(config-router-neighbor-af)# rewrite-evpn-rt-asn 9K-1(config-router-neighbor-af)# end
Verify all of this:
eBGP with R1 to send/receive loopbacks for VTEPs and BGWs
9K-1# sh bgp ip un sum .... Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.0.101.1 4 65000 51 40 29 0 0 00:36:50 7
NVE Interface is up
9K-1# sh nve int Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [not-notified] Local Router MAC: 0c1f.d9f8.3307 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.1.254.1, secondary: 0.0.0.0)
Now we jump to 9K-9 to setup the route-server which is pretty simple
9K-9(config)# feature bgp 9K-9(config)# feature nv overlay 9K-9(config)# nv overlay evpn
It needs to peer with R2 so it can learn the loopback IPs from site 1 and site 2
9K-9(config)# router bgp 1001
9K-9(config-router)# nei 10.0.106.1
9K-9(config-router-neighbor)# remote-as 65000
9K-9(config-router-neighbor)# add ipv4 un
! Verification
9K-9(config-router-neighbor-af)# sh bgp ip un sum
.......
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.106.1 4 65000 8 4 15 0 0 00:00:28 12
!
9K-9(config-router-neighbor-af)# do sh ip route 10.1.255.1
........
10.1.255.1/32, ubest/mbest: 1/0
*via 10.0.106.1, [20/0], 00:01:05, bgp-1001, external, tag 65000
Next, configure a route-map to ensure the next hops do not get changed. This ensures the NVE peerings are not trying to get setup between the sites and this router.
9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED permit 10 9K-9(config-route-map)# set ip next-hop unchanged
Now setup the BGP EVPN peerings to Site 1 and Site 2. First configure the L2 EVPN address family to retain all route-targets. The route server doesn’t have any route targets imported/exported, so without this it will drop all routes instead of send them to the other sites.
9K-9(config-router)# add l2 ev 9K-9(config-router-af)# retain route-target all ! 9K-9(config-router-af)# nei 10.1.255.1 9K-9(config-router-neighbor)# remote-as 100 9K-9(config-router-neighbor)# eb 5 9K-9(config-router-neighbor)# ebgp-multihop 5 9K-9(config-router-neighbor)# add l2 ev 9K-9(config-router-neighbor-af)# send-community both 9K-9(config-router-neighbor-af)# rewrite-evpn-rt-asn 9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED out ! 9K-9(config-router-neighbor-af)# nei 10.2.255.1 9K-9(config-router-neighbor)# remote-as 200 9K-9(config-router-neighbor)# eb 5 9K-9(config-router-neighbor)# ebgp-multihop 5 9K-9(config-router-neighbor)# add l2 ev 9K-9(config-router-neighbor-af)# send-community both 9K-9(config-router-neighbor-af)# rewrite-evpn-rt-asn 9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED out
Now to check and see if site 1 has peered, and learned the routes from site 2
K-1# sh bgp l2 ev sum
......
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.106.2 4 1001 149 61 277 0 0 00:02:55 8
10.1.255.2 4 100 1257 1221 277 0 0 18:54:38 2
10.1.255.3 4 100 1254 1213 277 0 0 18:54:07 2
Neighbor T AS PfxRcd Type-2 Type-3 Type-4 Type-5
10.0.106.2 E 1001 8 4 2 1 1
10.1.255.2 I 100 2 2 0 0 0
10.1.255.3 I 100 2 2 0 0 0
!
9K-1# sh ip route vrf TENANT1
.....
192.168.0.0/16, ubest/mbest: 1/0
*via 10.2.254.1%default, [20/0], 00:00:09, bgp-100, external, tag 1001 (evpn
) segid: 1000 tunnelid: 0xa02fe01 encap: VXLAN
192.168.100.12/32, ubest/mbest: 1/0
*via 10.1.254.2%default, [200/0], 00:44:35, bgp-100, internal, tag 100 (evpn
) segid: 1000 tunnelid: 0xa01fe02 encap: VXLAN
192.168.100.15/32, ubest/mbest: 1/0
*via 10.2.253.1%default, [20/0], 00:00:09, bgp-100, external, tag 1001 (evpn
) segid: 1000 tunnelid: 0xa02fd01 encap: VXLAN
192.168.200.13/32, ubest/mbest: 1/0
*via 10.1.254.3%default, [200/0], 00:44:35, bgp-100, internal, tag 100 (evpn
) segid: 1000 tunnelid: 0xa01fe03 encap: VXLAN
Now to test to see if PC5 (site 2 = 192.168.100.15) can ping the hosts at site 1
PC2 = 192.168.100.12
PC3 = 192.168.200.13
PC-5> ping 192.168.100.12 84 bytes from 192.168.100.12 icmp_seq=1 ttl=64 time=50.520 ms 84 bytes from 192.168.100.12 icmp_seq=2 ttl=64 time=176.138 ms ^C PC-5> ping 192.168.200.13 84 bytes from 192.168.200.13 icmp_seq=1 ttl=60 time=61.637 ms 84 bytes from 192.168.200.13 icmp_seq=2 ttl=60 time=68.457 ms
[…] – Setup simple L2 VXLAN within a single site Part 2 – Setup L3 VNI within a single site Part 3 – Setup multisite between 2 sites using a route server Part 4, we will be configuring […]
Have you done the Multisite without the route-server?
that’s easy, you just need a full mesh of ebgp between all border leafs
So, a route server should “provide AS-path, MED, and nexthop transparency” but we only configured nexthop transparency.
Med is not really used in EVPN, but what about AS-PATH ? your outputs do not show the BGP table, so i wonder what effects it has on best route calculation