VXLAN EVPN Multisite Setup – Part 3

In part 3, I’ll be setting up the multisite between site 1 and 2 (Site 2 has been pre-configured) using a route server (9K-9) seen in the diagram below.

Spine needs to have the vlans configured with VNIs to it knows which to send to internal peers (Route targets)

9K-1(config)# feature vn-segment-vlan-based
9K-1(config)# vlan 100
9K-1(config-vlan)# vn 100
9K-1(config-vlan)# vlan 200
9K-1(config-vlan)# vn 200

EVPN to generate RD/RTs for Importing/Exporting into BGP

9K-1(config-vlan)# evpn
9K-1(config-evpn)# vni 100 l2
9K-1(config-evpn-evi)# rd auto
9K-1(config-evpn-evi)# route-target both auto
9K-1(config-evpn-evi)# vni 200 l2
9K-1(config-evpn-evi)# rd auto
9K-1(config-evpn-evi)# route-target both auto

L3 VNI to create the VRF Routing Table for Tenant1

9K-1(config-evpn-evi)# vlan 1000
9K-1(config-vlan)# vn 1000
!
9K-1(config-vlan)# vrf context TENANT1
9K-1(config-vrf)# vn 1000
9K-1(config-vrf)# rd auto
9K-1(config-vrf)# add ipv4 un
9K-1(config-vrf-af-ipv4)# route-target both auto evpn
9K-1(config-vrf-af-ipv4)#
!
9K-1(config-vrf-af-ipv4)# feature interface-vlan
9K-1(config)# int vl1000
9K-1(config-if)# vrf mem TENANT1
Warning: Deleted all L3 config on interface Vlan1000
9K-1(config-if)# ip for
9K-1(config-if)# no shut

Next, configuring the multisite site number, which is 1 in this case. This site number is shared with any other BGWs at the site. Only a single node in my case.

9K-1(config-if-nve)# evpn multisite border-gateway 1

This will not work without specifying interfaces to be used for fabric-tracking (internal facing) and dci-tracking (external facing). These should be used to track the interfaces and bring down the BGW if it loses internal/external connectivity. The 9000v i’m using doesn’t allow these commands on sub-interfaces that i’m using. I had to apply them on other interfaces i’m not using which doesn’t seem to matter as long as they are routed interfaces (don’t need IPs configured) and up/up.

9K-1(config-evpn-msite-bgw)# int eth1/4
9K-1(config-if)# no switchport
9K-1(config-if)# evpn multisite fabric-tracking
9K-1(config-if)# no shut
9K-1(config-if)# int eth1/5
9K-1(config-if)# no switchport
9K-1(config-if)# evpn multisite dci-tracking
9K-1(config-if)# no shut

This node will act as an NVE peer for inter-site traffic, and external connectivity

9K-1(config-if)# int nve1
9K-1(config-if-nve)# host-reachability protocol bgp
9K-1(config-if-nve)# source-interface lo1
9K-1(config-if-nve)# multisite border-gateway interface lo2
9K-1(config-if-nve)# mem vni 1000 associate-vrf
9K-1(config-if-nve)# mem vni 100
9K-1(config-if-nve-vni)# mcast 239.1.1.1
9K-1(config-if-nve-vni)# multisite ingress-replication
9K-1(config-if-nve-vni)# suppress-arp
9K-1(config-if-nve-vni)# mem vni 200
9K-1(config-if-nve-vni)# mcast 239.1.1.1
9K-1(config-if-nve-vni)# multisite ingress-replication
9K-1(config-if-nve-vni)# suppress-arp

Next, eBGP is neighbored with R1 (10.0.101.1) to advertise Loopbacks 0 – 2. Loopbacks were already configured within BGP in part 1.

9K-1(config-if-nve-vni)# router bgp 100
9K-1(config-router)# nei 10.0.101.1
9K-1(config-router-neighbor)# remote-as 65000
9K-1(config-router-neighbor)# ebgp-multihop 5
9K-1(config-router-neighbor)# add ipv4 un
9K-1(config-router-neighbor-af)# exit

The route server located is configured as 10.0.106.2

9K-1(config-router)# nei 10.0.106.2
9K-1(config-router-neighbor)# remote-as 1001
9K-1(config-router-neighbor)# update-so lo0
9K-1(config-router-neighbor)# ebgp-multihop 5
9K-1(config-router-neighbor)# peer-type fabric-external
9K-1(config-router-neighbor)# add l2 ev
9K-1(config-router-neighbor-af)# send-community both
9K-1(config-router-neighbor-af)# rewrite-evpn-rt-asn
9K-1(config-router-neighbor-af)# end

Verify all of this:
eBGP with R1 to send/receive loopbacks for VTEPs and BGWs

9K-1# sh bgp ip un sum
....
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.101.1      4 65000      51      40       29    0    0 00:36:50 7

NVE Interface is up

9K-1#  sh nve int
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [not-notified]
 Local Router MAC: 0c1f.d9f8.3307
 Host Learning Mode: Control-Plane
 Source-Interface: loopback1 (primary: 10.1.254.1, secondary: 0.0.0.0)

Now we jump to 9K-9 to setup the route-server which is pretty simple

9K-9(config)# feature bgp
9K-9(config)# feature nv overlay
9K-9(config)# nv overlay evpn

It needs to peer with R2 so it can learn the loopback IPs from site 1 and site 2

9K-9(config)# router bgp 1001
9K-9(config-router)# nei 10.0.106.1
9K-9(config-router-neighbor)# remote-as 65000
9K-9(config-router-neighbor)# add ipv4 un
! Verification
9K-9(config-router-neighbor-af)# sh bgp ip un sum
.......
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.106.1      4 65000       8       4       15    0    0 00:00:28 12
!
9K-9(config-router-neighbor-af)# do sh ip route 10.1.255.1
........
10.1.255.1/32, ubest/mbest: 1/0
    *via 10.0.106.1, [20/0], 00:01:05, bgp-1001, external, tag 65000

Next, configure a route-map to ensure the next hops do not get changed. This ensures the NVE peerings are not trying to get setup between the sites and this router.

9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED permit 10
9K-9(config-route-map)# set ip next-hop unchanged

Now setup the BGP EVPN peerings to Site 1 and Site 2. First configure the L2 EVPN address family to retain all route-targets. The route server doesn’t have any route targets imported/exported, so without this it will drop all routes instead of send them to the other sites.

9K-9(config-router)# add l2 ev
9K-9(config-router-af)# retain route-target all
!
9K-9(config-router-af)# nei 10.1.255.1
9K-9(config-router-neighbor)# remote-as 100
9K-9(config-router-neighbor)# eb 5
9K-9(config-router-neighbor)# ebgp-multihop 5
9K-9(config-router-neighbor)# add l2 ev
9K-9(config-router-neighbor-af)# send-community both
9K-9(config-router-neighbor-af)# rewrite-evpn-rt-asn
9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED out
!
9K-9(config-router-neighbor-af)# nei 10.2.255.1
9K-9(config-router-neighbor)# remote-as 200
9K-9(config-router-neighbor)# eb 5
9K-9(config-router-neighbor)# ebgp-multihop 5
9K-9(config-router-neighbor)# add l2 ev
9K-9(config-router-neighbor-af)# send-community both
9K-9(config-router-neighbor-af)# rewrite-evpn-rt-asn
9K-9(config-router-neighbor-af)# route-map NEXT-HOP-UNCHANGED out

Now to check and see if site 1 has peered, and learned the routes from site 2

K-1# sh bgp l2 ev sum
......
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.106.2      4  1001     149      61      277    0    0 00:02:55 8
10.1.255.2      4   100    1257    1221      277    0    0 18:54:38 2
10.1.255.3      4   100    1254    1213      277    0    0 18:54:07 2

Neighbor        T    AS PfxRcd     Type-2     Type-3     Type-4     Type-5
10.0.106.2      E  1001 8          4          2          1          1
10.1.255.2      I   100 2          2          0          0          0
10.1.255.3      I   100 2          2          0          0          0
!
9K-1# sh ip route vrf TENANT1
.....
192.168.0.0/16, ubest/mbest: 1/0
    *via 10.2.254.1%default, [20/0], 00:00:09, bgp-100, external, tag 1001 (evpn
) segid: 1000 tunnelid: 0xa02fe01 encap: VXLAN

192.168.100.12/32, ubest/mbest: 1/0
    *via 10.1.254.2%default, [200/0], 00:44:35, bgp-100, internal, tag 100 (evpn
) segid: 1000 tunnelid: 0xa01fe02 encap: VXLAN

192.168.100.15/32, ubest/mbest: 1/0
    *via 10.2.253.1%default, [20/0], 00:00:09, bgp-100, external, tag 1001 (evpn
) segid: 1000 tunnelid: 0xa02fd01 encap: VXLAN

192.168.200.13/32, ubest/mbest: 1/0
    *via 10.1.254.3%default, [200/0], 00:44:35, bgp-100, internal, tag 100 (evpn
) segid: 1000 tunnelid: 0xa01fe03 encap: VXLAN

Now to test to see if PC5 (site 2 = 192.168.100.15) can ping the hosts at site 1
PC2 = 192.168.100.12
PC3 = 192.168.200.13

PC-5> ping 192.168.100.12
84 bytes from 192.168.100.12 icmp_seq=1 ttl=64 time=50.520 ms
84 bytes from 192.168.100.12 icmp_seq=2 ttl=64 time=176.138 ms
^C
PC-5> ping 192.168.200.13
84 bytes from 192.168.200.13 icmp_seq=1 ttl=60 time=61.637 ms
84 bytes from 192.168.200.13 icmp_seq=2 ttl=60 time=68.457 ms

One thought on “VXLAN EVPN Multisite Setup – Part 3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.