OTV has been around for a while and whilst in general, I think that stretching broadcast domains (layer 2 VLANs) is not a great idea. However, in most enterprises I have worked in it is typically done for either long distance vMotion or server clustering, the later, specifically for Databases.
There are many great posts explaining why layer2 stretch VLANs (bridging) are not a good idea so I wont go into them here, but will just leave it with this.
Anyway if you do need to span a bridging domain between data centres, over a data centre interconnect (DCI) then one of the perhaps, more robust options, is Cisco’s Overlay Transport Protocol (OTV). It should be noted that whilst I’m discussing Cisco’s proprietary OTV implementation there is an RFC draft for OTV which can be found here.
NOTE: the Cisco and standard draft are likely to be non interoperable as they use different encapsulation methods.
Thus this post is specifically focused on the Cisco implementation of OTV which is actually Ethernet over MPLS over GRE over IP (EoMPLSoGREoIP). For OTV to work, assuming you have devices that support it (Nexus 7K & Cisco ASR), all that is needed from a transport perspective to run OTV is IP connectivity.
As OTV is an EoMPLSoGREoIP encapsulation it adds 42 bytes of header to the original packet which comprises of:
- 4 bytes – MPLS
- 4 bytes – GRE
- 20 bytes – IP Header
- 14 bytes – Outer Ethernet frame
One of the limitations of OTV currently, is that it cannot run in the same device or VDC as the layer 3 gateway or SVI. Thus when deploying OTV it must be in one of two topologies, being either ‘on a stick’ or ‘inline’. The on a stick method is typically preferred as it means that non OTV traffic does not need to traverse the OTV VDC and thus is more flexible, however the configuration is largely the same in either scenario.
- Red Line – represents the OTV traffic path.
- Purple Line – represents other (non OTV) traffic path.
Additionally OTV can be deployed in multicast mode or unicast mode, where the former requires that the DCI also supports and runs multicast. The benefit of multicast mode over unicast is when more than 2 DC’s are connected and thus the traffic can be more optimally sent between the source and destination rather than it being replicated to all OTV participants regardless if they need to see the traffic or not. However in most scenarios OTV is utilised to connect 2 sites, and thus unicast is generally fine, and arguably more simplistic to configure, as it does not require multicast in the DCI.
NOTE: The rest of this post will assume a 2 site DC connected via unicast OTV.
Whilst the diagram above only has a single device for the Layer3 and OTV functions these would each typically be on a pair of devices for redundancy purposes. Thus when OTV is formed it needs to decide which of the OTV pair is the Active Edge Device (AED). This is done via the AED election process, which in turn is negotiated over a ‘site VLAN’.
As mentioned, the site VLAN is an internal VLAN connecting the two local AED routers (not spanned over OTV) and utilised for the AED election. This allows each edge device to be active for a VLAN and redundant for another. When this process takes place, what would normally end up happening is that all even VLANs would use one AED, say edge device 1 as its active OTV device and all odd VLANs would use the other AED, say edge device 2 as its active OTV device, with the other AED being redundant (secondary). Each site is identified via a unique ID, which is shared locally between the AED’s at the same site and must be unique to a site.
The main components of OTV are an Overlay Interface, Join Interface and the Internal Interface. The Overlay interface is the logical OTV tunnel interface which performs the encapsulation, whilst the Join interface is the L3 physical link or portchannel which is used to route upstream towards the DCI. The Internal interface is the L2 interface, typically configured as a trunk or access port, which takes part in STP and learns MAC addresses per normal. This is typicall the interface that connects the L3 gateway device to the OTV device.
OTV uses ISIS as the routing protocol to form adjacencies between AEDs and to advertise MAC reachability information. OTV utilises ISIS in the background and thus no ISIS configuration is required. Additionally the OTV ISIS will not interfere with any other ISIS process which may be running on or between these devices.
OTV does have some limitations
- Cannot run OTV in the same VDC as Layer 3 SVI’s.
- OTV does not support Fragmentation, and sets the DF (Don’t Fragment) bit on all packets.
- OTV adds 42 bytes of header (So DCI needs to cater for this additional header).
- Limited number of extended VLANs per system across all configured overlays, which is around 1500 at time of writing this post.
One of the key benefits of OTV over other layer 2 stretch technologies, such as VPLS, is that it keeps the STP domain isolated. That is STP BPDUs are not transported over the OTV and thus a STP issue in one site should not affect the other.
The intent of this post is to provide a high level overview of OTV and its components. The next post will provide more detail into the configuration and troubleshooting of OTV.