When a tenant network is created in any project, Neutron will allocate a unique VLAN number (which OpenStack refer it as segment ID) for that tenant network. Note that this VLAN number is ONLY used in the physical network but NOT inside the OVS of the compute node. This is the most confusing thing, as OpenStack beginners will always have the misconception that the segment ID is used internally in the compute nodes.

Now let’s see what happen in the compute node. When you create a new VM and attach it to your tenant network, Nova component of the OpenStack will find a compute node to launch your VM. Then Neutron will try to allocate an “internal” VLAN of the OVS inside the compute node and put your VM to that internal VLAN. Neutron will instruct the OVS to map the internal VLAN traffic to the physical VLAN when going out to the physical NIC of the compute node and vice versa. Obvious, if 2 VMs are on the same tenant network, they will be put in the same internal VLAN.

The term “internal” here has the meaning of local, it means that the VLAN value of the OVS maybe different in different compute node, even if they belong to the same tenant network. But they will be translated to the same VLAN of the provider network when traffic goes onto the physical network. In short, the provider VLAN bridges the internal VLANs of all compute nodes for the same tenant network.

Why is it so complicated? Why not directly use the VLAN number inside the OVS so that translation is not required. From my point of view, if you use VLAN for the provider network, it seems that it’s really unnecessary. But what about VXLAN and GRE? As the address space of VLAN is not large enough to have a one-to-one mapping to the VXLAN or GRE segment ID, so it make sense to use a local VLAN number within the compute node, as you will never put all tenant networks in one single node. The number of VLANs should be good enough within one single node. In fact, Cisco DFA uses similar trick in the leaf switch.

Once you understand the above implementation, there should be no magic for the case of using GRE and VXLAN as the provider network.

OK, let’s see what happen if we use VXLAN (or GRE) for the physical network. Actually the only different is that when you create a tenant network, instead of allocate a VLAN for that network segment, OpenStack will allocate a VXLAN ID (VNID) (or in GRE it will allocate a key) for that network, once again this ID is referred as segment ID.

Then when you create an instance (VM) on that tenant network, once again OpenStack will find a compute node to launch your VM. However, the VM will be put onto one of the internal VLAN of the OVS in that compute node. As internally, OVS uses VLAN (not VXLAN nor GRE). Nonetheless, OpenStack will instruct the OVS to remember the mapping between the internal VLAN and the segment ID. So when the Ethernet frame goes out of the compute node to the physical network, the OVS will encapsulate it onto the corresponding VXLAN using the mapped VNID (segment ID). Similarly, in the case of GRE, the frame is encapsulated in the corresponding GRE tunnel with the mapped key.

In simple word, it just making use of VXLAN (or GRE) to bridge the internal VLANs of the compute nodes, so that all those VLANs are now in the same layer 2 domain.

Continue with my previous blog entry which I have mentioned that OpenStack can make use of VLANs in physical network for tenant network segregation.

Yet, what is the limitation of VLAN? What is the maximum number of VLANs you can use? Yes, only 4K, that needs to be shared with all the tenants in your cloud. Also, all your compute nodes’ physical NIC need to be on the same layer 2 network.

Then what’s the solution? If you have followed my previous blogs, you will figure out that VXLAN is one of the promising solutions. As the VNID of VXLAN supports 24 bits addressing space i.e. 16 million LAN segments. Moreover, by using VXLAN, the compute nodes’ physical NIC need not to be on the same layer 2, they can be in different subnets of the physical network, so that they can be anywhere in your data center.

Besides using VXLAN, there is another option that Neutron provides, which is the traditional GRE tunnel. GRE is just like VXLAN, both are tunneling technology that making use of IP network to encapsulate the Ethernet frames. However, GRE is point-to-point in nature, while VXLAN can make use of IP multicast to carry multi-destination Ethernet frames. In GRE header, there is 32 bit key field that can be used to identify different tenant network number.

To summary, you have 3 choices:

1. Use VLAN,
2. Use GRE
3. Use VXLAN.

Let me discuss the detail one by one.

If you want to use VLAN, your compute nodes should be reside on the same layer 2 domain of your physical network, the physical NIC of your compute nodes need to connected to a trunk port of the uplink switch. And all those trunk ports need to be the same layer 2, i.e. cannot be routed. Just like the figure below:

In the traditional Cisco 3-tier data center design, layer 2 domains are resided within the same aggregation block. As the layer 2 boundary is between the aggregation and the core, unless you extend your layer 2 over the core, otherwise, your compute nodes cannot be attached to access switches in different aggregation blocks.

That’s the reason for Cisco Nexus to provide the Fabric Path technology so that you can extend the layer 2 anywhere in your data center. Similar solution is the Cisco DFA and ACI.

Talking back to the OpenStack, let me discuss the relationship between the tenant network and the VLAN of your physical network.

When a tenant network is created in any project, Neutron will allocate a unique VLAN number (which OpenStack refer it as segment ID) for that tenant network. Note that this VLAN number is ONLY used in the physical network but NOT inside the OVS of the compute node. This is the most confusing thing, as OpenStack beginners will always have the misconception that the segment ID is used internally in the compute nodes.

Let me discuss the relationship between the tenant network and the VLAN of your physical network in next blog entry. Please follow here to the part 3 of this blog.

Recently, I have a consultation request for the OpenStack and Cisco DFA integration which is a relativity new solution, and I find that although there are documentations and blogs discussing the network architecture about OpenStack, most of them are focusing on the internal network inside the compute node (hypervisor). It seems that there is not much information about the relationship of the physical network (OpenStack refer it as provider network) and the tenant network (network that defined in OpenStack). Here in this blog I try to bridge the gap, so that I wish it will be easier for networking guys to understand how to design the physical network for the OpenStack.

At first, just want to have a brief introduction about OpenStack (in case you don’t know what it is). OpenStack is not a single vendor product, instead it’s an open source software developed by the community. They refer it as a cloud operating system. From my point of view, if you are familiar with VMware, OpenStack is somewhat like the vCenter. You can use OpenStack to manage different compute nodes, create and launch VMs (OpenStack refer it as instance) in those compute nodes. Of course, can also create different LAN segments (virtual in some sense) for the VMs.

In cloud environment, one of the important features is to support multi-tenants. OpenStack allows the administrator to create different tenants (which is referred as projects). As a tenant administrator, you can create different tenant networks in your project, and attach instances on different tenant network. Basically a tenant network is a layer 2 domain (like the traditional VLAN, or right now Cisco refer it as bridge domain in terms of the Cisco ACI). So VMs on different tenant networks should be in different subnets and need a layer 3 device to route traffic between them.

Following figure illustrates how OpenStack Web UI (dashboard) looks like after creating 2 tenant networks of each 2 VMs on it:

What I am going to discuss is the relationship of the tenant network and the provider (physical) network. Let’s first have a look on the whole network picture. Your compute nodes are actually connected to your physical network. Yet the VMs are not directly connected to the physical network, instead they are logically connected to an internal switch OVS within the compute node. So logically it’s like the following diagram (in fact it’s more complicate then that, internally it’s not only one single bridge, but I just want to simplify the picture so that networking people can understand it much quickly).

Now the question is, how does each tenant network mapped to the physical network? Recall that each tenant network (no matter within the same tenant or different tenants) is a layer 2 domain. So the most natural way is to use VLAN in the physical network for the layer 2 segregation – it’s true that one of the methods is to configure the OpenStack network component (they call it Neutron) to use different VLAN for different tenant network traffic.

However, besides VLAN, OpenStack provides 2 more solutions, please follow here to the part 2 of this blog.