Adding Physical Servers to Cisco ACI Fabric

A while ago I wrote an article about adding VMware integration to your ACI fabric. This is of course a very important step, as about 70% of people’s workloads are virtual. However, that number has been pretty static over the last few years. We don’t see many environments that are 100% virtual because some applications just run better on physical servers. They’re not even all legacy applications. There are new applications such as big data using Hadoop that require physical servers. So, I thought I’d write an article on how we connect physical, or bare metal, servers to the ACI Fabric. It’s not too difficult but we can’t just plug it in and expect it to work.

There are a few things that need to be done first, which I’ve covered in various articles already. If you haven’t already, you’ll need to configure Switch Profiles and Interface Policies for your bare metal servers. When you’re configuring these profiles you’ll need to figure out whether to use CDP or LLDP. You’ll also need to create a separate AEP for it. After that you can associate the AEP with a physical domain.

To configure a Physical Domain:

1. Login to the ACI APIC
2. Click on Fabric at the top
3. Click on Access Policies in the Sub Header
4. Expand Physical and External Domains in the left navigation tree
5. Click on Physical Domains
6. Click Actions on the right
7. Click Create Physical Domain
8. Give it a name
9. Use the pulldown menu to choose the AEP that you selected
10. Use the pulldown menu and select Create a VLAN Pool. Create the VLAN Pool according to the VLANs you’d like the physical servers to be able to connect to.

Figure 1

Most likely you already have tenants, VRFs, and bridge domains but if not you can reference this article to create them and understand what they are. Also keep in mind that all of the previous steps listed in this article, and for that matter the ones I’m about to go through can be automated and orchestrated by issuing API calls to the APIC using something like the Postman REST client. For more information on that check out this article.

Now that we’ve gotten all the basic stuff out of the way let’s go ahead and get our physical servers connected. Click on Tenants at the top and select the Tenant you’d like to work with (this is very important as you don’t want your bare metal servers getting added to the wrong tenant). In this case we’re going to create a separate Application Profile and End Point Group to put these servers in, but if you already have an EPG you’d like to add them to that will work as well.

Create Application Profile:

1. Click on Application Profiles
2. Click Actions and select Create Application Profile
3. Give the Application Profile a Name
4. Click Submit. You could go on and create an EPG here easily as well, but I’m going to do it in two different phases for the sake of the article.

Now that we have an Application Profile, we can expand Application Profiles in the navigation tree and select our new Application Profile. From here we will create an EPG.

Create an EPG:

1. Click Actions while the Application Profile is selected and select Create Application EPG
2. Give it a name
3. Give it an optional description
4. Tags, QoS Class, and Custom QoS are also optional items
5. Select the correct bridge domain using the pulldown menu
6. Select an optional Monitoring Policy
7. Click the + sign next to Associated Domain Profiles and select the Physical Domain you created earlier
8. Most likely you’ll also want to change the Deployment Immediacy and Resolution Immediacy to Immediate.
9. Click Update.
10. Click Finish

Figure 2

You’ll notice I changed the Deployment and Resolution Immediacy to Immediate instead of On Demand. Setting the Deployment Immediacy to Immediate will send the policy to the leaf right now and setting the Resolution Immediacy to Immediate will have the leaf apply the policy to the hardware right now. If you set it to On Demand it will wait until a workload appears that requires that policy. Now we have an EPG and we could go in and create other EPGs and assign Contracts (or Policies) to them, but that will be addressed in a future article.

The only thing really left to do is to create a static binding or static path for our server. A static binding tells us which physical port on a leaf switch should be associated with a given server. According to the Information window for Static Bindings (Paths) it is a… “static association with a path. The existence of this object implies that the corresponding set of policies will be resolved into the node to which the relationship points.”

Create a Static Binding

1. Expand Application EPGs
2. Expand the EPG you just created
3. Click on Static Bindings (Paths)
4. Click Actions and select Deploy Static EPG on PC, VPC, or Interface
5. Select the Path Type. This comes back to how you created your Switch Profiles and Interface Policies. Hopefully you created a VPC for redundancy. If so, select Virtual Port Channel. If you only have one connection from the server to one leaf then just choose Port.
6. Select the Path using the pulldown menu. Your path should be in there if you created the Switch Profiles and Interface Policies.
7. Put in the VLAN you’ll be using (make sure it’s a VLAN that’s in the pool you specified earlier).
8. Select Immediate for Deployment Immediacy
9. Select Untagged for Mode.
10. Click Submit

Figure 3

Why did we select Untagged…let’s reference the Design Guide again:

Follow these guidelines to assure that devices that require untagged packets operate as expected when they are connected to access ports of an ACI leaf switch.

• When an access port is configured with a single EPG in native 802.1p mode, it’s packets exit that port untagged.
• When an access port is configured with multiple EPGs, one in native 802.1p mode, and some with VLAN tags, all packets exiting that access port are tagged in the following manner:
• For the EPG configured in 802.1p mode, packets are tagged as VLAN zero
• For the other EPGs, packets exit with their respective VLAN tags.
• When a leaf switch is configured for an EPG to be untagged, for every port this EPG uses, the packets will exit the switch untagged.

Note:
When an EPG is deployed as untagged, do not deploy that EPG as tagged on other ports of the same switch.

At this point you should have connectivity for your server. Try pinging the gateway of the bridge domain to make sure you have connectivity. Also keep in mind that if you don’t have contracts created yet, you won’t be able to ping any servers that are in other EPGs, but you should be able to ping servers in the same EPG. This is because Cisco ACI uses a white list model, which again we will go over in another article.

If you have questions or comments please leave a comment below this article or reach out to me on Twitter @Malhoit.