Cisco ACI Multi-Site: Architecture & Design Guide

Cisco ACI Multi-Site: A Real-World Architecture and Design Analysis

Enterprise data center infrastructure is evolving rapidly. Traditionally, organizations hosted applications and infrastructure in centralized data centers. However, modern enterprises increasingly distribute workloads across multiple data centers, colocation facilities, remote locations, and public cloud environments.

As a result, managing network connectivity, security policies, and application availability across geographically separated environments has become more complex.

Cisco ACI Multi-Site addresses these challenges by connecting independent Cisco Application Centric Infrastructure (ACI) fabrics through a scalable data center interconnect architecture.

Furthermore, it enables organizations to coordinate network and security policies across supported on-premises and cloud environments while maintaining independent fabric control domains.

In this Cisco ACI Multi-Site architecture analysis, we examine a real-world reference topology and explore the key technical considerations involved in deployment.

Specifically, this guide covers:

• Cisco ACI Multi-Site architecture and topology
• MP-BGP EVPN control-plane communication
• VXLAN-based inter-site connectivity
• Inter-site network requirements
• Latency and infrastructure considerations
• Cisco Nexus Dashboard Orchestrator and centralized policy management
• Hybrid and multi-cloud connectivity considerations

By understanding these components, network architects and infrastructure engineers can make more informed decisions when designing resilient, scalable, and manageable multi-site data center networks.

From the above diagram, the controls plane function between the DC1 and DC2 is done forming the MP-BGP EVPN session between the SPINE nodes in each site. This allows the exchanging of MAC and IP address information for the endpoints that communicate across the sites. Furthermore, the control plane session can also provide encryption for the DCI link using cloudsec software function.

Additionally, the data plane function between the endpoint connected to DC1 and DC2 are achieved using the site to site VxLAN tunnels.

Key Requirements & Considerations

Latency Requirements: There are three different scenarios to consider when it comes to latency

No	Description	Max Latency
1	Control and data plane traffic latency between the sites	50 ms RTT
2	The latency between the MSO cluster nodes -Obviously, this comes only when you place the MSO nodes across different locations	150 ms RTT
3	Finally, the latency between the MSO  and the APIC clusters situated across the sites	1 sec

Software Requirements:  Please note, the MSO software hosting on VMware has the following resource requirements

Software	Resource requirements
Vmware ESXi 6.0 or later	Minimum of eight virtual CPUs (vCPUs), 24 Gbps of memory, and 100 GB of disk space

Form Factors	Description
Physical	Bare-metal servers installation with a.ISO from cisco.com
Virtual	For this there are two virtual appliance flavors are available 1) to be run on VMware ESXi hosts and the 2) on Linux KVM hypervisors.
Cloud Deployment	A specific .ami file is available; this allows you to deploy a cluster of three CASE VMs directly in a specific AWS region.

Topology Requirements

Inter-site spine connectivity: The connectivity between the sites can be done using back-to-back between the two sites or in the case of multiple sites that should be routed through ISN ( Intersite Network nodes). In fact, ISN can be any router or switch model with an L3 function with OSPF, sub-interface, and VLAN tagging support. To demonstrate, the below topology shows the connectivity requirement when routing through the ISN nodes.

MSO orchestrator to the APIC controllers connectivity: The Cisco ACI Multi-Site Orchestrator cluster should sit outside the Cisco ACI fabric – for instance, it could be connected to the APIC using OOB network

Cloud APIC Deployments: The cloud APIC is the software solution, only required to be part of your public cloud infrastructure if you need to extend the network & security policies. Therefore, Ideally, the required policies are globally defined on MSO and then distributed to the Cloud APIC. It accomplishes the task by translating the received policies from the MSO to the cloud-native network policies. For example, it can automate the provisioning of the needed cloud-native resources, such as VPCs, VNET, CSR 1000v deployments, and VGW, security groups, rules, etc.

Conclusion: Building a Resilient Cisco ACI Multi-Site Architecture

As enterprise applications become increasingly distributed, organizations need reliable ways to connect data centers while maintaining network visibility, consistent security policies, and operational flexibility.

Cisco ACI Multi-Site provides an architecture for connecting independent ACI fabrics across geographically separated locations.

However, successful deployment requires careful consideration of network connectivity, latency, routing, software compatibility, and application requirements.

This Cisco ACI Multi-Site analysis highlights five important design priorities:

• Architecture planning: First, evaluate whether Multi-Site or Multi-Pod better matches your infrastructure requirements.
• Inter-site connectivity: Additionally, ensure that the routed inter-site network supports the required bandwidth, MTU, and redundancy.
• Control-plane design: Furthermore, understand how MP-BGP EVPN supports endpoint reachability between participating sites.
• Policy management: Equally important, evaluate Cisco Nexus Dashboard Orchestrator for centralized policy coordination.
• Resilience: Finally, validate failure scenarios, operational dependencies, and application recovery requirements.

Although Cisco ACI Multi-Site can simplify policy coordination across distributed data centers, it does not eliminate the complexity of designing a resilient network.

Therefore, organizations should carefully evaluate their existing infrastructure, application dependencies, and long-term operational requirements before implementation.

Ultimately, a well-designed Cisco ACI Multi-Site architecture can help enterprises build a more consistent, scalable, and manageable multi-data-center networking environment.

By combining appropriate architecture, validated connectivity, and effective operational practices, businesses can better support their evolving infrastructure requirements.

Planning a Cisco ACI Multi-Site Deployment?

Connecting multiple data centers requires more than network connectivity. It demands careful architecture planning, policy design, resilience testing, and operational readiness.

TREZBON TECHNOLOGIES supports organizations with enterprise network architecture consulting, infrastructure design, and technology evaluation.

Whether you are assessing Cisco ACI Multi-Site, modernizing your data center, or reviewing your network architecture, our consultants can help you evaluate your requirements and identify practical solutions.

Ready to discuss your data center networking requirements?

👉 Connect with TREZBON TECHNOLOGIES

Build your infrastructure strategy with a trusted technology consulting partner.