The term “overlay” is thrown around a lot now with the software defined networking evolution. A layer 3 overlay is often used now to avoid the problems that come along with layer 2 technologies such as STP and blocking. Layer 3 networking (or routing) doesn’t have to deal with this. So, when we implement things like VXLAN or NVGRE (layer 3 network overlays) we can scale better for larger environments because we are limited to only 4096 VLANs.
Since virtualization solutions often rely on layer 2 adjacency to provide things like VM mobility, using VXLAN let’s us scale this better. In the case of ACI, the network overlay lets us normalize any encapsulation. For example, we can accept an NVGRE packet into the fabric, it is then changed to VXLAN, and upon egress it can be changed back to NVGRE or actually any encapsulation type we want it to be.
There’s a lot more to networking overlays, but the thing to remember is just that it provides scalability and flexibility.