Mid-size and large organizations are entering another cycle of Local Area Network upgrades due to two recent technology changes. One is that 10Gb Ethernet are now standard connections on servers. The other is the widespread adoption of wireless devices like the iPad that has put a strain on Wireless Local Area Networks. This is forcing the adoption of 10Gb Ethernet for server and access layer uplink connectivity. Because the LAN has to be upgraded, organizations are looking ahead a few years to see what other technologies are going to have to be accommodated, and working with experienced network designers to put together a comprehensive plan. There are many considerations to take into account in order to get an optimal design for the Core, Distribution, and Access layer upgrades, and specific Cisco switch choices are important in order to implement the design well.
Core Network Upgrades
The core network is the primary site where application servers are located. Most organizations now have a combination of dedicated application servers alongside servers configured for hosting virtual servers, usually running VMware ESXi. The older servers tend to have multiple Gigabit Ethernet connections, so the core network switches are similar to the Cisco 6500 or a stack of Cisco 3750 Gigabit switches, with 50-300 Gigabit Ethernet ports total.
New servers are shipping with 10 Gb Ethernet on the motherboard, and the price for 10 Gb Ethernet cards has dropped so much that 1 Gb for servers is becoming difficult to cost justify. Especially since 10 Gb connections in the datacenter can be made with copper twinax cabling, it is difficult to make the decision to purchase anything else.
This is combined with the trend to moving away from storage in the servers, usually called Direct Attached Storage, and moving all storage into a fast and reliable storage array accessed by iSCSI, Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), or ATA over Ethernet (AoE). Sometimes the storage array is accessed at the file level as Network Attached Storage (NAS) by either CIFS or NFS. It is much more cost-effective to access networked storage over 10G Ethernet connections than 1Gb Ethernet or even Fibre Channel because the price of a 10Gb Ethernet connection is much lower than an equivalent Fibre Channel connection. Because the storage array can be backed up and replicated to another site, it adds the additional benefit of improving business continuity capabilities.
If the organization is purchasing new storage arrays at the same time as the new servers, they can be specified for 10Gb iSCSI, 10Gb FCoE, or 10Gb AoE, but if the storage arrays are older Fibre Channel arrays, there has to be a provision made for connecting the new servers to the old storage. This is where the Cisco Nexus 5000 switches come into play. The Nexus 5000’s have the ability to connect to FC storage arrays, and merge the FC stream into a 10 Gb FCoE connection. This capability allows the organization to not have to purchase Fibre Channel Host Bus Adapters for every new server, and also limits the size of the Fibre Channel Storage Area Network that has to be maintained.
Not only can the Nexus 5000 provide Ethernet access to legacy storage for the new servers, it can connect the older 1 Gb servers into the system as well. This is done by connecting Nexus 2000 fabric extenders into the Nexus 5000 at 10 Gb speed, putting the Nexus 2000 boxes at the top of the rack of the older servers, and connecting the multitude of existing1 Gb Ethernet to the Nexus 2000 boxes. This architecture provides high speed server and storage access at the upgraded core of the network, as well as connectivity to new storage, legacy storage, and older servers.
All this server and storage connectivity should ideally be done at layer 2, without any layer 3 routing getting in the way and slowing things down. This is also the design recommendation for using Vmware with shared storage, because this allows for virtual server loads to be dynamically moved between physical servers while still accessing the same storage.
The core network upgrade can all be done independently of the access layer upgrade. If an organization has a large layer 3 switch like the Cisco 6500 or a stack of Cisco 3750’s at the core of their current network, the Nexus 5000 can be connected in with multiple 10 Gb Ethernet connections, providing the lowest cost upgrade while still retaining the core network upgrade performance benefits. For larger core networks, the Cisco Nexus 7000 can be used to provide a larger quantity of 10 Gb links to multiple Nexus 5000 switches.
Access Layer Switch Considerations
The access layer upgrade is sometimes usually driven by the need to have 1Gb desktop connectivity, but in many cases more by the desire of the organization to have 802.11N wireless access points. 802.11N full speed access requires 1Gb links, and multiple access points mean the uplinks from the switches have to then increase to 10Gb.
The access layer switches are used for workstation connectivity and are also used to power 802.11N wireless access points, 1 Gb IP phones, and Ethernet powered thin clients. In addition to high speed wireless, the access layer has to accommodate Voice, Video, and Virtual Desktop Infrastructure. For all these requirements, the access layer switches have to have more than just raw bandwidth. They also need to be:
- Secure – with voice, video, and desktop sessions on the LAN, the switches must have security features that can prevent them from getting attacked with MAC address floods, rogue DHCP servers, gratuitous ARP’s changing the default gateway, and other attacks that can be launched by malware. This security must be implemented at the switch level.
- Fast – As traffic goes through multiple switches, each hop can add latency. Instead of store and forward of the Ethernet frames, switches should use cut-through to move things along. At the choke points of the LAN, which are the uplinks, bonding multiple uplinks together can improve speed if done properly.
- Quality of Service – The switches should be able to reclassify traffic at the switch port level as it enters the LAN in order to prevent untrusted applications from claiming the highest traffic priority. Then throughout the LAN infrastructure, higher priority applications like voice, video, and virtual desktop sessions have to be given priority over other traffic like file transfers and print jobs.
- Reliable – Long Mean Time Between Failure, well tested code to limit bugs, good support from the manufacturer in case there is a software or hardware issue.
- Manageable – The switches have to be able to be managed remotely, have SNMP information, be able to log, and be configurable. GUI interfaces are OK, but there is nothing like a solid command line interface for rapid configuration, troubleshooting, and repair. Ideally the switch management should be integrated into a network management application.
- Power Density– Switches have to be able to support the power density of the planned devices. Most switches can not power all ports at the highest levels, so it is important to calculate the expected power load of the switches and specify the correct ones.
- Power and Cooling – Since many devices like access points, video cameras, and IP phones are powered from the switches, all access layer switches require properly sized Uninterruptable Power Systems. A basic switch consumes about 60 Watts. A 48 port switch with 15 Watt phones plugged into every port will require at least 600 Watts. Put a few of those switches in the closet an you are looking at not only upgrading to a much bigger UPS, but also better cooling.
- Redundancy Capable – The only place that there should be a single point of failure is at the access layer in the closets. If a switch fails, only the devices connected to that switch should lose connectivity – all others should work around the issue. In most cases that means dual uplinks from each closet to a redundant distribution layer at the core, and these uplinks should be able to link together into a port channel so that the full bandwidth of the uplinks can be used.
Distribution Switch Upgrades
The distribution layer is where all the access layer uplinks come together. Most of the organizations tend to have a large main campus with an extensive LAN. Many of them have a LAN that is set up in a fashion similar to the Cisco High Availability LAN designs of a few years back. This is 100 Mb Ethernet at the access layer and multiple1Gb fiber uplinks to the core/distribution switches. The larger networks have multiple distribution switches and separate core switches, but most mid-size organizations have the collapsed core/distribution model.
The Nexus 7000 enters the discussion when an organization considers upgrading the fiber uplinks on the access layer switches from 1 Gb to 10Gb Ethernet. As soon as an organization internalizes the need for 1Gb access layer switches and 10Gb uplinks, the place where these uplinks all come together has to be upgraded as well. The logical choice for this upgrade is the Nexus 7000.
The Cisco 6500 is an excellent switch that has versatility and speed. The problem is that most of the installed base has Sup720 supervisors. These supervisors have a maximum connection speed to any one line card of 40Gb, which means an entire 6509 can only have 32 10Gb Ethernet ports without oversubscription. It is more cost-effective to either replace the Cisco 6500 with a Cisco Nexus 7000, or change the 6500 to a 1Gb access-layer switch. Just about every 1Gb blade on the 6500 can be upgraded for Power over Ethernet, and with bigger power supplies, the 6500 makes an excellent access layer switch.
One of the catches in this 1Gb to 10Gb upgrade is the fiber issue from the access closets. The requirements for 10Gb fiber are different than for 1Gb fiber. 1 Gb fiber connections can be made for a considerable distance over multi-mode fiber with the use of long-haul SFP’s and mode conditioning cables. This does not work for 10Gb Ethernet over fiber! Longer multimode fiber runs have to be re-pulled with single mode fiber in order to support the 10Gb uplink upgrade.
Cisco Switch Models
Cisco switches have all of the above listed attributes, and sometimes more importantly, there are always good local Cisco resellers in every market that are able to create a custom design based on the organization’s needs, then assist with the installation, configuration, and ongoing support. Other brands of switches can be used, but having local experts available is an important consideration.
There are a few Cisco switches that we use in every design, because they have the right combination of price and capabilities. These are used in most LAN design situations, unless there are special requirements.
Cisco 2960-S. This is an excellent all around access layer switch. It can be set up as a standalone switch, or four of them can be stacked together with FlexStack when fitted with the optional stacking module. The uplink ports can be set up as either 1 Gb or 10Gb, and the stack uplink ports can be bonded together to create a reliable connection back to the distribution switch. The stack can be managed as one switch, and the stack connection speed is fast at 20 Gbps. This is a layer 2 switch.
Cisco 3750-X This is a great distribution switch or core/distribution switch. It is faster than the 2960-S and can route at high speed, making it an excellent layer 3 switch. The switch can have up to 9 in a StackWise stack, which can be managed as a single switch, and the stack connection speed is very fast at 64 Gbps. There is a model of this switch that has six 10Gb ports, so it can be used as a core/distribution switch for smaller environments.
Cisco 4500. This is a good access-layer switch. It can have a high density of 1Gb ports, and multiple 10Gb uplink ports. Even though it is a chassis switch, it should not usually used at the core layer of the network, since it can easily be oversubscribed, leading to potential server and storage performance issues.
Nexus 5000 This switch has 20-40 ports that can be used for 10Gb or 1Gb connections. It is used to connect servers, Nexus 2000 switch extenders, and to connect to Layer 3 switches. There is also a larger version of the Nexus 5000 that can be enabled for Layer 3 switching as well, the 5596.
Cisco Nexus 7000. This is the switch of choice for multiple 10G uplinks and very fast layer 3 routing. It works well with the Nexus 5000 and Nexus 2000 series switches, completing the Nexus core and distribution switch design.
Remote Site Switching
For remote sites that are away from the main campus, unless they are big, the LAN should be designed with the knowledge that 100 Mbs is usually faster than required. Most remote sites connect back to the network core over a Private Network such as a Metro Ethernet connection, and MPLS Wide Area Network, or the Internet through a Virtual Private Network. The speed requirements of remote site LAN’s can be much lower. This switches we usually use for this design are:
- Cisco 2960 with Gigabit uplinks
- Cisco 3750 with Gigabit ports for the uplinks
If 1Gb connections to the desktop are preferred, the Cisco 2960-S and Cisco 3750-X listed above are a good combination.
One of the most useful devices to increase reliability of the switching infrastructure is a redundant power supply. A good rule of thumb is that moving parts break first, so the most likely item to fail in the switch is the power supply or the cooling fans. Every single power supply stackable Cisco switch and most of the smaller routers have a DC port in the back. That is for backup power.
The Cisco RPS2300 can be used for redundant power. It has dual power supplies, and can connect to six different devices. If those devices ever lose their power supply, then the RPS box will provide power via the DC power port, and everything will continue to run.
Putting together a LAN upgrade design is a straightforward process. The difference between a good design and a poor one really come down to the details. No one wants to get a cheap network that will not handle the needs of the organization in the next few years and have to be replaced, and conversely most organizations would not want to pay for an oversized network that is too expensive.
It is best to get a design done from a reseller that regularly sells deploys the products they are recommending. Good Value Added Reseller’s will stay on top of the new products that are out, and will change their recommendations are based on the customer’s needs and budget. Many will do this at no cost as part of the sales process, and in many cases they are better than consultants or switch manufacturer’s, because the VAR’s are responsible for putting together designs that will work when they do the the deployment!