The power consumption of the ToR switches is a function of the features and capabilities designed into the switch. This comparison uses the value of 10 W per port for the 40 downlinks, plus the PHY power. The power consumed by the uplinks was excluded from the comparison. Our two ToR switches will consume 40 x 12 W = 480 W each when deployed with 10GBASE-T, or 40 x 10.5 W = 420 W with SFP+ direct attach copper.

In an overall power budget of about 13 KW, the difference between the two interconnect choices is less than 2% of the total. This scenario is somewhat simplistic and assumes that other energy efficiency features available in today’s PHYs are not utilized. Next, we look at the real life usage environment and how innovative 10GBASE-T features reduce the overall power consumption to be not only lower than SFP+ but also gigabit Ethernet for a given amount of bandwidth.

Enter utilization, virtualization and dynamic consolidation
The server power shown above reflects the power consumed when the system is at full load. But servers typically operate at full load for only short periods, otherwise operating at either light loads, or at idle.

Power does not drop to zero for servers at active idle. Processors remain clocked, memories are active and disks are spinning. The amount of power consumed at active idle will be implementation specific. To get an idea of actual power consumption, data was taken from www.spec.org which shows power at active idle at about half of full load power. Our analysis draws from this data, and will use a 50% power at active idle. Our servers are assumed to draw 300 W at full load, and 150 W when at idle. Dynamic consolidation and Wake on LAN

Virtualization offers the ability to consolidate virtual machines in order to minimize the number of physical servers required. Initially the concept of virtualization was applied in a static manner, by combining otherwise independent applications on the same server. Applying virtualization in this manner saved both capital costs and power.

The application of virtualization can also be used to consolidate active virtual machines onto a minimum number of physical servers through a concept known as dynamic consolidation. These now-idled machines can then be placed in sleep mode.

A technique used to permit a network application to control networked devices going into, and coming out of sleep mode is “Wake on LAN”. WoL was originally developed in the context of corporate IT, where client machines on desktops only needed to be active when they were attended by living, breathing humans but could be turned off at night and on weekends. However IT needed access to the machines off-hours to perform maintenance and install updates. WoL would permit a remote IT utility to awake a PC in sleep state to permit IT to perform maintenance.

WoL is ideal for use with dynamic consolidation in data center applications. WoL is used to wake networked elements based on the receipt of “magic packets”. There are a number of implementation details which ease implementation of WoL for 10GBASE-T and potentially can speed the implementation of dynamic consolidation in the market.

A key for successful deployment of WoL is to minimize the power consumed by the network controller while in sleep state. Some PHY manufacturers have designed into their products the ability to enter and exit low power modes which facilitate WoL on a systems level. The low power for WoL leverages the auto negotiation capabilities integral in copper PHY technology by permitting the link to perform its WoL monitoring function at the lower-power 100-megabit data rate. The sequence would be as follows:

The power reduction for the server delivered by entering sleep state is much higher than the power differences between 10GBASE-T and SFP+ PHYs. If we assume that the total power consumed by the server in sleep state is 3W (standby power will include supply inefficiency and ancillary standby power consumption), this is a savings of 147 W per server over the active idle power of 150 W which is otherwise consumed.

The systems benefit of permitting servers to enter sleep state begins to accumulate quickly. Given that 10GBASE-T started with a 240-W power disadvantage over SFP+, 10GBASE-T will demonstrate lower overall systems power once the second server enters a sleep state.

These differences become more apparent as the number of servers entering sleep state increase. Of course, the role of dynamic consolidation and the number of servers in sleep state will be highly application dependent, but the benefits are large. As dynamic consolidation is deployed statistics will become available on what the weighted average is of active vs. idled servers.

If the average utilization of server capacity over the typical work week drops below 95%, 10GBASE-T enjoys a power advantage that will only increase as the average power utilization decreases from the 95% threshold.

EEE creates the ability for the link to substitute a true idle when there is nothing to transmit. The system can present Low Power Idle (LPI) commands to the PHY, permitting the PHY to enjoy lower power for the duration of that LPI command.

The power savings will depend on the traffic density and on the device implementation. An example taken from a server with a gigabit uplink in use at the LLNL is shown in Figure 5. Peak bandwidth approaches the full gigabit capacity. However overall utilization is less than 1%, and there are gaps that exist between these peaks where large files are being exchanged.