Per-Destination vs. Per-Packet Load Balancing

You can set load-balancing to work per-destination or per-packet. Per-destination load balancing means the router distributes the packets based on the destination address. Given two paths to the same network, all packets for destination1 on that network go over the first path, all packets for destination2 on that network go over the second path, and so on. This preserves packet order, with potential unequal usage of the links. If one host receives the majority of the traffic all packets use one link, which leaves bandwidth on other links unused. A larger number of destination addresses leads to more equally used links. To achieve more equally used links use IOS software to build a route-cache entry for every destination address, instead of every destination network, as is the case when only a single path exists. Therefore traffic for different hosts on the same destination network can use different paths. The downside of this approach is that for core backbone routers carrying traffic for thousands of destination hosts, memory and processing requirements for maintaining the cache become very demanding.

Per-packet load-balancing means that the router sends one packet for destination1 over the first path, the second packet for (the same) destination1 over the second path, and so on. Per-packet load balancing guarantees equal load across all links. However, there is potential that the packets may arrive out of order at the destination because differential delay may exist within the network. In Cisco IOS software, except the release 11.1CC, per packet load balancing does disable the forwarding acceleration by a route cache, because the route cache information includes the outgoing interface. For per-packet load balancing, the forwarding process determines the outgoing interface for each packet by looking up the route table and picking the least used interface. This ensures equal utilization of the links, but is a processor intensive task and impacts the overall forwarding performance. This form of per-packet load balancing is not well suited for higher speed interfaces.

Per-destination or per-packet load-balancing depends on the type of switching scheme used for IP packets. By default, on most Cisco routers, fast switching is enabled under interfaces. This is a demand caching scheme that does per-destination load-balancing. To set per-packet load-balancing, enable process switching (or disable fast switching), use these commands:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# no ip route-cache
Router(config-if)# ^Z

Now the router CPU looks at every single packet and load balances on the number of routes in the routing table for the destination. This can crash a low-end router because the CPU must do all the processing. To re-enable fast switching, use these commands:

Router# config t
Router(config)# interface Ethernet 0
Router(config-if)# ip route-cache
Router(config-if)# ^Z

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094820.shtml