Cfrgtkky

Disclaimer: I'm not a network engineer; I just have to try to do a good-enough impression to solve this problem.

We have two rooms in our office. In the first, I would like to put a large copper 10-GbE switch to provide connectivity to several servers, something like this:

https://www.fs.com/products/69378.html

This switch has 48 10G-BASE-T ports and 4 40-GbE uplink ports.

In the second room, we have many devices that require 10-GbE SFP+ connectivity. It appears something like this could work:

https://www.fs.com/products/69226.html

This switch has 48 SFP+ ports and 6 40-GbE uplink ports.

Goal: To the greatest extent possible, I would like to provide 10-GbE line-rate performance between any pair of hosts in both rooms. As long as they are plugged into the same switch, this is straightforward, but I'm wondering how best to accomplish this across the two switches. I don't think I need any fancy features like QoS, VLANs, or anything like that; I just want one flat level 2 network. Can I simply connect 4 of the 40-GbE uplinks between the two switches and get ~160 Gbps of bandwidth between the two rooms?

My application consists of continuous, very high-rate UDP flows; that is, it's common for a particular flow to take between 2-8 Gbps of bandwidth on its own. That means that each host will only be communicating with 1-2 other hosts at any time, putting a limit on the total number of flows that are running simultaneously. I'm not sure whether this simplifies or complicates the issue: I need to ensure that the UDP datagrams are not reordered by any aggregation of the uplink ports.

Goal: To the greatest extent possible, I would like to provide 10-GbE line-rate performance between any pair of hosts in both rooms.

In order to truly guarantee 10G, you'll need 10G dedicated bandwidth between the rooms for each host. 10 hosts on each side would require a 10*10 = 100G link. Aggregated links might not be enough as the flows are balanced based on source/destination addresses and ports - two random flows can easily land on the same physical link and fight for bandwidth while another link is idle.

That said, LAG trunks most often work fine unless there's a very busy network or an extreme necessity to guarantee bandwidth at all times.

Can I simply connect 4 of the 40-GbE uplinks between the two switches and get ~160 Gbps of bandwidth between the two rooms?

No. Running multiple connections between switches causes bridge loops which in turn cause broadcast storms that'll bring down the network.

One solution is using a spanning tree protocol (RSTP/MSTP) but that's only useful for redundant links (all but one link is operationally deactivated).

What you need is link aggregation (LAG), preferrably LACP which is an IEEE, vendor-agnostic aggregation protocol. Put the desired interfaces in an LACP trunk group on both sides and then connect the ports. Note that you can only run LACP trunks between two switches - you can't split and recombine a trunk between three or more switches.

For more than two switches there are various proprietary solutions or Shortest Path Bridging (IEEE 802.1aq) which sadly hasn't caught on that much yet.

edit

I need to ensure that the UDP datagrams are not reordered by any aggregation of the uplink ports.

That is exactly why traffic is distributed based on SA/DA hashing: so that each datagram in a flow always uses the same physical path. If you also need to avoid overtaking across flows between the same two end nodes you need to make sure to use only the source/destination IP addresses and not port numbers as well.

/edit

Whether you also trunk VLANs across the LAG trunk doesn't matter. Using STP on all ports is always a good idea in case something goes wrong with the LAG trunk or someone puts up yet another link.