Thousands of hosts on a gigabit LAN, maybe not

[johnl at iecc.com (John Levine)]

Some people I know (yes really) are building a system that will have
several thousand little computers in some racks.  Each of the
computers runs Linux and has a gigabit ethernet interface.  It occurs
to me that it is unlikely that I can buy an ethernet switch with
thousands of ports, and even if I could, would I want a Linux system
to have 10,000 entries or more in its ARP table.

Most of the traffic will be from one node to another, with
considerably less to the outside.  Physical distance shouldn't be a
problem since everything's in the same room, maybe the same rack.

What's the rule of thumb for number of hosts per switch, cascaded
switches vs. routers, and whatever else one needs to design a dense
network like this?  TIA

R's,
John