I know exactly what you mean. In my house, we have 10 computers connected via ethernet and a half dozen or so wireless devices (Xbox 360, PS3, Wii, etc.). I have Comcast cable with a 16 Mbps connection. My cable modem goes into a Linksys WRT54GL router that I have running v1.28 of tomato (a 3rd party, Linux based firmware update for WRT based routers). That, in turn is connected to a Zonet 10/100 16-port switch. If I run a computer straight into the modem, then on average I get about 10 Mbps down and 1.5 Mbps up. If I run through my router and switch, I only get about 3 Mbps down and 750 Kbps up (and that's with no one else using the connection).
The problem is the little home networking routers just don't cut it for more than a couple of devices and more than 5 Mbps connections. At this point, you'll need to upgrade to an "enterprise grade" router. Personally, I like the UTMs by Fortinet. They give you the most bang for your buck. I just ordered a FortiGate 50B, and a Zonet 24-port Gigabit switch from Newegg. In your case, I would recommend the FortiGate 30B. It's a bit pricy at $350, but well worth it.
Most companies these days don't use the old firewalls anymore. Now we use UTMs, or Unified Threat Management device. These puppies can not only handle routing, setting up separate vlans, multiple VPN connections, and thousands of simultaneous sessions, but they also incorporate IDS, on the fly anti-malware, traffic shaping, QoS and more! Don't worry if you don't know what most of that stuff is. You can use a much or as little of it as you like. Anyway, that's what I believe your issue to be based on your description.
Good luck!