No way for me to know for sure from the given information, but I suspect it's the router.
I'm not familiar with that particular router, but most consumer-level routers will become bogged down after a you try to put more than (roughly) a few dozen typical users on them, depending on the model. If those users are making a lot of TCP/UDP connections or using a lot of bandwidth, then the capacity is even less. Even a single user running Bittorrent can bring some of the cheaper/older models to their knees.
You're also dealing with the fact that you've got a single access point being used by 50+ people. Wi-fi bandwidth is not infinite; everyone using the AP has to share the total available bandwidth. If you throw enough wireless devices on a single AP, the available bandwidth for each individual user can quickly diminish as a result. That's before you even take the problems in the previous paragraph into account.
Most home routers use the 192.168.1.xxx block of private IP addresses and will tell you that they can support 253 users (or a similar number). This is technically true, as there are 253 possible addresses to choose from (everything except 192.168.1.0, 192.168.1.255, and the router's IP), but I've never seen a home router that can actually handle that sort of situation in the real world unless the vast majority of the connected devices sit idle at any given time.
If possible, look into third party firmware for the router (DD-WRT and Tomato come to mind). If the router is not under warranty or the warranty isn't important, you may be able to squeeze a little more performance out of it. They also make it easier to set up multiple access points on the same local network, if you want to try buying a few more similar routers to spread out the load. Be aware that this will take some technical know-how to install and configure.
Edit: I use a different cable ISP and a residential connection, but if your connection is anything like mine, then your download speeds will usually decrease sharply when you start to approach the upload speed cap. This is probably because downloading over a TCP connection requires the downloader to send acknowledgement (ACK) packets back to the uploader as they download. If those packets are fighting for space with other uploads, it will slow your downloads as well. I don't think that's causing your current problem, but it's something to be aware of.
Edit2: Joel is right. The default "maximum number of users" (I'm assuming one "user" means one DHCP license here) is 50 on the WRT400N. You'll need to adjust that (and possibly the starting IP address from its default of 192.168.1.100, depending on how high you want to go) to fit more users.