You've got several potential issues, many of which the others here have addressed.
1. Multiple users logging off can cause a slowdown, as prevously mentioned. But only a dozen users? Not likely.
2. Do you have DFS enabled? If someone is closing out a massive database at 5 and DFS has to replicate, that could hit your bandwidth pretty hard. But in a small business? Also not likely.
3. Intermittent hardware failure. A switch/router going bad can cause intermittent problems that are hard to trace. But regularly at 5 pm? Again, not likely.
4. Bad NIC on the server. We've had problems with the built-in Intel Gig Nics on some of our server motherboards. Though the NIC is "automatic", we've seen terrible performance connected to a 10/100 network. Changing the NIC from auto to 10/100 sometimes helped, sometimes not. Replacing the NIC with another did solve the problem. However, the slowdown was constant, not just at a specific time. NICs are cheap and easy to try as a fix, but this is also unlikely.
5. Denial of Service attack. As he goes home for the evening, some hacker is pushing a button to crash your network on a daily basis. Either from inside or outside your organization.
I assume you've used performance monitor to monitor your network traffic through the server. Are you getting a big hit around 5 PM on your server's network interface? If the hit on performance is at your server, then SOMETHING unrelated to your business function is banging it pretty hard. It could be a deliberate attack, a rogue program installed unintentionally, or something misconfigured. If the problem's at your server, then trace where the network traffic is coming from. Then you'll have your culprit. Single system-- rogue program/attack. Internet--attack/rogue program on server (check processes in task manager). Multiple systems-- misconfigured server system.
6. If the hit is not at your server, do you have the ability to monitor your network traffic in promiscuous mode, particularly your internet connection? If not, the culprit will be hard to trace. It's probably someone with a peer-to-peer file sharing program (like Limewire). I'd throw another NIC in the server and route all network traffic through it. Then you can monitor your network traffic without promiscuous mode capability. Trace where the network traffic is coming from/going to, and find your culprit.