The server type, i.e. shared or dedicated has a huge impact. It simply isn't right to say that it's not a case of what the server is doing.
Imagine your server is trying to serve up 100 websites, that puts a far higher strain on the server than a dedicated server only dealing with your website. If the server is over subscribed, or serving a high-traffic site then the page generation times incease quite quickly.
The type of jobs you're asking your server to do also affect it. Ideally you want a server handling your database requests independent, but geographically close, to your webserver so that each server has a single job to do.
What can affect the speed of interaction:
- Client speed
- Server speed
- Server load
- Line speed (client side - available bandwidth versus theoretical max)
- Server bandwidth
- Number of database requests per page
- Encryption
- Etc... that list is by no means exhaustive.
I've touched on your second question above, but if you're separating out your servers, i.e. webserver, database server, payment gateway etc, having them geographically close and linked with fibre lines will make a huge difference because you effectively negate the communication time between them.
If they were taking a long time to communicate with each other you'd see that as a compounded delay at the client side. If you've got them in the same building then communication via CAT5 is almost instant for the size of packets you'd be talking about.
What they consist i'm assuming means server type and hardware. Just like your PC at home, how quick it is and how quickly it can execute tasks has a massive impact on how quickly you can work and what you can do. The same is true with servers. The more powerful the server the quicker it can serve your site. That's true at all levels.
The database management system you're using will also have an impact. Using something heavy-weight when you don't really need it will slow your DB server down. Choose something light, like MySQL and you'll see performance improvements at the server.