First, your network equipment can make the biggest impact on your internet and network performance. Faulty cables, ethernet cards, routers, and modems can cause huge latency issues without letting you know there's an issue from the way indicator lights appear. With wi-fi, the network channel and number of concurrently connected devices can make a huge impact on performance. Also, keep in mind that a wireless network defaults back to whatever the slowest device on the network is; if you have 3 wireless N computers, a wireless G iPod Touch, and a wireless B Nintendo DS/DSLite, everything is going to be running at the wireless B 11 mbps speed and not the faster 54 mbps G speed or 150/300/450 mbps N speed. One trick that can help wireless performance is if you have a dual-band N router and A/B/G/N capable cards in your PCs, use the 5 GHz band for N devices and the 2.4 GHz band for everything else.
Second, the kind of connection you have can determine your ping times. Cable and DSL generally should run between 25 and 60 ms pings, where 4G wireless will run about twice to three times of that and satellite internet will often range up to 2 seconds.
Third, your best ping times with your connection will come with a direct connection to the broadband modem you have, and not through a router. Routers can add between 5 and 40 ms to your ping time, dependent on a number of factors (processor speed, switch throughput, installed firmware). Some providers (usually DSL) will generally give you an integrated router/modem device; if there's no way to enable "bridge mode", you may want to swap your combo modem/router to a stand-alone modem. Wireless connections will also generally have a bit slower ping time than wired connections while going through a router.
Forth, what's going on with your network while you're playing games? If you have bittorrent or other downloads running, these can negatively affect your ping times. Multiple users on a single network can also be detrimental to your connection latency. Any time you're in a situation where you're using most of your bandwidth, you will experience high ping times. There's also the effects that trojans, viruses, and other forms of malware can have on your connection; if your ping times to Google.com are above 150-200 ms and nothing else is running on your computer, you very well may have a malware infection.