There are two passwords to be concerned about here. The first password is the encryption on your wireless access point. For example, your SSID is "linksys" and your password to connect the computer to the internet is "dudeawesome."
And then there is the username/password for the administration of the router. By default, most routers use admin/admin.
If a user has your access point's SSID and password, they can gain access to your wireless internet connection and, depending on your shared files, potentially have access to your personal documents. They could potentially do a lot worse depending on their experience/knowledge. Maybe access your emails if you have that personal information stored somewhere on your computer. Or steal your identity. It really depends on how much info you have saved on your computer. In fact I'm sure if someone wanted to, they could do a lot with the right knowledge.
And if anyone does anything illegal using your connection - the blame goes to you. Like downloading/sharing tons of music, movies, or games. Or teaching themselves how to make WMDs or something stupid.
If a user has your router's username and password, they probably already have the username/password for your access point. They could change the SSID and password of your access point. They could change the username/password of your router's administration. They could add/remove MAC filters to only allow certain computers to connect to your access point. They could change every little setting about your router - and possibly render it totally useless.
If this happened, however, then you could always hit the reset button for like 30 seconds and your default factory settings would be restored.
In most cases, you are probably fine. Especially if you live out in the country and have only a few neighbors. But it comes down to knowing your area you live in, and whether or not you trust any wandering access leechers. It's better to be safe than sorry. Take the 2 minutes to add some extra security to your network.