Port has two separate meanings in the world of networking. One is the electrical socket into which a data cable can be plugged. For most home routers, there are four Ethernet ports for connecting to devices such as PCs, TVs and Games Consoles. Although the ports are numbered on the sockets and on the lights that indicate which ports are being used, the actual ports all serve the same purpose, and it makes no difference what port each device is plugged in to.
The other meaning of port identifies which particular software within a device is to handle the connection. For example, port 80 is used for connections to a HTTP server program. When a connection is made between two computers, it involves an IP address, which identifies the specific computer or network device that the connection is going to and a port number, which identifies the particular software that the connection will be using on that target computer.
When you connect to a website such as Yahoo Answers, your computer connects to the IP address of the Yahoo Answer server, and sends the connection request to port 80 so that the correct software at YA is used. If the connection is to be secured by changing from HTTP to HTTPS, then the port at YA changes from 80 to 443.
In the messages that go to YA are the IP address to identify your device and port number of the software (your Internet browser). Operating systems perform a little trick. When the browser wants to open a connection to a website, it asks the operating system to supply a port number it can use for any returning information to be directed back to the correct software for the browser window. You can open multiple browser windows to the same website doing different things. Each response from the website will be directed to the port associated with the window that sent in the request.
If you set up a server process at home, such as a web site or a minecraft server, then you may need to configure a mechanism called port forwarding in your router. You might have multiple computers on your internal network. When an unsolicited connection arrives from a client for your server, then you need a port forward setting that tells the router which computer has the software for serving the connection.
When a connection is established in either direction (incoming or outgoing), the router remembers which ports for each remote IP address are associated with each of your computers.
Various port numbers between 0 and 1023 are reserved for special standard protocols including FTP, SSH, Telnet, SMTP, POP, IMAP, NTP, DNS, HTTP, HTTPS, etc. Ports over 1024 up to 65535 are used for certain games and for source ports for outgoing connections.
I am not aware of any settings associated with port numbers that can improve download performance.
I hope this helps.