It's probably related to contention ratio. Many ISPs only have a certain bandwidth of connection to the Internet backbone. This bandwidth might be sufficient to support perhaps 10% of their customers if those customers are all using their own maximum connection speeds. In the evenings, people come home from work and start streaming movies and TV catchup shows, and the total bandwidth of all the ISP's customers reaches the limit of the ISP's connection to the Internet backbone. At this point, the speed that can be supplied to each customer has to drop.
As an analogy, if you turn on the cold water to a single basin in your home, you will get a certain flow rate. Now if you turn on the cold water to every other basin, bath and flush the toilets at the same time, the flow rate to that first basin will probably slow down. The limit is the connection to the water backbone, the supply pipe connecting the home to the main water pipes in the street. Each water outlet is like an Internet customer, the pipe connecting the home to the street is like your ISP, and the main water pipe in the street is like the Internet backbone.
For many people, the drop in speed is not too serious as they rarely use the full bandwidth of their Internet connection. For example, my maximum download speed is around 40 Mbps, but apart from downloading large files from fast servers or performing Internet speed tests, probably I rarely reach 20% of the maximum download speed.
If the drop in speed is causing you problems, talk to your ISP. If it is simply that speed tests show a slow down, then you should not worry too much. Many people have similar symptoms but never realise it as they do not perform speed tests.
I hope this helps.