Mercury was responsible for what was known as "Mad Hatter's Disease". It was used in the blocking of chapeaux by haberdashers and over time rose to toxic concentrations. It had direct effects on their brains, and in time they became permanently insane.
The problem is, and is further explained in the article, that mercury accumulates in the body, it isn't excreted ever. That's why the accumulations rise among organisms higher along the food chain. The first step, say algae, may contain nontoxic levels, but that's eaten by small freshwater marine life, which accumulate it during their life times. In turn they are eaten by larger creatures, and so on. So by the time a fish large enough to provide a meal for a human eats its fill over its life, it contains a much higher concentration of mercury. The more we eat, the more we accumulate, too.