No you shouldnt have to turn avatars off to have things speed up, but on the same side, the avatars only load as fast as the site that they are on.

If that site is slow, it slows down the rest of the page.

Since we do not have control over anything that is not directly off this server, we cannot control how fast and/or slow those things load.

But we have plans to deal with even avatars.