Issue turned out to be with my ISP. Even though online speed tests were giving correct speeds, NNTP download speeds were still at about 1/2. I chatted online with my ISP and they confirmed I was paying for the 30Mbps package. Then as I am chatting with them, my connection drops and when it comes back, I am miraculously getting full NTP download speeds.
Anyway, thanks for the support!