Slow Header Download Speed

[ppan]

When downloading files I am getting > 100 Mbps. However, with header download, I am only at 3-4 Mbps!
Is that "normal"?

I am using Newshosting & Newsbin 6.71

p.s. Disabling Antivirus (ESET NOD32) doesn't help

[Quade]

I don't know if they support compressed header downloads or not. Might want to check the server advanced options and try enabling XFeatures if it's disabled.

I suspect this is server dependent. I typically update multiple groups at once and get > 100 to giga and AW. They both support header compression so the actual speed is faster. One change between older and newer Newsbin is that older Newsbin showed the decompressed speed, meaning it could show a speed up to 10 times faster than your line speed. It's not any slower now. The reporting is just different.

[ppan]

I don't know if they support compressed header downloads or not. Might want to check the server advanced options and try enabling XFeatures if it's disabled.

I suspect this is server dependent. I typically update multiple groups at once and get > 100 to giga and AW. They both support header compression so the actual speed is faster. One change between older and newer Newsbin is that older Newsbin showed the decompressed speed, meaning it could show a speed up to 10 times faster than your line speed. It's not any slower now. The reporting is just different.

There is no check box for enabling" XFeatures", only a checkbox for disabling it (and it is unchecked).
With the correct reporting option (showing real speed now), shouldn't the header speed download be the same as for binary file downloads?

p.s. I will post this question in the Newshosting Forum as well.

[ppan]

I just found out that it appears that each header download connection maxes out at 15-25 Mbps. Can that be correct?
Thus, if I set the number of connections >1 and download headers for 1 group, I may get total ~20 Mbps. If I download headers for 2 groups , I may get a total of ~40 Mbps.

Can someone please test with their provider?

[itimpi]

The speed you get for a header download connection is normally highly dependent on your provider (assuming you are not hitting your ISP overall bandwidth limit). Most providers have servers dedicated to handling headers and they can have very different performance characteristics to the servers that handle file downloads.

[ppan]

I just received a response from Newshosting. They confirmed that the download speed is only about 15-25 Mbps per connection.
They stated that in order to increase the header download speed, I need to use a Newsreader that allows multiple connections for header downloads.
However, it appears that Newsbin only allows one connection per group?

[ppan]

I did some more testing and Headers are only downloaded at 15-25 Mbps with XFeatures on but 53 Mbps with it off.
I asked them why the speed was slower with XFeatures and they suggested switching to another Newsreader...!?

Here is their response:
"Thank you for your patience. While our servers support xfeatures, having this feature enabled may cause slower speeds in many newsreaders. If this becomes an issue for you I would recommend switching to an alternative newsreader such as the newest release of Newsleecher (7.0 final)."

Is this correct? Why would the Header download speed be slower in Newsbin than in Newsleecher?

[itimpi]

I think that the compression ratio for headers is around 10 to 1, so the effective speed with Xfeatures enabled is MUCH faster once you multiply the Xfeatures enabled speed by 10.

[ppan]

I think that the compression ratio for headers is around 10 to 1, so the effective speed with Xfeatures enabled is MUCH faster once you multiply the Xfeatures enabled speed by 10.

Yes, but why is the speed slower when transmitting compressed vs non-compressed headers? According to the response from Newshosting, it is the fault of the Newsreader.
I would like to download compressed headers at the same speed (53 Mbps in my case) as uncompresssed headers.

Currently, I am not able to download compressed headers from large groups in a reasonable time (e.g. less than 12 hrs).

[itimpi]

The speed is controlled at the server end so you have no control over that. The slower transmission speed is probably due to the overheads of compression, but since 15Mbps when compressed is an effective speed of 150 Mbps, then from the speeds quoted it will take even longer if you run with xFeatures disabled.

[ppan]

The speed is controlled at the server end so you have no control over that. The slower transmission speed is probably due to the overheads of compression, but since 15Mbps when compressed is an effective speed of 150 Mbps, then from the speeds quoted it will take even longer if you run with xFeatures disabled.

I don't intend to download with XFeatures disabled. That was not the point. I just need to get a higher download speed with it enabled.

Here is a simple test: On a "good day", to download 1 M posts it takes about:
55 sec with XFeaures off (running at 53 Mbps)
19 sec with XFeatures on (running at ~24 Mbps)

However, some large groups have in excess over 18,000 M posts. Thus, it would take more than 95 hrs to download the headers for such a group (with XFeatures enabled)!

p.s. I just tried Newsleecher but it didn't help. Same speed as with Newsbin: 15-25 Mbps.

[Quade]

However, some large groups have in excess over 18,000 M posts. Thus, it would take more than 95 hrs to download the headers for such a group (with XFeatures enabled)!

Yeah it takes awhile. It'll probably consume 50-60 gigs of disk space for that one group when it finally finishes importing too.

18 billion * 300 = 5.7ish TB of downloaded data.

Most people don't realize how much they're downloading. XFeatures uses compression so, over the wire, you're probably talking about 500-600 Gigs of data from the server.

[itimpi]

Large groups like that are not really amenable to manipulating by downloading headers. That is one of the reasons why NZB indexing services that provide a search front-end to such groups have become popular.

[Quade]

Large groups like that are not really amenable to manipulating by downloading headers. That is one of the reasons why NZB indexing services that provide a search front-end to such groups have become popular.

That's my opinion too.