Slow header d/l after v5.59 updgrade to v6

[PSPDWB]

I upgraded from 5.59 to 6 and then to 6.01B1. Since upgrading to 6, it takes many hours to update my groups using the "download latest" option, even if I "download latest" almost every day. I did the "download all headers" to completion after upgrading, so it's not like it's trying to do that, but it takes about as long as if it was. I do subscribe to some large groups, like a.b.dvd, but with version 5.59, it took maybe an hour max to complete "download latest" whereas now it takes several hours. My dl speeds fluctuate, but can get up to 120Mpbs, so that doesn't seem to be the problem, and nothing much else has changed with my pc, ISP, or news server, other that upgrading to v6.

Any ideas what might be wrong?

[itimpi]

You should expect v6 to take longer to download and save headers.

With v5 when headers were downloaded they were basically just written to disk.

With v6 after actually downloading the headers there is then considerable extra work done to process these headers into the .db3 file that is used for loading into post lists. It is this additional processing that allows v6 to load groups into post lists much after, and run with a much smaller memory footprint.

Having said that I do not know if the times you are experiencing are typical. If you tend to leave Newsbin running you could set NB to automatically download headers at more frequent intervals to spread the load. If doing this there are techniques you can use to limit the number of connections allowed to download headers thus minimizing the impact on file downloads running in parallel.

[Quade]

In the options, you can turn on "Background Update". This will speed up header downloads but, it'll take longer for the new records to become visible in the post lists.

[rh]

Turning on that option made my v6 header downloads work just like v5.59. Much better. Not sure why this is slower in v6 than v5.59 without the option.

As far as taking longer to appear in the post lists, I recall a thread that mentioned that. I think there was a desire for some sort of progress indicator. Is there one? How would one know when the background update is complete?

[DThor]

No matter what, v6 takes longer than v5 to process spools simply because 5 was more of a raw dump to disk. The upside with v6 is that with that processing you get a much more condensed spool that lets you load significantly more spool a lot faster with a lot less memory use. It's assumed if you want to bother storing spools locally in this world of nzb and Internet search like the kind newsbin offers, searching and loading spools fast is the repeated process you want optimized. The foreground / background thing is really just there to give an option for those that use newsbin differently. I like it in the foreground because that is effectively your feedback to progress, background means 'just get to it when you can, I'm doing something else'.
I agree it wouldn't hurt to have some sort of indication of how it's doing, though.

DT

[rh]

Thanks for the explanation! That certainly explains why search is so much faster. I just finished a typical search session and the difference from 5.59 is enormous.

Yeah, I'm old school for now, everything locally. Definitely need some sort of way (in my use case) to know when it's safe (ready) to load a recently updated spool.

Maybe my machine is quick enough that there's not much lag. I download headers and go do something else. When I come back, it's done and there's no out of the ordinary disk i/o present on any drives.

[Quade]

On my machine, background mode's pretty quick too for normal update. If you do something like "Download all Headers" on a significant number of groups AND you have a fast connection, it's pretty easy to get pretty far behind. With compression, my header downloads hit 400 Mbps. Trying to insert that in real time is pretty unrealistic.

I've added an indicator for import state. Not sure it it's enough or if it's in the correct place but, as a prototype, it's doing what I expect.

[rh]

My connection is only 20Mb, not "fast" compared to others I've seen mentioned here. I only download about 20 groups at any one time and almost never "Download All Headers".

So I'm probably safe. In the next update I'll observe the indicator and it will likely verify that I'm not pushing any envelope.

[Quade]

With 250 groups, I see it queue 150-160 blocks of data for insert. When I just update a couple groups, like a double-click on a topic it's pretty much real time.