32-bit barrier in 64-bit NewsBin?

[RayMark]

This may not be a version specific issue, but I see with 6.50 B16 (64 bit) this picture:

With servers: giganews + easynews, the add group dialog shows for alt.binaries.boneless:
Post Count: 8.01 GB
When actually trying to download headers (download all headers), the Size tab shows:
giganews: 4,218,015,649
easynews: 1,661,666,902

I think the size number should be about 8 billion, not 4.2. This 4.2 billion number is very close to the 32-bit threshold.
In other groups the post count in the add group dialog closely matches the size number when downloading all headers.


So what is the problem: does NewsBin wrongly displays the number but still handles those numbers of posts correctly, or the server simply does not give more headers, or some advanced settings in NewsBin should be selected?

BTW, I really did not indend to download all the headers of bonelesse , but I started just to see the number.
On the other hand, I might download them, to see what happens when loading them all at once

[RayMark]

Or perhaps NewsBin downloads headers in 4 billion steps, and will start another 4 billion after finishes the first one but never shows the whole number?
I think it is misleading in that case, it should show the whole number.

[DThor]

I believe this is a server-end issue, I seem to recall when the counter for a group ticks over at 4 billion, it wraps around and gets reset. A year or so ago that happened with boneless and people started getting anomalies downloading headers from that group. Quade tweaked something to deal with it.

If you try to download boneless to memory, your system will lock up long before it's done.

DT

[Quade]

[08:00:58] HIGH news.giganews.com alt.binaries.boneless GROUP alt.binaries.boneless
|211 8515193824
1613525768
10128719591
alt.binaries.boneless

According to this, it should be showing 8 billion posts in a "download all".

[RayMark]

but it shows only 4 billion. - right after I started "download all"
So something is wrong either with NewsBin or with the server. Perhaps I need to do something in advanced settings?

I also noticed that giganews has 5 x more headers than easynews, at least in sereval groups I compared in the last couple of days: a.b.cores, a.b.mom, a.b.town, a.b.boneless
While advertised retention is 1800 days at easynews, 1889 days at giganews. So there is something wrong with easynews numbers.
Perhaps they have 1800 days retention for some groups only. But recently some tv stuff is more likely to be found in those groups.

Speaking of that, what is the best way to add easynews headers to already existing giganews headers?

[Quade]

It's probably just a cosmetic problem and nothing is functionally wrong.

As for the easynews headers. Just download them. The combining is automatic.

Turn on "Show Network Commands" and you can see what Easynews counts are actually returned. Like the giganews counts I posted above. They'll tell you directly how many headers there are.

I tracked it down. It's just cosmetic. "All" pulls all 8 billion.

[RayMark]

Ok, thanks. Maybe I will try. I think boneless headers should take about 120-150 GB

[RayMark]

As for the easynews headers. Just download them. The combining is automatic.

Which one of the following 5 ways, if any, should I use:

1-2. Is it enough to do "Download latest" for both servers / easynews only?
3-4. Or to delete Range.db3 and do "Download latest" for both servers / easynews only?
5. "Download all" for easynews only.

?

[dexter]

Depends how many days you want. If you want a certain number of days then set those days in the Download Age setting under Options/Setup, then right-click the group and select Post Storage/Use Download Age, then right-click again and select Download Latest. You'll probably want to disable the Giganews server before you do this if you only want to download from Easynews. Then you can re-enable Giganews and then they'll both stay updated after that.

If you want to really download all headers for Easynews only then disable the Giganews server, right-click the group, and use Download All. Actually, just doing a Download Latest with both servers active would do the same thing if you set a really high Download Age (i.e. the supposed retention of the group) and if you've never downloaded headers from Easynews before for this group. The first way is a brute force approach.

For either of these, make sure your Storage Age setting is set high enough to accommodate old headers. Newsbin wil purge any headers from disk that are older than the Storage Age.

[RayMark]

I have a different problem now - how not to download headers:

When I tried to start boneless download, just to see the header count, then I deleted boneless from the download list. Turns out - it is being downloaded anyway.
So now I deleted the whole group altogether - it is still being downloaded.
Then I deleted all boneless headers - they appeared again and keep growing.
I don't want to restart newsbin because I am currently downloading headers of another rather large group (a.b.mom), I want to finish it first.
But network commands show boneless still being downloaded, and the directory with boneless headers has files appearing again if deleted / increasing in size.
But those files are not locked - I can delete them from time to time. But the bandwidth and processing power is wasted, the other group will take longer now.

[dexter]

It's a known bug in beta. Header downloads get deleted from the Download List but don't actually stop. It'll be fixed in B17. Only way to stop it is to shut down Newsbin at the moment.

[RayMark]

OK, thanks.
I am reluctant to shut down Newsbin because the headers of another group are being downloaded.
But perhaps it is safe to restart newsbin and the header download of the other group (which is actually in the download list) will continue without any corruption?

Maybe I should set header overlapping to something higher than 0?

[dexter]

Header downloads do not survive a restart. But it will pick up where it left off if you do a download latest for that group after you start Newsbin back up.

I wouldn't mess with header overlap. It's really there for servers that backfill.

[RayMark]

Interesting.
So giganews and easynews do not backfill?

Anyway, an overlap probably does not cause corruption or duplications, just unnecessary but harmles header download if those headers are already present?

[Quade]

Anyway, an overlap probably does not cause corruption or duplications, just unnecessary but harmles header download if those headers are already present?

It depends on you. If you're in the habit of deleting all the posts before you close a tab, an overlap will make some number re-appear because you're downloading some of the old when you download the new headers. If you're like me and you just let the groups grow, it won't hurt anything.

[RayMark]

No, I never delete headers deliberately. Only accidentally
Because yesterday I was adding a post to the download queue and my fingers slipped somehow, hit a wrong key and the post disappeared.
I'm not sure what happened, but there was no question "are you sure?"
Oh, maybe I locked out the poster - P instead of Y.