beta4 b1747 bug: header download stuck @ XOVER 2147483647

[Tyador Borlu]

This issue exists at least in all the 6.30 betas to date; unfortunately I switched ISPs at around the same time as I upgraded from a 5.xx version to 6.xx so can't say whether this is ISP specific or not.

Anyway, one of the binaries groups they have (a.b.nl) is stuck at XOVER 2147483647-2147483647 i.e. 32-bit int max while downloading headers; '224 data follows' and 'connected' are reported in the Connections view (netstat confirms this state of affairs). There's no way to get past this point as every time you kill the connection or even go and nuke everything db3 or otherwise that's Newsbin storage, etc. and (re)download the headers, NB will work up to this point and then get that connection stuck, meaning your view of the newsgroup will show up empty and the connection will display that message forever (at least 48 hours on a stretch) while the other connections go and fetch other stuff.

a.b.nl is a rather busy newsgroup; I read in this forum that another 32-bit maxint issue was resolved so I fetched B4 today and this effect is still there. Time to report it.

Server config in NB has 'disable XFeatures' ticked. I'm running the 64-bit build of NB Pro.



If you need access to the ISP (one of the major ones in the Netherlands: ziggo) to test this, PM/email me to get account info for testing/reproducing the issue.

[Quade]

Giganews has well over 2.1 Billion posts in a number of their groups. I wonder if the problem is that the server needs updating and simply can't deliver over the 2 billion point?

If you want to PM me an address and password, I'll be happy to try it.

In Newsbin's network options, you can check "Show server Commands" and see what the server is telling Newsbin.


[09:11:10] HIGH NNTPSocket - GROUP alt.binaries.nl
[09:11:10] HIGH NNTPSocket - 211 1549415757 858176747 2407592503 alt.binaries.nl
[09:11:11] HIGH NNTPSocket - 211 1542020463 871408535 2413428997 alt.binaries.nl

This is giganews and astraweb.

1,542,020,463
8,71408,535
2,413,428,997

Newsbin uses all 64 bit numbers for these counters. Still I might have missed something.

[alexander038]

Don't know if my reply helps but i'm also from the netherlands and have ziggo and i think i have no problems
with ab.nl (astra headers) my StorageData.db3 is over 15gigs and it updates every day.

[Tyador Borlu]

Odd so you don't have the issue. (I take it you're also running the 64bit version of newsbin?)
Ho! You mention 'astra headers' - so I guess you're using that pay server? I'm talking about the NNTP server of ziggo itself (just making sure here we're talking about the very same thing).

Environment here:
ziggo ISP, no pay server
newsbin pro (registered), 6.30b4
os: XP64 English

Okay, turned on the show server response setting; when NB is done with the current queue (ETA in a couple of hours) I'll delete the entire newsbin directory + db3 files and start with the .nbi file only and see what it does again (I have done this before, but that was with beta1).

If that doesn't work out -- i.e. bug recurs -- , I'll PM the uid+passwd so you can have a look-see. (Hm, might as well PM the .nbi file too then, I guess)


[EDIT: to clear up possible confusion: 'you' in first half of this message is @alexander038, 'you' after the environment bla bla is @Quade. Ambiguous addressing by me there.]

[Quade]

Odd so you don't have the issue. (I take it you're also running the 64bit version of newsbin?)
Ho! You mention 'astra headers' - so I guess you're using that pay server? I'm talking about the NNTP server of ziggo itself (just making sure here we're talking about the very same thing).

I'm simply pointing out that on 2 other servers that I use with records over 2.1 billion, it seems to work. In the data folder, in the group, there's a file called "Range.db3". You might try deleting it, then set a "Download Age" to 2. Then right click the group and "Download latest" and see what happens. Range is what tracks the data ranges for the groups.

I test with both the 32 bit and 64 bit versions. They use the same code (not that they always act the same).

You could select everything in the download just and hit the pause button. Then turn on the command and try a header download.

[alexander038]

@Tyador Borlu: yes i ment the isp from Ziggo i think i have to read better before replying next time sorry.

[Tyador Borlu]

Odd so you don't have the issue. (I take it you're also running the 64bit version of newsbin?)
Ho! You mention 'astra headers' - so I guess you're using that pay server? I'm talking about the NNTP server of ziggo itself (just making sure here we're talking about the very same thing).

I'm simply pointing out that on 2 other servers that I use with records over 2.1 billion, it seems to work.

My mistake; that first bit of mine was addressed at alexander038.

[Tyador Borlu]

Okay, I reduced the group set to test a.b.nl and a few small ones to see what the NB log would say.

It starts with

[20:13:05] HIGH NNTPSocket - DATE
[20:13:05] HIGH NNTPSocket - 111 20120206191305 UTC
[20:13:05] HIGH NNTPSocket - GROUP alt.binaries.pictures.manga
[20:13:05] HIGH NNTPSocket - 211 134512 1888992 2023503 alt.binaries.pictures.manga
[20:13:06] HIGH NNTPSocket - GROUP alt.binaries.nl
[20:13:06] HIGH NNTPSocket - 211 1900000001 247483647 2147483647 alt.binaries.nl
[20:13:06] HIGH NNTPSocket - GROUP alt.binaries.nl
[20:13:06] HIGH NNTPSocket - 211 1900000001 247483647 2147483647 alt.binaries.nl

when I invoke 'update headers'. Sounds like the NNTP server of the ISP is stuck at 32bit maxint there in the 211 response.

I have to mention that fetching data from that group via nzb files does fine, and given that this maxint issue was visible here for about 4 weeks now (I was looking for the how and what and a way around; didn't get far, that much is obvious), my _guess_ is that there's a bug in the GROUP response code there; I have googled for this issue too, but I must be completely down on my google fu as I haven't found this XOVER issue being mentioned anywhere in public forums or elsewhere.

Before I did the last test now, I deleted everything in the newsbin data directory except the nbi files there, so all db3 files and everything have been nuked before the test. a.b.nl headers are fetched and given the 211 response there the last reaction is

[20:13:18] HIGH NNTPSocket - XOVER 2147479923-2147479933
[20:13:18] HIGH NNTPSocket - 224 data follows
[20:13:18] HIGH NNTPSocket - XOVER 2147483547-2147483557
[20:13:18] HIGH NNTPSocket - 224 data follows
[20:13:18] HIGH NNTPSocket - XOVER 2147479923-2147483647
[20:13:18] HIGH NNTPSocket - 224 data follows

and a.b.nl 'headers download' is 'stuck' in the Download List view. (Second run indicates that the server took a little while longer to arrive at that last 224 response, BTW)
Which to me seems there's something going wrong again once NB gets to the header at id maxint.

I've also run a third test with your Range.db3 delete tip; dialed down the download days to 1. No change: server fetches a few thousand headers (or so it seems to state) and then it's stuck at maxint again.


Haven't checked with other header download tools yet to see if they survive, but is sure looks like a NNTP server bug (or two).


Anyway, will also 7zip my .nbi file and PM it to you (@Quade) so you can verify. Will dial the #connections down to 2 so we don't collide due to #connections limit at 4.

Thanks for the advice and the help; it's much appreciated.

[Quade]

I have to mention that fetching data from that group via nzb files does fine, and given that this maxint issue was visible here for about 4 weeks now (I was looking for the how and what and a way around; didn't get far, that much is obvious), my _guess_ is that there's a bug in the GROUP response code there; I have googled for this issue too, but I must be completely on my google fu as I haven't found this XOVER issue being mentioned anywhere in public forums or elsewhere.

Record numbers are sort of a fiction. They only apply to a specific group for header downloads and aren't used for downloading posts any more. The message-id is used instead (a unique identifier). Headers aren't required to cover the entire span of the group's contents. In fact, other than Astra, Giga and their resellers, it's common for servers to only have a limited subset of headers compared to the files they can deliver.

[20:13:06] HIGH NNTPSocket - 211
1,900,000,001
247,483,647
2,147,483,647
alt.binaries.nl

Looks like the group is stuck at maxint/2

20:13:18] HIGH NNTPSocket - XOVER 2147479923-2147479933
[20:13:18] HIGH NNTPSocket - 224 data follows
[20:13:18] HIGH NNTPSocket - XOVER 2147483547-2147483557
[20:13:18] HIGH NNTPSocket - 224 data follows
[20:13:18] HIGH NNTPSocket - XOVER 2147479923-2147483647
[20:13:18] HIGH NNTPSocket - 224 data follows

This is Newsbin hunting around looking for records less than 1 day old. The problem here is that with the server stuck at maxint, it's probably delivering headers from months ago instead of 1 day. That's likely why things are stuck. You could probably unstick it by either, "Download all Headers" or setting the download age to before where the post gets stuck. Might try 180 days to start. It's likely a bug in Newsbin but, I never anticipated a broken server that stops delivering current posts.

You've already shown where the problem is. The server is broken. It can't deliver header information for newer posts any more. Until that's fixed, you're probably going to need to use another server for headers.

[Tyador Borlu]

As I replied in the PM:

From the perspective of Newsbin Pro, I'd say 'case closed'. I'll have to see if rattling at the cage of the ISP support desk will give us any forward motion on this one; I've tested the header fetching with a few other nntp tools to see if someone had come up with something wicked to circumvent this sort of bug, but 3 out of 3 tools gives a resounding 'nay'.

So it's payserver or nzb for me. Bloody [censored].

For the folks that come in after me and stumble across this issue: as of the date of this posting, know that the ziggo NL NNTP server (news.ziggo.nl) is broken regarding the delivery of newsgroup headers: they fail to deliver on very busy newsgroups, such as alt.binaries.nl and alt.binaries.boneless.

The only way around the issue is either to use another news server for downloading newsgroup headers or to not download headers at all and just use nzb's where available to access the contents of the newsgroup: the data of alt.binaries.nl, etc. is available at the stated server, it's just that you won't be able to get headers any more recent than somewhere around december 8th, 2011 for alt.binaries.nl. (I can't say exactly what the 'edge date' is for a.b.boneless, but you'll get the point.)

Technical guestimate (mine): looks like the server has a 32-bit signed integer overrun issue in its code; there's nothing you can do about it client side. notice the 'GROUP' request 211 responses as shown before in this thread: that's where the issue first pops up.

Thanks is due to Quade for helping me out; the final diagnosis isn't the one I hoped for, but it is the unfortunate current reality.