boneless broken in b14/b15?

[mho]

It seems boneless cannot be stored, anymore - maybe 'cause it broke through the 1Gheader barrier (astra).

I noticed it would try to re-download everything, last few days, so I deleted it and did a "Download All Headers". It took a bit over a day, but it actually completed on first try! Things are getting stable:-)

I exited newsbin (b15) and started and dhit "Update All Groups", only to find a boneless (currently stuck) at 0/1030636448. (Was at something like 1028xxxxxx when I started the previous download, so it broke 1 Gheader fairly recently)

Is it working for anyone else?

- mho

[mho]

Just noticed, it seems to be looping:

Code: Select all

astra-eu,Connected,a.b.boneless,XOVER 4405272352-4405572352,224 data follows [COMPRESS=GZIP]

I hit pause (to capture the log that was scrolling madly with identical lines saying the same as the above connection status line), and then it suddenly started updating the db; it quickly reached 263000000/1030636448 and the log now scrolls with 1000s of

Code: Select all

[01:22:09] DEBUG  NRF  - Spool: P:\NewsBin\Data\SPOOL_V6\alt.binaries.boneless\Storage.db3 Loaded: 432074 Bytes

- mho

[mho]

According to the server,

Code: Select all

281 Ok
group alt.binaries.boneless
                           211 1030772338 4143972352 5174744689 alt.binaries.boneless


so I can't quite see why newsbin seems to be repeatedly downloading the same 300000 headers over and over again (in 50Mbit/s...)

- mho

EDIT: It has now come unstuck and is getting new headers. Basic problem that it seems intent on re-XOVER-ing all of boneless remains, though.

[Quade]

Code: Select all

[19:42:40] HIGH   NNTPSocket news.giganews.com  - MODE READER

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 200 reading enabled
[19:42:40] HIGH   NNTPSocket news.giganews.com  - XFEATURE COMPRESS GZIP

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 290 feature enabled
[19:42:40] HIGH   NNTPSocket news.giganews.com  - DATE

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 111 20110528234253
[19:42:40] HIGH   NNTPSocket news.giganews.com  - GROUP alt.binaries.boneless

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 211 3549438299 1563366622 5112804920 alt.binaries.boneless
[19:42:40] HIGH   NNTPSocket news.giganews.com  - XOVER 5111495505-5111795505

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 224 xover information follows [COMPRESS=GZIP]
[19:42:50] HIGH   NNTPSocket news.giganews.com  - XOVER 5111795505-5112095505

[19:42:50] HIGH   NNTPSocket news.giganews.com  - 224 xover information follows [COMPRESS=GZIP]
[19:43:02] HIGH   NNTPSocket news.giganews.com  - XOVER 5112095505-5112395505

[19:43:02] HIGH   NNTPSocket news.giganews.com  - 224 xover information follows [COMPRESS=GZIP]
[19:43:12] HIGH   NNTPSocket news.giganews.com  - XOVER 5112395505-5112695505

[19:43:12] HIGH   NNTPSocket news.giganews.com  - 224 xover information follows [COMPRESS=GZIP]
[19:43:20] HIGH   NNTPSocket news.giganews.com  - XOVER 5112695505-5112804920

[19:43:20] HIGH   NNTPSocket news.giganews.com  - 224 xover information follows [COMPRESS=GZIP]




t'aint seeing it here.

[mho]

t'aint seeing it here.

Hmm.. Maybe astra's db is corrupted? Anyone using astra's eu servers for boneless?

- mho

[mho]

I just realized that the 8-ish GB the db uses is not nearly enough for boneless, so it seems to have failed to store it properly in the first place.

(Yes, I have a weird setup, but it worked last week and other big groups are still fine)

- mho

[mho]

Comparing our numbers, it seems more than likely that it's astra that's broken.
They should be at 1020 days of boneless (which I guess is not far behind giga), so 1030772338 vs 3549438298 articles seem to indicate some serious problems. (Thinking about it, I seem to recall there were more headers, before - I guess I just got fooled by the fact that the claimed number of headers was just over 1 G:-))

Sorry for the (probably) false alarm!

- mho

[Quade]

Mines's 7-8 GB. I'm not sure how you can make any of these assumptions to be honest. Sounds like there might have been a short term AW glitch and that the problem has gone away.

Record numbers have no real meaning. You can't compare one servers record indexes to another. You also have to keep in mind that most servers count "Retention" as being able to deliver files, not headers. You could have a server with 1000 days retention but, only 300 days worth of headers. In this case, AW seems to have 1/4 the headers of Giga but, that doesn't signify a problem.

Giga is the only server I know of that can deliver the same range of headers as they can files.

[mho]

Mines's 7-8 GB. I'm not sure how you can make any of these assumptions to be honest. Sounds like there might have been a short term AW glitch and that the problem has gone away.

I compared with alt.binaries.hdtv.x264 which is (obviously) much smaller, and that is 13GB.

Record numbers have no real meaning. You can't compare one servers record indexes to another. You also have to keep in mind that most servers count "Retention" as being able to deliver files, not headers.

I was, of course, looking at the number of headers, not the article numbers. So far, astra has delivered full headers for all groups I've tried... (eu farm; I have seen some indications that the us farm has not always had full headers. I guess they might have lost the eu boneless index and copied it from the us, or something)

- mho

[mho]

Mines's 7-8 GB. I'm not sure how you can make any of these assumptions to be honest.

I just checked on the machine I used before I virtualized newsbin; it was last updated 2011-04-22, and its boneless (also from astra) used 30GB:-)

- mho

[mho]

astra's problems with boneless seem to be over; they now (again) claim 3Gheaders.

Code: Select all

211 2987444608 2256535552 5243980159 alt.binaries.boneless

Unfortunately, I still cannot seem to get newsbin (now rc2) to successfully store it...

I have tried all I can think of, removing every trace of the group (spool directory, culling it from GROUPS.db3, etc), but no luck; it still will try to download it from scratch all the time.

I wonder if there is a regression in the recent nb builds regarding large article numbers?
Is NN_MaxGroup in GroupRange really supposed to be negative?
In my (recently created) SPOOL_V6\alt.binaries.boneless\Range.db3, I get

Code: Select all

astra-eu|alt.binaries.boneless|2256535552|-2037681744

after getting a few headers - looks fishy to me, but it's possibly an artifact of download not being complete...

- mho

EDIT: To me it looks like newsbin downloaded 750k headers but failed to store it properly:
2256535552+2037681744
4294217296
4294217296-2^32
-750000

[mho]

My analysis (guesswork?:-)) is that it works for you on giga due to their low article number being below 2^31. astra's, for some reason, are much higher...

Did you, perhaps, use a "creative approach" when fixing article numbers above 2^32?:-)
(I have now more or less confirmed that my NN_MaxGroup gets set to ((highest downloaded article number) - 2^32))

- mho

[Quade]

Astra is apparently doing header maintenance. Whether that relates to this or not is the question. Newsbin doesn't use the record numbers when storing headers, just to know what range to download. Essentially the record numbers are thrown away after download.

19:42:50] HIGH NNTPSocket news.giganews.com - XOVER 5111795505-5112095505

astra-eu,Connected,a.b.boneless,XOVER 4405272352-4405572352

5,111,795,505 - Giga
4,405,572,352 - Astra

If your theory is that Giga has lower record numbers....That doesn't appear to be the case to me.

Download Headers: AW - a.b.boneless,Data Folder:\Spool\alt.binaries.boneless,814215/52530856,Downloading,None,0d:0h:2m

[10:34:51] HIGH NNTPSocket ssl.astraweb.com - 211 2988416515 2256535552 5244952066 alt.binaries.boneless
[10:35:39] HIGH NNTPSocket ssl.astraweb.com - XOVER 5192721210-5193021210
[10:35:39] HIGH NNTPSocket ssl.astraweb.com - 224 data follows [COMPRESS=GZIP]
[10:36:17] HIGH NNTPSocket ssl.astraweb.com - XOVER 5193021210-5193321210
[10:36:17] HIGH NNTPSocket ssl.astraweb.com - 224 data follows [COMPRESS=GZIP]
[10:37:02] HIGH NNTPSocket ssl.astraweb.com - XOVER 5193321210-5193621210
[10:37:03] HIGH NNTPSocket ssl.astraweb.com - 224 data follows [COMPRESS=GZIP]

I purged the group, did a download latest and after it probed around, it's downloading 50 million of a claimed 3 billion

SQLite version 3.7.4
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema
CREATE TABLE GroupRange (NN_Server TEXT,NN_Group INTEGER, NN_MaxGroup INTEGER, PRIMARY KEY(NN_Server,NN_Group));
sqlite> select * from GroupRange;
AW|alt.binaries.boneless|5192421210|5196855730
sqlite>

[mho]

5,111,795,505 - Giga
4,405,572,352 - Astra[/qoute]

Code: Select all

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 211 3549438299 1563366622 5112804920 alt.binaries.boneless

If your theory is that Giga has lower record numbers....That doesn't appear to be the case to me.

Only for the (claimed) lowest article number (your earlier post:)

Code: Select all

[19:42:40] HIGH   NNTPSocket news.giganews.com  - 211 3549438299 1563366622 5112804920 alt.binaries.boneless

[10:34:51] HIGH NNTPSocket ssl.astraweb.com - 211 2988416515 2256535552 5244952066 alt.binaries.boneless

1563366622 vs 2256535552. It's the only difference I can find:-(
Do you agree that my negative NN_MaxGroup is an anomaly, or is it expected behaviour under some circumstances?

Thanks for looking; I'll try to come up with some more tests...

- mho

[mho]

Finally some progress!

I lowered my Download Age (1200 -> 100), which, btw, required a newsbin restart before it took, and "Download Latest" (after purging) works fine. So, the problems seem to be limited to "Download All Headers".

Could you perhaps try try a "Download All Headers" and see if you get a negative NN_MaxGroup? (When I do it, it will immediately show in Range.db3, so there is no need to leave it running for more than a few seconds)

- mho

EDIT: NN_GroupMax -> NN_MaxGroup

[Quade]

Do you agree that my negative NN_MaxGroup is an anomaly, or is it expected behaviour under some circumstances?

What are you using to display it? It might be a problem or it might be that the software you're using can't display really large numbers.

Other than that I have no opinion. Both giga and AW worked for me. Without more to go on, there really nothing for me to do.

Edit: I did download all headers and then reverted to "Download Latest". No problems.

[mho]

Do you agree that my negative NN_MaxGroup is an anomaly, or is it expected behaviour under some circumstances?

What are you using to display it? It might be a problem or it might be that the software you're using can't display really large numbers.

Latest sqlite windows binary (3.7.6.3 - win32 build; didn't find a 64 bit build, and I don't know how to compile on windows). (Big numbers work fine; my 100 day download (in progress, 57M/496M) is showing

Code: Select all

astra-eu|alt.binaries.boneless|4750364857|511361807

)

- mho

EDIT: Specify version.

[mho]

*sniff*
I restarted newsbin and hit ctrl-G (Update All Groups), and boneless start over, again:

Code: Select all

astra-eu|alt.binaries.boneless|4750364857|-2034825997

6050000/2991942482

So, it's either me or ssl-eu.astraweb.com:563

- mho

[Quade]

I guess it's possible the group command lies to me every now and then and I don't see it because it's essentially a random occurrence. It's also possible, I guess that something happens with the returned record numbers so, they're not atomically incrementing like they're supposed to.

I'll look at the code and see.

[mho]

Hmm... Doesn't look random. It looks like it's trying to store the correct number, but somehow ends up subtracting 4294967296, first, sometimes, and when that is done, it gets confused (for rather obvious reasons). Looks like a 32-bit temporary store of a 64 bit value...

I just wonder what could trigger something like that...

- mho

[mho]

Status on rc3 (so far):
I did a "Download All Headers", but NN_MaxGroup stuck at 0, which didn't go much better than negative numbers:-/

I canceled that download and did an "Update All Groups"; that considered the group to be empty and proceeded to download 100 days' worth (Download Age) of headers (in progress). I will report back when those 500Mheaders are done...

Thanks for trying a fix!

- mho

[mho]

Seems like the update failed with "Unspecified Network Error" just before it should have finished and somehow ended up dumping the _number of downloaded headers_ in NN_MaxGroup. It is now stuck on 493745775 even if newsbin is currently (new update) doing XOVERs in the 2.2G range:

Code: Select all

[23:35:18] DEBUG  NNTPServerWorker  - XOVER: alt.binaries.boneless 2267035552:2267335552

Looks like it is now trying to store MAX(old NN_MaxGroup, ((new NN_MaxGroup) - 2^32)) or something:-)

What sqlite3 version do you build against?

- mho

[Quade]

I wiped boneless and did a "download all headers" using astraweb.

sqlite> .schema
CREATE TABLE GroupRange (NN_Server TEXT,NN_Group TEXT,NN_MinGroup INTEGER, NN_MaxGroup INTEGER, PRIMARY KEY(NN_Server,NN_Group));

sqlite> select * from GroupRange;
AW|alt.binaries.boneless|2256535552|2256742695
sqlite>

Couple minutes later.

sqlite> select * from GroupRange;
AW|alt.binaries.boneless|2256535552|2257057705
sqlite>

AW|alt.binaries.boneless|2256535552|2257265575
sqlite>

Then I went and ate dinner

AW|alt.binaries.boneless|2256535552|2259731258
sqlite>

So, I see no issue.

[mho]

I wiped boneless and did a "download all headers" using astraweb

What server are you using? I use ssl-eu.astrawen.com:563...

I understand how it must look to you, but the results I'm getting are really weird:-)
I'm pretty stumped.

- mho

[Quade]

ssl.astraweb.com.

I'm not understanding where your assumptions are coming from.

Looks like it is now trying to store MAX(old NN_MaxGroup, ((new NN_MaxGroup) - 2^32)) or something:-)

I mean, where did this come from?

[23:35:18] DEBUG NNTPServerWorker - XOVER: alt.binaries.boneless 2267035552:2267335552

2267335552 - 2267035552 = 3000000

Newsbin asked the server for 300,000 headers.

You told it to "download all headers" and that's what it's trying to do. ALL, 300,000 at a time.

[mho]

Aha!

I switched servers, and from ssl-us.astraweb.com, it seems to work fine! (just a bit slow)

Code: Select all

astra-us|alt.binaries.boneless|2256535552|2258232358

I can even stop the update and it will resume in the proper place, next time.

What are the chances for an option to have a per group setting for header server? :-)

(I'd appreciate it if you could try the eu server and tell me if I'm going crazy or not)

- mho

[mho]

I mean, where did this come from?

[23:35:18] DEBUG NNTPServerWorker - XOVER: alt.binaries.boneless 2267035552:2267335552

2267335552 - 2267035552 = 3000000

Newsbin asked the server for 300,000 headers.

You told it to "download all headers" and that's what it's trying to do. ALL, 300,000 at a time.

Yes, that is the problem. It downloads fine, but it messes up the Ranges.db3.
(It used to get negative values, then, after your rc3 fix, it was first stuck at 0, then on 490M...)

- mho

[Quade]

Server's F'd up then.

I imagine if you let it run long enough, you'd get out of the bad range eventually. Range.db3 is only used when re-downloading from a group. It's not used during download other than to update it from time to time. So, during download it doesn't matter what happens to the range values.

[mho]

I finally figured out how to copy text from cmd:

Code: Select all

P:\Newsbin>
P:\Newsbin>sqlite3.exe Data\SPOOL_V6\alt.binaries.boneless\Range.db3
SQLite version 3.7.5
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .sc
CREATE TABLE GroupRange (NN_Server                              TEXT,NN_Group
                        TEXT,NN_MinGroup                        INTEGER, NN_MaxG
roup                    INTEGER, PRIMARY KEY(NN_Server,NN_Group));
sqlite> select * from GroupRange;
astra-eu|alt.binaries.boneless|2256535552|0
astra-us|alt.binaries.boneless|2256535552|2256742695
sqlite>

Notice the slight difference in bahviour? :-)
(I just cleaned the group out, enabled both eu and us servers and started a "Download All Headers")

- mho

[mho]

Server's F'd up then.

So it seems:-( It all looked good to my eye, though... That's why I couldn't figure it out.

I imagine if you let it run long enough, you'd get out of the bad range eventually.

Unfortunately not. I downloaded all 3 billion headers (in a late beta or possibly rc1), but even after it all went well, I had a negative NN_MaxGroup, so the next update, it started over from the first post, again...

Thanks for you patience!

- mho

[Quade]

Unfortunately not. I downloaded all 3 billion headers (in a late beta or possibly rc1), but even after it all went well, I had a negative NN_MaxGroup, so the next update, it started over from the first post, again...

Yeah, if you were still running RC1, that might apply...

[mho]

rc3:

Code: Select all

Download Headers: astra-us - a.b.boneless,Data Folder:\Spool\alt.binaries.boneless,20750000/3025354383,Downloading,None,0d:0h:45m
Download Headers: astra-eu - a.b.boneless,Data Folder:\Spool\alt.binaries.boneless,54650000/3025354071,Downloading,None,0d:0h:45m

Code: Select all

sqlite> select * from GroupRange;
astra-eu|alt.binaries.boneless|2256535552|0
astra-us|alt.binaries.boneless|2256535552|2277638637
sqlite>

I call same behaviour as rc1, except that the safeguard you added to rc3 stores 0 instead of a (changing) negative number for astra-eu.

- mho

[mho]

Here is what the corrupted server returns:

Code: Select all

group alt.binaries.boneless
                           211 3025602580 2256535552 5282138131 alt.binaries.boneless
xover 2256535552-2256535555
                           224 data follows
-2038431744     ZACK AND MIRI MAKE A PORNO DVD 5 THE ILLUMINATI [087/109] - "ZACK AND MIRI MAKE A PORNO DVD 5 THE ILLUMINATI.part086.rar" yEnc (151/157)       WWFMIDDELBURG@power-post.org (WWFMIDDELBURG)     14 May 2009 13:22:51 GMT       <part151of157.Hhcvx7Co0YSmMcUYQpx3@powerpost2000AA.local>                891    2538     Xref: news-big.astraweb.com alt.binaries.boneless:2256535552
-2038431743     (068/100) "jpxpsv950.part067.rar" yEnc (120/134)        JBinUp.com <JBinUp@JBinUp.local>        14 May 2009 13:22:51 GMT        <Qbye6Lq79bNtRXoCwtFj@JBinUp.local>             772     3062    Xref: news-big.astraweb.com alt.binaries.boneless:2256535553
-2038431742     [048/101] "flashpoint.S02.dvd04.tvpinda.part047.rar" yEnc (113/131)     pinda53 <me@privacy.net>        14 May 2009 13:22:51 GMT        <IYxH1NMkmhj1urNv99O8@JBinUp.local>             789     3061    Xref: news-big.astraweb.com alt.binaries.boneless:2256535554
-2038431741     Charlies Angels [007/139] - "Charlies Angels.part005.rar" yEnc (108/201)        Yenc@power-post.org (suparo)    14 May 2009 13:27:00 GMT       <part108of201.z8DeuJUEc4X$hlmIW9aY@powerpost2000AA.local>                778    1984     Xref: news-big.astraweb.com alt.binaries.boneless:2256535555 alt.binaries.ftd:100460254 alt.binaries.test:176568083

Code: Select all

mhoria$~:bc
-2038431744+2^32
2256535552


So: XOVER from astra-eu returns ((article number) - 2^32), which obviously confuses newsbin. Perhaps you can add a sanity check to incoming XOVER data, as, even if the article number is thrown away, it is still used to keep track on what is seen in Ranges.db3. It's probably better to abort such a download than wasting time/bandwidth/server capacity for not much gain.

(Though I suspect the resulting spool is still fine, it's just that it will start over and over and over again on next update:-()

- mho

[mho]

Just noticed: The article number in Xref is correct, it's just the leading article number that's bad.

- mho

[KilJaden]

According to stevef , Astraweb are upgrading/changing their header servers , and until the upgrade is complete headers on some groups might not work as expected .
Try downloading headers from another news-provider and see if the issue remains , but my best guess would be is to wait for Astraweb to finish their job.

[Quade]

I call same behaviour as rc1, except that the safeguard you added to rc3 stores 0 instead of a (changing) negative number for astra-eu.

My thought is that the negative number was like poison. Once it gets set, it can never be reset, whereas with a 0 any valid positive value can replace it.

You're right about the sanity check. I added some when this started but, they assume the server never returns negative numbers.

I'm going to limit the used record numbers to the range between requested min/max ranges so, anything outside that range is just ignored. The data is still used but, the record numbers are ignored.

I'll give EU a try.

[mho]

I'm going to limit the used record numbers to the range between requested min/max ranges so, anything outside that range is just ignored. The data is still used but, the record numbers are ignored.

Sounds great! Perhaps update NN_GroupMax to (if previously lower) the first number in the XOVER range that successfully returned a result, even if the returned numbers are nonsense? (As headers were returned, it should be safe. Hopefully, in the astra-eu case, it would just be one "batch" overlap, and things would work)

- mho

[mho]

Just an explanation why it took me so long to realize what was wrong:-)

I hadn't read the XOVER RFC for several years, so, when I first (using telnet) looked at the XOVER data return (a couple of weeks ago), I didn't realize that the leading negative numbers were supposed to be the article numbers - everything looked OK...
Also, it didn't occur to me to try the US server, as all other groups seem fine from the EU server (and boneless also worked previously).

- mho

[Quade]

I've talked to AW about this. They tell me a fix is in the works. Still, I'm moving forward with the range limiting in Newsbin to be sure.

[mho]

Everything seems to work fine, in rc4.

Unfortunately (:-)), astra seems to have fixed the problem in their end, so I cannot tell if whatever sanity checks you put in help, or not...

- mho