Can't download headers on some groups

[zork]

I got couple days back errors on some groups. Sadly I deleted them but i remember that they were "Server Reports bogus MaxSpool > MaxGroup" errors. I searched some info about them and set short Download age, set "Use download age" for group and got latest headers. No error message anymore but it stays forever downloading headers. Downloading files window shows no progress, headers are 0/0 and I waited maximum some 15 minutes when I deleted it from download list. But even then connection wasn't dropped and I had to do "Kill all connections". I set "Show NNTP commands" and tried again. Excerpts from log window are below. I just can't figure out if this is Newsbin problem or Astrawebs. Most of groups are okay but 7 out of 92 subscribed groups show this kind of behavior. And I even unsubscribed and deleted group and subscribed again but nothing changed.
Newsbin version is 6.52 Build 3289
------------------
Download age - 5 days
Use download age
Download all headers
[16:20:06] HIGH astraweb alt.binaries.emulators GROUP alt.binaries.emulators|211 4826635 1158582 5985216 alt.binaries.emulators
[16:20:06] HIGH astraweb alt.binaries.emulators XOVER 1158582-2158582|224 data follows [compress=gzip]
[16:20:07] HIGH astraweb alt.binaries.emulators XOVER 2158582-3158582|224 data follows [compress=gzip]
[16:20:08] HIGH astraweb alt.binaries.emulators XOVER 3158582-4158582|224 data follows [compress=gzip]
[16:20:08] HIGH astraweb alt.binaries.emulators XOVER 4158582-5158582|224 data follows [compress=gzip]
[16:20:09] HIGH astraweb alt.binaries.emulators XOVER 5158582-5985216|224 data follows [compress=gzip]
Group was empty if I opened it with CTRL+L

download 50,000 older posts
[16:21:56] HIGH astraweb alt.binaries.emulators GROUP alt.binaries.emulators|211 4826635 1158582 5985216 alt.binaries.emulators
[16:21:56] HIGH astraweb alt.binaries.emulators XOVER 5935216-5985216|224 data follows [compress=gzip]
Group was still empty

Download latest headers
no progress shown in "Downloading files" window
Remove from download list
log window shows that there is still connection up and XOVER commands are going on
Only when I did "Kill all connections" then it stops

Download latest headers log
[16:22:40] HIGH astraweb alt.binaries.emulators GROUP alt.binaries.emulators|211 4826635 1158582 5985216 alt.binaries.emulators
[16:22:41] HIGH astraweb alt.binaries.emulators XOVER 1182715-1182765|Done
[16:22:41] HIGH astraweb alt.binaries.emulators XOVER 1206848-1206898|Done
...
[16:24:26] HIGH astraweb alt.binaries.emulators XOVER 5888650-5888700|Done
[16:24:27] HIGH astraweb alt.binaries.emulators XOVER 5912783-5912833|Done
[16:24:27] HIGH astraweb alt.binaries.emulators XOVER 5936916-5936966|Done
[16:24:28] HIGH astraweb alt.binaries.emulators XOVER 5961049-5961099|Done
[16:24:28] HIGH astraweb alt.binaries.emulators XOVER 5985182-5985232|Done
[16:24:28] HIGH astraweb alt.binaries.emulators XOVER 6009315-6009365|Done
...
Remove from download list was somewhere here
...
[16:25:52] HIGH astraweb alt.binaries.emulators XOVER 54564911-54564961
|
Kill all connections was made

[Quade]

Download age - 5 days
Use download age
Download all headers

Which means you told it wipe the existing headers and then download ALL headers. ALL always means ALL. It doesn't matter what the download age is set to.

Download latest headers
no progress shown in "Downloading files" window
download 50,000 older posts

Because you already downloaded all the headers. There aren't 50K older header to download anymore.

Here's what's going one.

1 - You purged the group and told Newsbin to download all the headers.
2 - They're all downloading and are being stored in the import folder.
3 - At the same time a process in Newsbin is importing and feeding these headers into a database.
4 - If you look down at the "Status" bar you'll probably see a number in parens (XX). This number represents the number of blocks of data to import.

None of this seems to be what you want.

- I'd look in the import folder, in the data folder and delete all the GZ files you found there.
- I'd right click the group "Post Storage/Use Download Age".
- Double-click the group to make it download the last 5 days worth of headers and to display them.

This group is pretty small. Only had 5 million posts so, having the entire thing on disk is very doable.

[zork]

Import folder is empty and somehow same procedure works on other groups normally.

I did per your instructions but "Downloading files" windows shows:

Download Headers: astraweb - a.b.e-books,Data Folder:\Spool\alt.binaries.e-books,0/0,Downloading,None,5.08.2014 17:51

Its second group with same problem and has also some 5 million posts.

And download progress stays at 0/0, Import folder stays empty and log window shows XOVER commands going and going and going and ...
[17:56:39] HIGH astraweb alt.binaries.e-books XOVER 95168914-95168964|Done
...
[17:59:21] HIGH astraweb alt.binaries.e-books XOVER 172060918-172060968|Done

Its 172 million already and I guess could wait a hour or more to see how high it can go or does it get some error message at last.

[Quade]

[17:56:39] HIGH astraweb alt.binaries.e-books XOVER 95168914-95168964|Done
...
[17:59:21] HIGH astraweb alt.binaries.e-books XOVER 172060918-172060968|Done

It's probing the group trying to determine what date range to pull. It's not actually downloading and storing headers at this point. There's no mechanism for pulling by age built into usenet so, you have to probe the groups. You can tell it's probing because it's only pulling 50 posts at a time. You can also tell because the range is bouncing around while it's trying to zero in on the proper range.

95168964
95168914-
50

[zork]

But why it probes indefinitely for some groups and not for most groups? I do exact moves on other group and it works fine but I have now 7 groups for which it stays probing indefinitely. How do I fix that and how I can download headers for these groups? Or is it problem on Astranews side and I must ask for them?

[Quade]

[11:48:13] HIGH AW alt.binaries.ebooks GROUP alt.binaries.ebooks|211 2571806 115296 2687101 alt.binaries.ebooks
[11:48:14] DEBUG AW alt.binaries.ebooks XOVER 1401198-1401248|Done
[11:48:15] DEBUG AW alt.binaries.ebooks XOVER 2039149-2039199|Done
[11:48:16] DEBUG AW alt.binaries.ebooks XOVER 2358125-2358175|Done
[11:48:16] DEBUG AW alt.binaries.ebooks XOVER 2517613-2517663|Done
[11:48:17] DEBUG AW alt.binaries.ebooks XOVER 2597357-2597407|Done
[11:48:18] DEBUG AW alt.binaries.ebooks XOVER 2637229-2637279|Done
[11:48:18] DEBUG AW alt.binaries.ebooks XOVER 2657165-2657215|Done
[11:48:19] DEBUG AW alt.binaries.ebooks XOVER 2667133-2667183|Done
[11:48:19] DEBUG Total XOVER Size:alt.binaries.ebooks - 2,667,133:2,687,101
[11:48:19] HIGH AW alt.binaries.ebooks XOVER 2667133-2687101|224 data follows [compress=gzip]
[11:48:19] DEBUG XOVER: alt.binaries.ebooks 2667133:2687101

This is how long it took for me. I had a 30 day download age set. 5 days of that group is about 119 files. The line in bold is the one where it actually downloaded headers.

You can always "Download all Headers". It doesn't probe. In the server options, you could try disabling "XFEATURES". If that makes it work, I'm going to guess you firewall or security software is blocking the download.

[syshog]

I'm having this exact same problem with the exact same group..a.b.e-books... I posted in the astraweb section..

I'm trying to download headers for a.b.e-books

I'm getting error

[12:22:05] ERROR Server:AW Server Reports bogus MaxSpool > MaxGroup Numbers :518470:4600972 - alt.binaries.e-books

I've tried deleting the existing headers for the group and I get the same problem. I'm using server ssl-eu.astraweb.com

I'm using 6.53rc2 of newsbin upgraded from beta3. 11 other groups are reporting this problem

problem started 5 days ago...


tried with x-feature disabled

[12:40:23] HIGH AW alt.binaries.e-books GROUP alt.binaries.e-books
|211 4082503 518470 4600972 alt.binaries.e-books
[12:40:23] HIGH AW alt.binaries.e-books XOVER 518470-1518470
|224 data follows
[12:40:24] HIGH AW alt.binaries.e-books XOVER 1518470-2518470
|224 data follows
[12:40:24] HIGH AW alt.binaries.e-books XOVER 2518470-3518470
|224 data follows
[12:40:25] HIGH AW alt.binaries.e-books XOVER 3518470-4518470
|224 data follows
[12:40:25] HIGH AW alt.binaries.e-books XOVER 4518470-4600972
|224 data follows


It starts the header download but stops and leaves the queue.

[Quade]

Try the US server.

It's a tiny group. I'd "Download all Headers" on it too.

[syshog]

Try the US server.

It's a tiny group. I'd "Download all Headers" on it too.

just tried to download all headers when I enable the US server and select download all headers it tries to pull from both EU and US .. The EU server lists a header count in the download window but the US server does not.

both disappear from queue and the log shows the following for the US server like everything is normal.

[12:53:52] HIGH ssl-us.astraweb.com alt.binaries.e-books XOVER 2518470-3518470
|224 data follows [compress=gzip]
[12:53:53] HIGH ssl-us.astraweb.com alt.binaries.e-books XOVER 3518470-4518470
|224 data follows [compress=gzip]
[12:53:53] HIGH ssl-us.astraweb.com alt.binaries.e-books XOVER 4518470-4600972
|224 data follows [compress=gzip]


the spool folder only contains the range.db3 and rangedb3-journal no storage files are created.

I was thinking of trying to go back to 6.52 final.

[Quade]

What's the number in parens showing at the bottom? (XX) That's the real indicator. It's how many blocks of headers are waiting to import. Your logs show the headers downloading just fine.

You could wipe out the "import" folder in the data folder then then try some more downloads. As long as there's a number in parens, it's importing header. They don't go directly to the group.

[zork]

[11:48:13] HIGH AW alt.binaries.ebooks GROUP alt.binaries.ebooks|211 2571806 115296 2687101 alt.binaries.ebooks
[11:48:14] DEBUG AW alt.binaries.ebooks XOVER 1401198-1401248|Done
[11:48:15] DEBUG AW alt.binaries.ebooks XOVER 2039149-2039199|Done
[11:48:16] DEBUG AW alt.binaries.ebooks XOVER 2358125-2358175|Done
[11:48:16] DEBUG AW alt.binaries.ebooks XOVER 2517613-2517663|Done
[11:48:17] DEBUG AW alt.binaries.ebooks XOVER 2597357-2597407|Done
[11:48:18] DEBUG AW alt.binaries.ebooks XOVER 2637229-2637279|Done
[11:48:18] DEBUG AW alt.binaries.ebooks XOVER 2657165-2657215|Done
[11:48:19] DEBUG AW alt.binaries.ebooks XOVER 2667133-2667183|Done
[11:48:19] DEBUG Total XOVER Size:alt.binaries.ebooks - 2,667,133:2,687,101
[11:48:19] HIGH AW alt.binaries.ebooks XOVER 2667133-2687101|224 data follows [compress=gzip]
[11:48:19] DEBUG XOVER: alt.binaries.ebooks 2667133:2687101

This is how long it took for me. I had a 30 day download age set. 5 days of that group is about 119 files. The line in bold is the one where it actually downloaded headers.

You can always "Download all Headers". It doesn't probe. In the server options, you could try disabling "XFEATURES". If that makes it work, I'm going to guess you firewall or security software is blocking the download.

Group is actually alt.binaries.e-books. Disablinf XFEATURES didn't change a thing. Disabled AV and looked at firewall logs and it didn't block anything and these moves didn't also change anything. Newsbin only probes group indefinitely. Group command gives min max article numbers
[13:52:34] HIGH astraweb alt.binaries.e-books GROUP alt.binaries.e-books|211 4082503 518470 4600972 alt.binaries.e-books

And then it start probing, goes over max and gets going on and on and on.
...
[13:54:25] HIGH astraweb alt.binaries.e-books XOVER 4600870-4600920|Done
[13:54:25] HIGH astraweb alt.binaries.e-books XOVER 4621282-4621332|Done
[13:54:25] HIGH astraweb alt.binaries.e-books XOVER 4641694-4641744|Done
...

But... I switched over to astraweb-us instead of astraweb-eu and things started working. I guess there is something weird going on in EU server.

[Quade]

[17:16:14] HIGH AW alt.binaries.e-books GROUP alt.binaries.e-books|211 4129631 472911 4602541 alt.binaries.e-books
[17:16:15] HIGH AW alt.binaries.e-books XOVER 493559-493609|Done
[17:16:16] HIGH AW alt.binaries.e-books XOVER 4581893-4581943|Done
[17:16:16] HIGH AW alt.binaries.e-books XOVER 4581893-4602541|224 data follows [compress=gzip]

Same thing. Using US server. It's a tiny group. There's not much to probe.

[zork]

[17:16:14] HIGH AW alt.binaries.e-books GROUP alt.binaries.e-books|211 4129631 472911 4602541 alt.binaries.e-books
[17:16:15] HIGH AW alt.binaries.e-books XOVER 493559-493609|Done
[17:16:16] HIGH AW alt.binaries.e-books XOVER 4581893-4581943|Done
[17:16:16] HIGH AW alt.binaries.e-books XOVER 4581893-4602541|224 data follows [compress=gzip]

Same thing. Using US server. It's a tiny group. There's not much to probe.

Which begs the question - why there is so much to probe on EU servers? I let it probe some time and it went up to hundred of millions and still didn't stop.

[Quade]

The simple answer is that the EU server is broken. I talked to a contact at AW and they think they might have found something.

[ozzie]

I was having this problem too, my server was ssl-eu.astraweb.com
Changed to ssl.astraweb.com the same groups wouldn't update.
Changed to ssl-us.astraweb.com and all groups now update ok.

[syshog]

The simple answer is that the EU server is broken. I talked to a contact at AW and they think they might have found something.

Groups started to download headers again from the EU server. Thanks for the help!