Help can't download headers for just certain groups!

[Izzy2]

Hello,

I have been downloading headers daily from alt.binaries.teevee, among other groups. Two days ago a.b.teevee stopped downloading automatically and it had a green dot next to it. If I clicked on it I could manually download the headers. So, I deleted the group and added it again. Now I can't download headers at all!! It just sits there. I don't get any critical errors. I tried adding two other groups to see if that would happen with all new groups I added. It happened with one, but not the other. I don't think I have any filters enabled. I have posted below what it shows on logging. It keeps popping up over and over again. I will also attach some screen shots. I would really appreciate any help you can give. This is so frustrating!!!! Update as I was typing this I received a critical error, I am also attaching it. Not sure the critical error is related.




Logging:
[13:22:36] HIGH - My_Server alt.binaries.teevee XOVER 370297480-370297530
|Done
[13:22:41] HIGH - My_Server alt.binaries.multimedia XOVER 1217300554-1217300604
|Done
[13:22:41] HIGH - My_Server alt.binaries.multimedia XOVER 1217310554-1217310604
|Done
[13:22:41] HIGH - My_Server alt.binaries.teevee XOVER 370307480-370307530
|Done
[13:22:41] HIGH - My_Server alt.binaries.teevee XOVER 370317480-370317530
|Done
[13:22:46] HIGH - My_Server alt.binaries.multimedia XOVER 1217320554-1217320604
|Done
[13:22:46] HIGH - My_Server alt.binaries.multimedia XOVER 1217330554-1217330604
|Done
[13:22:47] HIGH - My_Server alt.binaries.teevee XOVER 370327480-370327530
|Done
[13:22:47] HIGH - My_Server alt.binaries.teevee XOVER 370337480-370337530
|Done

[Quade]

Couple things I'd try:

1 - Just for grins, in the server options check "Disable XFEATURES" then restart and try again.

2 - Reduce your connections in 1/2 restart and try again.

3 - If you're using SSL, disable it, restart and try again.

The error your seeing is a DNS error which is very strange in this context. It's another thing that suggests you might be using too many connections.

[Traveler2530]

I've also had difficulty with "teevee" one of my favorite groups. This trouble began recently, MAYBE it was when I upgraded to 6.42. A coincidence?

Even though I spend hours re-downloading the headers, and the spool directory shows a huge database, none will show up in a search or display.

Points: I DL headers from supernews.

I have my old-old "free agent" from about 8 years ago. It does not have ANY trouble with teevee or my other subscribed groups causing the same trouble. That is, I can DL the headers, search, and DL files flawlessly.

What's going on? Can I do a work-around ?

[Quade]

What do you see in the "Cache" line in the status bar? A number in parens represents number of records waiting to be imported. If you hammer away at header downloads and don't let that number go to zero, you'll never get all the headers fed into the database. The issue is that headers download much faster than they can be imported so, Newsbin imports them in the background.

If the number in parens never goes down then it suggests and import is stuck.

[Traveler2530]

What do you see in the "Cache" line in the status bar? A number in parens represents number of records waiting to be imported. If you hammer away at header downloads and don't let that number go to zero, you'll never get all the headers fed into the database. The issue is that headers download much faster than they can be imported so, Newsbin imports them in the background.

If the number in parens never goes down then it suggests and import is stuck.

The cache numbers drop about 100 to 200 per minute. I see the "New Records" counts for the groups going up. Very slow considering the cache status shows "Cache: 92/100 (107653)"

How can this be sped up?

[DThor]

The number in brackets shows your backlog, which is significant. Let it catch up (bracket number goes away) before throwing more IO at it. Odds are you're thrashing like mad which simply slows everything down and takes significantly longer overall.

DT

[Quade]

The cache numbers drop about 100 to 200 per minute. I see the "New Records" counts for the groups going up. Very slow considering the cache status shows "Cache: 92/100 (107653)"

It's pretty fucked. I'm not really sure what to advise at this point. Maybe to wipe the spool_v6 folder and start over. Leave Newsbin running and see if this number "(107653)" goes down. Don't do a another header download till that number clears. If it seems like it's going to take too long, you might want to start over from scratch.

How fast is this machine? Just as a reference, I've never seen a count this large and 200 a minute is a pretty decent rate for importing.

[Ace]

The cache numbers drop about 100 to 200 per minute. I see the "New Records" counts for the groups going up. Very slow considering the cache status shows "Cache: 92/100 (107653)"

How can this be sped up?

I've only been on version 6.42 for a few days, but I don't think your problem is with version 6.42 because I just downloaded the same group from the same server, and on the download tab headers were coming in at the rate of about a million a minute. The "new records" count seems like a less reliable measure, I would guess because maybe the new records are different sizes? But it was at least 1500 a minute.

My cache status showed 90/100 during the download, though sometimes there was a (1) and very rarely a (2) after it, so I have no idea how you get (107653)? I never saw it go above (2). I built that computer in 2006 which is pretty ancient by now I suppose, so hopefully you have a newer computer. I do, but I haven't used it for Newsbin yet since the old one still works so well.

Since I'm so new to version 6.42 I can't say how much it fragments the disk, but with some past Newsbin versions, when I had disk I/O problems, I could trace them to disk fragmentation issues. It was a bit of a pain to defrag my entire C-drive, so I actually created a separate partition, mostly for Newsbin, but I also included a couple other apps that caused more than their fair share of fragmentation. So now I can defrag the partition with apps like Newbin as often as I want without defragging the entire hard drive, which helped my I/O a lot.

Of course I have no idea if your issue is related to fragmentation or not, but thought I'd throw that in the pile of suggestions as one possibility you might want to look at.

[Quade]

My cache status showed 90/100 during the download, though sometimes there was a (1) and very rarely a (2) after it, so I have no idea how you get (107653)? I never saw it go above (2). I built that computer in 2006 which is pretty ancient by now I suppose, so hopefully you have a newer computer. I do, but I haven't used it for Newsbin yet since the old one still works so well.

It's mostly a function of your download speed. I get header downloads in the 400-500 Mbps range from Astra and even my machine can't keep up. Still, a typical large backlog for me is 500. I've never seen > 1000 before.

[Ace]

It's mostly a function of your download speed. I get header downloads in the 400-500 Mbps range from Astra ..

Maybe Astra is better than supernews at delivering headers then?

I can download the attachments using my full bandwidth so supernews delivers the bandwidth there, but when it comes to delivering headers, it's only at about a quarter of my available bandwidth. So does this mean supernews is the bottleneck on headers, and not my PC, since I have such a low cache number, and since most of my bandwidth is unused? I was never really sure why header downloads can't use up more of my bandwidth, like the attachment downloads do. Sorry if I strayed a little from the topic but you got me curious, since I would like to get faster header downloads if possible.

[Quade]

Giganews throttles header downloads too. AW is at least 5 times faster than Giga. Typically, if you have a slow header server, you can update them all at once or set Newsbin to update every 15-30 minutes so, you never have many to download.

[Ace]

Typically, if you have a slow header server, you can update them all at once

Thanks for the tip, I didn't know I could do that. I got an error when I did it but it seems to be working anyway.

[14:42:32] ERROR - Error: Socket Connect Failed, Host: news.supernews.com Error Code: 10057 A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied.

I just ignore the error, and it solves the problem...I didn't know Giga throttled header bandwidth, but that's a good solution. Since each one uses a quarter of the bandwidth, only 4 simultaneous header downloads are needed to use all my bandwidth, so I didn't need to do all of them at once.
This will help a lot, thanks. (And now I see higher numbers in the cache also, but not too high)

[Traveler2530]

I went back to 6.32, and the problem has been worked around in poor way, but it works. I've downloaded headers for teevee, cores, boneless and several other very large groups from supernews. When I update them then search them, they have the current posts even up to a few minutes back from when I stopped downloading. So the problem of downloading hundreds of millions of headers but having display and searches turn up nada has been solved but at a price.

Apparently, the header compression/decompression routines for downloading are much less efficient in 6.32. Searches are MUCH slower, as well, taking hours. Also, Norton's Security Suite often sends a pop-up warning "Using Excessive CPU" for Newsbin 6.32 when downloading.

About 2-3 weeks ago, Comcast bumped everyone's(locally??) basic high-speed cable connection from 27Mbps to 50Mbps. When I saw Newsbin 6.42 was released, crossed my fingers and installed it. Newsbin 6.42 showed I was getting 350 to 450 Mbps!! (decompressed rate after downloading compressed, of course.) Kool! Later, I re-downloaded teevee with just under 500 million headers in about 12 hours.

But for large groups including teevee, NOTHING was turning up on searches when I should have seen hundreds and thousands of hits. I think this is related to the problem I had a few months ago with slow header updates. You suggested slow PC then, as well. I worked around by re-installing 6.3x and unchecking "update in background" in the options.

My PC is: MoBo GA-EP45-U3P, Q6600 CPU, 8GB RAM, boot drive is a Patriot Torx2 60GB, the HDD that Newsbin resides on is two WD-640AAKS in RAID0. Sustained write of a single 15GB from two Barracudas in RAID0 to the WDs was 210MB/sec reported by explorer. It took about a minute and 15-20 secs to move the entire 15GB file.

My PC's speed is not an issue.

I think the newer versions of Newsbin are choking on fast downloads of large groups. These header problems began happening ONLY when I updated to the more efficient download/decompress/store in 6.4x?? versions a few months ago. Is this related to a new storage and/downloading compression routine?

What's strange, is I still have an original version of Forte's "Free Agent" that does fine with teevee from Supernews! It does, however, DL at the same speed as Newsbin 6.32

What's up with all this?

[Quade]

I think the newer versions of Newsbin are choking on fast downloads of large groups. These header problems began happening ONLY when I updated to the more efficient download/decompress/store in 6.4x?? versions a few months ago. Is this related to a new storage and/downloading compression routine?

Not really enough data to know.

In 6.42:

1 - Headers download to RV4 files in the spool_v6 folder.

2 - They're then imported into the DB files. The number "(NNN)" in the cache area of the status bar tells you how many records are being imported.

3 - Until all the records are imported, you won't see all the headers.

In all the cases so far with people not seeing their headers, they seem to still be in the import stage. When you download headers at 350 Mbps, they download far faster than they can be imported so they queue up.

If you want to trouble-shoot this. Try one group. Download all headers, watch the import line. And when they all import, load the group and see what you can see.

[Traveler2530]

I think the newer versions of Newsbin are choking on fast downloads of large groups. These header problems began happening ONLY when I updated to the more efficient download/decompress/store in 6.4x?? versions a few months ago. Is this related to a new storage and/downloading compression routine?

Not really enough data to know.

In 6.42:

-snip-

If you want to trouble-shoot this. Try one group. Download all headers, watch the import line. And when they all import, load the group and see what you can see.

I already did it with teevee alone using several different increasingly more severe approaches.

Later today for my final try with 6.42, I'll disconnect ALL HDDs containing anything to do with Newsbin(except the user profile directory on c:\). This is easy to do because I have the entire Newsbin application, .nbi, spool, downloads etc... on the e:\ drive. Next, I'll install a dust-collecting 150GB VRaptor as e:\, then do a complete new install of Newsbin 6.42 to the e:\ drive. Next, I'll download ONLY teevee headers from Supernews.

I'll post what happens in a day or two. If you or anyone here has any suggestions I should add before I begin the 14-16 hour procedure, please post them!

[Quade]

Later today for my final try with 6.42, I'll disconnect ALL HDDs containing anything to do with Newsbin(except the user profile directory on c:\). This is easy to do because I have the entire Newsbin application, .nbi, spool, downloads etc... on the e:\ drive. Next, I'll install a dust-collecting 150GB VRaptor as e:\, then do a complete new install of Newsbin 6.42 to the e:\ drive. Next, I'll download ONLY teevee headers from Supernews.

Most of the time trouble-shooting like this fails when people can't do things in discrete steps. Click click clickity click is what I call it. When people can't just sit back and watch what's going on without messing with the process. I don't think you need to do anything other than what I suggested above. You can watch the process, look in the teevee folder for instance but, I'd just let it run and see what happens.

The only thing you should probably check before "Download All" is what your "Storage Age" is set to. If it's too short, Newsbin will simply delete the headers you're downloading till you hit the download age range. I'd set it to 2000 before the "Download all". If you have numbers in parens in the status bar (next to cache), don't start till they're gone. Either that or wipe spool_v6 before you start.

[Traveler2530]

Later today for my final try with 6.42, I'll disconnect ALL HDDs containing anything to do with Newsbin(except the user profile directory on c:\). This is easy to do because I have the entire Newsbin application, .nbi, spool, downloads etc... on the e:\ drive. Next, I'll install a dust-collecting 150GB VRaptor as e:\, then do a complete new install of Newsbin 6.42 to the e:\ drive. Next, I'll download ONLY teevee headers from Supernews.

Most of the time trouble-shooting like this fails when people can't do things in discrete steps. Click click clickity click is what I call it. When people can't just sit back and watch what's going on without messing with the process. I don't think you need to do anything other than what I suggested above. You can watch the process, look in the teevee folder for instance but, I'd just let it run and see what happens.

The only thing you should probably check before "Download All" is what your "Storage Age" is set to. If it's too short, Newsbin will simply delete the headers you're downloading till you hit the download age range. I'd set it to 2000 before the "Download all". If you have numbers in parens in the status bar (next to cache), don't start till they're gone. Either that or wipe spool_v6 before you start.

Yeah, obviously I'm fishing to work the problem out. Not even sure what I'm fishing for.

I have always had download, storage and display ages all set to 9999. On your suggestion, I'll change download, storage and display ages all to 2000 also on your suggestion. Then I'll un-subscribe from teevee, exit newsbin, delete the teevee directory, edit out any references from newsbin.nbi, re-subscribe to teevee and do a download all.

Wiping spool_v6 directory is not an option. It will take several weeks to rebuild the thousands of binary groups on it. I have already tried deleting the teevee directory within spool_v6 and all references to it in newsbin.nbi and then re-downloaded the entire 485+ million headers several times this week. Using 6.42, always the same problem! Doing either a display(last few days) or searching yield zero headers! This is even though the teevee directory shows over 5 GBs of storage.

[Quade]

Wiping spool_v6 directory is not an option. It will take several weeks to rebuild the thousands of binary groups on it. I have already tried deleting the teevee directory within spool_v6 and all references to it in newsbin.nbi and then re-downloaded the entire 485+ million headers several times this week. Using 6.42, always the same problem! Doing either a display(last few days) or searching yield zero headers! This is even though the teevee directory shows over 5 GBs of storage.

You can always just rename it for the test.

My teevee folder is about 6 gigs. I'd have a look in there and see if you see any RV4 files. If not, if all the data is in the Db3 files then that suggests there's actually nothing wrong with importing and that the headers are there. When you right click the group and "Show all Files". Does the progress bar cycle?

Even on my machine, it takes about 30 seconds to load up all of "teevee". That's about 500,000 sets. If it takes about that or longer and you see the progress bar cycle. The problem is in the filters and not in the loading or display of the group.

If you get to this point, the headers seem to be loading but, aren't visible. I'd exit and rename the filters.db3 file to something else. Then restart and try again.

If its loads quickly, there's no data. If it loads slowly, the display of the data is being filtered out.

[Traveler2530]

I went ahead and disconnected the 1.3TB RAID with my current 6.32(working) Newsbin installation. THen I installed an old but 100% working VRaptor, then installed Newsbin 6.42 on it. I set all of the retention dates to 2000, down from my previous 9999 I've used for years with no trouble.

Also, since the downloaded headers folder(spool_v6) can't be changed in 6.42's options, I manually edit newsbin.nbi so it is on the e:\ drive with the remainder of the Newsbin installation. This is because I don't want 800GB-1,100GB of headers on my c:\ drive.

[Settings]
AutoOLD=0
AutoSave=0
AutoShutdown=0
AutoShutdownMode=0
BwAutoOff=0
DataPath=E:\Users\Quad\AppData\Local\Newsbin\
DisplayAge=2000
DownloadPath=E:\DL\$(GROUP)\
HeaderUpdateWhenStarted=0
.
.
.


It appears to be working now, the only group subscribed to, teevee, showed 10million headers downloaded, then I stopped it. I did a "display all" and it showed about 30k headers. I'm sure a search would work, too.

It must be tested on the full teevee group, but it take into the wee hours to completely re-build teevee. So I'm not sure it is working until I try.

RV4 files flash by every few seconds, then disappear.

[Traveler2530]

I was wrongly using 6.32 and misled into thinking it worked. Nothing has worked, downloading several new groups fail to get and keep headers properly.