Why don't downloaded headers show?

[Traveler2530]

I just installed 6.40 after using 6.35. Server is GigaNews, OS Win7.

This occurs for many many groups, will use alt.binaries.madcow.highspeed as an example:

When adding a new group, NB says there are 41.3 kb of headers in alt.binaries.madcow.highspeed.

"Download all Headers" says 930577 headers are being downloaded to alt.binaries.madcow.highspeed.

When I look at my list of subscribed groups, alt.binaries.madcow.highspeed says "0" new records.

When I enter the group and request "Show Posts," "Show All Posts," or "Show Posts Special.." nothing comes up!

Is there a setting I missed when I went to version 6.40?

Other possibilities?

[DThor]

Check Options/Setup/Storage Age. It's possible if that was set to something odd that could cause this. You would still see the posts the first time you load them up in the same session you downloaded, though. Also, probably more importantly, completely disable filters when you display them. That's the button on the top left of the post tab that looks like a power icon. Your global filter might be hiding things, that disables it and shows all the posts you have on disk.

DT

[Traveler2530]

Thanks for the suggestions. I clicked the filters on/off button (on upper left, looks like a power button) so the four filter options disappeared up into the border. Still does not work!

Other try: I had entered 9999 days in all three of the download/display/storage slots to get all possible headers. Changed it to 6000 just in case NB saw 9999 as "something odd" re-downloaded all in the example group madcow.highspeed, but still does not work! No headers show.

More guesses?

Again, thanks

[Quade]

Look down at the status bar, where it says "Cache" do you see a number in parens? That's the number of records waiting to be imported. That number should go down as time passes. If not, it suggest that Newsbin might not be able to save the headers to disk.

1 - Exit Newsbin, right click the shortcut that starts it and select "Run as Administrator" then try the header download again. See if the symptoms change.

2 - In Newsbin, in the options, click "Open Data Folder" then open "Spool_V6". The groups you download headers should be in there and you should be able to look in one and see data.

I took a look in madcow. It's mostly dead. Full of spam. Seeing 200 day gaps in postings. You should see something but, as groups go, there's not much there.

[Traveler2530]

Says Cache: 50/100 (248 (nothing else follows until the "Ready" notice in another block.)

Have always run as admin, and set up as such in properties. Did as you said, no changes.

I see the same problem in 100s of groups, madcow is only an example. Here is what is in the directory:

Thanks for trying, more suggestions?

Directory of E:\Newsbin\SPOOL_V6\alt.binaries.madcow.highspeed

10/28/2012 06:36 PM <DIR> .
10/28/2012 06:36 PM <DIR> ..
10/28/2012 04:03 PM 6,416,607 news.giganews.com-22879325.RV4
10/28/2012 04:03 PM 6,397,655 news.giganews.com-22882195.RV4
10/28/2012 04:03 PM 6,302,812 news.giganews.com-22886001.RV4
10/28/2012 04:03 PM 912 news.giganews.com-22886672.RV4
10/28/2012 04:03 PM 6,270,883 news.giganews.com-22888809.RV4
10/28/2012 04:03 PM 6,338,161 news.giganews.com-22890978.RV4
10/28/2012 04:03 PM 6,958,651 news.giganews.com-22894363.RV4
10/28/2012 04:03 PM 2,237 news.giganews.com-22895845.RV4
10/28/2012 04:03 PM 6,816,944 news.giganews.com-22899339.RV4
10/28/2012 04:03 PM 6,834,839 news.giganews.com-22902101.RV4
10/28/2012 04:03 PM 6,848,650 news.giganews.com-22905221.RV4
10/28/2012 04:03 PM 2,470 news.giganews.com-22905548.RV4
10/28/2012 04:03 PM 2,047,915 news.giganews.com-22906703.RV4
10/28/2012 04:03 PM 3,072 Range.db3
10/28/2012 04:03 PM 0 Range.db3-journal
10/28/2012 04:08 PM 81,920 Storage.db3
10/28/2012 04:08 PM 0 Storage.db3-journal
10/28/2012 04:08 PM 32,768 StorageData.db3
10/28/2012 04:08 PM 512 StorageData.db3-journal
19 File(s) 61,357,008 bytes
2 Dir(s) 966,004,518,912 bytes free

E:\Newsbin\SPOOL_V6\alt.binaries.madcow.highspeed>

[Traveler2530]

I just used an old copy of Agent (before my NewsBin days) and got into the group. There are tens of thousands of posts, some as recent as the 24th. Certainly a good bit of spam, but also games, movies, and so on.

Why can't NewsBin 6.40 load the headers?

I have not had this trouble before. If I put 6.34 back on, how can I do it without losing my new downloads?

[Quade]

Compared to a more heavily traveled group, it's a backwater. Just say'in... Some groups get more posts in one day than this group has had in the past 4 year. That's neither here nor there though.

I see what the problem is. The headers are downloading, they are landing on disk. I can see them there in your folder. I can also tell you're running 50 connections, which again isn't the issue here but, isn't recommended. You ought to be able to hit full speed with 10-20. What kind of machine and disk is that?

Here's the problem. The records that are downloaded don't seem to be getting imported into Newsbin's database.

1 - I'd exit Newsbin.

2 - Wipe that group folder.

3 - Start Newsbin, right click the group and "Download all Headers".

4 - Sit back and watch.

When it finishes, look at the cache line. Watch it for awhile and the number in parens should either not be there or should shrink over time. When it goes away, all the records are imported.

At that point, if you load the group, you should see all the headers. On my machine, I don't have to wait, up to about 100 Mbps the headers import in real time. I'm trying to figure out why yours don't seem to import. That's why I'm taking this step by step. If it's just a corrupt file, it should be fixed with this.

[Traveler2530]

Good suggestions, but has not worked. Same problem, no changes. I used the add/delete groups utility to remove spool_v6/madcows, closed NB, manually deleted the madcows dir, rebooted.

Started NB, added the group, downloaded all, same problem.

PC is home made 2008. Q6600 CPU, 8GB RAM, mobo is a Gigabyte GB-P35-UDP with the NB disk two WDC-640 in RAID0.

Same setup and equipment I've been using for NB for 4-1/2 years.

I just got curious today and saw 100k and more headers DL'ed to many groups, but "0" showed new..

Maybe this problem always existed, I just looked at it today and discovered WTF??

[Traveler2530]

I noticed other groups are slowly updating the count of "New Records" even though the view at the bottom of NB's connections pane show no activity.

It looks like the records are not being imported into NewsBin's database real time, but background.

Is there a setting I missed?

[Traveler2530]

Got it! When you said on your computer "headers import in real time" and I saw updates in record count but but no connection... I looked around for settings for "realtime" or "background." Found it in the .nbi file.

Thanks for pointing me in the right direction.

I closed NB, searched the .nbi file, changed "BackgroundInsert=1" to "BackgroundInsert=0" saved, re-started NB, now I have 56,000 headers in madcows and 100s of other groups that showed "0" new records previousy, now have 10,000s of new headers/records each.

Is this in an option or setting I missed? See below.

[Settings]
AutoOLD=0
AutoSave=1
AutoShutdown=0
AutoShutdownMode=0
BackgroundInsert=1 <== CHANGED to "0"
BwAutoOff=0
DataPath=E:\Newsbin\
DisplayAge=9999
DownloadFinishedSets=0
DownloadPath=E:\EDownloads\Downloads\
HeaderUpdateWhenStarted=0
HideProgress=1

[Quade]

Hmm. I thought I removed that as an option. This is build 2059?

Digging through the code, I don't see anything looking at "Backgroundinsert" any more which suggests this is an older 6.40. Not the release (or it's just a co-incidence).

[Traveler2530]

I think you are correct about version. I found the slow update problem and corrected the 1 to a 0, but that was AFTER I re-installed 6.34.

I figgered I had fixed the problem, so I re-installed 6.40. Went back to a very slow. Changing BackgroundInsert to a "0" did not help, it was still a "0".

So I went back to 6.34. Goes fast as before!

Why was the background option removed in 6.40? It made my NB databases slo as molasses! It's intolerable. I'll have to keep 6.34 so it does real-time updates unless other fixes come along.

[tgeery]

I'm seeing incredibly slow header loading performance now on 6.40 as well. takes literally an hour or longer to load some groups. I've tried setting display age back to just October and storage age is 30 days. I used to be able to leave display age much older and storage age at 90 days.

I've been reading through a lot of these slow headers posts and I might try blowing away the folder for the group to see if it improves anything... my cache numbers show 90/100 (455)

this is on a 3 ghz quad core with 12 gb ram and the data drive is dedicated to NB so not being used by other apps. no other system processing going on to slow things down.

[Quade]

Several potential factors

1 - If you "delete all posts from poster", this bangs away at the database files too.

2 - The more "poster lockouts" you have, the slower the load will be.

3 - Some groups are getting hammered with spam currently.

4 - If you see a number in parens "(NN)" down in the cache line. Newsbin is still importing headers. Until it's done, header loading will be slower.

I can load 60 days of 30 large groups at once in under 30 seconds so, you clearly have something majorly wrong going on. I might try loading one group at a time and see if you can isolate the group. It might be a sign that your disk drive is going south too or needs to be defragged.

[tgeery]

the drive is regularly defragged and was just defragged to the point that the defragger said there was nothing to do. SMART analysis on the drive shows no issues. swap file is on a completely different drive and the download drive is another completely different drive. all different drives, not just partitions.

now, I do have a ton of blocked posters, I'll admit that. I don't generally delete posts from a given poster so that shouldn't be a problem... I can always get rid of all of the blocked posters just to see what difference that makes.

beyond that, an example of a problem group is boneless. yeah, I know that is a huge group with a ton of headers but it used to load in minutes. now, it takes 2 hours. the progress bar at the bottom of the header window will scroll over and over and then finally go away and the CPU load will go down and you'll think it is finally done and then all of a sudden the CPU load goes up again and the progress bar starts all over again.

it is weird...

[Quade]

the progress bar at the bottom of the header window will scroll over

If you're double-clicking the group, it's both importing new headers, downloading and loading all at once. Just for grins you might start from scratch, right click and "Show Posts" and see how long it takes.

CPU load will go down and you'll think it is finally done and then all of a sudden the CPU load goes up again and the progress bar starts all over again.

Suggests it's your poster list or filters too.

The first progress bar is the actual load. The second is the filter pass. Boneless is the largest group of them all by a large margin.

[tgeery]

yeah, I know boneless is a bad example... it happens on a number of other groups for me too. and the point of comparison between previous version behavior and current behavior, too.

to open the group I use Control-L hotkey. I don't think that downloads headers but you'd know better than me!

[tgeery]

so, no blocked posters at all (deleted them all), I have 26 items in the subject reject filter and nothing else. I started loading a group over an hour ago and it is still loading now and only shows a count of 76 headers in the tab so far and it is still scrolling the progress bar.

so, if nothing really changed in NB that would cause this, can you provide me with what you'd suggest as the "best" settings for faster load times in heavy traffic groups? stuff to maybe limit the overall size of the DB but retain enough to still keep relevance, etc?

[Quade]

I'm not sure where do go with this.

1 - I might pick a slow but, not particularly large group and purge it and re-download headers and see of that changes anything.

2 - I move move the filters.db3 completely out of the data folder and run without one.

Not sure what else to suggest, you results are way over the edge. I'm still wondering if it's hardware or disk or something like that.

Edit:

You could also right click a group/port storage/compacts database and see if this does anything. Might want to set it to run overnight.

[Quade]

I may have seen this happen. I was looking at a group that Newsbin would normally load in 10 seconds and it got 1/2 through a load then stalled out.

it's looking like the spammers are hitting the group with random "from" addresses and that's slowing the load down. I'll look into working around the problem.

[tgeery]

thanks for the suggestions. I've checked all logs (event viewer, SMART data) and the drive for NB data is not reporting any issues. I've used the suggested setup where my data drive is not used for anything else, swap file drive is by itself, download drive is by itself.

none of the drives are showing any issues from everything I can see.

[Quade]

At this point, I think it's the data and specific to the group. Basically the random from fields act like a DOS attack. I'm already testing a solution

[tgeery]

cool! and thank you for being willing to look at it from all sides.

[Quade]

http://www.newsbin.com/downloads/nb641B3.exe

How about trying this one and letting me know. It's not seen much testing.

[tgeery]

sure...

[tgeery]

a very large group loaded in minutes. looks like a good fix so far!

[tgeery]

so, I am seeing one issue though: I blew away the a.b.boneless group and deleted the associated folder on the data drive as a part of troubleshooting the header load issue. in trying to get that group set back up, I've re-subscribed to the group and I set a download age of 30 days for empty groups.

I'm getting the following errors and the headers do not download.

[11:03:05] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Compressed Headers - Timeout Waiting for Server to send Data: Socket - Read or Write Timeout
[11:03:05] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write Timeout

any ideas?

[tgeery]

once I right clicked the group and selected download all headers, it seems to be starting to get the headers so I'll see if it was just that I never "filled" the new empty group and that was what was causing the problem.

[tgeery]

never have been able to get the header download for a.b.boneless to complete. always gets partway into it and then stops. next time it starts or I start NB back up, it just repetitively logs:

[09:26:25] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Compressed Headers - Timeout Waiting for Server to send Data: Socket - Read or Write Timeout
[09:26:25] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write Timeout
[09:27:27] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Compressed Headers - Timeout Waiting for Server to send Data: Socket - Read or Write Timeout
[09:27:27] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write Timeout
[09:28:29] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Compressed Headers - Timeout Waiting for Server to send Data: Socket - Read or Write Timeout
[09:28:29] ERROR NNTPSocket - Server:ssl-us.astraweb.com NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write Timeout

[Quade]

Try it without SSL and see what happens. It's going to take a week to get all of boneless downloaded anyway. As long as it doesn't restart from zero when it starts again. I'm not sure this is an issue.

[tgeery]

yeah, it does restart from zero. the group is empty. I'll try it without SSL if it keeps up. I also turned off the use header download age just to see if it would make a difference. I would have thought that the limit on the age would have made the task easier but who knows...

[Quade]

"Download all Headers" always means all. If you do this over and over again then it'll never finish.

"Download all Headers" one time. Then if you have to restart it, "download latest" from that time forward will make it download where it left off.

The most common cause of the timeout issues is your firewall/V scanner but, it could be server load too. You'll never be able to pull that group in one attempt. If only because the server will disconnection you after X amount of download.

[tgeery]

good to know, thanks Quade.

[tongueboy]

6.41B for me and still the same problem.
I know people with pirated versions who don't have this problem.
This needs to be fixed.
It is a problem with the program, not usenet.

[dexter]

Exactly what problem are you reporting? If you can provide some details about what you are doing and seeing, we might be able to help.

[tgeery]

this problem is back for me as well - trying to load headers in a group and it is taking hours to load. last version (Link was provided to me in this thread by Quade) worked better it seemed.

I've been waiting on headers to finish loading in a group for at least 3 hours now and it is only about 3/4 of the way done by the progress bar at the bottom and the cache shows "90/100 (342)" and that hasn't changed much over time.

this is an issue that I raised earlier in this thread and has been raised by others. I never used to see this issue in earlier versions of NB... don't know what other info I can provide to help

[Quade]

Maybe the group that causes it?

The numbers at the bottom tell me you're way behind importing headers. You probably want to stop downloading headers till that number clears. It's kind of pointless to download more headers at this point till that clears out.

You might find that you need to track fewer headers. a.b.teevee alone gets 800 megs of new headers a day. If you track a number of large groups, you might have to figure out how much your PC can take.

[tgeery]

yeah, we've had that discussion. Quad core, 3 Ghz, 12 Gig ram. it's a current PC and there was a header download going on with a different group that I wasn't trying to open. so, what you're saying is that the header download in the background was slowing my attempt to open another group with just a Control L display headers command?

[DThor]

Yup, I have a similar computer and if I wanted to load a massive group while another massive group was being processed, well, I wouldn't. It's mostly about IO bandwidth. I'm sure an SSD would be snappy, but you'd still be thrashing. I tend to let the download /consolidation process get over with before I start digging through headers.

I mean, assuming in this case Quade's theory is correct.

DT

[Quade]

It's just a theory. Not knowing the group, I don't know if it would happen to me too so, I'm left with guesses. Dex has noticed that the Warez groups are getting absolutely hammered with spam. To the point, the search server feeding has slowed down significantly feeding them.

Since it seems group dependent, it would make sense that it's related to the group. Importing headers at the same time you load will impact load speed but, I've never seen "impact" to mean 3 hours. I've not had any group that I look at cause anything like that. I don't look in all the groups thought. I don't look in the warez groups at all.

Until I know the group, there's nothing for me to test.

[tgeery]

roger that... I see the spam too. I've had to reset some filters to try and get rid of more and more and maybe that's what a lot of this is... groups that I load for music and such load quickly.

[Quade]

Filters won't help. Best way to handle it is select and "Shift-Delete" to remove it from disk.