Header Import problem sometimes

[Calahan]

Hello,

I updated from Beta 7 to Beta 9 and had a strange header import problem with a group.
At the bottom in () there was a (2) but at the same time 24 headers files were downloaded,
but none were imported. I stopped the header download and the headers weren't imported either.
I restarted Newsbin but no change.
I then installed Beta 7 and it showed (24) files at the bottom.
About every 5 seconds one .gz file was imported. I stopped at (19) and installed Beta 8.
Beta 8 seemed to do nothing, but after minutes (well it felt like 2 or 3 minutes, I didn't stop the time)
it went to 18. I tried Beta 7 again and the files were imported again every 5 seconds.
So when all files were imported, I downloaded headers again and tried Beta 9 again but
no headers were imported. Well I waited about 5 or 10 minutes.
When headers were downloaded with Beta 9 but weren't imported, the number in () at the
bottom didn't change. It weren't increased when the .gz files were downloaded.
After some trying with Beta 7, 8 and now 9 again, it worked fast with Beta 9 as with Beta 7.
Very strange. Maybe someone has the same problem and can report more details?

All the best,
Calahan

P.S. Well thanks for supporting long filenames now. I waited a long time for that feature

[Moondawgie]

With the hope that I'm not hijacking this message thread, but only adding to it.

I downloaded the latest headers for the newsgroup alt.binaries.hdtv 52 hours ago. Until 4 hours ago, the chunk cache use state had been displayed as "Cache: 200/200 (2)." But I found that there are 50 files, all in the format of alt.binaries.hdtv-Astraweb-XXXXXXXXXXXX-XXXXXXXXXXXX or alt.binaries.hdtv-Easynews-XXXXXXXXXXXX-XXXXXXXXXXXX , in the import folder waiting to be imported.

The only entries in the Logging tab were

"[10/14 13:XX:XX] DEBUG NZB FILE Load Checking"
"[10/14 13:XX:XX] DEBUG Periodic Search: Waiting for next Period XXXX Secs"
"[10/14 13:XX:XX] DEBUG Periodic Search: Waiting for next Period XXXX Secs"

repeated over, and over again.

I then exited from and then restarted Newsbin. Three hours later the cache use state was being displayed as "Cache: 200/200 (50)."; the same 50 files were still waiting to be imported, and the exact same string of entries were being made to the logging tab.

I then exited from Newsbin, deleted the files from the Import folder, restarted Newsbin and downloaded the latest headers for all of my subscribed newsgroups, except alt.binaries.hdtv. The new headers were downloaded, and imported in the usual manner, leaving the cache use state as "Cache: 200/200 (0), with the same string of entries being written to the logging tab.

I then changed the post storage settings for the newsgroup alt.binaries.hdtv to "Use Download Age" and with my installation's "Download Age" set to "7" (More than enough time since the last successful update of that newsgroup) and initiated another download of the latest headers. Everything seemed to go as it should, with headers appearing to have been downloaded and imported, and the Cache State changing as things progressed until about an hour ago. Now Newsbin is stuck with 62 files for alt.binaries.hdtv in it's import folder, and the Cache state being displayed as 200/200 (2).

I assume that there is something about the new import files that are choking Newsbin. But how can I overcome the problem?

The file in the import folder currently locked for use by Newsbin is alt.binaries.hdtv-Astraweb-XXXXXXXXXXXX-XXXXXXXXXXXX and is of a size of 40,265 KB. I'm familiar with the process of filtering headers with Newsbin, but I don't know what header is causing the problem and so I don't know what syntax to use to create an entry in my headers filter.

I am running Newsbin 6.8.0B9 Build 4978 under Windows 10 Pro Build 17004 64-bit with 32 GB of RAM.

[Calahan]

I think we had the same problem.

I think when I installed Beta 7, the downloaded header .gz files which stopped the import on Beta 9 were imported,
so when I tried Beta 9 again, there were other newer .gz files which could be imported at last.
Maybe some header .gz files contain something which stops Beta 9 from importing?

By the way, header import is very slow again.
Instead of 5 seconds for one import file, I measured times between 30 and 70 seconds for the import of one .gz file.
So the import folder is filling up now... On the other hand, Newsbin downloaded over 500 million headers by now, without stopping.

[kirm]

I had a similar situation a couple of betas back so I just went back to an earlier version which cleared things up for me. It always was the same group that the gz file got stuck on. Must admit I am not seeing this on beta 9 though. This has happened to me on 2 separate builds already.

Very curious though what triggers this behavior.

[Quade]

Very curious though what triggers this behavior.

Me too. I'm not having any import issues but then again, I might not be hitting the right group. Switching back and forth between B8 and B9 isn't recommended. B7 and prior to B9 is fine. B8 changes the header DB which is slow and when you switch to B9, it changes back so, I'd suggest not using B8 in any of your testing.

[Calahan]

B8 changes the header DB which is slow and when you switch to B9, it changes back

Sorry Quade, my English seems to be not good enough to understand what you wrote.
I don't understand what you mean with "when you switch to B9, it changes back".

I have now installed B9 and the header import is slow.
Do I have to install Beta 7 once more, do maybe a header download and then switch back to Beta 9 again?

If you ment, when I switch to B9, it changes back to the header DB how it was before B8 and so it should be fast again, why is the header import so slow now?

[dexter]

He means that B8 should be avoided. There was a change to the header database index in B8 as an experiment, which failed. If you go from any other version to B8, it will remove an index from the header database which will take some time. If you go from B8 to B9, the index gets re-added to the database which will again take some time. Once the header databases are all updated it should be back to normal. So basically stay away from B8 if you are downloading headers.

[Moondawgie]

With the hope that I'm not hijacking this message thread, but only adding to it.

I downloaded the latest headers for the newsgroup alt.binaries.hdtv 52 hours ago. Until 4 hours ago, the chunk cache use state had been displayed as "Cache: 200/200 (2)." But I found that there are 50 files, all in the format of alt.binaries.hdtv-Astraweb-XXXXXXXXXXXX-XXXXXXXXXXXX or alt.binaries.hdtv-Easynews-XXXXXXXXXXXX-XXXXXXXXXXXX , in the import folder waiting to be imported.

The only entries in the Logging tab were

"[10/14 13:XX:XX] DEBUG NZB FILE Load Checking"
"[10/14 13:XX:XX] DEBUG Periodic Search: Waiting for next Period XXXX Secs"
"[10/14 13:XX:XX] DEBUG Periodic Search: Waiting for next Period XXXX Secs"

repeated over, and over again.

I then exited from and then restarted Newsbin. Three hours later the cache use state was being displayed as "Cache: 200/200 (50)."; the same 50 files were still waiting to be imported, and the exact same string of entries were being made to the logging tab.

I then exited from Newsbin, deleted the files from the Import folder, restarted Newsbin and downloaded the latest headers for all of my subscribed newsgroups, except alt.binaries.hdtv. The new headers were downloaded, and imported in the usual manner, leaving the cache use state as "Cache: 200/200 (0), with the same string of entries being written to the logging tab.

I then changed the post storage settings for the newsgroup alt.binaries.hdtv to "Use Download Age" and with my installation's "Download Age" set to "7" (More than enough time since the last successful update of that newsgroup) and initiated another download of the latest headers. Everything seemed to go as it should, with headers appearing to have been downloaded and imported, and the Cache State changing as things progressed until about an hour ago. Now Newsbin is stuck with 62 files for alt.binaries.hdtv in it's import folder, and the Cache state being displayed as 200/200 (2).

I assume that there is something about the new import files that are choking Newsbin. But how can I overcome the problem?

The file in the import folder currently locked for use by Newsbin is alt.binaries.hdtv-Astraweb-XXXXXXXXXXXX-XXXXXXXXXXXX and is of a size of 40,265 KB. I'm familiar with the process of filtering headers with Newsbin, but I don't know what header is causing the problem and so I don't know what syntax to use to create an entry in my headers filter.

I am running Newsbin 6.8.0B9 Build 4978 under Windows 10 Pro Build 17004 64-bit with 32 GB of RAM.

UPDATE with Additional information added 10/15/2017:

My CPU is an Intel I7-3770K, and my installation of Windows 10 Pro 64-bit has now been updated to Build 17017.

Last night, In frustration, and after exiting from Newsbin, I scanned my PC's system partition, and the partition with my Newsbin Data files for errors (none), deleted all of the .gz files from my import folder, changed the post storage settings for the affected newsgroups to "Use Download Age" (7 days), and deleted the AutoPAR2.db3 and DownloadsV2.db3 files. I then restarted Newsbin and initiated a download of all headers for all of my subscribed newsgroups, except alt.binaries.hdtv hoping to avoid the issue detailed above. Things seem to be moving along so I left Newsbin running and went to bed for the night.

This morning I found that Newsbin had crashed, causing the following event to be written to my PC's Application Event Log file:
Log Name: Application
Source: Application Error
Date: 10/15/2017 10:20:55 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: DesktopPC-1
Description:
Faulting application name: NewsbinPro64.exe, version: 6.8.0.0, time stamp: 0x59c57725
Faulting module name: ntdll.dll, version: 10.0.17017.1000, time stamp: 0xb9bdfd0b
Exception code: 0xc0000374
Fault offset: 0x00000000000f4edb
Faulting process id: 0x9a4
Faulting application start time: 0x01d345d870fd688d
Faulting application path: C:\Program Files\Newsbin\NewsbinPro64.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 35cebff5-f644-445e-bb07-49f37c9d6c0c
Faulting package full name:
Faulting package-relative application ID:
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2017-10-15T17:20:55.780208000Z" />
<EventRecordID>444</EventRecordID>
<Channel>Application</Channel>
<Computer>DesktopPC-1</Computer>
<Security />
</System>
<EventData>
<Data>NewsbinPro64.exe</Data>
<Data>6.8.0.0</Data>
<Data>59c57725</Data>
<Data>ntdll.dll</Data>
<Data>10.0.17017.1000</Data>
<Data>b9bdfd0b</Data>
<Data>c0000374</Data>
<Data>00000000000f4edb</Data>
<Data>9a4</Data>
<Data>01d345d870fd688d</Data>
<Data>C:\Program Files\Newsbin\NewsbinPro64.exe</Data>
<Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
<Data>35cebff5-f644-445e-bb07-49f37c9d6c0c</Data>
<Data>
</Data>
<Data>
</Data>
</EventData>
</Event>

I again scanned the system partition and the partition with my Newsbin Data files for errors. None found.

I then uninstalled and reinstalled Newsbin, retaining all of the files in the Newsbin Data folder on a separate drive. And then restarted Newsbin and had just enough time to notice that the Cache State was being displayed as 156/200 (141), and that there are no files in the Downloading Files or Files List tabs, before Newsbin crashed. Now, after scanning for file errors, Newsbin won't run at all without crashing.

I've run the command ' sqlite3 DownloadMarker.db3 "pragma integrity_check;" ' from an elevated command prompt for all of the .db3 files in the Newsbin Data (parent) folder and, despite no problems being reported for any of those files, I then ran the command ' sqlite3 database_name "VACUUM;" ' for the same files.

Yet, Newsbin continued to crash within a few seconds of every subsequent restart.

In complete frustration I've renamed my Newsbin Data folder to "Newsbin Data From Hell", created a new "Newsbin Data" folder, copied the Newsbin.ini file from the old folder to the new, and am now in the process of downloading all new headers for all of my subscribed newsgroups. The last time I did this, which was a couple of years ago, it took about 103 hours to finish. Fortunately, my internet service has since been increased from 60/5 Mbps to 100/10 Mbps, but that really won't matter much as it's the importation of the new .gz files that will consume my time and system's resources. <sigh>

In the meantime, I'll be watching the Newsbin forum with my fingers crossed to see if anyone comes up with a realistic remedy to my problems which might allow me to return to the use of the files in my old Newsbin Data folder. And I'll probably also copy the old files to another computer and check the integrity and vacuum all of the *.db3 files in that folder's Spool_V3 folder.

Or, perhaps it's time to abandon the use of message headers and utilize NZB files. What's presently a good source for NZB files? I'd appreciate receiving any and all legitimate suggestions and recommendations via private message. Thanks!

[Quade]

n and initiated a download of all headers for all of my subscribed newsgroups,

I hope you didn't "download all headers" because that bypasses the "Download Age". Hopefully you mean you downloaded headers for all groups.

I'd switch back to the other data folder. Remove all the GZ files from the import folder. Re-do "Post Storage/Use Download Age" set "Download age" to 7, then download headers normally (not "ALL").

[Calahan]

Again sitting on 623 .gz import files of 2 groups. Restarted B9, shows 623 import files at the bottom, does nothing more.
I bet, if I go back to B7, they will be imported.
What else can I do?

[Quade]

You could try moving them all out of the import folder then moving them in a couple at a time till you find one that doesn't import.

I'm not having any import problems in B9 or the currently unreleased B10. You could use "Procmon" and see if Newsbin is still loading the GZ. 623, I assume 30-40 meg GZ files is 186 TB of data to chunk through (when uncompressed).

[Calahan]

I left Newsbin alone, hanging on the imports over the night and now 3 .gz files were imported.
Seems to be very very slow..., I think it's too slow to live with.

At the moment it's importing a 46 MB .gz file from the 16th.
(Cannot unpack, because it's locked, but the one after that (nearly the same size) contains about 300 MB of headers)

In the folder are now 632 files, the cache info still shows (620).
Maybe it will be updated, when the import of the current file is done.

I'm using a 500 GB Samsung SSD for Newsbin, so I don't think it could be the hardware.
I could copy all files to a normal harddrive, but on the other hand, if it's fast again with B7, it can't be the SSD, I think.

So now we know it's not stopping the import, what else can I do?
Should I switch to B7 to see, if it's fast again?

[Quade]

Are you using anything like header filters?

What group is slow to import? Some groups are simply unusable for headers these days and one of the symptoms is really slow import. They've basically stopped posting files to these groups and are only posting ungroupable posts. So, instead of Newsbin importing 10,000 files into the DB, it's 100 million. Some groups like TeeVee were attacked around last Christmas and the headers posted will bog Newsbin down.

I have 200 some odd groups. If I update them every 30 minutes. Headers import almost instantly. If I wait a week, it takes as much as a full day to import all the headers. A week of headers these days is easily a TB of headers particularly because some of the posting groups are purposely posting hard to use posts (basically they're NZB only).

You might need to review your groups and see if people are still posting meaningful files there.

I'll do some back to back import testing too. Seat of the pants, I'm not seeing any difference in import speed but, I have a test block of GZ's which allows me to do fully instrumented tests.

[Calahan]

I don't think I use header filters as I don't know how to do it.

I've made backup copies of the .gz files which seem to make problems.
I switched back to B7 and the files were imported with a speed of one file every 10 to 15 seconds.
(The database is about 85 GB.)

Then I made a new data folder with a new Newsbin ini file.
Installed B9 again and copied the problem .gz files into the import folder.
I waited a really long time but it seems it needs an hour or more to import one of the files.

I installed B7 and tried again and every .gz file was imported in about 3 to 8 second.
So I think, if I could send you one or more of the .gz files, you should be able to analyse, why it's so very slow with B9.

The files have a size of about 40-50 MB each.
I think the first, which is so slow has a size of 45 MB.

Do you have a ftp server, I could upload them.
I could post them to usenet, maybe.
Or I could give you access to my own ftp server through a PM.
Or any other idea?

[Quade]

I'm back so I can do more testing today. B10 should be up but the header part, while more robust than B9 won't be any faster. I have an idea to allow you to set the protection level of the DB to manage speed. Basically there are three different ways to write to the DB.

1 - Journal
2 - WAL
3 - Direct.

Using journal and wal modes, the data written to the DB is first staged and then fed in at certain times in a way that a crash wouldn't damage the DB. On the other hand, "Direct" is the fastest but if you crash 1/2 way through writing, the DB is likely to become corrupt.

B10 should be in the forum today. I suspect some of the crash reports are in header download import so, I've added another layer to make the feed more robust.

[Calahan]

There must be an issue with B9, when the same import files are fast in B7 (seconds) and unspeakable slow in B9 (hour(s)).
So you won't test with my .gz file(s)?

[Quade]

There must be an issue with B9, when the same import files are fast in B7 (seconds) and unspeakable slow in B9 (hour(s)).
So you won't test with my .gz file(s)?

Yeah, no real clue. I have a feeling it's a journal/wal mode thing but without actual testing it's just a theory. It's not clear why it doesn't seem to impact me too.

[Calahan]

It's not clear why it doesn't seem to impact me too

That's the reason why I offered you to test with .gz files which makes problems obviously.

I don't know, what you changed between B7 and B9, that could make the header import so slow with SOME import files.
We don't talk about B7 needs 15 seconds and B9 needs 20 or 30 seconds.
B9 needs an hour or even more to import ONE .gz file for what B7 only needs some seconds!

Newsbin has to import header files in a reasonable amount of time
I have some groups with auto-download to get all what is posted, before it's gone.
Now Newsbin needs hours or even days to import files of some groups and when it's done, the new posted files were just deleted.
So I loose downloads. When the headers are there, the newly posted files are gone!
It's a big issue for me.

[Quade]

I've spent the past day or so testing. B7 and B10 + tweaking B10 and exposing some internals in the NBI which will be in B10. I deleted the spool_v6 between runs and optimized the disk between run (trimming the deleted files).

I had 299 40-90 megs GZ files I used over and over again testing the two versions and various safety modes.

B9 really did 2 things differently than B7.

1 - It uses WAL mode
2 - It writes smaller blocks of data to the DB file.

In my testing, the difference between WAL mode and Journal mode was negligible. On the other hand, increasing the block size that gets written to the DB is faster. So B10 uses a bigger block size, bigger even than B7. I switched to the smaller blocks size to make the GUI more responsive but it seems like it's not worth the performance lose.

Oddly I found B7 to be significantly slower than B10 processing these 299 GZs. It was processing them in about 12-15 seconds each but that was still slower then B10. B10 took almost exactly an hour to process all 299 files. B7 only did about 3/4th that in an hour.

Once B10 is up which should be today, I want to hear what your experience is. If it's still really slow for you, I'd love a couple of your slow GZ's. Now that I have a baseline, I can really compare them.

[Calahan]

Sorry, for being so late. So the good news is, B10 is importing my test .gz files much faster than before.

With DatabaseMode=2 the test file is imported in about 37 seconds.
Using DatabaseMode=4 the same file is imported in about 35 seconds.
So Using DatabaseMode set to 2 seems to make only a small difference.

Importing the same .gz file with B7 needs 9 seconds.
So it's 4 times faster for me than B10.

Tried the same, but deleted the group database before.
Time with B10 is about 20 seconds then.
B7 needs about 5-10 seconds (seems to vary much).


So I installed B10 again, set DatabaseMode=2 and continued my group update.
After some time, I saw the number of .gz files waiting to be imported, was rising.
So B10 is too slow for my 100mbit connection, to read and import the headers in real time.

So I can maybe update all groups that are "nearly" up-to-date, but not make group updates of new big groups, as they will fill the import folder, so the other group updates for auto downloading, are not imported fast enough, which will (again) result in loosing downloads...

Maybe it's a good idea, to import .gz files, of auto-downloading groups, with priority.
Or is there a priority setting for group updates, which I don't know?

Maybe I should use a second data folder instance of Newsbin for download headers of new big groups and then copy the Database files to my first installation...
Do I have to update the ini file of the first installation with the group data of the second ini?

Still don't understand why your test .gz files are faster with B10.

[Quade]

So B10 is too slow for my 100mbit connection, to read and import the headers in real time.

Well yeah, that's why it writes the GZ's to disk. With XFeatures, the effective header download speed can be as much as 9-10 times faster than your connection speed. There's basically two ways to handle this.

1 - Download headers and feed them in real time. Which means I'd have to slow the header download.

2 - Download headers at full speed and store them off in GZ files and allow the import to happen in the background.

I chose #2. You could slow the header download by turning XFeatures off if you're troubled by a collection of GZ files.

Still don't understand why your test .gz files are faster with B10.

1 - I run no security software except the MS supplied stuff.

2 - All my testing was to a 1TB SATA SSD.

3 - Fast PC though, It might not be any faster than yours.

4 - The disk was essentially idle in my testing. I didn't do anything else with any of my disks what it was processing.

This might be group specific too. It could be some pattern in the of groups that B10 tries harder to group than B7 does too.

[Lurker]

I am on B11 right now and thought I was having a header issue, but it seems that there is a real discrepancy for how fast the .gz file gets imported depending on the group. I tried several ~40 MB .gz files and some took around 30secs to process but others it took as long as 13 min each.

Is there a way to show an indicator or maybe an entry in the log window that newsbin is processing header files in the background? I thought I was having issues with header downloads not showing up but apparently it was just not showing up because it was still processing a bunch of header files I downloaded a couple of days ago and was just slowing down the updating of the group I was reading at the moment. There were 10.5 GB of header files waiting to be processed .

[dexter]

Newsbin displays the backlog of header data to be processed in the Cache display at the bottom of the Newsbin window in the Status Bar. It looks something like "Cache 400/400(0)" when idle. If the number in the parenthesis is non-zero it indicates a backlog of header data to be processed.

[syshog]

Newsbin displays the backlog of header data to be processed in the Cache display at the bottom of the Newsbin window in the Status Bar. It looks something like "Cache 400/400(0)" when idle. If the number in the parenthesis is non-zero it indicates a backlog of header data to be processed.

On RC5 I've had cases where the number in the parenthesis never reaches zero. Sometimes it will be (15) or (5), but there will be no disk activity on the data folder showing it's writing anything to disk other than GuiIteims.db3. I used windows resource monitor to check. The debug log just shows the amount of seconds to check watch list. Also it they a away to disable the constant checking it does of the watch list? I don't have the watch list setup and I don't use it, but there are constant updates to the log and if i could stop that it would save cpu cycles.

I should add that I can restart newsbin and the cache does clear or start processing again.

[Quade]

constant updates to the log and if i could stop that it would save cpu cycles.

I hope you're kidding. The polling of the watch time is pretty inconsequential processor wise compared to the other internal processing Newsbin does. I have removed the message though for RC6 and forward. It served its purpose.

[syshog]

constant updates to the log and if i could stop that it would save cpu cycles.

I hope you're kidding. The polling of the watch time is pretty inconsequential processor wise compared to the other internal processing Newsbin does. I have removed the message though for RC6 and forward. It served its purpose.

I'll look forward to to testing RC6.