NB 6.70B6 seems slow to process headers:

[saintsinner]

Perhaps it's due to an increased amount of spam posting but I've noticed that NB 6.70B6 is quite slow to sort through headers (as in more than 22 minutes to process a.b.mm, a.b.hdtv, and a.b.teevee together)

At the moment of posting it's been 38 minutes since initiating a 16 hour header download, NB reports (4) under cache, is using 1.13GB (and rising) of memory, 7-11MB/s of disk and ranging between 5 and 14% of my cpu (I7-4790) FWIW disk access on the drive that the headers are stored can peak at 120MB+ (when I'm compressing headers for example)

After this processing finally finishes I will go back to 6.6X to see if it's any faster for the same selection of groups, but on the bright side, I can post again

Note: activated verbose logging and bounced the program (before going back to 6.6x) it started with a (3) in cache and is using the cpu and disk to presumably process the already downloaded headers as things propagate into the log I'll post here:
[14:22:55] DEBUG High Speed File Creation Mode: Disabled - "Run as Administrator" to Activate this mode
[14:22:55] HIGH PAR Pausing: Enabled
[14:22:55] HIGH Par2Repair - Multi-core 1.5
[14:22:55] HIGH MessageId Filter: Disabled
[14:22:55] HIGH FileType Version: 2.01
[14:41:28] DEBUG Checking Autodownload: alt.binaries.hdtv
[14:43:58] DEBUG Checking Autodownload: alt.binaries.hdtv
[14:48:57] DEBUG Checking Autodownload: alt.binaries.multimedia


(Note, I don't use autodownload....) sometime prior to 1504 (but after 1448) it finished as cache now shows (0) and the group list shows a combined total of 92798 New files under the main screen (before choosing to view the groups) so it *might* be down to so much spam being posted, but that sort of delay is still ghastly. Will try 6.6X when I grab more headers in 7 or 8 hours.

[saintsinner]

Ahhhh, 6.60 is also insanely slow these days (it took 19 minutes for "30,674" files) so perhaps it is in fact all the useless spam posts, but really, should 64 bit newsbin be a bit faster even with so much crap to sort through?
Apparently when it hits checking autodownload it's finished processing the headers...
log:
(edit: snipped after the fact)
[20:10:55] DEBUG Download: Idle Connection Closed:usenetbucket
[20:10:56] DEBUG Download: Idle Connection Closed:usenetbucket
[20:10:56] DEBUG Download: Idle Connection Closed:usenetbucket
[20:19:36] DEBUG Checking Autodownload: alt.binaries.hdtv
[20:24:05] DEBUG Checking Autodownload: alt.binaries.multimedia
[20:29:11] DEBUG Checking Autodownload: alt.binaries.teevee

[Quade]

It's a bunch of data to grind through. I've been recently looking into multi-coring the process but I'm not sure how much speed improvement that would and whether most PC's are up to the load. The goal is to feed them as quickly as possible without impacting the performance of the PC.

I suspect I'm running into the limits of the DB.

[saintsinner]

Ahhhhh, yeah that would explain it Another option is instead of having one giant database in each group file for all headers, you go (back?) to having databases based on dates (for example, monthly) that way unless someone is searching through or loading old headers there's no reason to touch headers from more than say 30 days old if you went monthly. (Edit: or allow manual separation of the headers, but that's a lot more effort for you if you also would then allow manual searching of the separated headers)