Newsbin Slow (DB Background Handling)

[OMG2011]

Hi Guys,
in the later versions (I guess starting in one of the RCs of 610) Newsbin started to become very slow, grabbing between 5% to 50% of my CPU Time (even if Newsbin isn't downloading anything). I tought it was due to the debug-build, so I didn't take any further notice. But after the release of 610 nothing changed. So I investigated this with "Process Explorer". All the time Newsbin is fiddling with the db3-Files in the Spool_v6 Directory. I remember that I had the same problem before I unchecked the "Insert header data into database in the background" option. I tried to find it in the options, but it seems to be gone. It used to be on the switches pane of the options dialog.
I assume that this background tasks is now always on causing my problems. Could you bring back the option to disable it again?
Switching back to v6.00, until I get a computer with better IO performance.
OMG2011

[DThor]

It's in Options/Switches/Background Update.

I think it's the number of threads processing the spools(which needs to happen no matter what), causing thrashing on your current hardware. Pulling to foreground might help, but oddly I've found the opposite for me - I believe it's limited to 2 threads doing the processing in BG. Not 100% on that, however.

DT

[OMG2011]

It's in Options/Switches/Background Update.

I found that, but it doesn't have the same effect. Or should I say, I doesn't seem to make any difference at all.
With both settings, the same DB activities in the background at all times. (Yes, I restarted NewBin 6.11RC1 several times to check. ).
Still, only 6.00 works for me.
OMG

[DThor]

Hmm, well it does make a difference, in that there are entries in the download list while Newsbin is processing. Once they're gone, the processing is done, with background, the only clue you get is disk activity. If I follow you, though, what you're saying is just two threads of processing are enough to grind your disk pretty heavily. I'm not sure if there's a way to pull that down to 1, or why using the earlier version makes a difference.

DT

[OMG2011]

Hmm, well it does make a difference, in that there are entries in the download list while Newsbin is processing. Once they're gone, the processing is done, with background, the only clue you get is disk activity. If I follow you, though, what you're saying is just two threads of processing are enough to grind your disk pretty heavily. I'm not sure if there's a way to pull that down to 1, or why using the earlier version makes a difference.

DT

Well, I looked a bit further now, an after an update of all headers, 6.00 now also starts to get sluggish. It seems there is a folder which contains 114 RV4 files (all around 20 megs in size) in the SPOOL_V6 folder. Newsbin seems to have a hard time coping with that.
All other folders are clean in that respect. In Process Explorer I see that all db3-files in this folder are all open and active. While I'm writing this the number of RV4 files reduced to 109, so at least there seems to be some progress. I now keep it working. Lets see where this goes. Maybe my impatience (restarting Newsbin with different versions, when it seems to lag) might cause the problem in the first place.
OMG

[itimpi]

That behavior would be normal if you have background processing enabled. In that mode Newsbin downloads headers as fast it can and stores them in the rv4 files, and then works through the rv4 files processing them into the database.

[Quade]

Switching back to v6.00, until I get a computer with better IO performance

DB stuff in all the 6's is pretty much the same. Only the foreground/background stuff has changed. In that background mode is the default now. Background mode means a single worker thread does all the inserting. My machine is pretty powerful and it'll grind some inserting RV4's from 300 different groups. Not sure there's much if any way to make it faster with software changes. The schema is pretty simple in the DB, everything is indexed. I think the problem is one of sheer volume of data and typically slow disk drives. On thing you can do is let Newsbin update once and hour so, there's never that much data to insert between updates.

[OMG2011]

Switching back to v6.00, until I get a computer with better IO performance

DB stuff in all the 6's is pretty much the same. Only the foreground/background stuff has changed. In that background mode is the default now. Background mode means a single worker thread does all the inserting. My machine is pretty powerful and it'll grind some inserting RV4's from 300 different groups. Not sure there's much if any way to make it faster with software changes. The schema is pretty simple in the DB, everything is indexed. I think the problem is one of sheer volume of data and typically slow disk drives. On thing you can do is let Newsbin update once and hour so, there's never that much data to insert between updates.

I'm now on 6.11RC1, the RV files have all been processed (4 Gigs of header data, it took over 6 hours). Speed is fine now with background mode switched off. I don't know how long these files where stored in that directory, but it surely caused the sluggish behavior of Newsbin. The switch from version to version was just coincidentally. Debug logging is a huge help (could have used that earlier...)
Thanks guys!

[OMG2011]

Hi guys,
reviving my old thread here. But I maybe have an maybe simple idea for my problem with a very sluggish computer while any db-handling tasks runs on NewsBin (620ß1, 1423). Is it possible to give the DB thread a lower priority (or maybe a configurable priority like "idle, low, normal, high"). DB handling might take a bit longer, but I wouldn't care, losing control over my computer is worse for me (I'm an impatient guy ).
OMG!