Header Import Tying Up Entire Application

[thuvia]

It's going to take me several days at least to import nearly 100GB of RV4 format group data (I like my local searches!). It would be nice if the silly progress bar didn't prevent access to any other part of NewsBin (even the minimize button!!).

[rh]

I did a few, saw how long it was taking, saved the rest for bed time. Woke up, all done. Will never need it again. Glad it's there. No need to put more work into it.

[thuvia]

I wouldn't have minded so much either if it would all be done in one night, but having seen it struggle for hours through a single group, that's not going to be the case for me.

Also, I find this use of a modal dialog to be very sub-optimal UI design, and remember seeing it elsewhere in a later revision of NewsBin 5, but at least there was an option to disable it (post deletion progress).

[rh]

Granted, not an optimal design, but it's a one shot utility - not something you'll use every day.

Besides, you were warned

http://www.newsbin.com/beta.php

"A progress screen will pop up and you'll have to wait for it to finish before continuing."

[thuvia]

Yes, it probably should have said "you may have to wait a few days for it to finish before continuing"...

[DThor]

Personally, I can't be bothered exercising my disk that much, but then I'm not a hard core spool downloader. I just redownload.

Btw, a tip once you're done, I found that with all six of my connections downloading headers every day, the disk IO was getting pretty crazy. There was obviously a lot of thrashing going on, and that simply increases the overall time to insert new spool. I found the time to update them lessened significantly when I only used two connections. I copied that server, and set it to be a slave server, which won't attempt to get spools, so the upshot was I had the same number of download connections, but only two trying to insert spool at a given time, overall time to update spools lessened.

DT

[rh]

Btw, a tip once you're done
DT

Thanks for that tip! I was trying last night to figure out how to ask a question about the thrashing I was seeing - that I could not ever recall experiencing 5.x, who's headers would come down quickly. I decided to use 6.x for a while to see if it was persistent or just a 1st time occurrence.

[rh]

Let me see if I have this right.

Starting with a single server and 10 connections. Change that to a slave with 8 connections. Then create a second master server with 2 connections?

So header downloads will use the master and article downloads with use ?both?

[itimpi]

Correct.

[DThor]

Yeah, sorry I wasn't totally clear.

There was some discussion on irc that perhaps in future this could be folded into Newsbin so that in a server configuration, you could set, say, 'maximum number of spool insert threads ' or something perhaps a little less klingon-ese, to save having to hack around this. Two threads still keeps the IO pretty constant on a decent pipe, frankly I can't imagine why anyone would want more unless they're running a raid or ssd. Hmm, or perhaps if they're running a really slow connection too, I guess.

DT

[Quade]

You can also use "Background Mode" which can download headers with all connections but, uses a single thread to insert the records in the DB. This will also reduce disk IO load.

[rh]

Newsbin Options / Switches / Background Update

http://forums.newsbin.com/viewtopic.php?f=43&t=28453&p=174288&hilit=background+update#p174219

I'll try that too.

Thanks!