CPU running 90-100% when getting headers

[bruce73]

Sorry if this has been brought up before, but is it normal for the CPU to be running so high during header downloads? This is on an XP/SP3, P4/2.40 GHz machine with 1GB RAM. NB version 6.01B1 (1127).

[Quade]

In the options there's a mode called "Background Update". You might enable that mode. What that does is spread the work of the header downloads and insert over a longer period of time. The problem is that in 6.0, you can't see when it's done. In the upcoming 6.01B3, there's a progress indicator showing when it's done.

[bruce73]

Thanks, found it. So, the high CPU usage is not unusual then? BTW, while I was in Options, I noticed that in the Filename Options window, the "Advanced ... " button doesn't do anything. FYI.

[Quade]

It's SO Advanced, I haven't written it yet.

The high CPU usage is a function of downloading and trying to insert data into a database in real time. BackGround Update, defers the insert. A worker thread does the insert in the background which is lower load than 20 threads in the downloaders doing it. In the 6.01B3 that's currently posted here in the forum. An indicator next to the cache entry on the status bar shows my many blocks of records still need to be inserted.

Another option would be to add a second server, copy of your existing header server, give it 2 connections say, and make the primary server a "slave". In that way, you never use more than 2 connections to download headers. If you do this, set the "Download Age" short, say 1 day so, it's not re-downloading many headers. Alternatively you could make your main server entry, 2 connections and then add the rest of the connections to the copy and make the copy a "slave" so it doesn't download headers.

[RayMark]

I have never tried to download with V6.0 headers of a group that contains a large number of very small posts before.
But now I deliberately tried a picture group with a large number of headers (> 8,000,000) with 6.0.1 beta 3:

Downloading from giganews

Frankly, I had no patience to finish downloading those headers, but it looks like this:

- downloading for a few seconds with low (2%-4% or so) CPU usage
- and than stalls at 0 speed for a long long time with 25% CPU usage (with a quad-core CPU = 100% with a single core)
- and that repeats again, a short burst of downloading and a long long pause.
99.99% of the time speed is 0 and CPU usage is 25-26% - it is almost impossible to catch the moment when it is not 0.

It seems that the headers are downloaded zipped in some chunks.
I also tried to disable XFeatures - still the same picture.
I did not try the background mode yet. Perhaps it is good for such groups, not sure, maybe worth trying.
Perhaps I just chose a really large group and I should not have expectations to download headers quickly.
But it is pretty demoralizing - 0 speed and high CPU usage all the time, and no progress to speak of.
I think, with V5 the header download was much more continuous.

With multimedia groups things look very differently - a steady progress and almost never 0 speed pauses.


Anyway, no big deal as I did not even try such groups with v6.0 until now, but interesting if a smoother/faster(?) header downloading can be achieved with some lines in ini file, etc.
Probably it is more important the overall required time, not smoothness.
Interesting if the background mode is slower of faster? Probably it should be even slower.

[Quade]

I did not try the background mode yet. Perhaps it is good for such groups, not sure, maybe worth trying.

Maybe you should. I sustain 450 Mbps header downloads to giganews when in background mode.

[RayMark]

Yes, the background mode is working nicely - continuously downloading, CPU usage increased from 25% to 25%-35%, but it seems that the overall time needed to download headers is much shorter.
In fact, many many times shorter.

So I am trying it on a large picture group.
Would this background mode be useful for multimedia groups as well, where foreground mode seems to be working OK?

UPDATE: the background mode seems to be working well with multimedia groups too.
Probably worth keeping the background mode always on.
The question is - why it is not a default mode and when it is useful to switch it off?

UPDATE 2: actually, it is not so clear if it is better or worse with a.b.hdtv
I will have to try it on and off with that group several times, I think.

Does switching this mode on/off in the middle of header downloads takes effect on the headers that are currently being downloaded?

[Quade]

The downside is the posts are slower to show up in the post lists.

If you try 6.01B3, you'll see a number next to the cache indicator. That number represents the number of blocks of headers that are currently being inserted. The total amount of effort to insert header is the same between the two modes. Background mode just spreads it out over a longer time.

[RayMark]

yes, I see something like:

Cache: 180/200(53)

Interesting, is it dangerous to switch the background mode on/off during header download, and does this change even takes effect on the headers that are already being downloaded?

[mho]

The question is - why it is not a default mode and when it is useful to switch it off?

Apart from some strange behaviour when still inserting headers, and it [used to] being hard to know when it was done, it was also a lot slower for some of us. (It used to be default when it was first added.) I guess it depends on which group you have and probably speed of disks, etc.

- mho

[RayMark]

I am trying the background mode for the 1st time.
I assume, I will still have to wait for the header download to really end even after the header download ends....
The progress indicator is a bit misleading in the background mode then.

[RayMark]

I have now finished downloading that large picture group.
Watching the cache indicator now:

180/200(75)
180/200(74)
180/200(73)
180/200(72)

I got the picture
For the number to decrease by 1 it takes about 1 minute, so 1 hour and 10 minutes more to wait.

With the multimedia groups (I am currently updating a.b.hdtv and a.b.hdtv.x264 - with a separate instance of NewsBin) - this number stays in the 0-3 range - when 0, then simply disappears.
So clearly, (almost) no additional wait will be needed.

[Quade]

There's only one importer which means the picture group can hold up the other imports...just say'in.

Might be worth compacting that picture group and see if you can speed it up. Large multimedia groups are actually much easier on the DB than picture groups because the number of writes to the DB is an order of magnitude higher for pictures.

[RayMark]

I don't know what can be compacted - i just downloaded that group with "Dowload All Headers".

Anyway, I am updating multimedia groups with a completely separate instance of NewsBin, so those groups are not affected.
In fact, already finished.
It is really not clear if it was better or worse using background mode for the multimedia groups.
For the picture group - the background mode is clearly better, even with those 75 additional minutes, still much much faster, maybe 10 times faster.

[bruce73]

Quade, I created another server like you suggested and am getting headers now, but it's a slow go (I'm doing it with 5 connections instead of the usual 10, and I disabled the background update). CPU usage is all over the place, but it's probably averaging about 50%. It's an interesting trick, though (I didn't know you split a server like that).

But maybe enabling background update it the way to go -- it's definitely faster and the CPU usage is ~20%. So I'm assuming that files can't be downloaded until everything is inserted in the DB, correct?

Also, can you explain how to read the Cache numbers? I see 175/200, then (1) pops up. Not sure what this means.

[Quade]

If you don't use background mode and have CPU issues, I might just use 2 connections. I've taken to using background mode myself. It's much better now that B3 has a progress indicator.

[Moondawgie]

I did not try the background mode yet. Perhaps it is good for such groups, not sure, maybe worth trying.

Maybe you should. I sustain 450 Mbps header downloads to giganews when in background mode.

Wow!

Who is your ISP; What kind of an internet connection do you have; And, can we talk them into providing service to Southern California?

[Quade]

I have a 100 Mbps comcast connection. Remember Giganews and Astra both support compressed headers so, you should be able to download headers faster than your connection.