Lockups / Loading download lists?

[Chris1701A]

I'm currently using version 6.42 build 2146 and I've been having a couple issues lately.

For some reason I've been building up a considerable list of files in the download queue, at one point there was over 1,000,000 records. At times Newsbin appears to become unresponsive, the main program Window becomes white (although I can still see the groups and download files lists) and if I hover the mouse over the program window all I get is the hourglass (or in my case the spinning circle). I've left it in this condition for as long as six hours and the problem did not resolve itself in that time, during that time I've looked using Process Explorer and Newsbin seems to be using anywhere from 50% to 75% of CPU 0 core 0.

Since I can't seem to close the program normally I have no choice but to terminate it. Previously when I had over a million records and restarted Newsbin it would take over six hours for "Load the download list" to complete and downloading of files to start. I gave up trying to complete that download list so I renamed the downloads.db3 file and started again. Everything seemed to work fine for a day but the download list continued to grow faster than files were downloaded, when the list grew to over 545,000 files I tried bumping up my connections from 8 to 10 (the max allowed by my usenet server) and for a while it seemed like it was growing much slower and in some instances getting a little smaller.

I left it running overnight and it seemed like it was still running in the morning but the files list had grown to over 637,000 files, I usually delete files from the download list marked in red (that never seem to complete) at intervals and when I marked a couple hundred and hit delete and then said Yes to the dialog Newsbin again went into this unresponsive state. I left it sitting like that for more than an hour but it never resolved itself.

Any idea's or advise on how to keep this from happening?

- Chris

[Chris1701A]

Sorry, one other quick note: I had recently started using Newsbin again after a long hiatus and I believe that I started with the previous stable release 6.41 and I didn't notice this problem with that version.

- Chris

[DThor]

Well, the obvious answer is "don't push your system so hard", and I'm not at all trying to be a smart ass about it. If by a record you mean a file, that is indeed a crapload of files. If you mean a million *chunks*, I. E. actual individual usenet posts that are virtually combined by newsbin to make a file, that's not that much, though. I'm guessing you mean a million files, though. That's a whole lotta usenet.

Things to consider:

What's your hardware? You mention a core, so you're running at least a dual core. You might scoff but plenty of times I've seen posts by users complaining their 5 year old dual core laptop can't handle half a terabyte of data. So, just so we know the baseline...

Your disk is probably the bottleneck, however. Much of newsbin is IO bound, so throwing a lot of posts at it might be choking.

Ensure newsbin isn't in the middle of updating the spool files when you're doing big load operations like this. Check the cache at the bottom of the program and report what it says. If there's a number in brackets, newsbin is inserting spool and this will hammer your disks. Adding a huge download on top of this will thrash like mad. Try letting all disk IO finish before doing anything.

You might well have a state of the art platform, but it's useful to know that before moving forward. I've never even tried to add a million files all at once, to be honest. You might be getting bleeding edge behaviour. Quade has torture tested newsbin, though, he might have some comments about throwing that much around.

DT

P. S. Just got your added note. It's unlikely the version change would be directly related, I'm guessing it's not apples and apples. I guess I'm wondering what exactly 'didn't have a problem' means. The same download queue loaded... immediately?

[Quade]

It shouldn't do that but, by the same token 500,000-1 mil files is alot of files.

My first thought is to not feed 1/2 mil files into the download list.

What kind of machine is this? I wonder if you're just hitting some natural limit of the machine?

[Chris1701A]

This is a pretty good computer, Socket 1366 i7-960, 12GB ram, both my boot and data drives are Western Digital Black drives, plenty of free space on both.

After I initially got started again with Newsbin I switched most newsgroups over to auto mode and they've been running that way for a couple months, what I've outlined has only been an issue within the last week.

In regards to "If you mean a million *chunks*, I. E. actual individual usenet posts that are virtually combined by newsbin to make a file, that's not that much, though" I'm talking about posts listed in the "Downloading Files" tab. From what I was seeing in the failed files and files list tabs 95% of those million files, then the 637,000 files were being failed as duplicates by the dup detector so in actuality out of 155,000 posts in the download list (which downloaded before the program lock) probably less than 3500 files were actually downloaded and the rest were failed as dups.

As soon as I can gain access to the program I'll switch off "Auto Mode" and try and pace this a little. But just to give you an idea, I had to terminate Newsbin around 8am this morning and restart it and it's now 3pm and "Load the Download list" is about 90% completed so that's 7 hours I've been waiting for that to finish so I can try disabling auto mode.

- Chris

[itimpi]

Every item in a Download/Failed files list will be consuming RAM regardless of whether the file that underlies that entry is a duplicate. With the number of Download list entries you are talking about it is not surprising that RAM usage is high.

[Chris1701A]

Now that I've turned off auto on all groups things seem to be running a little smoother and I'll just have to pace catching up on these groups so as not to overload newsbin.

One other odd thing that I've noticed and I've seen this before several times, while in the process of downloading posts suddenly start showing a red circle with an exclamation and all columns which previously had text in them i.e. the Download Description, Download Path, Size, Status and progress suddenly are blank and as I watch this starts happening to more and more posts going down the download list. If I exit newsbin and start it again everything goes back to normal and downloads continue normally.

- Chris