30,000 files in Chunks Folder; Newsbin very slow

[ppan]

When I download a lot of files, I often see the speed at 0 Kbps but files are slowly being added to my download folder. It can take several hours for the files to be added. Looking in my chunks folder there are over 30,000 chunk files (many, but not all, are 247 KB). The number is slowly decreasing as files are added to the download folder.

I assume that everything is so slow because there are so many files in the chunks folder!?
Why are there so many chunks there?

BTW, I have my data folder (and therefore Chunks Folder) on a different drive from the download drive. Would things speed up if I move the data folder to the download drive so that they chunks can be assembled on the same drive (it is much faster to move files on the same drive than to move between drives)?

[Quade]

I assume that everything is so slow because there are so many files in the chunks folder!?
Why are there so many chunks there?

You have it backwards, you have a bunch of chunks in the folder because unrar and repair are slowing everything down but the download continues. If you tell Newsbin to stop downloading during unrar and repair (which will make both of them faster) you won't collect chunks.

Downloading to one drive and unraring to another is probably fastest.

[ppan]

[
You have it backwards, you have a bunch of chunks in the folder because unrar and repair are slowing everything down but the download continues. If you tell Newsbin to stop downloading during unrar and repair (which will make both of them faster) you won't collect chunks.

Downloading to one drive and unraring to another is probably fastest.

I have already told Newsbin stop downloading during unrar and repair. I am also not unraring (only PAR2 repair).

1. I still have a lot of chunks. Currently Newsbin has 652 files in the download queue, but nothing is downloading. Many, if not all, of these files are already downloaded. Not sure why the files are not removed from the queue ?
The chunks folder has 12,802 chunk files. Why? Should I delete them?

2. The question was if it would be better to have the data (chunks) folder and download folder on the same drive (not the unrar folder). If I understand correctly the chunks are being assembled and moved from the chunks folder to the download folder. It is faster to move files on the same drive than between drives.

[Quade]

but files are slowly being added to my download folder. It can take several hours for the files to be added.

1 - If this is what's happening then it's because your machine fell behind the download speed.

2 - If it's repairing files, its just as or even more intensive than unraring.

Should be easy to tell if it pauses during repair. Not sure I've ever tested that mode with just repairing though.

As for the chunks location. I don't know if it matters. Because you fell behind, after each file assembles Newsbin has to determine if it's any good. If it's damaged then the scan process is much slower than if the file is perfect. If you find yourself repairing mostly everything. I wouldn't be shocked by the slowdown. I seldom have to repair anything. You should note how much you're repairing. If it's every file, you have a problem to solve.

[ppan]

I am really confused over what is going on. Newsbin is extremely sluggish now and consumes 25% of the CPU (4 cores).
In the download list I have, for example,

1. A single rar file ABC.part01.rar;
Size 5.3 MB/53.2 MB
Status [Par: 0 of 1719] D:1 DL:1 Retry 30 N:5073

2. A group of 8 PAR2 files (7 paused):
Size 337 KB/31.7 MB
Status [PAR 0 of 771] D:0 DL:7 Retry 28 N:1206

What do the status symbols mean and why does Newsbin sit there in an endless loop trying fix the unfixable?
Is there some setting that would prevent this?

(Note again that I am not unraring; only PAR2 repair).

[Quade]

You're a heavy downloader so you might have to do cleanup that a normal person doesn't have to do. Things like the signature and downloadmarker files.

You're an outlier so you're probably running into issues most people never see.

As for repair, as long as more PARS download, Newsbin will continue to try to repair. I'm surprised those pars didn't go to the failed list already with those retry counts. Unless you actually see it repairing, I'm not sure you can assume that repair is the load.

[ppan]

1. What does "Status [PAR 0 of 771] D:0 DL:7 Retry 28 N:1206" mean?

2. There is no repair going on because the corresponding rars or PAR2 files are missing. Newsbin just sits there trying. I have set Max retries to 2, but that does not have any bearing on PAR2 repair. Can we have an option for max PAR2 retries?

3. I guess that the only solution is to delete the files from the download list?

4. I don't want to delete the signature and download marker files as I don't want duplicates, and I need to know what has been downloaded (although the later doesn't work so well - more on that later)

[Quade]

What does "Status [PAR 0 of 771] D:0 DL:7 Retry 28 N:1206" mean?

No par blocks meaning no files associated with those pars have downloaded. DL:7 means 7 files are waiting to finish downloading. DL:0 means no files of that set completely downloaded. I think the rest is self explanatory.

I don't want to delete the signature and download marker files as I don't want duplicates, and I need to know what has been downloaded (although the later doesn't work so well - more on that later)

If they get too big, performance will fall. You'll have no choice but to delete them eventually.

[ppan]

1. Retry 28 I understand, but what does N:1206 mean?
2. Can we have an option for max PAR2 retries?

[Quade]

How many retries do you have set in the options? Honestly I don't think this has anything to do with your performance issue. As far as I can tell it's just sitting there idling.

[ppan]

1. Retry 28 I understand, but what does N:1206 mean?

2. As mentioned above, retry is set to 2. As also mentioned above, Newsbin is consuming 25% of my (4-core) CPU while the only thing that is happening is that N is increasing for several files (now at N~ 4,830 for the top three files in the download list)

[Quade]

1200 some odd second before the next retry. it's basically sitting there doing nothing.

25% CPU usage normally suggests it's importing headers. Check the cache line for a number in parens.

[ppan]

1200 some odd second before the next retry. it's basically sitting there doing nothing.

Why is N constantly increasing (it is now at 9030)?

25% CPU usage normally suggests it's importing headers. Check the cache line for a number in parens.

What is the "cache line"?
I am not downloading headers or anything else. There is also nothing in the logging Tab.

--- EDIT---
Newbin became so slow that it was impossible to use. Just trying to opening up the Settings dialogue took 5-10 min.

I deleted all the 50 or so files in the download list (they all had retries > 30, and it took 5-10 min just to get the delete confirmation dialogue!).
After the files were deleted, the CPU went back to 0% and Newsbin became usable again.

There must be some bug that causes Newsbin to go into some infinite loop when trying to repair.

[Quade]

Why are you fixated on repair? Nothing you've shown indicates anything is being repaired. If it was repairing, it would be using 100% (or pretty close) of all 4 cores.

You should try what I suggested earlier. The cache line is at the bottom of the status bar. Where is says "Cache".

[ppan]

I am sorry, but I don't understand what you meant by " importing headers"?

As I wrote above in the EDIT above, I deleted all the files in the download list, and the CPU went back to 0% and Newsbin became usable again.
When I look now, the Cache line shows 200/200 (0). I don't know what it showed before.

[ppan]

I downloaded several new files and the problem returned. Newsbin uses 25% CPU, is extremely sluggish, and very little is happening. Note that I currently have 12,009 chunks and the number is not decreasing anymore. I have turned off Auto UNRAR; only running AutoPAR.

(Because, for unknown reasons, you cannot see inline images here, I have attached links to the images instead):

1. Status of the Files left in the Download tab:
https://i.gyazo.com/dee599515ab2f6aafa1b00f8b5f09e4d.png

2. Example1 - the top set expanded:
https://i.gyazo.com/91b7aaabb1c049d1939106414064f4f8.png
In this example, the small PAR2 file and RAR files are downloaded; the large PAR2 files are paused. If I run Quickpar on the small PAR2 file, nothing is found.

3. Example2 - the 3rd set expanded:
https://i.gyazo.com/4d1de1a7e5c7d302de300d061b4d8f8a.png
In this example, the set only consists of PAR2 files; only the small one is downloaded (the large ones are are paused). If If I run Quickpar on the small PAR2 file, all the RAR files are found except for a small support RAR file (1 block is missing). If Newsbin downloaded one of the large PAR2 files, it could probably be repaired (why doesn't Newsbin download this?).

4. Example4 - the last set
This example is similar to example3. However, in this case, when I run Quickpar all the RAR files are complete (no repair needed).

5. The "cache line":
https://i.gyazo.com/0a03765e8eb545fdb7fdd8935451907b.png

How can I fix this?

p.s. I am not sure when this issue started, but I don't think it happened "in the past".

[Quade]

Sounds like something is getting stuck. You need to try to characterize what file types cause it, tell me and I can try to reproduce it.

[ppan]

Sounds like something is getting stuck. You need to try to characterize what file types cause it, tell me and I can try to reproduce it.

The file types and circumstances were given in the three examples above. Do you need more details?

[Quade]

The file types and circumstances were given in the three examples above. Do you need more details?

Nothing really usable there. I'm not going to download all of usenet to try to figure out what you download that stalls it. If you figure out what post does it and then tell me, I'll try to reproduce it.

How much are you repairing? It should be under 2-5% of your downloads. I asked you that before and you didn't answer me.

[ppan]

How much are you repairing? It should be under 2-5% of your downloads. I asked you that before and you didn't answer me.

I don't know how much is being repaired as I have AutoPAR enabled. Is there some way to find out after the fact? I wonder myself...

[astrax]

I don't want to delete the signature and download marker files as I don't want duplicates, and I need to know what has been downloaded (although the later doesn't work so well - more on that later)

If they get too big, performance will fall. You'll have no choice but to delete them eventually.

Since I noticed similar behaviour on my machine - what do you consider 'too big'? NB usually stalls, when I want to display a group (double clicking on it, opening a new window - no filters). It takes really long for the new downloaded articles to show up, get sorted. I can still scroll through the groub, but I can't add files to the DL, or delete any posts etc. until NB has it all sorted out.
My DownloadMarker.db3 is currently about 575mb and the SIGNATURE.db3 ist about 744mb.

[Quade]

My DownloadMarker.db3 is currently about 575mb and the SIGNATURE.db3 ist about 744mb.

Pretty big.

As an experiment exit Newsbin move them out of the data folder and then restart. You can always move them back if there's no improvement.