Fast download slow par check

[Jeru]

I've been using Newsbin for quite some time now and it used to work just fine.
I recently did a reinstall and after that it seems that the par check is horribbly slow, especially when I load in multiple NZB's it almost fully checks them, but then seems to wait untill all other nzb par checks are done too, and then actually finishes the check and starts repairs if needed.

I've set these under performance:
ChunkCacheSize=4048
MemCacheLimit=4048

and set the performance in normal settings to high powered pc(which didnt change anything for me related to this problem)
I've set up the main download folder to an ssd, then download folder for downloads to an HDD(used to have this as SSD, but tried this hoping it would fix it, but didnt), and unrar folder to a third HDD.

During par check no CPU is being used, just RAM. Which seems kinda strange to me.
I run 6.72 build 4776 with win 10.
Am I missing a setting or something?

[Quade]

1 - I'd probably try the latest 6.73RC

2 - You might want to enable aggressive mode in the autopar options and make sure the retries are only 2

3 - You might want to tell Newsbin to stall the download while unraring and repairing (it's in the performance options). You certainly can download and assemble files are the same time but doing that, particularly on a spinning disk will slow down assembly and scanning the files.

I think it doesn't sound like par speed is the issue. Instead I think the download is getting way ahead of unrar and repair. Having a monsterous cache is not pointless but you hit a point of diminishing returns. I'd probably set it to 1000 or less.

PAR scanning happens as the files are assembled so, while not free, it's not that expensive since the data is already wizzing by for the assembly. I suspect your real issue is trying to download and repair simultaneously. If you connection is really fast, the downloaded chunks will backlog and you'll see them finish assembly well after the download completes.

How fast is "fast" download?

I've set up the main download folder to an ssd, then download folder for downloads to an HDD(used to have this as SSD, but tried this hoping it would fix it, but didnt), and unrar folder to a third HDD.

You mention the download folder twice. Best download performance is likely to happen when the data and download folder are on SSD. Then you can set the unrar folder to some other driver, SSD or not. If you're not using a local internal disk for the download path, you're probably crippling unrar and repair performance.

[Jeru]

1 - I'd probably try the latest 6.73RC

2 - You might want to enable aggressive mode in the autopar options and make sure the retries are only 2

3 - You might want to tell Newsbin to stall the download while unraring and repairing (it's in the performance options). You certainly can download and assemble files are the same time but doing that, particularly on a spinning disk will slow down assembly and scanning the files.

I think it doesn't sound like par speed is the issue. Instead I think the download is getting way ahead of unrar and repair. Having a monsterous cache is not pointless but you hit a point of diminishing returns. I'd probably set it to 1000 or less.

PAR scanning happens as the files are assembled so, while not free, it's not that expensive since the data is already wizzing by for the assembly. I suspect your real issue is trying to download and repair simultaneously. If you connection is really fast, the downloaded chunks will backlog and you'll see them finish assembly well after the download completes.

How fast is "fast" download?

I've set up the main download folder to an ssd, then download folder for downloads to an HDD(used to have this as SSD, but tried this hoping it would fix it, but didnt), and unrar folder to a third HDD.

You mention the download folder twice. Best download performance is likely to happen when the data and download folder are on SSD. Then you can set the unrar folder to some other driver, SSD or not. If you're not using a local internal disk for the download path, you're probably crippling unrar and repair performance.

Fast download is gigabit speed so around the speed of 120Mb/s so the limit of a normal hdd. All disks are internal on this pc.
How do I set the standard amount of retries to 2? I think its already set to 2 in Basic settings tab.
It's just strange that I've not had this problem before, and only seems to happen after the reinstall. I'll also try upgrading to the newer version.

You can delete my other topic btw, I didn't see that it needed to be approved the first time I tried to post this problem.

[Quade]

Fast download is gigabit speed so around the speed of 120Mb/s so the limit of a normal hdd

120 Mb/s isn't particularly fast. 120 MB/s is pretty blazing fast so I think we need to clarify big B or little b. Bytes or bits. If you really are getting 120 MB/s then my "download getting ahead of assembly" theory is probably correct.

It's just strange that I've not had this problem before, and only seems to happen after the reinstall. I'll also try upgrading to the newer version.

Ideally all of your downloads will be perfect. If that's the case then PAR processing doesn't actually do anything. On the other hand if you find Newsbin is repairing every download then this will crush your performance. With a decent server(s) and proper reliable setup, you should be getting mostly perfect downloads. If everything has to be repaired, there's a problem somewhere. It sounds like there's a decent amount of repairing going on.

[Jeru]

Fast download is gigabit speed so around the speed of 120Mb/s so the limit of a normal hdd

120 Mb/s isn't particularly fast. 120 MB/s is pretty blazing fast so I think we need to clarify big B or little b. Bytes or bits. If you really are getting 120 MB/s then my "download getting ahead of assembly" theory is probably correct.

It's just strange that I've not had this problem before, and only seems to happen after the reinstall. I'll also try upgrading to the newer version.

Ideally all of your downloads will be perfect. If that's the case then PAR processing doesn't actually do anything. On the other hand if you find Newsbin is repairing every download then this will crush your performance. With a decent server(s) and proper reliable setup, you should be getting mostly perfect downloads. If everything has to be repaired, there's a problem somewhere. It sounds like there's a decent amount of repairing going on.

120MB/s it is yes, sorry. The problem with your theory is though that it hasnt been a problem before, with the same download speeds and the same providers.
Well I'm definitely not getting perfect downloads, most of them need to be somewhat repaired, but all downloads will be par checked first.

Are there any other settings that I can look at? I changed the cache and the other one to around 1000 now.

[Quade]

The problem with your theory is though that it hasnt been a problem before, with the same download speeds and the same providers.

1 - Prior versions automatically stalled the download while unraring/repairing. Since at least 6.72 and probably a couple versions before the default is to download and repair/unrar at the same time. It's optional.

2 - If you weren't repairing as much before, that would explain this too.

Well I'm definitely not getting perfect downloads, most of them need to be somewhat repaired, but all downloads will be par checked first.

Most of what I download doesn't need repair. If most of your files need repair this could easily explain why your unrar/repair performance has gone bad. Repair is an intensive process. Bad downloads will stall processing while it waits for retries to expire. If you finish a set of downloads, and watch the downloads are sitting there with the retry counters counting down, you knew Newsbin is doing retries.

Sounds like you're observing the downloads sitting there so, likely the par counts are showing missing blocks meaning that retries are going to happen a couple times before it starts to download the pars an attempt a repair.

Assuming you don't see 430 errors in the logs during download, this suggests there's something up on your PC causing bad files. The most likely culprit is your security software. It could be bad RAM or too much overclocking too. If you're seeing 430 errors, this means the server is unable to deliver parts of the files.

Probably the next step is to look for 430 errors in the logs during download.

[Jeru]

I don't get any 430 errors.

This is for example what the par status list looks like

http://imgur.com/XlZ3f8y

I've done everything you said. But it seems to me that it has something to do with the newer builds.

[Quade]

I've done everything you said. But it seems to me that it has something to do with the newer builds.

Could be lots of things.

Just from your image, I can see these files all show incomplete PAR counts and all show that some files are still downloading. (The DL in the details sections) I'd set it to pause download during unrar and repair and see if your symptoms change.

[Jeru]

I've done everything you said. But it seems to me that it has something to do with the newer builds.

Could be lots of things.

Just from your image, I can see these files all show incomplete PAR counts and all show that some files are still downloading. (The DL in the details sections) I'd set it to pause download during unrar and repair and see if your symptoms change.

That does kinda help, but in the big scheme of things doesnt speed it up that much. The pars still seem to be stuck 1 or a couple of pars away from finishing the normal check before maybe needing to repair. Thats what confuses me the most. It's all there for the check, but it doesn't seem to prioritize to finish the checking before moving on.

[Quade]

but it doesn't seem to prioritize to finish the checking before moving on.

I think you might have the wrong idea about what it's doing.

By default PARS remain paused till they're needed. When the download completes and it doesn't have enough PAR blocks, it pauses and waits till it's time to retry then it retries a couple times. One the retry expires it'll start to download PARS (assuming they're complete or it retries the PARS too).

I think the delay you're seeing is just that it's in retry mode. Typically a minute or two between retries.

At your download speed, you might want to uncheck "Pause Pars" so they download with the files and repair will kick off as soon as you have enough PAR blocks.

The combination of "Aggressive Mode" and unchecking "Pause Pars" is the quickest way to get the downloads to repair and assemble incompletes. Some people prioritize bandwidth of the downloads over the speed of the assembly and repair.

[Jeru]

I agree that I don't fully understand what the proces is of whats happening.
I'll try that, thank you. And also thanks for all the other suggestions so far.