How to auto-assemble incompletes for PAR repair?

[optiman]

I notice that Newsbin does not automatically repair incompletes. I have to right-click on the "downloading files" file, select "assemble incompletes", then the PAR auto-repair process begins. Do I have an option unchecked somewhere? I have aggressive mode unchecked but that shouldn't be the problem...
Optiman

[Quade]

The number of retries determines when Newsbin gives up, assembles and repairs. The default is two retries. It won't do it unless 90% of the file is available.

[Valmont]

The number of retries determines when Newsbin gives up, assembles and repairs. The default is two retries. It won't do it unless 90% of the file is available.

Thanks for the answer. Can the percentage level be variable in a some next release?

Cheers!

[Mannypl]

I have my default retries set at 2 and it still won't auto assemble incomlpetes

[Quade]

What version are you using?

[Mannypl]

Was using v6.56. Have now upgraded to v6.6B9 as I also had that "out of memory" message.

[Quade]

6.60 is more aggressive but still Newsbin won't assemble them if they're really in bad shape (though I'm reconsidering that).

[Valmont]

6.60 is more aggressive but still Newsbin won't assemble them if they're really in bad shape (though I'm reconsidering that).

I'm using an external program, MultiPar, to check if certain downloads really are beyond repair. Often they can be repaired though. Most of the time not, but often it is. You can see it coming, but I forgot what the logic (so to speak) was.

[Co6aka]

Any chance of this getting "fixed" ever? Running 6.72 and it's still happening. I'm able to fix most of the incompletes with MultiPar, that is, after struggling with Newsbin for a while. I don't understand why when a user manually commands a program to do something it can't just do what the user commanded; this has nothing to do with the autopar feature as I have it turned off.

Incompletes move to the "Failed Files" tab, I highlight them there and select "Assemble Incompletes" and they move back to the "Downloading Files" tab where Newsbin seems to be re-attempting to download missing chunks, which fails, and then Newsbin moves them back to the "Failed Files" tab. Rinse and repeat ad infinitum. WHY??? For what reason??? Once I actually get the assembled incompletes most of the time I can fix them with MultiPar, and sometimes I can find a component file or get one from a friend on another USENET host.

[Quade]

Yeah, you can only manually assemble them from the download list. I agree that it's misleading and needs to be fixed.

Dex and I are discussing removing the failed list altogether and just leaving the failed files in the download list until you manually assemble or dispose of them.

If you leave autopar repair turned on, they'll assemble and repair automatically but not unrar. You can also tell it to repair and remove the entry from the list too. If you want it to be even faster to repair, disable "Pause Pars" in the autopar options. It'll repair as soon as it has enough blocks.

[Co6aka]

Yeah, you can only manually assemble them from the download list.

It would be fine if they went to the download list and then assembled, which wouldn't be misleading, instead of re-trying and re-failing and re-moving to the failed list ad infinitum.

Dex and I are discussing removing the failed list altogether and just leaving the failed files in the download list until you manually assemble or dispose of them.

The failed tab serves a purpose; keeps the downloading list from getting gunked up with failed downloads. Sometimes I swap temporary server logins with friends so we can try to clear our failed download lists, and having the separate failed tab is handy. Also, we've made a NZB of all files on our failed tab and then sent each other the NZB.

If you leave autopar repair turned on, they'll assemble and repair automatically...as soon as it has enough blocks.

Not always; too often I've found that I can manually assemble and repair with MultiPar (what NZB seems to think is unrepairable.)

[Quade]

Not always; too often I've found that I can manually assemble and repair with MultiPar (what NZB seems to think is unrepairable.)

Turn on aggressive assembly mode and uncheck "Pause Pars". This is the quickest route to assembly and repair. If you use NZB files, this should be the default mode.

As for the failed list, I'm not sure "reducing clutter" justifies the problems. Like the problem you're having right now. Most people don't even know there is a failed list.

[Co6aka]

Turn on aggressive assembly mode and uncheck "Pause Pars"

Pause pars was already off, so I checked aggressive assembly. Will advise.

Also, before making the above change, I realized that incomplete multipart components fail before everything is downloaded, which doesn't seem right. And, from the incomplete tab, instead of selecting assemble incompletes, if I "download-bypass filters" and then on the downloading files tab quickly select assemble incompletes I can get enough good blocks for MultiPar to fix the lot; it seems Newsbin isn't actually writing out all good data available when assembling incompletes. For example, just before posting this, MultiPar reported "Useless" for a few of the PARs, but after re-downloading and assembling incompletes (again, but as I just described) MultiPar reported damaged blocks in the PARs that previously were declared useless. Methinks something in Newsbin isn't exactly as it should be. In case it matters, I'm running 6.72.4776-64 under Wine-64.

Addendum/Update: Doing some experimenting here... I've just tried downloading and assembling-incompletes three times for the same post, moving the assembled files to another folder each time, and then I ran my dupe checker...which reports different files for all of the same incompletes! WTF??? Something is hosed. (The files that were complete were reported as dupes.)

[Quade]

Methinks something in Newsbin isn't exactly as it should be. In case it matters, I'm running 6.72.4776-64 under Wine-64.

I wonder if you're confusing different failure modes. Sometimes it'll immediately fail the downloads if the spam filter kicks in or the duplicate checker kicks. Sounds to me like you're getting partial downloads because Newsbin is failing it for non-download related reasons. The fact the files are different with each assemble suggests they are failing for non-download related reasons.

Might be worth disabling the duplicate detector (advanced options) and spam filters and see if your symptoms change. If you're leaving autopar off, you can't even watch the actual PAR block by par block download progress so, it's hard to know what you're seeing or if the downloads are any good.

How about setting a baseline? 2 retries, aggressive with autopar enabled but perhaps unrar disabled. Disable the filters then try some downloads and see what you see. The par blocks with be listed next to the size. How many you need, how many you end up with.

[Co6aka]

Sounds to me like you're getting partial downloads because Newsbin is failing it for non-download related reasons.

"Error" message for all failed files is "Download Failed: File retry count exceeded"

Might be worth disabling the duplicate detector...

Dupe detector was enabled, disabling it now. Spam filter is and was disabled, leaving it disabled. Retries was set to 3, setting it to 2 now.

AutoPar had been disabled, enabled it yesterday along with aggressive, and behavior is now slightly different: Appears that autopar is trying to do something but MultiPar-repairable incompletes still fail. When I re-download from the Failed Files tab (ctrl-Y) it seems I have a little more time on the Downloading Files tab to highlight and assemble-incompletes before they're moved back to the Failed tab, which is better.

I've also realized that sometimes Newsbin indicates files to be complete on the server, but after downloading they have missing chunks, and when several are missing they fail. I'm going to try another binary downloader to see if it can download without errors. Also, over the weekend I'll try to get a friend to log in to my server and see what happens; he's running Newsbin under Windows-7-64.

Another thing: With autopar disabled, why aren't files under the "+" downloaded when I highlight a bunch that includes a "+" grouping? For example, in a music group, under the "+" is a nfo, a par2, and several par2 volumes, and then outside the "+" grouping are the flac, png jpg and nzb files... If I highlight them all and select "Download" or "Download-Bypass" none of the par2 files (under the "+") are downloaded, so I have to go back and expand the "+" and select them and download. Sometimes a jpg or png or nfo will be under the "+" and it won't be downloaded either, until I go back and expand and download. With autopar enabled they download as expected.

FWIW, another unrelated issue, a UI thing: I can't expand "+" groupings by clicking on them; I have to highlight the row and use the right arrow key. And to close, I have to similarly use the left arrow key. Same everywhere in Newsbin.

[Quade]

I've also realized that sometimes Newsbin indicates files to be complete on the server, but after downloading they have missing chunks, and when several are missing they fail. I'm going to try another binary downloader to see if it can download without errors. Also, over the weekend I'll try to get a friend to log in to my server and see what happens; he's running Newsbin under Windows-7-64.

It's pretty easy to tell from the logging if the server isn't returning chunks. You'll see errors like "430" meaning the server says it doesn't have parts of the file. It mostly depends on your server and whether the files have been DMCA'd or not (or are too old for the server). When Newsbin says a set is complete, that just means it has all the information it needs to download. If the server won't deliver the chunks, there's nothing it can do about it. You can test the downloads before you attempt to download, look in the right click menu. I believe a manual "retry download" on the test will turn the test into a download.

Are you seeing the PAR blocks count up with auto enabled? With autopar enabled, and some pars downloaded, it'll seldom fail files. Files with pars are retried extra amount. The thing is aggressive assemble will download, retry and them assemble with the assumption that you'll have enough blocks to repair. If you servers are shitty and you're getting a bunch of 430's then there may simply not be enough there.

These days I use Usenetserver as my primary. It's the only server that seems to really have the retention they claim they do. Both Giga and Astra are over-claiming what they can actually deliver.

"Download-Bypass" none of the par2 files (under the "+") are downloaded, so I have to go back and expand the "+" and select them and download.

Uncheck "Pause Pars" in the autopar options. You might have to enable it to uncheck it.

[Co6aka]

Tried lots of stuff, but Newsbin is still failing repairable sets. Just now manually ctrl-Y-ed from the Failed tab and manually assembled on the Downloading tab, AND THEN Newsbin began repairing the set, which it did: Newsbin is failing set component files BEFORE auto-repair has a chance to go to work, so auto-repair never happens.

"Download-Bypass" none of the par2 files (under the "+") are downloaded, so I have to go back and expand the "+" and select them and download.

Uncheck "Pause Pars" in the autopar options. You might have to enable it to uncheck it.

Pause Pars was/is unchecked; they don't download.

[Quade]

Might be time to wipe and re-install then. I'm not having any issues like this.

I'd leave all the options set to default except the unrar and download paths after the re-install and then try some downloads without changing any other settings.

[Co6aka]

Might be time to wipe and re-install then.

Same results. It seems the root of the problem is Newsbin failing incompletes before the entire set is downloaded. Could this be an out of sync multithreading issue?

[Quade]

I don't rightly know. Why can I download 100 gigs in one sitting and you can't?

With a fresh install, I assume you have it set to pause PARS and with autopar enabled?

[Co6aka]

With a fresh install, I assume you have it set to pause PARS and with autopar enabled?

Yes.

I've just tried something and the result is quite interesting: With a set I know has incompletes that were being failed, I "manually" (highlight and ctrl-Y) downloaded all of the pars and then the rest of the set, and as soon as Newsbin finished downloading autopar jumped into action and repaired the download. The pars need to be downloaded FIRST, before the rest of the files. This is with Newsbin 6.72.4776-64 on Wine-64 1.9.20 on Xubuntu 16.04-64.

[Quade]

The pars need to be downloaded FIRST, before the rest of the files. This is with Newsbin 6.72.4776-64 on Wine-64 1.9.20 on Xubuntu 16.04-64.

If you add compacted sets to the download list, this is automatic. Still sounds like you're doing something odd like expanding the sets with the + button and adding the individual files instead of adding the entire compacted block.

[Co6aka]

Still sounds like you're doing something odd like expanding the sets with the + button and adding the individual files instead of adding the entire compacted block.

Actually, that's what doesn't work. I have to expand the sets and download the pars first, then download the rest of the set, which is a small nuisance because clicking on the "+" doesn't work under Wine (only highlight and right-arrow.) Also, again, sometimes set components are being failed when (by wrestling with Newsbin to download all it can) I can get enough to repair manually with Multipar.

Could you make a new option to just automatically assemble incompletes after a certain number of retires, instead of failing?

[Quade]

If you pause, add a set to the download, then hit the + on it in the download list, you should see that the small PAR2 is at the top of the set of files. Is that small par failing to download? The only time it'll really fail a download is if the there are no PAR files. In your case it sounds like the small PAR2 isn't downloading.

What I'd probably do is remove the failed list altogether so, they'll turn red but it'll be up to you to manage the failures. Then you'll be able to retry or assemble to your hearts content.

[Co6aka]

you should see that the small PAR2 is at the top of the set of files

Typically it isn't; typically it's a .txt .nfo .sfv or even the .rar files making up the set (with the .par below, and the repair volumes below the .par file.) Also, in music groups, typically the only files in the collapsed block are the .txt .nfo and .par files.

Is that small par failing to download?

It's downloading, and watching the download progress it seems to download in the order it appears, which is after the .rar files.

The only time it'll really fail a download is if the there are no PAR files.

Perhaps, because often the .rar files (and others) are download before the .par files. Also, as I wrote before, it seems that when there aren't enough repair blocks ALREADY downloaded it fails, so it actually works when I manually download all of the pars first.

What I'd probably do is remove the failed list altogether so, they'll turn red but it'll be up to you to manage the failures. Then you'll be able to retry or assemble to your hearts content.

Could you also add an option to just automatically assemble after a configurable number of retries, and maybe instead of the filed tab have a failed files logfile? Seems that about half of the time Multipar is able to repair what Newsbin declares unrepairable.

[Quade]

Could you also add an option to just automatically assemble after a configurable number of retries, and maybe instead of the filed tab have a failed files logfile? Seems that about half of the time Multipar is able to repair what Newsbin declares unrepairable.

I can consider it. Most people won't want this so it might be a hidden option.

repair what Newsbin declares unrepairable

Just to be clear, if they end up in the failed list, it's because Newsbin didn't see a par for these files. It's not because it think's they're unrepairable. Unrepairable but par containing sets remain in the download list.

To fix your specific issue, maybe I should automatically move all PAR files to the top of any sets of files you add to the download list. With compacted sets, containing only rars and pars, it's not needed. Maybe it is for loose files.