Aggressive assembly mode not being aggressive.

[kalzekdor]

I'm using v6,62b1. I have options set to always download pars, and Aggressive Assembly Mode is set. From my understanding, once enough blocks are downloaded, download should cease and repair/unrar should begin,

This works sometimes, but other times it does not. Sometimes a download is mostly complete, except for a couple of missing blocks. There are enough blocks to assemble the file. I know this, because manually choosing "Assemble Incompletes" will flush the cache and then begin the repair process successfully. However, the aggressive assembly mode is not kicking in for these cases.

So, when a download is stuck with a couple of missing blocks, my only choice is to manually assemble incompletes and hope I have enough. If I don't, it becomes a veritable pain to repair, even as parts later become available, as automatic retries don't work at that point. Why isn't the aggressive assembly mode doing its job, instead leaving me to guess whether or not the partially complete files have enough blocks?

[Quade]

How many retries do you have set? The default it 2.

It should be assembling incompletes on its own after the second retry. You can make it work more quickly by setting the retries to 1 but that might be too aggressive depending on your news servers. If it's higher than 2, I'd try 2.

[kalzekdor]

I thought the aggressive assembly mode bypassed retries? That's what the option says: "bypass retries and assemble when you have enough PARs".

Isn't that the whole point? If it doesn't bypass retries, then how is it any different from normal operation?

[Quade]

I'll be honest with you. People who don't really know how usenet works, were setting "aggressive" and then bitching when they got partial unrepairable files. Because of that aggressive isn't as aggressive as it once was.

The retries now gate everything. 1 retry means a 10 second delay before it retries and then on the second retry it'll assemble and attempt repair IF it thinks enough of the file is there to repair. If you only have 60% of the file available, it won't assemble because it knows repair is impossible. It probably means your server doesn't have the chunks yet and you need retries to actually complete the download.

Yes. The aggressive option is misleading right now.

There is a more aggressive mode still built in though. If you have enough of the file already downloaded it'll simply assemble and repair. This handles cases where you're just missing a couple chunks from the files. It's still gated by the retries though. I think it's best to leave this control in you hands.

[kalzekdor]

I see. That's unfortunate, but I see where you're coming from. From what I've seen, it'll repair/unrar once it has enough fully complete files, but it doesn't actually take partially complete files into account. I've had instances where just one or two chunks were missing from each of 4 or 5 files. Overall, ~98% of the download was complete, more than enough with parity files. However, only about ~75% of the files were flushed to disk, and it wouldn't assemble unless I clicked "Assemble Incompletes" to manually flush them to disk.

I'll try changing the retries and see if that helps, but most of time chunks are missing is because the server hasn't finished receiving all the files. Lowering the retries will shorten the amount of time a download will wait for the server to finish receiving.

[Quade]

I'll try changing the retries and see if that helps, but most of time chunks are missing is because the server hasn't finished receiving all the files. Lowering the retries will shorten the amount of time a download will wait for the server to finish receiving.

I'd leave it at two. Then if the server is still being fed, each time a new chunk that downloads on a retry resets the retry count. It'll continue to try while the chunks trickle in. Particularly if you download the very newest files and you know your news server is getting the file chunks delayed compared to say search or the NZB, you'll want it to try harder and longer.

I have ideas to improve this in the next major version. It's just too late for 6.62.

[kalzekdor]

I'll try changing the retries and see if that helps, but most of time chunks are missing is because the server hasn't finished receiving all the files. Lowering the retries will shorten the amount of time a download will wait for the server to finish receiving.

I'd leave it at two. Then if the server is still being fed, each time a new chunk that downloads on a retry resets the retry count. It'll continue to try while the chunks trickle in. Particularly if you download the very newest files and you know your news server is getting the file chunks delayed compared to say search or the NZB, you'll want it to try harder and longer.

I have ideas to improve this in the next major version. It's just too late for 6.62.

Ah, that explains some of the behavior I've seen. I thought the retry limit was being ignored. Alright, if it's on the roadmap, that's good enough for me. You can consider this thread closed.

[kalzekdor]

Lowering the retries didn't help.



I set max retries to 2, but as you can see, despite having essentially all the data necessary, it's gone to retry 4 as of that screenshot, and finally retry 6 before I got annoyed and clicked assemble incompletes manually. Moments later (enough time to flush the cache to disk):



It wasn't as though it was finding additional chunks either, which you said could reset the retries.

I really don't get what's going on here.

[Quade]

I agree, that should have assembled and repaired. Let me look at some things. I might want to send you a test version of Newsbin to try. I should know before the end of today.

I'm thinking I might have the assembly threshold set to high. My plan is to expose that setting so, you can set it to a lower threshold.

[Quade]

If you look in the logging tab do you see messages like:

Check Assembly: Assemble Files Current files have been processed - ParChunks > nMissing chunks, assemble and lets pars take over:

or

Check Assembly: No Assembly Percent downloaded hasn't exceeded min threshold for repair:

Really any "Check Assembly:" messages

[kalzekdor]

If you look in the logging tab do you see messages like:

Check Assembly: Assemble Files Current files have been processed - ParChunks > nMissing chunks, assemble and lets pars take over:

or

Check Assembly: No Assembly Percent downloaded hasn't exceeded min threshold for repair:

Really any "Check Assembly:" messages

Code: Select all

[22:15:52]  DEBUG Current files have been processed - Retries have exceeded max retries
[22:15:52]  DEBUG Check Assembly: Assembly Files still seem to be downloading so files won't auto-assemble

There are a couple of those, each time a new retry triggered. There are also a bunch of idle connection closed messages in between. So, despite data not coming through, the connection being open is enough to prevent assembly. But why are new connections being opened after max retries has been reached?

This might explain why this only happens sometimes. It'll only assemble if a retry triggers when connections are closed. Since it reopens the connections after they timeout, that's essentially random.

[Quade]

DEBUG Check Assembly: Assembly Files still seem to be downloading so files won't auto-assemble

This is probably the key one. It thinks some files are still downloading. Maybe the PAR files. I'm still looking at it.

[salxs]

This is my issue as well. Set retries to 2, hopefully that works better. Upgraded to latest in hope finally resolves
incompletes automatically as have to do it manually all the time.

[Quade]

http://www.newsbin.com/downloads/nb662RC2.exe

How about you two giving this one a try?

[kalzekdor]

I think that may have done it.

Code: Select all

[01:18:02]  DEBUG Current files have been processed - ParChunks > nMissing chunks, assemble and lets pars take over
<snip>
[01:18:05]  DEBUG AutoPAR: ScanFile 32 BadFiles : [Download].r07
[01:18:05]  DEBUG AutoPAR: Scan 32 Blocks File: [Download].R07
[01:18:06]  DEBUG AutoPARPlugin: Not enough files or blocks to do anything yet
<repeat for all incomplete files>
[01:18:22]  DEBUG Combining BAD Complete
[01:18:22]  DEBUG PAR Repair: Processing with: a single core
<PARs loaded>
[01:18:22]  DEBUG PAR Repair: There are a total of 834 data blocks.
[01:18:22]  DEBUG PAR Repair: The total size of the data files is 1229973 bytes.
<Find damaged files>
[01:18:26]  DEBUG PAR Repair: 2 file(s) are missing.
[01:18:26]  DEBUG PAR Repair: 5 file(s) exist but are damaged.
[01:18:26]  DEBUG PAR Repair: 21 file(s) are ok.
[01:18:26]  DEBUG PAR Repair: Have Blocks:757 Need Blocks: 834
[01:18:26]  DEBUG PAR Repair: Have 83 Repair Blocks
[01:18:26]  DEBUG PAR Repair: Extra Repair Blocks: 6
[01:18:26]  DEBUG PAR Repair: Repair Blocks: 77 used to repair.


I'll keep an eye out in case it happens again, but I think that may have worked.

[Quade]

Ok great. Thanks for the feedback.

[salxs]

So far looks like MASSIVE improvement! Been assembling incompletes for what feels like years...so used to doing it, but lately usenet provider much much worse...thanks a lot (will update again if have any problems)

[Quade]

That's good to hear. Thanks.

[Questar_99]

Giving this one a try, I have files that NB keeps trying to repair over and over. Typically they are a block short. If I repair them with quickpar then NB attempts another repair and unrars them.

[Quade]

Thanks. I'll check it out.

I run into something like that once in a blue moon. Newsbin thinks the files are good but, in my case a single bit had gotten flipped between the time when Newsbin wrote the file and the time the repair was attempted. I need to verify that it re-scans the files if the unrar fails.