6.50 Beta 13, Incomplete files being discarded

[mjpierce]

I've just now tried the 6.50 version (beta 13).

I'm download files with missing parts. Instead of assembling and saving the incomplete file when max retries is reached, Newsbin is just discarding the whole thing.

The file shows up in the "Failed Files" tab and says "[PAR Process] Retry Failure: Ran out of retries". It also shows that it did download about 95% of the file data.

I have AutoPar off.

[Quade]

It's because you disabled Autopar. Newsbin says "no parid, I probably shouldn't assemble the files". It doesn't factor in external processing.

For now, you can add them back to the download list and manually assemble them. I'll have to think about it. It's not a bug, it's doing what I want it to do but, I'll have to think about your usage pattern.

[bobsmo]

Would be very worrying if made to be like this..........almost like NB is being to be made to automated.....

[Shine]

Yes, please change this behavior back so that incomplete files are actually being assembled instead of discarded. The current behavior makes NB completely unusable with AutoPAR disabled (or when no PAR files are available)

Re-adding the files to the download list doesn't work, as it simply re-downloads (!), fails and deletes them again (using beta 14).

Also, while doing so, it fills up my C drive, because it thinks it's a good idea to download all incomplete files to C:\Users\MyUsername\AppData\Local\Newsbin\chunks instead of to the download directory that I specified in the options... can you please fix that as well? My C drive is not large enough to hold Usenet downloads. That's why I specified a different drive in the first place!!

[DThor]

It's rare chunks ever build to the size of the download directory (they are two different things so you shouldn't be expecting them to match by default), but if you're having trouble with this you can change the data directory to wherever you want, which will carry all the data files including spools. You need to manually edit the nbi file to wherever you want it when newsbin isn't running(back up first), move the data over, then restart newsbin.

The chunks folder is a temporary work area that clears out over time. That's unrelated to where you want your downloaded files to go.

DT

[Shine]

I don't want my data directory anywhere else. I just don't want any downloads - not even temporary ones - in the data directory. Data != Downloads. NB 6.5b14 just filled up all free 15GB of my system drive with incomplete downloads while I was re-re-re-trying downloads that NB kept deleting due to at least 1 of 80 files being incomplete...

NB 6.42 puts incomplete files in the download directory (as .nb2 files) and leaves my data directory alone. That's how it should be.

Anyway, this thread is about NB 6.5b14 deleting/discarding downloads instead of assembling them, so let's get back on topic. NB 6.42 keeps retrying incompletes forever regardless of the max retry count in the settings (which I already asked to be fixed long ago). Now this is fixed, but NB 6.5 just deletes the whole set instead, which is even worse

Back to 6.42 for now, since 6.5 is unusable that way.

[revco]

I can confirm the drive space issue reported in this thread. It appears that if you have a large number of files in a download that are partially incomplete, beta 14 will continue to download additional files into the chunks directory and will not reassemble, ultimately filling free space. If you run a relatively lean amount of free space (i.e. smaller than the files you're downloading), it's very easy to exhaust the free space on the installation drive. (It's happened to me twice all ready and I've only been running the beta for about 48 hours.) I would echo the desire to utilize the download directory as opposed to the installation directory, if possible...or at least allow it to assemble incompletes to the data directory as they are determined unfulfillable, to await a (potentially) successful autopar.

I was also reviewing the chunks directory and it appears things don't get cleaned out perfectly, but I'm not 100% sure why. I had previous download chunks that had completed successfully located in the chunks area. This was even after shutting down/restarting Newsbin. Unfortunately, due to the download completion, I'm not aware of why the chunks were remaining. (Download auto-par'd and un-rar'd successfully.) I will admin I experienced some sort of fault that caused Newsbin to stop running, so that may (and probably is) related to the issue.

P.S. Thanks for the server priority options, exactly what I wanted.

[Quade]

Newsbin assembles the incompletes:

1 - If you have PAR file for the files

2 - If Newsbin thinks you have enough of the file to repair with PAR files.

3 - When the retries expires. So, large values for retries might mean it never assembles.

So, I'm thinking you either don't have enough to repair or you have a high number of retries.

[revco]

Quade,

I can confirm you're correct in at least one of my cases that I've analyzed. I have noticed, however, that the chunk files are not deleting properly in this condition. (Or at least my case, anyway.) The file was moved to failed downloads, Newsbin was restarted, and original chunk files remained. (Had to clear manually.)

Respectfully, I slightly disagree with the philosophy here. Most times, Newsbin works perfectly. But, I've had the occasional odd file (in previous versions) that fails to autopar/unrar and I have to do so manually. As far as I've been able to tell, the failures have to do with filenames or odd (usually incorrect) PAR structures. In many cases, I've been able to recover after the fact using manual methods. (e.g. CRC correction on a bad RAR set, something Newsbin doesn't handle at all.) Sometimes I have to hunt down an alternate post pulling some files from a different found set...which tends to confuse the heck out of Autopar. With the current method, there's absolutely no way that I can think of to recover the file after the fact, should Newsbin not react properly to the RAR set. This is mostly because we now don't have access to the actual RAR files (and no way to manually reassemble), even if incomplete, but rather some non-assembled chunks that won't be recognized by anything but Newsbin. Don't get me wrong...I LOVE autopar, but I do see this as a limiting design in some cases. I at least like the option of being able to try my hand at manual recovery methods rather than leaving it all up to Newsbin.

[Quade]

that the chunk files are not deleting properly in this condition. (Or at least my case, anyway.) The file was moved to failed downloads, Newsbin was restarted, and original chunk files remained. (Had to clear manually.)

Moving to failed isn't supposed to delete the chunks. If you were to start a secondary news server for example and move them back from failed to download to try to get the missing chunks, you wouldn't want to re-download the data you already have.

This is mostly because we now don't have access to the actual RAR files (and no way to manually reassemble), even if incomplete, but rather some non-assembled chunks that won't be recognized by anything but Newsbin.

Add them back to the download list, right click "Assemble Incompletes". You might have to pause, assemble then unpause to make it actually assemble and not go back to the failed list. I typically don't have to do that though. If you hit the + icon, select the individual files and "Assemble" it'll assemble them no matter what state the download is in. If you assemble on the parent item, it tries to be smart about it and won't assemble if chunks are still downloading.

The only time they should go to the failed list is if you don't have a PAR file for the set though.

[revco]

Ah-ha! I get it now. That's perfect, at least there's a way to force it to do what I want. I wasn't quite thinking through that process. How about a possible feature request to add "Assemble Incompletes" to the failed download list? (That's just nitpicking, though.)

Things have sure changed in my time with Newsbin from way back in the day. This is no different! Appreciate your stellar work, as always!

[bobsmo]

Newsbin assembles the incompletes:

1 - If you have PAR file for the files

2 - If Newsbin thinks you have enough of the file to repair with PAR files.

3 - When the retries expires. So, large values for retries might mean it never assembles.

So, I'm thinking you either don't have enough to repair or you have a high number of retries.

Is that 1 or 2 or 3 OR 1 and 2 and 3?

[Quade]

AND for all of them. If all conditions are met, it'll assemble and attempt repair.

[bobsmo]

Even if AutoPar is off? Why does ths new version assume anything if autopar is off? What happened to just plakn downloading what is asked to be downloaded?

Thanks.

[Quade]

Unless I missed something, you said nothing about disabling autopar in your first post on this. If autopar is off, it should just dump them into the failed list after the last retry.

[bobsmo]

Unless I missed something, you said nothing about disabling autopar in your first post on this. If autopar is off, it should just dump them into the failed list after the last retry.

Sorry, i was referring to the original post, where they stated Autopar was set to off.

In any case, i guess i just didn't get or understand how NB knows/calculates that they are not enough PARS if a)autopar is off and/or b) some par files themselves can be incomplete (hence not being assembled but in fact if/when they are, there are in fact enough).

Or in other cases i have seen, where you have to use the PAR2 file to rename the rars, where in fact if one of the rar files is incomplete, it can still repair ok if the nb2 file is manually renamed.

[Quade]

If autopar is off, it should just dump them into the failed list after the last retry.

Here you go, in bold. If autopar is off, you're on your own. It doesn't even look at the PAR files and makes no decisions other than to fail incomplete files and leave it to you to assemble them.

You can manually assemble the files by adding them back the download list then right clicking and assembling them.

[bobsmo]

If autopar is off, it should just dump them into the failed list after the last retry.

Here you go, in bold. If autopar is off, you're on your own. It doesn't even look at the PAR files and makes no decisions other than to fail incomplete files and leave it to you to assemble them.

You can manually assemble the files by adding them back the download list then right clicking and assembling them.

Ok fair play, but why not just assemble the incompletes as is ? (Like now). Guess i just dont understand why the change in coding? Was this some enhancement request or alike? I ask as maybe I'm missing something good i just do not see.

Thanks.

[Quade]

Assembled incomplete files that don't have pars are trash. If you have autopar turned off then Newsbin doesn't know or care if pars are there or not so, it reverts to "don't have pars" logic. People who do use autopar, don't want incomplete and unrepairable files sitting in the download folder, they'd rather they fail. Then they can consider bringing another server online and try to finish the download that way.

Keep in mind, your mode, "autopar off", isn't even on my rader. It gets no development or concern till someone complains. So, you ask why I made the decision, it's because it's the right answer for people who use autopar. Because of these posts though, I'm considering making "Autopar off" mode just assemble when retries expire. It won't hurt the people who use autopar so, there's no real reason not to do it.

[Busty]

If we download data then it should stay in the download folder with rar/par and all parts until we decide ourselves what to do with it considering you can often repair files with quickpar that have failed with with newsbin par.

[bobsmo]

Assembled incomplete files that don't have pars are trash. If you have autopar turned off then Newsbin doesn't know or care if pars are there or not so, it reverts to "don't have pars" logic. People who do use autopar, don't want incomplete and unrepairable files sitting in the download folder, they'd rather they fail. Then they can consider bringing another server online and try to finish the download that way.

Keep in mind, your mode, "autopar off", isn't even on my rader. It gets no development or concern till someone complains. So, you ask why I made the decision, it's because it's the right answer for people who use autopar. Because of these posts though, I'm considering making "Autopar off" mode just assemble when retries expire. It won't hurt the people who use autopar so, there's no real reason not to do it.

OK, thanks, then, so maybe keep as is, as certainly right now, that (just assemble when retries expire) happens now.

[Quade]

as certainly right now, that (just assemble when retries expire) happens now.

Ok great, maybe I already implemented it. For modes I don't use. I always have to re-check what the logic is doing.

[Quade]

If we download data then it should stay in the download folder with rar/par and all parts until we decide ourselves what to do with it considering you can often repair files with quickpar that have failed with with newsbin par.

I'm not clear what you're complaining about. The files remain.

Moving to the failed list doesn't delete anything. It just gets it out of the way of the good files. You can manually repair from the download folder or the files list or do anything you want with the downloaded files. Files with PARS and with autopar enabled never get moved to the failed list anyway. They remain in the download list and continue trying till you do something about it.

[itimpi]

Maybe the most useful option would be to allow the "Assemble Incompletes" to be allowed directly from the Failed Files list. I would find this useful myself at times even though I DO use AutoPar.

[Quade]

Yeah, it would have to include adding it back to the download list because Assemble Incompletes uses the standard assembly code. I added a note.

[Plankton]

If we download data then it should stay in the download folder with rar/par and all parts until we decide ourselves what to do with it considering you can often repair files with quickpar that have failed with with newsbin par.

I'm not clear what you're complaining about. The files remain.

Moving to the failed list doesn't delete anything. It just gets it out of the way of the good files. You can manually repair from the download folder or the files list or do anything you want with the downloaded files. Files with PARS and with autopar enabled never get moved to the failed list anyway. They remain in the download list and continue trying till you do something about it.

Often with failed / stuck downloads you might get the first par in the download folder but the rest are missing so i'm not sure if that is what Busty means?

I've often found myself loading up another client to re-download a NZB so you get all the parts as it's quicker than waiting around sometimes for parts to be Assembled from the Incompletes

[Shine]

Because of these posts though, I'm considering making "Autopar off" mode just assemble when retries expire. It won't hurt the people who use autopar so, there's no real reason not to do it.

Yes please, that would be great! I actually loved that old behavior of NewsBin when it auto-assembled incompletes/unrepairables, and named them in capital letters, so I could easily see which of the files need special attention!

Thanks!

[mjpierce]

Because of these posts though, I'm considering making "Autopar off" mode just assemble when retries expire. It won't hurt the people who use autopar so, there's no real reason not to do it.

That would be great. Pars are really optional for binary news postings and many such postings don't have them.

[Quade]

I implemented this in B16. When retries expire and autopar is disabled it should just assemble.