UnRxxxxx.tmp folders all over the place :(

[UlrichKliegis]

Quade and folks,

Generally, autopar and autorar seem to work fine. But then, after starting some pretty hefty downloads that brought the HDD to its limit (NB stopped appropriately!), I now find a large numer of folders below the nzb folder level (meanwhile I deactivated that nzb, but the problem persists, just one level higher in the folder hierarchy now): Lots of UnRxxxx.tmp folders that partly contain partials of the wanted download, partially unrared, partially not, and all requiring lots of manual interaction. What - except having not yet replaced the 1 TB by a 2 TB HDD (which just will shift the problem again, I think) - else can I do?
Ah well, all downloads where this happens are controlled by nzb files generated by the http://www.binsearch.info server. I only very rarely do still invoke downloads from the genuine group window in NB itself (logistic background, basement machine... )

Am I wrong, or was the world (also the usenet) simpler in older days?

Cheers,
U.

[Quade]

Newsbin unrars to a temp folder and copies the files to the final destination of there's room for the files and if the filenames don't already exist in the destination folder. So, you need to see if either one of these conditions might have prevented the move to the final destination.

[UlrichKliegis]

I guess the first condition was temporarily responsible for leaving the files there in the temp-folders. It would be nice if NB would remember to continue its housekeeping tasks after the preventing condition has been cleared. So, I'll have to do that myself? Hmmm.

[UlrichKliegis]

Quade,
is it possible to instruct NB to change the sequence / priority of the single different tasks that it has to accomplish?
I expect there would be fewer file zombies (tmp partials) around if NB could first finish the PARing and unRARing after it has everything necessary belonging to one article together, and then, after having done that, start d'loading the next task. A small buffer to bridge some activity gaps would be ok, but I have the impression that it is just overloaded with all the tasks - even a 1 TB or 2 TB disk will fill faster, even more, the faster the online connection speed is, than NB can.

The overall time - using 100% CPU time - would be the same like that in the now pretty complex interleaved mode of doing the jobs. Or am I wrong?

So, again: Is there a way to limit the multi-multi-tasking levels and branches?

Thanks again, overall, for a very nice program, the limits out there in this harsh world are probably often like here: I would love to invest in a 12 GB RAM i7 countless TB machine, but "just to download from the usenet? " - Hmmm...

Cheers, and keep up the spirits!

U.

[itimpi]

As far as I know there is no way to do that. Note that the limit on throughput is normally disk I/O rather than CPU so if you have choices you might want to spread the disk load across multiple drives by separating out the Newsbin DATA and DOWNLOAd folders, and possibly unpacking to a thrir drive.

One option that might achieve the result you were asking about is to set the Newsbin Speed Limiter to a value that your system seems to be able to keep up with.

[Quade]

I expect there would be fewer file zombies (tmp partials) around if NB could first finish the PARing and unRARing after it has everything necessary belonging to one article together, and then, after having done that, start d'loading the next task. A small buffer to bridge some activity gaps would be ok, but I have the impression that it is just overloaded with all the tasks - even a 1 TB or 2 TB disk will fill faster, even more, the faster the online connection speed is, than NB can.

I told you why you get these temp folders and it has nothing to do with load. Either, it fails to copy all the files to the destination folder or files with the same name already exist in the destination folder. I suspect the later. The tmp folder holds the complete unrar, there's no "partial" unrar in there. Some releases come with common text files in every release so, every release has some identical file names in them. I'm thinking about disabling the "Same Filename" restriction for junk files, like txt and HTML files so, the copy should be more likely in the presence of these releases with common junk files.

You could probably prevent the temp folder if you used NZB filenames as part of the unrar folder (options settings) so, you're just not likely to get duplicate filenames in the final destination folder. There's only one unrar going on at a time and it only unrars when it has all the files on hand. There is no unrar interleaving.

[UlrichKliegis]

[data] and [download] live on separate drives here already (not only separate partitions, but real drives). Reducing the transfer speed - well, a little unusual, but if it helps? At least worth a try, thanks for that idea!

The third drive for results - hmmmm. Maybe there are some other options - e.g., how does NB set the sequence of unRARing and unPARing jobs? My impression is that it does that time-multiplex, i.e., as many processes active as possible, rather than one after the other, in the sequence that they appear in from outside. Mouth too full, so to speak... Can I limit the count of instances of the unraring / paring process like I do for the server channels? The latter may be even more unpredictable...

Cheers,
U.

[Quade]

One unrar/repair at a time.

[UlrichKliegis]

ok, saw your previous post only after sending my last one. I just was happy to reduce the filename complexity (thus also reducing the incidence of "filename_too_long - errors where the sender wrote a novel in the header), but will re-enable that nzb-part in the structure.

Thanks.

Cheers,
U.

[UlrichKliegis]

I'm thinking about disabling the "Same Filename" restriction for junk files, like txt and HTML files so, the copy should be more likely in the presence of these releases with common junk files.

Multiple equally-named .nfo files are another case...

[Quade]

Typically NFO's have a real filename. Still it could apply to NFO's too, they're junk to me.

[UlrichKliegis]

and possibly unpacking to a thrir drive.

Creating an "unrared" folder to store the final results in and selecting that option has led to a clarity of what is ready and what not never seen before on that machine. Plus a convenient combination of the folder name components set in NB. Recommended!

Cheers,

U.

[Quade]

I've changed behavior to just rename the existing files in the destination folder before the files are copied out of the tmp folders so, all the tmp folders should clear now.

[UlrichKliegis]

Another step on the way to usenet heaven... Image link not allowed for unregistered users Available in the next beta, I guess?

[UlrichKliegis]

Stuck again: I installed the "third folder" to collect all unrared files. Works/ed like a charme.

Until - the incoming folder slowly but surely filled up with rars where just one or a few blocks of many were missing. If I understand the NB philosophy right, NB would recognize the necessity to load sufficient PAR files to repair the bad or missing data. This seems to work perfectly - until the disk is full.

Now my question is: How can I help NB to get back on the track again?

As soon as I make some room on the HDD, NB will either start to load more new files, or, if I stop that, will become totally inactive, at least here, it will not continue to get repair-PARs to continue the automatic reconstruction first, then and only then start with new data.

If I move any of the partial data to a different disk, NB will continue its work - but then those moved data are sort of stranded, their needed repair-PARs could only be retrieved with much user interaction.

I am not sure if the following proposals would be useful or would work at all.

a) a higher threshold of free disk space would at least grant some data mobility. If, b) only the input stream would then be stopped but the background processing of the already downloaded partially to be repaired-data could and should continue - allowing also to load additional PARs. Spare space for this could also be allocated on a different partition / drive.

The dream would be to feed NB even previously loaded but not yet fully decoded or repaired RARs and PARs which it wold process then automagically as far as possible. This should at least work for files that NB once retreived itself but which had to be moved somewhere else temporarily to create disk space, for example. I understand that NB, however, would at least have to know where the PARs came from...

But back to my presently "stuck" situation: How can I get out there in the most elegant and system-conform way?

Maybe the solution to my problem is already implemented, and I just don't see it?

Thanks to all who bear with me... Image link not allowed for unregistered users

Ulli

[itimpi]

I increase my Disk free space threshold to about 10GB. The instructions on how to set the threshold to whatever value you want are in the Advanced section of the User Guide.

[UlrichKliegis]

Thanks, itimpi, increasing the threshold was an easy one. The page you named in your sig (I knew it generally, of course) also mentioned how to tweak the way NB handles PARs and the decoders' relatve priority. I did not find any of these, though, in version 6.latest that I am presently using (not the beta). I turned down the speed limit to a valium-like 0.5 Mb/s.
In previous versions, there was a pretty transparent display of how far the decoding process had proceeded for each file. I can't find any of that in 6.xx now. Is it me or my glasses?

[itimpi]

In previous versions, there was a pretty transparent display of how far the decoding process had proceeded for each file. I can't find any of that in 6.xx now. Is it me or my glasses?

It is much clearer in v6! When a file starts decoding the progress bar column in the Download list shows progress in Bright Green (or Yellow if repairing). This should be eay to identify as long as you have not used either of these colors to identify a particular server. The Status column should also display a percentage figure.

[UlrichKliegis]

Thanks, ok, but how can I control the degree of priority for the decoding process(es)? I also looked in the newsbin.nbi file but found no self-explaining parameter name.

[taowelie]

I'm experiencing the same issue since some 6.xx version. I've just dealt with it, but it's becoming really annoying now since it happens with EVERY single download. It's not intermittent... has there been any resolution to this issue?

Thank You!

[Quade]

6.2 has the fix discussed here.

[UlrichKliegis]

Thanks, Quade. I installed 6.2 immediately after its release and did not see this problem any more.

Cheers,
U.

[Quade]

That's what I hoped to hear.

[taowelie]

Hi Quade, I'm still experiencing the issue Image link not allowed for unregistered users I'm running 6.30B3 and have tried every version since 6.2 to get it to stop. Please let me know what I can try to do. Thank you.

[Quade]

Is anything in these folders? I wonder if you're really running the latest versions. Maybe you're installing them but the old ones run? The reason I say that is I'm not seeing this at all.

[taowelie]

Is anything in these folders? I wonder if you're really running the latest versions. Maybe you're installing them but the old ones run? The reason I say that is I'm not seeing this at all.

Sorry for the late response, Quade. It appears to be functioning correctly now. I'm running 6.31RC2 B1836 and about to update to 631RC3

[UlrichKliegis]

That's what I hoped to hear.

Quade, I don't want to ruin your week, but the folders persist to exist, although they are all admittedly empty now. I don't object to some manual work, but have the feeling that this is still not totally what was in the specs.
BTW - all these wonderful improvements - do they generally work on previously but not yet fully processed downloads too?

Cheers,
U.

[Quade]

What version are you currently using? I was just doing my monthly cleanup and I think I saw one of these folders. Considering I debug though, it's possible that I simply left the debugger 1/2 way through an unrar.

As for whether you existing downloads get the fixes. It depends. For something like an unrar though, the answer is yes.

[UlrichKliegis]

Quade, as posted yesterday in another thread ( viewtopic.php?f=43&t=30116#p183329 ), I use 6.31 (final RC)

Cheers,
U.
(Proposal: let users generate a footer / sig here where they can enter their system parameters - would save a lot of back questions. In my case, it would read XP, NB 6.31, nzbs generated by nzbindex.nl....)

[itimpi]

(Proposal: let users generate a footer / sig here where they can enter their system parameters - would save a lot of back questions. In my case, it would read XP, NB 6.31, nzbs generated by nzbindex.nl....)

You can set up a signature in your forum profile.