6.80 RC7 keeps downloading @ low disk space pause

[RayMark]

I posted recently about a problem with NewsBin resuming downloading on its own even when previously it was paused manually - caused by this red pop-up low disk space pause.

Now I have a (perhaps related?) severe problem: Newsbin continues downloading even during the low-disk space automatic pause - and there is no way to stop it!

I have the low disk space pause set to 20 GB - because If it is allowed to go close to 0, then damaged/empty files are assembled from the cache (which in itself is wrong). So it is a safe margin allowing to avoid such corrupt files.
Unfortunately I just had a horror experience - The low disk space went to 20 GB, the automatic red pause popped-up.
I continued to delete/move some unnecessary files one by one, so - it seems - the red pop-up went off and on and off and on again several times.
Then I noticed a rather high network utilization - newsbin was still downloading at 300 Mbps or more even during the automatic pause, with the red pop-up glaring.
At that, it still was showing the free disk space close to 20 GB. But when I checked - it really was just a few MB left. And that means corrupt files. Even worse - there is no way to stop it.
Apparently, to stop downloading, I need to pause it manually, but there is no way to do it with the red pop-up. It is not possible to get rid of it for long enough. The only way - to delete enough stuff (so, about 20 GB at least at that point) for the auto-pause to go away.
And when I finally deleted more than 20 GB at once in a hurry - The red pop-up did not go off at once - instead it blinked several times off-on-off rapidly, as if it was popping multiple red alerts from a stack - maybe that will tell you something.

So, even with the 20 GB threshold I got lots of incomplete and 0-length files, because of this unacceptable and unpredictable behavior of NewsBin 6.80 RC 7 build 5054
So you see, this implementation of the auto-pause pop-ups causes various problems - but this is a really serious one - it may cause corrupt data not only in Newsbin downloads but somewhere else, not related to NewsBin, if it runs the disk space to 0 without any way to stop it other than to kill the Newsbin process. It is also misleading because during that red pop-up auto-pause NewsBin does not update the free space value and keeps showing close to the threshold (20 GB in my case) value when in fact it is rapidly approaching zero.

I think the whole implementation can be simplified:
1. No need to monitor the disk space when paused - paused either manually or automatically.
2. (follows from 1) the auto pause should not go away by itself, the user should have to un-pause it.
It would make NewsBin much more predictable and safe to use.
I don't think that many people are running some complicated scripts capable of making more free space when needed. So, if the user has to do it manually either deleting/moving something or changing the download drive (path) or physically replacing a drive - he then can un-pause NewsBin manually as well.

[Quade]

I continued to delete/move some unnecessary files one by one, so - it seems - the red pop-up went off and on and off and on again several times.

I'm caught between a rock and a hard place here. Windows occasionally lies about free space so, the popup automatically closes when the free space increases but in your case you free up some space which then unpops the windows, then with your high speed connection you re-fill the space back up. Ideally you should hit the pause when the popup drops but I can see where you might not be quick enough to do that. Maybe it needs some hysteresis. Pops when it hits a specific threshold then don't unpop until 25% of the threshold limit becomes available on disk. That would prevent if from quickly popping and unpoping.

I've not seen it continue to download with the low space window popped up but I'll verify. I wonder if it's simply that Newsbin is filling in the disk as fast as you free up space. Header downloads don't pause too. if your disk is set to "Recycle" the files then delete isn't really a delete. These days I run out of space mainly because of the recycle bin. It cleans itself out slower than I can download even though I only have a 100 Mbps connection.

[Calahan]

It's not only with RC7. I have such problems for a long time, I think with every version I used before.
I think it happened about 5 or 6 times in the last years, but as it doesn't happen so often, I never reported that problem...

I can't confirm the pause behavior, because when I saw Newsbin downloading, while there was no free space anymore,
I don't remember if there was a pause dialog or not. But I remember I had to delete all those many 0-length files and I had to delete
all incomplete unrar folders. Then after cleaning and creating free space I had to download all files again. I remember it was much work every time.

Instead of only looking for how much free space is available to pause the download, you should insert additional checks, wether all files are created with the correct size. Not only when downloading files, also when unpacking archives.
If I write to a file and there is no free space anymore for the size of the file, I get an error, that the file couldn't created.
Maybe you are missing some status checks when creating files?

[RayMark]

I don't use the recycle bin at all. It is always empty.
And I was not deleting any files one-by-one when I noticed NewsBin downloading while paused with the red pop-up. At that time it was really downloading at high speed when well below the threshold. And the only way to stop it was to kill the NewsBin process - or to make lots of free space at once.
I don't know what would have happened if the free space actually reached 0. But it was pretty close to that when I suddenly deleted more than 20 GB at once and then I paused it. But that puts the user under pressure and who knows what may get deleted.

I remind you that there was another perhaps related problem: NewsBin un-paused itself after a manual pause because thanks to other work I was doing on that drive while NewsBin was paused the free space dropped below 20 GB and then increased back above 20 GB - it never occurred to me that I needed to worry about NewsBin waking up on its own once I paused it.
Newsbin should never monitor free space when it is paused manually, or at least not un-pause itself. Because it continues doing other stuff when paused, perhaps it needs to monitor free space for other purposes - but it seems it makes no good use of that monitoring then - because it creates incomplete and 0-length files when assembling/decoding files from the cache. It should never happen! NewsBin should know that there is not enough free space to assemble the next file from the cache.

And also, it should be possible to pause everything. Perhaps I need to do some time or resource critical work and I cannot afford NewsBin doing anything in the background.
Can you add also a STOP button - for complete stop of all activities? I guess, NewsBin may be just closed. But when I have some local search results or some group tabs open which takes long to load, I want to avoid closing it completely.

Here's a thought:
There should be "Pause downloading" and "Pause everything" buttons. But perhaps "Pause downloading" we already have, sort of - the speed limiter - if it could be set to 0. Currently it cannot.
Another way to pause downloading would be to select all the items in the download list and pause them.
So you see - the complete pause is more important because we don't have any workaround for it.


Another thing:
When incomplete and 0-length files are created, there is no way to re-download them. It does not matter if I leave them or delete them manually, the right-click and "Retray Download" does not cause them to be downloaded again.
So I have to delete those posts from the download list and to add them to the download list again.
It is an additional hassle trying to find them - the group tabs or search results might be no longer open at that time.

[Calahan]

Another thing:
When incomplete and 0-length files are created, there is no way to re-download them. It does not matter if I leave them or delete them manually, the right-click and "Retray Download" does not cause them to be downloaded again.

If the files were still in the download list, my workaround was to close Newsbin, delete the downloaded 0-length files and restart Newsbin.
Newsbin then downloaded the files again. BUT that only worked if the files were still in the download list.

All other 0-length downloaded files and unrared folders had to be deleted and the nzb files loaded again or I had to choose download, on posts in the group posts list, again.
In any case, much work...

[Quade]

I noticed NewsBin downloading while paused with the red pop-up.

I'm skeptical but will verify. On the other hand you do download faster than autopar can process so even with no downloading and paused, you'll probably see a bunch of download list activity as it works to catch up. Pause pauses download, it doesn't pause the auto-par processing. Maybe it should pause both.

[Moondawgie]

You've not torn your hair out until you've used "Download to a new folder" to download files to a drive partition other than Newsbin's Main Download Folder that end up exceeding the remaining space on the chosen partition. Newsbin just keeps going and you end up with a bunch of zero byte files.

This is not a new behavior. Just one to beware of.

[RayMark]

If the files were still in the download list, my workaround was to close Newsbin, delete the downloaded 0-length files and restart Newsbin.
Newsbin then downloaded the files again. BUT that only worked if the files were still in the download list.

Yes, this is true. I forgot to mention it. I used this workaround myself. But why "Retry Download" does nothing in such situations? It could.

[RayMark]

I noticed NewsBin downloading while paused with the red pop-up.

I'm skeptical but will verify. On the other hand you do download faster than autopar can process so even with no downloading and paused, you'll probably see a bunch of download list activity as it works to catch up. Pause pauses download, it doesn't pause the auto-par processing. Maybe it should pause both.

I'm not sure if it is easy to reproduce. It probably takes some number of those red pop-ups to appear. But most certainly NewsBin was downloading at full speed while showing the red popup.
And yes, the download is faster than the subsequent processing, so I am using "Pause Download during UnRAR/Repair" to prevent the queue of already downloaded to get out of control.

BTW, it is my other complaint: If I have lots of stuff already downloaded but not yet processed, how can I download something in a hurry with high priority? Putting that item to the top of the download list does not help.
For processing - it is still the last one. It should become the first one for processing as well.
If I need to download something very quickly, I have either to start another instance of NewsBin or to delete the already downloaded but not yet processed items from the download list and to re-add and re-download them again later.

And yes, maybe it should pause both. I sometimes use Pause to allow NewsBin to catch up with processing, but there are other ways to do this - with the speed limiter set to the minimal speed, for example.

[RayMark]

You've not torn your hair out until you've used "Download to a new folder" to download files to a drive partition other than Newsbin's Main Download Folder that end up exceeding the remaining space on the chosen partition. Newsbin just keeps going and you end up with a bunch of zero byte files.

This is not a new behavior. Just one to beware of.

Yes, ideally NewsBin should monitor the free space on all the "Download to New Folder" drives as well.
And on all "Download to Existing Folder", "Assign a New Path" and "Assign a New UnRAR Path" drives.
And on download from NZBs drives, etc.
In other words, NewsBin should never write data blindly to any drive without checking if there is enough free space for it. Perhaps easier to implement than to keep track of all those potentially different drives.

And yet another thing. The download cache (the Chunks folder) is inside the Data folder, where all the headers and configuration files are stored.
It really should be possible to specify its location separately. And then NewsBin should be aware of the free space remaining on the Chunks drive.
For example, I could keep the headers on an internal drive where there is not so much free space remaining, and the cache on the Download drive. When processing is so far behind the downloading, this cache folder grows in size significantly.
And it has nothing to do with the headers or with the configuration data.
Perhaps I am wrong, but I seem to remember times when NewsBin had a checkbox to keep unprocessed chunks together with the downloads.
So I am forced to use a workaround. I really have the cache on the download drive but pretend that it is still on the NBData drive. For that I am using a junction point.
But then NewsBin is not aware how much of free space really is left for the chunks.
So, because I set a big enough threshold for the Download drive, everything is fine. Until NewsBin decides to keep downloading even when free space is below that threshold!
So now also the non-processed items in the Chunks folder are incomplete or possibly 0 length. But who cares when processed items would be 0-length anyway.

BTW, having the cache (chunks) on the Download drive improved the speed as well - at least for me.

[Quade]

So I've got 3 things I'm going to look at right now.

1 - Verify Raymarks report that it isn't pausing

2 - Look into making the limiter pop up window less poppy by making the unpop not happen exactly when there's enough free space (like require at least 25% > free space limit.

3 - Look into making pause apply to autopar.

6.80 is too late in the game to make any radical changes.

[Moondawgie]

Would it be possible to add the option to a future version of NewsBin which would pre-allocate space for files to be downloaded, inflated/extracted, and the ability to warn when there isn't enough remaining space to allocate, in the same or similar manner used by uTorrent?

[RayMark]

So I've got 3 things I'm going to look at right now.
1 - Verify Raymarks report that it isn't pausing

It is true, but it never occurred to me before. That tells me it might not be easy to reproduce.
But what I was doing this time is pretty clear: When NewsBin was at or around the threshold, I was moving some files to another drive - which is slower than downloading.
So, when one file is finally moved, the free space jumps above the threshold and soon goes below again because, again, downloading is faster than moving files to an external drive.
Then the red alert stays while the next file is finished moving, and so on. So there was a series of below the threshold alerts. And then I finished moving the group of files, naturally - again below the threshold, the red pop-up is on the screen.
I think NewsBin was showing fixed download speed and remaining free space - I think it does not update those values during the red pop-up.
But the task manager showed high network utilization so I looked at the speed through the interface used for NewsBin (the default one is different and connected to vpn and nothing was going on through it at the time).
Then I checked the remaining free space - and it was well below 20 GB and rapidly decreasing.
So I deleted in a hurry a bunch of files, more than 20 GB of them at once, and then I was able to pause NewsBin. But the red pop-up alert sort of blinked several times before disappearing for good, as if it was popping from a stack or something.
Oh, possibly it was caused by me tricking to click on buttons on the alert dialog box while it was still on - I may have done so in a futile attempt to stop Newsbin somehow when I saw it was downloading. So the blinking before disappearing might have been related to this rather than to going multiple times below the threshold - not sure.
Anyway, I did not expect NewsBin to keep downloading when paused so I was startled and not thinking clearly how better to diagnose the problem, I was hoping to avoid incomplete files and I acted quickly by deleting a whole folder of no longer needed files. But not quickly enough.