Deleting files takes forever

[RayMark]

Sorry, I encountered this problem with 6.71 beta 4 (build 4710). Not tested with RC1 yet, but the change log does not say that the issue was addressed.


When unrar finishes (the green bar reaches the right end). the bar goes to white again, and says unrar 0%. And stays that way for a while - during that time rar parts are being deleted from the disk. Then that line finally is removed from the download list.

Unfortunately, this stage takes very very long. Longer even than downloading. Several minutes. It is completely waisted time - nothing else is being downloaded or extracted during that time.
I thought, perhaps my huge descripti.ion file was to blame (turns out, it grew really huge). I deleted it. No, exactly the same.
It is not the slow hard drive either - I can delete all those rar parts manually much faster, practically instantly.
So it is hardly my antivirus interferes - again, I can delete the same files manually very quickly.

Perhaps some db file became very large?
Something is very wrong, anyway. The unrar at this point in complete. I see that unrared files are all here.
It should take a second , not several minutes to delete those rar files.

For a post of only 500 MB (but with rather small rar parts, 100 parts) - it takes 1.5 minutes to delete those rar parts
If the post is DVD, or worse, BluRay - where sizes may be 4.5 GB or 24 GB or more - then what? An hour?

BTW, it seems that descript.ion is not updated when rar parts are deleted? They stay listed in the descript.ion file - therefore it grows and grows. If so, it is another issue to fix.

[Quade]

BTW, it seems that descript.ion is not updated when rar parts are deleted? They stay listed in the descript.ion file - therefore it grows and grows. If so, it is another issue to fix.

Nothing ever gets deleted from this file. It's just a record of what you downloaded.

I just downloaded a 4 TB set. Took about 15 seconds to delete. I was watching in Procmon and the slow part seemed to be feeding the recycle bin.

I tried it without the recycle bin and it wasn't much faster.

I saw it feeding the the signature.db3. You might try moving the signature.db3 to a different folder then start Newsbin and see if it makes a difference.

[RayMark]

SIGNATURE.db3 - 843,181,056 bytes

As it was not possible to move it without exiting NewsBin first, switched to RC1 as well (2 changes - not good for debugging).

Yes!!!
Deleting unrared rars now is instantaneous.
Not only that, but the decoding from the cache (creating files after downloading, probably par checking as well) - it is much much faster as well. Previously creating files took significantly longer than downloading them. Now it so much faster, that most of it is done during the downloading of the same file. Previously the download finished first, before PAR counter moved from 0. And that counter now moves maybe 100 times faster.

The whole downloading is so much faster, it (almost?) feels like using the good old NewsBin 6.55

Maybe now the other issues will be less painful too? "Display update pending" - or whatever. It used to be this way: after expanding a post in the download list, most elements, or even all of them where shown as "pending" and even selecting a line did not make it resolved. Several minutes of waiting before it is resolved. So far, it seems to be fixed as well.
And simply deleting a post (a single line) from the download list - it used to throw NewsBin into "unresponding" state for several minutes as well. I hope it was affected as well.

And another issue: moving something to the top if there are already downloaded items - perhaps the backlog will be almost non-existing.

Unfortunately, there is another bad issue when some multi-part rars show up on disk in uppercase as damaged or even 0 length. Usually first parts are ok, but from some point till the end they all are bad.
But it does not happen always. After deleting and re-downloading the same post - everything is OK.
It is difficult to reproduce, but I had several such cases just today. Happened before as well. Maybe this was fixed as well?

...................
descript.ion - maybe it should have a different name, not descript.ion?
descript.ion is supposed to be kept always synchronized with the contents of the directory.
If some software creates, renames, copies, moves, deletes files, it is responsible for updating descript.ion accordingly.
You are aware that file managers such as FAR can show the corresponding to a file description from descript.ion, changing it accordingly as you move from file to file in the list of files.
Not updating descript.ion disrupts that functionality.

On the other hand, if autopar/unrar is switched off, probably descript.ion is mostly OK.
Oh, the renaming issue, the -(0001) etc. files - they may cause descript.ion to go out of sync as well.

[Quade]

descript.ion - maybe it should have a different name, not descript.ion?

It's a file you can parse though to see a record of what you downloaded. That's all it is. All it's ever going to be. I'm never going to try to make it match the folder contents. It's named that way for legacy reasons.

SIGNATURE.db3 - 843,181,056 bytes

Yeah, it's a crazy large file. I can't purge it so it's kinda on you to keep an eye on it. You might want to just disable the duplicate checker. If performance is what you care about and you download in bulk (which seems like a fair guess). You might not want to take the performance hit.

No you can't delete it while Newsbin is running. That's to be expected though. It's an active database file.

[RayMark]

And how to disable the duplicate checker?

I have "Use Duplicate Detector" unchecked.
Still, SIGNATURE.db3 was created again, and, even though it is still quite small (12,288 bytes) - already both file creation/par check and file deleting are significantly slower than immediately after removing SIGNATURE.db3.
For 700 MB 64 rar part post file deletion now takes again about 25 seconds.
In order to keep removing SIGNATURE.db3 I would need to close NewsBin frequently. How to disable it completely?
After removing that db3 file, everything went much faster.
Unless there are other factors involved - then I tried downloading immediately after restarting NewsBin. Now I have a group loaded which takes a bit over 3 GB of memory. But that should not be an issue?

[Quade]

Still, SIGNATURE.db3 was created again, and, even though it is still quite small (12,288 bytes) - already both file creation/par check and file deleting are significantly slower than immediately after removing SIGNATURE.db3.

I doubt it's related. Yes that's how you disable it. It will create a new one but won't use it.

[RayMark]

So far the size of SIGNATURE.db3 did not change, still 12.288 bytes.
So I will see if it grows or not.
However, its usage already was switched off, I don't even remember for how long - for years, I think.
It is hard for me to believe that it was so big for years. Especially, that removing it had such a dramatic effect
Or did it? Perhaps re-starting NewsBin and not loading any groups into memory had this effect.
So I will try to observe if the file grows and if restarting (or perhaps closing groups and freeing memory) speeds things up.

[Quade]

What matters is what's really happening. If you see it growing when it's disabled, it's a problem.

Older versions would still store the signature even if you disabled it but wouldn't obey the result. Newer versions don't work that way.

[RayMark]

With 6.71 RC3 - worse than even. So - the same.

Perhaps I forgot to mention - while unrared line from the download list is being deleted - NewsBin GUI is not responding. So, practically, NewsBin almost always is not responding. Because it looks to me that it always is deleting. Everything else is very fast in comparison. But GUI should run on a separate thread and should not be affected? So I could do other stuff while waiting.

I see that the following files are being modified during that time:

Downloaded.db3 - 221,986,816 bytes
DownloadMarker.DB3 - 163,708,928 bytes
DownloadsV2.db3 - 107,544,576 bytes

Question: May I delete some of the above files? I want NewsBin to remember what is already downloaded and show that to me in the group / search tabs. So what I can remove without affecting NewsBin's memory?

I am guessing that DownloadsV2.db3 replaces Downloads.db3, which is present but not updated. So it is safe to remove it, but it probably will not speed things up. I simply have lots of stuff in the download queue (currently: 1264), but most of them are paused anyway. So I can easily remove it - let's say, after exporting its contents to a nzb first. But I suspect that will not help. The top two files are more interesting.

[Quade]

You can delete them all. DownloadsV2 contains the download, wish and failed list.

[RayMark]

But where is the information required to mark posts as already downloaded?
Not in headers, because the dates of the last write to headers are older.

[Quade]

"DownloadMarker" contains "Downloaded" status.