6.62 - Chunks multiplying at a mind-boggling speed!

[RayMark]

This is with 6.62 final (build 4358), but probably with older builds and versions as well.

Just in a few short hours my Chunks folder grew from 0 (after I previously deleted all the chunks manually - today!) to 73 GB - and everything in the download list is successfully downloaded, unrarred and deleted, those chunks do not belong to something that is not finished downloading yet.
Why (so many) chunks are not deleted automatically? Seems like a bug.

To grew orphan chunks so quickly, try downloading something posted by d@mn.dymyca(DD) in a.b.tv group. Remove the two letters "y" to get the correct poster - I added them to avoid triggering hits by search engines. Besides, it is a very offensive word

BTW, I asked ages ago to add "Poster contains" into custom filters. Well, at least a few versions ago. Still not done in 6.62. Because of that, you will have to figure out yourself how to find posts by that poster
(a hint: you may search for strings containing a date, for example, 151201, or 151116, etc.

BTW-2, off-topic questions (sorry for that, but I will try to make them relevant to this topic):
1. how to specify the poster in the search string (such as the above mentioned DD)?
2. why search for 151116_ returns all the same garbage as 15116 - an underscore is not white space nor a delimiter, this search string should result (hopefully) only in the posts by the above mentioned DD ?

UPDATE: 10 GB of orphan chunks again literally in one hour. Downloading @ 270-290 Mbps

[Quade]

BTW, I asked ages ago to add "Poster contains" into custom filters. Well, at least a few versions ago. Still not done in 6.62. Because of that, you will have to figure out yourself how to find posts by that poster
(a hint: you may search for strings containing a date, for example, 151201, or 151116, etc.

I just search by poster using the search engine.

Anyway, I downloaded 2 of the sets and most of the chunks vanished when they finished. The part 1's are kept for up to 3 days.

After one set which consisted of 2310 chunks, I had 27 part 01 chunks sitting in the chunks folder. My files were 100% complete though. Are yours complete or are you needing repair?

[RayMark]

BTW, I asked ages ago to add "Poster contains" into custom filters. Well, at least a few versions ago. Still not done in 6.62. Because of that, you will have to figure out yourself how to find posts by that poster
(a hint: you may search for strings containing a date, for example, 151201, or 151116, etc.

I just search by poster using the search engine.

Anyway, I downloaded 2 of the sets and most of the chunks vanished when they finished. The part 1's are kept for up to 3 days.

After one set which consisted of 2310 chunks, I had 27 part 01 chunks sitting in the chunks folder. My files were 100% complete though. Are yours complete or are you needing repair?

I clearly need to learn how to search by poster

My posts seem to be all complete, although some are repaired before extracted. And I don't see any 01 chunks. I see some 1.xx chunks such as .1.14 or 1.66 but no xx.1 or xx.01 or 01.xx.
Now the number of the chunks increased to almost 25 GB (more than 32,400 chunks) - most of them are left after completed posts, although about 36 posts are currently in progress, so they account for some number of chunks.
As the number of blocks (sections) per post I am currently downloading is mostly 66, the number of chunks of the posts in progress should not exceed 2400 even if all of them were a single section chunks, So at least 30,000 are left to be deleted manually.

Have you tried specifically those posts that I suggested?
They seem to generate many orphan chunks - 30,000 orphan chunks in less than 4 hours in my case.

Here how all my chunks (from that specific poster) look like, which one of them is part 01 chunk?

AD4E18B2.18.67
B01050AD.30.50
E6686431.24.53

Their names are meaningless - seem to be CRC in hex or something, not rar part01 - if that is what you mean.
Maybe you generate such names only for chunks of some specifically named posts and then are not able to handle them correctly?
Again, try that poster!

[Quade]

The naming for all the chunks is the same. There's nothing special about the files this guy posts other than the obscure names and even then, they're not that obscure. 01 chunks start with 01 in the extension

bla.01.XX

Each filename for each chunk for each downloading file will have the same CRC value. So if you have a 66 chunk files you'll see

<bla>01.66
<bla>02.66
...

And so on.

I downloaded one of your sets and ended up with 37 part 1 chunks in the chunks folder. At this point I have no idea what your issue is. It looks like delete is failing. I'd suggest as I always do, making your security software ignore the Newsbin data folder.

You could clear out the chunks folder and download a single multi-part file. Maybe a single rar from a set. Then watch what happens when it finishes.

[RayMark]

On the other hand, maybe the problem surfaces when you put into the download queue more of those posts at once?
Try downloading the whole day of his posts at once? Some days are quite large, for example, 151003.
Or perhaps the orphan chunks are left only when rars have to be repaired? I don't know how many of them are actually being repaired.

I don't see any bla.01.xx chunks but there are some bla.1.xx chunks, however, more chunks with other numbers.

So now, my download list is again completely empty - everything was downloaded, repaired (when needed), extracted and deleted.
So, I assume, the Chunks folder should be empty?

Instead:

41,875 chunks (almost 32 GB total), of them, bla.1.xx chunks: 9,154
So you see, more than 3/4 of all the orphan chunks are not bla.1.xx chunks. But clearly, there are more 1st chunks than with any other specific number.
And what does the last number mean? bla.39.50 probably means bla 39th-of-50?
I don't see any partially combined chunks. Each holds only a single section, it seems.


UPDATE: forget it, I have lots of left chunks after downloading a single post, and it certainly was not being repaired - green at once.
After downloading a single post, approximately 1 GB in size, I have 94 chunks left, almost 63 MB in size. With various numbers, not only 1st.

Trying again. Properties: 67 records, size: 50.8 MB
Unrared: 3 files (mkv, nfo, srr) less than 1 GB
Chunks left: 106 files, about 80 MB

[RayMark]

How to specify the path to the Chunks folder?
I would like to move Chunks to the same hard drive where my downloads are located, but I want to keep Data on a separate drive - because the hard drive to which I download stuff is frequently changed, and the headers, etc., should be kept on the drive that is always present.
Maybe that would solve the chunks problem? How about the speed, would it be better or worse if chunks and downloads are on the same hard drive?
The hard drive for downloads is fast (connected usually as internal SATA 6 - I have an integrated hot swap internal SATA docking station on the top of the computer tower case).
My Data folder is on an external USB 3.0, so probably fast enough as well.

I see this option: UseChunks=1. Maybe disabling it means that chunks are put directly on the download path?
I probably should try it.

PROBLEM: UseChunks=0 still creates (and leaves) chunks in the data folder!!!
I would like to try keeping Chunks in the Downloads folder or at least on the same drive as downloads. Is it even possible?
Data folder, obviously, must be on a separate drive. But chunks are not data, they are downloads in the intermediate form.

[Quade]

I'd rather you try a single say, 10 meg file and see what it does.

You have two much going on at once to even guess.

[RayMark]

Did you miss this from above:

UPDATE: forget it, I have lots of left chunks after downloading a single post, and it certainly was not being repaired - green at once.
After downloading a single post, approximately 1 GB in size, I have 94 chunks left, almost 63 MB in size. With various numbers, not only 1st.

Trying again. Properties: 67 records, size: 50.8 MB
Unrared: 3 files (mkv, nfo, srr) less than 1 GB
Chunks left: 106 files, about 80 MB


I guess, I could also try files from other posters, but this was on a single multi-part rar file. And many chunks were left undeleted.

Ok, here is a single 67 MB file. Deleting all the chunks. The download list is empty. Downloading a single file.
Downloaded successfully. File is downloaded. The download list is empty.
Chunks folder: 126 chunks, total size 96 MB !!!
I cannot believe this - 96 MB of chunks for a 67 MB file.

Trying again - chunks deleted and double checked - Chunks folder is really empty.
Downloading a similar size file, 67.8 MB
Chunks: one 1 file only.

Hm, trying again with the previous file, the one that resulted in 96 MB
Also one 1 file only. I don't think I failed to delete chunks though. But perhaps. What else.

Trying on 10 single files of various sizes, up to 400 MB - ten 1st chunks only.

Trying also with a single file which is accompanied by a par2 files: in this case no chunks left, not even the 1st.

Ok. I suggest let's go to multi-part rars. Perhaps from a different poster?

I now downloaded a 1.58 GB multi-part rar post, just one of web-dl 720p variety.
Downloaded successfully, resulted in 3 files (mkv, nfo, srr).
Chunks left - 169 files, about 128 MB. 32 of them - 1st parts.
This post was not from the poster I began this topic with.

So you see - as long as it is a single file, everything is ok: if par2 shows that OK, - no chunks left, if there is no par2 to confirm that, the 1st chunk is left (for 4 days?)
But with multi-part rars serious problems. Perhaps there are cases when even multi-part rars are handled correctly, I am not sure.
Let's not search for such good cases. Let's take a bad one and if you can replicate it you can debug it.

So bag to the poster DD: I am downloading, let's say this one:

[ 150818_01 ] - [03/41] - "150818_01.part01.rar" yEnc

OK, downloaded, mkv, nfo, srr files
Chunks left: 173 chunks, about 131 MB, Of them - 25 1st parts. I even saved the list of all them.
Please, can you try on exactly the same file?

And how about moving Chunks to the different hard drive than Data files? Is it possible?

[Quade]

UPDATE: forget it, I have lots of left chunks after downloading a single post, and it certainly was not being repaired - green at once.
After downloading a single post, approximately 1 GB in size, I have 94 chunks left, almost 63 MB in size. With various numbers, not only 1st.

Did you miss this from above:

I can't keep track of what you're doing. I'm asking for a single file download but it sounds like you're still downloading sets of files.

Start with an empty chunks folder, hit the plus icon, see a list of PARS and RARS and download one RAR. Look in the chunks folder. As far as I can tell you still haven't performed the test I asked for.

[RayMark]

Oh, you want one rar part from a multi-part rar?

OK. Downloaded. the rar part is ok. Chunks: 64 chunks left. Very interesting that there are 3 different file names here. On with the most chunks left: 47 of 52 left. But where the other two came from?

I have deleted all the chunks but when I try downloading another single rar part from the same rar, - it shows up downloaded but no chunks appear in the Chunks folder even temporarily.
Another one - the same, rar part downloaded, no chunks showed up.

I downloaded: part01.rar, part20.rar, part30.rar - so not even in order. And they contain real data, seem to be correct.

How to explain that?


Another attempt, with a different multi-part rar. This time downloading part15.rar first. not the 1st one.
OK, downloaded.
Chunks: 48 for 4 different files this time, it seems.



I don't know if it is a good idea to download a single rar part, who knows how much you are actually downloading, and what is supposed to be left after such a download.

But if we download a complete single multi-part rar post - and any chunks are left afterwards - that is wrong.
I gave you such an example above, again: 150818_01.part01.rar - 173 chunks wrongly left.

[Quade]

How to explain that?

Caching. Not all chunks end up on disk. In fact that's the slowest way to download. If you download files and the chunks are all in memory, that gives you the highest performance.

The mix of filenames is because of caching again. I didn't think of it when I suggested this test. What I should have suggested was:

- Help/About - to flush the cache.
- Clear the chunks folder.
- Do the single file download.
- Help/About - to flush the cache again.
- Look in the chunks folder.

You're using 6.62 and not the 6.63 test version right? 6.63 has some extra cache flushing for testing so it might cause some strange issues.

[RayMark]

No, I am using 6.62 final. No 6.63 on the beta page yet.

OK.
Help / About
Cleared the Chunks folder
Downloaded a single rar part: 150807_02.part10.rar
Help / About
Chunks folder: empty

Again, part25.rar
Chunks folder: empty

2 parts at once
Chunks folder: empty

3 parts at once
Chunks folder: 2 different 1st chunks left

4 parts at once
Chunks folder: 4 different 1st chunks left - I assume, of all the 4 rar parts

5 parts at once
Chunks folder: 5 different 1st chunks left

10 parts at once
Chunks folder: 10 different 1st chunks left

25 parts at once (from 150807_03)
Chunks folder: 25 different 1st chunks left

The whole 150807_4 consisting of 32 rar parts + nfo + srr + par2 files
Chunks folder BEFORE Help / About : 32 different 1st rar parts left.
Chunks folder AFTER help / About: 186 various chunks
This was the first case when the cache did not return to 200 but remained at 46 before Help / About, and so flushing it with Help / About produced additional chunks.

So, until cache during download did not go down to 0, no chunks were left, when it did go down to 0 but returned back to 200 - only the first parts were left, when it went to 0 and did not return back to 200 - in addition to 1st parts also other parts were left.

But really, only the last case is important, because normally I would not be downloading separate rar parts by one or by several.

BTW, perhaps it would be interesting to play with the size of the cache - to increase it above 200. With 64-bit version and 32 GB of memory, why not? To see if overall download speed / throughput increases.

[RayMark]

Again deleted manually 93,813 chunks (almost 74 GB). That is the practical side of it.

Please, how to specify that chunks are to be placed not on the Data drive but on the Downloads drive?
That would simplify for me at least the empty space management.

How to increase the size of the cache from 200 to, let's say 2000? Maybe that would be enough not to generate orphan chunks.

[Quade]

Yeah, been thinking about your issue. No real answer off the top of my head.

[Performance]
SpoolPath=X:\Chunks\
ChunkCacheSize=2000

You can hand edit the NBI and add this. It's relatively untested. I put it in there for testing only. You really want the chunks folder to be on the fastest drive like an SSD or internal.

Did you ever tell your security software to ignore the data folder?

[RayMark]

1) SpoolPath - does it really specify the Chunks folder and not the headers? Because Spool folder (SPOOL_V6) holds all the headers - I don't want to change the location of headers. So I did not try this yet.

2) I finally made an exclusion for the Chunks folder for my security software - it did not help. Orphan chunks continue to be left. No difference.

3) I have increased the size of the cache to 2000

First tried with one multi-part rar with pars.
Chunks folder - empty. But the cache did not return to 2000. It was left at 1781 (it was down to about 1400). That is not good. That probably means that with more downloads eventually it all will be used up.
So, after flushing the cache: 219 chunks, almost 168 MB.

Trying with 5 multi-part rars at once.
Went down to 0 while downloading the last one, 5th rar.
Returned back to 750 after the downloads were finished.
However, the Chunks folder still empty. Even reaching 0 did not create any chunks.
Flushing: 1250 chunks left, about 953 MB

Memory usage: with the 2,000 chunks cache, the memory usage kept about 1,663 MB. After flushing the cache, it went down to 134 MB.
So, I guess I could probably get away with 20,000 or more instead of 2,000. It would use perhaps about 16 GB of memory.
But eventually the cache would still go to 0, if it does not return back to empty even after all the downloads are complete and the download list is empty. Perhaps you could enforce flushing of the cache at least when the download list becomes empty (better - after a download is completed, to flush all the related chunks. Actually, to delete them, if not directly from the cache (better) then at least after flushing).

Trying again, this time with 20 rars at once (from the same DD poster, so the rar sizes are similar: 1.6 - 2.1 GB including pars)
After downloading, the cache back to 590. I did not notice it going all the way to 0, just to 200, but I may have missed it.
Chunks: 3,520 (about 2.7 GB)
After flushing:
Chunks: 4,930 (about 3.8 GB)