V6 requires more par2 blocks to fix incomplete files

[RayMark]

Recently I was getting quite a lot of incomplete files with 6.01 beta 1, even though "All Posts Available" in properties, so I tried to download the same files with V.5
Turns out, also incomplete, but different sizes.

The whole post consisted of 29 rar parts (rar, r00-r27), 12 of them were incomplete, here are the downloaded sizes:

Complete rar part: 50,000,000 bytes

9 Incomplete with V.5: 49,999,999 bytes
9 Incomplete with V.6: 49,919,999 bytes

and

1 Incomplete with V.5: 49,999,998 bytes
1 Incomplete with V.6: 49,919,998 bytes

and

1 Incomplete with V.5: 49,999,996 bytes
1 Incomplete with V.6: 49,919,996 bytes

and the last, smaller rar part:

should be: 27.896,188 bytes

- if downloaded with V5: 27,896,187 bytes
- if downloaded with V6: 27,647,999 bytes

The problem seems to be bad YEnc CRC. It was reported by V5, but nothing was reported by V6 - just an incomplete file assembled.

Because of the slightly different sizes of those incomplete files, for the whole post consisting of 27 rar parts of which 12 were incomplete, the required par2 block number to fix the rar is as follows:

- if downloaded with V5: 17 blocks needed
- if downloaded with V6: 27 blocks needed

Come on, it is more than 50% more !!!

Sometimes it so happens that the number of par2 blocks available is barely enough when downloaded with V5, so in such cases V6 becomes useless.
I guess, then I need to generate the nzb and try to download with V5, this way there would be no need to maintain headers up-to-date also with V5.

But perhaps V6 can be fixed not to waste blocks?

[itimpi]

Interesting! It looks like there is some difference between the two releases in the handling of bad posts. If you have not already done so I would PM Quade with the details of the bad posts that gave you the discrepancies you mentioned so that he can look into why the difference is occurring. It is always hard to get particular error cases that one can reproduce in a controlled testing environment.

Regarding the yEnc error messages I believe that Quade removed them because there was nothing the user could do about it as the problem was inherent in the post.

I have a couple of items I want to download that did not have quite enough repair blocks when done with v6. I might now try the same ones in v5 to see if I can get the same results as you and thus be able to repair these posts.

[RayMark]

Ok, done.
Sent PM to Quade

Actually, I wanted to attach the nzb file to PM, but every extension I tried was rejected.
Interesting, what types of files can be attached.

By the way, I tried several times with both V6 and V5.
I had headers so I used the headers. Then updated headers and tried again.

Now I also tried to use the also posted nzb file - the same result, also incomplete, the same sizes.

As to YEnc error messages, this one (bad CRC) is not shown with V6, but it shows sometimes some other messages, such as "bad YEnc format" or similar.
Not in this case, though.

Notice also, that in this case all the segments were present, only some of them had bad YEnc CRC.
I don't know what would happen if some segments were missing, would the resulting file be also less complete if downloaded with V6?
TO DO: to try with incomplete posts.

[Quade]

[07:51:02] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.rar_yEnc.nb2
[07:51:15] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r02_yEnc.nb2
[07:51:19] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r03_yEnc.nb2
[07:51:45] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r08_yEnc.nb2
[07:51:51] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r09_yEnc.nb2
[07:52:05] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r12_yEnc.nb2
[07:52:25] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r18_yEnc.nb2
[07:52:31] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r19_yEnc.nb2
[07:52:43] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r23_yEnc.nb2
[07:52:44] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r23_yEnc.nb2
[07:52:45] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r23_yEnc.nb2
[07:52:46] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r23_yEnc.nb2
[07:52:53] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r25_yEnc.nb2
[07:52:54] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r25_yEnc.nb2
[07:52:55] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r26_yEnc.nb2
[07:52:58] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.r27_yEnc.nb2

Each one of these chunks of the file is short on data. Meaning the file is bad sitting on the server. Some files have two damaged posts.

When I was done, I had 612 out out of 632 blocks. It downloaded one, 22 block repair files and repaired the files. It doesn't matter if a post is missing or a post is damaged. To PAR it's the same thing. All PAR cares about is blocks. Missing will lose you at least one block, can be more, a one byte change in the data or short data will all lose you blocks.

even though "All Posts Available" in properties

This means little if the data sitting on disk or the server is bad. It's not enough that it be there, it has to be byte by byte correct AND the bytes have to be in the correct order. So basically what you're showing here is that the file is F'd up on the server, which it is. The only question is whether Newsbin 6 really isn't getting as many blocks as 5. In my testing, the difference from that you claim is 3 blocks. The set was posted with 60 repair blocks.

such as "bad YEnc format" or similar.

This error means the required trailer for a yEnc post is missing. The CRC and size indicator for checking the post isn't even there.

So to me, the only question is, why might there be a 3 block difference in what you claim 5 found and 6 found. Other than that it worked perfectly.

[RayMark]

I just updated headers and downloaded with v6 again, nothing changed:

source block count: 632
damaged files: 12
complete files: 20
Ready to recover using 27/58 recovery blocks

With v5 the number was 17, so the difference is 10 blocks, not 3.

Maybe you are using a different par2 tool (built-in?)?

I got those results with QuickPar 0.9.1
It would be interesting if it requires more blocks than some other tools.

Still, the sizes of the incomplete files should be the same.
With v6:

rar, r02-r03,r08-r09,r12,r18-r19,r26: 49,919,999 bytes
r25: 49,919,998 bytes
r23: 49,919,996 bytes

BTW, this your example with
[07:51:02] DEBUG - WriteChunk -Warning - Chunk seems short:X:\[Püssy]\Dry and Boring-orenji.rar_yEnc.nb2

What is that?
My test was not with files with such names, so perhaps your test is with different files?
That would explain the 3 block difference instead of 10.
But the block count is the same: 632.
So, I guess, you just renamed the files yourself...

[Quade]

So, I guess, you just renamed the files yourself...

Haha, you don't say? I even left you a hint.

As far as I can see, you've not shown the good block count with 5 versus 6. That would be an interesting number.

[RayMark]

So, I guess, you just renamed the files yourself...

Haha, you don't say? I even left you a hint.

Except perhaps New Rules

QuickPar does not give the good block count, but it should be total count - bad count?

Actually, bad block count is also not given specifically.
It says "Ready to recover using N1/N2 recovery blocks.

The number of recovery blocks needed perhaps is not the same as bad block count.
Although probably is.

[Quade]

It says "Source block Count" and "Needed Blocks" so, the difference = missing blocks.

I ran the test again.
632 total blocks.
612 Newsbin found in the RAR's
3 blocks were in the NFO/SRR and SFV files which I didn't download. If you'd downloaded them, you'd have needed 17 blocks, I didn't download them, I needed 20 blocks. Newsbin again, downloaded a 22 block PAR file.

I don't see anything wrong. Your numbers don't match mine.

I've pretty much decided this is a non-issue. Newsbin's internal numbers match Quickpar. You claim that version 5 found more blocks in the data somehow. I'm skeptical. I even built a special test jig to save the raw data off and still that data is the same a what 6 is saving.

[RayMark]

I've pretty much decided this is a non-issue. Newsbin's internal numbers match Quickpar. You claim that version 5 found more blocks in the data somehow. I'm skeptical.

What about the sizes of the incomplete files, with V5 the files should be larger.

As to a non-issue, I think it is a VERY serious issue:

as you know, Giganews or somebody is removing first segments of each file in certain posts.

I tried such a post consisting of 12 rar parts:

with V5: Ready to recover using 12/146 recovery blocks. - as it should be
with V6: Ready to recover using 47/143 recovery blocks - what ???

As you see, V6 recquires 4 times more blocks (!!!) than V5, and the number of available blocks is also lower (because par2 blocks also have first segments removed)
I assume that V6 requires 47 and not 48 because the last rar part is smaller than all the others, and it needs 3 blocks instead of 4.
But when downloaded with V5, 1 block is enough for one removed segment, not 4!

This particular post had lots of par2 blocks and additional par2 blocks were posted as well.
Usually things are not so good, sometimes the number of par2 blocks is barely enough to restore those removed first segments, even when downloaded with V5.
With V6 it would be hopeless.
So it is not a non-issue.

I will PM you the info about this post

[Quade]

I'm not totally discounting your comments. In fact I've been re-working the "chunk assembler" because of what you're saying and because some asian poon didn't download completely. Both issues are related I think. In one case, the posting software is using variable length chunks (it's french). In your case, I've not actually managed to make it break the same way you did but, I'm still looking at it. In your case, the files were posted badly. Still not matching your numbers though. The new chunk assembler might end up being more reliable in these "files posted broken" cases.

[RayMark]

Quade, look at my test #2 - I PMed its details to you.

In my test #2, Giganews deliberately removed the first segment of each posted file.
But everything was posted correctly.

if downloaded with V5 - 1 block is required to restore such a file with the very first segment missing.
if downloaded with V6 - 4 blocks are required.

That is certainly a very serious issue, many such "deliberately expired" posts become incomplete with V6, but are OK with V5.
Because I remember often reconstructing such posts with the very last available par2 blocks or close to that.
In most cases there is no 4x reserve of par2 blocks.
And, again, these posts are posted correctly, they were not "posted badly".
Please try my example #2 or ANY other deliberately expired by Giganews post.
You will reproduce this huge difference.

[Quade]

That is certainly a very serious issue,

It's a serious issue if the files can't be repaired. If they can still be repaired it's something to look at but, not serious.

If you find me a set that works in 5 and doesn't work in 6. I'll upgrade it to "Serious".

[RayMark]

I mean, it is serious because it prevents successfully repairing many of those posts that have first segments artificially removed.

But I am emphasizing here the fact that when the 1st segment is missing, then 4 blocks are needed to repair it instead of just 1 as with V5.
Here are two things:

1. It should be easy for you to reproduce this issue. Until now you keep saying that your results do not match mine, but that was with bad CRC case.
What about when in previously correct post 1st segment is missing?

2. This 4x factor is mind boggling in itself, must be some obvious bug. Maybe too obvious to find
V5 proves, that 1 block is enough.

[Quade]

Just so we're clear, at this second we're talking about files that have been DMCA's and had their first chunks removed.

If I have a set of 20 files and the first chunk is missing from all 20, you have a minimum of a 20 block loss off the top. Nothing will prevent that. Then you can also count 1 blocks loss at least from every repair block. So, if a file says

vol00+01.par2 - there won't be any repair blocks in it.

vol15+14.par2 - will only have 13 or fewer in it.

What this means is there's a good chance Newsbin 6 will download more pars than it needs because the par files don't contain as many repair blocks as they claim.

Here's something for you to try. Exit Newsbin then, in the configuration file

[SETTINGS]
MemCacheLimit=100

Save then try a problem download and see if the symptoms change.

[RayMark]

I will try MemCacheLimit a bit later - downloading headers for a group for 17 hours already, almost 3/4 downloaded, I don't want to interrupt.

But look in my previous posts here, I already did such a test with the files with 1st chunks removed.
Not with a set of 20 files, but with a set of 12 files:
.rar, .r00-.r10 - where r10 is the last smaller one.

And I repeat what QuickPar says:

with V5: Ready to recover using 12/146 recovery blocks
with V6: Ready to recover using 47/143 recovery blocks

So you see that when downloaded with V5, exactly 12 par2 blocks needed to repair 12 files.
But why does V6 require 4x more? (I assume that 47 and not 48 required because the last .r10 is smaller - somehow the number of required par2 blocks is related to the size of the whole file - or to the number of chunks in it - but should not be!?)
BTW, the sizes of those files are 47 MB each, except .r10 which is 34 MB.


Also you see, that 3 more par2 blocks are available when downloaded with V5, that is because the first chunks are removed from par2 files as well, and somehow V6 assembles par2 files badly as well.
I PMed to you that specific case a couple of days ago.
But I think a similar picture would be with any such file set.

So you think that MemCacheLimit could affect this?
I hope I will be able to test in about 6 hours.

[billynews]

So you see that when downloaded with V5, exactly 12 par2 blocks needed to repair 12 files.
But why does V6 require 4x more?

I assume, Newsbin 6 doesn't require more Par-Blocks, it only downloads more of them.
If blocks are missing, Newsbin 6 downloads Pars from bottom to top (biggest Pars first,
smaller ones later).
Lets says Pars have 1, 2, 4, 8, 16, 32, 32, 32 repair blocks and you're missing 10 blocks.
Newsbin 5 would download 1+2+4+8 = 15 repair blocks
Newsbin 6 would download 32 repair blocks
But in both cases you only need 10 blocks.

with V5: Ready to recover using 12/146 recovery blocks
with V6: Ready to recover using 47/143 recovery blocks

This does not mean you require 12 resp. 47 repair blocks.
This means there are 12 resp. 47 repair blocks available but you may need only 1.

[Quade]

assume, Newsbin 6 doesn't require more Par-Blocks, it only downloads more of them.
If blocks are missing, Newsbin 6 downloads Pars from bottom to top (biggest Pars first,
smaller ones later).

Not really. The first thing it does is figure out how many it needs, then ask for a par file with a couple more blocks than it needs. If something goes wrong though, it will start pulling the larger PAR files looking for a good one.

This does not mean you require 12 resp. 47 repair blocks.
This means there are 12 resp. 47 repair blocks available but you may need only 1.

Really? I assumed it did. If it doesn't then you theory sounds more plausible. I've made some significant code changes in a related area because of some crappy new posting software so, the next beta might give different results. We'll see. I'm not ignoring the issue though.

[billynews]

assume, Newsbin 6 doesn't require more Par-Blocks, it only downloads more of them.
If blocks are missing, Newsbin 6 downloads Pars from bottom to top (biggest Pars first,
smaller ones later).

Not really. The first thing it does is figure out how many it needs, then ask for a par file with a couple more blocks than it needs.

Ahh, ok.
The few times I saw Newsbin downloading Par-Files it worked that way. But usually I'm not watching what Newsbin
is doing

This does not mean you require 12 resp. 47 repair blocks.
This means there are 12 resp. 47 repair blocks available but you may need only 1.

Really? I assumed it did.

You're right. It was totally nonsense I've written. Don't know what I thought there.

[RayMark]

My first test (with bad YEnc CRC) required somewhere between 1.5x -2x blocks, here - almost 4x (compared to V5).
Is it possible that when the very first segment is missing, then the file is assembled worse than when the gap is in the middle?

I tried
MemCacheLimit=100

There are two par2 sets posted, perhaps not the best test, and for some reason the required block number is different, probably one par2 set is somewhere incorrect.
Anyway, I haven't noticed that before, sorry.
So now I am getting with those two par2 sets, when downloaded with V6 using MemCacheLimit=100

1) Ready to recover using 24/143 recovery blocks

2) Ready to recover using 22/129 recovery blocks

The difference between 143 and 129 is not important, simply a different number of recovery blocks available in those two sets.

So, I assume the required block number was reduced from 47 to 24 (judging from the number of available blocks).
What this MemCacheLimit actually does?
Should I increase its value further, perhaps to 200, to achieve 12 blocks?

Interesting, this
MemCacheLimit=100
line disappeared from the ini file when I closed NewsBin !
But when I tried to download that test again, the results are the same, as if that line still is there!
Was it translated into something else in the ini file?
No, I don't see new suspect lines there.
Is it possible, that today NewsBin somehow assembles better than the last time, that a restart changed the results?
Or perhaps even rebooting of the computer?
But why the line was deleted? Not recognized, not supported? The ini file restored from a backup?

[Quade]

If you edited the NBI while Newsbin was running, "MemCacheLimit" wasn't actually used then. When Newsbin exited, it would have overwritten it. The other possibility is that it was moved onto the other "SETTINGS" section.


Your first example "Dry and Boring" Above

Set: 632 blocks
Version 6: 612 blocks downloaded
Version 5: 612 blocks downloaded

So, no difference. In both cases, 20 blocks were needed.

Your second example:

Set: 1536
Version 6: 1524 blocks downloaded
Version 5: 1524 blocks downloaded

12 blocks were needed, one per file.

So, my testing shows on my PC that 5 and 6 download the same block counts for the same damaged sets.

Your repair block stuff is neither here nor there. What's important is how many blocks of each set each version downloaded. I was hoping you'd come up with some real version 5 blocks versus version 6 blocks. You seem to think the repair block thing is that information but, it's not to me. As Billy points out, we're not really sure what QuickPar is seeing or reporting. You could also have worse results because your servers aren't as good as mine. I don't know. You can try it with B3 of you want. Dex is going to make me one for the forum.

[RayMark]

I am downloading from Giganews, and the posts are 388+ days old, not changing.

As to MemCacheLimit=100 in the ini file, yes, turns out I added that line while NewsBin was still running, and then I closed it and started again.
Sorry.

Trying again this time really with
MemCacheLimit=100

result:

Ready to recover using 12/144 recovery blocks.
with V5 it was:
Ready to recover using 12/146 recovery blocks.

So the same as with V5 now. Although the number of available blocks somehow still was 2 more with V5.
Sometimes even 1 block means all the difference.

But it is strange that a few days ago I needed 47 blocks and now only 24 even without
MemCacheLimit=100 line.

Perhaps the difference was in available memory or resources, because NewsBin was restarted and even the computer rebooted.
And/or perhaps different other applications running simultaneously and using different memory/resources.
Still, the result should not depend on such things?

What exactly MemCacheLimit does and what should be the optimal value?