Anybody ready for ... "PAR3" ??

[jginnane]

In the last few days I'd been seeing some files that looked like PAR2s but with a ".PA3" extension. They couldn't be decoded by QuickPar.

Today I found out that since QuickPar "hasn't been updated since 2004", some programmer decided to come out with his new-and-improved version of a parity check program. You find/locate/install a program called "MultiPAR" (Japanese website) to replace some or all of QuickPar.

IMO, it's slow as molasses.

Under "Options" within this program you can do QuickPar-like associations with PAR and PAR2 files, integrate in a shell, etc etc. After seeing how slow it was I made sure to keep QuickPar for everything except the PAR3s.

Questions -- Since I did a search and no one's mentioned this in the forums yet -- how likely are we to see a mass conversion to PA3? Will it take six months, like PAR2 (or h264 codecs)? Is there a simple way to add lines to our Newsbin nbi files to decode .PA3 extensions *ONLY* with the external program "MultiPar"?? (Since MultiPar seems to behave almost exactly like QuickPar.)

Perhaps a final question is whether this is all someone's idea of an elaborate prank, since we're still in the April Fool month and MultiPar is so lame in its present incarnantion. I don't want to be cruel to some guy working hard to make our world a little better, but this effort is not at a Phil Katz quality level, shall we say.

[Quade]

I think the problem with PAR3 is that there's no open source version of it so, the only way to actually do a repair is to use multi-par. I'm keeping my eyes open and will support it when it becomes popular. Maybe sooner than later. It's hard to say. At the very least Newsbin should be able to parse the files and be able to determine if a repair is needed.

[jginnane]

I just did a quick check with Binsearch and see several PA3s from Dutch posters in a.b.dvd, a.b.movies, and boneless ... what I would call a coordinated effort to quickly push the extension into popular use.

From what I understand of the technical rationale, there''s good logic behind updating parity programs, especially for multi-core systems. Presently QuickPar on one BD-50 can take >20 minutes to check PAR2s on a decent system; I haven't timed your AutoPAR results because it always happens when I'm sleeping!

[Quade]

Newsbin checks on the fly so, there's never a need for a massive par scan post-process. Even the repair process doesn't' require full scan of the downloaded files. Newsbin and the repair DLL work together to avoid unnecessary scanning. This is one of the things Newsbin does that the competition doesn't. Most of them download and then have to post process scan all the files. Newsbin handles par validation of the files as they download in real time. So, it knows if the files need repair or not as the download finishes. It can download and scan the files at well over 400 Mbps assuming you have a server and network that can feed it that fast.

The lack of an open source version though doesn't bode well for it as a replacement for PAR2. It's not like yEnc where there was a clear technical advantage. Supposedly multi-par is hand tuned and about as fast as it can get. Par2 repair has supported the use of multiple cores for a couple years now.

[Shine]

It's alarming that people are already using PAR3 in the wild. The author himself says (dated June 2011):

My PAR3 sample in MultiPar is public for testing only. The proposal specifications were changed many times ago, and they will be changed in future, too. There is no compatibility between current proposal and future version. You should not use it for transport with others.

and (dated February 2012), in the release notes of the current MultiPar version 1.2:

In this release, sample PAR3 isn't available by default, as the specification is being updated.

So we can expect that these posts will not be repairable in the future unless you know exactly which version of MultiPar has been used to create the PARs. Actually, the current PAR3 spec is already obsolete, as it's being changed right now.

Quotes from MultiPar forum.

@Quade: There's no open source yet, afaik, but the author is offering it on request. Also, the (obsolete) spec for PAR3 is public.

[Semel]

that the competition doesn't.

the competition does it, all right.
usenet explorer for instance. + i have to say that its kinda more efficient with memory management and disk operations.(at least in 32bit environment)

[Quade]

Yeah? What does "it" do exactly? I was pretty specific about what I was talking about. I'm wondering if you actually understand what I said?

- Does it help the PAR library so, the data files don't have to be re-scanned for repair?

- Does it do par block detection on the fly during download?

- Does is tell the repair library to use more ram than standard to speed up repair?

I wouldn't be surprised if it did but, your comment is pretty empty. I was actually surprised that most of my competition waited till the download completes to start par checking.

[possamai]

Seeing more and more morons using par3 now...
I don't get it.. Maybe when you have an SSD but for par2 my disk has always been the bottleneck, not the cpu, not the ram, and multicore won't do shit for disk-oi as far as I know..

[Ace]

It's not like yEnc where there was a clear technical advantage.

I was hoping the clear technical advantage is that they will fix this problem with Par2, which just happened to me again on an 11GB set of files, that I downloaded and have enough blocks to repair, but newsbin and quickpar won't repair it, apparently because of this:

http://help.newsbin.com/index.php/V550-AutoPAR
"AutoPAR will not repair a file: This has been known to happen. If it does then the best solution is to use QuickPAR to check/repair the files. It is also possible to get files that cannot be repaired even by QuickPAR. This is due to a very obscure bug in the algorithm used to create the PAR2 files."

I've seen that bug enough times that for me it doesn't seem so obscure, and if Par3 doesn't fix that, I don't see the point. But wouldn't fixing that bug be a technical advantage?

Apparently the problem is worse with large block counts or large block sizes, so with an 11GB archive, you tend to get at least one or the other, if not both.

But I agree with those who said that Par3 doesn't even officially exist yet, so it's way premature to implement it in Newsbin. However I do see a need for an improvement over par2 to get rid of that supposedly "obscure" bug with par2, that's not obscure enough for me. So whenever Par3 does officially exist, I'll have some motivation to use it.

[Quade]

Sounds like the issue is making the pars not using them for repair. Considering it's a user problem most of the time. I suspect par3 won't be any better. I've seen stuff the claimed to fail to repair but actually did. The problem was the filename contained unicode and Quick par didn't support unicode (Newsbin does).

[Ace]

PAR3 should address the unicode problem because supposedly it will support unicode whereas par2 didn't (unless they end up dropping that from the new par3 spec but that would be nuts).

Aside from that, if they don't fix the "obscure bug in the algorithm" in par2 that site mentioned, when they come out with par3, then I don't see the point in coming out with par3. That and unicode support are the two biggies.

You sound a little skeptical of that claim about the bug, but if you doubt it, or haven't seen it before, I could pm you the file details if you want to see the bug (just let me know)...I don't see how this could be a user "error", though the user wasn't the brightest bulb in the box...they basically sized the blocks so the par2 files function similarly to par1 files, defeating the advantage of par2 which can use smaller blocks. However, despite the not so smart configuration of block size, it still should have worked if there wasn't a bug.

[Quade]

PAR3 should address the unicode problem because supposedly it will support unicode whereas par2 didn't (unless they end up dropping that from the new par3 spec but that would be nuts).

Unicode is supported in PAR2 through UTF8. What I'm saying is that Quickpar wasn't coded to use UTF-8. The library Newsbin uses and my own interface to it (which isn't quickpar) supports UTF-8 thus any form of unicode that can be converted to UTF-8. I've tested repair of files in full asian languages and it works fine. It works in many cases where QuickPAR won't.

I'm not skeptical that there's a bug but, I do wonder if the repair lib which is enhanced over Quickpar, newer and multi-core has this bug. If the files were generated with quickpar then there's nothing a lib trying to use these pars can do to get around the bug. On the other hand I think mode bulk posters aren't using Quickpar to gen the repair blocks. Right now there's one guy doing PAR3 and he's not releasing code. To me, that suggests it'll never be as successful as PAR2.

[Ace]

I'm not skeptical that there's a bug but, I do wonder if the repair lib which is enhanced over Quickpar, newer and multi-core has this bug.

That's a good question. I wish I knew the answer, but I don't.

If the files were generated with quickpar then there's nothing a lib trying to use these pars can do to get around the bug. On the other hand I think mode bulk posters aren't using Quickpar to gen the repair blocks.

I opened the par2 file that says it has enough blocks to repair, but then the repair never completely succeeds. (When it's done "repairing", quickpar still says it's ready to repair). That par2 file says at the end: "PAR 2.0 Creator QuickPar 0.9" so you're probably right there's no way around that bug.

I'll have to open some par2 files from bulk posters and see if they say at the end what created them like this one...I don't normally look inside the par files.

Regarding the ultimate success of par3, I have to agree it doesn't seem to be off to a very good start, but shine said the developer IS releasing code on request:

There's no open source yet, afaik, but the author is offering it on request.

[driverdude]

Though I wrote tools to do Par2 and rar for my own needs, and distrust output of ANY program for usenet that seem to overly rely on Yenc and coverup issues at times, I do have a knee jerk opinon on PAr3 after looking at its spec very briefly :

PAR3 IS A PILE OF CRAP!

Its faster in theory (huge files) because its DEFECTIVE and dangerous. It was not thought out well.

A couple reasons proposed Par3 is worse than GENUINE and only legitimate Par2 (my thoughts) :

1> You can safely scan a hard drive of millions of files and sort them into par2 sets first by 16K md5 checksum (billions multiplied by billions of times more unique than crappy 32 bit CRC3 from the 1980s) because MD5 initial fingerprint in Par2 is 256 bits not 32bits like crappy Par3. This allows you to distrust filename in OS (filenames typically mangled if Japanese or Russian, sometimes french too messed up like an accented version of "Sènégal.mp3") and if file is slightly damaged, par2 set will match up because you can ignore the EXPECTED filelength too. Only crappy standards such as SFV uses crc32. Antoher problem with CRC-32 standard is it allows two files containing a datastream of pure nulls, to yield the SAME crc-32 even if one file has a thousand or a million more null characters in the stream than the smaller file.... because the CRC-32 standard does not salt or force programers to add some 111111's or invert initial seed buffer for speed. Admittedly , and humorously many crc-32 libraries invert the initial buffer so a stream of 0's will be unique, but that has been a pet peeve of mine for decades, even though I always initialized and seeded with 1's. This Par3 spec specifies CRC-32-IEEE802.3, which by law means not to EOR (exclusive OR) or invert but rather to properly seed with 32 1's and "unreflected". At least THAT part is not retarded, and XOR with 0xffffffff constantly in EVERY call, in case programmers were newbies, is avoided for ultimate crc32 speed.

2> Par2 HAS to have a universal ASCII approximation file name MANDATORY unique per set, and an OPTION full blown UNICODE-16 support. Crappy Par3 proposal has only UTF-8, though both UTF-8 and UTF-16 can of course hold within them UTF-32 and UTF-16. UTF-8 is risky because most UTF-8 libraries do not do filename matches on decomposed tokens (accents separate from letter e for example, and various stroke marks separate from root japanese). Mac OSX for spite decomposes Unicode a lot jsut to keep programmers on their toes. I wrote my own decomposition library tools years ago, for string compares, but the BEST is to use ASCII FILENAME APPROXIMATION and force programmers to use it and that is what PAr2 does as one of TWO simultaneous filenames. Par3 dangerously insists solely on buggy UTF-8 because it is a spec written by a Japanese guy angry at having to deal with ROMAN usage of words in programming world. Worse... becaue he does not understand dangers of sorting decomposed unicode, he actually insists file numbers be assigned based on file name SORT order and allows UNICODE to participate. This means running areference test benchmark on two OSes (Mac OSX vs Windows 7( could result in two different sets of par3 results that would have been less likely if files were not sorted based on RAW UNICODE BIT STREAM!!! (Unicode allows OPTIONAL BOM markers at start and middle of streams, though UTF-8 has no need of BOM markers, it has many other pesky things. U+00E9 is equal to U+0065 followed by U+0301, but sorting in THIS crappy spec would make a missort, without knowing which way an OS decided to decompose or not. Its not a MAc problem, in this case its LINUX issue, Mac is always 100% pure and decomposed unicode for 10 years but linux is arbitrary and almost random.

3> The maximum file size storable in Par2 spec is TWICE the size of PAR3, hardly a step forward. Yup. Par2 uses UNSIGNED 64 bit lengths for all vital fields such as filelengths.

4> FFT in Par3 made grave shortcuts for Bluray rip archivers of public domain bluray copies of their own wedding videos. Long explaination : FFT of 2's powers can be done with GPUS and GPU libraries (Free in Mac OS via Apple for NVIDIA and AMD(ATI), fallback OPenCL instead of from GPU if needed : http://en.wikipedia.org/wiki/OpenCL), and free in ARM in iPad and iPhone (OpenCL , + vDSP API , among others), and Intel corp of course has CPU assisted vector code for FFT on Windows, and 3rd party FFT on NVidia and AMD. Also in Windows you can make a joke out of traditional GPU or vector CPU FFT assists and get 106 GIGAFLOPS :http://www.bealto.com/gpu-fft_intro.html with NVidia CUFFT . FFT is used in Par3 and Par2 because of POSSBILE hardware acceleration even if no programmers do it yet in most Par2 implementations, but the sad part is Par3 makes more shortcuts sacrificing the future, for options of assiting FFT. One laziness example : Par2 allows 65534 repair slices of data so that if you want to you could take a 45 gigabyte file, break it up conceptually, and is 25% pars desired, then 65536 108K (108 kiloBYTES sized) repair parity records can be provided in Par2 to go with the precious problematic 45 gigabyte source data (also broken up conceptually in 32K pieces usually). In Par3 you are only allowed HALF the number of repair slices, another step backwards!

5 > In Par2 a recovery set ID is critical for file parings and is 128 bit MD5.In Par2 it is some sort of perverted joke that it is a paltry 32 bit CRC-32! Not kidding! Par3 files are NEVER supposed to be mixed with other par3 files on the same computer directory ! Ha!

6> In Par2 EVERY packet , every record, nearly every field is protected for safety by many 128 bit MD5 checksums, in Par3 it is billions times billions of times less secure! (Hilarious CRC-64). Yes I always check every on in my par2 code.

7> in Par2 a misanamed par2 header file with no file extension such as mywedding_par2 instead of mywedding.par2, can be identified rapidly by OS tools by a special 16 byte header at the top of the file. Par3 arrogantly has NO VERSION number of ANY KIND in the par3 even though par2 has a 2.0 in it that could have been changed, and par3 has.... I m not joking... only 6 bytes in file front as an identifier. 6 bytes a person could accidentally type in a text file header in amessage to a buddy, because historically though NULL is rare in a text file, it is not illegal and is a "NOP" on lineprinters and teletypewriters. PAr2 planeed on safe 16 byte identifiers not insane 6 byte file headers for critical main vital par file. (true, both allow multiple copies in multiple files, but what a pain).



Half here, half there , half half half! Everything about Par3 is cutting Par2 quality in half for THEORETICAL speed gains at expense of critical accuracy and quality. True..... BOTH allow full file MD5 of files up to 63 bits in size, (Par2 allows double)... and thats good enough for ULTIMATE final safety. But too many intermediate steps are skimped on buy this half baked 2001 standard of Par3.


I admit i see a FEW advantages in Par3 that could have been added to Par2, but the LIES and laughable clinging at straws in the spec is stomach churning. For example, in cyptography you don't rely on MD5 alone for digital signatures int eh last 5 years because tools exist to create a SPECIAL datatastring that has special padding at end of TWO filestreams (not ONE), to specially make two files have the same MD5, just for funny proof on concept, so long as you can control the padding of the end of the datastream.... This author uses that to beat his chest and claim his speci is BETTER than par2 slightly because he offers a ridiculous 1980s crc-32 checksum in addition to MD5 full stream that both par2 and par3 offer. He types this claim as "it may be more difficult to create a fake file with same MD5 and checksums". Hah. The effort to pad the datastream with with output from a crc32 spoofer and a MD5 is less trivial and therefore "harder" but hardly cryptographically secure! Its the almost the same endeavor overall.

I apologize for all the typos... And I am NOT a Par2 authority, only a one time implementor of sorts. But I am not wrong, Its just that it is afact that this 2001 Par3 originated spec is a NOT AN IMPROVEMENT over Par2 and should be IGNORED and killed off.

I beg the developers of Newsbin to resist supporting PAr3 only to handle the error DETECTION aspect of it (it is easy to support if for error detection, in fact is is backward compatible slightly for that)0, but 100% ban this abomination. It should also warn people to stay away from adopting crappy so-called Par3.

Par3 is a crappy abomination.

Par3 is a travesty and a pack of dangerous lies. I am not a luddite, I do not resist change or technology, its just that Par3 is a crappy spec compared to par2.

=driverDude

[driverdude]

I never proofread the thing I just sent 100 seconds ago, but now reading it, swap "'128" for MD5 of course in the above. MD5 is of course 16 bytes as always. typo.

[Quade]

Your comments were interesting if a bit hard to read. The bottom line though is that, I'll support par3 as soon as it becomes popular enough. I don't care about whether it's good or bad or not.

If the users use it, I'll support it. I have no agenda. I ignored the protest against yEnc and just did it. Par3 will be the same way.

[julie11111]

You guys, they aren't 'PAR3' files. They're 'pa3' files. There's no 'R'.

[DThor]

That's just window's standard for a file suffix, which is 3 characters. They are par3 files, the actual suffix is pa3, yes. It's no different than '.txt' is a text file.

DT