Sandy Bridge and v6

[twitch]

Any thoughts on using the new instructions in sandy bridge with v6?

[Quade]

For what? AES encryption?

For things like Gzip, I'm already using some Intel libs that switch instructions depending on CPU. For some of the heavy duty stuff like Gzip. There's not much hard core number crunching except during repair. You may be surprised to to know that Newsbin spends more time allocating memory than actually crunching numbers. 6 uses far less ram than 5.59 so, everything is faster as a result.

If you have some specific suggestions, I could look into it.

[twitch]

The two broad areas where the additional cpu muscle could be used - rar repair and rar decompression. So far the numbers look excellent per the anandtech report. Definately using the multithreaded par should help. Specifically, Advanced Vector Extensions should help for floating point calculations, though I'm not sure that would directly effect the code. Ideally if it could be used with the two broad areas - rar repair and decompression that would provide the best bang for the buck.

[Quade]

Yeah, no floating point in repair.

Been toying with the idea of contacting RarLabs and see if I can license the multi-core version of the rar lib. Currently the one on the site isn't multi-core (and seems to be broken, I'm not using the latest).

[twitch]

What about using 7zip (gnu lgpl source code available) for decompression and par2cmdline multithreaded from churchsoft (not sure if source code for that is available, though they just modified the par2cmdline IIRC). These numbers are pretty impressive: (http://www.anandtech.com/show/4083/the-sandy-bridge-review-intel-core-i7-2600k-i5-2500k-core-i3-2100-tested/18)

[Quade]

Newsbin's had multi-core repair for a couple versions now. That caption's of all screwed up though. PAR isn't compression. It's repair.

As for 7zip. It's not on my radar yet. Luckily mostly everything is RARs. I'll add 7z when it gets more popular. Mostly, I hope it goes away though. For example, I loaded "Boneless" in 6 and out of 3.1 millions sets of files (about 900 days worth) there were 7000 7z sets. I guess I can look into it after 6 is out.

Mostly everything on usenet's already compressed so, compressing it again doesn't buy you much. I understand RAR files but, you're probably better off using them for storage instead of as a compressed container. Just an example, I rar'd up a 2.4 GB MKV file and it was about 40 megs smaller rared up. Barely worth the effort if you ask me.

[br0adband]

Don't mean to butt in here, but I think twitch's perspective is from using the 7Zip multithreaded ability because it can decode RAR files as well and not just for using it to compress - this is strictly about making it useful to decompress RAR sets. Just using 7Zip does not automagically imply using it because 7Zip is slightly more efficient - I'm pretty sure he was meaning that you could implement 7Zip's native multithreading decoding ability as a benefit strictly for decompressing RARs.

We all know that 7Zip is way way behind in the popularity race and it's not gaining any traction at all (even that example provided by Quade himself would make 7Zip's spread less than a drop of water in an Olympic sized swimming pool known as RAR).

But I got his point immediately: find a way to implement 7Zip's native ability to decompress RARs in a multithreaded sense. I'm not 100% sure about this, but considering they offer an x86 and x64 version, I do believe that 7Zip actually decompresses RAR sets faster than RAR itself does - I think I saw a benchmark someplace that showed something like a 12-15% speed boost by using 7Zip to decompress a RAR set over using RAR itself to do the same task.

I'm still hoping to get v6 at some point, guess I missed the sign up for the alpha or it was just given to people that contribute more to the forum, I don't know, but I've been using Newsbin for many many years now, but I thought I'd offer that perspective that 7Zip can be utilized strictly for decompressing the RARs, and potentially offer more efficient operation than a native RAR client or patch.

[Quade]

I guess you have to be a 7zip user or mind reader to get that from his comments. If he'd said, "you know, 7zip does multi-core unrars", it would have made more sense. Knowing that, I can look at the source for 7zip and see how they do it. Mostly everyone uses the RAR library which RARLabs has on their site and it's not multi-core. WinRAR 4.X does multi-core unrars.

People who've spent time on the IRC channel and contributed have been running the alpha's for a couple months now. There was no alpha signup. Still shooting to get a beta out in the next couple days. No show stoppers at the moment though there are some missing parts in the beta. Retries and automatic assembly of bad files for one thing.

Edit: That 7zip stuff looks pretty interesting.

[br0adband]

Well, I mean he did cover his idea pretty succinctly in the first sentence there:

What about using 7zip (gnu lgpl source code available) for decompression ..."

(my highlighting, of course)

which is precisely why I chimed in, seemed for some reason you missed it.

He specifically inquired about using 7zip for decompression only, not compression (which I can't imagine anyone would considering the prevalence of RAR but we just covered that multiple times in the past few posts).

And I get the thing about the IRC channel, but I was on IRC back in late 1988 before it even went "public" and helped code the development for years - in fact I was on IRC pretty much every single day from that time until around 2007 or so... almost 20 years of that is enough for this lifetime, so I guess I just miss out on the alphas for v6.

I tried to do some makeshift testing myself earlier today with a few gigs of RARs I made from a DVD5 ISO to see which could decompress it faster, 7zip or WinRAR (which is what I use, can't stand command lines anymore). On this ThinkPad T60 I have with 2GB of RAM and a 5400 rpm drive, 7zip was about 6% faster in the decompression, so not a lot to write home about but it was somewhat more efficient at getting the job done (32 bit version of the latest 7zip, on Windows 7 Professional x86, Core Duo 1.66 GHz CPU).

[Quade]

I meant specifically on our private IRC server though.

[Semel]

Retries and automatic assembly of bad files for one thing.

do u mean automatic retries of failed(for various reasons, for instance stuff is still being processed by the servers, though all headers are there, happens quite often with astraweb, it gives u 430 error) files?

UE has "retry failed articles task every" xxx
ur interval(1 min minimum). it would be nice to have it in nbpro..

[twitch]

Thanks for the clarification semel and sorry for the late reply, just got busy with school.

[-pk-]

I've used 7zip and winrar side by side for a couple years and I have also noticed that 7zip is slightly faster at decompressing rar's. But I've always been happy with Newsbin's repair & decode speeds.

For compression, 7z has slightly better compression and is faster with multicore, until you start playing with Ultra or higher settings. Normally though I still choose to use 7z over zip for quick backups, and winrar for multi-part file storage.

The one thing Winrar has over 7zip is prompting for the next file when a file is missing. 7zip just says the files are corrupt or missing and quits. This is really inconvenient if you backup a large set to 2 dvd's. With winrar you can extract from the disc without copying the parts over to the hard drive; or in this case, you should be able to extract a rar set spanning over two groups (which might be useful for older reposts). With 7zip all files must be in the same folder before you decompress. I'm not sure if this is an oversight in the lib or gui implementation, but it is something to be aware of.

[Quade]

That's good to know. I wasn't sure whether to go any farther with 7zip. The open source version doesn't seem to have multi-core support so, it didn't seem like it was worth the effort.