newsbin.com

https://s3.us-east-1.amazonaws.com/downloads.newsbin.com/downloads/beta/nb691B7-5731.exe

Still unsigned so windows will complain during install. The new cert is on the way. I


Two fixes

1 - Column sort on download list crash

2 - Remove length restriction on obscured file renamer.

Installed and tested under Linux too. Ran some Sonarr through it as well.

Installed with no complaints from windows 11.

That's good to hear. It should have complained.

Autorar is not showing the progress of the unrar or repair.

I believe I mentioned that in the release notes that when using the 7zlib to decode, progress barely works. I haven't solved that problem yet.

Thanks for the report.

Quade wrote:That's good to hear. It should have complained.

didnt complain for me either on 10 pro

I'm finding that this build won't clean up PARs, it also seems to be d/l'ing the full PARset if it needs some PARs.
I checked that the cleanup setting is enabled. It does delete the RARs.

BTW b6 installed without any Win10 warnings, this builds' installer however was halted by Win10. (W10Pro x64) Haven't altered my AV or its' settings between b6 & b7.

Thanks for your hard work guys.

I'll check it out. It might depend on whether the PARS are grouped with the files or not.

Windows 11 balked at the install. Had to tell it to install anyway. Thanks for all the hard work you put into Newsbin. I've been with you since 2002. Damn, now I feel old.

I

wiggins09 wrote:I'm finding that this build won't clean up PARs, it also seems to be d/l'ing the full PARset if it needs some PARs.
I checked that the cleanup setting is enabled. It does delete the RARs.

BTW b6 installed without any Win10 warnings, this builds' installer however was halted by Win10. (W10Pro x64) Haven't altered my AV or its' settings between b6 & b7.

Thanks for your hard work guys. Image link not allowed for unregistered users

Hi Quade,

First of all, keep up the great work! Really love Newsbin Pro's UI and D/L speed performance!

I do want to point out 2 behaviors, which seems to have started with the 5731 build (or in one of the builds after 5651 - the last build I had installed prior to 5731):
(Note -I did not upgrade between 5651 and 5731, so not sure how 5672 and 5726 behave)

1) I noticed something along the same lines as wiggins09 above, only with build 5731:
It seems to be downloading ALL PARs, even when they are not needed. I suspect this might be why it seems to take a while now, even after the download is almost done / done (e.g. at 98% or 100%) to actually complete the download & unrar process with the latest version?
Prior versions did not have this issue.

2) Also, apparently only with this latest beta, I noticed that the PAR repair process is maxing my CPU. I have a 32GB i7 13700K and still the system / CPU temps go to very high levels during the PAR repair process.
Even though I never used it before, I even enabled the option to "Reduce the amount of CPU that PAR repair and UnRAR can use", but that also did not help.

Going to downgrade to a prior build and see if the above still happen...
I did not experience 1 previously and also, to my knowledge, do not recall experiencing 2 with prior beta versions, but not sure how many PAR repairs have been done in the past (I do not track it closely - only noticed now because my fans kick into high gear and my system fans turn red (I have it set to change colors so, aide from fans ramping up, I can easily know when CPU temps are high).

also deinstalled this "beta"

nin wrote:also deinstalled this "beta"

True but this may seem a bit harsh, Newsbin is hands down my number 1 app i use EVERY day.
I feel that the whole topic may become a bit distant to new users since there are a few conditions.....
Usenet is still one of the best things that "internet" has to offer and newsbin the best app.

PAR2's that aren't grouped with RARS are now downloaded by default. Because Newsbin has no idea whether they'll ever be used. If the files are coming in the future or never coming. The old way was to let the pars linger in the download list needing to be manually cleaned out. The old way never worked well but the new way keeps the download list cleared out at the expensive of more par downloads.

As an experiment I downloaded sets that weren't grouped properly. If I added the PARS first, they all downloaded, then the file downloaded. The file was perfect so the PARS weren't needed.

Then I added the file and the pars afterwords. In that case the file downloaded, one PAR downloaded and the rest were removed from the download list.

For the first mode, Newsbin should probably delete the downloaded pars that weren't needed because it is tracking both the PARS and other files.

Then I added a complete set of properly grouped files. When the set, about 14 mkv's finished, there were no extra PARS.

File types that don't group are problematic. Maybe I should force the ungrouped PARS to download last in cases like that. For those file types the PARS tend to be small.

There doesn't seem to be a perfect answer for all file types.

Quade wrote:PAR2's that aren't grouped with RARS are now downloaded by default. Because Newsbin has no idea whether they'll ever be used. If the files are coming in the future or never coming. The old way was to let the pars linger in the download list needing to be manually cleaned out. The old way never worked well but the new way keeps the download list cleared out at the expensive of more par downloads.

As an experiment I downloaded sets that weren't grouped properly. If I added the PARS first, they all downloaded, then the file downloaded. The file was perfect so the PARS weren't needed.

Then I added the file and the pars afterwords. In that case the file downloaded, one PAR downloaded and the rest were removed from the download list.

For the first mode, Newsbin should probably delete the downloaded pars that weren't needed because it is tracking both the PARS and other files.

Then I added a complete set of properly grouped files. When the set, about 14 mkv's finished, there were no extra PARS.

File types that don't group are problematic. Maybe I should force the ungrouped PARS to download last in cases like that. For those file types the PARS tend to be small.

There doesn't seem to be a perfect answer for all file types.

Thanks for the detailed explanation and updates Quade.

1. I like your proposal of forcing the ungrouped PARs to download last in cases for 'file types that don't group'.
I hope that this 'ungrouped PARs' scenario is indeed my specific use case that is causing the PARs to download, even when not needed. I say 'I hope' this is what is causing it, since I do not recall PARs every getting stuck in the download list with prior versions.

Hopefully related to the above, I noticed the main file download process is sitting (won't say stuck) at 98% for a while, but not sure (yet) if this is due to PAR download being pending or something else; will have to monitor more closely to get a better idea of what is / may be causing it.


2. Also, any idea why the CPU is maxing out during PAR repairs?
To provide more background, as noted above, I noticed that the PAR repair process is maxing my CPU. I have a 32GB i7 13700K and still the system / CPU temps go to very high levels during the PAR repair process.
Even though I never used it before, I recently enabled the option to "Reduce the amount of CPU that PAR repair and UnRAR can use", but that also did not help to stop it from maxing out.


Last and most importantly, keep up the great work!

Also, any idea why the CPU is maxing out during PAR repairs?

Your really shouldn't be seeing many repairs. I might get one repair a day.

That's said, repair is an intensive process and the multi-core repair code uses as much CPU as it can. Nothing has changed with the multicore repair code in forever.

So I guess I don't know. I'd have to try to trigger some repairs. It's possible the multi-core repair code wasn't working before and now it is. Maybe before it was just using one core.

Starting Downloads manually seems to work fine again... but AutoPAR/AutoUNRAR doesn't work for manually downloaded files at all?! Why is that?

Everything worked fine in 6.8x, at least for me. Then you broke it?!

Adding automatic downloads for those who want help with obfuscated NZBs is fine, but why did you have to break manual downloads completely in the progress?

I hardly ever encountered any NZB I couldn't download manually and unpack automatically with 6.8x. Now AutoUnrar seems to have stopped working completely for manually started downloads. Can you please fix that?
I'd go back to 6.8x but it was unbelievably slow to update the download list. At least that is fixed in 6.9x...

Btw. I have set AutoUnrar to use WinRAR in newsbin.nbi. Is the problem related to that?

Quade wrote:

Also, any idea why the CPU is maxing out during PAR repairs?

Your really shouldn't be seeing many repairs. I might get one repair a day.

That's said, repair is an intensive process and the multi-core repair code uses as much CPU as it can. Nothing has changed with the multicore repair code in forever.

So I guess I don't know. I'd have to try to trigger some repairs. It's possible the multi-core repair code wasn't working before and now it is. Maybe before it was just using one core.

Thanks. True, I do not get a lot of repairs percentagewise, but when they happen I know it right away.
The theory about it perhaps only using one core in prior versions might make sense.
If you do try some repairs, would love to hear your experience.



EDIT - Ironically, shortly after I replied I had a PAR repair occur and sure enough the CPU stayed maxed and CPU temps skyrocketed during the entire PAR repair.
So I edited this post, just to add that.

EDIT2 - The 'Decode' step is sitting at 99% completed for quite a while - several minutes. This happens with most decodes now with Build 5731.

Which is the most stable beta prior to 5731?

but AutoPAR/AutoUNRAR doesn't work for manually downloaded files at all?! Why is that?

I'd said that modern NZB's work best went sent directly to the download list but, for whatever reason people don't seem to believe me.

If you want to send me some of these NZB's to "ts at newsbin dot com" I can give them a try. Maybe reference this post in the subject so I know it's you.

Are all the files ending up on the same folder after download?

EDIT - Ironically, shortly after I replied I had a PAR repair occur and sure enough the CPU stayed maxed and CPU temps skyrocketed during the entire PAR repair.
So I edited this post, just to add that.

I consider this to be 100% normal. It should have always worked this way. Even if you tell Newsbin to use one less core, you'll still see high CPU usage. There should just be a core left over for other processing.

EDIT2 - The 'Decode' step is sitting at 99% completed for quite a while - several minutes. This happens with most decodes now with Build 5731.

The progress bar isn't reliable when using 7z to decode RARS.

https://forums.newsbin.com/viewtopic.php?f=43&t=48155

There's a hidden option to make it use the older WinRAR code for decoding. The progress should be correct there.

I'd hoped for a large performance boost using 7z to decode but in my testing, I couldn't measure much if any difference. I'm working on a method to try to feed the 7z library data faster. Maybe that'll make it faster.

Quade wrote:

EDIT - Ironically, shortly after I replied I had a PAR repair occur and sure enough the CPU stayed maxed and CPU temps skyrocketed during the entire PAR repair.
So I edited this post, just to add that.

I consider this to be 100% normal. It should have always worked this way. Even if you tell Newsbin to use one less core, you'll still see high CPU usage. There should just be a core left over for other processing.

EDIT2 - The 'Decode' step is sitting at 99% completed for quite a while - several minutes. This happens with most decodes now with Build 5731.

The progress bar isn't reliable when using 7z to decode RARS.

https://forums.newsbin.com/viewtopic.php?f=43&t=48155

There's a hidden option to make it use the older WinRAR code for decoding. The progress should be correct there.

I'd hoped for a large performance boost using 7z to decode but in my testing, I couldn't measure much if any difference. I'm working on a method to try to feed the 7z library data faster. Maybe that'll make it faster.

Thanks on both counts Quade!

1. Thanks for confirming the PAR repair CPU usage is normal (even when setting the Newsbin Pro option to limit CPU usage).

2. Appreciate the clarification and updates on the 'decode' step - For now will just leave the setting alone then (as opposed to changing to WinRAR) since you noted it is just a display issue with the progress bar, especially since you are trying to improve the 7z decoding speed. Image link not allowed for unregistered users Looking forward to that!

Quade wrote:I'd said that modern NZB's work best went sent directly to the download list but, for whatever reason people don't seem to believe me.
[...]
Are all the files ending up on the same folder after download?

It's not that I don't believe you. But I hardly ever encounter such weird "modern" NZBs. My Sources are mostly bulletin boards in combination with NZB search sites on the web. You simply enter a filename to get the the full set including PARs.

ALL these files have rather regular names like "ntvwGhe7srlI9rDojpIDcu5wfbrKRjhB.part01.rar" to .part15.rar. All PARs have the same basic filename as well. And all of them end up in the same folder. The only "obfuscation" is the filename with seemingly random characters. And the RAR set requires a password. My NZB Filenames are usually "title{{password}}.nzb".
I can send you some files, if that helps or I can send you some names with passwords and links to the search sites in a private message, if that is okay.

All my NZBs unpack almost without a problem when started automatically but when started manually, AutoPar/AutoUnrar doesn't work AT ALL in 6.91b7. It never even starts. Newsbin does not download additional pars but simply removes the download from the download list and the first PAR and the RAR set remain untouched in the download folder.
The only problem when started automatically is that an empty temp folder from unpacking remains in every download folder.

All these NZBs still work flawlessly in Newsbin 6.8x. I'd still use that, if it hadn't become so unbelievably slow. (Pending download list updates often take longer than the actual download. 6.9x fixed that.)

It seems to me that your attempts and focus to better handle weird, obfuscated NZBs has broken classic features and functionality of previous Newsbin versions.
What I would like to have is both. Better handling of what you call "modern" NZBs in automatic downloads, going straight to the download list, AND the functionality of version 6.8x for people like me, who prefer manually started downloads of "classic" NZBs.

And as I mentioned before. My problem is not so much that NZBs automatically go to the download list, it's that I can't choose a download folder when Newsbin is doing that.
I don't want all downloads to go to the same default folder and I don't want to change the default folder every time I want one or two downloads to go somewhere else...

Your detailed message has given me a clue about what's going on. Thanks.

Passwords are handled fundamentally differently between 6.8 and 6.9. 6.8 would dump passwords into a common password list. When it was time to unrar Newsbin would figure out what password to use from the list.

This mechanism works but, once you collect up enough passwords, the performance gets really bad.

6.9 assigns the password to each download. The old mechanism still works but ideally the new mechanism will handle most passwords. What I suspect you're seeing is that manually loading the NZB is stripping the password away from the download. It's not that autopar isn't working. It's that after repair and verification, there's no password for the unrar.

It becomes a real problem if multiple NZB's with different passwords are added to the same post listing. I might need to remove the ability to load multiple NZB's into the same post list.

I'll have to look to see if my theory is correct.

You might be right. The behavior mirrors that of previous versions if the password was missing. Then Newsbin also left the files untouched.
Can't you just use the old list for manual downloads and purge it once in a while?

Also pretty much all of my downloads use a password so the list in 6.8x should be really, really long. But I never noticed any slowdown before unpacking. Maybe a second or two between AutoPar and AutoUnrar. Unless the endless updating of the download list before a download even starts is related to that as well.

Either way, simply emptying the password list should help with such slowdowns. You could do that automatically every time the download list is completely empty. Every newly started download will have it's password added again anyway.

Whatever you do, thanks for your efforts to fix this!

Can't you just use the old list for manual downloads and purge it once in a while?

The point is NZB's don't add to the list anymore. The list now is really for just manually added passwords.

I'd rather fix it properly then hack something in there. I just need to figure out how.

I'm seeing lots of files with multiple RARS that don't uncompress automatically (multiple files). I can click one of the files and the contained contents is fine and extracts fine. It's just that Newsbin isn't extracting those and cleaning up the compressed files.

That said it DOES work on some.

What am I doing wrong?

I have currently the app crashing when it's unpacking certain files (I'm only seeing this at specific nzb's at the moment, the majority is working fine).
The log file unfotunately isn't recording any information about this crash - it's only visible within the Windows Event log (see below).

After this it's crashing every time I'm opening NB - I then have to manually cleanup the nzb files in the localappdata dir - after this it's working fine again.
I have also set "UseWinRAR=1" in the config file.
Can send you the nzb and log files privatly, if required.


Also - the exe has the wrong version number - I'm running 6.91B7, and it's showing 6.8

Faulting application name: NewsbinPro64.exe, version: 6.8.0.0, time stamp: 0x6489c53c
Faulting module name: NewsbinPro64.exe, version: 6.8.0.0, time stamp: 0x6489c53c
Exception code: 0xc0000409
Fault offset: 0x0000000000729959
Faulting process id: 0x3394
Faulting application start time: 0x01d9b48dccc0da0f

That said it DOES work on some.

If is headers or NZB files?
If NZB's are you loading the NZB into a post list before feeding them to the download list?

Can send you the nzb and log files privatly, if required.

If the NZB is crashing Newsbin, I'd love to get a copy. "ts at newsbin dot com".

How are you loading the NZB?

Thanks.

If the NZB is crashing Newsbin, I'd love to get a copy. "ts at newsbin dot com".

Mail with additional infos and files sent

Got it. Thanks.

RE crashing NZB.

I started checking. One thing I noticed is that at least one of the downloaded files ended up in "NZBAutoload" somehow. Newsbin should never download to the data folder. What might have caused the crash is Newsbin trying to read an NZB out of that file since Newsbin looks inside encoded files.

I'm going to feed the NZB in the Sonarr interface by hand and see what happens. I noticed there was no unrar folder in the NBI so I wonder if that's related. I'd going to drop that downloaded file into my NZB Autoload folder and see if it crashes too.

Quade wrote:

That said it DOES work on some.

If is headers or NZB files?
If NZB's are you loading the NZB into a post list before feeding them to the download list?

Can send you the nzb and log files privatly, if required.

It's the file clicked directly from the headers for the group list. Some don't even seem to have an nzb file. Many start with *.000 instead of ***.rar They are all multiple file downloads.

You can generate your own NZB by selecting the files, right click and "Create NZB from Posts".

You can also PM me the subject and group so it's not shown in public and I can look for it here.

Quade wrote:RE crashing NZB.
I'm going to feed the NZB in the Sonarr interface by hand and see what happens. I noticed there was no unrar folder in the NBI so I wonder if that's related. I'd going to drop that downloaded file into my NZB Autoload folder and see if it crashes too.

I think it's not the integration - it seems to be the NZB itself.
I have downloaded it directly - and the same is happening.
But it's only one particular show from one particular poster

Hi Quade,

Just reading about the recent WinRAR vulnerability and wondering if this would affect Newbin's RAR decoding as well?
https://www.bleepingcomputer.com/news/s ... -archives/

Cheers,
Rob

I'll have to look into it.

By default the current betas don't use the rarlabs code for decoding.

You can re-enable RAR processing which might be vulnerable to this. It's pretty rare to post something on Usenet with RAR recovery files. PAR's are normally used in place of recovery volumes.

Thanks for the report.

Quade wrote:I'll have to look into it.

By default the current betas don't use the rarlabs code for decoding.

You can re-enable RAR processing which might be vulnerable to this. It's pretty rare to post something on Usenet with RAR recovery files. PAR's are normally used in place of recovery volumes.

Thanks for the report.

Thanks Quade.
Can you please clarify the following (just to make sure I am not missing something): "You can re-enable RAR processing which might be vulnerable to this."
Are you referring to changing the settings back to default so it uses 7zip for decoding? Or do you mean something else?

I was thinking of just editing the .nbi file to set the Winrar option back to default (either delete it or just set it to 0).
(I had set 'use WinRAR' = 1 in the .nbi file recently due to the 7zip issue and your recommendation).

Thanks!

Yes I put it that way because I'd already suggested switching back to the WinRAR decoder.

You need to exit Newsbin first before editing the NBI but yes, that's how I'd do it.

I looked at the RAR Labs web site. It's not clear if the current source code has the fix in it or not. Or even if this source is vulnerable to this exploit at all. The bug report suggests they have to fool the user somehow to trigger the bug.

Quade wrote:Yes I put it that way because I'd already suggested switching back to the WinRAR decoder.

You need to exit Newsbin first before editing the NBI but yes, that's how I'd do it.

I looked at the RAR Labs web site. It's not clear if the current source code has the fix in it or not. Or even if this source is vulnerable to this exploit at all. The bug report suggests they have to fool the user somehow to trigger the bug.

Thanks a lot!

If I am reading it correctly, it appears that the article below (bleepingcomputer link) is claiming that RARlabs has already addressed the vulnerability (CVE-2023-40477) in WinRAR v6.23 (earlier this month)?

So I assume, given that (per your confirmation in another thread pasted below - I cannot link to it since the limit is one URL per post and I already have another link below) WinRAR is integrated into Newsbin, I assume you may need to make adjustments to the Newsbin source code where it integrates with WinRAR (ONLY IF needed - this depends of course on what in the WinRAR code was specifically changed).
Regardless, based on my understanding of the below thread (since WinRAR is built into Newsbin, I assume you would need to re-compile Newsbin's source code in order to integrate the latest updated WinRAR source code (v6.23 codebase) into Newsbin; correct me if I am wrong?

Quote from other thread:

Quade wrote:You don't have to install anything for Newsbin to be able to unrar. It's all built in. We tend to recommend having WinRAR installed simply for cases where Newsbin doesn't manage to unrar the files but it's completely optional.

https://www.bleepingcomputer.com/news/s ... -archives/
<BEGIN QUOTE>
"Mitigating the risk
RARLAB released WinRAR version 6.23 on August 2nd, 2023, effectively addressing CVE-2023-40477. Therefore, WinRAR users are strongly advised to apply the available security update immediately.


Apart from the RAR4 recovery volumes processing code fix, version 6.23 addresses an issue with specially crafted archives leading to wrong file initiation, which is also considered a high-severity problem.

It should also be noted that Microsoft is now testing native support on Windows 11 for RAR, 7-Zip, and GZ files, so third-party software like WinRAR will no longer be required in this version unless its advanced features are needed."
Those continuing to use WinRAR must keep the software updated, as similar flaws in the past were abused by hackers to install malware.
<END QUOTE>

RAR Labs publishes source code you can use in your application. That's what I'm using. It's not "WinRAR" but some of the code in WInRAR is the same.

I have the latest source but it's not published with any release notes so it's hard for me to tell of it's got the fix in it, or whether it ever has the vulnerability in it.

The 7z decoder doesn't use any WinRAR code so, you're probably safe just using the default 7z decoder in the current beta.

It sounds like unrar.dll and unrar64.dll aren't affected by this vulnerability.

https://www.rarlab.com/vuln_rev3_names.html

That's great. It's effectively what Newsbin is using.

Thanks.

This is really minor. But when downloading multiple NZB files as a ZIP file, the ZIP file no longer gets automatically deleted after being read/input. It used to always go away on its own!

Are you moving it to a watch folder or just double-clicking it?

I'm not sure if removing the zip was intentional or an accident.

Quade wrote:Are you moving it to a watch folder or just double-clicking it?

I'm not sure if removing the zip was intentional or an accident.

Its in a watch folder. And its still there after the files are loaded into Newsbin. Again, no biggie, it just used to go away on its own!

alanstarr wrote:

Quade wrote:Are you moving it to a watch folder or just double-clicking it?

I'm not sure if removing the zip was intentional or an accident.

Its in a watch folder. And its still there after the files are loaded into Newsbin. Again, no biggie, it just used to go away on its own!

Using 6.91B5 Build 5672, the NZBs from the watch folder are moved to my N:\Newsbin\Data\NzbAutoLoad directory after processing.
They are also copied to directory N:\Newsbin\Data\Nzbs\NzbAutoLoad, which contains a duplicate of everything in the above folder.
There is also another directory N:\Newsbin\Data\Nzbs\Temp, that contains a copy of NZBs that are loaded via the Load NZBs icon or the File -> Load NZBs menu item.

The contents of all three of these directories are tidied up by Newbin periodically, most likley by
[SETTINGS]
PurgeDays = X
mentioned in another thread recently (https://forums.newsbin.com/viewtopic.ph ... 73#p233073), as I have never manually tidied them and they only have a few days worth of NZBs in them.

stavros wrote:
Using 6.91B5 Build 5672, the NZBs from the watch folder are moved to my N:\Newsbin\Data\NzbAutoLoad directory after processing.

Yup, the NZBs disappear from the watch, just like they always have. Its the ZIP file that sticks around now.

alanstarr wrote:

stavros wrote:
Using 6.91B5 Build 5672, the NZBs from the watch folder are moved to my N:\Newsbin\Data\NzbAutoLoad directory after processing.

Yup, the NZBs disappear from the watch, just like they always have. Its the ZIP file that sticks around now.

Sorry, I forgot about the zipped NZBs using 6.91B5 Build 5672.

The zip file of NZBs is moved from the watch folder to my N:\Newsbin\Data\NzbAutoLoad directory after processing.
The NZBs contained in the zip file are also copied to directory N:\Newsbin\Data\Nzbs\NzbAutoLoad, but not the zip file itself.

When the zip file is loaded via the Load NZBs icon or the File -> Load NZBs menu item, the NZBs are loaded for processing,
and then the NZBs contained in the zip file are also copied to directory N:\Newsbin\Data\Nzbs\Temp, but not the zip file itself.

I haven't run version 6.91 B6 or B7 yet, so I can't say when the change of behaviour slipped in.

Thanks for all the hard work, keeping us little monsters fed!

Difficult to explain but ill try

NZB's containing passworded .7z files
download fine, look like they extract, as an extra folder is created and also deletes all the .7z files (they are in the bin)
But the folder it creates in the download folder is empty with no sign of the extracted file.

Not sure if its not extracting or putting the file some where else. Probably the former as it creates a new folder in the right place

If I use winrar and the password from the nzb it extracts fine

Does the NZB contain the password?

In the filename or the meta data?

"nzbfilename{{password}}.nzb"

Is often how they're formatted.