newsbin.com

Version 6.80 Beta 1 is now available for public beta testing. It has an improved download engine that will do a better job plowing through some of the funkiness that has been showing up on Usenet lately. It may even be a tick faster than it used to be. Other than that, mostly fixes that have been reported in the 6.7x versions.

It also has changes to better handle posts coming in from Sonarr or SickBeard with obfuscated subjects. We would like to hear from people using Newsbin with Sonarr and SickBeard to see if Newsbin is working better with these style posts.

Change Summary:

• Fixed issue where Newsbin couldn't download if there was an illegal filename in subject.
• Fixed issue where the "Look in Groups" dropdown in a Watch Topic wasn't remembering the Group chosen and defaulted to searching all newsgroups.
• Removed confirmation prompt when performaing a "Mark All Old" since it is non-destructive and reversable.
• Fixed issue in AutoPar when the first par2 file is corrupt. Now triggers other par downloads.
• Now compacts all NZB records that come from Sonarr or SickBeard to solve issue displaying post sets with obscured filenames.
• Fixed issue where automatic assemble incompletes wasn't working correctly.
• Resolved intermittent stalling issue when retries fail.
• Reduced logging in some cases. Caused the disks to get hammered.
• NZB Autoload no longer looks at rar files larger than 5MB. There is no limit in the size of NZB files
• Change AutoPAR to go ahead and unrar 0 byte files. Sometimes they are meaningful files.
• The performance option "Fail Chunks with bad yEnc Formats" is now on by default. Most are correct these days. Bad ones are usually spam.
• In cases where the yEnc filename doesn't contain an extension now uses the filename from the Subject.
• Change in the download engine to download raw data and send to decoder after the fact instead of processing on the fly.
• Speed and reliabilty enhancements to yEnc, Mime, and UU decoding functions since they are now decoupled from the download engine.

The download link is on the Newsbin Beta Page. If you find any issues, please reply to this thread or use our Technical Support Contact Form.

Different par behaviour when downloading.

Loaded 22 new files to the Download list from different groups/gog and NBPro downloaded them all in sequence - all progress bars turned blue - without reassembling them.

Then went back to the top of the list and counted all the blocks downloaded for each file in sequence and then went back to the top of the list and started adding any needed par files and reassembling the posts.

Just tried another series of downloads a few hours later and the par and reassembling were back to the usual behaviour of previous versions.

Maybe something was reset when the new beta was installed and after NBPro was shut down for a few hours??? - puzzling!

You can see this if the download speed is really fast and the program is set to not pause the download during unrar/repair. Basically you run out of disk bandwidth so assembling files and handling the par processing is delayed.

In the performance options, if you tell it to pause download while doing unrar/repair this probably won't happen.

Thanks for the reply - I've got a 200MB connection so that would explain it.

I installed 6.80b1 and went to d/l a file i had previously downloaded,it wouldn't d/l no matter did,right clicked on it and selected the bypass filter and had dupe file unchecked.Reinstalled 6.73 4821 and it downloaded the file with no issues,tried this with several files with 6.8 and no go.

Is there a setting I'm missing in 6.8b1?

Is the file still sitting on disk?

Nope,was deleted a while ago.

I'll have to try it then. Sounds like it's not bypassing the dup check. As a work around you could disable the dup check in the advanced options.

I did uncheck the dupe check in options/settings/advanced so i could d/l files that got caught in the dupe check because they had the same first few minutes of video as the previous week.

I did some further test on 6.80 and found that it would d/l files which have been rared but not files which are whole(not rared) such as a mp4.

If we're talking search based, I have a fix in the works for this. It'll be in B2. Hopefully early next week.

[edit: solution to problem presented in Edit3 at the end]

Hey man!

This broke something weird for me. When I download a group of headers (18+ x2 servers for overlap) it completely freezes my entire system until the downloading data clears whatever its clogging up.

Yes, the CPU (4 cores + HT = 8 ) pegs at 100%, but no other multithreaded program gives me the same system freeze. And I do a lot of video compression, 3d rendering, etc.

The best way to describe it is that it "feels" like one of those full DMA system freezes from many years ago.

I tried SSD and sata HD and nothing changed. Went back to the previous release version and the problem went away, so it's clearly something odd with this new beta.

My best guess is something in the beta itself is freezing up when CPU usage is maxed.

This is W10x64 CU, 12 GB ram, Intel 975 extreme CPU. The only thing really customized on this would be the settings for the Realtek PCIe gigabit network adapter, which have been optimized to reduce bufferbloat across my 350 megabit DL connection. But, again, this freeze does not happen except with the beta.

Do you have an obvious culprit?

I hope this is useful beta feedback.

Edit: Back the release version, I can make the same thing happen for a few fractions of a second, so there's something about the beta that more efficient about gumming up the works. hehe I'll try making some network card changes to see if some kind of network offloading is being hammered. Will advise.

Edit2: GOT IT! It was X-features. No change on the LAN card seems to have made any difference. Based on the previous comments, I figured I'd try this option. The system comes to a complete crawl with BOTH giganews and astraweb and xfeatures enabled (disable option unchecked), in BOTH the release version and this new beta, so maybe that can help you track this down (if possible). This also explains why there's no problem with downloading actual files or any other network access. I hope this helps. Thanks again for all of your amazing work.

Edit3: I was able to find a solution at my end. Moving Newsbin to a BelowNormal task priority (via task manager/details pane) solved the problem with xfeatures on header import. The best way I can describe the problem is that the system freezes as if Newsbin's header import suddenly jumps to realtime task priority, causing everything else in my system to fight for resources. Not sure if you need to look at anything at your end (or if this is just something wrong with my system at a core level), but I thought I would post a solution in case you or anyone else finds this. In the interim, I have created a shortcut that can launch Newsbin at the lower task priority. Best regards.

I get a hard crash when downloading headers, but only for some (one?) group. I can't tell which group it is.
Logfile.txt is empty.

Also newsbin shows: Cache 200/200 (1), when there are 2 (~45MB) gz files in the Import folder. Interestingly it also showed the same when there was only 1 gz file. Each file took 30-60s to process.

Also, I have 3 providers, and I have one group (from only provider usenetexpress), that doesn't seem to download the headers. Every time I perform Download Latest Headers, it shows 0/292,467 for about 15s, then 0/157,352 for about 10s, then it goes away (for that one provider).
This occurs everytime I download latest for that group. The other providers download (just the 100 XOVER) almost instantly.

I renamed the group to alt.test in the log file:

Code: Select all

[05/02 08:43:00]  HIGH  PAR Pausing:  Disabled
[05/02 08:43:00]  HIGH  Par2Repair - Multi-core VC2015 2.0
[05/02 08:43:00]  HIGH  MessageId Filter: Disabled
[05/02 08:43:00]  HIGH  FileTypeId Version: 3.00
[05/02 08:43:18]  DEBUG Connecting: news.frugalusenet.com Index:2
[05/02 08:43:18]  DEBUG Connecting: news.usenetexpress.com Index:3
[05/02 08:43:18]  DEBUG Connecting: ssl-us.astraweb.com Index:5
[05/02 08:43:18]  DEBUG Connection Status: news.frugalusenet.com SSL Version: TLSv1.2 TLS-RSA-WITH-AES-128-CBC-SHA
[05/02 08:43:18]  DEBUG Connection Status: news.usenetexpress.com SSL Version: TLSv1.2 TLS-RSA-WITH-AES-128-CBC-SHA
[05/02 08:43:18]  DEBUG Connected: news.frugalusenet.com Index:2
[05/02 08:43:18]  DEBUG Stored Range: Min: 480373 Max: 480373 alt.test news.frugalusenet.com
[05/02 08:43:18]  DEBUG Group Numbers: alt.test Min: 432477 Max: 480373 news.frugalusenet.com
[05/02 08:43:18]  DEBUG Server:news.frugalusenet.com No new records since last header update :432477 : 480373 - alt.test
[05/02 08:43:18]  DEBUG Connected: news.usenetexpress.com Index:3
[05/02 08:43:18]  DEBUG Stored Range: Min: 135215 Max: 135215 alt.test news.usenetexpress.com
[05/02 08:43:18]  DEBUG Group Numbers: alt.test Min: 1 Max: 292467 news.usenetexpress.com
[05/02 08:43:18]  DEBUG Total XOVER Size:alt.test - 135,115:292,467
[05/02 08:43:18]  DEBUG Group Numbers: alt.test Reps: 1 news.usenetexpress.com
[05/02 08:43:18]  DEBUG XOVER Numbers: alt.test Min: 135115 Max: 292467 news.usenetexpress.com
[05/02 08:43:18]  DEBUG Connection Status: ssl-us.astraweb.com SSL Version: TLSv1.0 TLS-RSA-WITH-AES-256-CBC-SHA
[05/02 08:43:18]  DEBUG news.usenetexpress.com alt.test XOVER: alt.test 135115:292467
[05/02 08:43:19]  DEBUG Connected: ssl-us.astraweb.com Index:5
[05/02 08:43:19]  DEBUG Stored Range: Min: 683599 Max: 683599 alt.test ssl-us.astraweb.com
[05/02 08:43:19]  DEBUG Group Numbers: alt.test Min: 208177 Max: 683599 ssl-us.astraweb.com
[05/02 08:43:19]  DEBUG Server:ssl-us.astraweb.com No new records since last header update :208177 : 683599 - alt.test
[05/02 08:45:19]  DEBUG Download: Idle Connection Closed:news.frugalusenet.com
[05/02 08:45:19]  DEBUG Download: Idle Connection Closed:news.usenetexpress.com
[05/02 08:45:20]  DEBUG Download: Idle Connection Closed:ssl-us.astraweb.com

EDIT: Still occurs in B2

Both 1 & 2 crashed during updating headers so fast I didn't have a chance to tweak settings or get any clues.
Had to install 6.73 and rebuild everything to get running again.

Not getting any other reports so it's going to be hard to track down. If I had to guess, I'd guess there's some issue reading the GZ files out of the data folder. The fact the beta's crash and not the release makes me wonder if you've told you security package to ignore All three versions when reading and writing data to the data folder?

Security package?
I also went back to 6.73 and it is fine.
6.80 B2 still crashes for me.

The GZ filename will tell you which group the problem might be from. It clearly might be group specific so it might be a group I don't use.

If you figure out what group you think it is, I can try it here.

Disabled X features in nbi and now it works.

Tnx
Dave

Disabled X features in nbi and now it works.

Yeah, typically this means your security package is messing with the compressed connection. New versions of Newsbin mean you have to tell your security software (Virus scanners/Firewalls) to ignore Newsbin again. New versions look like a new program to many security suites so, you have to reset them to also ignore the new version.

Avast and Kaspersky see to be the most aggressive about messing with the compressed header connections.

finding "Fixed issue where automatic assemble incompletes wasn't working correctly" is not working using v6.80 b2

having this same issue for months with 6.7 and now 6.8;

files loaded via nzb - no passwords/no disk space issue

example:
a part of a rar file has an error downloading
no auto 'assemble incompletes' happens
expanding the file shows the rar/par files, one rar (or more) shows as incomplete
right click file and select 'assemble incompletes'
assemble incompletes finishes
auto-unrar happens

starting forcing myself to use nzbget to dl overnight vs letting newsbin dl overnight so i don't come back to see long list of incompletes (lol)

-----
oh, confirmed turning on pause download during unrar helps when moving to a FAST Dl connection, went from 150 to 1Gb and had A LOT of Blues cached (lol)

When I download in debug mode, I purposely damage every file (in software) so I force it to assemble and repair every set on its own.

You can control when it gives up and assembles with the retry setting and with "aggressive mode". I'd suggest 2 retries and turning on aggressive mode and see what happens.

Your issue isn't "assemble incomplete". The fact you can assemble them from the right click, shows that it works. You issue is letting Newsbin know when to give up and assemble the files to repair on it's own. This is gated by the retries. "Aggressive Mode" pulls the smarts out of it. It becomes mechanical. After retries expire, it assembles what it has. If the PARS are also missing chunks, they go through the same process. They retry 2 times then they're assembled too till eventually the download has enough blocks to repair or all the PARS download.

New beta should be up today. It has a work around for some obscured files that are purposely posted damaged.

thanks!
will make the change and report back
years of using (has been over a decade now? (lol)) NewbinPro and it's the best there is....

Update:
Upgraded to v6.80b4
Found Aggressive Mode was already turned on(i kind of remember reading a long ago post about turning it on to help unrar and guess i turned it on then), turned it off to see if a difference, closed newsbin, tried dl again, same issue when turned off (as expected since the idea was to turn it on). turned Aggressive Mode back on.
Thoughts?

Update 2:
with Quade's help, i found the re-tries were the issue, changing the setting to 2, with aggressive mode on was the charm...i now bow my head and walk away