Version 6.54 Release Candidate 1 Available...

[dexter]

We are moving into Release Candidate status. This one fixes a few more issues and we're ready to move into production so Quade can focus on 6.60 and other projects.

Here is a summary of changes:

• Fix for potential crash issue when processing NZB files produced by NewsMangler
• Fix issue where icons on Search tab would change if you switch to a post list and back.
• Fix Read Post Body to show header text if there is no body data - specifically (0/1) posts.

You can download this release from the Newsbin Beta Page. If you find any issues or have questions, either reply to this thread or send us a message through our Technical Support Form.

[magician1962]

There seems to be a delay between selecting from a group and appearing in the download tab at first start. Sometimes it can be minutes. Not sure if it is a problem, or if it is background housekeeping taking place.

[Quade]

I'm not seeing it. How much do you have in the download list?

[magician1962]

There are 500 in the wish list and the Download Files tab is empty at start. Today files added went straight in, so it is variable. I will monitor to see if any pattern occurs.

[Quade]

It's probably a function of how busy the disk drives are.

[magician1962]

The drive is only used for download data, so shouldn't be an issue, with almost zero fragmentation.

It does seem to be time related. On one occasion I added 5 files and they began downloading. I then proceeded to mark dozens more, but they did not appear in the list till the list had cleared. That may be coincidence.

Gut feeling is that background tasks are happening after start up, causing the delay of appearing..

One other thing I have seen a couple of times now, which may or may not be related.
All parts downloaded, but sitting there. On right click run quick par it indicates that several parts are missing (red)

[magician1962]

More info on the issue.

Opened Newsbin for daily download session. Added 1 file for it to get on with. Went straight into download files tab. Deleted from Group tab.

Now added one at a time about a dozen more. Each indicated it was queued for download, Deleted from Group Tab. I also deleted all the SPAM from the Group, and stuff of no interest, bringing down from some 400 to around 130 items in the Group Tab. Queued items still not in Download tab after several minutes.

Decided to shut Newsbin down, and restart. On Restart all the SPAM and files I had deleted reappeared in the Group Tab. Added all the files I wanted again as they had also re-appeared in the Group tab. It was about 7 minutes before the files appeared in the download Files tab.

Looking at the Log
[07:21:51] HIGH Aggressive Assembly mode: Enabled
[07:21:51] HIGH PAR Pausing: Enabled
[07:21:51] HIGH Par2Repair - Multi-core 1.5
[07:21:51] HIGH MessageId Filter: Disabled
[07:21:51] HIGH FileType Version: 2.01
[07:28:11] LOW Connection Status: SSL Connection Server: news-europe.giganews.com SSL Cert: news-europe.giganews.com Stats: AES256-SHA 256 TLSv1/SSLv3

I have been using Newsbin since I believe V4 but never encountered this issue.

[Quade]

. Deleted from Group tab.
Now added one at a time about a dozen more

As I suggested, I'm pretty sure it's disk IO. Your comment just tells me what probably triggers it though. When you add something to download, the data to download has to be pulled from the group database. When you delete something, it also has to be done to the group database. My guess is your method of add then immediately delete is what's causing the slow feed.

Not sure there's anything I can do to make it faster. As an experiment, how about adding without deleting and see if it remains fast?

[magician1962]

The problem seems to have gone away, and my normal method of use works fine. Weird!

[Co6aka]

Some weird problems have been going on for a while now, and continue with this version. But, I'm running under WINE so I realize MMMV... FWIW:

Currently: Newsbin 6.54RC1 64bit on Xubuntu 14.04 64bit. Latest WINE from WineHQ PPA.

With current and last few versions, at first everything seems to be working as expected, but then a 10060 error message displays and Newsbin gets funky: I start seeing "ERROR AutoPAR: PAR file is Damaged, Unable to Add Parfile" messages even though AutoPAR is disabled; and, then I start getting lots of incompletes, most of which only have one or two missing chunks; and, when I eventually select "assemble incompletes" it doesn't work -- the incompletes remain onn the downloading tab; and, then if I highlight and select delete it asks if I want to delete the incompletes from disk. Deleting them and everything in the queue and restarting clears the problem, until the next 10060 error message.

Also, starting with the current version there are no new headers for the last three days in any newsgroup! (Astraweb) But, I can DL 0-day stuff via NZB files though.

Also, other weirdnesses: Even though I select some files to DL with Ctrl-Y (bypass) the duplicate detector prevents their download. And, sometimes when I type Ctrl-Y the group window jump-scrolls to the top instead of downloading the highlighted files.

[Quade]

How many connections and what news servers are you using?

Had a user report somewhat similar issues which went away when he used fewer than 10 connections.

Also, other weirdnesses: Even though I select some files to DL with Ctrl-Y (bypass) the duplicate detector prevents their download. And, sometimes when I type Ctrl-Y the group window jump-scrolls to the top instead of downloading the highlighted files.

If the files already exist in disk. 6.54 won't re-download them so, that might be what you're seeing. Newsbin checks the first 600K or so of the file to make sure it's just not the same filename.

[biosteel]

An issue which has been around since v5, incomplete files, doesn't matter if the nzb is missing chunks or the server, do not get assembled, clog up the pipeline and have to be assembled manually.
Overflowing the chunks folder despite ram caching, making any double digit gb download with a hefty amount of incompletes (I am looking at giganews) a chore to download with quite a bit of manual labor involved.

Ideally the incompletes should get written with padding, filename all uppercase, chunks always remaining in ram, _never_ hitting the chunks folder.
Ie after retries on all servers have gone through and if it is certain the net or connection hasn't gone down of course.

Hitting the chunks folder can put too much strain on an ssd for what it is worth. Especially nowadays with MLC gaining traction.
I've had to clear hundreds of thousands of chunks from that folder on too many occasions, which could have been avoided.
Not to mention the slowdown and the windows I/O issue of too many files in a folder.

Also an issue, because incomplete files are not written to disk as they used to, par repair takes longer as it prefers to fill a missing file completely rather than the chunks, making repair impossible unless a manual "assemble incomplete" is done.
Aggressive assembly mode does not help as one would expect or want it to.
On low power cpu systems the added par2 calculations also hinder performance.
There really is no need to excessively waste cpu processing time this way.


Another issue is the passwords, as is, the lack of assigning passwords to a nzb set slows down things a lot, with ie a password file of 15+kb it is apparent that newsbin no longer directly tries the assigned password but rather all passwords in the list.
It should be possible to change this back to first use an assigned password then the list if needed.
We need to be able to verify assigned passwords, too. Having a column to display it directly is also a nice idea, but i guess overkill for the added comfort. If it isn't it would be really nice to see.


Par2 processing lacks gui feedback, it looks as if Newsbin is stalled, while it does verification/password cracking tasks, this should be adressed, really.


Anyhow, looking forward to a robust fully working Newsbin release.

[Quade]

I understand your core issue which is that incompletes aren't assembling. I'm unclear why that's happening to you.

For assemble incompletes to kick in:

1 - Has to have PAR files (files that exceed retries with no pars are failed to the failed list).

2 - Won't try until retries have expired (I use 2 retries).

3 - > 90% of the file blocks have to be present (file blocks, not PAR blocks).

The key is that retries gate Assemble incompletes. If all these things are true, it'll assemble incompletes and attempt a repair. So, knowing this, the question is why does it decide not to assemble? If you look at these incompletes that are hanging out, why do you think they're not assembling? More than 10% missing? Retries not expiring?

I'm not averse to removing #3. Download what I have, run the retries and if I have pars assemble and attempt repair. If you only use one server then you're sort of at their mercy anyway so retries are unlikely to fix anything.

When incompletes are assembled, they're padded.

Another issue is the passwords, as is, the lack of assigning passwords to a nzb set slows down things a lot, with ie a password file of 15+kb it is apparent that newsbin no longer directly tries the assigned password but rather all passwords in the list.
It should be possible to change this back to first use an assigned password then the list if needed.
We need to be able to verify assigned passwords, too. Having a column to display it directly is also a nice idea, but i guess overkill for the added comfort. If it isn't it would be really nice to see.

Do you actually have 15K of valid passwords in the list? Assigning passwords does nothing these days. The password is assigned using the first chunk of the first par file. There's no cracking. If you don't have a valid password in your list, it's going to be failed.

We need to be able to verify assigned passwords, too.

Any files with invalid passwords get kicked into the failed list as spam automatically. Unless you force the download that is.

Also an issue, because incomplete files are not written to disk as they used to, par repair takes longer as it prefers to fill a missing file completely rather than the chunks, making repair impossible unless a manual "assemble incomplete" is done.

Chunks on disk have nothing to do with par repair. Same for chunks in memory. Only completed files get scanned for par blocks. Where the chunks are is transparent to that process. Maybe you're talking about re-constituting files using just the PAR files.

SSD's aren't as delicate as you might think:
http://techreport.com/review/27062/the- ... fter-1-5pb


I'm not averse to making changes for you. This just doesn't happen to me, probably because I use multiple servers.

[Co6aka]

How many connections and what news servers are you using?

Astraweb; 8 to the main, and 2 to the European. Gonna try reducing it to 6 and 2 or 1.

If the files already exist in disk...

They don't, that's what I'm trying to accomplish.

The ctrl-Y jumps to the top thing is something else though. Seems to happen every time it's first invoked in a group or GoG, then seemingly randomly thereafter.

[CoolStream]

For assemble incompletes to kick in:

1 - Has to have PAR files (files that exceed retries with no pars are failed to the failed list).

2 - Won't try until retries have expired (I use 2 retries).

3 - > 90% of the file blocks have to be present (file blocks, not PAR blocks).

A couple of things I observe is that before download is attempted the main PAR file has to have successfully downloaded. If that main PAR doesn't download completely, then it's a no-go for any further downloads of parts to automatically proceed. I can't seem to find anything in the settings that can change that.

SPAM FILTERS
I appreciate the great amount of work that has been done to filter out the great amount of crap that now exists on UseNet, however I have noticed a few recently that Newsbin has flagged as suspicious or containing executables. I only use Newsbin to process NZBs which I spot and link to on a Dutch site. My checking tools are Easynews site which allows me to search for completed posts. Easynews creates a thumbnail of the completed post - if the thumbnail is good, I know that it is safe to attempt to download. I create the NZBs from information in the post headers on external NZB sites.

Here is an example NZB. It shows on Easynews as being an album of MP3s. The post comprises RARs and PARs which Newbin 654RC1 flags as

I hope this is useful.

[Quade]

A couple of things I observe is that before download is attempted the main PAR file has to have successfully downloaded. If that main PAR doesn't download completely, then it's a no-go for any further downloads of parts to automatically proceed. I can't seem to find anything in the settings that can change that.

I've addressed this already in 6.60. Basically any errors in the par files will trigger the next PAR file to download.


It's against the rules to post what you're downloading here. I'd rather you PM'd it to me so, it's not out in the open.

I'll check out that file. The spam filters are a running battle so, I'm not surprised that it misses sometimes. Just something that needs constant improvement.

Thanks.

[CoolStream]

Oops sorry. Thanks for modding my post.

I'll PM you with such details in future

[magician1962]

Quade, looks like I finally found the error I was having, wasn't due to the way I was selecting, but was effectively an intermittent I/O issue. It looks like it has been interfering on and off for some time, but finally became more permanent, so easier to find. A Block failed on the disk, right where the recycle bin was, so as soon as Newsbin tried to write there after decoding, it had been failed. System back to normal after drive replaced.

Keep up the good work.

[Quade]

Nice. Good trouble-shooting.