Newsbin Beta 6.90B9 - build 5353

[Quade]

http://downloads.newsbin.com/downloads/ ... 9-5353.exe

Lots of changes, mostly related to improving repair. Some to reduce stalling. Sometimes a badly corrupted file can cause a stall while it's being scanned. It's on my todo list.

- AutoPAR: Complete re-write - using more efficient methods. Re-includes good unit tests.
- AutoPAR: Cases were PARS were embedded in PARS could be handled better. Prior to the change the download would stick in the download list now, the bad PARS are ignored and the file will still attempt to unrar. You can’t repair a “PAR inside PAR” set.
- AutoPAR: Redo par tracker to key on parid.
- AutoPAR: Remove the ability to send pars to the wish list.
- AutoPAR: Seen a case where an existing filename in the middle of being scanned prevents assembling a file.
- AutoPAR: If real names are known from PAR files, use them instead of the filename from the file itself.
- AutoPAR: Newsbin doesn’t handle sets that all download with the same name. Rename hides them. Rename needs to rename the new files, not the old.
- AutoPAR: Remove “aggressive assembly”. - It’s the default now.
- AutoPAR: PAR DLL updated with improves PAR2 detection. Saw cases where obscured PAR filenames would be missed by the PAR DLL.
- AutoPAR: Repair can be disabled the same way unrar can be disabled but Newsbin will continue to track the files internally. It’s not persistent between runs.
- AutoPAR: Combined MP3 sets, like found in MP3 NZB and Spotnet posts are downloaded and repaired into the correct folder including the JPG’s mentioned in the PAR
- AutoPAR: Repaired MP3’s end up in both locations too.
- AutoPAR: QPMS files assembled by Newsbin can now be reported as "good" to the repair DLL. This saves a block search scan by the par dll.

-Filters: New Filter profile with subject-accept was loaded into a profile with a FIlterType of none.
-Filters: Make a new sample filters file showing examples of how to filter in and out by size, poster and subject.
- Watch: Feeding multiple groups to a watch results in too many files getting through.

-Display: Release build was stally.
-Display: When you enter watch list that contain a GOG if you don’t enter a GOG every time, it’ll clear the field. It’s not setting the GOG when the watch it initializing
-Display: Sampling from files list doesn’t work. Doesn’t generate proper samples in list.
-Display: Make servers show session/total count in the column.
-Display: Help/About shows “unlimited” when not registered.
-Display: When Download list is expanded the files are showing individual files. It’s showing a copy of file#1
-Display: Enabling SSL on an existing connection causes the server display to go wonky.
-Display: Expandable connection is back due to popular demand.

-Download: Make the unrar processor clean out all downloaded files.
-Download: File based dup-detector now renames the filename to rename the downloaded files instead of renaming the existing file on disk. To help autopar among other things.
-Download: Change how autopar rename works. Check both the par name and filename for existence.
-Download: Report that changing the download folder in the download list doesn’t persist across restart.
-Download: Using the same re-compactor logic for reading data from local header data. In that way local header data can be re-compacted with the latest code.

- NZB Files: Force compact of NZB’s might be compacting MP3’s together too. If that’s the case, maybe a more nuanced compaction is in order. No MP3’s or samples
Added a recompactor stage that’s smart enough to pull samples out of the data and give the samples their own download. Samples can still be removed with the NZB Loading filter.
- NZB Files: Certain NZB paths thrrough the code weren’t being filtered with the NZB Loading filter.
- Base: Lower cased all the DB3 extensions
- Base: Mis-counting the “rar” index of old school RARs Causes download sorting to fail. Non-fatal Just didn’t like how it looked.
- Installer: The default filter DB3 isn’t very usable.

[kenr]

B9 does't seen to follow the GOG download folder.
I have different folders for each GOG but it has downloaded to the main download folder using the GOG name instead.
These are mp3's if it makes a difference

[Quade]

Do they have PARS? PAR file containing MP3 sets end up in the unrar folders.

[jcthorne]

Thanks for the update! I am noticing much improved handling and completion of UnRAR sets.

[Quade]

Thanks for the feedback. The fact the PAR dll was missing some PAR files was pretty major.

[dot.net]

I'm having an issue where the file downloads but fails. All RARs are downloaded, but I cannot find any PAR files (in download or unrar folders). The RAR can be decompressed without error. The NZB appears to contain PAR entries.

What's stranger is that when the file fails, there is no entry in the Error column as to why. And there are no entries in the Logging tab. In fact, I haven't had a single entry in the Logging tab since I ran this program two days ago.

I've emailed the NZB file to you in case that helps. Out of a set of 8 files, 5 failed and 3 succeeded. I've sent one failed and one successful file (in case the positive case is any help).

[Moondawgie]

After updating to Beta 6.90B9 - build 5353, automatic downloads of files matching watch topics are all going to a new folder named, Watch_Topics, instead of to the individual download folders designated in the properties for each of my Watch Topics.

Sample entry in a Watch Topic for download folder:

F:\NB-Downloads\Poster - Moondawgie\\$(YEAR)-$(MONTH)-$(DAYS)\

[Quade]

I've emailed the NZB file to you in case that helps.

Thanks. I'll check it out. I wonder if maybe you have a filter that's filtering them out?

After updating to Beta 6.90B9 - build 5353,

What's the download folder set for? Sounds like maybe $(GROUP) is included.

[Moondawgie]

After updating to Beta 6.90B9 - build 5353, automatic downloads of files matching watch topics are all going to a new folder named, Watch_Topics, instead of to the individual download folders designated in the properties for each of my Watch Topics.

Sample entry in a Watch Topic for download folder:

F:\NB-Downloads\Poster - Moondawgie\\$(YEAR)-$(MONTH)-$(DAYS)\

This wasn't the behavior in previous builds of Beta 6.90, and I haven't made any changes in any filters or settings since the previous build.

Is there a quick/easy way to temporarily disable filters?

If so, I'll perform my daily run tomorrow with filters off as a test...

[Quade]

Assuming you sent me the RARs with 4 NZB's in them, none of them have PARS listed in the NZB's. I've seen and worked with sets like this before, they often have PARS but they have no regular filename.

Code: Select all

<file poster="baee5@31c12f67d2.com" date="1619141551" subject="f808f9d4eee05f8085276b6acefc9061 yEnc (1/1)">

For example, this was a PAR or NFO.

B9 build 5353 was specifically tested against these and seems to work with them. All 4 of the sets of files you sent worked for me. I just drag and dropped the RAR onto the download list. I saw par blocks count up and when it was done, they unrared. These will only work if autopar is enabled.

I'd make sure you're using 5353 because, I did post an earlier B9 last week which I quickly pulled back down after issues were discovered in on IRC.

[Quade]

F:\NB-Downloads\Poster - Moondawgie\\$(YEAR)-$(MONTH)-$(DAYS)\

Do these files contain PARS? Any files that use PARS will get moved to the unrar folder. I don't doubt what you're seeing. I'm just trying to gather information about why it's happening. I have a watch topic set to collect images for example from all groups I download headers from and they all end up in the proper folder.

I didn't say anything about filters. I don't think this is related to filtering.

[Moondawgie]

One multi-part .7z file in 7 parts (7z.001, 7z.002, etc., with one Par2 file downloaded. I don't use autounrar (though I may start given the recent changes IF they work with ebook files) and they all ended up in the new Watch_Topics folder. The other 33 files were .jpg files with no Par files.

If I'm the only one reporting this problem, please put it on the back burner while I delete and rewrite my Watch Topics after working with the now enhanced PAR and AutoUnRar features.

[Quade]

If I'm the only one reporting this problem, please put it on the back burner while I delete and rewrite my Watch Topics after working with the now enhanced PAR and AutoUnRar features.

I don't want to pretend it's not happening though.

Were the files already in the download list or added there since you updated?

I'm assuming this is a header based watch list since you're catching images?

It might be a good idea to enable AutoPAR and see if the symptoms change. I don't test much with it disabled.

[Moondawgie]

If I'm the only one reporting this problem, please put it on the back burner while I delete and rewrite my Watch Topics after working with the now enhanced PAR and AutoUnRar features.

I don't want to pretend it's not happening though.

Were the files already in the download list or added there since you updated?

I'm assuming this is a header based watch list since you're catching images?

It might be a good idea to enable AutoPAR and see if the symptoms change. I don't test much with it disabled.

The Download list was empty.

The watch lists watch for specific posters regardless of file type.

After reading the release notes I will try using AutoPar again. Not only to see if the symptoms change but also to see if that will serve my needs better than earlier versions/builds.

Good questions. I appreciate your interest and responses.

[BZee]

Tested b9 for one feature.

from: Re: Newsbin Beta 6.90B6 - Build 5303

Newsbin 6.90B6 no longer writes the descript.ion data to DownloadedFiles.txt.
I periodically move this file to a storage folder and have several years of data. Every so often I need to search through them for one thing or another.
Can this functionality be restored? I'll revert back to NB 6.82 for now.

I'll just stay on NB 6.82 unless forced to upgrade by some "must have" feature. I am working on a way to easily concentrate the day's descript.ion files into one text file.

[Quade]

I'd be happy to do away with descript.ion files altogether and replace it with a single txt file in the data folder. I'd just revamp the options so, it talks about this data file instead of the files spread all over the place.

[BZee]

I'd be happy to do away with descript.ion files altogether and replace it with a single txt file in the data folder. I'd just revamp the options so, it talks about this data file instead of the files spread all over the place.

That would mean manually adding hundreds of captions to ACDSee unless I could copy the applicable portion of the txt file, rename the portion to decript.ion, and import it into the ACDSee database. If so, I could live with just the txt file if that would be simpler for you.

[kenr]

Do they have PARS? PAR file containing MP3 sets end up in the unrar folders.

Unrar folder is unset
Yes but until now they always went into the GOG folder, nothing changed on my side.

Also doesn't seem to be autosorting headers any more.

[holtehero]

Thanks Quade for adding back the filtering on the servers, much appreciated

[Quade]

NP. It just turned out to be a failed experiment.

[kenr]

OK further investigation suggests it is only "loose" mp3's that are not downloaded into the GOG folder.
One that have par2 files work fine. (i,e. the ones that are collapsed into one item) Ones that have .svf's don't.
The par files are there but not shown in NB if I use the nzb file then it downloads all the files including par2's, but if I select the files individually including the svf it puts them in the main download folder\($GOG) instead of the GOG folder.

I can send you a sample if you need it.

[Quade]

but if I select the files individually including the svf it puts them in the main download folder\($GOG) instead of the GOG folder.

Lets get back to basics. Are you downloading from NZB files or headers/search?

[Moondawgie]

I'd be happy to do away with descript.ion files altogether and replace it with a single txt file in the data folder. I'd just revamp the options so, it talks about this data file instead of the files spread all over the place.

FWIW - It's my long-time and continuing use of ACDSee and its use of Descript.ion files to import descriptive information about files to its database which led me to first purchase a license for Newsbin way back when.

[BZee]

FWIW - It's my long-time and continuing use of ACDSee and its use of Descript.ion files to import descriptive information about files to its database which led me to first purchase a license for Newsbin way back when.

Me too. If the single txt file could be used to import the caption description into ACDSee {then exported as a descript.ion file for safe keeping} I could live with that. Too many pics are posted with names like 01.jpg - the description data is needed to differentiate and organize.

[kenr]

but if I select the files individually including the svf it puts them in the main download folder\($GOG) instead of the GOG folder.

Lets get back to basics. Are you downloading from NZB files or headers/search?

headers always headers

[Quade]

Ok I'm seeing some pathing problems now. In the midst of testing changes. Thanks.

[alexon]

I want to clarify right away: I'm comparing the previous release 6.82 with the 9th beta, skipping all the previous beta versions of the program. So I may have missed something, and please excuse me for that.
You already know about sorting problems, so I won't write about them.
I noticed that after selecting "Update All Groups" from the drop-down menu, I have to open a specific group three times to display all the messages.
The first time you open a group, the messages for the last year are displayed.
The second time you open it, the messages for the last week only are displayed.
The third time you open it, all messages are displayed.
I also regret to find that when reading the message, the service header information is no longer displayed. This information allowed me to distinguish fake posts from real ones.
Thank you for your work.

[Quade]

I noticed that after selecting "Update All Groups" from the drop-down menu, I have to open a specific group three times to display all the messages.

Particularly when you do all the groups, the headers will collect up and take some time to process. If you look down where it says "Cache: X/Y (Z)" Z is the number of header blocks still importing. Until that hits zero, you won't have all headers available. Might be interesting to wait till it hits zero then see if you have the same symptoms.

I think I know what's happening. Loading and updating simultaneously is aborting the load. I'll be if you waited till the Z count hits zero, it would load the first time. It is a bug though, I'll check it out thanks.

I also regret to find that when reading the message, the service header information is no longer displayed. This information allowed me to distinguish fake posts from real ones.

I don't know what this means. You do a Ctrl-R and you're not seeing the headers when it pops up your text editor?

[alexon]

Thank You Quade for reply.

I found out what happened.
After the update from 6.82 to beta-9, the value in the "Display Age" field has changed.
I set the value I need and now all messages are displayed.

About message headers:
I was referring to the RFC message headers.
Now by selecting "Read Post Body" or pressing "CTRL+R" I see only the text of the message or the name of the attached file,
but not the name and address of the poster and the name of its server.

[Quade]

I see headers here. Do you have your server set to "body only" download? It's in the "really advanced" options.

[Vimes]

"-Display: Expandable connection is back due to popular demand."


Most appreciated, as well as the humongous list of other fixes etc.

not had chance to test, just wanted to say thanks for that

[alexon]

I see headers here. Do you have your server set to "body only" download? It's in the "really advanced" options.

Thanks Quade.

I'm use *weka for a long time.
I don't have any checkbox marked in the "Really Advanced" window.
I didn't change anything and the "RFC" headers were always displayed in full.
Now I just upgraded from 6.82 to beta 9 and the " RFC " headers stopped showing, the message text shows only.
I went back to 6.82 and the "RFC" headers are back.
I installed beta 9 again and again I don't see the "RFC" information.

Maybe I should check some values in the " nbi " file?
I changed the computer configuration several times, switched to new operating systems,
but the "Data Folder" located on another disk never changed (I need my databases).

Version 6.9 has become so good that I don't want to go back to 6.82, but I need the "RFC" information.

[mesadude]

Loaded 5353. One problem I've noticed with the current betas, including this one, is that when I download text files, and go to open them, they have been saved with NO extension. And, some of them are blank, or empty.
When I replace the betas with the last released version, and repeat those downloads, the text files are saved properly and open as expected. Not sure if anyone else has noticed this.
Regards,
Mesa

[Quade]

I'll see if I can reproduce it. I'm not seeing either one with my setup using usenetserver.

[mesadude]

Thanks c:

[alexon]

I'll see if I can reproduce it. I'm not seeing either one with my setup using usenetserver.

Hello Quade.
I do not know what happened, but the "RFC" headers suddenly began to display correctly.
But I did something before that.
I found in the "spool_v6" folder a subfolder "Newsbin", inside which there was one folder with the name of one of the groups that I subscribed to.
A folder with the same name is located in the root directory of the "spool_v6" folder.
In the duplicate of the original folder there was a file "Storage.db3" filled with zeros.
I deleted the fake folder "Newsbin" from the root directory of folder "spool_v6" and restarted "Newsbin".
After that, I opened a random message in one of the groups and it was displayed completely.
I think during the database conversion, while updating to version 6.9, there was a crash,
after which I had a reset value in the "Display Age" field and there were duplicate folders inside "spool_v6".
Thank you so much for your work and updating "Newsbin".
And I'm sorry for your time.
-----
P.S.
Along the way, I want to ask a question about the configuration file.
I found there information about a server "Altopia" which I have not used for several years and that is not in the list of available servers in the program settings.
Is this normal or should I delete this section?

Thanks.
------
Added:

My guess was wrong. The subfolder "Newsbin" reappeared, but that's probably how it should be.
However, the headers are displayed correctly.
I don't have any ideas.

[Quade]

Altopia went out of business sadly.

Newsbin's configuration file for servers is:

[Servers]
Server1
Server2
Server3
....

[Server1]
Stuff

[Server2]
Stuff

If the server isn't listed in "Servers" it won't show up in the Newsbin server listing even if the old data still exist in the NBI files. You can hand edit the NBI file to clean out data you don't want to keep anymore.

I'd look inside the NBI and see what folder the data folder setting is set to. Maybe the NBI got corrupted and that's why you're seeing spurious folders.

[alexon]

Thanks a lot Quade.
I will definitely check this out a bit later,
as I was forced to go back to release 6.82 again due to a broken function for renaming duplicate files.
It turned out that files with the same name are not only not renamed, but are not even saved to disk.
If you check in the message list two groups of files with the same names (01,02,etc.) but different subjects for downloads, then the first group of files will be saved only.
However, all files are marked as downloaded.

[Quade]

It turned out that files with the same name are not only not renamed, but are not even saved to disk.

This is true if the files are identical. Newsbin compares the file on disk to the file being downloaded and it won't re-save one that's it knows is already sitting there. It's just another type of duplicate detection.

If you check in the message list two groups of files with the same names (01,02,etc.) but different subjects for downloads, then the first group of files will be saved only.

Newsbin doesn't care what the subject is in this case. It compares the contents of the file to the first chunk downloaded for the next file. If they are identical, not just the name but the data, it'll abort the download and keep the existing file.

I just verified this by downloading 2 files. Renaming file #1 to " File#1.org" then renamed file #2 to File #1's name. Then re-downloaded file #1. I got "Copy(1) of File#1" in the folder along with file #1. Because it noticed while the files had the same name, they didn't have the same contents.

[alexon]

This is true if the files are identical. Newsbin compares the file on disk to the file being downloaded and it won't re-save one that's it knows is already sitting there. It's just another type of duplicate detection.

Newsbin doesn't care what the subject is in this case. It compares the contents of the file to the first chunk downloaded for the next file. If they are identical, not just the name but the data, it'll abort the download and keep the existing file.

I just verified this by downloading 2 files. Renaming file #1 to " File#1.org" then renamed file #2 to File #1's name. Then re-downloaded file #1. I got "Copy(1) of File#1" in the folder along with file #1. Because it noticed while the files had the same name, they didn't have the same contents.

Thanks Quade.
I think I've figured out why this isn't happening for me.
I have disabled duplicate detection by content.
Once upon a time, this function did not work correctly for me and I turned it off.
But earlier, when the check was disabled", NB " saved both files by default. Perhaps now the logic of behavior has changed.
Now I'll try to turn it on and check it out.
Thank you again!

[alexon]

I did an experiment.
So there are two groups of images sequentially was uploaded at high speed to the server.
All images are different.
The poster is the same.
All images was captured in a short period of time by a Canon camera and have a huge block of metadata.
The file size in the message list is shown from 5 to 10 megabytes (the actual file size is from 2 to 6 megabytes).
The actual file names in both groups are: 01.jpg, 02.jpg etc.
Message template of the first group of images (as they are shown in the message list): ABC Abcdefg 123 - 01.jpg
Message template of the second group of images (as they are shown in the message list): ABC Abcdefgh-223_abcdefgh_12 a - 01.jpg
For this case i can download both groups of images only if select the checkbox for the "Short Filename Mode..." field in the "Filename Options" settings and set it to a value higher than 6 or equal to 6.
With any other configuration option, the images of the first group are downloaded only.
Version 6.82 downloads all files and automatically renames them if necessary without the above manipulations.

[Quade]

PM me the group and subject and I'll check it out. It's possible for the detector to misfire but it would require the whole first chunk of the file to be identical to some other file with the same name.

As an experiment, I tried Ctrl-Y to force the download but it didn't bypass this the duplicate check. Maybe it should.

[gonzalu]

Looks like Add Password from context menu (right click on download list) is still not working (was not working in previous two betas) ... it does work from Menu Options > Password Manager...

Thank you.

[Quade]

- Add password from the download list is to assign a password to a specific download. You can select from the list or type it in. It doesn't store the password in the password list.

- Add password from the menu stores the password in the password list. It's then checked if you add an encrypted download that doesn't already have a password.

The problem with the old way, just adding the password to the list, is the more passwords you add, the slower the checking and unraring of files will be. So the new way prefers that the password be assigned to the download. Now if a single password can be used for many downloads, then it's better to add it to the list. If the downloads use a unique password per download, it's better to assign it directly to the download.

NZB's that contain passwords, use the new mechanism and don't add the password to the list, like they used to.

[alexon]

PM me the group and subject and I'll check it out. It's possible for the detector to misfire but it would require the whole first chunk of the file to be identical to some other file with the same name.

Thank You Quade.
Unfortunately, I can't tell you the group and subject, because I don't remember where I downloaded these files from.
One of the "test" groups.
When I wrote the message, I had already deleted the files, reinstall NB and restored the "Data Folder" from the backup.
I got these files by accident during a test download.
Perhaps files of this type are quite rare. But, as I said, earlier I abandoned the function of checking for duplicates, due to false definitions of duplicates.
Specifically, these files differed from the usual files obtained from "DSLR" cameras by an unusually large block of metadata,
and it seems that they were two series of images taken by the same camera at high speed and having almost the same and large blocks of metadata at the beginning of the file.
In addition, both groups of files were uploaded to the server almost simultaneously - within 1~2 seconds.
Therefore, I assumed that the problem is precisely in similar blocks of metadata.

But the main problem is that:
Release 6.82 with the duplicate check function disabled, by default download all files, guaranteed to rename duplicates regardless of the file content.
The version 6.9 with the duplicate check function disabled, can not guarantee the download of all files for one reason or another.

P.S.
If I manage to find the group and the topic from where I downloaded those files again, I will definitely show them to you.

[gonzalu]

- Add password from the download list is to assign a password to a specific download. You can select from the list or type it in. It doesn't store the password in the password list.

- Add password from the menu stores the password in the password list. It's then checked if you add an encrypted download that doesn't already have a password.

The problem with the old way, just adding the password to the list, is the more passwords you add, the slower the checking and unraring of files will be. So the new way prefers that the password be assigned to the download. Now if a single password can be used for many downloads, then it's better to add it to the list. If the downloads use a unique password per download, it's better to assign it directly to the download.

NZB's that contain passwords, use the new mechanism and don't add the password to the list, like they used to.

Ahh, thanks... didn;t realize it changed. My bad.

As for NZBs that have the password, is this something I can look for or it is a hit or miss? My typical workflow is to search on a forum like Usenet4all or NZB.ru search engine.

Thanks as always.

[lmcmann]

Just a quick note. I'm running 6.90B9: 00 E9 C6 57 65 1E and it has a horrible memory leak for me. Never had this problem with your software before, figured I'd mention it.

I'm on Winblows 8.1, Dell laptop. If you want any specifics on anything, just let me know. I'd be more than happy to help.

After running for a few hours, it'll take more than 7 gigs of ram up. I'll try to keep track of how much data I've pulled during a session. Less than 100gigs though, and I used to pull way more than that at a time.

Larry

[lmcmann]

Quick update. Just opening NB takes 6 gigs. Once I start pulling things down, it goes to about 7. Could just be my laptop, but I don't think so. I don't have anything else that does this.

I'd backrev, but the autopar just doesn't work right in the released version. This one still gets wonky sometimes, but not nearly as often. If I run into wonky ones now, I'll send them your way.

Again, if you want any info from me, just let me know.

Thanks, Larry

[taanru]

Hi - my first time posting in beta feedback. Forgive me if it is stupid.

I download from a.b.tv. The poster I download posts encrypted files where the file names are 8 numbers (for example 20072902.part1.rar). The password is always the file name (the 8 numbers) of the file. Previous versions of Newsbin, and even previous betas would automatically decrypt the files. By "automatically" I mean that it did not require me to enter a password in the password manager. I don't know if this change was intended or not, but I thought I would mention it. I really liked the feature.

Regards
taanru

[Quade]

It's odd I just wrote an item in my todo list to try to use the filename as the password when looking at encrypted files. If it worked before it was by accident. I do plan on adding this feature but I'm not sure it'll make B10.

What version did it automatically?