newsbin.com

Lots of bug fixes in this one. Getting ready for feature freeze and move into Release Candidate mode. Next version willl probably be RC1

Summary of changes:

• Fixed long standing issue where Newsbin would hang when you press tab when focus is on the Search field.
• Fixed auto-download mode for Watch List items in Search Engine mode.
• Addressed issue where artifacts would display on the screen when toggling filters on a large post list. Not perfect but better than it was.
• Fixed Local Search on poster function.
• Fixed issue where Download List find wasn't populating the drop down.
• Added the enable filters button back to toolbar.
• Fixed issue where Files were showing par state and not download state during download
• Fixed issue in Watch tab. If you load, then subsearch and hit enter, it wouldn't re-display the data
• Fixed issue with Posting
• Fixed issue deleting files from the files list that were renamed during unrar.
• Fixed crash issue that could occur if you close a tab and there is still background processing going on that needs access to it.
• Fixed issue where the files list sometimes stops updating.
• Fixed issue where you can't add the same file to download even if it's removed from the list.
• Fixed issue with search results not becoming complete if the first search returned an incomplete.
• Made changes so a CTRL-R of the post and download of the posted file can be queued at the same time.
• Fixed issue where disabling servers from main menu wasn't re-painting the servers tab.
• Changed to how new files are counted. Only really new files are counted now. Files that get updated with new data aren't counted.
• Some changes to header loading to improve responsiveness.
• Sonarr now correctly handles when Newsbin reports files failed to download.
• Hardened header import logic.
• Changed from GZ import files to text import files. It will import both types.
• Change to how chunk cache flushes. It always flushes out at least 10%

.

The complete list of changes and the download link is on the Newsbin Beta Page

Any questions or issues, either reply to this thread or contact us through the Newsbin Contact Page

Seems like with B14 I have to close my post lists and reopen to get it to display new headers. I usually leave everything open and running 24/7. The post list will show a bunch of incomplete posts even after manually updating headers, If I close and reopen the post list they show complete.

edit: I downgraded to B13 and when manually updating the headers I can see the size increasing without closing and reopening the post list. With B14 the sizes will not increase until the post list is closed and reopened.

I've found the post-unrar filter ( e.g. [.]srr etc) no longer seems to be working for me.
Tried re-entering the details - no dice.
Generally happier with this build

Jack, I'll check it out.

Wiggins, I was just testing this the other day and didn't see any problems. I'll have to look again.

Thanks Quade.

There is a new logging entry :

ALWAYS CNNTPWorkerResolver

This repeats sometime forever, the log count reaches to sometimes 3000-4000 or more.

How we can turn this off ?

Thanks


PS : Dear Dex and Quade, may ask the on-off option for the nzb filename prefix in the next release ?

I'll be off in the next version. I left some logging on that's normally removed in the release build.

Quade wrote:I'll be off in the next version. I left some logging on that's normally removed in the release build.

Thank you

B14 was almost RC1. 6.60 is getting close to release. Basically I'm going to feature freeze it and get it ready for release so no more major changes.

1 - Found a bug in B14 during header import. I can use too much CPU if one headers files is still downloading while the rest have already imported. It's fixed it.
Has no impact on search or NZB downloaders.

Quade wrote:1 - Found a bug in B14 during header import. I can use too much CPU if one headers files is still downloading while the rest have already imported. It's fixed it.

That's not the only performance problem with B14, I tried using it today and it's completely unusable for me.

Basically anything related to viewing group headers is insanely slow, somewhere beyond 10x slower than B13 and the special 4026 version I tested, I've switched back and forth so I can confirm that the only difference is B14. No header download was going on and the cache usage was (0) so it doesn't exactly match what you describe so I though I needed to report it.

Example:
I loaded up a fairly small group, about 10k items displayed in "Hide Old" mode, 25k in "Show All", almost all "1 file". Loading it up in B13 takes perhaps 0.5 second, B14 takes 7+ seconds before it shows the first entries, with one core pegged at max during the entire time .

With the group loaded the act of clicking anywhere inside the Group window triggers another 7+ second pause with one core pegged with Windows sometimes tagging NBPro as "not responding", in B13 there's not even a cpu spike visible when I do the same!

This was with the group default sorted by Subject, if I tried to change sort I again got the 7+ second pause (similar to the group load), still with further 7+ second delays each time I clicked inside the group window. If you have a newer version you want me to test to check if the problem is still there you can PM me.

How many days are you loading into memory? I'm not having any issues. I load up 60 days at a time so, I'm trying to figure out why you might be having problems.

Click and drag to select has stopped working in beta 14 in the search tab, working fine in the download tab, reverted back to beta 13, working in search, back to 14, not working. Thanks, Rob

Windows 8.1 x64

You can make it work in B14 is you resize the columns to give you some free space on the right and start the drag from there.

I'm probably turning old drag select back on for the next version.

Quade wrote:You can make it work in B14 is you resize the columns to give you some free space on the right and start the drag from there.

I'm probably turning old drag select back on for the next version.

Yep that works thanks.

Quade wrote:How many days are you loading into memory? I'm not having any issues. I load up 60 days at a time so, I'm trying to figure out why you might be having problems.

In this case, "Display Age" is set to 45 days on a smallish picture group, IE not that much data.

I can switch back and forth between B13 and B14 and B13 is always much faster on loading that group and the GUI is close to unusable once any group is loaded in B14.

Just to confirm this, I just now loaded up 200 days worth of data from that group, B13 showed the article list after 3 seconds and after another second it showed the total post number, selecting window or specific row are instantaneous and changing sort takes < 0.5 seconds.

I then installed B14, it took 10 seconds before showing the post list, after one more second it showed the total post count but blanked the post list, after another 5-6 seconds it finally showed that everything. My focus at that point was on my browser, so I tabbed over to the Newsbin Pro window which then hung for ~7 seconds before it was usable. At that point, selecting a different row on visible part of the displayed group list takes 1-3 seconds, scrolling one window up/down one page takes 6-7 seconds, changing sort takes 8 seconds.

Next I closed the group window, set Display Age to 10 days and loaded it up again. Astonishingly it was very close to being as slow as the 200 day result, probably less than 20% faster. That means it's clearly not the amount of headers causing this.

The only thing special I can think of is that I'm doing this via remote desktop (RDP), I guess it's possible you somehow managed to totally wreck performance via RDP in B14, I will check that tomorrow when I'm in front of the machine. Other things I can think of, I do have a 192 MB DownloadMarker.db3 and 1.3 GB SIGNATURE.db3 which I guess is larger than many but not all installs.

I'm not seeing anything like that so, I'm at a loss. I loaded up 950,000 lines of image files. It loaded in 15 seconds. Then acted like normal. Quick scroll and the like.

If I had to guess, I'd guess that it's importing headers at the same time so, the import and reading the downloadmarker.db3 are fighting with each other. How about going into options, then select "Open Data Folder" then close Newsbin. Move "DownloadMarker.db3 and "Posters.txt" out of the data folder. Then restart and try again. You can move them back in when you finish the experiment. Exit Newsbin first.

Do you have any filters assigned to this group?

I kinda developed a theory. B14 loads new posts from header downloads to the top of the list instead of the old way, which was to the bottom. This will trigger an immediate read from the downloadmarker where the old version wouldn't read till you re-sorted.

Quade wrote:If I had to guess, I'd guess that it's importing headers at the same time so, the import and reading the downloadmarker.db3 are fighting with each other. How about going into options, then select "Open Data Folder" then close Newsbin. Move "DownloadMarker.db3 and "Posters.txt" out of the data folder. Then restart and try again. You can move them back in when you finish the experiment. Exit Newsbin first.

That didn't help but I kept on moving away files until it suddenly became fast again. I then moved back ALL other files except that one, still fast.

I then moved back... DownloadsV2.db3 (19.5 MB) and the start-up took noticeably longer and all the other things also moved back to being slow (group load, clicking on rows).

A quick peek inside with SQLite shows that it has 4387 Download and 4387 FileData records and it appears to contained the "Failed Files" tab. SQLite3 "Vacuum" brought it down to 6.7MB, NBPro was still equally slow with that one. I then tested the original file, selected "Clear List" which made NBPro fast immediately (without restart even) and the file was still 19.5 MB.

So it's purely the number of items in the Failed Files tab that makes B14 slow, as far as I can see there's no other factors involved. A quick check says most/all? the entries in my list are single file items which means 4387 entries aren't that much. I know both B13 and 4026 doesn't have this issue which ought to make it relatively easy to pin down. I still have all db3 files so if you have more tests you want me to run I can do that.

So it's purely the number of items in the Failed Files tab that makes B14 slow, as far as I can see there's no other factors involved. A quick check says most/all? the entries in my list are single file items which means 4387 entries aren't that much. I know both B13 and 4026 doesn't have this issue which ought to make it relatively easy to pin down. I still have all db3 files so if you have more tests you want me to run I can do that.

Good trouble-shooting. That makes sense to me too. I just have to decide how to fix it.

I changed how I check the failed list. It's much faster now.

Quade wrote:I changed how I check the failed list. It's much faster now.

By any chance is that change what has lead to there no longer being any errors/reasons displayed in the Error Column of the Failed Files Tab of Newsbin 6.60 Beta 14 for files that have failed to download? I see lots of failed files, but no reason for them having failed.

It's unrelated but a bug on my list to fix.

NbPro as stopped working......

seen this about 5 times now, seems no pattern to it. Rolling back to 13.

gkar wrote:NbPro as stopped working......

seen this about 5 times now, seems no pattern to it. Rolling back to 13.

This happened to me 4-5 times without any reason or log entry. Now it works flawless for 3 days. The first day the memory consumption was 1.8Gb now it dropped to 400-500kb

Still seeing this intermittent issue with the last PAR file I post not showing complete, pull headers again and reload group it's complete but properties show incomplete. Post is fully complete pulling headers via Forte Agent and on binsearch, no issues in 6.56 so it's something in 6.60 but no idea what as the posting is not through NB!

If you reload the group is it still incomplete?

Fisherking wrote:

gkar wrote:NbPro as stopped working......

This happened to me 4-5 times without any reason or log entry

Is it possible that it happens when it has a lot of entries in the Failed Files tab?
Then it may be the bug Quade and I was discussing earlier.

In my case 4387 entries was enough to hang NBP for 7+ seconds on an Intel i5-2500 and trigger occasional "Application is not responding" warnings from Windows 7/64-bit. Depending on how it scales (big-O) it may or may not require much higher numbers (I didn't explore how it scaled) to reach the levels you describe.

Based on what he posted this specific problem will probably be fixed in the next version, but without more data it could be something completely different too.

thx for the info, currently have nothing in the failed list though.

Quade wrote:If you reload the group is it still incomplete?

Yes it is!

If I pull headers again and reload, the icon will change to the blue PAR icon, the file will show the green IDLE/NEW icon but the file properties for the file still show a missing part! Downloading the file clearly shows a black block for the last part so NB is not reporting the status correctly as it's remembering the incomplete status even when it shows complete! If I pull headers in forte agent the file is complete and can be downloaded.

It's as though the last PAR file NB receives when downloading the headers get's the last part marked as missing as it's always the last part of the last PAR file I posted?

If I pull headers again and reload, the icon will change to the blue PAR icon, the file will show the green IDLE/NEW icon but the file properties for the file still show a missing part! Downloading the file clearly shows a black block for the last part so NB is not reporting the status correctly as it's remembering the incomplete status even when it shows complete! If I pull headers in forte agent the file is complete and can be downloaded.

I'd guess your posting program is posting an empty last part of that file. Newsbin notices it's empty and doesn't include it. There are gobs of PARS on Usenet that don't show the "last part missing" thing you're describing so, I'm not convinced this is solely a Newsbin issue. How about using agent, looking in this last part and seeing what it contains. If you download this last PAR in Newsbin does it show all all valid repair blocks?

As I mentioned, using NB 6.56 (or earlier) I don't get this issue and it's only appeared in the 6.60 betas. I've posted for years using Camelsystem Power Post (currently v2.4) and no one in the group I post to complains unless AW starts dropping parts like it's prone to doing, I also generate the PARS within CPP though!

Splitting the PAR file in agent, and downloading the last part shows it as a PAR block, PAR2.PKT header so it's got all 14 parts of the file. In NB, I can only download if I assemble incompletes, then it shows as a damaged file in QuickPAR as it only has 13/14 blocks, so NB isn't happy about something as the file has posted complete? Maybe I'll try generating the PARS in QuickPAR and posting them instead of the ones CPP generates and see what NB makes of those!!

EDIT: Posted again with PARS created with QuickPAR and this time the second to last part got missed of the last PAR file posted! Pulling headers again produces the same complete status as before with file properties showing the same part is still missing. Pull headers via Agent and the post is complete, so at a loss as to why NB is showing it as complete yet remembering it's incomplete? Nothing has changed on my end regards posting method/software that I'm aware. I'm quite happy to go back to 6.56 and try the final build of 6.60 to see if anything changes as this is such an edge case it might be best to leave it for a while.

Quade wrote:

If I pull headers again and reload, the icon will change to the blue PAR icon, the file will show the green IDLE/NEW icon but the file properties for the file still show a missing part! Downloading the file clearly shows a black block for the last part so NB is not reporting the status correctly as it's remembering the incomplete status even when it shows complete! If I pull headers in forte agent the file is complete and can be downloaded.

I'd guess your posting program is posting an empty last part of that file. Newsbin notices it's empty and doesn't include it. There are gobs of PARS on Usenet that don't show the "last part missing" thing you're describing so, I'm not convinced this is solely a Newsbin issue. How about using agent, looking in this last part and seeing what it contains. If you download this last PAR in Newsbin does it show all all valid repair blocks?

+ Engine: Failed list wasn't being preservered between runs.
+ Engine: Having many failed list items slows down all post list processing. Probably related to "failure" tagging.
* Added a new data structure. Much faster.
+ Engine: Import thread is running balls out with no delay between polls. Do I need to clean out the sema between runs?
* Added a failsafe
* Require that a file to monitor be either GZ or TXT
+ Engine: During header download updates, files aren't switched to complete - headers.
* Still an issue where posts from dead groups show up if there's overlap.
+ AutoPAR: Consider removing sliding window scanner and just leap boundary to boundary because of padding
+ Engine: Wasn't updating download status on exit.
+ GUI: Failed list doesn't display error
+ GUI: Report that if you only have one header download, the count doesn't update properly.
+ GUI: Renaming files in the files list makes the list freak out.
+ Engine: Groups list downloads don't work.
+ Engine: Unable to clear and save a cleared path.
+ Engine: Problem parsing flac files
+ GUI: Size sorting in download list is inconsistent with test files.
+ GUI: Size sorting doesn't set the download list order properly.
+ GUI: Size sorting arrows for columns are backwards

Some fixes from this week. I want the next version to be an RC so, it's cleanup time now.

Splitting the PAR file in agent, and downloading the last part shows it as a PAR block, PAR2.PKT header so it's got all 14 parts of the file. In NB, I can only download if I assemble incompletes, then it shows as a damaged file in QuickPAR as it only has 13/14 blocks, so NB isn't happy about something as the file has posted complete? Maybe I'll try generating the PARS in QuickPAR and posting them instead of the ones CPP generates and see what NB makes of those!!

PM me the name of one of these PAR sets.

Could you please explain why after a "Download all headers" on a group some files dont show? I dont have filters of anykind active. I had to run download all headers again for the files to show. BTW all headers had imported before i loaded the group(number in parenthesis showing 0)

Im downloading headers from one server AW EU

many thx

PM me some examples.

What's your "storage age" set to? How old are the files?

gkar wrote:Could you please explain why after a "Download all headers" on a group some files dont show? I dont have filters of anykind active. I had to run download all headers again for the files to show. BTW all headers had imported before i loaded the group(number in parenthesis showing 0)

Im downloading headers from one server AW EU

many thx

Will they appear if you close the post list and reopen? This is what I see with B14. I update headers, nothing shows, close and reopen the group and they all appear. This is different behavior from B13.

Quade, as per your PM, disabling XFeatures in the server options seems to have cured the 'last PAR incomplete' issue! Have posted some files and all are complete with no incomplete last PAR file this time!

Great work on figuring it out!

jackholexxxx wrote:

gkar wrote:Could you please explain why after a "Download all headers" on a group some files dont show? I dont have filters of anykind active. I had to run download all headers again for the files to show. BTW all headers had imported before i loaded the group(number in parenthesis showing 0)

Im downloading headers from one server AW EU

many thx

Will they appear if you close the post list and reopen? This is what I see with B14. I update headers, nothing shows, close and reopen the group and they all appear. This is different behavior from B13.

the files are 2400 days old ish... my storage is set to 2500 im displaying all headers. I dont ever have the group list open whilst NB is updating. I always wait for it to download then import all headers before i view.

PM me the group. I'll give it a try. Might be worth changing the "storage age" to 3000 and see if it makes a difference.

the files are 2400 days old ish... my storage is set to 2500 im displaying all headers. I dont ever have the group list open whilst NB is updating. I always wait for it to download then import all headers before i view.

I tried the group on both AW US and Giga, 2400 days +- a month or two is all there is.

working with a.b.pictures.wallpapers, but this is happening across all groups.
from what ive read "download all headers" is suppose to delete all stored posts then redownload all headers again.
If this is true then why on running it twice in a row is NB saying theres no new headers since last update(debug lvl logging). Am letting NB import all headers before running again.
If i delete a group then readd it and run "download all headers" i get the same message no new headers.
Came across after having problems getting NB to fetch all the headers in a group. Sometimes id get 800 days worth sometimes 1000 sometimes all off then(2442 days)
Not sure whats going on here, anyone any ideas?

I think your expectations are somewhat unrealistic.

If this is true then why on running it twice in a row is NB saying theres no new headers since last update(debug lvl logging). Am letting NB import all headers before running again.

1 - If you Download all Headers and get 2400 days worth, be happy and never do it again. There's never a reason to download all again unless you screw something up with the headers you already have.

2 - After a "download all headers". You have them all. "Download Latest" isn't going to pull more until someone posts more. The group you pointed me at last night is a low traffic group. The one you just mentioned sounds like a low traffic group. If nobody is posting to it, you're not going to get new headers.

I'm not sure what you're trying to do. Repeated "Download all" on a group is just going to download the same headers over and over again.

I already told you I thought your server was acting up.

am well aware what download latest does and ive never sais im using that.

As this being a server issue ive tried this on Astraweb EU and US, also on blocknews. Now I guess it could be all the servers and all the different grps im trying that are having problems, but that seams unlikely. The only reason im rerunning "download all headers" is to try to fetch all the headers and get past NB only fetching the 1st 800 days or so. If this is a problem generally with Usenet then so be it.

All the servers will have different retention. 2400 is about the farthest back in time you're going to go for binary groups. Most other servers are lower. AW and Giga are pretty close to each other.

I'm still not sure what you're trying to accomplish. It sounds to me like you've managed to get 2400 days a couple times. When you do it again and only get 800 or 1000. You're basically wiping out the 2400 days you managed to get when you re-"download all".

You could also set a 2400 day "download age" and reset the group with a right click "Post Storage/Use Download Age" and then download latest. It should pull 2400 days worth, assuming the server will actually deliver it. Doing it this way won't delete the existing headers. Using AW and Giga both I got 2400 days for the group yesterday.

thx 4 the info. After NB failed to pull headers past 800 days i was simply trying to find out why.

It appears the post counters are working properly again in beta 14.

Newsbin is still showing some small files as complete but when dl, they come up incomplete and go into failed files after my set retries. Granted this is on small files that were posted within couple of hours, but there is still a descrepency.

Newsbin is still showing some small files as complete but when dl, they come up incomplete and go into failed files after my set retries. Granted this is on small files that were posted within couple of hours, but there is still a descrepency.

Just because all the data is there in the headers, that doesn't mean the files can be downloaded. What kind of errors are you seeing in the logging tab?

Headers/NZB/Search are just the map to how to get the file. That doesn't mean the bridge isn't out 1/2 way there. Only during download do you actually know if all the file is available. If you have multiple servers, it might be on one server and not made it to the other one yet.

You kinda have to watch the log and characterize what's going on.

Just because all the data is there in the headers, that doesn't mean the files can be downloaded. What kind of errors are you seeing in the logging tab?

Came back later to the same group, seems newshosting is giving me alot of incompletes in this particular group. It almost seems if I go to this group shortly after files were posted and they were not completed, newsbin will show some, but not all files as updated or they actually are incomplete. After 8 hours one would think the server should have all the parts.

This is what logging shows me on retrying dl.

[21:49:32] ALWAYS CNNTPWorkerResolver: Set Status: 13684 2147500045

Newsbin showed this particular file as complete, granted it was around 300mb compared to say the average around 50mb, that did show as incomplete. Since I am dealing with apk files, they are generally small.

[21:49:32] ALWAYS CNNTPWorkerResolver: Set Status: 13684 2147500045

That's just a status message.

Red should probably be 430 errors in the logging tab. Black means data is missing from the headers.

Another issue!

Queued up about 12GB of files to post, posted about 5 and pulled headers in NB to see if they were complete on AW, which they were, so carried on posting overnight and shut the PC down when it had completed posting. Have now pulled headers again some 12 hours later and no new posts show up only the ones I already posted? Checked binsearch and pulling headers in Agent and the post is there and complete, but NB doesn't fetch anymore headers? Tried Download Special 50,000 Older Posts but nothing showed, is this supposed to re-download headers regardless if they've already been downloaded, not sure how that option actually works?

Right click groups "Post Storage/Use Download Age". Set the download age to a couple days. I'd guess something corrupted the range.db3 for that group.