Newsbin Version 6 Beta 13 - Possible bugs.

[sourceavenger]

Hello,
I am posting this message because I have been running into a few repeated problems that are making it difficult to download content using the "Auto" mode lately. These problems are the following.

1) When I have newsbin sort the files by the largest file size first it appears as if a bunch of par files end up being there which never assemble or error out. Basically it downloads half of the data it says is for the whole download(theres two identical files to be downloaded) and then it just stops. After a couple hundred of these get downloaded my downloads come to a complete halt. The bar is always purple similar to downloading files that are a group of files via the search function.

2) When I have newsbin arrange by the smallest file size I get files that download for example 30kb/1.1mb and the download stays in the queue and never get removed due to errors or anything. Basically they get up to 15 retries then just sit there and bring my downloads to a halt. The bar showing the file progress is black and grey(not sure if that means anything).

3) Newsbin has been having issues with having multiple servers work correctly. For example Giganews + Newshosting works fine but Giganews + Newshosting as master servers and Tweaknews and xsnews as slave servers(priority doesnt effect this bug so far in my testing). Adding tweaknews, or xsusenet always ends up killing all my downloads. Basically ill go from 60-70 Mbit/s to 0 and my giganews and newshosting connections all die. This isnt fixed until I disable the tweaknews or xsusenet servers and then disable and re-enable giganews and newshosting. I have tried multiple priority levels(1-10) none of those effected if the setup worked or not and setting them as slave servers didnt change anything either. I cannot figure out why newsbin doesnt want to work with those two services but I am thinking somehow newsbin is getting "stuck" on connecting to either of those services and it endlessly keeps trying to connect, making me lose my giganews and newshosting connections.

4) Somewhat of a bug, I have to use a memory cleaning/releasing program to automatically free newsbins memory because it goes from 30megs of memory to 6gb's within just a few hours otherwise. I tried changing the cache chunk size but it didnt change much. I am unsure if theres a memory leak somewhere or if newsbin just queues up way more memory than needed but I really cant do any automated downloads without a memory optimizer.

Thank you for your consideration of these problems and I look forward to a response .

EDIT: I forgot to mention one more possible bug I ran into that was rather important. I was downloading using the auto setting allot over the last few days and I decided to try and have everything save to one folder. After an hour or so of downloading newsbin crashed, and after multiple reruns of the application it still kept crashing. When I changed the save settings back to saving the files based on the groups there in(so its different folders) it fixed the problem. Is there a possible bug in a limit on how many files newsbin can save in one given directory? I am just curious as to if thats a possibility.

Heres my system information to see if it helps:
Windows 7 x64 Ultimate Edition
VyprVPN service using the OpenVPN 256bit protocol.
8gb Ram
Amd Phenom x6 1100T Black Edition
Nvidia 580

[dexter]

1&2) What kind of filters do you have on auto mode? It sounds like you are trying to download all kinds of garbage posts. Black in a progress bar means the post is incomplete. You could turn aggressive assembly mode on to make Newsbin assemble these posts but I think you are using auto mode too broadly. Auto mode works best if you give some criteria for the type of content you are looking to download.

3) If you check the Logging tab when this happens, it should give a clue as to what's going on. Could be it is erroring out on those servers.

4) Similar to #1 and 2, I think you are just pushing it too hard downloading everything in sight. All this stuff gets queued into memory when processing. If you have a high traffic group and you are loading up millions and millions of headers, you are going to run yourself out of RAM no matter how much you have. You are doing commercial grade processing on a consumer grade system.

As for the folder thing, Windows has an upper limit to the number of files that can go in one folder if you are running FAT32, like if you are downloading to an external USB drive or something. I'm not aware of a limitation like this in Newsbin but, again, you are really pushing the limits here.

Quade may have more to contribute to this but that's my thoughts.

[sourceavenger]

1&2) I am using a rather aggressive Pictures filter, what I mean by that is I am accepting 6-7 image formats and then denying 48 filename and subject name words and file extensions. Specificially I block .rar, .par, etc. The groups I am downloading from is 1 at a time and their picture groups.

3) I finally managed to figure out a temporary fix for this one, I have to let newsbin boot and establish the connections to Giganews and Newshosting first and then enable xsusenet and tweaknews after.

4) I definatly agree that im pushing the ram really hard but how many groups I have, the memory freeing software works well though to counteract this problem(unless freeing newsbins memory is somehow causing problems I am unaware of).

I agree about the files per folder limit thats what my original guess was aswell. Its a NTFS external drive I am using and my chunk cache is stored on my internal drive. As a fix I store the files under \$(LONGGROUP)\$(Subject)

[Quade]

3) Newsbin has been having issues with having multiple servers work correctly. For example Giganews + Newshosting works fine but Giganews + Newshosting as master servers and Tweaknews and xsnews as slave servers(priority doesnt effect this bug so far in my testing). Adding tweaknews, or xsusenet always ends up killing all my downloads. Basically ill go from 60-70 Mbit/s to 0 and my giganews and newshosting connections all die. This isnt fixed until I disable the tweaknews or xsusenet servers and then disable and re-enable giganews and newshosting. I have tried multiple priority levels(1-10) none of those effected if the setup worked or not and setting them as slave servers didnt change anything either. I cannot figure out why newsbin doesnt want to work with those two services but I am thinking somehow newsbin is getting "stuck" on connecting to either of those services and it endlessly keeps trying to connect, making me lose my giganews and newshosting connections.

Shitty servers can absolutely gate your downloads. Here's the issue.

Newsbin won't declare a chunk to be "bad" till it fails all active servers. If you have a couple servers that are hard to connect to or slow. Everything will wait for them to finish processing. Newsbin assumes either the server can deliver a chunk or it can't deliver a chunk. It assumes that connectivity is good and that the servers will respond in a timely fashion. I don't use either one of these servers and you do have a bunch of other stuff going on so, it might not be just the servers but, you sort of imply they aren't the best servers.

The acid test might be, how will does it download when using only those servers?

As for the ram use. The biggest RAM user is the download/wish/failed lists. How many items do you have in the lists? Just as a test, you might try removing your virus scanner/firewall, reboot and see if ram use still goes out of control. Most security software simply isn't designed for industrial levels of network traffic and sometimes they cause problems like this. If the problem goes away, you might re-install and see which one is doing it. Maybe upgrade or try a different security package.

[sourceavenger]

I agree that the XSusenet and tweaknews servers may be why everythings coming to a halt I will delete those ones at once(there free accounts anyways, just kept repeatidly signing up after each one finished its days or data quota lol). And it downloads way bette rjust using giganews and newshosting than when I add the extra ones in there.

I did figure out though whats going on with the multi-part downloads with the Auto function. Theres 2 files with the same exact name, size, etc which are exactly the same. Newsbin downloads the first one but the second one stays in a downloading state. Basically the second file never downloads and never gets removed.

As for the ram issue I have my firewall and anti-virus software all disabled during downloads, I have any uneeded system processes closed and any background software that is unneeded aswell. I actually have my windows 7 box down to 35 or 36 processes before I turn on my vpn and newsbin pro. Is there any harm in me freeing newsbins memory during auto-downloads? My download queue hits sometimes 200,000 to 300,000 at times. I thought about an easier solution to limit ram usage but its up to the developers of newsbin if you all would consider them. Here was my ideas.

Hidden Settings that could improve performance if they were added:
1) Allow disabling of the Failed File list via a setting under Performance.
2) Ability to disable logging all together via a performance setting(between this and the failed file list I think performance for heavy auto-downloaders could greatly increase)
3) Ability to limit the queue for the downloads list to a set limit, any other downloads above that limit would just be marked to be added later, I think if a check interval of 5 minutes to add new downloads after the limit was hit would be an effective means of not having a continous check and adding being done.
4) Ability to filter multi-file downloads using a filter so auto-downloads of pictures can avoid a descent portion of the "junk" posts that halt newsbins downloading.
5) As for more performance options, is there a way a setting could be added to the Performance section of the nbi file and be able to turn off displaying of the server connections in the gui(under server tab).
6) Ability to disable thumbnails all together(a "none" or "0") option under the switches and number of thumbnails tab.
7) Ability to disable listing of the "files" tab, this would help kill the eventual huge list of downloaded files that hog memory.

[Quade]

As for the ram issue I have my firewall and anti-virus software all disabled during downloads,

Disabling isn't enough. They use kernel drivers, you can't really disable them.

If Newsbin doesn't crash when you free this ram, it means Newsbin isn't really using that much ram and something else is. You can't just remove ram that's being used without also causing a crash. I'm assuming something else is going on and that this 6 gigs isn't really being used (or that you're not really freeing the RAM but, forcing it to page to disk).


1 - You can tell Newsbin not to save the failed list. I had it self trimming but, many people bitched.

2 - Logging tab is self cleaning. You never get that many entries.

3 - No, this is your responsibility. Newsbin does what you tell it to do. That's what people want. If you want to reduce what goes into the download list. Add some filters. A size filter is very effective.

4 -You can already to this with filter profiles assigned to the auto-download group.

5 - How about clicking the X button to close the window? You might have to unlocks the GUI. The connections tab really isn't performance intensive. It only updates every 4 seconds.

6 - I can consider this.

7 - Files list is self cleaning. Once it hits a number of entries, it purges the oldest.

[sourceavenger]

I agree the kernel drivers are still loaded. I'll uninstall my av/firewall and see if theres a noticeable difference.

And I agree about the newsbin ram usage it doesnt crash from cleaning it so somewhere its allocating more ram than needed.

1&2) Yeah I have the failed file list not save on close. Ahh I see so you had it auto-cleanup before and people disliked that? Thats okay then I completly understand. And yeah the logging tab can only hold 4000 entries then it flushes a few hundred or so. I was only curious as to if a person were to disable logging, failed file list, files listing, and thumbnails if memory usage would be greatly reduced and if performance would increase(less disk io, less cpu usage, etc).
3&4) I have a size filter already in place, is the cross-post filter a way to filter multi-file posts I assume?
5) I always have it closed, but tnx for the headsup on its refresh time.
7) I was unaware of this, I normally have it closed but thanks for the headsup .

[Quade]

If by multi-part you mean RAR files. You can add a subject reject entry.

"[.]rar"

for example will reject any file with ".rar" in it. Don't use the quotes. Make sure you use "subject" based filters, filename filters only apply after the download completes.

[sourceavenger]

Heres actually what I meant, the top image is an example of what I mean by multi-part downloads getting stuck. Those 2 files in that one download never get completed. The first image out of the 2 gets downloaded and the second one just sits there with "Downloading" as its status endlessly. After enough of these my downloads completly freeze up till I manually remove them.

The second image is what I meant by a download that gets stuck and keeps retrying over and over, it gets up to 15 or 16 retries normally and still keeps counting down. The downloads get completly halted after enough of these are at the top of the dl list(unless I baby sit and manually remove them).






As for the [.]par and [.]rar files in the filter, I block those along with other files I thought would end up being multipart downloads.

[sourceavenger]

I know this is a double post but I wanted to update you. I am still running into the problem of my downloads getting stuck. Basically I uninstalled my anti-virus and firewall just to see if that was causing my issues but it had no effect. The multiple downloads of pictures in 1 post still eventually locks up my downloads. Also, the incomplete downloads(the ones where it downloads for example 60kb/1.1mb) is still locking up my downloads as well. I have to sit and remove them after around 200 of them reach the top of the queue because it always stops the downloads completly.

I also uninstalled and reinstalled Newsbin Pro, rebooted the pc, I even messed around with a bunch of the filter settings(cross post ammount, Age, file sizes, etc).
I basically just want to completly block multi-part downloads because for some reason they get stuck with the images. I already block [.]rar and [.]par it doesnt seem to stop the downloads from still getting added.

[Quade]

I'd probably re-download headers for the group and see if you still get the doubled up images.

[sourceavenger]

I went ahead and took your advice about redownloading the headers. I tried that with a few groups but still hit the same problem. I know it has to be something small but no matter what I do I cannot figure out a solution lol.

[Quade]

PM me the group. I seemed to have missed it here. I'll download headers and see if I can figure out why they're doubling. It's probably because they were re-posted but, re-posts shouldn't combine.