Unstable Speed (Slow downs)

[Tanoshimi]

I have always had a problem with Newsbin Flaring up at start to (close to) my internet connection speed, then throttling down to about 50% of it or so. About 2 weeks ago, I upgraded to Verizon FIOS 150Mbps. I also upgraded my Newrazor account to the Unlimited with 50 connections, and changed Newsbin to reflect that. I'm DLing to an internal hard drive, and I'm connected directly to the router. Speedtests consistently come in about 155Mbps. So what am I getting in Newsbin? About 27Mbps. Again it starts off with a burst of about 90, then drops down and sits around 48Mbps or 27Mbps, which can be slower than I had before. Here's what I've tried, to save time:
- All the various ports
- SSL
- Using less than the full 50 connections
- Contacting Newsrazor for help

What I haven't tried yet:
- Mixing the local and European server with 25 connections each
- Mixing the local and European server with 20 connections each
- Giganews

I don't rightfully expect to get the full on 150Mbps. I'd settle for 100Mbps. Heck, I'd settle for a consistently high value (70-100, if it stayed there).

Just wondering if others have this kind of fluctuations in their speed, and if anyone's got ideas for me to get stable, higher numbers.
-Chris

[Screedo]

I only use 10 connections with SSL enabled, I have 20 in total, with astraweb and i'm maxing my 100 Mbit connection, try to use less connections.

[Tanoshimi]

Nope. No good. Tried SSL with 25 out of 50 connections. Same result. 40 and 27.

[dexter]

We know Newsbin can handle those speeds, I'd look at the following things:

1) Your ISP, do they throttle Usenet connections? Many in Europe do. Sometimes you can get around it by running on an alternate port. If your news server supports it, try port 443 instead of port 563 for SSL

2) Your news server may not be up to the task. Try a free trial with Giganews and see if you get your full speed. If you do then it's time to consider a different news service provider

3) A local virus scanner can cause download speed fluctuations. Temporarily disable any firewall, virus scanner, or Internet Security software and see if you get full speed.

4) Is your machine up to the task? Look in the status bar at the bottom of Newsbin and see if it is eating into cache or if you have backlogged header imports slowing you down. Look for something that looks like "Cache: 80/100 (xx)" in the status bar. If the first number is a 0 then you have a disk I/O bottleneck and your download speed will be impacted. If you have number in parenthesis to the right, it means Newsbin is trying to import headers in the background. That could slow things down as well.

[Quade]

First thing I'd do, is I'd try a different server. Maybe Giga or AW euro server. Giga has a 3 day trial so, you can test for free.

You ought to be able to get the full 150. It might take two servers to do it. I regularly test Newsbin at 400 Mbps download speeds but, my machine is a beast. I have 100 Mbps comcast to the net and, other than Sunday nights, it's always humming along at 100+ Mbps with 20 connections.

Keep in mind connections don't equal speed. If the server can't feed you fast enough then no number of connections can solve that. If you watch the download, look down at the status bar at the "Cache" line. If it drops down to zero during the download, the problem is disk IO. If it stay at (100-connection) count (50 to 50 connections, 80 with 20 connections) then the problem is the net speed.

[Tanoshimi]

All good information. I was going to try Giganews, but before I switched to them (they're more expensive, and I've had Newsrazor for years), I wanted to make sure it's not Newsbin, mostly due to the fluctuations in speed.

I have 75/100 for the Cache, not sure if that's good or bad.

I can say the html direct download (when I find a server fast enough) gets me the full on 150, so I don't believe it's the PC, the drives, or the internet connection.

Verizon doesn't throttle or bottleneck things in this area. At least, not in my experience.

I've disabled firewall and virus scanners when I read it in previous threads. Again, to no avail.

So basically, it boils down to whether or not the server can handle giving me the speeds. I certainly do not want to pay for 2 services to get the speed I want, since most of my downloading takes place in the evening when I'm not at the machine. I do grab stuff during the day manually, and for that, the faster the better.

I appreciate everyone's input. I'll see if I can try Giganews for free, and if that's the issue then I'll post back. But if Giganews (or Giganews + Newsrazor) can't get me stable speeds, or speeds above 1/4th of my actual ISP speed, then I'll need to come back to Newsbin as a potential culprit.

{EDIT}
So Giganews gave me more over the 100 than I was getting before. Actually Giganews + Newsrazor, since I didn't deactivate it. But I still got a lot of slowdowns. I'm starting to think it might be the drive I'm sending them to. I've recorded my speed meter, so you can see what speeds I'm getting, and maybe why it's slowing down, but right now my server is down for maintenance. I'll change the Default download location to another place, and then try again. If I still cannot whip this, I'll post my meter video.

[itimpi]

If the left hand figure for the cache value stays above zero, then it is not disk I/O that is slowing you down.

Note that quite a few ISP's give have the concept of 'burst' mode where a connection runs at one speed for a few seconds, and then after that is slowed down. This is intended to give excellent response for interactive traffic such as web pages, but limit the load any downloading software with long-lasting connections can impose on the network. Not saying this is your problem, but you might want to carry out tests other than how fast HTML pages can be retrieved.

[Tanoshimi]

Well, with some experimentation, I believe I've determined it's a combination of Newsrazor not putting out big speed, and the drive I was saving it to. So we'll switch to Giganews for better speed, and download to the local HD.

Question: Is there a way to move completed downloads to another folder? I know I can unrar things to a different folder, but I want to download to C:\Downloads, then move the completed folder to S:\Downloads.

[Quade]

Just spec the download path in the options, then add an unrar path to the options too. Then the downloads will go to the download path and the unrars to the unrar path.

Maybe I'm not getting the spirit of what you're asking. Why not just download to the final destination?

[Tanoshimi]

I can't do that. Well, I could, but I don't want to. Here's the situation. Originally, I set S:\Downloads (on my network drive) as the download location, because that's where I want everything to be. I recently discovered that my download speeds to that drive are about 1/5th of what they are if I download to the local hard drive. I want to download to the local drive, because I want the speed, but I need the items to end up on the network drive so the local drive doesn't fill up. As for UNRARing to the network drive from the completed download, that leaves behind everything that is not RARed. In the case of video media, the folder, fanart, nzb, sfv, and nfo files would all stay on that local hard drive, filling it up slowly, while only the video itself would be extracted to the final location. Further, not everything I download gets UNRARed. Some things just download straight in, and they would never make it to that network drive.

No, what I need is for Newsbin to move the entire folder after the download is complete (preferably after the UNRAR process). I'm a decent programmer, and I have written plugins for other software, if that's an option here? I would think Move Completed Downloads to xx Folder would be a standard item, or at least something others would've requested by now.

Any help?

[Quade]

In the case of video media, the folder, fanart, nzb, sfv, and nfo files would all stay on that local hard drive,

Newsbin moves anything mentioned in the PAR file to the unrar destination too.

Seems like you have to eat the slow network drive on the front end or the back. You either take the hit during download or take the hit in the subsequent file copy. The only way you'd get full copy speed is if you didn't do it while Newsbin was also downloading. There's only so much disk bandwidth available. I imagine the process of copying the files off the local disk to the net drive would slow download too.

In the performance options:

1 - You could try turning off buffer mode and see if it makes any difference.

2 - You could tell Newsbin to stop writing the download to disk while it's repairing/unraring.

3- You could increase the internal chunk buffer size so, it potentially keeps much more of the file in memory. That's the thing in the cache line. I have mine set to 2000 right now. That gives Newsbin up to 1G of memory to buffer the downloads into.

I'd restart Newsbin to be sure the settings took.

In 6.50, you'll be able to decouple the download chunks from the assembly. So, you could download chunks to the internal drive and have them assemble on the network drive. It's not clear how much of the slowdown is because of PAR processing. The file has to be scanned to find all the par blocks. That might be what's slow about the network drive..

[Semel]

i would advise against giganews.

if you really wanna go giganewish route then go for supernews. its the same giganews only without crap features they use to justify outrages prices.

[Tanoshimi]

Well I found out that with my existing Newsrazor account I was able to get those 150Mbps speeds as long as it downloaded to the local hard drive, not the Network Attached Storage drive I want them on.

Quade: Some pretty technical advice. I'll go through and try it later today. As far as the UNRAR goes, I just know that I ended up with two download directories full of stuff. The final destination had the stuff in the RAR, and the original download location had what I call the garbage files that aren't needed. Maybe those were all from people who did not list all the files in the PAR, but it is what it is. I also occasionally download files that are not compressed/broken into RAR. So that download directory would contain those items, as opposed to the final destination. To be honest, if Newsbin moved the fully downloaded folder over, I wouldn't care if it did it before or after the UNRAR, but it seems like after would be a good time. I could use third party software (or write a small utility) to monitor that directory and move files over, but that's starting to overcomplicate what should be simple.

So I guess, at least for now, the final answer is, Newsbin can't move the entire download upon completion? Just the UNRAR location?

[DThor]

I think the problem is that the notion of 'complete' isn't trivial. Of course, you can pick a specific post and say "it's done", but there's typically another post right behind it that's already started before you finish saying that. So, you're still going to have downloads and local file copies stomping all over each other, right back where you started, no? The only way around that is to clear out the download list first, which to me means when newsbin is finished, move all the contents of the download/extract folder to the NAS, and don't start up any more downloads until it's done. That's a pretty specific behaviour, and even a 3rd party approach, like running rsync, will still collide with downloads.

Maybe, like Quade, I'm not getting the point, but the goal is to... keep download speeds maxed out all the time? That's a reasonable expectation, to me it needs either a faster NAS/network (I have things copying to my WDTV, which is hardly a workhorse, and I don't notice any hits. Network? Then again, this may be related to your fatter pipes...), or keep the copying process completely separate from the downloading.

DT

[Tanoshimi]

Well all of my devices on my network, including the NAS, are 10/100/1000, so I wasn't expecting the problems I'm getting. Going directly to the NAS drops speeds to like 1/5th of what I get downloading to the local C: drive.

As long as there are other processes in place, aside from downloading, you're going to take a hit. So whether it's unraring, moving completed items, or an anti-virus scan, you have to accept those things, or turn them off. I will say this. When I put everything to the local drive, there was no slowdown or lag, not even during an unrar to the same local drive. That is, Newsbin was downloading and unraring at the same time, and my speed remained consistent.

As for knowing the download is done, I'm pretty sure Newsbin can tell when all the files from the NZB are complete. It moves them from the download to the completed tab file. So, simply put, when it moves it to the Completed tab, it could also copy it to the Completed folder. That's all I want.

Adding a third party to the equation would be a nightmare! For another piece of software to be able to tell when something is done downloading is not a simple thing. About all they can do is watch all the file sizes in a folder, wait until there's been no increases for something like 5-10 minutes, then TRY to move the files over. Even then, no guarantees. This is the kind of thing that could be handled better internally.

So maybe there will be some collision with copying files while downloads come in (or an option can be checked/unchecked to suspend downloads during copies or unrar), but it's no different than the collision we already get during an unrar. If fact, I'd venture to say copying would be significantly less detrimental to the download process/speed, as a simple copy is much less processor intensive than an unrar operation. Especially, if that unrar's to another location.

[dexter]

Version 6.50 will support what you want. You'll be able to specify a location for downloading parts and a different location for assembly into files.

We are working on version 6.41 now but the focus of this version is minor bug fixes for 6.40, not major functionality changes. No ETA on when 6.50 will enter the Beta process.

[Tanoshimi]

Excellent! Maybe I could join the team and assist in some of the lighter lifting.