Download rate problem

[kai]

Hello,
First post here.
I noticed the next behavior when I download binaries with Newsbin: downloads fast let say 1 GB, then take pause while chunks are merged into a single rar file, then download fast again and so on. I looked in Options and didn't find something related to this. With Alt.Binz, the speed is constantly 30-35 MB/s for the entire download session and chunks are merged into final files in parallel with downloads.

[Quade]

What kind of drive are you downloading too?

1 - In the options select "Open data folder".
2 - Exit Newsbin.
3 - Look for "Newsbin.nbi"
4 - Edit it with wordpad and add:

[PERFORMANCE]
ChunkCacheSize=400

To the file. Then save, exit and start Newsbin. Down at the bottom it should say 400/400 near the cache line. Try some downloads.

Newsbin tries to keep partial files in memory so unraring can happen during download without there being any impact on the download speed. This CacheSize controls how many decoded blocks of the file it'll keep in memory.

[kai]

Before I have started this topic, I've tested on 1 TB HDD Samsung F3 1TB filled at 80% from its entire capacity and on RAID0 SSD (2xOCZ Vector 128 GB each one). Download was 35 MB/s and after 30 seconds it stopped to allow those 1GB chunks to be merged (even on RAID0 SSD, it took 20 seconds or more to merge the chunks). I looked on Task Manager and the CPU never exceeded 20% load during chunks merging.
With
[PERFORMANCE]
ChunkCacheSize=400

in Newsbin.nbi, pause is shortened to 10 seconds (tested on HDD with data folder on HDD not on RAID0 SSD), but right now, my news server capped the bandwidth to 11 MB/s.
I'll try again after midnight when the speed is 35 MB/s or more.

Edit: I forgot to write that with
[PERFORMANCE]
ChunkCacheSize=400
the chunk folder will still be filled to 1GB before chunks will be merged into a final file and the unwanted download pause will occur.

[Quade]

How many chunks does each individual RAR file have? I'm guessing something like 1700 or so. I that's the case, 400 isn't enough. 2000 might be too many depending on how much RAM you have.

Are you saying each RAR is 1G?

[kai]

It looks like rar size from queue is 1GB. I got 16 GB RAM.

Edit: after the midnight, when the cap is gone, I saw 60 MB/s speed and more, using Newsbin. Alt.Binz can't go beyond 34 MB/s (that was the average speed measured with a netmeter for 1 minute for Alt.Binz).
Newsbin is the tool to go, but it need some tweaks to take advantage of those fast 16 GB RAM (2400 MHz, CL10).

On the last attempt with Newsbin, after downloading 4 files (1GB/file), download had stopped and almost all files from queue becomes red (spam), even if I have disabled all checkboxes from Spam Filter Settings. Testing the same nzb with Alt.binz had worked fine.

[Quade]

It looks like rar size from queue is 1GB. I got 16 GB RAM.

Hit the pause button. Then add a set of files to the download list. Down in the download list hit the + icon to see the individual files. How big is each one of those files? 1G seems pretty big. The cache only has to hold one or at most 2 files at a time. With 16 gigs of RAM. You might want to try 2000 chunks in the cache.

On the last attempt with Newsbin, after downloading 4 files (1GB/file), download had stopped and almost all files from queue becomes red (spam), even if I have disabled all checkboxes from Spam Filter Settings. Testing the same nzb with Alt.binz had worked fine.

Red doesn't mean spam. Look in the "Logging" tab. My guess is you're getting errors of some kind. Maybe too many connections errors.

[kai]

It looks like rar size from queue is 1GB. I got 16 GB RAM.

Red doesn't mean spam. Look in the "Logging" tab. My guess is you're getting errors of some kind. Maybe too many connections errors.

It was written spam and files was red.
I disabled some dup check and the errors disappeared.

Last tests:

Without editing of Newsbin.nbi, I got 34 MB/s on RAID0 SSD, average speed during 5 minutes, measured with a netmeter application (10.8 GB download in 5 minutes, which should be divided to 1.07 to obtain the correct amount of data downloaded, from my observation during years).

Using ChunkCacheSize=2000 in Newsbin.nbi, I got 40 MB/s average speed during 5 minutes. In this case, I saw peak speed of 72 MB/s.

The problem is that I didn't saw in Task Manager from Windows 8.1 write speeds more then 90 MB/s on the RAID0 SSD array, which means that Newsbin can easily continue to download even when chunks are merged in a final rar file, because the storage has plenty of resources (RAID0 SSD array can write 800 MB/s using 256K files even on qdepth 2).
So, I think that the unwanted download pause during chunks merging can be eliminated when downloads are processed entirely on SSDs.

From my tests, if I use ChunkCacheSize=2000 in Newsbin.nbi, Newsbin and Alt.Binz are on par in terms of speed if tests are done on SSDs.

These tests was done using a gigabit connection @home.

Thanks for reading and replies!

[Quade]

If you watch the "cache" it'll show you if it's running out of blocks. When it unrars, it keeps downloading but it doesn't write the downloads to disk. It keeps them in memory till the download completes. My experience is that high speed download and unraring at the same time will cause the unrar to sometimes take twice as much time. One thing you can do it download to one drive and unrar to another. That should be the fastest.

[kai]

If you watch the "cache" it'll show you if it's running out of blocks. When it unrars, it keeps downloading but it doesn't write the downloads to disk. It keeps them in memory till the download completes. My experience is that high speed download and unraring at the same time will cause the unrar to sometimes take twice as much time. One thing you can do it download to one drive and unrar to another. That should be the fastest.

I already disabled par/unrar.

[kai]

Using ChunkCacheSize=2000, chunks were still written to disk.

Using ChunkCacheSize=10000 I reached 42 MB/s average speed during 5 minutes test (data folder and download folder were located on the same HDD). That average speed was achieved because chunks were not written to HDD, but on RAM. Even if chunks are not written to HDD, when the final file is written to HDD, I saw a drop speed to 3 MB/s.
After 10 minutes from the beginning of download, I saw that Newsbin started to written chunks to HDD and the speed drops pretty seriously (with peaks over 40 MB/s, but which can drop to ~200 KB/s).

[Quade]

I have a feeling no chunk cache size will be large enough. That the download speed will always exceed how quickly the files are assembled so, you'll eventually run into a full cache. The only way to fix it will be to try to speed up file assembly. I might have to make the cache smarter so, I can find the chunks faster.


Windows actually writes to files twice.

1 - Internally it zero's out the data on the disk before writing data to it.

2 - Then your data gets written to disk.

The reason Windows does this is to prevent applications from seeing old data that might be left over on blank parts of the disk. I don't actually know if windows does this on SSD drives. It wouldn't make sense to because the OS has no idea where the SSD writes data. Actual data layout on an SSD is completely controlled by the SSD.

Newsbin had a mode where it disabled this double-write. You had to run it in Administrator mode to give it permissions to disable it. I'm looking to re-instate this mode in 6.60. For your 1 Gb RAR files. I think the double-write might simply be too expensive. Newsbin also had a mode where it would pre-allocate the file so, it was almost always contiguous. For SSD's this doesn't matter but for spinning disks it does. Pre-allocating will hurt performance on SSD's. That's one reason I disabled it.

[kai]

If it's necessary, I can send to you a nzb file that contain those 999MB size files anytime you want and if you want. It's not a hurry.
Thanks!

[farmwald]

I saw this post about pre-allocate being bad for SSDs. I have put in a fast SSD as the target for Newsbin downloads, and so I thought I'd clear the pre-allocate flag in Performance.

I can't unset it! If I uncheck the box, exit options and then go back and look, the box is checked again. Is this right?

Also, a related question, I have "Pause Downloads during RAR/PAR" unchecked. I would expect it to continue writing RARs for the next post while unRARing the current. This should be ok, as the SSD is pretty fast, and is unRARing to another, slower, spinning disk, so the SSD should have extra write BW. I'm pretty sure Newsbin is pausing the download anyway, at least when the Cache goes to zero. It doesn't start writing until the unRAR finishes.

Thanks.

[Quade]

can't unset it! If I uncheck the box, exit options and then go back and look, the box is checked again. Is this right?

I'll check it out. You should be able to clear it.

We've been discussing this in another topic. If you want to have both happen you'll probably need to run 6.56 till I get the code changed to allow it. I honestly wasn't expecting it to be the big deal it's become. I don't download large files and didn't take that into account.

[Fisherking]

can't unset it! If I uncheck the box, exit options and then go back and look, the box is checked again. Is this right?

I'll check it out. You should be able to clear it.

We've been discussing this in another topic. If you want to have both happen you'll probably need to run 6.56 till I get the code changed to allow it. I honestly wasn't expecting it to be the big deal it's become. I don't download large files and didn't take that into account.

Same here. Can't unset it. Also I've sent a message to support about a problem for sorting Download Path in the DL list. It doesn't work. Dex replied me about it. Any news about this ?

Thanks

[Quade]

Same here. Can't unset it. Also I've sent a message to support about a problem for sorting Download Path in the DL list. It doesn't work. Dex replied me about it. Any news about this ?

I'm sure Dex said something like "we're looking into it". I'm not clear what news you're looking for. To fix it, code will need to change meaning it's probably not an NBI tweak. My guess is the download path is sorting on "pre-rendered" path. Meaning $(GROUP) instead of the actual group name.

[Fisherking]

Same here. Can't unset it. Also I've sent a message to support about a problem for sorting Download Path in the DL list. It doesn't work. Dex replied me about it. Any news about this ?

I'm sure Dex said something like "we're looking into it". I'm not clear what news you're looking for. To fix it, code will need to change meaning it's probably not an NBI tweak. My guess is the download path is sorting on "pre-rendered" path. Meaning $(GROUP) instead of the actual group name.

The news which I'm looking for is if that bug/error is found/confirmed, or is it just me, that's all.

[Quade]

Yeah, I wasn't questioning your report. I assumed it was correct.