Speeddrops with 6.50 RC 3 Build 3095

[useneter]

hello people, I have speedrops in the new versions. since the beta I remember that on already. for me the speed is not constant. but falls off very often. no matter how many connections I use. the speed varies greatly from 100-170 mbit. with the stable version 6:42 everything is always perfect. always constant speed. could test one of the pacemakers newsbin the times. I have the rest ever presented. for me the most stable version is 6:42. sorry for my english

[DThor]

Usually this means some sort of antivirus program is interfering, assuming everything else is the same (your usenet server, the port you're using, etc). Sometimes they see an upgrade as a new program and interfere with the downloading. Simplest test is to completely disable all anti - malware software and see if that helps. If it does, you need to configure it to ignore the new release.

DT

[itimpi]

It is a good idea to indicate if there are any error messages being displayed in the Logging tab. Certain types of error (e.g. 430 for post not found) can have the side-effect of slowing down the effective download speed as Newsbin looks for posts that the server appears to not have..

[Shine]

I noticed the same.

I narrowed it down to the fact that Newsbin 6.50 saves each single post to the "chunks" folder, then after finishing all parts of one file, loads all these single posts again and assembles the final file. So you have 3x the disk access compared to version 6.42: (1) save single posts, (2) load single posts, (3) save assembled file. This causes major disk thrashing and slows down Newsbin considerably. It completely escapes me why this change has been made - the 6.42 and earlier behavior of saving the data directly to the final file was much much faster.

In my case, depending on the size of the individual files (apparently smaller files can be handled better by Newsbin's or Windows' cache), I get slowdowns from straight 100MBit in 6.42 (this is my ISP cap) to 60-70MBit in 6.50 with individual files of 50-100MB or larger. Almost no slowdown is happening for me when the individual files are smaller, e.g. 10MB.

[Quade]

In the cache line down in the status bar, how many cache entries do you have? That controls how much of the file Newsbin keeps in memory. If everything goes perfectly and the file downloads clean, the files are 100% cached in RAM and assembled out of RAM. If you typically download files with more than cache size number of chunks then you might want to increase it. Some people use as high as 2000. I typically use 400.

I'm going to guess that the files aren't downloading clean so, new file data is flushing partial old file data out to disk. I'm a bit skeptical that this is the cause of your slowdown. I've run the cache really low in testing and don't notice a performance decrease. I guess it depends on what else you're doing with the machine at the same time though.

[PERFORMANCE]
ChunkCacheSize=n

In the NBI file is how you set it.

Exit Newsbin before you edit the file.

[Shine]

Thanks for the description, Quade. While I was already aware of how to configure it, others might find adjusting the chunk cache useful.

I can't really go much higher than the default of 200 without slowing my box to crawl anyway by maxing all available RAM (this is 32bit Windows with 2GB RAM). Also, even if I could, I find it pretty much overkill to give Newsbin 2000MB (ie. two gigabytes!!) of download cache, especially since the previous version basically didn't need any download cache at all and could constantly download at my 100MBit cap without any slowdown whatsoever.

The new way of assembling files is definitely the cause of the slowdown I'm experiencing. I can see the chunks folder fill up with thousands of tiny files (already causing some disk thrashing), then all the sudden I can see (resp. hear) major disk thrashing and the download speed drops to close to 0 for a few seconds. Then one additional RAR part appears in the destination download folder and the download speed starts to pick up again. This repeats for every downloaded RAR part, unless the parts are rather small.

Since common Usenet RAR sizes are 5, 10, 50 or 100MB per part, I could observe that with 5 or 10MB parts, it works as you say. Assembly appears to happen in the cache, nothing gets written to the chunks folder. Larger sizes, e.g. 50MB are hit and miss. With 100MB or larger parts, the slowdown as described above happens always. I tried a chunk cache size of 200 and 300, couldn't see any noticeable difference. Perhaps 300 is already too much for my box and the chunk cache itself gets paged out...

[Quade]

I can't really go much higher than the default of 200 without slowing my box to crawl anyway by maxing all available RAM (this is 32bit Windows with 2GB RAM). Also, even if I could, I find it pretty much overkill to give Newsbin 2000MB (ie. two gigabytes!!) of download cache, especially since the previous version basically didn't need any download cache at all and could constantly download at my 100MBit cap without any slowdown whatsoever.

The model is different than before. The old version ALWAYS wrote the data to disk into the NB2 file. There were a number of problems with that including restarting and that sometimes the block sizes weren't uniform. I believe it used to perform better that's why I wrote it that way but, I think reliability trumps performance. Particularly when you can fix the performance with more RAM. The new scheme is much more reliable. I still get 300-400 Mbps download speeds to a local 7200 RPM hard disk using my test server so, the performance is still there.

Cache blocks end up being from 400-600K in length so, 200 of them is about 120 megs of RAM. 300 would be 180 ish megs.

I'm not planning on switching back. This new method is more reliable than the old. Supports restarting much better and in general works better than the old NB2 scheme. When Newsbin is idle you can actually flush the cache blocks by opening the Help/About windows. It's something I use for testing. RC4 which is in the wings reduces RAM usage significantly. I found an issue with the logging stuff.

[Shine]

Cache blocks end up being from 400-600K in length so, 200 of them is about 120 megs of RAM. 300 would be 180 ish megs.

So chunk cache size is counted in "posts", not "MB"? At least this clears up the "hit and miss" part - the smaller the post size, the higher the chance of the cache filling up, regardless of the actual destination file size, correct?

Have you thought of reserving a configurable amout of memory for chunk cache instead? This would make it easier to predict Newsbin's memory consumption and to adjust the cache to match the actual amount of RAM one can "spare".

[Quade]

Have you thought of reserving a configurable amout of memory for chunk cache instead? This would make it easier to predict Newsbin's memory consumption and to adjust the cache to match the actual amount of RAM one can "spare".

600K per cache block is how I size it in my head. I know sometimes it's bigger but, most of the time it's smaller. It's caching posts so, posts are the atomic unit. Not bytes. I don't see how reserving memory would be a benefit. I'd still have to code it as:

Reserved RAM Size / 600K = Cache blocks.

Which is about the same as just specifying cache blocks directly.

I'm wondering if your cache is getting purged because your download speed is faster than Newsbin can actually write the files. During assembly, available cache blocks are used to store newly downloaded chunks of the next file so, if your assembly is slow, the cache will get pushed to disk. Only one thread reads or writes cache blocks to disk so, it's likely the download will stall when the cache gets down to zero. That's probably what you're seeing.

Are you running Newsbin inside a VM?

[Shine]

Reserved RAM Size / 600K = Cache blocks.

I was thinking more like dynamic cache block count instead of fixed, with an upper RAM consumption limit. So larger posts equal less total cache chunks and the other way round.

I'm wondering if your cache is getting purged because your download speed is faster than Newsbin can actually write the files. During assembly, available cache blocks are used to store newly downloaded chunks of the next file so, if your assembly is slow, the cache will get pushed to disk. Only one thread reads or writes cache blocks to disk so, it's likely the download will stall when the cache gets down to zero. That's probably what you're seeing.

Yes, that makes sense. In fact, it looks like the cache is already down to zero before assembly even starts, because I'm seeing hundreds of chunk files in the chunk folder. Speed picks up again once assembly is done (be it because the file r/w thread is released or some cache blocks freed). A second r/w thread could help I guess, but might also overload the HD with accessing all those tiny 600k files. HD is a 7k2 single drive, just like in your test setup.

Are you running Newsbin inside a VM?

Unfortunately not, otherwise I would've tried assigning more RAM already.

[useneter]

what can I do for now. the problem is persists with the new version. I do not remember more. with 6:42 everything is going great. constant speed. with the new version speed drops. my machine: core7i (haswell) @ 4.2 ghz .... 16gb RAM. ssd Sasmung 840 per. i think its have enough speed for newsbin. please help me so i can use the newer version. but with this speed drops is it not useful.

thnx for some help

[Quade]

I already made a suggestion.

[PERFORMANCE]
ChunkCacheSize=n

Exit Newsbin, Set it to 500, start Newsbin, see what happens. You should see "500" down in the cache line. If it uses too much RAM, try reducing the number. Are you downloading to the SSD? I get 400 Mbps downloading to a 7200 RPM hard disk. With your PC, there should be no speed dips at all.

[useneter]

i cant find the line ChunkCacheSize. i calls SaveChunksMode by me. it is right? or where can i find that line?

[Quade]

You add it. It's not in there already.

[useneter]

You add it. It's not in there already.

Thnxxxxxxx. Problem solved. Every Time full speed and fine caching without drops. im so happy with new version now.

[lordkinbote]

I set my cache size to 400 and dropped my number of connections to 12 (I was going 48 'cause GigaNews' limit is 50 and I was still only getting 1/2 my full speed of 30Mbps with that many connections) and it didn't help me. I'm getting speeds of 5Mbps now at 12 connections. I feel it is something in this configuration that is giving me slower speeds 'cause I got this .nbi file from my laptop, which was also getting slower speeds than my PC and I couldn't figure out why. I just rebuilt my PC and used the laptop .nbi and now I can't get full 30Mbps speeds. At 48 connections I was getting 18-19Mbps at the highest. Before I rebuilt my PC, I was getting the full speeds with 8 connections. By the way, I'm using v6.51b3 now.

[Quade]

It's probably Giganews. Sunday seems to be a bad day for most news servers. If there's any day I can't get full speed it's sunday. I've also noticed that older posts seem slower to download than newer. Might give a brand new file a try and see if anything changes.

Are you using both giganews servers? Might add the euro server and split the connections between them.

[lordkinbote]

These are new files, things posted in the last day or so. Right now, my speed is about 3Mbps with 8 connections and I should be getting 10x that.
I should add the euro server as an additional one?

[lordkinbote]

When I added the euro server, also with 8 connections, it bumped the speed up to almost max so I made it 10 connections on the euro and now it's maxed out. Thanks for the suggestion, although I've been using Newsbin, GigaNews and WideOpenWest for years and never had to add a server before.

[jackholexxxx]

I've also noticed that older posts seem slower to download than newer.

I have noticed this quite a bit with GigaNews. Enabling my Astraweb block account gets me full speed again. Not sure why Giga is so slow on older posts.