6.8 final HDD load

[jonas]

Hi,

I just upgraded to 6.8 final from a previous beta (not quite sure which one) and I noticed that since updating, the load on my HDD where the Newsbin profile is stored and the chunks get written to before the actual files are created (the download folder is on a different HDD) goes to 100% while downloading and my download speed drops by almost half. I didn't have this problem before so I assume it might be related to the new RAR processing? Anything I can do on my end?

[Quade]

You mean the amount of disk space used? How much free space do you have on this drive?

[jonas]

No, I don't mean the amount of disk space (the HDD has around 200GB free if that still matters). I'm talking about simultaneous read/write operations. I did some further tests last night and I noticed, that the problem does not occur with every file I tried. Sometimes Newsbin behaves as I would expect: checking in Task Manager the activity on HDD 1 (where the chunks are stored) goes to below 20%, activity on HDD 2 (where the actual files are created) goes to around 40% and I get a speed of around 400Mbit downloading.
But with a lot of files, I get the problem I described above: Activity on HDD 1 spikes and the speed drops to around 250Mbit for the entire download. It is definitely not random, since re-downloading the files I tested always leads to the the same behaviour.

EDIT:
I did some more tests and it seems it is actually the minority of files it happens with, so it is quite possible, that the problem is not new but that I simply did not notice it until yesterday.

[Quade]

Newsbin buffers chunks in memory but if the chunk count of a file > memory chunks the chunks get flushed out to disk. The chunk count is down where it says "Cache" at the bottom. The default is 400 chunks in 6.80. If you're downloading files that are > 400 chunks per file, some of that partial file will hit disk.

Your symptoms suggest high chunk count files.


[PERFORMANCE]
ChunkCacheSize=X

If you add this to the NBI file and set the chunk cache to larger than the per file chunk size, you might make this go away. It's about 1 meg per chunk block worst case. Some people set it to 1000 or 2000. There's a point of diminishing returns too. If you're using an SSD for this drive, you probably don't need to change this.

[jonas]

Thanks for the suggestion, I've set it to 2000 and performance seems to be improved while still not perfect. Setting it to a higher value makes no more difference.

[jonas]

Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.

[Quade]

You might try playing with the URL for the download. I think Dex leaves the older ones on the site.

I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.

[Quade]

One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.

[jonas]

You might try playing with the URL for the download. I think Dex leaves the older ones on the site.

I've tried that, didn't work unfortunately.

I'm kinda surprised you're seeing any performance difference in the last couple beta's. I might uninstall any non-Microsoft security software then reboot and try some downloads. More often then not sub-standard performance when everything else is the same is the result of security software stepping in.

There is no security software apart from windows defender installed on my machine.

One of the users on the IRC channel was hitting 800 with peaks to nearly 1 Gbps with the 6.80 release. It's just a data point but it means it's possible to get high speeds with it.

I'm sure if you have a faster HDD or SSD it won't be as much of a bottleneck as it is for me, but now that I've watched performance the last couple of days I am sure that the problem is definitely new for me. Might be related to something other than a change in newsbin but without the previous betas I can't test it...

[dexter]

Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.

Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Archive Page

[jonas]

Would it be possible for me to download older beta releases? I would really like to test, when the performance drop was introduced.

Sorry, I cleaned out all the beta releases once the final release was out. You can run the last production release which was 6.73 and see if there is a difference if you like. It is on the Links not allowed for unregistered users

ok, so I installed version 6.73 as suggested and infact I don't experience any unusual activity on my HDD and get a consistent 450 mbps. Installed 6.8 final again and HDD activity spikes and speeds go down to about 250 mbps. Something in the latest version does not play nice with my setup Image link not allowed for unregistered users

[Quade]

You could use "Procmon" to examine disk IO.

[jonas]

You could use "Procmon" to examine disk IO.

The only thing that jumps out at me is, that in both versions I'm seeing a lot of "QueryOpen" operations on chunk files that return "NAME NOT FOUND" and while the old version followed by performing a single "WriteFile" operation on the Logfile.txt, the new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?

[Quade]

he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?

Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.

[jonas]

he new version performs two of those and inbetween a "FlushBuffersFile" operation. Seems like that could be a factor in the increased activity I'm seeing?

Yeah, this one is a possibility.

Are you logging server commands? You might want to disable logging server commands in the network options and try turning down the logging in the advanced page.

Setting logging to normal under advanced options makes no difference. With disable logging server commands you mean the "show server commands" option? Other than that I don't see an option to control logging in the network tab and that option was disabled to begin with.

[Quade]

Yeah "Show server commands". On a high speed connection logging that might consume more disk and CPU than you want.

[jonas]

So I moved the DataPath from one of my HDDs to my system drive, which is an SSD, and that eliminates the bottleneck I was experiencing. Somewhere between version 6.73 and 6.8 Newbin got a lot less efficient and it seems to download at high speeds you now need to use an SSD at least for the profile folder.

[LaRokka]

hello

pretty similar issue. just updated from 6.72 (release) to 6.80 (release) and my hard drive is getting hammered. in 6.72 all was smooth.
using a quite normal samsung 7200rpm sata 3 hdd as my download drive.
my download speed is officially 150mbit, usualy sits between 150 and 160mbit.

since 6.80 the hdd is constantly hammering when downloading. the drive LED is permanetly on, just stops for nanosecs every few secs.
the larger the files it is downloading are, the worse it gets.

1-3gb i am still getting 150-160 mbits.
3-6gb the harddrive is hit so hard, download goes down to 120ish mbit.
8gb down to 100ish mbit.
didnt test bigger files yet as i just installed 6.80.

i have "pre-allocate files for spinnign discs" enabled. tried to disable it, does not change behaviour.
it somehow seems as 6.80 is not caching (just from belly feeling, cant say for sure). but i know it wasnt this bad at any time before this release in the past years.

help greatly appreciated !

no native speaker, plz excuse me

[Quade]

I suspect it has something to do with the logging doing "FlushFileBuffers". I've disabled that in code but you won't see it till 6.81ish.

It's looking like we need a 6.81 because some people are reporting "Add to download list and it stalls" problems.

[LaRokka]

i just downloaded 6.73 from the archive. can i install it over 6.80 like a normal update, with my settings kept ?
i would then continue to use 6.73 until 6.81 is out. didnt have issues with the older version anyhow, just updated to be up-to-date

[dexter]

Yes, you can install 6.73 over 6.80 with no issues. I've done it myself during the testing phase. If you are a header downloader, I'd make sure the import folder is clean before downgrading since they are no longer compressed in 6.80 so 6.73 won't process them.

[LaRokka]

ok guys, thank you for your quick replys.

see you in 6.81

[jonas]

Looks like the just released 6.81 beta fixes the issue.

[Quade]

Thanks for the feedback. I removed some "flush file buffer" calls.

[LaRokka]

updated from 6.73 to 6.81, currently downloading a 15gb file.
download jumping between 120 and 150 Mbps, should be 150-160. high HDD activity happening when its jumping.

doanloaded several 1-3gb files last night, wasnt as appearant then. seems it gets worse the bigger the files are.

it is better than 6.80 but not as good as 6.73

[LaRokka]

please ignore my last post. currently downloading a 39gb file @steady 150-160Mbps.
so all is well for my serrver´s hdd in 6.81

maybe something funny was going on with the files last weekend, or my isp/usenetprovider was acting ... /shrug