Cache Miss - Larger chunk cache might perform better?

[Moondawgie]

I see quite a few entries of "Cache Miss: Larger chunk cache might perform better" in my Newsbin Data Logfile.txt file.

How do I increase the size of the chunk cache?

This is with Newsbin Pro Version 6.20 Beta 3 Build 1447, under Windows 7 Ultimate 64-bit, on a PC with 4GB of Ram and a pair of SATA 2TB 7200RPM hard drives.

[Quade]

I should probably remove that warning. What it's telling you is that your disk drive can't keep up and that you're downloading faster than the data can be laid out on disk. Not sure more cache is the answer here though.

Tell me about your PC. Changing the "MemCacheLimit" might be more beneficial.

[Moondawgie]

I can't think of anything more to say about my PC that wasn't included in my post, or in my forum signature.

I guess I need a RAID, or faster hard drives. Either way, I'd better buy them quick. (See http://www.tomshardware.com/news/hdd-th ... 13936.html )

Thanks.

[DThor]

Hmm, just throwing it out there, but I wonder, if you happen to have a really fast pipe to the internet, and are using ssl to boot(both guesses), if this symptom might show up? SSL compression starts to make less sense as connections start to approach internal network speeds.

DT

[Moondawgie]

Hmm, just throwing it out there, but I wonder, if you happen to have a really fast pipe to the internet, and are using ssl to boot(both guesses), if this symptom might show up? SSL compression starts to make less sense as connections start to approach internal network speeds.

DT

I have a 12mbps cable internet connection, which tests and consistently provides 14mbps service. And I do use a VPN connection (OverPlay) in order to avoid throttling by my ISP.

The use of a RAID, using three 7200 RPM SATA drives, is looking like the best solution for me.

For the record, and with the exceptions of the inability to hide downloaded files from view, and the problem with automatically extracted RAR files being found in folders named like "UnR1A13.tmp" rather than with the title of the RAR file, I am really quite happy with my use of Newsbin 6.20 Beta 4 Build 1456.

[Quade]

I have a 12mbps cable internet connection, which tests and consistently provides 14mbps service. And I do use a VPN connection (OverPlay) in order to avoid throttling by my ISP

Hmm, that's not very fast. I have 100Mbps here and I'm going to a single encrypted 7200 RPM drive that I both download, repair and unrar too and I don't run out of cache packets. In my testing, Newsbin can sustain 300-400 Mbps to my single disk without running out of cache. Wonder if your virus scanner is just killing performance. To me "fast" is > 100 Mbps.

Are your drives in IDE or AHCI mode?

[Moondawgie]

I have a 12mbps cable internet connection, which tests and consistently provides 14mbps service. And I do use a VPN connection (OverPlay) in order to avoid throttling by my ISP

Hmm, that's not very fast. I have 100Mbps here and I'm going to a single encrypted 7200 RPM drive that I both download, repair and unrar too and I don't run out of cache packets. In my testing, Newsbin can sustain 300-400 Mbps to my single disk without running out of cache. Wonder if your virus scanner is just killing performance. To me "fast" is > 100 Mbps.

Are your drives in IDE or AHCI mode?

AHCI.

[Bismark]

Be very careful about using RAID. I have three 7500 RPM 3 GIG disks and had a RAID set. They worked very well until one of them got some sort of temporary error resulting in a slow response. the RAID controller in the mother board decided that the RAID lost a disk and went to backup mode. It then took 45 hours to rebuild the set. Reading about RAID I found that your disks have to have fast error recovery or this will continue to happen. In short, after the 6th time, I backed up the RAID set and broke it apart. No problems since. The disks needed for RAID are more expensive than the run of the mill PC disks.

[Quade]

I use a 4 drive software raid on my linux box. I've had weird problems with hardware raid like this.

Honestly though, I don't think he needs raid. He needs to find out why Newsbin has a hard time writing to the disk. If there are other things going on, even to another disk, this can impact performance.

I was working on the BW limiter yesterday and my download speed wouldn't go over 23 Mbps. I did everything, rebooted routers, network resets and the like. Turned out something I was doing on another disk was killing download performance. I still don't know why.

[Moondawgie]

If and when I might figure out what the problem is, I'll return to this message thread with my findings.

Is there a description somewhere here in the Newsbin forums of the components of the computer system you use to run Newsbin, Quade?

[Quade]

http://www.boxxtech.com/products/3dboxx ... odid=4860x

It's basically an older version of this. Quad core i7. My download disk is a 1TB WD black encrypted with truecrypt. I don't use raid. The drive is just hanging off a single SATA connector on the MB and it's in AHCI mode. My development disk is a 320 Gb spinning disk. The only thing on it is development stuff.

No virus scanner or firewall. I have MS Security essentials installed but, disabled. I turn it on from time to time to perform a scan.

I can't remember if you set the "MemcacheLimit" or not. If not, you could try setting it to 200

[SETTINGS]
MemCacheLimit=200

This will make Newbin buffer files in memory before writing them out in one shot. It's actually better than the cache setting. If you have many incomplete files though, it can run you out of RAM. That's why it's off by default in the current beta's. You have to exit Newsbin, add this then restart Newsbin for it to take.

[Moondawgie]

I've determined that this problem is due to my use of the Group, "Automatic Download. Add all new posts to the download list", option.

Newsbin starts downloading and processing new files added to the download list before it has finished the process of downloading new message headers, processing the the applicable filtering of undesirable/unwanted posts, and adding new posts to the download list. The problem is most acute when Newsbin runs the AutoPar and Unrar processes for newly downloaded files while news posts are still being filtered and added to the download list. As things are now, the process of adding new posts to the download list usually continues for several minutes after Newsbin has finished. Once that process appears to have finished, and it's often very hard to determine when that process has really finished, I'll pause downloads and look through the download list removing posts that I'm not interested in before resuming the download process.

The only reason I use the automatic download feature is that it's MUCH easier to automatically add new posts to the download list, knowing that they are all new posts, rather than having to go back and individually pick out new posts from the clutter of new, failed/duplicate, and downloaded posts listed in each newsgroup's, or group of newsgroups, list of posts. If failed/duplicate, and downloaded posts could be hidden from view in the same manner as idle/old posts, I probably wouldn't use the automatic download feature.

The MemCacheLimit setting is, and has been, 200 (MemCacheLimit=200)

It seems that I need to disable the automatic downloading of files, while hoping for the addition of the capability to hide failed/duplicate and downloaded posts from the list of posts in the individual newsgroup and group(s) of newsgroups.

Is it possible to provide an option which would delay the downloading and processing (AutoPar and AutoUnRar) of new files until after the the addition of new posts to the Download list has finished? Or perhaps add such new posts to the wish list rather than the download list (oooohhh....)

Thank you for your attention, suggestions, and assistance.

[Quade]

Yeah, you're beating the disks.

1 - Header download - Disk IO
2 - Packing the data into a DB - Disk IO
3 - Adding the files to the download list DB - Disk IO
4 - Downloading - Disk IO
5 - Repair and Unrar - Disk IO.

Another option is to just not care about your cache and let if just do it's thing in it's own time.

The only reason I use the automatic download feature is that it's MUCH easier to automatically add new posts to the download list,

Assuming you do this every day, I'd set a 1 day display age so, out of the box older posts are invisible. Then maybe use "Autoold" so, each load of the group, sets the records in that load to "Old". If you have lots of crossposting. You might want to add all those groups into a common topic and load them all into a single list. The "Autoold" will apply to all the loaded groups at once.

[Moondawgie]

Assuming you do this every day, I'd set a 1 day display age so, out of the box older posts are invisible. Then maybe use "Autoold" so, each load of the group, sets the records in that load to "Old". If you have lots of crossposting. You might want to add all those groups into a common topic and load them all into a single list. The "Autoold" will apply to all the loaded groups at once.

For once, I'm ahead of you. I'm relishing the moment. I know it won't last long.

Stupid hard drives...