Possible memory leak in AutoPAR???

[slee]

I downloaded a set of files from alt.binaries.hdtv.x264. Came back to my computer several hours later to find that I had a critical message saying Windows was out of memory. I've never seen that message before. There were messages in Newsbin saying resolver error, etc. A bunch of blank lines. It was obvious that this was not how it was meant to look. I could tell that Newsbin was having difficulty with AutoPAR. Newsbin downloaded every single par file it could and still reported errors. I checked task manager, and sure enough, Newsbin had consumed nearly all the ram. I've never seen Newsbin use so much ram. I started QuickPar on the files, and it was showing that the only thing wrong with the files where the file names were incorrect. Almost half way through the scan, it appeared that Windows rebooted. Didn't seem like a normal reboot though, but it came back to the login screen, and everything opened again like I had just powered off the machine and turned it on again. So, once again I started QuickPar. Sure enough, it confirmed, that the files were fine, just mislabeled. Is it possible that Newsbin's AutoPAR doesn't know how to handle mislabeled files?

[DareDevil ©]

Not sure if it's in AutoPar but it's somewhere. Just looked at my Task Manager to see why my system seemed a bit bogged. Newsbin was actually using 1.8GB of RAM. I've never seen anything much over 7ooMB in the past - no matter what version. This is for v6.10 RC2. When I noticed it, AutoPar had been going previously to unrar a DVD but it was finished that and just downloading the next.

[Quade]

How many incomplete downloads are waiting in the download list not written to disk yet? if any. My current theory is that it's partial files waiting to get written out to disk. It's possible to make Newsbin not cache these files in memory by setting the "MemCacheLimit" to zero. Might be worth trying that to see if your symptoms change.

[SETTINGS]
MemcacheLimit=0

In the NBI file.

[DareDevil ©]

Trying the MemcacheLimit=0 setting. Will let you know. Thanks.

[slee]

Quade... I just had this same thing happen with a different set of files. Didn't have the reboot issue as mentioned above, but otherwise the same. If you'd like to confirm this yourself, I can give you the set of files that caused this.

[FriedLocust]

Quade, I have similiar a issue. Memcache ist set to 161 the parfiles are 159000. No file gets written to the disk, as intended. The first part of each rar file is corrupt.
Newsbin is doing the parrepair and everything obviously in memory. And then unrars and writes the file to disk. So far so good.
Newsbin eats up 7 gigabyte of ram and never releases the memory after everything is ready and written to disk. Download list is empty and all files are on disk.

The full ram of course slows my system down.

I'm not complaining about the usage of ram when needed by newsbin, my problem is newsbin never releases allocated and not more in use memory. Newsbin 6.10rc3

Edit: Set memcache to 61. Files were written to the disk instead kept in memory. Now memoryusage of news bin is only 3,5 gigbabyte, but still not released after everything is done.
One more issue showed up. After parrepair and autopar, unrar wrote the mkv to disk - rar files got deleted but rar.1 files were still on disk. Had to delete them manually.

[Quade]

I saw it myself last night. It ended up fixing itself. Was using about 2 gigs of ram and then dropped back down. I have increased the amount of RAM Repair is allowed to use so, it's possible it is Repair that's causing this. I'll look into it further.

I suspect there's more than one thing going on here though. If you have the memcache limit set to say 100. Partial files get stored in ram until they're written out to disk. If you have a bunch of partial files, you will use a bunch of memory. So, that's one thing to look for, partial files that haven't assembled yet. I have a technique for testing this but, it's not obvious. Telnet into the remote control part and type in

"debug cache"

It'll list how many partial files are in memory and how much RAM they're using. If this is low or then you know it's not the memcache that's doing this. It might be the repair DLL. When I saw my Newsbin using 2 GB of RAM, "debug cache" showed that it wasn't the memory cache. That's why I'm thinking repair now.

[FriedLocust]

Hm I must doing something wrong. Under programs and features telnet client check box is on.

Opened an elevated cmp prompt I typed TELNET 127.0.0.1 118 but nothing. No reply. I'm doing something wrong I presume?

€: OK, I had to check the remote part in options

[FriedLocust]

OK, it seems it does not flush the cache after it is finished.

Screenshot taken while par repair/unraring using debug cache command


and after finishing using debug cache command again


Cache and ram stay the same. Shouldn't it be cleared? As you can see on the side ram usage stays the same.

[Quade]

Mine doesn't do that so, now the question is, what makes them stay there. There shouldn't be any at this point. So, something prevents them from getting closed out and disposed up. I'll have to see if I can figure out what though. Some path through the code is different for you than for me.

Thanks for the shots. You've proven that in your case that it's things stuck in the cache. My cache was empty so, it is looking like multiple potential issues.

[Quade]

I added more details to "Debug Cache" and what I caught, is that if a file fails to assemble, it remains in memory. In this case, I didn't have a drive mounted so, the file downloaded but, it couldn't be saved. I'm figuring out how to handle it.

[FriedLocust]

Hm, my files got written to disk and are fully functional. The files got repaired and then were written to the disk but stayed in cache.

I'm sure you'll sort this out a way or another. Glad to be of help.

[slee]

Quade.. Have you been able to confirm the incorrectly named files/par problem? This happened to me again last night. The files were fine, just labeled different than what the pars show. QuickPAR is able to fix the file names. Newsbin seems to think there is a problem and grabs all the PARs it can. At that point, Newsbin continues to consume all available system memory until Windows becomes unusable. This is happening with the 64bit version of RC3.

I imagine a simple test would be to create an rar set of files. Then create a set of par files for the rar set and then rename all the rar files to something different. Post the entire set of rar's and par's to alt.binaries.test. Download with Newsbin and see what happens.

I love the AutoPAR feature of Newsbin, but I guess I've come across a poster that can't seem to leave the file names alone after creating the par files.

[Quade]

As long as the par downloads, the files should get renamed in Newsbin. In particular I test a case where the RAR files are named "JPG" and I watch Newsbin rename them.

Dex gave we a weird NZB that Newsbin repaired over and over but, quickpar says is fine. The repair would work, the files were perfect but, the repair library in Newsbin thought there were still mis-named files. I just shelved it as "weird" for now since it works fine with other mis-named files.