Running out of memory after updating to 6.55

[abalian]

Ever since I updated to 6.55, most of the time, not always, when I add a long list of files to the download list at once, my RAM goes up to 7GB in seconds. It says out of memory and it either flushes out by itself immediately back to ~200MB of RAM used, OR my computer slows down and starts freezing and I have to restart in order to use it.

I don't understand what could even cause 7GB to fill the RAM. Sure, the files I'm downloading amount to that much, but my download speed is only 7MB/s. Am I the only one who gets this? Thanks.

[Quade]

I'm not seeing it. How are you downloading? Headers, Search or NZB Files?

[abalian]

NZB files.

[Quade]

How many gigs worth of files are you feeding? I mean the size in the download list, not the size of the NZB Files. Though knowing the size of the NZB's might be a good data point too.

[abalian]

Downloads are up to 50GB with each NZB file (up to 10MB themselves). I think it's happening when I add to the download list when it's not empty, but I'm not sure. Download list can sometimes go up to 200-300GB. Used to be fine, though.

"Write downloaded files with No memory buffering" is unchecked if that makes a difference.

[Quah]

I'm running into what sounds like the same issue as abalian.

On windows 7, with 8GB's of memory, Newsbin jumps up to the max (about 97% physical memory), and then I can hear the hard drive swapping files. Everything slows down.

I tried renaming the downloads.db3 file, deleting the spool files (they had 120G in them), and degrading from Version 6.56 to 6.55 and then 6.54, and it still maxs out the memory.

If I just download the headers, it goes to about 600MB, then within a minute after it completes, it goes up to about 3GB, then just increases to the max slowly. If I download a file, the increases faster. Of course as I am writing this, it started to drop instead :/
Within a minute, it jumped back to 3GB, and is on the steady climb.

It does keep my CPU running at 25% or above all the time now.
I regularly run anti-virus/spyware programs, but can't hurt to run AVG, Avira and Spybot again.

Any suggestions? Any additional info you could use? Screenshots? Webex?
Thanks :)

[Quade]

If you exit. Do you get all your RAM back?

If you look down at the "Cache" line do you see a number in parens? "(XXX)"

In the data folder, do you have anything in the "Import" folder?

[Quah]

Sorry for the delayed response, my computer decided to stop allowing inbound packets.

Yes, there was a delay, based on the downloads.db3 files size, but all memory was returned.

Since last night, I ran AVG, (nothing found), and spybot (normal tracking junk found).
I upgraded Java RE, from 8u31 to 8u40. I doubt this matters though.
I also upgraded back to 6.56.

It seems to be more stable, only using 1-3 GB on average.

Cache 107/200 (44) memory usage at 2.6GB

The import folder has 49 files, totaling about 500MB.

[Quade]

Until the import folder hits zero, they'll be a fairish amount of processing happening. That shouldn't consume 8 gigs of RAM though.

Keep an eye on that number. If it doesn't go down on it's own while Newsbin is running, it might indicate a corrupt file. You could exit, move all the files out of the import folder then add a few at a time back till you find one that gets stuck (assume that actually is a problem). The number will go up with header downloads and down as Newsbin processes the files. It should hit zero relatively quickly. 500 megs of GZ files is actually 5 gigs of data to process because the files are compressed.

[Quah]

Hmmm, submitted a response, and don't see it, so I'll recap.

Yes, that was the issue.
The CPU is below 4%, and memory below 400KB.

I renamed the entire Imports folder, and created a new empty one.
Is there any reason I need to restore any of these files?

Thanks for your help,
Quah

[Quah]

I meant 'under 400MB' - doh!

[Quah]

I decided to do as you suggested, and I'm crunching through the files one at a time. I found three files, that cause my computer to go into memory lala-land. I have 18 more to go.

I suspect that these are related to the filters, as a few files from my filter list showed up in the download list while doing this.

[Quade]

I renamed the entire Imports folder, and created a new empty one.
Is there any reason I need to restore any of these files?

These are headers that haven't made it into your header database yet. You can delete them but then you'll be missing some headers for these groups. If they're groups you don't much care about it might not matter.

It's a good find. Now I just have to figure out how that causes memory usage to go crazy.

[Quah]

I can send you the ones that fail, if that would help. Two are about 4MB big. I did run them through a 7zip archive test, but they aren't bad as far as it can see. I still have 7 that are over 50MB big. I suspect they'll take a long time to go through.

They are all from a newsgroup that is very active.

[Quah]

I tried disabling auto mode and the filters I had applied to the groups, and re-imported one of the small files, but it didn't help.

I went a step further and split one of the files in half (after unzipping, splitting, fixing the title, rezipping and fixing the file name), and was able to get it to go through. Each half took more than half the memory I have, but it completed in about 4 minutes each. So I guess that would mean the file doesn't have any specifically bad stuff in it. The first gzipped file was 1.464 MB, and took over 4GB to process, so that can't be good.

I could split the same file down further to see if the memory usage dropped in half too, but before I do that, will it cause problems by re-importing the same articles twice? Wait, I found that answer. There is already a 10 article overlap in the smaller files. I'll try this after I wake tomorrow.

[Quade]

How about PMing me the name of the group and I'll check it out.

Re-importing won't hurt anything. It's just a bit slower than the initial import.

[Quade]

I see it here too.

Boneless is under attack of some sort. Not sure the group is even usable anymore. What it needs is some header download filter to delete the attack posts before they hit disk.

I'll look at it.

[Quade]

I figured out what's going on and have a fix. It'll be in the next beta. It's not a traditional attack. Just acts like one.

[Quah]

That's great.

The only import files that affected me were the ones from 4-7-15 (the same day this thread was created), so I'm going ignore them. If the issue reoccurs I'll know how to deal with it and I'll go the beta route. I might do that anyway with these bad files, but need to find the time.

Is it possible to use a subject or filename reject filter as a workaround?
I've done a lot of looking through the import files in the last days, Dora's posts were the only ones using the references field, but decided that was unlikely the cause. Can you share what you found?

Thanks for your help,
Quah

[Quade]

The issue issue is caused by posts that like this:

XYZABCDEFG (10000/30000) <Poster 1>
XYZABCDEFG (10001/30000) <Poster 2>
XYZABCDEFG (10002/30000) <Poster 3>
....

Basically they looked like a multipart post but because the poster address was random. Newsbin didn't group them. A memory optimization in Newsbin was triggered by these posts and so, each one consumed 30,000 posts worth of RAM. I put a fix in place to handle them.