6.10 RC5 crashes

[RayMark]

I get lots of these with 6.10 RC5

[18:40:29] ERROR - MF_AssembleFileUtil - Thread Crash

I few days ago, I was using RC1 (back from RC4) and there were other thread crashes, different ones, I think, and the speed dropped to 0.
Now, the speed is still over 90 Mbps, but the files are NOT assembled - they are just missing altogether as if not downloaded.

The problems so far seem to be limited to a single multi-rar post. But even if there is something wrong with it, Newsbin perhaps should not crash this way.
I can PM the details about this post.
BTW, its nzb downloaded successfully, I will try to re-download it from nzb instead of from my local headers.

UPDATE 1:

No, there are other posts with exactly the same problem. This is with RC5.
I can go back to RC1 and see if it is any better.

And also these:

[19:02:51] ERROR - DbThread - internal crash

UPDATE 2:

Closed Newsbin and restarted - so far so good, no crashes, but haven't tested yet on those problematic posts.
Just as I am typing this, I got the following:

[19:27:55] ERROR - INT_AssembleFile: Unable to MAP file ......

This was a single message so far, not an internal crash at least

UPDATE 3:

Newsbin is gone. Crashed and closed completely, it seems, all by itself, its process no longer running. Only a number of .nb2 files remaining unfinished.
Started again.
A number of "INT_AssembleFile: Unable to MAP file" messages immediately.
Then most of .nb2 files were deleted, it seems, I don't see any assembled files appearing instead of them.

My current download list is almost empty, when finished, I will try those vanishing posts again.
Maybe only those posts cause problems but not most of others.

UPDATE 4:

Now again many of pairs of those:

[03:21:16] ERROR - MF_AssembleFileUtil - Thread Crash
[03:21:16] ERROR - INT_AssembleFile: Assembly Failed .....

On yet another different post, unfortunately

UPDATE 5:

Possibly, those problems are with incomplete posts only. I will look into that theory.

[Quade]

You have some weird problems. I'm not sure where to begin to figure them out at this point.

[RayMark]

Now I am trying to download those posts again, the ones that have nzb files posted - from nzb.

1st downloaded successfully.
2nd - it was badly incomplete, also downloaded but could not repair.
3rd - downloaded as if ok, but no files appeared at all, not even .nb2. - still indicated downloading speed and then marked them as downloaded.
Closed Newsbin, started again - started downloading the same files again, but this time for real - everything seems to be ok.
Files correctly downloaded and assembled.
So far no crashes. Except that weird downloading to nowhere which was fixed by restarting of Newsbin.

So the next step will be to try to download again from my headers, not from nzb.
Previously all the problems were when downloading using local headers.


BTW, that search in groups is suddenly working in RC5, was not working in RC3 and RC4


UPDATE 1:

I had 5 such posts that caused crashes. Downloaded > 200 other posts without problems (close to 200 GB).
Of those 5 posts, 4 had .nzb files posted.
I downloaded them from nzb - everything ok, except two things:
- one post was badly incomplete, but probably it has nothing to do with Newsbin.
- one post was downloaded somewhere to void, after restarting Newsbin - downloaded again successfully.

The last post, without nzb file - I downloaded it again using my local headers.
This time also without problems. Downloaded correctly.
It is great, because I was worried that my headers were corrupt.

So no more crashes.
BTW, I started those downloads with RC1 then switched to RC5 in the middle, probably with some .nb2 files partially assembled.
Crashes were already occurring with RC1, though.
Restarting Newsbin did not help while those files were already in the download list.
After the download list became empty, adding the same posts does not cause more crashes.
So I suspect that the download list was somehow corrupted and caused those problems.

[dunkem]

All of my files are also getting ERROR - INT_AssembleFile: Unable to MAP file

All the files remain with the nb2 extension. Files are marked as "downloaded" in the download list but fail assemble. Running quickpar, all files are good but need renaming. I'm on 6.10 build 1337. It started occuring about two beta releases ago I believe. Shutting down newsbin and restarting has no effect. Shutting down and restarting is actually worse because when the files are scanned, all are still nb2 files so it thinks it hasn't downloaded it redownloads.

[DThor]

Any chance your download folder is 'deep', i. e. has a long pathname? If you change to a much more shallow one, does it start to work?

DT

[Quade]

One quick workaround is to set the "MemCacheLimit" to something like 100 so, it doesn't have to map the files. This makes Newsbin use more ram and cache the entire file in memory.

[SETTINGS]
MemCacheLimit=100

In the NBI file.

If the files you typically download are large you might make the number larger to match the file size. I mean the size of the individual files, not the entire set. You could try this too Raymark. You really want to keep ram use well below 3 gb in 32 bit windows. The problem is, you might have plenty of RAM because of swap but, that 32 bit applications run out of address space.


Tell me about how your downloads Dunkem, I'm assuming you have something unusual in your setup. How much RAM is Newsbin using at the time? If this is 32 bit windows and you're close to 3 GB of memory use, the problem is that you're close to out of RAM and that's probably why the map fails.

[RayMark]

I started a whole bunch of new downloads and again I got those, but only 3 times:

[10:07:45] ERROR - INT_AssembleFile ...

I see that in the related post, exactly those three files remain as .nb2 - but they are par2 files, not rars, not needed, in this case.
Still, if there is some issue with assembling par2 files rather than rars, this info might useful.

My download path is not too deep, I think:

U:\NNTP\MULTIMEDIA_TWO\...

MULTIMEDIA_TWO was added by Newsbin itself because it is a a folder containing several multimedia groups, and I loaded the whole folder, not a separate group.

As to MemCacheLimit, I already had this line in .ini:

MemCacheLimit=100

Should I give it a larger number than 100?
I think that in Windows32 Newsbin could use up to 2 GB (its allowed address space) without using swap as long as at least 2 GB are free for it to use at the time.
Now, I see, it is using 519,004 K
Total memory available: 3,487,969,280
Free memory: 1,044,586,496
The other two main memory users are:
Opera: 616,872 K
and
uTorrent: 289,428 K

So you see, if I close Opera, or at least its many no longer needed tabs, I might have just about 2 GB for NewsBin directly, without swapping.

[dunkem]

I actually think it has something to do with memory and my 32 bit XP system. I guess the download list size has a lot to do with it because I had a massive amount queued up and my mem usage was high at around 700,000K per task manager. When I reduced the download list to maybe 80 GB, mem usage was like 200,000K and files started to assemble.

I haven't tried the memcachelimit setting yet. I'll try that. In general, does the download list amount affect memory usage? I'm just speculating since as I mentioned, things cleared up when I reduced it. I know, I should get rid of this old rig..

Oh.. and no folder is not deep and some file sizes were kind of large so memcachelimit would help there.

[Quade]

I'd say the download list is the #1 user of RAM in Newsbin. Might have to re-design that in the future. I could leave some of that data in the DB3 and only load it right before download for instance.

If you see "Fail to Map" then that means the file was too big for the "MemCache". It only maps the files if they're not cached in RAM. The problem is, some people post single file 600 meg files. If you tried to set a MemCach that big, you'll just run out of real ram instead of virtual. That's why there is a memcachelimit, so really large files don't consume all of your RAM.

The memory problem I'm talking about is lack of address space. Not lack of RAM.

[----------------------------------------------------------] = 3G - address space limit in Win32 (32 bit windows)
[+++++++++++++++++++++++++++++++++++---------] = 3G + 2.5G ram actually used
[+++++++++++++++++++++++++++++++++++******************] - 3G + 2.5G used + 600M filemap

When you file map, if the file is say 600 megs (example) and you're using 2.g gigs of address space already. You can see there's not enough virtual address space. This isn't related to how much RAM you have but how much address space a 32 bit program is permitted to use.

[RayMark]

I think that in Windows 32 (XP at least) each process gets 2 GB address space.
3 GB if a special /3GB switch is used in boot.ini
Even then, only applications that use IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header can access those 3 GB.
Not sure if Newsbin supports 3GB.
But the /3GB switch has some disadvantages as well and I am not using it anyway.
So the address space for Newsbin is 2 GB.
And it is possible to get it all without resorting to virtual memory:
I have about 2.8 GB of free physical memory when almost no other applications are running.
But then you have to be careful what else you are running, better not to get too close to 2 GB with Newsbin.

What about loading large groups, headers for long day ranges at once?
Can memory be controlled in Newsbin in .ini for groups and for download list separately?

[Quade]

Well there you go. Add /3gb and your mapping problems will probably go away.

Yeah, you have the problem I described. On an XP box, I'm assuming you're put /3gb in the appropriate line of boot.ini? Replace the /3gb with /USERVA=2700 (and if it persists, lower that number until it stops)

So, it looks like you can set it to any arbitrary value. I'd try "/USERVA=2500" to start. 32 bit newsbin's been large address aware for awhile now.

You have complete control over how much RAM Newsbin uses.

[RayMark]

As far as I know, /Userva has to be used together with /3GB, not instead of it.

Furthermore, 2700 seems to be a very low value.
In some versions of windows, values lower than 2970 are ignored

I put the following lines into my boot.ini file:

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 2 GB" /noexecute=optin /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 3 GB" /noexecute=optin /fastdetect /3GB
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 3030 MB" /noexecute=optin /fastdetect /3GB /Userva=3030
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 2970 MB" /noexecute=optin /fastdetect /3GB /Userva=2970
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 2900 MB" /noexecute=optin /fastdetect /3GB /Userva=2900
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 2800 MB" /noexecute=optin /fastdetect /3GB /Userva=2800
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP 2700 MB" /noexecute=optin /fastdetect /3GB /Userva=2700

This way I can choose during booting whichever value I want.
This might be useful at least initially until the best value is determined.
This way you will never have to use safe mode to edit the boot.ini, if windows does not start normally, just use a lower value, or no /3GB at all.
BTW, 3030 is considered to be a pretty safe value.
Now I am booted with 3030 and Far (I use Far for absolutely everything) is showing 3,177,054,208 of total virtual memory - so the switches and the choosing during booting are working, I think. Because 3030 MB = 3,177,185,280 bytes - close enough.

[dunkem]

Thanks Quade. Makes sense on the memory / download list. yeah, some of the files are large so I absolutely must fiddle with memcachelimit.

I guess NB6 is far more memory intensive than NB5 in general. I think another thread mentioned long "scan" times when loading NB6 because it has to check all the files to see if they're there. It'll sit and scan each file for a while to mark them "downloaded" before it finds a file that needs to gets downloaded. Same 32 bit memory problems that cause the assemble failure is probably what's slowing me down at startup. Hopefully limiting dowload list size and increasing cachelimit will allow my dinosaur to survive as Newbin becomes more resource heavy.

[Quade]

Same 32 bit memory problems that cause the assemble failure is probably what's slowing me down at startup.

What it does is download the first chunk of every file and check to see if it's on disk and whether autopar knows about it. It should be pretty quick. It might depend on your machine though. I'd suggest making the /3gb change to your setup too. It'll give you an extra gig of memory to play with. You can turn the memory cache completely off by setting it to zero then as long as you keep your ram use below 1.5 gigs or so, or make the /3gb change, you ought to be good to go.

I was under the impression one of the XP service packs turned on /3gb mode but, apparently that's not correct.

[RayMark]

/3GB is only available from SP2, but not turned on by default, not even in SP3

UPDATE:

with /3GB /Userva=3030
and with
MemCacheLimit=100

still was not able to map a single file with the size of 1,397,434,839 bytes.
I had to rename it manually.


Unfortunately, now I keep getting in uTorrent messages "Insufficient system resources exist to complete the requested service" so clearly 3030 is not working for me.
For Newsbin not enough anyway, or a different reason.