newsbin.com

6.4 was pretty stable for me, but I recently ran into two strange new issues:

1. Some downloads appear stuck. Inside, there are red bars, so that means that there are missing pieces. However, it doesn't try to download them again - these downloads stay stuck. If I right click them and select "retry download" they finish downloading properly almost immediately... If a download is stuck for long enough, shouldn't NBP do a retry on the missing pieces?

2. I had two instances where NBP suddenly shut down on its own. I did not turn on the feature that does auto shutdown. I noticed nothing was new, so I saw the program was not running. I ran it again and 5 minutes later (it was not immediate), it happened again. It hasn't happened since, but I suspect something could be wrong here.

What's your server config? Multiple servers, any fills, or just the one? Also, what are your retries set to?

The suddenly quitting thing is certainly an odd one, there's been no reports I've seen of anything like that. The fact that it happened in the same windows session and hasn't happened since suggests it was a bad state your computer got into. There's insufficient information to do much about it, about all you can do is keep an eye peeled for it and try to establish any common parameters. There's nothing persistent about newsbin sessions - when it quits, it's gone, just like any other non-service based application, so the 'solution' (reboot) points to the generic culprit (the OS). Newsbin doesn't have permission to do anything outside of user space.

DT

one server right now. Retries set to the default 10 (nothing changed).

What I've seen in other situations is that when a serious exception happens, a log file gets saved to try and figure out where this happened.

In any case, I will keep my eyes peeled to see if this happens again. It hasn't happened again since those two instances.

A couple of thoughts about the stuck download :

An in-progress upload(or something wrong on the server end) that had done or was in the middle of the retries (there's a wait between tries) when it wasn't available and when you kicked it, it just happened to be finished uploading? The fact that you got red initially and suddenly not later suggests this might be a reasonable scenario.

Server connection issues, where the server has locked up the connection but not long enough to kick you off (thus triggering newsbin to reconnect), and your kick forced a new set of connections? This, I have seen myself.

DT

I am pretty sure your first scenario is what happened in this case. I am using filters, so if someone is in the middle of uploading - it can cause this problem.

So, what you think happened is that that their uploader might have had a problem mid-way, and it took it long enough to cause NBP to give up on retrying?

I was thinking the server had a problem. I have seen this on occasion, it just hangs there. Newsbin is waiting to hear back, but nothing. It's pretty rare, assuming you have a decent news server. I just rack this down to the internet. If you consistently have a problem, then it might be something else.

DT

Nbp 6.40 unexpectedly quit again last night and again this morning.

Looks like its a recurring issue...

How do i help you diagnose it? It's the 32 bit version running on windows 2003

You pushing things really really hard, memory-wise? Just trying to eliminate the 32 bit ceiling, but you really have to work at that in newsbin.

Other than that, I'd just start doing some old fashioned problem solving. If it's all that common, look for common events,like massive download lists. Make sure logging is turned on, check to see the last event. Disable all 3rd party apps like antivirus or non-default firewalls(obviously just for diagnostics). Any unusual things like downloading to network drives, raids, etc. - lose it all to see if that stops the issue. It's always possible it's an issue with 2003 server, I suppose, but I'd be doubting a recent upgrade would be behind it.

DT

not really.

nothing has changed - save A/V, same memory issue. I download locally, but decode to a NAS. Everything on that server is running perfectly.

My NBP config is pretty complex. I have about 4 groups I download everything from, but running them through a long list of regex filters.

No issues on the Win2003 - works perfectly and other stuff is running on it just fine.

The issue seems to have started pretty recently, with one of the latest betas or final 6.4 release.

It happens consistently. I have to run the software all the time - it just bugs out.

How do I read the log file without running the software (as soon as the software loads, I get a lot of items logged on).

In the Newsbin DATA folder there's a log file called logfile.txt. It gets overwritten with each run of Newsbin unless you set a hidden NBI variable to make it persistent.

[LOGGING]
Append=1

How much do you have in the download list?

actually, it would be better not to make it persistent, and just to take a look once NBP craps out.

I'll keep you posted.

Quade,

Not much. It happened when I had 1-2 items too.

How do I read the log file without running the software (as soon as the software loads, I get a lot of items logged on)

You might have to drag and drop to make a copy of the log file while Newsbin is running. You don't want to try to read the actual file while Newsbin is running. The log file is just a text file. As long as it doesn't get too big, you can read it with WordPad.

I only care about it after nbp is no longer running.

As I mentioned above, you'll want to problem solve, that is, start disabling things, one at a time, that are specific to your environment. In your case, I would:

1. Disable the AV. AV's change their behaviour with every update. I think twice in as many weeks there's been catastrophic behaviour reported here that got traced back to AV updates.

2. Disable the complex filters. Still happen?

3. Same memory issue? Don't follow. If you mean the memory hasn't changed, cool, but monitoring memory use is useful. That might help track down an issue. The fact this tends to only show up when running for long periods might be a clue. One AV issue that happened was the AV was snatching data and holding on to it, then passing to newsbin. Memory exploded.

DT

I monitored memory. I don't think that's the problem but I will keep an eye out.

ok, it happened again at 1:38am.

logfile doesn't show anything special:

[01:22:48] DEBUG - Autodownload Added: 0 Out of 19
[01:22:48] DEBUG - DbWorker: Db Updated:alt.binaries.hdtv:ssl-39190953.rv4
[01:24:44] DEBUG NNTPServerWorker - Download: Idle Connection Closed:SSL
[01:24:44] DEBUG NNTPServerWorker - Download: Idle Connection Closed:SSL
[01:24:44] DEBUG NNTPServerWorker - Download: Idle Connection Closed:SSL
[01:24:44] DEBUG NNTPServerWorker - Download: Idle Connection Closed:SSL
[01:24:44] DEBUG NNTPServerWorker - Download: Idle Connection Closed:SSL
[01:37:57] LOW - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5 168 SSLv2
[01:37:57] LOW - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5 168 SSLv2
[01:37:57] LOW - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5 168 SSLv2
[01:37:57] LOW - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5 168 SSLv2
[01:37:57] LOW - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5 168 SSLv2
[01:37:58] DEBUG NNTPServerWorker - DownloadRecords - Total XOVER Size:alt.binaries.multimedia - 1055531670:1055564312
[01:37:58] DEBUG NNTPServerWorker - DownloadRecords - Total XOVER Size:alt.binaries.hdtv.x264 - 2172741382:2172773434
[01:37:58] DEBUG NNTPServerWorker - DownloadRecords - Total XOVER Size:alt.binaries.tv - 518286080:518288272
[01:37:58] DEBUG NNTPServerWorker - DownloadRecords - Total XOVER Size:alt.binaries.hdtv - 2573801861:2573841144
[01:37:58] DEBUG NNTPServerWorker - DownloadRecords - Total XOVER Size:alt.binaries.teevee - 429031737:429035937
[01:37:58] DEBUG NNTPServerWorker - XOVER: alt.binaries.multimedia 1055531670:1055564312
[01:37:58] DEBUG NNTPServerWorker - XOVER: alt.binaries.hdtv.x264 2172741382:2172773434
[01:37:58] DEBUG NNTPServerWorker - XOVER: alt.binaries.tv 518286080:518288272
[01:37:58] DEBUG NNTPServerWorker - XOVER: alt.binaries.hdtv 2573801861:2573841144
[01:37:58] DEBUG NNTPServerWorker - XOVER: alt.binaries.teevee 429031737:429035937
[01:37:59] DEBUG - WriteRecords - WriteRecords Elapsed:0:48
[01:37:59] DEBUG - Checking Autodownload: alt.binaries.tv
[01:37:59] DEBUG - Autodownload Enabled: alt.binaries.tv
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 [ *** ]-[18/25] - "***" yEnc
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 (241/351) "***" - 5,20 GB - yEnc
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 (***) [368/732] - "***" yEnc
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 (227/351) "***" - 5,20 GB - yEnc
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 (255/351) "***" - 5,20 GB - yEnc
[01:37:59] HIGH - Autodownload File Filtered out 2147500036 [ *** ]-[03/25] - "***" yEnc
[01:37:59] DEBUG - Autodownload Added: 0 Out of 6
[01:37:59] DEBUG - DbWorker: Db Updated:alt.binaries.tv:ssl-40104031.rv4
[01:37:59] DEBUG - WriteRecords - WriteRecords Elapsed:31:184
[01:38:00] DEBUG - Checking Autodownload: alt.binaries.teevee
[01:38:00] DEBUG - Autodownload Enabled: alt.binaries.teevee
[01:38:00] HIGH - Autodownload File Filtered out 2147500036 [112981]-[FULL]-[#a.b.teevee@EFNet]-[ *** ]-[14/50] - "***" yEnc
[01:38:00] HIGH - Autodownload File Filtered out 2147500036 [112979]-[FULL]-[#a.b.teevee@EFNet]-[ *** ]-[11/31] - "***" yEnc
[01:38:00] HIGH - Autodownload File Filtered out 2147500036 [112980]-[FULL]-[#a.b.teevee@EFNet]-[ *** ]-[21/26] - "***" yEnc
[01:38:00] HIGH - Autodownload File Filtered out 2147500036 [112981]-[FULL]-[#a.b.teevee@EFNet]-[ *** ]-[37/50] - "***" yEnc

I have filtered out file names from this log, let me know if you want me to send the full thing by email.

I have turned the A/V off.

Hmm, wonder if it's related to the filter approach - typically one tries to define things they want, as opposed to get everything but let me define what I don't want. I mean, it's unlikely, but I wonder if you're hitting a complexity ceiling?

I can understand if you don't want to test it, but I wonder if you instead just added enough stuff you know you want to keep the pipes busy overnight, disable autodownload and all those filters and see if it repeats. Seems like a long shot but if you could repeat the problem without those settings it would at least eliminate them.

Did you have a chance to monitor memory before you hit the hay? I'm wondering if you saw any creeping increase in memory use. It wouldn't necessarily be newsbin eating memory. It fact, I'm doubting it would. At least it's not the AV, as long you're certain it was off off off. Some of them are pretty insidious. Which one you using?

DT

Autoload isn't particularly heavily tested. I use it in groups. I do use filters but, they aren't that complicated. The question is whether Newsbin is exiting or crashing. I might set the PC in power management so, it never goes to sleep (as an experiment) and see if that makes any difference. I'd verify in the advanced options that you don't have autoclose enabled or windows
shutdown.

Have you tried loading the group and then applying the filters to the visible posts? Does it seem to work?

Dex and I are getting closer with a 6.41. If this is still going on when it goes to beta. I can add additional logging to see if we can figure this out.

What I do is set auto download and add some regex filters that specify just the stuff I am looking for.

Would it help to send the nbi file to you?

Memory consumption is about 128Mb. It is pretty consistent. I don't think it's a memory leak.

Computer never sleeps. It is a server.

I checked. The auto close option is off and auto shutdown is also off.

Loading a group and applying filters works perfectly, it's my favorite part of the software. I used it just now and no problems.

ok, no A/V and it happened about an hour ago.

Code: Select all

[15:15:19] DEBUG   - Autodownload Added: 0 Out of 20
[15:15:19] DEBUG   - DbWorker: Db Updated:alt.binaries.tv:ssl-89142750.rv4
[15:15:20] DEBUG   - FileIOThread - AssembleFile: Got AVI XVID? [05/12] - "***" [Recent XVID - Please post Chelsea Lately - (please index as torrent)]
[15:15:20] DEBUG   - AssembleFiles: ***
[15:15:20] DEBUG   - FileIOThread - AssembleFile: Got AVI XVID? [06/12] - "***" [Recent XVID - Please post Chelsea Lately - (please index as torrent)]
[15:15:20] DEBUG   - AutoPAR: ScanFile Sliced:1 File is Perfect: ***
[15:15:20] DEBUG   - AutoPAR: Scan 1 Blocks File: ***
[15:15:20] DEBUG   - Par File added :***
[15:15:20] DEBUG   - AssembleFiles: ***
[15:15:20] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:21] DEBUG   - FileIOThread - AssembleFile: Got AVI XVID? [00/12] - "***" [Recent XVID - Please post Chelsea Lately - (please index as torrent)]
[15:15:21] DEBUG   - AssembleFiles: ***
[15:15:21] DEBUG   - WriteRecords - WriteRecords Elapsed:93:202
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] LOW     - Connection Status: SSL Connection Server: ssl.astraweb.com Stats: DES-CBC3-MD5  168 SSLv2
[15:15:22] DEBUG   - WriteRecords - WriteRecords Elapsed:407:396
[15:15:23] DEBUG   - Checking Autodownload: alt.binaries.hdtv

Just an update. After a few days where the issue recurred twice daily or more, it stopped. It has not happened again.

I am convinced it is related to something on the server side.

It's back. Just had two instances of nbp shutting down unexpectedly.

it just crashed in front of me (not responsive for a few seconds, then dissapears).

It was downloading headers from two groups while it happened.

Updating to 6.41

I think it's worthwhile keeping the discussion here rather than splintering it over to the release announcement thread. Quade reads all new messages constantly so no need to echo, makes it easier to keep track of what you've tried.

As far as further diagnostics, I think you need to pull back on the throttle until it stops breaking. You appear to have a very special case, it's hard to say whether it's specifically newsbin or the environment it's running in, or a combination. I would be running the same setup, but remove one thing at a time - like remote extraction, autoload, or complicated filters, until it suddenly is rock solid. I realize that's a pain, but until the root cause, or at least a strong clue, is revealed, there's too many variables to debug. Quade has mentioned autoload hasn't been heavily tested, that's probably a good starting place. Quade might have some other thoughts, but that's how I debug...

DT

DThor.

Normally, I would agree, but in this case - I had two weeks of no issues without any change to the configuration. The crashing is so rare and erratic that I unless I run in each configuration for a few weeks, we won't really know for sure if the problem is related to a particular setting or just because something's changed on the back end...

I believe the issue is related more to communication with the server rather than extraction or something like that. I watched it crash twice today while updating a particular group.

Could be. Perhaps a combination of downloading and injecting headers. As usual, unless quade can repro it, that makes it something on your end. Not really sure what to suggest next, except perhaps try and get it to crash while interacting with it, or maybe bypass filters? You had mentioned they were particularly dense...

DT

Would it be possible to send you my configuration (NBI) or something similar?

Quade mentioned a version of the software that would log much more, so we can find out what triggers it.

Not me, but Quade perhaps. I'm just a user.

Probably want the filters.db3 too, no?

DT

Just suggesting. I can send anything that's needed.

This sounds similar to an issue I have been having, with lockups, it stops happening when you turn off autopar but after some trials I suspect the its to do with the testing newsbin runs on the first par2 file having missing parts or newsbin finding something it doesn't like and rather than carrying on with the download it locks up until you close and restart it.

What's your RAM usage look like when this happens?

I'd go into the performance options and check the "Write Downloaded Files with no Memory Buffer" option and see how it reacts. You'll want to restart for it to take.

The problem continues with 6.41b3.

I've been having numerous lockups (the software appears to be running, but it is frozen) or drop outs (I see in the morning that it stopped working during the night).

I did not notice any issues with RAM, but the problem started around 6.40 I believe.

I also don't think it's something local that's triggering it. I think it's something that happens server side - as the software runs flawlessly for a few days, then continually drops out or crashes. I have a feeling it has something to do with DMCA takedowns that happen while the software tries to download something.

I turned on the "no memory buffer" checkbox. Lets see if that does anything.

OK, it just happened right in front of me (third time today). First there was "program not responding" and then in dissapeared.

This time it happened with "no memory buffering" on and the autoPAR turned off...

I'm outta ideas. It happens constantly now...

I can see in the event viewer the following message:

Faulting application newsbinpro.exe, version 6.4.1.0, faulting module newsbinpro.exe, version 6.4.1.0, fault address 0x00330f08.

I would go back prior to the release you think the problem started at, see if you can reproduce. My guess is that it still happens. As much as it's not what one wants to hear, I think something has become pooched on your system. If this was a common occurrence then it's a bug, but in this case it's really sounding like a local issue. Quade is happy to squash bugs, but he needs that repro otherwise he's just guessing.

If it was me, I'd do a clean install of newsbin, and run it in the most virgin mode possible. No heavy filters, all local. If it still happened, I've got problems. I'd try a system reinstall, frankly. It might even be something like a NIC driver issue, disk controller, memory or power supply. But, OS reinstall first. Then, *before* I customized my setup, tweaking services and all that crap I do with every install, reinstall newsbin and test it clean. Many times I've discovered that system customization I've done has unexpected side effects.

Maybe someone else has some ideas, but that's where my head would be at if I were in your scenario. A clean OS install can make the world of difference. Sometimes software leaves an odious trail of taint that breaks things even though you've uninstalled it.

DT

Dude, this is a heavy system - I run tons of stuff on it without a problem...

If there were NIC issues, memory issues, I would have seen them long before.

The system is stable as a rock...

NBP bombs out once in a while and I should spend a week reinstalling my OS? That's nuts.

Maybe it's time to wipe and re-install Newsbin? I mean complete wipe, maybe keep the NBI file.


You're having an issue that's no generally reported so, either something is in a bad state or you're using is in a way I'm not expecting.

OK, I cleaned everything up.

reinstalled 6.41b3

Then returned the nbi, and filters.

Everything else is clean.

I disabled autodownload for the groups I follow. autoPAR also disabled.

It's catching up on the headers now. Next, I'll start without the auto download and autoPAR and slowly turn these features on over the next few days.

So far so good. No failures since the cleanup.

Nice

Everything worked for a week and then the issue returned.

The program just crashed out twice...

Seems like you hit some critical threshold. Maybe you can note how much you had in the download list before and after the crashing?

It is not that big. I have had 20 or 30 items. Now, it was about 10 items,mostly ones stuck in there.

Since this happened, no further crashes...

As I said before, I think this has something with the communication to the server. Issa sure this was nailed when I ran clean...

Maybe someone else has some ideas, but that's where my head would be at if I were in your scenario. A clean OS install can make the world of difference. Sometimes software leaves an odious trail of taint that breaks things even though you've uninstalled it.

Gotta agree with you on this one. Been using computers since dos 3 days, and over the years, with all OS, I have found sometimes you just cant fix a prob. It may be just one program, but a fresh install can give amazing results.

I have always done a clean install with a several backup images as I progress with installing all my programs. Takes 2-5 minutes to boot back from an image where you know you are good. All it takes is one program conflict, say one using older dll file, you have probs. Sometimes, I have found it all depends on what order you install certain software. It has gotten much better these days compared to the past.

Not gonna happen.

I have too much on this computer to afford this. I think this type of solution is a cop out. Are you going to rebuild your house if you can't fix a leaky faucet? Probably get another plumber if he tells you to rebuild your house...

The os is stable, it's not exhibiting any other issues except one program crashing 2-4 times a week.

oferlaor wrote:Are you going to rebuild your house if you can't fix a leaky faucet? Probably get another plumber if he tells you to rebuild your house...

I think a more appropriate analogy is that your house is riddled with termites. It is still standing during fair weather (most applications with low resource requirements) but when a hurricane comes along (Newsbin downloading at full speed and possibly performing complex file repair and unrar functions) your house could get blown down. In that case, you should rebuild your house, it's beyond repair because of the damage riddled throughout.

If your crash issues were really with Newsbin then we'd be getting 1000's of crash reports per day. As it is we get a couple a week maybe, and it's usually from people running older versions with known issues.

I understand and that's why I am offering to send the nbi file.

There are no problems with that computer. It is rock solid and has been running flawlessly with various apps.

I watched nbp crash when it had 5 downloads and nothing too heavy running.

On the other hand it runs fine with lots of downloads and pars at full speed.

I am a developer too. I don't remember ever asking my customers to format their pc. If something crashed I would send them a debug version and figure out what was causing the issues.

I don't have a handle on the crashing myself. One thing worth trying is perhaps trying it on a different PC.

I wasn't recommending a re-install. I really don't know what the issue is. I'm skeptical that additional logging would help.