newsbin stalling

[catflyer]

Just bought Newsbin.

I queue up some files to download and newsbin works for a while and then just......stalls. The only way I can start it downloading again is to quit and restart. Is there anything I can do to diagnose it?

[Quade]

What version are you running? I'd try 6.2 if you're not already running it.

[catflyer]

I'm using 6.11. Will my current subscription allow me to upgrade to a newer version?

[itimpi]

I'm using 6.11. Will my current subscription allow me to upgrade to a newer version?

Yes - all Newsbin upgrades are free.

[DThor]

It's unlikely the latest version will solve your problem, so to anticipate your post, you should try completely disabling your antivirus and see if that helps. Most of the time, they are the cause of unexplained stoppages. Also, check the logging tab for any meaningful messages.

Running the latest is best, though. Should there be a problem with the software, that's the baseline.

DT

[Swede1]

Also have a stalling issue with 6.20 which can be "eliminated" by reverting to 6.11.
No changes have been made in any configuration between 6.11 -> 6.20.

Basically I start up Newsbin and it downloads for a while what's in the queue and then halts. Subsequently I can't download headers or restart stalled posts.
Shutting down Newsbin and restarting it repeats the procedure.

The log from startup to fail looks like this:

Code: Select all

[11:56:30] HIGH    - High Speed File Creation Mode: Enabled
[11:58:24] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[11:58:24] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[11:58:24] HIGH   Download Engine news-europe.giganews.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[11:58:29] ERROR  NNTPSocket  - NNTP Perform Auth - news.giganews.com-The server rejected your login - contact your news service's support department. It's not a Problem with Newsbin : 481 (remote) (aucl:gn;gnxxxxxx) exceeded maximum number of connections per user
[12:02:31] ERROR  NNTPSocket  - SendCommand Worker - Server:europe.newsdemon.comSendCommand - failed to Read Response From server: Socket - Read From Server Failed
[12:02:31] ERROR  NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Disabled Service on connection:
[12:02:37] ERROR  NNTPSocket  - NNTP Perform Auth - news.giganews.com-The server rejected your login - contact your news service's support department. It's not a Problem with Newsbin : 481 (remote) (aucl:gn;gnxxxxxx) exceeded maximum number of connections per user
[12:03:03] ERROR  InterSocket  - Error: Connection Failed, Host: europe.newsdemon.com Error Code: 10060 A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
[12:04:49] ERROR  InterSocket  - Error: Connection Failed, Host: news-europe.giganews.com Error Code: 10060 A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
[12:05:51] ERROR  NNTPSocket  - SendCommand Worker - Server:europe.newsdemon.comSendCommand - failed to Read Response From server: Socket - Read From Server Failed
[12:05:51] ERROR  NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Disabled Service on connection:


Newsbin : 6.20 00 E9 EC 3E A0 64
Windows7-64 bit, 16 GB RAM

I have 4 servers from 2 providers, 1 master from each provider and 1 slave from each provider (yeah, I know it's overkill).
Looking at the configuration they are obviously the same as in 6.11. Noticed 1 thing though, only one server (europe.newsdemon.com) has the option "Disable Xfeatures" checked (why I don't know).

So I unchecked it and re-started and the problem remains, this time with no errors in the log file.
EDIT: The checkmark re-appears for the one server upon re-start even if I uncheck it and save the configuration.

I then set the logging level in the logging TAB to DEBUG and started NewsBin again (and get flooded with messages) but maybe the last messages can shed some light on the problem:
These just repeats for ever after dowloading halts:

Code: Select all

.
.
.
[12:49:16] DEBUG   - Cache Miss: Larger chunk cache might perform better
[12:49:16] DEBUG   - Cache Miss: Larger chunk cache might perform better
[12:49:16] DEBUG   - Cache Miss: Larger chunk cache might perform better
[12:49:16] DEBUG   - Cache Miss: Larger chunk cache might perform better
[12:49:16] DEBUG   - Cache Miss: Larger chunk cache might perform better
.
.
.

So I set the DEBUG level to Verbose but that now only generates one entry now (downloading still comes to a halt):

Code: Select all

[12:52:05] HIGH    - High Speed File Creation Mode: Enabled

Any suggestions ?

[DThor]

The debug comments aren't important, the meat is in your first information. This is likely unrelated to this thread.

Code: Select all

Server Reports that a Post isn't on the server: 430 No Such Article

This is from the server, which means you're actually successfully communicating with it. What it says: that particular post isn't there.

Code: Select all

contact your news service's support department. It's not a Problem with Newsbin

What it says. Contact them and find out what's up. Your account, login name, password or server is wrong.

So these are two different things, and neither indicates a Newsbin problem. I'm a little skeptical after that last one you can downgrade to the previous version and all is well, I'm suspecting this happened around the time you upgraded and you're assuming it's related. Note that the 'not on server' message means all was well at that time. So, get that sorted first, ensure your account with them is in order. If somehow it's OK, I suppose it's possible that your antivirus in getting between you and the server, somehow munging the communication. Try disabling it(completely) and see if that helps.

DT

[Quade]

How many connections are you running? What I see is that you're getting disconnected and that eventually Giganews says "you have too many connections".

A 430 shouldn't result in a disconnect but, maybe it is. Rapid disconnects confuse the server.

So I see 2 things.

1 - Trying to download a file that Giganews says it can't deliver.

2 - Giganews eventually saying "You now have too many connections".

I'll check the "does 430 cause disconnect" thing. That might be a problem.

[Swede1]

The debug comments aren't important, the meat is in your first information. This is likely unrelated to this thread.

Code: Select all

Server Reports that a Post isn't on the server: 430 No Such Article

This is from the server, which means you're actually successfully communicating with it. What it says: that particular post isn't there.

Code: Select all

contact your news service's support department. It's not a Problem with Newsbin

What it says. Contact them and find out what's up. Your account, login name, password or server is wrong.

So these are two different things, and neither indicates a Newsbin problem. I'm a little skeptical after that last one you can downgrade to the previous version and all is well, I'm suspecting this happened around the time you upgraded and you're assuming it's related. Note that the 'not on server' message means all was well at that time. So, get that sorted first, ensure your account with them is in order. If somehow it's OK, I suppose it's possible that your antivirus in getting between you and the server, somehow munging the communication. Try disabling it(completely) and see if that helps.

DT

Thanks for your reply.
I'm well aware of the potential host problems (login failed, exceeded number of connections, post not on server etc.) That's not part of the problem. That particular(first) log entry was just because of the particular files I tried downloading at the moment. Subsequent downloads have no problems communicating with the server nor having "posts not on the server" problems. Nor do I have login problems with any of the two providers yet the problem om "slowly coming to a halt" persists.

All I get now in the log is the "[12:52:05] HIGH - High Speed File Creation Mode: Enabled" message and then the download slowly dies and comes to a halt.

My entire point is that 6.20 slowly dies to a halt when 6.11 doesn't. If the proverbial "disable your anti-virus" is the solution then the symptoms should be the same on 6.11.
Yes, just to be sure, I now did disable the real-time virus checking on Microsoft Security Essentials, ran 6.20 and things still slowly comes to a halt. Anti-virus is NOT the problem.

It was suggested in earlier posts that the DEBUG level "cache chunk size" message was due to disk throughput. Not in my case, SPOOL_V6 folder is one one RAID0 disk set, the download directory is on a separate RAID0 disk set.

If the problem only was related to the point in time that 6.11 was upgraded to 6.20 (as you say) then why can I at will eliminate the problem by reverting to 6.11 ?

Most likely Newsbin 6.20 related problem, OR an existing server problem brought into light by NewsBin 6.20 ...

I'm not out to point fingers at NewsBin, I'm looking for clues on how to debug further.

[DThor]

OK, now you've pointed out the relevant part, the problem is almost certainly your AV.

If the proverbial "disable your anti-virus" is the solution then the symptoms should be the same on 6.11.

Nope. The latest version can seem like a new App to AV. My point is, completely disable your AV, see if that helps. My guess is, it will.

I missed the 'too many connections' thing first time around. If they still keep showing up, that's important, that will cause you problems downloading. Set your connections lower and see if that helps.

DT

[Swede1]

How many connections are you running? What I see is that you're getting disconnected and that eventually Giganews says "you have too many connections".

A 430 shouldn't result in a disconnect but, maybe it is. Rapid disconnects confuse the server.

So I see 2 things.

1 - Trying to download a file that Giganews says it can't deliver.

2 - Giganews eventually saying "You now have too many connections".

I'll check the "does 430 cause disconnect" thing. That might be a problem.

I have the following configuration (both my providers allow 50 connections on the plans I purchase):

europe.newsdemon.com - 50 connections
news-europe.giganews.com 50 connections
news.giganews.com 50 connections (No header download, Fill server)
useast.newsdemon.com 50 connections (No header download, Fill server)

Yes, I realize that all 50 connections could be used on one "Master" server and then a "fill server" kicks in and I get the "Too many connections" error.
But if that scenario happens, should that then eventually (slowly) disable communication ? And if so, why don't I see any error messages of that nature ?
EDIT : And why wasn't this a problem on 6.11 (same server configuration) ?

As I said before on the "Verbose" log setting, all I see is this "[15:39:05] HIGH - High Speed File Creation Mode: Enabled" and then it comes to a halt (takes a few minutes each time).

Looking at TCPView (or via netstat command), The TCP/IP connections to the servers are still "ESTABLISHED".
EDIT: All ports eventually close after corresponding TCP time-out.

Maybe I should switch on the NNTP commands logging to see what happens ?

[Swede1]

OK, now you've pointed out the relevant part, the problem is almost certainly your AV.
Nope. The latest version can seem like a new App to AV. My point is, completely disable your AV, see if that helps. My guess is, it will.
DT

If you read the previous post I did just that to no avail.
Besides why would an anti-virus slowly disable communication ?

[Quade]

Yes, I realize that all 50 connections could be used on one "Master" server and then a "fill server" kicks in and I get the "Too many connections" error.
But if that scenario happens, should that then eventually (slowly) disable communication ? And if so, why don't I see any error messages of that nature ?
EDIT : And why wasn't this a problem on 6.11 (same server configuration) ?

I'm not saying it's not a problem in 6.2. So, you don't have to repeat "6.11 works" over and over again. I explained what I think is happening though.

430,430,430...too many connections.

That's what your logging suggests. I'm going to look and see if 430's cause disconnects. Servers can't take rapid connects and disconnects. They all get confused when that happens. They mis-count. 430's in general look like a stall. Particularly if all the servers are returning them.


50 connections is pretty silly though. I'll bet if you set each server to 20 it would be no slower and your CPU load and RAM use would be vastly reduced. And yeah I know it works under 6.11. When you run the connections at the max like this, there's no margin for error. Meaning it's real easy to get "too many connection errors" because you're already at the limit. Newsbin was never tested with 200 connections so, you're sort of on your own here.

[Swede1]

I'm not saying it's not a problem in 6.2. So, you don't have to repeat "6.11 works" over and over again. I explained what I think is happening though.

50 connections is pretty silly though. I'll bet if you set each server to 20 it would be no slower and your CPU load and RAM use would be vastly reduced. And yeah I know it works under 6.11. When you run the connections at the max like this, there's no margin for error. Meaning it's real easy to get "too many connection errors" because you're already at the limit. Newsbin was never tested with 200 connections so, you're sort of on your own here.

Point taken

I backed down the connections to 25 per server and so far so good!
I get 100+ Mbitps average download speed (my connection is 100 Mbitps fiber).
No stalling but varying download speed especially during the UnRAR assembly process, when download speed of other items in the download queue dips to 0 bps, then comes back up.
Yes, I have ample CPU, RAM and Disk I/O power.

The log file now looks like this:

Code: Select all

[16:45:49] HIGH    - High Speed File Creation Mode: Enabled
[16:51:56] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[16:51:56] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[16:51:56] HIGH   Download Engine news-europe.giganews.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[16:55:39] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[16:55:39] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[16:55:39] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[17:02:57] HIGH   InterSocket  - SSL Zlib Compression Added
[17:02:57] LOW    InterSocket  - SSL Connection Server: is.newsbin.com Stats: DHE-RSA-AES256-SHA zlib compression 256 TLSv1/SSLv3
[17:03:30] HIGH   InterSocket  - SSL Zlib Compression Added
[17:03:30] LOW    InterSocket  - SSL Connection Server: is.newsbin.com Stats: DHE-RSA-AES256-SHA zlib compression 256 TLSv1/SSLv3
[17:03:57] HIGH   InterSocket  - SSL Zlib Compression Added
[17:03:57] LOW    InterSocket  - SSL Connection Server: is.newsbin.com Stats: DHE-RSA-AES256-SHA zlib compression 256 TLSv1/SSLv3
[17:04:12] HIGH   InterSocket  - SSL Zlib Compression Added
[17:04:12] LOW    InterSocket  - SSL Connection Server: is.newsbin.com Stats: DHE-RSA-AES256-SHA zlib compression 256 TLSv1/SSLv3
[17:06:25] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:06:25] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:06:25] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com

<Numerous similar messages deleted>

[17:16:07] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[17:17:40] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:17:40] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:17:40] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[17:17:40] ERROR  Socket  - Error: WSARecv Socket Error Code: 10054 An existing connection was forcibly closed by the remote host.

[17:17:40] ERROR  NNTPSocket  - SendCommand Worker - Server:news.newsdemon.comSendCommand - failed to Read Response From server: Socket - Read From Server Failed
[17:17:40] ERROR  NNTPSocket  - Socket HandleError - Server: news.newsdemon.com SendCommand - Server Disabled Service on connection:
[17:19:04] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:19:04] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:19:04] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[17:20:38] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:20:38] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:20:38] HIGH   Download Engine europe.newsdemon.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com
[17:22:55] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:22:55] HIGH   NNTPSocket  - Socket HandleError - Server: europe.newsdemon.com SendCommand - Server Reports that a Post isn't on the server: 430 No Such Article
[17:22:55] HIGH   Download Engine news-europe.giganews.com  - INT_OpenItem -DL_DI - Failed to retrieve Article from Server::europe.newsdemon.com

Anywy, I can live with this.
Case closed, this is no longer an issue.









...and it had nothing to do with Anti-virus ...

[Quade]

I've confirmed that 430's don't disconnect the connection.

No stalling but varying download speed especially during the UnRAR assembly process, when download speed of other items in the download queue dips to 0 bps, then comes back up.

There are actually some hidden options in Newsbin that are off by default that can tend to keep the speed up when unraring. It depends on your disk drives though.

[SETTINGS]
MemCacheLimit=300


If you exit Newsbin and add this to the configuration file, Newsbin will stage up to 300 megs of the downloaded files in memory before writing them out on one shot. It's turned off because people running 32 bit machines don't have enough address space or RAM for it.

On my machine, I can sustain 400 Mbps file downloads, which drops to 250ish when unraring. That's with 10 connections to my local server.

[Swede1]

I've confirmed that 430's don't disconnect the connection.
There are actually some hidden options in Newsbin that are off by default that can tend to keep the speed up when unraring. It depends on your disk drives though.
[SETTINGS]
MemCacheLimit=300
If you exit Newsbin and add this to the configuration file, Newsbin will stage up to 300 megs of the downloaded files in memory before writing them out on one shot. It's turned off because people running 32 bit machines don't have enough address space or RAM for it.
On my machine, I can sustain 400 Mbps file downloads, which drops to 250ish when unraring. That's with 10 connections to my local server.

I added the settting you suggested and i get about 160 Mbps on a pure header download. Around 100 Mbps while downloading file as well as unRARing.
This is from "over the internet" servers and not a local one as yours.
Yes, improvement!