Inability to download articles

[Tarkus]

I am on 6.72 Build 4776 accessing Newsgroupdirect. Everything has been working fine with this service since August 2016 when I switched to it. Over the last few weeks we have seen an enormous amount of spamming that has seriously increased the download header time and then the refresh time for the group. I somewhat fixed this issue by reducing the display and storage age down to 15 days. Over the last few days nothing downloads at all. I see the first file, (the first par), as downloaded and then everything else is 'paused'. If I 'resume download' the status bar turns eventually red as if the articles have been deleted.

Now I know that NGD has a bit of a bad reputation but in reality their download feed is EasyNews so one would expect the articles to be there. I will get the "host rejected your login it is not Newsbin" message but why is it possible for NB to login and download the headers? Also why is it possible that if I use Agent that I can download files that NB cannot? Clearly the login credentials are working and more importantly NGD is not pulling some scam of letting me download the headers but then refusing the file download.

So I am wondering if all of these spamming messages have somehow corrupted the data file. I have deleted one group and then reloaded it but the same failure occurred.

Any suggestions would be appreciated.

[Quade]

. If I 'resume download' the status bar turns eventually red as if the articles have been deleted.

If what you're seeing is purely 430 errors on the logging tab then this is a server issue with the servers removing files. The only fix is to download quicker or find another (probably non-US) based server which doesn't DMCA or doesn't DMCA so fast.

Servers do change. So you might find a server that was fine, all of a sudden be more aggressive about removing files.

[Tarkus]

. If I 'resume download' the status bar turns eventually red as if the articles have been deleted.

If what you're seeing is purely 430 errors on the logging tab then this is a server issue with the servers removing files. The only fix is to download quicker or find another (probably non-US) based server which doesn't DMCA or doesn't DMCA so fast.

Servers do change. So you might find a server that was fine, all of a sudden be more aggressive about removing files.

I guess I didn't explain things that well.

1. The status bar turns red as if to suggest the post has been deleted.
2. However if I then go to Agent, (which I essentially use just to poll what is in large groups and not for download since it is so slow), I can download the post even though NB is seemingly indicating that the post is gone, (not to be a braggart but I've been using NB since around 2000 and NNTP since the late 80's so I know when a post has been removed).

This is why I am wondering if the data base has become corrupt or if the sheer number of posts is messing up its ability to download so after the program thinks it hasn't gotten any articles it assumes that the post is not there.

I have seen your solution to the spamming to avoid having those headers added to the data file but I am a little unclear how this can be a preventative solution since the poster doesn't just post one topic but rather thousands of topics. So that said I would have to load NB and then the group to see what username(s) he is using today and make the change the NBI file then quit NB and reload. I do not see a way to set up the filters in advance simply because I would not know his posting setup until I have loaded the Newsgroup.

[Quade]

Need to check the logging tab when this happens.

Most common is 430 errors. If you have some other error, there's no way for me to tell until I know what errors they are.

I have seen your solution to the spamming to avoid having those headers added to the data file but I am a little unclear how this can be a preventative solution since the poster doesn't just post one topic but rather thousands of topics. So that said I would have to load NB and then the group to see what username(s) he is using today and make the change the NBI file then quit NB and reload. I do not see a way to set up the filters in advance simply because I would not know his posting setup until I have loaded the Newsgroup.

You can either ask someone or look in the group and figure it out. I have no idea what you want to filter out or filter in. If I was doing TeeVee I might do a "subject accept" for "\[FULL\]" instead of a poster reject and simply only accept the posts I know are probably good. You can filter IN posts as well as filter them OUT. Filter "IN" is probably the best way.

[Tarkus]

Need to check the logging tab when this happens.

Most common is 430 errors. If you have some other error, there's no way for me to tell until I know what errors they are.

I have seen your solution to the spamming to avoid having those headers added to the data file but I am a little unclear how this can be a preventative solution since the poster doesn't just post one topic but rather thousands of topics. So that said I would have to load NB and then the group to see what username(s) he is using today and make the change the NBI file then quit NB and reload. I do not see a way to set up the filters in advance simply because I would not know his posting setup until I have loaded the Newsgroup.

You can either ask someone or look in the group and figure it out. I have no idea what you want to filter out or filter in. If I was doing TeeVee I might do a "subject accept" for "\[FULL\]" instead of a poster reject and simply only accept the posts I know are probably good. You can filter IN posts as well as filter them OUT. Filter "IN" is probably the best way.

I just attempted a download a download:

a.b.teevee <something>.S01E01.DVDrip.HVEC.x265.F1rebladerunner [03/22] - "<something>.S01E01.DVDrip.HVEC.x265.F1rebladerunner.part01.rar" yEnc

The following errors occurred:

[16:41:51] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:42:46] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:08] ERROR NNTP Perform Auth - NEWS SERVER ERROR: NGD_US-The server rejected your login - contact your news service's support department. It's not a Problem with Newsbin : 481 be a cool kid
[16:43:38] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:38] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:38] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:38] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:40] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:40] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address
[16:43:43] ERROR Server:NGD_US NNTPServer: Timeout - Waiting for Server to send Data: Socket - Read or Write TimeoutSocket - Already ClosedSocket - General ErrorSocket - Connection To Server FailedSocket - Write To Server FailedSocket - Read From Server FailedDNS - Unable to Find Server's AddressDNS - Unable to Find ServiceDNS - Unable to Find Peer Address

Of note if my login was rejected why did the group download the headers?

Then I attempted the same download using Forte Agent. Granted it takes forever, (not sure why Agent is so horrid at downloading), but the file downloads completely and assembles properly and plays.

So I am confused as to why NB is reacting so differently? Could it be that Newgroupdirect is treating NB differently? How would they even do that and if so why would NGD accept header download requests from NewsBin but then reject article downloads?

I am fully aware that one has to be careful about making false correlations when it comes to diagnosing technical issues but on the surface I have had zero issues with NewsBin and NewsGroupDirect, (recognizing as you point out that some posts are DMCA deleted quickly but that is beyond the control of either NB or NGD), until this massive spam attack.

Would you suggest that I physically delete all of the data files for the groups and do a reload? Would it make sense to first change the storage and display duration to maybe 1 day and then purge the groups or again why not just delete everything and start fresh?


Regarding the filters I do realize that it is a difficult process to strategize, (not really hard to do once a template is set ... > btw may not a bad idea for you to build a text file with your example of the filter so people can just cut and paste and then edit), the filter since you are essentially having to figure out what NB is going to encounter in the future. Maybe you can explain to me what I am not getting with your logic on this one. In your post about filters you suggest:

----
For example I wanted to filter out:
"Anime Tosho <usenet.bot@animetosho.org>"

I just made the filter "animetosho[.]org".
Even "animetosho" would have worked.
----

Except is this example what the spammer is in fact posting as? It seems to me that the company, and let's be honest it has to be a company because how else would the NNTP provider continue to let this idiot destroy their product, is not using the same Poster or Subject information. Unless I missed something I am not sure how one can create a filter without first downloading the group and seeing what the name is and then Deleting All Posts From User and then adding that user information to the .nbi file.

[dexter]

Please do not tell everyone what you are downloading. I edited it out for you.

We have several people reporting the same errors and so far they are all using newsgroupdirect. Do you have SSL turned on? As a test, turn it off and see if there is a difference.

[Quade]

What address is in the server address field?

[Tarkus]

What address is in the server address field?

news-ssl.newsgroupdirect.com

I tried your suggestion of removing SSL and I changed the address to the non-SSL address. Same result so I switched it back, (Forte Agent is accessing NGD with the same SSL config and credentials).

So that's not it!

[dexter]

We have a handful of people with this issue, and they are all using newsgroupdirect, so something changed with their servers. We'll reach out to them today and see if we can get a resolution.

[Quade]

(Forte Agent is accessing NGD with the same SSL config and credentials).

This always suggests something is blocking Newsbin specifically from the internet.

https://www.reddit.com/r/usenet/comment ... tcom_down/

Suggests they're having some issues.

[Tarkus]

(Forte Agent is accessing NGD with the same SSL config and credentials).

This always suggests something is blocking Newsbin specifically from the internet.

https://www.reddit.com/r/usenet/comment ... tcom_down/

Suggests they're having some issues.

Well I have taken the risk to have my cheap NNTP account canceled by sending in a complaint to NGB. No doubt they will follow protocol and charge my credit card for a couple of months and cancel my account but let us see what they say.

I will play with the German and Netherlands servers as well.

Like I said before it is risky to diagnose similar factors although you are getting lots of complaints, (not saying it is NB's fault but a combination of factors most specifically the spammers and potential affects), and I cannot imagine that everyone one of them is on NGD?

[Quade]

Like I said before it is risky to diagnose similar factors although you are getting lots of complaints, (not saying it is NB's fault but a combination of factors most specifically the spammers and potential affects), and I cannot imagine that everyone one of them is on NGD?

For this particular DNS related issue, we've mostly seen problems with NGD over the past couple days.

If the software is trundling along and working just fine then suddenly stops being able to connect to the news server, typically it's some external issue. Considering the number of people using 6.72 and 6.73RC who aren't having issues, this also suggests it's not an inherent problem in the software. In your case, if I couldn't connect to the news server, I'd probably then try a couple internet searches and see if they work. If they do, that tells you the software isn't blocked off the internet. If they don't and specifically because the searches aren't talking to the news server but still fail, it typically means Newsbin has been blocked.

Those guys on Reddit typically aren't even using Newsbin so, if they're having problems also suggests it's not a software issue.

You have to also figure that if we get a bunch of complaints (which we haven't, it's just a smattering of users) if could also be that some security package has updated itself and has now broken something. So, having a number of people complain still doesn't means it's a software problem. In cases like that we have to ask people what security package they use.

[Tarkus]

Like I said before it is risky to diagnose similar factors although you are getting lots of complaints, (not saying it is NB's fault but a combination of factors most specifically the spammers and potential affects), and I cannot imagine that everyone one of them is on NGD?

For this particular DNS related issue, we've mostly seen problems with NGD over the past couple days.

If the software is trundling along and working just fine then suddenly stops being able to connect to the news server, typically it's some external issue. Considering the number of people using 6.72 and 6.73RC who aren't having issues, this also suggests it's not an inherent problem in the software. In your case, if I couldn't connect to the news server, I'd probably then try a couple internet searches and see if they work. If they do, that tells you the software isn't blocked off the internet. If they don't and specifically because the searches aren't talking to the news server but still fail, it typically means Newsbin has been blocked.

Those guys on Reddit typically aren't even using Newsbin so, if they're having problems also suggests it's not a software issue.

You have to also figure that if we get a bunch of complaints (which we haven't, it's just a smattering of users) if could also be that some security package has updated itself and has now broken something. So, having a number of people complain still doesn't means it's a software problem. In cases like that we have to ask people what security package they use.

I received a response from NGD indicating "that they have been having server issues over the last 24 hours and they are largely corrected". I am uncertain as to whether or not their definition of largely is the same as mine but we will see.

That said I will wait and see if this flushes out but as I indicated before one has to be careful about mixing diagnoses when it comes to technical issues so is NGD being a little misleading when they say 24 hours or has this been an ever increasing issue over the last five days which would explain the problem.

Clearly this is a big issue since in the response to me NGD admitted that they are getting an enormous amount of tickets which I must confess surprised me given that their Support system is very poorly rated. So for the moment let us put this on hold and see if it resolves itself.

[Tarkus]

Like I said before it is risky to diagnose similar factors although you are getting lots of complaints, (not saying it is NB's fault but a combination of factors most specifically the spammers and potential affects), and I cannot imagine that everyone one of them is on NGD?

For this particular DNS related issue, we've mostly seen problems with NGD over the past couple days.

If the software is trundling along and working just fine then suddenly stops being able to connect to the news server, typically it's some external issue. Considering the number of people using 6.72 and 6.73RC who aren't having issues, this also suggests it's not an inherent problem in the software. In your case, if I couldn't connect to the news server, I'd probably then try a couple internet searches and see if they work. If they do, that tells you the software isn't blocked off the internet. If they don't and specifically because the searches aren't talking to the news server but still fail, it typically means Newsbin has been blocked.

Those guys on Reddit typically aren't even using Newsbin so, if they're having problems also suggests it's not a software issue.

You have to also figure that if we get a bunch of complaints (which we haven't, it's just a smattering of users) if could also be that some security package has updated itself and has now broken something. So, having a number of people complain still doesn't means it's a software problem. In cases like that we have to ask people what security package they use.

I have confirmed that the connectivity issues are NGD related and specifically to the US server. As a result testing is impossible, (or very slow), since the European Server only obtains speeds in North America around 3-5 Mb/s so downloading a few million headers is a bit ridiculous.

That said I have question about your filter. I see that it is entered in the NBI file under UNSORTED however will the filter apply all of the subs under UNSORTED even if you download a single group or only if you download latest headers from UNSORTED? In simpler terms if I want to download from alt.binaries.stuff and not the other groups under UNSORTED do I have add the filter to alt.binaries.stuff as well or will the Parent group UNSORTED automatically apply its filter to everything. The reason I asked is I attempted to download with the filter but all of the spam still came in but I realized later that there was a filter for UNSORTED but nothing was indicated for any of the individual groups.

If you say the UNSORTED fitler should automatically apply even if I download headers from a single group within the UNSORTED group then I must have made an error imputing the filter.

[dexter]

You should upgrade to version 6.73RC2, it allows you to apply the filter either to a specific newsgroup or to a group folder of newsgroups. You don't have to mess with the .nbi file anymore either.

Download the beta from http://www.newsbin.com/beta.php and follow the instructions from http://help.newsbin.com/index.php/V673-FilterDownloads

[Tarkus]

You should upgrade to version 6.73RC2, it allows you to apply the filter either to a specific newsgroup or to a group folder of newsgroups. You don't have to mess with the .nbi file anymore either.

Download the beta from http://www.newsbin.com/beta.php and follow the instructions from http://help.newsbin.com/index.php/V673-FilterDownloads

I have downloaded the new version, (btw thanks for you speedy effort on reacting to this issue), and installed the filter. The result is that the old Newsbin is back. Groups article lists are loaded essentially immediately so clearly your change and process has eliminated the unwanted headers.

Many thanks.