Can download headers but often cannot get message bodies

[Jerry2KC]

Roadrunner provides users with a limited connection to newshosting.com as a replacement for their own news server which has been gone for a couple of years. Connections do not require authentication, and all has worked well until recently. Newshosting made some sort of change that required users to download all of the newsgroups again, replacing whatever you had already subscribed to. Failure to do this brought the error:
"[12:02:55] ERROR NNTPSocket - NNTP Perform Auth - isp5.newshosting.com -Server Password: Server asked for Username and Password but, They aren't set in the options"

Although I followed instructions and downloaded all the newsgroups and just to be sure deleted those I was subscribed to. This seemed to have partially done the job. Within a newly subscribed newsgroup I can download headers (usually) without a problem, and sometimes I can download attachments, but more often I get the Server Password error (above). Sometimes I get an error about a group not existing when I try to download headers, so it make me wonder if all the old groups were actually purged when I downloaded the new ones, but I don't see any way I can check this. I have no clue why some groups work sometimes and not others. I am running NewsBin 6.42 build 2148.

Oddly, I noted that version 4.3.2.0 has the same problem, but not nearly as bad.

Both versions also seem to have trouble letting go of a connection after downloading an attachment. If I walk away on either while it's running, when I return I often find it stalled out at some point waiting for the next download to begin. Manually closing the connection on completed tasks usually will get it started again if the password problem doesn't stand in the way.

Any and all help will be appreciated!

Thanks,
Jerry2KC

[Ace]

Although I followed instructions and downloaded all the newsgroups and just to be sure deleted those I was subscribed to.

You just deleted the groups from the group subscription list? Did you also delete the spool files? Or did you set up a new server name after the change to keep the spool files separate (though I'm not sure why you'd want the old spool files if they giving you problems now)? I'm not quite clear on what exactly you deleted.

Both versions also seem to have trouble letting go of a connection after downloading an attachment. If I walk away on either while it's running, when I return I often find it stalled out at some point waiting for the next download to begin. Manually closing the connection on completed tasks usually will get it started again if the password problem doesn't stand in the way.

I see this "sticky connection" thing from time to time. But never once did I think it was a problem with Newsbin 4 or Newsbin 6. I assumed the cause was elsewhere, probably with the news server, but I'm not really sure. I typically run 8-10 connections and one or two may get stuck from time to time but if all the rest are working it's usually not a big deal.

In version 4 sometimes what I'd do is kill the sticky connection by clicking the kill icon in the thumbnail window, but I don't know how to do this in version 6 or if it's even possible, since the thumbnail window doesn't have the kill icons. But even with version 4, that only fixed it sometimes.

[DThor]

I'm not sure the thing about groups is related to your issue. First guess is that you're allowed a limited number of connections and you're exceeding it. Keep in mind if you are allowed, say, 5 connections, don't set newsbin to 5 - there are always scenarios where something ends and the server end takes a while to disconnect. Perhaps what's happening is you're going over your limit and once that happens the server switches your account to a regular account - where you would require an account and password.

To my knowledge recent versions of newsbin haven't had any real changes in how it communicates with the server, so I'm also guessing there's no relationship there.

DT

[Jerry2KC]

Thanks to all for your responses. Here is more information. I should have probably broken this into two threads because I can see that I've confused things a little.

Inability To Download Attachments
The instructions I saw said only that I needed to download all the groups again and to replace any that were already subscribed. I did not do anything with spool files but had no way to know whether anything needed to be done with them or not. The only thing I specifically deleted were the subscribed groups. I did this by selecting the list, hitting control-A, then delete. After doing that I downloaded the complete list of groups, but there was no option I could find to completely purge what was already there. I did create a new server, but there were no differences in any of the entries. I gave it a new name but whether anything new was really created I can't say.

Hung connections:
The failure to let go of connections isn't tied to the primary problem I'm having, which is inability to download message attachments. Roadrunner subscribers using the Newshosting server are permitted only four connections, each limited to 500Kbps, and I have newsbin setup to use four. When this was first imposed it wasn't so easy to see the problem because the error messages say something to the effect that you're attempting to exceed your max download rate or similar, and I initially gave newshosting no end of grief because it sounded like they were the problem. The heart of the problem seems to be that Newsbin ignores stuck connections and when it happens tries to start another. Newshosting keeps count and won't allow that. To see what's happening I now usually use the command line netstat utility that comes with Windows.

netstat -a -b

will show you all the connections and listening ports, and will show the executable file "involved in creating each connection or listening port". As someone mentioned, clicking on the disconnect button will often release the line in version 4. YOu can also press the pause key and go to Utilities/Kill all connections, wait several seconds before un-pausing the download, and that will usually work, too. Somewhere that I can't find at the moment there is an option to have Newsbin release connections whenever possible. That helps as well, but the option warns that this may create more errors. I haven't been able to find anything like this in version 6.

With version 6 you can go to the connections panel and under Servers right click one of them. One of the options is to kill the connection. In my experience so far this works quickly, but often the problem returns immediately after the next attachment downloads.

It's not so bad when only one or two lines get into this state,it just takes twice as long. But when I get four in this state there aren't any left to use.

Jerry2KC

[Ace]

Inability To Download Attachments
The instructions I saw said only that I needed to download all the groups again and to replace any that were already subscribed. I did not do anything with spool files but had no way to know whether anything needed to be done with them or not.

If by "download all the groups again" is they are referring to a need to re-download the headers, that translates to "delete all the spool files and download all the groups again", because if you don't delete the spool files, I don't know how you can download all the headers again. The alternative explanation of just downloading the group list, I wouldn't think would do much except maybe add or remove some groups available in the new setup.

You could try moving all the files in your spool folder to a temp-backup folder, and re-downloading the headers from scratch for a group that's giving you trouble, and see if that helps. If it doesn't, you can move your spool files back where they were.

[Jerry2KC]

I'm not sure the thing about groups is related to your issue. First guess is that you're allowed a limited number of connections and you're exceeding it. Keep in mind if you are allowed, say, 5 connections, don't set newsbin to 5 - there are always scenarios where something ends and the server end takes a while to disconnect. Perhaps what's happening is you're going over your limit and once that happens the server switches your account to a regular account - where you would require an account and password.
DT

If I have to go from 4 to 3 connections that means downloads will take 25% longer, so I'm hoping that won't be necessary. Still, I gave it a try. I changed the max number of connections to 3 and went to the download list, highlighted all th4e files and told it to try again. It looked like it might work, but then windows reported a fault and newsbin was shut down. When I started it back up it immediately began logging the null password error, even though I haven't even told Newsbin to start downloading. In fact, I cleared the log window and as soon as I did it immediately filled back up with the same message. Max connections is still set at 3.

Thanks,
Jerry2KC

[Jerry2KC]

Inability To Download Attachments
The instructions I saw said only that I needed to download all the groups again and to replace any that were already subscribed. I did not do anything with spool files but had no way to know whether anything needed to be done with them or not.

If by "download all the groups again" is they are referring to a need to re-download the headers, that translates to "delete all the spool files and download all the groups again", because if you don't delete the spool files, I don't know how you can download all the headers again. The alternative explanation of just downloading the group list, I wouldn't think would do much except maybe add or remove some groups available in the new setup.

You could try moving all the files in your spool folder to a temp-backup folder, and re-downloading the headers from scratch for a group that's giving you trouble, and see if that helps. If it doesn't, you can move your spool files back where they were.

I had the impression that I was supposed to reload the complete list of groups, replacing the list I already had. I think the reason is that they've changed the identifier for each group, whatever that may be. It wouldn't have made sense if it was talking about headers, since you have top download headers each time before you begin a new session. At least that's how I saw it.

I just tried that and it seems to have worked. Now I'm getting a lot of other errors, though. The most frequent is either "failed to retreive article from server" or "Error resolver unable to find download data." It went along like this for several seconds and I think it may have downloaded a few attachments. But then it quit, and now it's just sitting there with 570 files in the download list. The server still says "active" but nothing is happening.

Thanks,
Jerry2KC

[Quade]

I had the impression that I was supposed to reload the complete list of groups, replacing the list I already had. I think the reason is that they've changed the identifier for each group, whatever that may be. It wouldn't have made sense if it was talking about headers, since you have top download headers each time before you begin a new session. At least that's how I saw it.

All you had to do is select all the groups, right click and select "Post Storage/Use Download Age". You didn't need to delete your existing headers. Even with the server change, they'll still work fine. The only time you might have wanted to delete them is if you wanted to re-download the last 1500 days worth of headers.

Then set a download age of say 10 days. Then download the last 10 days of headers.

"Use Download Age" resets the records counts back to zero and "Download Age" tell Newsbin how many days worth to download when you update.

Dex posted about this in the message of the day (MOTD).

I just tried that and it seems to have worked. Now I'm getting a lot of other errors, though. The most frequent is either "failed to retreive article from server" or "Error resolver unable to find download data." It went along like this for several seconds and I think it may have downloaded a few attachments. But then it quit, and now it's just sitting there with 570 files in the download list. The server still says "active" but nothing is happening.

The resolver errors are because you deleted the headers out from under the existing downloads so, when Newsbin goes to look for the information it needs to download, it's not there any more.

[12:02:55] ERROR NNTPSocket - NNTP Perform Auth - isp5.newshosting.com -Server Password: Server asked for Username and Password but, They aren't set in the options"

This error says the username and password fields for at least one server are blank.

[Jerry2KC]

I had the impression that I was supposed to reload the complete list of groups, replacing the list I already had. I think the reason is that they've changed the identifier for each group, whatever that may be. It wouldn't have made sense if it was talking about headers, since you have top download headers each time before you begin a new session. At least that's how I saw it.

All you had to do is select all the groups, right click and select "Post Storage/Use Download Age". You didn't need to delete your existing headers. Even with the server change, they'll still work fine. The only time you might have wanted to delete them is if you wanted to re-download the last 1500 days worth of headers.

Then set a download age of say 10 days. Then download the last 10 days of headers.

Actually I forgot to mention it but I ran into this advice somewhere on the Newsbin website a couple of days ago. I did as instructed but it didn't help that I could tell.

I just tried that and it seems to have worked. Now I'm getting a lot of other errors, though. The most frequent is either "failed to retreive article from server" or "Error resolver unable to find download data." It went along like this for several seconds and I think it may have downloaded a few attachments. But then it quit, and now it's just sitting there with 570 files in the download list. The server still says "active" but nothing is happening.

The resolver errors are because you deleted the headers out from under the existing downloads so, when Newsbin goes to look for the information it needs to download, it's there any more.

That makes sense, but I had no way to know that. Prior to making these deletions Newsbin would sometimes work okay for awhile and sometimes go into error mode. I thought it might be loading old information from somewhere but had no way to determine where.

I'm tempted to suggest that Newsbin might have warned me about deleting the headers. It seems like everything works differently than it used to. Nothing is where it used to be and it's often not obvious how to get what you need displayed. A good example is the fact that you have to tell the group to display posts after downloading headers. Seems a bit too retro to me, especially after using earlier versions that did this automatically. The one that I really don't understand is when downloads begin immediately when the software boots. With older versions you could set the software to start downloading headers from subscribed groups, and then to download all the attachments upon startup. The new version starts in downloading something right away, and I think it may be downloads that didn't complete previously although this isn't immediately clear. It doesn't ask if I want to do it, and I haven't found the option (I'm sure it exists) to stop it from doing that.

[12:02:55] ERROR NNTPSocket - NNTP Perform Auth - isp5.newshosting.com -Server Password: Server asked for Username and Password but, They aren't set in the options"

This error says the username and password fields for at least one server are blank.

Yes, that's what I thought. The trouble is that no username or password are required by the server, so why it asks is a mystery.

Be that as it may, I have at last solved my own problem using the brute force approach. I uninstalled Newsbin and let it delete everything related. That cost me a few years worth of history data, but everything works as it should, now, and I'm happy to have the problem solved. Sincere thanks to all those who helped or tried. I learned something from every one of you.

Jerry2kC

[Quade]

[12:02:55] ERROR NNTPSocket - NNTP Perform Auth - isp5.newshosting.com -Server Password: Server asked for Username and Password but, They aren't set in the options"

Servers that don't need passwords (which is rare these days) use the 480 error, which means send a password, as a way to tell you you're using too many connections. So, you hit the connection limit, server sends 480 error, Newsbin thinks the server is asking for a password which is what the 480 actually means and it reports this error. Usenet doesn't have an official "too many connections error" so, 480 does double-duty. On an ISP based free server like this, the connection limit might not just be you. It might include the other people on your ISP using the server at the same time. It depends on what kind of deal the ISP has with the news server .

Actually I forgot to mention it but I ran into this advice somewhere on the Newsbin website a couple of days ago. I did as instructed but it didn't help that I could tell.

It works. In the cases where it doesn't work, people are typically trying it while they have 10 other things going on at the same time. For example, doing this while header downloads are still processing will prevent it from working. It deletes the range.db3 which resets the group.

[Jerry2KC]

It looks like I spoke too soon about having things fixed. I also neglected to say that the way I fixed the problem was to do a fresh installation of version 6 after wiping out all remnants of the previous one. It worked great right after I did it, but today the problems resurfaced.

Quade, I follow you on the 480 error. I discussed this with Newshosting some time ago, and I want to say that the number of connections limit is on a per user basis, but I'm going to have to check that out again because I'm not certain.

After reinstallation last night I had Newsbin going great guns when I suddenly got a windows error stating that Newsbin had to close. I looked at the data and didn't see anything I recognized. There was a long list of DLL's and very little of anything else. Event viewer had an error logged against Newsbin that I think represents the same event. It said, "Faulting application newsbinpro.exe, version 6.4.1.0, faulting module newsbinpro.exe, version 6.4.1.0, fault address 0x00331b48".
(I'm running Windows XP Pro, SP3, 32-bit)

That hasn't happened today, but I'm getting a lot of errors stating that the message had an empty body. A new error that began popping up states "Unable to convert server name to Internet address". The most common error today has been "unable to retrieve article from server. 430 no such article."
The first error sounds like something may be malfunctioning with newshosting, I guess. I'll have to see what they have to say about it. I assume the second means that Newsbin was unable to get a reply from the DNS, but I'm going to have to watch that because I'm not aware that there was an outage. The Third confuses me because I'm used to seeing an error that the article has been purged, which is what this error seems to imply. There must be a distinction between the two but I don't know what it is.

I've noticed that when newsbin is unable to retrieve a file on the first attempt it tries again, regardless of the reason, and that slows it way down because I think it goes four times before giving up. Aren't there some errors, like a purged article, that make it a wasted effort to retry?

As before, any suggestions on this will be appreciated. Any guidance as to the origination of the errors would be helpful too. I'm about ready to call newshosting to see if they will admit to anything malfunctioning, but I'd like to have my ducks lined up before I do.

Thanks,
Jerry2KC

[Jerry2KC]

Actually I forgot to mention it but I ran into this advice somewhere on the Newsbin website a couple of days ago. I did as instructed but it didn't help that I could tell.

It works. In the cases where it doesn't work, people are typically trying it while they have 10 other things going on at the same time. For example, doing this while header downloads are still processing will prevent it from working. It deletes the range.db3 which resets the group.

[/quote]

I don't think I was doing anything like that, but I didn't really understand what I was trying to do, so may have fat-fingered it somehow. Is this something I should try in version 6 as well?

Thanks,
Jerry2KC

[Jerry2KC]

After running automatically and generating the above referenced errors for several minutes, Newsbin seems to go into a paused mode. I have a downloading list page containing several hundred items. The uppermost two items are composites--Newsbin is collecting the elements of par files or maybe rars, and none of these are complete. They are in various stages of completion. They all had a status of paused. A lot of the files below these are shown with a 2-piece horizontal red line under current progress.

When I saw the stalled files, I attempted to restart the downloads by right clicking and selecting resume downloads, and when that didn't work I selected retry download, and then I tried the vcr-style controls at the top of the page. Throughout this process a few of the composite files did download, but very quickly everything stopped again. Now the status of most of the components of the top two items say that they are downloading, but nothing has changed in the last half hour. There are a few that say they are paused. If I select one of the items and tell it to continue the status changes to downloading, but nothing happens. The connections page shows all of the connections as disconnected.

I closed and restarted Newsbin, and this appeared to get the downloads started again, but I noticed that nothing was accumulating in the files list, but nothing was going to the failed files list either. The status list looks like it might have a long list of the same file names, saying failed to retrieve article from server; 430 no such article. As I was writing this the task completed, and now everything in the download list has the red horizontal lines. I guess I need to manually remove these from the list, but I don't understand why. Maybe because I might want to try to get the files from another server?

I'm not convinced that these file attachments have really be deleted from the server. If they have the administrator left the message but deleted only the attachmentsd, I guess I'll find out if I try another download.

Jerry2KC

[itimpi]

If you get a string of 430 type errors, and the progress bars start turning solid red, then this almost certainly means that the item in question has had the message bodies removed due to DCMA takedown request. This is becoming increasingly common, particularily for the newer TV shows or films.

At this point you need to remove the items to stop Newsbin wasting time trying to retrieve what are now non-existent message bodies. If not Newsbin is going to slowly work through every chunk finding it cannot get it, and this can take a while.

[Jerry2KC]

If you get a string of 430 type errors, and the progress bars start turning solid red, then this almost certainly means that the item in question has had the message bodies removed due to DCMA takedown request. This is becoming increasingly common, particularily for the newer TV shows or films.

At this point you need to remove the items to stop Newsbin wasting time trying to retrieve what are now non-existent message bodies. If not Newsbin is going to slowly work through every chunk finding it cannot get it, and this can take a while.

For several days Newsbin behaved perfectly, doing everything perfectly. I think I went for three days, but then the trouble started again in a new way. I did delete the files that had the red bars, and I think that was what started the three day run of good luck. Each day I accumulated at least a few new red bar items, which I've deleted immediately, but yesterday for no apparent reason, newsbin just wouldn't work. The logging list was full of password requests, so I paused everything that was trying to happen and I went to each connection and killed it.

I waited a little bit, then unpaused Newsbin, but it continued to post the password message in the log. All of the connections said disconnected, so I used netstat -nob and found that Newsbin had seven connections open-waiting, for what I don't know. I went back to newsbin and used the command under options to kill all connections. This got rid of one of them, but the rest remained. I shut down newsbin and netstat reported the connections released. I restarted Newsbin, again tried downloading just three or four small files, but they wouldn't download, and checking netstat I found that four connections were open again and waiting. Again it was not possible to release the connections from within newsbin, and finally I gave up.

It's looking a bit like Newsbin isn't respecting the number of connections limit. I had this set to four during the days that all was working well, and changing it to three had no effect when it was not.

I'd like to add that it's a major step going from version 4 to version 6, with a bit of a learning curve to get there. Version 6 is a much more powerful implementation, though, and I've been coming to see that. You guys have really thought of some impressive tools, and nothing else I've seen comes anywhere close. When the bugs are gone it's going to be awesome! (Assuming that the bugs are in Newsbin, which certainly is not a given.)

Thanks,
Jerry2KC

[DThor]

As far as I can tell, nothing has really changed - my take is as follows: you're running on a very limited usenet server resource (free), and you sort of 'get what you get' from that. Newsbin is designed to work with fairly consistent usenet servers, that's not to say you can't use free servers, but it's optimized for pay servers. What appears to be happening is the server is (understandably) taking some time to reset when a connection is dropped. This happens with pay servers too, you'll typically get 'exceeded connections' messages, but in your case the free server happily shunts you over to a pay account connection once some central overlord server detects you trying to connect after (it thinks) you've used up all the free connections. The solution for other users is to drop the number of connections - at some point this gives enough time for the server to reset connections. I would argue it's in their best interest to not put a lot of work in keeping track accurately since that costs them, and otherwise encourages you to give them some money.

I've not seen a scenario where newsbin uses more connections than you've set, but don't forget a connection, by definition, requires both ends to participate. If one end doesn't get back, eventually the other needs to assume the connection is dead otherwise it sits there forever. For most servers, just staying well below the limit works. Unfortunately, with so few connections to start with, that doesn't give you a lot of room.

FWIW, I wouldn't be micromanaging connections.

DT

[Quade]

How many connections are you allowed to use? The problem with News servers is once a connection disconnects, it takes awhile for it to permit you to re-use it again. That's why it's always a good idea to use fewer than max connections. Once you get into this "server thinks you have too many connections when you don't" state, it can take as much as 10-15 minutes for the server to free up all the connections from it's end.

Connections in "time wait" or "close wait" state are already closed. Normally you don't count them as connections anymore.

[mmonroe]

6.50b8
Giganews

I am also running into the "Failed to Retrieve Article" error with Giganews. I assume it is related to the DMCA takedowns I have read about. Is there going to be a way around this or is Giganews worthless now?

[20:12:58] HIGH Failed to retrieve Article from Server:news.giganews.com ARTICLE <Part4of41.04AA3733339844EE8AA5E85CEAD23FC1@1367224720.local>
| 430 no such article
[20:12:58] HIGH Failed to retrieve Article from Server:news.giganews.com ARTICLE <Part5of41.5571E06A05BF402C88C5FED4A8D3A248@1367224720.local>
| 430 no such article

If they are going to remove the backend posts they should also remove the headers as well. I hope something can be figured out - else the usenet and readers are going to have some tough times.

[dougm1]

At this point you need to remove the items to stop Newsbin wasting time trying to retrieve what are now non-existent message bodies. If not Newsbin is going to slowly work through every chunk finding it cannot get it, and this can take a while.

Is the number hard coded into Newsbin or is there a way to manually set the retry download timeout on "430 No Such Article" so Newsbin isn't wasting time trying to retrieve what are now non-existent message bodies? I can't seem to find a selection setting for it.

[Quade]

I'm looking to adding an option to automatically fail files that have more than 10% errors.

Currently if no par files download and the retry count expires the failed sets should get sent to the failed list, at least in 6.50B8.

[dougm1]

So there's no way to manually set the retry download timeout on "430 No Such Article." Just out of curiosity, what is(are) the threshold value(s) for a timeout/fail?

[Quade]

You can set the retries in the options. The default is 2 which probably about 2 minutes

[dougm1]

Thank you.

[Quade]

If you set it much below two, you might run into problems. I'd suggest no less than 2.