6.42 - Not all headers are being downloaded

[besommer]

This has been obvious to me with a.b.teevee as this is the primary newsgroup that I've been using. The server is easynews.com. I've noticed gaps in the time period in the date column where the expected files should be. I can go to an nzb indexing site or the easynews.com web site, download an nzb file for the missing headers/files, load the nzb file and successfully download the associated files. When I delete the stored posts of the group and re-download the headers, I may see some additional headers populate the time period where the gap occurred, but not the headers for the specific file I was looking for -- that I was able to download through Newsbin using the nzb file. When this first started happening, I set server options --> header overlap: 1000.

[Quade]

First thing I'd try is in the server options, check "Disable XFeatures" then restart Newsbin and see if the symptoms are the same.

[besommer]

First thing I'd try is in the server options, check "Disable XFeatures" then restart Newsbin and see if the symptoms are the same.

XFeatures is now disabled. We'll see if that was an issue for Easynews.com. My understanding was that was an issue for other servers, but not Easynews.

A related symptom I forgot to mention is that at times, even if all records for a file appear to be complete in properties, at times they will show as incomplete when downloaded. This will also be corrected when I load the nzb file and redownload using that loaded nzb file.

[besommer]

First thing I'd try is in the server options, check "Disable XFeatures" then restart Newsbin and see if the symptoms are the same.

After disabling XFeatures, I am still seeing this behavior. It should be noted that th behavior is less frequent, but is evident, especially when complete series are posted as that makes it easy to see gaps in the posts. Once again, downloading the NZB files, loading them and using them to download the target files is successful.

[Quade]

I might try signing up with easynews and see of I can see this. I'm not seeing it with AW or Giga which makes me suspect it's the server. It sounds like a server farm that's out of sync.

[Ace]

This is probably not related but thought I'd mention it as an FYI.

I was using 6.40B2 and couldn't see all the headers. At first I thought it wasn't downloading all of them. In the group I was using there are about a million headers a year so three years worth was about three million headers. But I wasn't seeing the last year of headers (most recent).

I deleted all the header records and db files for that group and tried to re-download all the headers....same thing happened.

I think about 5 of my 6 GB ram were already in use when I started Newsbin, and Newsbin used up the last gig so ram was pretty full. I closed another application that was using 1GB and suddenly more headers started appearing in Newsbin... which told me it was likely a memory issue, but I wasn't getting any messages from Newsbin about being out of memory.

So, in a not very careful troubleshooting mode, (I just wanted to get it working), I did three things.
1. Freed up even more memory
2. Only downloaded a year and a half of headers instead of three years
3. Upgraded from 6.40 to 6.42

Now I'm not seeing any problems with displaying the most recent year of headers...but because I did three things at once I can't say for sure which one fixed it. If I get time I might see if 6.42 has the same problem as 6.40B2 of running out of memory but not telling you about it, but in my case I think this is why not all my headers displayed. besommer's problem sounds different though.

[besommer]

This is probably not related but thought I'd mention it as an FYI.

I was using 6.40B2 and couldn't see all the headers. At first I thought it wasn't downloading all of them. In the group I was using there are about a million headers a year so three years worth was about three million headers. But I wasn't seeing the last year of headers (most recent).

I deleted all the header records and db files for that group and tried to re-download all the headers....same thing happened.

I think about 5 of my 6 GB ram were already in use when I started Newsbin, and Newsbin used up the last gig so ram was pretty full. I closed another application that was using 1GB and suddenly more headers started appearing in Newsbin... which told me it was likely a memory issue, but I wasn't getting any messages from Newsbin about being out of memory.

So, in a not very careful troubleshooting mode, (I just wanted to get it working), I did three things.
1. Freed up even more memory
2. Only downloaded a year and a half of headers instead of three years
3. Upgraded from 6.40 to 6.42

Now I'm not seeing any problems with displaying the most recent year of headers...but because I did three things at once I can't say for sure which one fixed it. If I get time I might see if 6.42 has the same problem as 6.40B2 of running out of memory but not telling you about it, but in my case I think this is why not all my headers displayed. besommer's problem sounds different though.

I believe this is an unrelated issue, as I only display headers for the last 30 days and I did purge the headers from alt.binaries.teevee recently in an attempt to troubleshoot this very issue.

I might try signing up with easynews and see of I can see this. I'm not seeing it with AW or Giga which makes me suspect it's the server. It sounds like a server farm that's out of sync.

If you do, check out alt.binaries.teevee, specifically the recent <some shit we don't speak of> as that is where I saw this issue most recently presented.

[Quade]

File you mentioned is perfect on Astraweb.

[besommer]

File you mentioned is perfect on Astraweb.

As this appears to be strictly a header issue, I have no idea if we would even expect things to be the same or different from server to server. As I'm able to download an NZB file from Easynews.com, load that file in Newsbin and subsequently successfully download the affected files, the files themselves are complete on Easynews.

In the past, I was subscribing to both Easynews.com and Usenetserver.com and never saw this issue as I'm sure one server was doing a good job of filling whatever "header gaps" the other server had. With my deciding to save money and subscribe to only one private server, the absence of that "safety net" has made what are most likely shortcomings in the server readily apparent.

[Quade]

It's still in my list to see what headers Easynews is delivering. Easynews is a highwinds server. I'm not clear if they're running the latest version of the highwinds software though. Reports I've seen before suggest the counts of the headers aren't accurate from their server so, you end up with gaps in the headers.

- One thing to try would be a large overlap. Say, 20,000 and see if that makes a difference. Large overlaps can cause other artifacts but, if it solves this problem for you maybe it's worth it.

- When you notice this, another thing you could try is to set a 2 day "download age" then right click the groups "Post Storage/Use Download Age" and then download headers again. This is like setting a really large overlap. It'll re-download the last 2 days worth of headers.

I'm curious if you try this whether the headers become complete or not.

[confusednewb]

I haven't tried the 2 day trick - just saw this thread. But I'll second - I'm seeing empty spaces in my header listings. The files are there - if I use the Easynews web interface, the headers and articles are there; if I download nzbs, like the op, I'm also

I can't tell you why, but I'll confirm that this IS happening. I started noticing it in the last couple updates of Newsbin, if that means anything.

[Quade]

NZB's aren't headers. Just say'in. The fact an NZB works doesn't mean the headers are all there. Easynews web interface doesn't use headers either.

The only question is whether it's the server not delivering headers. I'm leaning in that direction because I have all headers when I use either giganews or Astra for headers.

[Traveler2530]

I've had similar problems with Newsbin downloading, displaying, and updating headers since they released 6.4x in production. I have been using Giganews for years, then went to Supernews a year ago. I just added AstraNews to see if this helps the problem. It does not!

I suggest trying Newsbin 6.3x and if possible, different newsreaders. I have an old Forte Free Agent from about 2004. Newsbin 6.32 and Free Agent both download, search, display all headers and attachments properly. See if the problem goes away.

As of the release of 6.4x, didn't Newsbin begin using a new compression/decompression routine for downloading headers? I noticed I get a decompressed 450 Mbps from a Comcast 50 Mbps connection.

Nice speed! Is the problem in the decompression then conversion to db3 files?

I've submitted a ticket to tech support. I like 6.42 and Newsbin, I've been a user for 7 years. I'd rather they work this problem out than I need to use 6.3x or another reader.

[confusednewb]

NZB's aren't headers. Just say'in. The fact an NZB works doesn't mean the headers are all there. Easynews web interface doesn't use headers either.

The only question is whether it's the server not delivering headers. I'm leaning in that direction because I have all headers when I use either giganews or Astra for headers.

Agreed/ maybe I need to change my name, I'm not quite as confused anymore.

Only mentioned that as proof that the articles do exist on Easynews - Newsbin Pro 6.42 Build 2143 simply isn't grabbing the headers, and I can't find a way to get them.

I had all weekend to play - did the 2 day trick, and then moved the download age to 8 days. As you suggested - still have a 33 day gap in headers. I've disabled Xfeatures - which made no difference.

As above, I haven't had this problem before; you could be right, that its only on Easynews - but otoh, your new code isn't playing well WITH Easynews. Any chance you could simply look into the problem and fix it?

[Quade]

The code isn't really that new. There's only so many ways to download headers. The 2 day trick will only fill in the past 2 days. If you're trying to fill in a gap from say, 60 days ago, you'll have to set it to 60 or more and repeat. You know, you could try a giganews free trial, load up on headers and see if there are any gaps. Headers from any server will work with easynews. I'm curious if this would fill in the missing bits.

Traveler2530 is subscribed to 9000 groups and has 100K+ header files to import before he's going to see all the headers so, I consider his problem completely different than yours. Basically the machine is flooded with imports and nothing will work properly till the import process completes. He's trying to do more than his machine is capable of.

[Traveler2530]

Try going back to an earlier version, I had success with 6.32. It downloads all the headers, displays them in searches, so on.

Trouble is, it's much slower on the downloads and searches than 6.42. For example, when I re-downloaded teevee with 6.42 onto a new NewsBin installation, I was surprised it took only eight hours or so. This was for around 480 million headers!

But the cache was showing a backup of over 100k, decreasing at only about 200 a minute. The latest posts I saw were weeks back. Completely unusable....

So I reinstalled 6.32 and re-downloaded teevee. It took overnight, maybe 24 hrs or more. But all of the posts in teevee appeared, including some as recent as a few minutes earlier than the download.

It appears the large amount of headers on Usenet and rapid downloads are swamping the capabilities of NewsBin for 6.4x versions.

[Quade]

It appears the large amount of headers on Usenet and rapid downloads are swamping the capabilities of NewsBin for 6.4x versions.

Naw, your usage pattern is just way out in left field. When you start hitting your 9000 groups with 6.32, the problem will happen in that version too. If you start clean with Teevee in 6.42 like you did here, it'll work the same as 6.32. You've already demonstrated that earlier.

Your problems are self induced. Massive amount of data and a slow PC. Nobody else is downloading like you are so, it's not really strange that you'd have odd problems.

[Traveler2530]

It appears the large amount of headers on Usenet and rapid downloads are swamping the capabilities of NewsBin for 6.4x versions.

Naw, your usage pattern is just way out in left field. When you start hitting your 9000 groups with 6.32, the problem will happen in that version too. If you start clean with Teevee in 6.42 like you did here, it'll work the same as 6.32. You've already demonstrated that earlier.

Your problems are self induced. Massive amount of data and a slow PC. Nobody else is downloading like you are so, it's not really strange that you'd have odd problems.

No, I'm standing firm in my position. If you remember, I set up a new blank drive and downloaded teevee only using 6.42. It was really fast downloading. Posts were up to date, within the last few minutes. Then I reported 6.42 was working great. But I was wrong, I discovered I was using 6.32 after the 6.42 download. I still had two installations of NewsBin on my PC!

I'm disappointed to say I even tried using 6.42 to download headers from just a few much smaller binary groups, and most headers were missing.

When I made sure I was using 6.32, re-downloaded the headers, I could view thousands in the groups!

I'm sticking with 6.3x versions. They work!