Posts are missing from all groups

[Streetwolf]

For about a month starting with a beta of v6.40 until now with 6.40 b2059 I am missing a massive number of posts from all groups. My group list only shows a high of 116.MB for any group although in the past I've seen Gigabytes of posts for many groups. It's not just that the display might be incorrect but actually downloading posts from any group produces very few.

Am i missing something obvious here? I am using NewsHosting, can it be them?

[Quade]

Newhosting has had problems with headers for their groups over the past couple weeks. Dex has reported a couple groups to them they've had to rebuild.

I've had a couple reports from people that they have to "right click/post storage/use download age" in order to get all posts. It wasn't consistent though. Just for grins, how about setting a 10 day "download age" then "use download age" and download headers and see if more show up?

As a data point, I use headers exclusively with giganews and I'm not missing any. I imagine I'd have the same results with astraweb.

Is it specific to some groups?

[Streetwolf]

It's affected all the groups I tried.

[Quade]

Well, I'd try my suggestion and then you could also try a free 3 day giganews trial and see if you get the same results. That should tell you whether it's Newsbin or the server. You could pick a group, purge it and then download headers and see if it has the same symptoms too.

What's the chance the records are being filtered out by the global filter?

[Streetwolf]

I have no filters. I'll give GigaNews a try.

[Streetwolf]

Tried Giganews and it shows a lot more posts than NH. For example a.b.boneless is the largest group on my list at 5.99 GB. NH showed something like 150MB. Is 5.99GB what you are seeing Quade?

[Streetwolf]

I contacted NH and here is what I got back from them. Seeing as I used to get many more headers up until a few weeks ago their answer is somewhat suspect if you ask me. Comments?


Brad Biel commented on NHS-103058 at 05/Nov/12 1:17 PM:
-------------------------------------------------------

Greetings,

I will say that one difference between us and Giganews, is that you can only download 500 Days worth of headers through us currently - I assume that the file sizes you're referring to are related to header storage. This would make sense.

We advertise over 1500 days of retention, and that is available, but you'll need an NZB file, or a newsreader that pulls content by Message ID rather than article. Our administrators are working to resolve the 500 day header limitation, but at this time, those two options are the only work around.

If you have any questions, please let us know.

Regards,
Brad
NH Support Team

-------------------------------------------------------

------------------------------------
Your original description appears below:
------------------------------------

For the past 3 weeks or so I’ve noticed that many groups contain less posts than they should. I use NewBin Pro as my News Reader. Take for example the group a.b.boneless. It shows something like 150MB worth of posts. As a test I signed up with Giganews for a free trial and it shows 5.99 GB worth of posts.

Given that Giganews appears to show the correct number of posts I can only assume the problem is on your end.

[Quade]

How many total headers you can download has nothing to do with the issue at hand. NH and all the highwinds servers had a 125 million post cap which means in some groups you might have 1400 days but, in other active groups you might have a couple months of headers.

This has nothing to do with what I think your problem is. I think it sounds like you're downloading headers and not seeing them all. I assume you mean latest headers, not old headers. If you're talking about total headers in the group then yeah, both giga and astra have far more headers than NH.

[alexander038]

Tried Giganews and it shows a lot more posts than NH. For example a.b.boneless is the largest group on my list at 5.99 GB. NH showed something like 150MB. Is 5.99GB what you are seeing Quade?

a.b. boneless is 50+ gb.

[Quade]

In post count, it's about 6 billion, in on disk size, it's 50 gigs like Alexander says.

[Streetwolf]

I emailed NH a screenshot of the largest groups as they appear under NH and GN. NH said the numbers are about right for NH because they only keep headers back to Oct. 19th or so. The thing is, I was seeing much larger header numbers about 3-4 weeks ago.

Do you think it's time to change to Giganews? Unfortunately, I just renewed a six month subscription to NH and it doesn't seem like they offer any refunds.

[Quade]

Do you need 6 billion headers? It'll take you two weeks to download them on a decent connection. Most people don't bother, they use NZB's or search. Newhosting has had this cap for a long time. If you used some other server before NH you might have been seeing their headers mixed in with the NH headers. If you then purged the group, that would explain why you suddenly are missing posts, the old ones got purged away too.

[Busty]

Shame you can't download the 50gig of headers as a binary as it would only take just over an hour.

[DThor]

Most (many?) providers allow compressed header download, which newsbin supports, so really you're getting pretty good compression in that regard. Of course it still needs to be parsed and injected into newsbin's database. The biggest thing slowing you down apart from that is that it's linear, you can't download in simultaneous chunks like file posts. One connection per group is all you can get.

Users trying to be a mini usenet server and hoard entire corners of usenet as locally searchable databases are probably always going to be the exception to the rule. It's a crapload of stuff that is constantly updating and is arguably 50%+ spam. Like most things on the Internet, it's about data management and finding ways to cut through the crap and find what you want, which is why a built in usenet search and nzb's are always going to be more popular. Not criticizing, just sayin'...

I use spools for numerous groups, but they're relatively low volume compared to the monsters.

DT