Groups Not Displaying New Headers

[GolfNut]

Strange situation. Not sure if it technical or other issue.

Have at least two groups that are not showing newer headers even after updating. No filters are turned on either group. The dates of display are slightly different though both are roughly showing posts 85 days or older ... no headers newer then that appear. I have tried showing all posts and also tried turning filters on to show headers newer then 2 months and still nothing.

The reason it seems to be technical is if do a search I can find posts for the two groups with dates as recent as a few hours ago.

Any thoughts?

The two groups are:
a.b.dvd.erotica
a.b.dvd.erotica.classics

[DThor]

Ensure when you say no filters are applied, you tried literally disabling filters completely, not just selecting none. At least one of those groups, I see posts in there fine. You can always download all headers to start over in case you've got corrupt files, but I'm suspecting not since it's two, similarly named groups.

Oh, right click on the groups or he gog they're in and check for any default settings that could be affecting things.
DT

[Quade]

As far as I can tell, dvd.erotica.classics hasn't had much posted to it in the past couple days. Last DVD sized thing was 2 days ago. The other group had a new posting 9 hours ago.

Might be worth purging the groups and re-downloading. Maybe the DB behind each of the groups is damaged. Right click/Post Storage/Delete Stored Posts

[GolfNut]

I will try the purge and reload. Thanks for the prompt reply (as always). Your level of service is appreciated.

[jloveguy]

I am having a similar problem with one group so far. Nothing shows up in the download list at all. I tried deleting the posts but that didn't help. The group is large, but five hours later, nothing is displayed in the window. I have turned off all filters. I'm not having the problem with any other group that I have tried so far. The group that I'm having trouble with is alt.binaries.erotica.

[mho]

five hours later, nothing is displayed in the window.

Depending on your circumstances, I'm not sure 5 hours is necessarily nearly enough for a Download All.
Firstly, I'd suggest _not_ double clicking on the group, but rather right-click and select Download All Headers. Updating a group while displaying it consumes a lot more CPU and memory, and may really make a difference for initial load.
You might also want to do this in a freshly started newsbin, so nothing else is eating memory.

You might also want to check out the Background header mode option.

You should be able to get some sense of what is happening from the Connections and Logging tabs, depending on your current log level and Show NNTP commands settings.

If the update job dies prematurely, restart it with Download Latest. (Note: _NOT_ Download All Headers or double-clicking.)

Good luck!

- mho

[mho]

Also, make sure you are not doing something silly with download age/display age/storage age, etc. It is quite possible to download headers that will just be thrown away instead of stored.

- mho

[Quade]

Yeah. The "Storage Age" setting will throw posts away if what's downloaded is older then the "Storage Age".

[GolfNut]

Deleting the groups and then pulling the headers again fixed the problem. I should have thought of that. Still the level of support is second to none and is appreciated.

[RayMark]

I also have this problem, the problem noticed in 6.10 RC3 (not tested with RC2).
There is another issue with search not working in a group tab - this was also introduced RC2, or RC3, because in RC1 search works fine.

As to headers not showing up, I am afraid, they are not downloaded at all, even though the number is shown and the progress indicator is working.
But the Storage.db3 and StorageData.db3 files remain unchanged.
If the headers are downloaded with RC1, then everything is ok - new headers are downloaded.
But if a later version was already used for "Download latest" - then RC1 is not able to rescue from the situation either.
Perhaps the only way to get the headers then - to redownload them all with RC1, for example, with the same "Download latest" but with removed range files.
But that could take many hours for large groups.
Perhaps "Download special ->" can be used to get only the newer missing headers? (Quade ????)
"Download all headers" option should be avoided because (am I correct?) then the already downloaded files are no longer marked as such.


NOTE: if you are using the background downloading, do not forget to wait until all the header downloads are actually completed.
If not, you may have a false missing headers situation (you may check if you still have .RV4 files)

In conclusion - using RC1 there should be no problems with headers or with searching in group tabs.


UPDATE 1:

No, "download 10,000,000 older posts" -is not working - nothing is downloaded.
And I see that in many groups I am missing last two weeks - from the time I switched from RC1 to RC3....
So now I have 1000 days of headers ending at day 15th, and then some headers from day 0.
No headers at all from days 14-1.
And no way to download them without downloading the headers of the rest of 1000 days?
Quade, is the a way?

Is it possible to set somehow manually a date in range.db3 file, for example?
An option in Newsbin to redownload headers from a certain date range would be a life saver.

Because of those 2 weeks missing in many groups I am now facing a huge huge task to redownload all the headers in all the groups (or at least to check carefully the last two weeks in each and every group).
This will take months.
And then how to be brave enough to use a newer version of Newsbin when the same may happen again?
To back up manually old headers before downloading new ones? But there are hundreds of GB of them.

I guess, it is enough to back-up the range files?


UPDATE 2:

OK, this seems to be working. Do everything with 6.10 RC1 !!!:

1. temporarily remove Range.db3 and Range.db3-journal files from the folder with the headers of the group where last two weeks of headers are missing.
2. use "Download special"-> "Download nnn Older Posts"
3. repeat step 2 if necessary as many times as necessary with values that seem to be necessary.
4. when you see that there is no longer a gap, move back Range.db3 and Range.db3-journal files (overwriting the newly created ones)

Quade, can you confirm that this is a correct approach?
I checked with one group and the gap between 0-14 days disappeared. Now all the headers seem to be present.

Again, I repeat: missing headers are from the last two weeks: day 14th is partially present in many groups.
That is, missing headers after I switched from RC1 to RC3 and later (I downloaded RC2 but did not install it)
Therefore use RC1, otherwise all the work might be in vain.

[mho]

UPDATE 2:

OK, this seems to be working. Do everything with 6.10 RC1 !!!:

1. temporarily remove Range.db3 and Range.db3-journal files from the folder with the headers of the group where last two weeks of headers are missing.
2. use "Download special"-> "Download nnn Older Posts"
3. repeat step 2 if necessary as many times as necessary with values that seem to be necessary.
4. when you see that there is no longer a gap, move back Range.db3 and Range.db3-journal files (overwriting the newly created ones)

Quade, can you confirm that this is a correct approach?

Not that I'm Quade, but I would have done:
* Remove the Range.db3 files.
* Set Download Age to 15 in options. (If you want to change it back, later, just note what your current value is)
* Download Latest.

(...and I would do it using rc6:-))

- mho

[jloveguy]

Just a few notes to clarify...

I always right click and select either "download all headers" or "download latest". I always get a count and it takes quite a while to download the headers. The five hours that I mentioned was for time waiting for the headers to appear once I selected "show posts". They never showed up for the one group. All others worked fine. I'm trying once again. I am using build 1019.

[mho]

Just a few notes to clarify...

I always right click and select either "download all headers" or "download latest". I always get a count and it takes quite a while to download the headers. The five hours that I mentioned was for time waiting for the headers to appear once I selected "show posts". They never showed up for the one group. All others worked fine. I'm trying once again. I am using build 1019.

Pretty old version, but I don't remember any specific problems like these.

* Did you check "Storage Age" settings?
* I'd do a "Download Latest" to check if the initial download wasn't cut short, in some way.
* What is your "Display Age"? If you only have partial headers on disk, there might just be nothing matching.
* Try loading just one month of headers using "Show Posts Special". Try a few different ones.
* Double check that you are always using the same group. Getting headers for, e.g., alt.binaries.test and then loading alt.binaires.test isn't likely to work well, so watch out for look-alikes!

* Maybe it is just _slow_? If you have a lower-end system, it might well be that you are trying to bite off more than you can handle (in one bite).
Are you using a netbook, a low memory system, USB disk, or something?

- mho

[RayMark]

not that I'm Quade, but I would have done:
* Remove the Range.db3 files.
* Set Download Age to 15 in options. (If you want to change it back, later, just note what your current value is)
* Download Latest.

(...and I would do it using rc6:-))

- mho

Thanks, mho. Of course. This is the perfect way.
But rc3-rc6 caused this problem in the first place.
rc1 at least actually downloads headers.
I am afraid to use anything above until Quade identifies and fixes the problem.

jloveguy,
If you are using "download all headers" then you are re-downloading all the headers each time unnecessarily. Furthermore, I think that the already downloaded posts are no longer marked as already downloaded - very inconvenient.
As to waiting times, perhaps you are running the header download in background and open the group tab too early, before all the headers are processed and the cache shows 0.


BTW, why files Range.db3-journal are permanently present in every group folder?
It is not a recent development and not a simultaneous one, some of them are from July.
Maybe they can be used to roll back to a state where an ordinary "Download latest" would work?
But I think there are newer than 14-15 days old ones. So probably not.

[jloveguy]

More info:

Windows 7 64 bit
NewsbinPro 64 bit build 1019
AMD Phenom 3.20 Ghz processor
16 Gb memory

Storage Age 900
Display Age 2000

I noticed that in the data folders for other groups contain 4 files:

Range.db3'
Range.db3-journal
Storage.db3
StorageData.db3

The data folder for the problem group contains the following files (it is currently downloading headers):

Range.db3
Range.db3-journal
Storage.db3
Storage.db3-shm
Storage.db3-wal
StorageData.db3
StorageData.db3-shm
StorageData.db3-wal

[Quade]

Not that I'm Quade, but I would have done:
* Remove the Range.db3 files.
* Set Download Age to 15 in options. (If you want to change it back, later, just note what your current value is)
* Download Latest.

If you have corrupt range.db3 files, this is how I'd handle it too. It wont fix cases where the storage DB's are damaged. If they get damaged you'll need to purge and re-download.

Storage Age 900
Display Age 2000

This is the opposite of what I'd do, I keep the Display age short for fast browsing and can always "Load all" from the dropdown. I see no reason to load the entire group up every time I browse. Of course your usage may vary. My display age is typically 30-60 days or shorter.

[mho]

The data folder for the problem group contains the following files (it is currently downloading headers):

Range.db3
Range.db3-journal
Storage.db3
Storage.db3-shm
Storage.db3-wal
StorageData.db3
StorageData.db3-shm
StorageData.db3-wal

Perfectly normal. Recent sqlite3 uses Write-Ahead Logging by default.
If you for some reason have a problem using shared memory in sqlite3, you should be able to tweak a database file back to the old way using[code][PRAGMA journal_mode=DELETE/code]Make sure the db is not open and do a copy/backup before you try such things!

- mho

See: http://www.sqlite.org/draft/wal.html

[RayMark]

Not that I'm Quade, but I would have done:
* Remove the Range.db3 files.
* Set Download Age to 15 in options. (If you want to change it back, later, just note what your current value is)
* Download Latest.

If you have corrupt range.db3 files, this is how I'd handle it too. It wont fix cases where the storage DB's are damaged. If they get damaged you'll need to purge and re-download.

It is not that range.db3 is corrupt. Simply 14 days of headers missing. In many groups I have last headers 15-14 days old and then 0 days old.
The missing headers were downloaded as usually, but, unfortunately with versions > RC1 so they simply disappeared without a trace.
Today's headers downloaded with RC1, so again successfully.
But I need to fill that gap of two weeks - in many groups, some very large ones, with range.db3 present it is not possible.
This seems to be the only gap - all versions of 6.x up to 6.10 RC1 were downloading headers correctly.
In a way, I was lucky - a gap in headers somewhere several months ago would cause much more trouble.

It would be great if Newsbin had something for such cases in "Download special" -> Download nnn headers mmm days old

[mho]

It is not that range.db3 is corrupt. Simply 14 days of headers missing. In many groups I have last headers 15-14 days old and then 0 days old.

But I need to fill that gap of two weeks - in many groups, some very large ones, with range.db3 present it is not possible.
This seems to be the only gap - all versions of 6.x up to 6.10 RC1 were downloading headers correctly.
In a way, I was lucky - a gap in headers somewhere several months ago would cause much more trouble.

* Remove the Range.db3 files.
* Set Download Age to 15 in options. (If you want to change it back, later, just note what your current value is)
* Download Latest.

[Quade]

Yeah...

Maybe I need a "Reset to use "Download Age" setting. I think it's pretty clear there's a problem with your range.db3's since they control how updates work and your updates aren't working.

[RayMark]

Yes please.
If I could set the range to a certain date - that would help.
But a more flexible approach would be to allow to download headers from a specified date range - this covers also the range from 0 as a partial case.

I think that might be useful if a server was missing some headers but later got them filled with the original date.
Not sure if something like that can happen, and if it does happen, would Newsbin leave the gap or somehow find it and fill it.
That's why I am proposing any range. If such temporary gaps on the server side are not possible, then reseting the range back by n days would be enough.

[Quade]

If I could set the range to a certain date - that would help.

Did you read what Mho? He's told you twice now how to reset the counts and how to control how many days of headers it pulls.

[RayMark]

Question:

Do I need to put the earlier removed Range.db3, Range.db3-journal files back after downloading missing headers?

When range.db3 was the same for all the groups and only one or some groups required to be updates this way, then - yes.
But now when each group has its own range files - probably no reason to replace the new ones by the old ones - it should be OK both ways?

[Quade]

No, they're probably corrupt and need to be deleted.

[RayMark]

If I could set the range to a certain date - that would help.

Did you read what Mho? He's told you twice now how to reset the counts and how to control how many days of headers it pulls.

And I thanked him after the first time.
My question was entirely different - you did not even read it.
Here, again:

Let's say for some reason a gap in headers develops (temporary gap from the server or like in my case - missing 12 days, but not noticed for a few months, for example).
So now, let's say, I notice a gap - I see that the headers from day 730 to day 700 are missing.
The solution which is currently available - to download the last 730 days of headers.

Could you implement a special header download where only the specified range of headers (between 730 and 700 days in this example) could be downloaded?
That would be very reassuring for everybody - if you suspect that headers for a certain period of time might be incomplete - you can re-download headers for that period.


Furthermore (perhaps more complicated): I am currently using one primary server for header download. If it has gaps for a certain period of time, perhaps such a range download could be used for the 2nd server, so that not all the headers are downloaded from the 2nd server, but only from that period of time when they are missing on the primary server?

[itimpi]

I suspect that it would not be very easy to implement a download for headers within a specific date range. The NNTP protocol used by Newsbin to communciate with news servers has no support for such a capability - it simply allows one to specify start and end record numbers.

It might be possible programtically to try getting a few headers at specific points in the available range to try and work out where a particular date starts and/or ends and after doing that ask for the range covering the days in question, but it would certainly be a non-trivial thing to implement.

[RayMark]

So, we have Message-IDs, but also record numbers, perhaps on a particular server in a particular group?
Those same record numbers could be used to specify the range directly.
If there were an option to display record numbers in the group tab, for example.
Or at least the record number for a message shown in right-click-> properties - then we could work out ourselves what record number range we need to specify.

This would have to be for a specific server, though.
Perhaps the simplest way to implement it would be this: if more than one server is used for header downloads, this range download is disabled, and properties are not showing the record number.
If the user disables all the servers except one, only one server from currently active servers is enabled for header downloads, then the record number is shown in properties and the range download is enabled.

[Quade]

Could you implement a special header download where only the specified range of headers (between 730 and 700 days in this example) could be downloaded?

I probably could. The question is whether it's worth the effort. That's what I'll have to think about.

[RayMark]

Newsbin gets a feature its competition probably does not have.
We get peace of mind.
We will be able to re-download a range of headers when in doubt (or when there is an obvious gap).
For large groups to get those headers could take days otherwise - Giganews is now at day 1148 and its retention growing - so the range feature may become increasingly welcome even for smaller groups.
Newsbin - Huge Retention Ready.

[Quade]

It's a feature nobody will use but, I'll have to maintain. That's why I have to think about it.

[jloveguy]

I'm still having the problem where I can't get anything to happen when I select "Show Posts" for one group only. This just started happening with version 6. Can I go back to version 5 and not lose anything?

[Quade]

Have you tried wiping the group and re-downloading?

What happens if you right click and "Show all Posts"?

You can always revert to 5 but, if you want to work on figuring this out. I'm game. It can only be a couple things.

[jloveguy]

I tried "Delete Stored Posts" and "Show All Posts". I've "Downloaded Latest" and "Download All". They both took a long time, but eventually finished. Then when I right-clicked on the group and tried to show the posts, the window never loads. Again, this only happens with one group.

Thanks

[Quade]

- PM me the group. I'll try it here.

- I'd look in the spool_v6 folder for this group and see how large the DB3 files are. A large group, they should be in the GB range. The "WAL" files indicate the DB is still open and something is being done to them.

- What's your "Storage Age" set to? As an experiment, I'd set it to 1500 and then re-download again.

- Do you see and numbers in the cache entry when you download headers (NNN).

[jloveguy]

The group is "alt.binaries.erotica"

There are 4 files in the Spool_V6 folder for this group. They are

Range.db3 3KB
Range.db3-journal 0KB
Storage.db3 88KB
StorageData.db3 32KB

Storage Age is set to 900, I will try 1500

Not sure what the cache entry is, but when I download, there are two sets of numbers, headers downloaded and number of headers to download and the first number does increase as does the size of the status bar.

[Quade]

Storage.db3 88KB
StorageData.db3 32KB

This is the key right here. They're empty.

Mine are 1.8 gigs and 10 gigs for the same group.

- Make sure your PC's date is correct.
- I'd delete the group in Newsbin.
- Set the "Storage Age" to 1500.
- Exit Newsbin.
- Delete this folder.
- If you know where your NBI is. I'd edit it with wordpad and remove the section [alt.binaries.erotica], assuming it's still there. It might not be.
- Start Newsbin, turn on "Show Server Commands" on the network options.
- Re-add the group.
- Right click the group "Download all Headers".
- Go back to this folder and watch the folder as the progress bar grows in Newsbin. You should see these files grow.

Right after you "download all Headers". Look in the logging tab. How about cutting and pasting the "group alt.binaries.erotica" line and the lines around it in the log and past them here.

[jloveguy]

Thanks, I did what you suggested and it is still downloading headers hours later. The difference is that the size of the db3 files is steadily growing.

[Quade]

Might want to back them up before you load the group the first time. That's when the "Storage Age" purge kicks in. If you see the files shrink back down after load, that means you're storage age or your PC's date is wrong.