Headers when group is open

[dunkem]

I just upgraded to 6.31 build 1858 and it seems that I've lost headers that were downloaded when the group was open.

For example, I had say.. a.b.hdtv open all day and my headers download twice a day. I recognized there were no new headers. I closed the group and loaded headers for the past two days for the group. No headers for the whole day. Latest headers were a day old. The other group I had open seems like the same thing happened.

I loaded headers for groups that weren't open throughout the day and they all had headers for the whole day so it seems like the problem is with headers not updating only if the group is open.

Just as a side note, not sure if other people would want this, but I wish there was a way to download headers for a particular day. Sometimes headers get flukey and I want to redownload headers for a particular day. Like in this latest case, if I could just download headers from yesterday, I could just "fill in the blanks". Can't download latest headers since newsbin thinks it's up to date and i don't really want to nuke all the headers and redownload again. *EDIT* I just saw the sticky about downloading old headers without purge.. that'll help fill in the blanks. If newsbin had a way to download a particular date range without having to delete the range.db3's would still be great.

Anyways, if you guys could see if it's not just a problem on my end regarding the headers for the open group, then I can figure out if it's just my end or not.

[Quade]

Do you have "Hide Old" set in the display for this group? I'm wondering if they just get tagged "Old". I use this mode all the time. Leave the group open, update periodically. I'm not losing any records.

The process of saving the records to disk happens before display, basically the chunks of records are written to disk and a worker thread reads them and places them in a database. Then after the new records are in the database a notification is sent to the display to read the new records in. So, intuitively I'm not sure how displaying them can impact storing them. Not saying it's impossible, it's just not obvious how it would happen.

[dunkem]

Hi Quade,

I do not have "hide old" on. I also tried the range.db delete trick and downloaded new headers for the past two days for the particular group (with it closed). After it was done, I tried loading and it and I still didn't have any headers from the recent day. So the newest headers was from two days ago. I tried loading the past two days and I got nothing. Just tried downloading latest and I got no headers being downloaded so something strange is going on for me.

I do keep a lot of headers so maybe that's an issue?

[Quade]

I have 140 gigs of header so, I don't think it's the problem. What downloaded when you deleted the range.db3? What's your download age set to.

Seems to be you have a more fundamental problem in that you're not getting any new records for this specific group even when you reset the group. What server are you using?

[dunkem]

I set my download age to 2 days (since I'm missing the last two days) and it spent a while downloading about 7 million headers from abhdtv. I'm using usenetserver. Since it spent the time downloading all the headers, I know they're there, but for some reason, Newsbin isn't updating it. I'm also sure it's not a server header issue since I got headers for the other groups I didn't have open at the time.

Guess it's a problem only on my side since there doesn't seem to be anyone else experiencing this. Maybe it's because of my 32 bit XP again.

[Quade]

Do you see numbers in parens next to the cache line "(NNN)". If so, that number represents the number of records Newsbin is importing. Until it's done importing, you wont be able to see the latest records.

I'll check the group out myself. I suspect it might be importing records still.

[dunkem]

You mean on the bottom of newsbin? I have cache: 81/100. I haven't had anything download for hours so everything should be "done". I tried again deleting the range.db3 and it still doesn't bring it in. I am getting recent headers though, but nothing for the period when I had the group open for those one or two days.

Do I delete the journal file too when deleting the range.db3 file to try and get those couple days?

I've filled the gap using my second astraweb server to grab those two days of headers, but I still don't know what caused this or why I'm not able to fill the blank with my usenetserver primary server (I use astra for fills).

I'm kind of afraid to keep my groups open after this happened... apparently just to me.

[Quade]

You know, as an experiment how about making Usenetserver a non-header download server, just use Astra for headers and then try some open group header downloads. I'm not 100% convinced this isn't a server problem. I think Astra has free headers and I know their implementation of "XFeatures" works meaning you should always get higher header download speeds from them.

If everything works properly, it might be some hiccup on UNS's end. Considering I tested this with Astra (typically get headers from astra) and it worked fine. That's where it's pointing.

[dunkem]

Thanks Quade. Will try to experiment with some groups I don't use so I don't muck up the headers for the ones I use. Ironically, the reason why I got usenetserver was due to periodic quirks in astraweb's headers (along with missing chunks that were were supposedly complete).

Will report back after experiment later tonight.

[dunkem]

Looks like it might be usenetserver missing headers for abhdtv.

I deleted the folder for it and tried to download 10 days worth using usenetserver again as my header server. After downloading millions of headers, I ended up with... one day. I deleted the range and changed download age to 20. I kept the folder open to see if the headers would update with 20 day old headers. I see it try to update the open group with the headers as they're downloading but nothing gets added. I should be seeing 20 day headers get added (display age is higher like 25).

Looking at the spool_v6 folder, when newsbin tries to update the headers for the open group, I see storagedata.db3-journal create a file and it disappears. Actual storagedata.db3 does not increase in size.. which is strange to me since I'd figure it'd be constantly growing as it is downloading older headers (20 days vs the previous "10 day" age).

Does newsbin have a problem when servers are missing a bunch of headers? So if there is a "gap", does newsbin somehow lose count when saving or displaying headers?

[itimpi]

The .db3 files for headers tend to reach a 'steady state' size. The size of the .db3 file does not reduce when headers are deleted - it is just that internally to the .db3 file space is marked as free for re-use.

[Quade]

In the network options, set "Show Server Commands" so you can watch what Newsbin has to say to the server. Then re-do your experiment You should see Newsbin talking to the server in the log tab. The counts from the group and XOVER commands should tell us something.

What happens if you do the same thing with Astra?

It's possible that UNS is doing something unexpected. I can't really code around it until I know what that something is.

[dunkem]

Astra downloads and displays all headers fine as the header server. I tell it 5 days and it gives me 5 days.

Doing the Usenetserver experiment again, I specified 5 days and it gave me far less headers than what astra did. like 4 million for usenetserver vs. 15 million for astra. I deleted the spool folder again and increased download age to 8 days and it would still only pick up 4 million.

After trying to download the intial amount, logging says:

GROUP alt.binaries.hdtv
211 4310322 1 4310322 alt.binaries.hdtv
GROUP alt.binaries.hdtv
211 4310323 1 4310323 alt.binaries.hdtv
XOVER 101-111
224 data follows
XOVER 101-111
224 data follows
XOVER 300101-600101
224 data follows.
etc.

Deleting range after it was done and updating download age to 15, I got the same 4 million header count again so somehow Newsbin can't get past the missing header area and it's basically the same headers downloading again. Newsbin must be thinking that once the headers go out of sequence, its "the end", Guess it'd be hard to find the next header number to resume especially with large header groups like this.

Hope this helps. Thanks

[itimpi]

That log shows the server reporting that the highest header number it HAS is 4 million. Looks like a definite problem at the server end. I wonder if they have a problem when the highest header number exceeds a 32-bit value or something like that?

[Quade]

Damn I didn't notice that. Good call. Yeah, looks like the server is capped at 4 billion posts.