Newsbin pro does not show all headers

[apsen]

I seem to be unable to make newsbin show all headers In the GUI.

For example if I query Storage.db3 with the following query "select distinct NN_Subject from Compact_V4 order by NN_Subject;" i see many more different subjects then in the GUI.

I have tried to search for single word of the subject via local search or visually and could not find it.

No filters has been setup for the group, tried "Show all files" and "Load all posts". Neither would help. What I am missing here?

[apsen]

Looks like it was related to filters after all.

I've been trying to fiddle with the group to fix it and have noticed that the list does not show anything older then 5 days. So i have enabled filters and set it to 7 days - and newsbin showed more headers. After that I have disabled filters and I could now see all of them

[DThor]

Yeah, it's easy to miss, I know I have. The habit I've gotten into is, whenever there's the slightest doubt that somehow posts have gone missing, I just hit the disable filters button altogether. That's guaranteed to show you all you have on disk.

DT

[apsen]

But I've been checking for the filters...

But looking on how it looks in NB 6 maybe I did make a mistake as enabled filters look kind of like they are disabled - all gray and the button does not have two positions. Lets call it interface bug/flaw/etc then.

[DThor]

Err, that I don't follow at all. Disable filters clearly shows no filters applied. Enabled, depending on what is set, it does show up fine for me. The only thing I'd call a bug is sometimes the status of the filter preset won't redraw on a new load unless you pass your mouse over it.

How would you prefer it to work? Do you want more aggressive indications that a given filter isn't off? Perhaps a gentle pastel background to a selector?

DT

[Quade]

Disabling the filters also disables the global filter profile.

[apsen]

I just caught this again:

1) Fresh start of NewsBin.
2) Right click the group and choose 'Show All Posts'
3) The group opens and shows no posts
4) Verified that filters are off - and this time I'm absolutely sure that the interface indicated them being off
5) In the group window change drop down from 'Display Age' to 'Show All Posts' - no change
6) Click 'Reload from disk' - 'Load All Posts' - no change
7) Enable filters - no posts shown as expected as filter is set to filter out posts older then 5 days and the headers haven't been updated for a week
8) Disable filters - now all posts are visible

I have no filters enabled/defined in Global profile and the behavior is consistent with regular filter being applied when it should not have been.

I do not know if there's some cache issues as this was the first start of the program after about a week. Closing the program and immediately starting it again causes no issues in step 3.


Other thoughts:
1) Sometimes the group has the filters disabled but shows grayed out '5 Days' in the age drop down, sometimes it's blank. Haven't figured out pattern yet.
2) After opening group via 'Show All Posts' the drop down indicates 'Display Age' - I then change it to 'Show All Files' - and NewsBin reloads the group.

[apsen]

How would you prefer it to work? Do you want more aggressive indications that a given filter isn't off? Perhaps a gentle pastel background to a selector?

One thing that would help is the filter button having two visually distinct states (on/off).

[Quade]

I was thinking it was sufficient that the filter controls grayed out when disabled. Maybe I need to re-think it.

[apsen]

I was thinking it was sufficient that the filter controls grayed out when disabled. Maybe I need to re-think it.

It's not that big of a deal. I was just thinking that I misidentified it first time but being able to see the same problem again restored my confidence in myself

Any ideas about my other post today?

[Quade]

- The drop down for loading, is no different than "Show all Files" from the right click. It just lets you load from within the post list. If you already "Show all Posts" then messing with that doesn't change anything.

- Do you see the progress bar progress when you "show all"?

- I'd enable the filters and reset any filters that are set. See if that makes a difference.

- Exit Newsbin and move the PosterLockout.txt and the "Filters.db3" out of the data folder. Then restart. If that does nothing. Exit, delete the "Workspace.xml" from the data folder and try again.

Either there's nothing to load or the posts are getting filtered out. It's not clear what's happening.

[apsen]

- The drop down for loading, is no different than "Show all Files" from the right click. It just lets you load from within the post list. If you already "Show all Posts" then messing with that doesn't change anything.

Maybe. But the program does not give me any indication it is showing all of them as the corresponding drop down shows 'Display Age'. I cannot be sure whether it's only display bug or the program actually shows posts based on 'Display Age'. So I do it again via the drop down.

- Do you see the progress bar progress when you "show all"?

Yes.

- I'd enable the filters and reset any filters that are set. See if that makes a difference.

Do you mean there's something else to do besides what I've already done. If so could you elaborate?

Either there's nothing to load or the posts are getting filtered out. It's not clear what's happening.

The post ARE getting filtered out but the filter is disabled. As you could see from my post there are posts in that group that will reappear after re-disabling the filter.
I actually do not use filters so it is bothersome to see this behavior.
Anyway I'll remove the mentioned files to see if that will prevent the bug re-occurrence.
But unfortunately I do not know how to reproduce it reliably so it may take a while for the bug to happen again.
I'll post if/when it happens again.

PS I've tried to provide enough details for you guys to analyze what may be happening.
But it seems that not everything I say is being understood.
If I'm not clear I'd be happy to elaborate - just let me know.

[apsen]

Here's another one for you. This one is reproducible every time:

1) open group via 'Show All Posts'
2) scroll down to posts with subject starting with '1/1' (quotes not included) - there are hundreds of screens of such posts
3) type '1' in search box (without quotes)
4) scroll down to posts with subject starting with '1/1' (quotes not included) - there are none of such posts
5) click 'clear the search entry'
6) scroll down to posts with subject starting with '1/1' (quotes not included) - there are none of such posts

You cold also replace search operation with changing display settings to 'Hide Old' and back to none or enabling and then disabling filters.

Probably more - I've only tested those as it takes time to go through it.

[Quade]

Not following.

I enter "1"

and

SOmething ~bY Someone [1/1] - ".....mkv.nzb" yEnc

Shows up in the listing

I enter "1/1"

the same thing shows up in the listing.

How about PMing me an example group? I'm not seeing any files like this in my groups. Where it's the first thing in the subject.

[apsen]

How about PMing me an example group? I'm not seeing any files like this in my groups. Where it's the first thing in the subject.

PMed

[RayMark]

I was telling in the beta forum about this issue, actually two issues:
1. search not working properly. Noticed in 6.10 RC3 (not sure about RC2)
2. new headers not showing up altogether (there is a different topic on this issue here).

As to 1. - I am not using ANY filters. Search in a group tab is not working properly.
However, when I go back to RC1 - search is again working properly - without any changes in the filters or anything.
Not working with RC3-RC6. I did not test with RC2.
So the bug was introduced either in RC2 or in RC3

Quade said that only I alone had such weird issues, turns out that others have them too.

So as to seach the issue is perfectly clear to me: with RC1 - working, with RC3 or higher - not working.
But there is another serious problem with new headers. This problem now spread to RC1 as well, unfortunately.
I will write about it in the other thread which is on-topic.

[Quade]

This isn't rocket science. I loaded up a topic, had 1.19 million file sets listed. I typed in "pussy". Hit enter. 10 seconds later, I had 7800 "pussy"'s listed. Then I typed in "tits" hit enter and 9 seconds later had 7400 "tits" listed.

Aspen told me a group and a post. I'm downloading that group as I write this.

This problem now spread to RC1 as well, unfortunately. I will write about it in the other thread which is on-topic

Software didn't change but, it stopped working. What does that tell you? It's not like the older version has changed. You're the common factor in all these problems. Seeing how you totally overcomplicated the /3gb switch thing, I suspect the problem is that you can't trouble-shoot this because you have too much going on at once.

[RayMark]

I think you don't need a specific group, it is not working with many of them.
A group does not even have to be large, or a large day range loaded.

I think the problem can be found with groups containing pics but also with groups containing BluRays.

Anyway, it is not a rocket science to me too:

If I type "pussy" and hit enter - I get 0 pusies with RC6.
If I then clear the search field and hit enter - I still have 0 results.

If I then switch to RC1 and do this with RC1:
If I type "pussy" and hit enter - I get all the pussies. If I then clear the search field and hit enter - I again have all the headers displayed with pussies and with tits together.

So, in conclusion: a bug was introduced in RC2 or in RC3.
I suppose I could install RC2 and see exactly which one.

But also a bug with header downloads - no new headers are downloaded or added after downloading (again - with RC1 - fine).
I am pretty sure of that. At least in some cases I did not forget to check if the background cashing is gone to 0

As to /3GB - the same difference between RC1 and RC6 without /3GB, with 3030 MB and with 2700 MB.
This switch changed nothing, except drained resources for uTorrent.

Having said that, yes, sometimes it is impossible to reproduce the problem and everything might work fine on your computer.
An earlier missed issue might surface under certain conditions, maybe above RC1 more system resources or more memory is used or something.

[apsen]

Aspen told me a group and a post. I'm downloading that group as I write this.

So any update on this yet?

[Quade]

So far, I can't reproduce what I think you're saying. I had some visible

[1/1]......

Files and entered "1", then hit entry. I still saw them when I scrolled around. Still thinking about it.

[apsen]

So far, I can't reproduce what I think you're saying. I had some visible

[1/1]......

Files and entered "1", then hit entry. I still saw them when I scrolled around. Still thinking about it.

In my case they start with

Code: Select all

1/1

not with

Code: Select all

[1/1]

they are about 700 days old, I do not have any posts newer than 400 days old yet and the group is converted from V5.
Also I could upload Storage.db3 if you'd like to see it. Just tell me where you want it.

[apsen]

Here's another way to look at it (ASSUMING post count has been fixed in V6):

1) open group - I see 874838 posts
2) change 'Display Setting' to 'Hide Old' - 45944 posts
3) change 'Display Setting' back to 'None' - 246084 (!!!) posts

[apsen]

Looks like I'm replying to myself...

It seems this is specific to import from V5. If I delete the data and download all headers I do not see immediately obvious problems. But if I delete the data and import V5 headers then I see the described behavior. I have reproduced this with two groups already (every group I've tried so far). That really sucks. I had newsbin mark all headers as new during last database change and haven't completely recovered from that screw up and the only fix to this problem so far seem to be ditching V5 data and redownloading all the headers from scratch

[apsen]

Another thought: why all the headers for the group are being downloaded in the single thread. Since NB downloads them in chunks it should be be possible to download them in parallel in multiple threads.

Or am I missing here something.

[Quade]

I haven't tried import recently. Import isn't that much different than header downloads so, I didn't expect it to be different. It's interesting that you think it's import related. I'll have to dig up some old data and try an import. As for the header downloads, that's just how it works. It's more reliable this way. If everything is perfect and you split headers downloads over multiple connections, that would be great but, if there are problems, it might leave the group in a bad state. That's what I'm avoiding it for now.

[apsen]

I haven't tried import recently. Import isn't that much different than header downloads so, I didn't expect it to be different. It's interesting that you think it's import related. I'll have to dig up some old data and try an import.

It looks like every group I import exhibit this behavior. It may not be limited to import but in this case it makes the problem very pronounced. I have not played much with new headers yet. May be it's because of a lot of old headers in different state.

[apsen]

This could be something related to old/new indicator - that's why it does not show up in group with freshly downloaded headers (probably even more then one bug interacting).

I've just noticed the following:

1) open group via 'Show All Posts' - Display Age shows 'Display Age', Display Setting shows 'Display settings', post count - 8657 posts
2) change 'Display Setting' to 'Hide Old' - Display Age shows 'Display Age', Display Setting shows 'Hide Old', post count - 3351 posts
3) change 'Display Setting' back to 'None' - Display Age shows 'Display Age', Display Setting shows 'Display settings', post count - 3351 posts
4) change 'Display Age' to 'Show All Files' - Display Age shows 'Show All Files', Display Setting shows 'Display settings', post count - 8657 posts
5) change 'Display Setting' to 'Hide Old' - Display Age shows 'Show All Files', Display Setting shows 'Hide Old', post count - 3351 posts
6) change 'Display Setting' back to 'None' - Display Age shows 'Show All Files', Display Setting shows 'Display settings', post count - 3351 posts
7) change 'Display Age' to 'Show All Files' (!!! the same setting as it is currently !!!) - list reloads - Display Age shows 'Show All Files', Display Setting shows 'Display settings', post count - 8657 posts
8) repeat from step 5 ad nauseam.

[apsen]

This could be something related to old/new indicator

Or display age - I just noticed that in the Display Age mode shows headers only to about 600 days old (the exact number is different for different groups) although the Display Age itself is set to 99999 days.

Also if in the this previous example after steps 2 and 4 I change Display Settings to 'Show All Files' then I could see all those '1/1' posts in steps 3 and 5.

1) open group via 'Show All Posts'
2) scroll down to posts with subject starting with '1/1' (quotes not included) - there are hundreds of screens of such posts
3) type '1' in search box (without quotes)
4) scroll down to posts with subject starting with '1/1' (quotes not included) - there are none of such posts
5) click 'clear the search entry'
6) scroll down to posts with subject starting with '1/1' (quotes not included) - there are none of such posts

So in essence it looks like I have found work around. Unfortunately that work around is awful as it takes long time to reload whole list...

[Quade]

I loaded up a group medium sized.

Test #1

1 - Show all - 229626 sets.
2 - Hide Old - 0 sets
3 - Remove Hide old (pick "None") - 229626 sets.


Test #2 - I set 3631 of the posts to be "New" again

1 - Show all - 229626
2 - Hide Old - 3631 sets (same number I set New).
3 - Removed hide old 229626


So basically, I don't know what you're seeing. You seemed to think this was related to importing the headers. 6 and 5.59 both store "old state" in the headers, though one is a DB and one is an RV4 file, "Downloaded" state though is stored in a different file in 6. Wonder if some of the weirdness you're reporting is because of the downloaded state?

You know if the date of your PC is incorrect, none of the "Display Age" options will work properly. The fact you see to have to always use "Show all" sort of suggests something like the date thing.

[apsen]

So basically, I don't know what you're seeing. You seemed to think this was related to importing the headers. 6 and 5.59 both store "old state" in the headers, though one is a DB and one is an RV4 file, "Downloaded" state though is stored in a different file in 6. Wonder if some of the weirdness you're reporting is because of the downloaded state?

You know if the date of your PC is incorrect, none of the "Display Age" options will work properly. The fact you see to have to always use "Show all" sort of suggests something like the date thing.

The date on my PC is fine.

And I do not know if it's related to import - just that I see it reliably in all imported groups. If I delete the group and download the headers from scratch I'm getting headers below the threshold so the problem may not show due to that.

If download state makes a difference it's not obvious how. All the problems that I've noticed were with not downloaded posts. But I was not looking specifically for the downloaded ones.

I've also offered to upload one of my groups for you to look at but you never took that offer.

Please look into this. V5 has a lot of problems - I was hoping V6 will be better. But so far it seems to be much worse.

[Quade]

When you download headers, Newsbin writes them to an RV4 and then a worker thread imports the headers in the background into the DB. When you import headers, Newsbin reads the existing RV4's in the spool_v1 and imports the headers into the DB.

It's essentially the same mechanism. You have the display age set to 273 years ago so, I don't know it that's screwing things up. Certainly it's nothing I've ever tested.

How big are the storage and storagedata.db3 files? The ones you're offering?

[apsen]

How big are the storage and storagedata.db3 files? The ones you're offering?

The whole directory for the group ranges from 125 Kb to 3.5 Gb.

The smallest one has just 10 headers and exhibit the same behavior:

Code: Select all

Storage.db3                             │   90 112
StorageData.db3                         │   32 768
Range.db3                               │    3 072


[apsen]

You have the display age set to 273 years ago so, I don't know it that's screwing things up.

I've lowered it to 9999 days and it seems to be working OK now (I've checked only one smallest group - would do some more testing).

So it seems there's a overflow bug somewhere in V6. But if it's in Display Age then why does it affect 'Show All Files' mode?

PS IIRC V5 had limit of 9999 days for that setting.

[Quade]

There's no reason to set unreasonable values in something like the display age. The time stamps used throughout Newsbin only go back to 1970. No news server has more than 1300 days at this point so, 2000 would be a reasonable number.

I mean yeah, setting it to be 270 years probably causes an overflow. I'd say that's "not recommended".

I just changed it to limit it internally to 5000 days. You can enter anything you want but, it'll always cap it at 5000.

[apsen]

There's no reason to set unreasonable values in something like the display age.

Putting 'reasonable' argument aside we have a bug. Actually it looks more like two:

1) NB allows entry of the value that causes overflow.
2) Setting 'Display Age' value affects 'Ignore Display Age' ('Show All Files') mode.

[Quade]

What are you trying to achieve with those large numbers? You might as well right click "Show all Posts" because it just ignores that value and loads everything up. When you set a large display age, you're basically saying "Always do a Show All". You were complaining about how long it takes to "Show All" but, large "Display Age" becomes virtually the same thing.

[apsen]

You were complaining about how long it takes to "Show All" but, large "Display Age" becomes virtually the same thing.

That was only in relation to reloading all posts after each search. I could wait once on initial load...

You might as well right click "Show all Posts" because it just ignores that value and loads everything up.

That's exactly what I do - but it turns out NB does not ignore that value. I do not use Display Age - it was set to some value and I wouldn't even think about it if it didn't cause this problem.



I'm able to use V6 now (assuming my further tests confirm this) so there's no longer an urgency to this.

I'm just trying to point out bugs that do need fixing in future versions:
1) The first one with allowing to set too high Display Age value is not as critical but should be very easy to fix.
2) The other one (Display Age affecting 'Show All Files/Show All Posts' mode - BTW it is named differently in different places - cosmetic bug) is probably much more serious and is not clear how easy it would be to fix.

[Quade]

It does ignore the value. My "Display Age" is set to 10 days. I regularly "Show all Posts" and see 1000 days worth of posts. I'd argue that you need to check your options again and any of them you've set to large values, you might want to consider setting them to normal values. If you find more like "Display Age" that need to be capped, then I'll cap them.

"Show All" only determines how much is loaded into memory. Any additional filters you might have set, in the post list or in the filters can override the display. So, you can load 1000 days into memory but, if you have filters set to only show 60 days of the 1000, that's all you'll see.

[apsen]

It does ignore the value.

Repeat my tests with that value set to 99999 and 9999 in Show All Files mode. The results are different - so how could you say NB does ignore it?

[apsen]

What is the highest safe value for all age limits in NB?

Now I have problems with Storage age being 9999.

[Quade]

I wouldn't go over 1500 normally. No server has that much binary retention yet.

I've added a hard cap of 5000 in the next version for people who set...unusual values. You're right, I need to protect people from themselves. The question to ask might be why you seem to have more problems with this than other people? It's looking now, like it has nothing to do with importing, probably has nothing to do with the global "Display Age". You can override the global display age in the group properties so, I'd make sure to look there too.

[apsen]

I wouldn't go over 1500 normally. No server has that much binary retention yet.

What if I store more then 1500 days worth of posts? Anyway my server has 1200 days retention. In a year it could be growing past 1500. If I set it to 1500 I would need to remember to change it. So I'd rather have it as high as possible so I never have to worry about it.

The question to ask might be why you seem to have more problems with this than other people?

Are you sure I'm HAVING more problems not just NOTICING them? Because of past (?) problems with NB search I do my search with external tools and then when I find it I search it in NB for specific stuff to download and could not find.
If I were using just using NB there would be no indication that I'm missing posts.

I've added a hard cap of 5000 in the next version for people who set...unusual values. You're right, I need to protect people from themselves.

I would call it protecting the program from the input that breaks it.

It's looking now, like it has nothing to do with importing, probably has nothing to do with the global "Display Age".

I do not know what actually causes the problem - I do not have access to source code to trace them. All I could offer is hypothesis. Only you could verify them. It looks like I was hitting the problem only with import because otherwise I did not have enough history to trigger it. As to 'Dispay Age' it is definitely plays some role in it as changing it changes the programs behavior.

Also my problem with 'Storage Age' may be similar but is triggered an group update. On update I'm loosing posts that are less then storage age from the group. This one should have probably be in it's own thread but it seemed reasonable to ask about it here at the time.

You can override the global display age in the group properties so, I'd make sure to look there too.

No, no overrides there.

PS I do not really know if you are actually looking into this problem but if I had access to source code I would be all over it trying to find the bug caused by high value of Display Age as it is very suspicious and could have much more consequences then we know of. There may be memory corruption for example. Also I do not like the fact that the oldest posts in each group had different age - it suggest that it is not as trivial as simple integer overflow.

[Quade]

but if I had access to source code I would be all over it trying to find the bug caused by high value of Display Age as it is very suspicious

I knew exactly what the problem was, the second you mention that reducing the value fixed it for you, It was trivial once I thought about it.

Unix timestamp - which starts at 1970.
Display Age - Set to 270 years ago.

Capping it to 5000 IS the fix. 5000 is 13 years ago.

I also looked to see if the display age has anything to do with "Show All" and the answer is, not in the loading part. It always loads all when you load all. It's likely though that display age acts as a filter in the display part.

Capping the display age at 5000 fixes that too.

I'm glad you found this weirdness. I'm glad it's fixed but, it's just a typical underwrap problem. If in the future, you decide you want to store more than 13 years worth of headers, let
me know. I'll expand it. In fact, maybe I'll set it to 26 years. That's safe enough.

Here you go, 26 years
static const int C_INIT_DAYS_MAX = 10000;

[Quade]

I took another look at this too. I see my changes weren't sufficient. In your case you need to limit

"Display Age"
"Storage Age"
"Download Age"

To no more than 10,000. 26 years. I've built the 26 year cap into the next beta so, it won't be possible for a user to set the values too long and cause this wrap problem.

Any number over 15000 will cause underwrap in the version you have.

[apsen]

I also looked to see if the display age has anything to do with "Show All" and the answer is, not in the loading part. It always loads all when you load all. It's likely though that display age acts as a filter in the display part.

If I understand this correctly you've missed my point: the post disappear not during load but after search or switching between None/Hide Old. If I reload posts after that via "Show All Files" - I'll get missing ones back. Something kicks in that filters the posts out after those operations. It is suspicious that changing Display Age affects it as it shouldn't be checked at all in "Show All Files" mode... Or do those operation reset the post list to "Display Age" mode?

PS Sorry for being annoying but I feel there're still issues to sort out.

[Quade]

It's hard to tell what state you're in because things are changing too much. We already know that having the age settings set too long is problematic. So, at this point, I'm starting from scratch. You need to tell me that the age settings are set to say, 2000 and then start over.

For example, if you had storage age set to a very large number and it under-wrapped. You might have lost some or even many of your imported headers. There's no telling. Essentially I'm saying there's too much going on at once to really trouble-shoot. Once the ages are set to 2000, you might even want purge and re-import the headers and see if the symptoms change. At that point, we're sort of at a common baseline and my testing will more closely match yours.

[apsen]

It's hard to tell what state you're in because things are changing too much. We already know that having the age settings set too long is problematic. So, at this point, I'm starting from scratch. You need to tell me that the age settings are set to say, 2000 and then start over.

I certainly understand that.

I'm getting more information on this meanwhile. Stupid me was changing Display and Storage age synchronously!!!
I should have also paid more attention to log - should have had it on DEBUG level all this time.

OK This is updated story: When you load all - you get all. Then if you have high STORAGE age set and change between None/Hide Old you'll see the following message in the log:

Code: Select all

DEBUG   - Attempted to Display Files older than StorageAge - they were hidden

And you were right Display Age does not affect things in Display All mode - I could have it set to 99999 and things work as long as Storage age is relatively low.

[apsen]

Any number over 15000 will cause underwrap in the version you have.

15255 to be exact. So 15000 would be the natural choice for the age settings limit instead of 10000, wouldn't you agree?

I'm curious, that does not seem to correspond to any integer limits even if I take into account that you're using Unix date there...

[Quade]

Unix time at Jan 1 1970 is 0. If you subtract 270 years worth of seconds from today's unix time you go less than zero, except time_t is an unsigned 64 bit number so, you underwrap to some large number.

10,000 is a nice number and it's more than large enough to hold anything you might possibly display or view. I see no point trying to make it exactly unix time stamp. Extra but, useless complexity. Honestly, 5000 is just as good. It's a circuit breaker. There's no need to it to do more than "stop crazy numbers".

[apsen]

If you subtract 270 years worth of seconds from today's unix time...

Got it - today is 15255 day of Unix epoch.

I see no point trying to make it exactly unix time stamp.

BTW I never suggested that. But rounding it 15000 would be nice.

Well, 10000 should work for a year or two as NNTP was introduced around 1985-86. I'll be back when that happens.