After update to v6.21 (from last v5) Local Search Is Limited

[Kaigi]

I just upgraded from the final v5 to v6.21 [I'll skip the why]. Since doing so, I re-added a group for NB to download headers from that I'd mistakenly deleted shortly before the upgrade. I had NB 'Download All Headers' which is going back between 600-1000 days: NB said there were 3.5M posts on my server. Since it clearly was downloading them all, it took letting NB run overnight for that process to complete.

Today I tried to do a Search/Local, and it only shows posts going back three months. Though I activated the filters, nothing in the pull-down menu for 'Age' could be selected, though Filters for Min/Max Size and 'Filter Profile' can be selected. The 'Age' area doesn't ALLOW a choice of 'NONE' (or anything else) for back headers locally.

With the limited date range of back headers I saw in the results, I looked, and the SPOOL_V6 directory for that group is only 1.3GB. When I had that group in v5 shortly before the upgrade, the last size for it's SPOOL directory was over 10Gb. So I compared the size of all the subdirectories of the v5 SPOOL_V1 folder that were saved on an external drive just before the upgrade, and compared them to their V6 counterparts. I found that many of the other folders in the SPOOL_V6 area are smaller than in _V1, but a few are even twice the size that the _V1 equivalent was. Yet this one group's SPOOL folder was 1/10th the size.

For comparison's sake, I tried doing a search for one of the other groups (one that had been in the folder when the upgrade took place). It's search only went back about six months, when I'd had over two years worth of headers available in NB v5. The 'Age' pull-down did not work for that group, either, though - oddly - ITS Filter settings were retained when turned on and off.

What has happened to Local Searches: why can't an Age range be selected in v6? [I'm assuming it has something to do with the limited headers that NB seems to have saved for that group. But that should not cause the 'Age' selection to be unavailable for Local Searches.]

What has happened to saved Headers: why the severely truncated saved list (unless there's a 90%-compressed listing of all of them, with something else wrong so that I can't access them via the Local Search function)?

I counted on local searches for many things, yet it appears that it no longer works in v6.


A secondary, less critical, but still very important issue (to me, at any rate): in v5 the last selections used for Search Filters would be retained, even if they were turned off. Now, each time they are turned off those selections are cleared and they have to be re-entered. This is a major annoyance, as most times I do a search I will switch back and forth between them being active and being inactive [if filtering out things seems to have eliminated something that I think may be there that I'd want (e.g. screens for a vid)].


Thank you for any help you can give on this.

[DThor]

Check Options/Setup/Storage Age. You can download as far back as you want, but the result saved to disk is trimmed to that. I'm guessing that's the problem.

DT

[Quade]

Pick a group, right click and "Show all Posts" and see how many days you see. "Local Search" will show no more than what "Show all Posts" shows so, if "Show all" only shows 30 days, so will local search. I was testing the hell out of local search yesterday. Made some tweeks to make it more responsive to column clicks subsearching while searching is still in progress. I was getting records from 1200 days ago. My storage age is 1500ish.

Keep in mind that records from 6 aren't automatically read by 6. You need to manually import if you want 5.X records in 6.

I'm not clear what you means with the filters. Give me a sequence to follow and I'll try it.

[Kaigi]

Sorry for the long delay, the 'why' for the delay is below (under the second part of Quade's comments).

Also apologies for the length. I've been accused at times of including too much information, because (with my head injuries) I've at times left out critical info so sometimes overcompensate for that.



At any rate... thank you both for the ideas/suggestions. To address them further:

Check Options/Setup/Storage Age. You can download as far back as you want, but the result saved to disk is trimmed to that. I'm guessing that's the problem.
DT

I had 'Storage Age' set to 600 days. But it only shows 106 days worth (adding up the days from first post showing to most recent). I just upped it to 1000, which is what my 'Download Age' and 'Display Age' both are. [Possibly the program got confused because I said to 'Display Age' 1000 days, but 'Storage Age' to only 600 originally? Those numbers worked with v5, so I doubt it, but then again, v6 was - I believe - a total rebuild, so possibly with the importing it got confused.]

1. Pick a group, right click and "Show all Posts" and see how many days you see. "Local Search" will show no more than what "Show all Posts" shows so, if "Show all" only shows 30 days, so will local search. I was testing the hell out of local search yesterday. Made some tweeks to make it more responsive to column clicks subsearching while searching is still in progress. I was getting records from 1200 days ago. My storage age is 1500ish.

2. Keep in mind that records from 6 aren't automatically read by 6. You need to manually import if you want 5.X records in 6.

3. I'm not clear what you means with the filters. Give me a sequence to follow and I'll try it.

1. The number of days showing differs for different groups. Some groups showed only a few months [Sept 2009], but I just checked a couple that had back to about 500 days [Aug 2010].

2. I wasn't aware that v5's Headers weren't auto-imported (since the install says, "saving all your settings, etc." I figured it was done automatically like prior 'full number' versions did). So...

I had a backup of everything from a week or so ago, prior to when I'd deleted the group with the 'three month' problem. So I copied that folder to the 'SPOOL_v1' directory and ran 'Groups/Post Storage/Import v5.xx Headers' for that group.

It took about eight hours to import the v5 headers into v6 [given the 10.3Gb size for that one group]. From that I got about 15 months worth of headers (likely what I had for v5), and in v6 they take up 13Gb of space.

[Now for why the delay to respond:] Then I thought, "hey, that seems to work, maybe in Options making the 'Storage Age', 'Download Age', and 'Display Age' the same 1500 days as what Quade used might make the downloaded headers work for everything my server has". So I tried that lengthy process again (about 6-8 hours) and again, even though the 'Size' in the window showed the full number of posts available (and downloading), I only got about 3 1/2 months worth of headers. That was despite the 1500 days for each of those three 'Age' ranges in my Options, and the hundreds of millions of posts showing in the 'Size' area during download.

So I had to RE-import the v5 headers to fill it back in (again, getting it only back up to about 15 months, when my server's retention is over two years). But what that all means is that the 'Download All Headers' function still isn't working properly for me.

3. I tried using Search for another topic, and it saved some of the search parameters (the 'Min Size' one), but only for that group or group set. [I have groups sorted into three categories/directories in the Groups window.]

Hopefully this picture will make it clearer what I am referring to with the search not allowing a change in 'Age' range (since the online pic account could potentially be traced to me, I eliminated the content of the search in the uploaded image):

As you can see, both "Age" and "Max Size" are greyed out (unselectable). I made a separate Filter profile called 'NONE' that I use, that simply sets everything at it's default (no limits to anything). Trying different filter sets didn't make a difference to the greyed-out 'Age'.



New [related] Question: Once any desired headers have been imported from v5, does the 'SPOOL_v1' directory's contents then become merely a digital paperweight? [i.e. is it not actively used by v6?] That'd clear a bit of room on the hard drive.


[My brain is so fried from thinking this intensely, I hope I didn't miss anything.]

[Quade]

[Now for why the delay to respond:] Then I thought, "hey, that seems to work, maybe in Options making the 'Storage Age', 'Download Age', and 'Display Age' the same 1500 days as what Quade used might make the downloaded headers work for everything my server has". So I tried that lengthy process again (about 6-8 hours) and again, even though the 'Size' in the window showed the full number of posts available (and downloading), I only got about 3 1/2 months worth of headers. That was despite the 1500 days for each of those three 'Age' ranges in my Options, and the hundreds of millions of posts showing in the 'Size' area during download.

- What server are you using? Only giga, astra and their reselllers can deliver headers for the full span of their retention. Many of the other servers cap the count to 125 million posts which is years on a low traffic group and months on a high traffic group. That's what this sounds like.

- Storage Age is what you use to purge older headers. If you have it set to 600 then anything older than 600 days will be purged.

- Display Age isn't used for local search. Only when you show posts is Display age applied.

Search doesn't seem to save state between runs. I'll have to check it out.

[Kaigi]

[Now for why the delay to respond:] Then I thought, "hey, that seems to work, maybe in Options making the 'Storage Age', 'Download Age', and 'Display Age' the same 1500 days as what Quade used might make the downloaded headers work for everything my server has". So I tried that lengthy process again (about 6-8 hours) and again, even though the 'Size' in the window showed the full number of posts available (and downloading), I only got about 3 1/2 months worth of headers. That was despite the 1500 days for each of those three 'Age' ranges in my Options, and the hundreds of millions of posts showing in the 'Size' area during download.

1- What server are you using? Only giga, astra and their reselllers can deliver headers for the full span of their retention. Many of the other servers cap the count to 125 million posts which is years on a low traffic group and months on a high traffic group. That's what this sounds like.

2- Storage Age is what you use to purge older headers. If you have it set to 600 then anything older than 600 days will be purged.

3- Display Age isn't used for local search. Only when you show posts is Display age applied.

Search doesn't seem to save state between runs. I'll have to check it out.

1- I've got UseNetServer. It could be their cap (if they aren't a giga or astra reseller). But it doesn't make sense: I can't believe that their cap would have changed dramatically at the exact time I change from v5 to v6. Because the 10+GB of stored headers I had from v5 were from the same newsserver; I haven't changed servers in well over a year.

2- Yes, that's why I changed Storage Age to 1500 days (along with 'Download Age' and 'Display Age') before trying to download the headers again. And I still got only about 3 1/2 months worth of headers from that.

3- That screenshot I posted was taken with a search within the 'Show All Posts' tab for that particular newsgroup. But the Age was still grayed out.

[Quade]

1- I've got UseNetServer. It could be their cap (if they aren't a giga or astra reseller). But it doesn't make sense: I can't believe that their cap would have changed dramatically at the exact time I change from v5 to v6. Because the 10+GB of stored headers I had from v5 were from the same newsserver; I haven't changed servers in well over a year.

It makes perfect sense. There's a 125 million cap so, new records are added and old records deleted. If you've been downloading headers regularly all year AND the storage age is long enough you end up having more headers than your server does. That's why "download all" only goes back a few months but, importing from 5 gets you much more span. It's just the headers that are limited anyway. Using Search, NZB or headers from another server would still let you download files going back 1000 days.

- I click the power button and nothing is unclickable. Selections will be grayed out if they're default or if the filters are disabled. Looks to me like the filters are disabled. As I said, search tab isn't saving state. I have to decide if that's a good idea or not.

[Kaigi]

1- I've got UseNetServer. It could be their cap (if they aren't a giga or astra reseller). But it doesn't make sense: I can't believe that their cap would have changed dramatically at the exact time I change from v5 to v6. Because the 10+GB of stored headers I had from v5 were from the same newsserver; I haven't changed servers in well over a year.

It makes perfect sense. There's a 125 million cap so, new records are added and old records deleted. If you've been downloading headers regularly all year AND the storage age is long enough you end up having more headers than your server does. That's why "download all" only goes back a few months but, importing from 5 gets you much more span. It's just the headers that are limited anyway. Using Search, NZB or headers from another server would still let you download files going back 1000 days.

- I click the power button and nothing is unclickable. Selections will be grayed out if they're default or if the filters are disabled. Looks to me like the filters are disabled. As I said, search tab isn't saving state. I have to decide if that's a good idea or not.

The explanation of why I've got more headers than what UseNetServer 'allows' in back-DLing makes sense: I got them while they were still 'fresh' and Newsbin - because of my settings - just never deleted them.

Actually, in that picture I put up (and when I've tried it), the filters have been on: the "None" and the "50 k" are in black and are fully selectable, the "Age" and "Max Size" are grayed out and unselectable. I still don't understand why that is.


I'd put in a vote that the search tab saves state. Because, if one turns on filters, then one has to select SOMEthing regardless; if what has been saved isn't what someone wants, they can change it, but they'd have to select something regardless if they turn them on (or there's no reason to turn them on).


To re-ask: New [related] Question: Once any desired headers have been imported from v5, does the 'SPOOL_v1' directory's contents then become merely a digital paperweight? [i.e. is it not actively used by v6?] That'd clear a bit of room on the hard drive.

Thank you!

[itimpi]

The SPOOL_V1 folder is no longer used by v6 after header import has completed.

[Kaigi]

The SPOOL_V1 folder is no longer used by v6 after header import has completed.

Thank you! There's a few gig saved.


NEVERMIND on the filter criteria not working. I found the problem [the problem was in my head (thanks to the head injuries): in v5 I had the default at 7 days for YEARS, and I was so used to having to occasionally change the Age range to 'None' that it ceased being apparent to me that 'no age limit' was the default setting ].

[EDIT: I realized on waking today that it wasn't JUST my head injuries (which these days I automatically assume are fully at fault when I make mistakes I wouldn't have in the past): in v5 the 'None' for the Age limit was selectable and would show in the window, while in v6, if 'None' is selected for the Age limit, the program just still shows the grayed-out word "Age". So what showed on the interface was different and suggested that 'nothing changed'.]

Sincere apologies!


But since I do have a number of searches I do roughly every 5-7 days, I do, again, vote for having the Search function retain state for whatever filters were selected last. Otherwise every week I'll have to re-set the four settings in the filter area for the searches again, and again, and.... Thanks!