WatchList Scans

[OldGuyWalking]

---Automatic Scanning---
If I update and enable several Watch List entries, only the first one that I enable appears to do a search immediately. For example, I just enabled 9 WatchList items and got one log entry for the first item only -
[15:46:34] DEBUG Periodic Search <WatchList Title> : <Search.Term.xyz.> No Results

Also, even though the log shows that NB search for that first item, the entry in the WatchList did not update. It still shows Never in the Last Updated column.

I watched the log and it was 30 minutes before NB searched for the rest of the items and updated the Last Updated column.

---Rescan Topic---
If I do a "Rescan Topic" on an item in the WatchList, the log does not show that it did a search. The only entry I get in the Log is:
[16:41:47] DEBUG Connection Status: SSL Connection Server: is.newsbin.com SSL Cert: is.newsbin.com Stats: DHE-RSA-AES256-SHA 256 TLSv1/SSLv3

Same as the above, nothing is updated in the Last Updated column after a Rescan.

I could not find any documentation on Rescan Topic in the User Manual so maybe I'm not using it correctly. I'm a bit confused about what the "Pick Topic" field does. If I've selected a WatchList item to scan, what does the "Pick Topic" option do? So far, I've not gotten any results by using Rescan Topic. If there's something documented that explains how it works I'd appreciate it if someone would point me in that direction.

Thanks,
Bill

[Quade]

1 - The watch item only updates if results are actually found.

2 - Rescan just does a normal search. The default is only 30 days. If updates the entry if results are found.

I just tested one of mine. I did a rescan, 20 results were found and both the count and timestamps were updated.

If you're not getting results, it suggest your watch isn't working.

[OldGuyWalking]

1 - The watch item only updates if results are actually found.

If by updates you're talking about the count I agree. But NB does update the timestamp even if it does not find anything. But it only does it if one of two things happens (as far as I can tell).

1. WatchList items are enabled before NB is started. NB will search for each of the enabled WatchList and update the timestamp for each item even if no results are found. It creates a log entry for each item that states
[17:09:29] DEBUG Periodic Search <Watch.Title> : <Search.String.xyz> No Results

2. NB is started with no WatchLists enabled. WatchList items are enabled. Wait until 30 minutes has elapsed then NB will update the timestamp on the WatchLists even if it doesn't find anything. It will, of course, update the count if it does find something.

I was expecting NB to start a search as soon as I enable a WatchList item and then have it do the automatic searches based on the timer (Using the default 30 minutes). To avoid having to wait 30 minutes for NB to start its search the workaround I'm now using is to enable those WatchList items I want NB to search for for that day and then close and then restart NB. At that point, that triggers NB to start searching for the enabled WatchList items.

2 - Rescan just does a normal search. The default is only 30 days. If updates the entry if results are found.

I just created a Test item in WatchList. The Source is the Search Engine. To keep it simple the Search For string is eggs. I did a search in the Search tab with Age set to 1 month for eggs and got 19 results with 1 being an Incomplete. The 30 minute timer hit right after I created this Test and the results I got were
[18:34:32] DEBUG Periodic Search <Test> : <eggs> 12 Results 6 Of them were new.
Note that 17 of the 19 results in Search showed a Status of Idle/New so I'm not understanding why there was a different of 7 items?

I deleted the stored posts in Test and changed the Search string to chicken. A search in the Search tab gave me 63 results with 59 Idle/New and 4 Idle/Old. I selected Rescan Topic. In the dropdown box titled "Pick Topic" I selected Watch_Topics. I have no idea if that's the correct selection but nothing happens when I leave the Pick Topic blank. The only items available in the dropdown box are 2 Group folders I created, the Watch_Topics folder, and the Unsorted folder. I left the default at 30 days. I press Start Scan and it displays 0 records and 0 matches. When I stopped and restarted NB and NB did the search on the Test WatchList it came up with 46 results with 23 of them new.

As I mentioned before I might be missing something but when you say Rescan is just doing a normal search I'm assuming you mean it's using the same Search Engine source that the WatchList I'm trying to rescan uses?

Any help is of course appreciated, and the more verbose, the better. Now that I've figured out that I can get the WatchLists to trigger a search by restarting NB that gives me a workaround so I don't have to wait 30 minutes for NB to start searching newly enabled WatchList items.

But I'm totally not getting what I'm doing wrong with Rescan Topic.

Totally separate curiousity item - The timestamp that's displayed in the WatchList is an hour ahead of the timestamp in the log. Not a big deal but I'm guessing the WatchList timestamp is coming from NB's system and the log time is using the system clock?

Thanks,

Bill

[Quade]

- Looks like "rescan" might be double-counting. I searched with search with the same settings as my watch and got 79 results. Then deleted stored posts on the watch item, did a rescan set for 3000 days and got a reported 158 hits. When you load the watch there are only 79 like the manual search.

was expecting NB to start a search as soon as I enable a WatchList item and then have it do the automatic searches based on the timer (Using the default 30 minutes). To avoid having to wait 30 minutes for NB to start its search the workaround I'm now using is to enable those WatchList items I want NB to search for for that day and then close and then restart NB. At that point, that triggers NB to start searching for the enabled WatchList items.

It doesn't. The timer fires, it starts the cycle of watch items and when they're done, it waits for the timer to kick over again. When you enable/disable you just remove/add them from the loop. If you add a new watch item. It'll break out of the long pause but won't interrupt the watches that are already running. "Rescan" is essentially how you trigger and insta-search. I need to move it to within the "add watch" window too.

I think the timestamp is a daylight savings issue.

[OldGuyWalking]

It doesn't. The timer fires, it starts the cycle of watch items and when they're done, it waits for the timer to kick over again. When you enable/disable you just remove/add them from the loop. If you add a new watch item. It'll break out of the long pause but won't interrupt the watches that are already running.

I'd been assuming the trigger that started the timer was somehow tied to the Watch List. If the timer starts when NB starts that would explain why nothing appears to happen when I enable a Watch item until the timer winds down.

It would be nice if there was a button that could be pressed to restart the timer in order to avoid having to wait for the timer to kick over when Watch List items are enabled. It would force a rescan of all of the enabled Watch List items. I'd prefer this as an alternative to the more limited "Rescan Topic" feature (see below). Maybe a button in the Watch list dialog box? Just a thought.

"Rescan" is essentially how you trigger and insta-search. I need to move it to within the "add watch" window too.

After experimenting it seems that Rescan Topic does not (1) use the data Source identified in the Watch item and (2) is limited to searching headers from a single Group selected in the Rescan dialog.

Maybe, under certain circumstances, Rescan Topic might be useful if the Watch List Source is set to Header and the Look in Groups field is set to a single Group and the user selects that specific Group when doing a Rescan.

Data being retrieved for a Watch List should be consistent regardless of whether it's being retrieved by the Watch List or using Rescan Topic. That can't happen when the Watch List Source is set to (1) Search Engine or (2) it's set to Header and the Look in Groups is set to All Groups. All of my Watch List items Source are set to "Search Engine" and that makes Rescan Topic unusable since it doesn't use the Search Engine.

Are you planning on linking Rescan Topic to use all of the search parameters and data source identified in the WatchList? I wasn't sure if that's what you meant when you said you need to move Rescan into the Add Watch window.

Thanks,
Bill

[Quade]

After experimenting it seems that Rescan Topic does not (1) use the data Source identified in the Watch item and (2) is limited to searching headers from a single Group selected in the Rescan dialog.

1 - I have my logging set to "Debug" (Advanced options).

2 - I pick one of my existing search based watch items and "Rescan". The only thing I changed was set it to 3000 days.

3 - I look in the logging tab

[08:50:07] DEBUG Query=<something>&MinAge=259200000&MinSize=209715200&Offset=0
[08:50:07] DEBUG Connection Status: SSL Connection Server: is.newsbin.com SSL Cert: is.newsbin.com Stats: DHE-RSA-AES256-SHA 256 TLSv1/SSLv3
[08:50:07] DEBUG AddToSearchDB Elapset:15:284

Is what I see. So, what I'm seeing is that watch list items set for "Search Engine" are using the search engine during rescan. That last log message tells me it add 284 hits to the local watch list storage. I also confirmed it used the groups that from the topic I'd selected in the watch list.

"Search Engine" searches all groups by default. You can also select a topic that contains a subset of the groups to limit it to searching in these groups. The "Look in group" drop down lets you pick the topic which contains the groups you want it to search for. I just set this in the watch options. I didn't change anything when I rescanned except the age range.

[OldGuyWalking]

I'd like to show some screenshots. Just read the FAQ and it says I have to use a publically available server. Any suggestions?
---------------------------------------------------------------------------------------------
Here's what I'm seeing -

1. Debug is on. I setup a "Test" Watch with the Source set to Internet.
[img]K:\Graphics\Temp\WatchList_Test_2015-11-06_105534.jpg[/img]
2. I select Rescan Topic in the WatchList for Test.
[img]K:\Graphics\Temp\Rescan_Topic_Option_2015-11-06_105952.jpg[/img]
3. Dialog pops up. I have 3 Groups setup. If I don't pick one of them I won't get any results. I can't leave "Pick Topic" blank because then nothing happens.
[img]K:\Graphics\Temp\WatchList_Scan_Dialog_2015-11-06_110225.jpg[/img]
4. For purposes of this test I use the "Unsorted" Group and also set the days to 3000. It finds 1020 results but they are all from Headers in the Unsorted Group and not from the Search Engine.
[img]K:\Graphics\Temp\WatchList_Scan_Result_2015-11-06_110640.jpg[/img]
5. After running the scan the only Debug entry that is displayed in the Log is the following. I'm not getting any of the other entries you show on yours.
[11:06:18] DEBUG Connection Status: SSL Connection Server: is.newsbin.com SSL Cert: is.newsbin.com Stats: DHE-RSA-AES256-SHA 256 TLSv1/SSLv3

[Quade]

In order to post images, you'd have to post them to something like Photobucket and then paste the link into the forum post. Posting the local path doesn't transport the images to here.

I actually don't need images though. Your description of your process was enough for me.

1 - A "Topic" is a sub section of the groups list. You can right click the groups list and select "New Folder" then make a topic other than "Unsorted". Then you can either drag and drop groups into this "Topic" or use the right click menu to "Send to Folder" groups you want to send to this topic. These "Topics" are what you select in the watch list setting if you want to scan more than one group. I'd suggest using them even if you're only scanning one group.

2 - You did discover a bug. "Rescan" won't search if there's no "Topic" selected in the watch settings. I'll have that fixed before the 6.62 release.

3 - The reason you only see the connection message is because I run Newsbin under the debugger and you're running a release.

I'd say the fact you selected "Unsorted", you got results AND you got the connecting to the search engine message, means it's using search for that rescan.

So my suggestion is to make a new topic. Move the 3 groups you were trying to use in the watch settings into this topic. The go into the watch setting and select this topic in the "Look in Groups" drop down. Then try a re-scan. I fixed the double-count while I was in there.

If you download headers too, if you load that "Topic" for the 3 groups, all 3 groups will load up into the same "Tab" in Newsbin.

[OldGuyWalking]

I understand everything you're saying but the results that are posted when I do a Rescan Topic does not show any data coming from the NB Search Engine even though I can get results by manually doing a search in the Search Tab and NB will also get data from the Search Engine once the timer restarts.

I only mentioned the Groups because Rescan Topic requires one. I use "Internet" as the Source in all of the Watch Lists.

Because Rescan Topic requires a Group I've tried each of the 3 I have setup (not just Unsorted). Because I don't normally use Groups I haven't updated any of them in months so the data is old. That's one reason I don't believe I'm getting any results from the Search Engine because I'm not getting any new results when I run a Rescan, just the results from the old headers in the Group I selected in Rescan.

I'll wait for the 6.62 release to see if it works any differently.

Thanks,
Bill

[Quade]

That's one reason I don't believe I'm getting any results from the Search Engine because I'm not getting any new results when I run a Rescan, just the results from the old headers in the Group I selected in Rescan.

[11:06:18] DEBUG Connection Status: SSL Connection Server: is.newsbin.com SSL Cert: is.newsbin.com Stats: DHE-RSA-AES256-SHA 256 TLSv1/SSLv3

This tells me it's hitting the search engine. Search and headers will essentially report the same files. The point of search is that you don't need to download headers if you use it.

So are you seeing results in the group headers that you don't see when you search?

I just want to be clear that search and headers should show pretty close to exactly the same data.

[OldGuyWalking]

Search and headers will essentially report the same files.

I think what's causing some confusion is that I started out saying that Rescan was not using the Source settings (i.e. Internet) from a Watch List and that became Rescan wasn't using the Search Engine which really wasn't what I meant. Also, I'm not using headers in any of the Watch List items and I probably mistakenly used that term in a couple of my ealier posts.

I did some additional testing to try to illustrate what I think is happening.

1. I have a Folder in the Groups tab called Audiobooks that contains 11 groups. (This is relevant to #3 below)
a.b.audiobooks
a.b.audioboooks
a.b.mp3.audiobooks
a.b.mp3.audiobooks.highspeed
a.b.mp3.audiobooks.repost
a.b.mp3.audiobooks.scifi-fantasy
a.b.sound.audiobooks
a.b.sounds.audiobooks
a.b.sounds.audiobooks.repost
a.b.sounds.audiobooks.scifi-fantasy
a.b.sounds.mp3.audiobooks

2. I setup a Watch List called "Test1" -
- Source set to Internet
- Search For is set to "egg"
- Look in Groups field is blank.
- No other settings changed.

3. I run Rescan Topic on Test1 and select Audiobooks in the Pick Topic field and use 3000 days.
- It says it finds 432 Matches but as you pointed out in an earlier post, it's actually 1/2 that (216).
- When I look at the Group column in the posted results it shows that the results came from the following Groups which are all included in the "Audiobooks" folder (as shown in #1 above).

a.b.mp3.audiobooks
a.b.sounds.audiobooks
a.b.sounds.audiobooks.repost
a.b.sounds.mp3.books

4. I waited for the timer to run down which triggered the Watch List search for Test1 and an additional 36 records were added to the post.
[16:03:23] DEBUG Periodic Search <Test-Internet> : <egg> 72 Results 36 Of them were new.

5. Looking at the Group column for the newly added 36 records it shows they came from the following Groups:
a.b.boneless
a.b.movies
a.b.multimedia
a.b.sounds.flac
a.b.sounds.mp3.progressive-country
a.b.sounds.world-music
a.b.teevee

So, to clarify what I'd meant to say in earlier posts (and I hope this is a more accurate statement) -

Rescan Topic uses the Search Engine but is limited to searches based on the Group selected under "Pick Topic" (see #3). Rescan does not do a separate "global" search through the Search Engine in the same way the Watch List does when the Watch List Source is set to "Internet" and nothing has been added to the "Look in Groups" field (see #4).

Using Rescan Topic, where I have to pick a Group, gives me a different set of records (as shown above), then what I get from the Watch List results. There's also a big difference in the number of records I can retrieve manually and what Watch List returns (see below).

- I ran a manual search for "egg" in the Search tab and it retrieved 1,412 results with post dates starting from 8/2008 through 11/6/2015 all with Status of Idle/New.
- The 36 results added by the Watch List only go back to 11/1/2015 and run through 11/6/2015. It didn't find the same 216 records added by Rescan or the 1,412 results found by doing a manual search.

There may be a logical explanation for that large of a difference between doing a manual search and what the Watch List retrieves but I'm not seeing it at the moment. I'm guessing there is something hidden away somewhere that limits the number of results returned for a Watch List item?

My opinion only, but I think that Rescan Topic should behave in the same way, and return results, based on how the Watch List that's being scanned would. Also, if there are age or some other restrictions on how many records a Watch List will retrieve that would be good to know.


Thanks,
Bill

[Quade]

Rescan Topic uses the Search Engine but is limited to searches based on the Group selected under "Pick Topic" (see #3). Rescan does not do a separate "global" search through the Search Engine in the same way the Watch List does when the Watch List Source is set to "Internet" and nothing has been added to the "Look in Groups" field (see #4).

I mentioned that this was a bug you discovered and that it would be fixed in the next version (it's already fixed but isn't public yet).

Thanks for clarifying.

I think where we are, is that you don't like having to do a group filter to do a rescan. I agree with you which is why I fixed it.

[OldGuyWalking]

Thanks. I look forward to the update.

Bill