Updating headers problem

[mimauk]

Ver6 Beta5.

I've noticed this happening over the last couple of days.

I boot my PC up of a morning and start NBPro and hit the 'Update' button - all the groups update. I'm subscribed to 54 groups, some unsorted and some in GOG folders. I go through the downloaded headers and sort out what I want to download and download them. When it's finished all the headers get marked 'old' by hitting the delete button or permanently deleted with shift+delete.

When I do a new 'Update' by hitting the button a few hours later and watch in the 'Download' tab, all the groups appear and then disappear one by one but without any of the progress bars colouring up except for the last few which are some of the larger groups. When I then right click each group/GOG and click on 'show posts' - there are no new posts showing on some of the 'groups' individual tabs and I know that there should be new posts - this doesn't happen with all the groups/GOG. I can shut the group tab down and redo the show posts and sometimes extra headers will appear.

I then go to the 'Groups' tab and left click and hit 'download latest' for the individual groups/GOGs - the headers download and when I left click and hit 'show post' all the headers for the 'Display Age' limit appear - in this case 2 days - and not the new headers since the last update a few hours ago. I did have the Display Age set to 6 days and all the headers for 6 days used to appear again.

It doesn't happen with all the groups/GOGs but only with a few of them.

[mho]

Ver6 Beta5.

When I then right click each group/GOG and click on 'show posts' - there are no new posts showing on some of the 'groups' individual tabs and I know that there should be new posts - this doesn't happen with all the groups/GOG. I can shut the group tab down and redo the show posts and sometimes extra headers will appear.

I think this is similar to what I'm seeing. In v6, headers are downloaded quickly, but the actual database updates take time and continues in the background. Until this is done (easily seen if you watch the log at level "Debug" ("DEBUG NRF...")), not everything works quite as expected.

In my case, if I open a new group/GoG, filters won't work until the deferred database updates have completed; a previously loaded group seems to work better. I'm not sure if this behaviour is new to b5, or just wasn't as noticeable, before.

- mho

[mimauk]

Last night I deleted everything in all the groups using shift+delete in case something was corrupted in the data bases - two of the groups would not delete 60+ older headers so I ended up deleting the two groups and adding them again.

When I booted up this morning the same problem happened - had to update some groups individually.

When I did an update a couple of hours later, I lowered the server connections to one from twenty and all the groups updated. It looks like it might be a speed/ processing problem on my machine. I'll check it a couple of times again today.

[itimpi]

As I understand it, the change in internal design in NB6 means that there is significantly more background processing to do after the headers have been downloaded. This is around handling all the downloaded headers being processed into database updates into .db3 files to later support faster loading/searching.

This raises, I guess, the question of whether there should be more visibility at the UI of the fact that this type of background activity is going on. I would think that perhaps having something visible on the Status bar might be a good idea. However any decisions on whether this is a good idea (and how hard it would be to implement) rest with Quade at the end of the day.

[DThor]

I've gone back and forth with this. Certainly my first reaction is that if Newsbin is still working on something, I'd like feedback, please. OTOH, in *theory* what it's working on is just needed for fast loading in the future - it still works fine if your spool hasn't been injected yet, just slower. Plus, one could argue the disk activity light is your feedback.

For that to work, though, I think that apart from speed Newsbin needs to respond to user requests the same whether the injection has happened or not. I admit that a part of me would still like to see something telling me Newsbin is actively doing something. Not as in your face as the download list, though.

DT

[mimauk]

I've updated a couple of times today with just the one connection to the server and the groups update ok.

Open each group up and the headers are there straight away - I'm not waiting for things to happen in the background or anything before I open the group, yet the headers are there.

With more than one connection something happens with the updates and some groups don't show headers.

[richy99]

its possible that with more than 1 connection you are updating faster than the records can be inserted

[mimauk]

Some more info.

I tried updating with 2 connections last night and again some of the groups did not update. When I went back to one connection and updated again, all the headers in some groups had to be downloaded again. It looks like an attempt was made to write to the group download file but nothing was written and the original file must have been dumped or some thing as all the headers were downloaded.

[richy99]

look in the spool folder to see if there are any rv4 files there, if there are then they havent been inserted into the db3

[mimauk]

In the Spool_VI folder in Newsbin Download/Data are all the newsgroup folders which contain other numbered folders which again contain rv files - all of which are dated various months in 2009.

The only file with todays date is a desktop.ini file.

[mho]

In the Spool_VI folder in Newsbin Download/Data are all the newsgroup folders which contain other numbered folders which again contain rv files - all of which are dated various months in 2009.

That is a v5 directory; the new one is called SPOOL_V6.
Perhaps it is in a different directory? Try "Open Data Folder" (Options->Options->Setup) and see if you end up somewhere else.

- mho

[mimauk]

Found the Spool_6 folder using the options link- thanks for that. It was in another folder called Application Data which Vista renames Roaming when looking in Windows Explorer.

I'd already decided to try and clean things up by purging all stored posts in the Groups drop down menu and I was able to do an update using 20 connections - only one group had to be manually updated.

As there seems to be so many different folders in different places I'm going to do a complete new install and start from scratch.