still not seeing headers as I expect!

[beany]

Could anyone let me know when the last headers were posted to a.b.multimedia group please?

[Quade]

There's pretty much a continuous feed.

30 gigs worth in the last hour alone.

[beany]

Well what I've done is not working then.....

I've spent the last few days downloading headers by firstly selecting download all and then periodically updating with the download the latest group updates (groups list screen red button).So I am expecting ALL the headers to be up to date.

Therefore when I double click a group I am expecting a new tab to open up with a list of all the headers for the displayed age (60 days default). Is that correct? On many groups I am not seeing anything even when I click to load earlier headers.

This is the sort of behavior I was experiencing when I first posted my question about possible gaps in headers. I've followed what you've suggested I do and tried a few things of my own but the bottom line is I ain't getting what I expect. So it appears not to be working as it should. I cant think of another explanation.

[Quade]

What so you see in the status line? "Cache Y/Z (XXX)" with numbers for X means they're not all imported yet.

[DThor]

Double click only loads as much as Options/Display Age is set to, as you say. This is unrelated to how many posts you may or may not have on disk. Set that higher, or right click Show All Posts to override that setting. Note this can also be set per group by right clicking/Properties, which overrides the global setting.

Lots of people doing this all the time, there are no 'gaps', I have no idea why you seem to be having problems, or perceive that. Right click and Show All Posts. Disable(meaning - turn OFF using the Power symbol a couple of buttons to the left of the search field) filters. You will see everything you have on disk. If you haven't allocated enough time for Newsbin to insert new header downloads, you won't see those yet.

DT

[beany]

What so you see in the status line? "Cache Y/Z (XXX)" with numbers for X means they're not all imported yet.

the a.b.multimedia group disappears from the download queue within seconds. 0/0. This would suggest to me that all headers "should" be downloaded.

[beany]

Lots of people doing this all the time, there are no 'gaps', I have no idea why you seem to be having problems, or perceive that.
DT

I spent nearly a week trying to understand why it isn't working. I would say its slightly more than a perception DT Image link not allowed for unregistered users <wink>

I may well be doing something wrong or have some wrong settings but its not apparent to me at this time.

The display all from disk has been running an hour or so and the blue status bar across the bottom is stuck at 3/4. I will leave until tomorrow if it don't see it make further progress I will do a 0-300 days and see what comes up then.

[beany]

What so you see in the status line? "Cache Y/Z (XXX)" with numbers for X means they're not all imported yet.

the a.b.multimedia group disappears from the download queue within seconds. 0/0. This would suggest to me that all headers "should" be downloaded yes?.

Sorry Quade I misunderstood at first. The figures in the bottom of the whole window for cache are 85/100(15081).
As I said if I update a.b.multimedia then it appears only briefly on the downloading files window.

[DThor]

It means you have a crapload of headers that need to be injected. If you try and load headers, or do any other heavy IO on your computer, things will thrash like mad and you'll be watching a pot until it boils. You need to leave it to process until that last number in brackets get to zero. At this point you might have so seriously backlogged things it might be some time, hard to tell. Newsbin will process them in order, it won't try to do them all at once, but other heavy IO will impact heavily. This is related to your hard drive speed and how much you've been pounding on the header downloads. Just the fact you didn't know your disk was grinding like mad suggests you don't have your IO light wired up on your computer, or you don't pay attention to it. Some people are like that, me, I'm a lot more obsessive-compulsive about knowing what my computer is up to.

DT

[Quade]

I'd suggest Restarting Newsbin and just leaving it alone till that number goes to zero.

If you keep downloading headers, it'll never catch up. I've never seen a PC get this far behind. Mine doesn't start falling behing till I'm well over 100 Mbps on the header downloads.

[beany]

Just the fact you didn't know your disk was grinding like mad suggests you don't have your IO light wired up on your computer, or you don't pay attention to it. Some people are like that, me, I'm a lot more obsessive-compulsive about knowing what my computer is up to.

DT

OMG its sooo spooky how well you know me DT. You must be psychic. !!!!!! (sort of a bit pissed about your tone but I'll let it go as I guess your trying to help me) : - )

you're right I don't pay attention to my disk light but only because this pc houses 4 seperate disk drives and acts as uPnP to most of the family's renderers. So a light in the front really doesn't signify a lot in that case when it's always being accesses from somewhere! Also its in the bottom of the rack so I cant hear if the disk is 'thrashing' or just constantly being accesses normally.

Being a little more scientific (even if not fully aware of what exactly I'm looking for/seeing) I have been running the resource monitor throughout. I have noted the cpu is quite low 30-40% (quad core) and that the Disk monitor is showing 2-8 MB/s with highest active time from 20-90% with the occasional 100% peek. what is a little more alarming is that I'm seeing higher than usual page faults into the 100's/s. Usually only 10s of.

non of the above would really point me in the direction of understanding what/how NB is behaving. All I see is that I have been asking it to do some stuff it is supposed to do (albeit very new to me as for past 12 months I've only ever used the internet search) but I'm not seeing what I expect. I don't know how long it takes for NB to process the headers. I see NB sitting there with nothing on the screen or in any of the queues and then I try to look at a group and I get nadda displayed. After trying to understand it myself I then post here.

I had asked in a previous post if NB should be run on another computer and was told (by Quade I think) that most modern PCs could hack NB running in background whilst I go off and Game!!!

You may be right and it's thrashing the drive or itself silly but how can I tell that from the application and how would I know what I did to cause it. I have just been following what I have been told to do here.

Now as I write (having closed the group tab which hopefully stopped the display all from disk) the cache is saying 85/100(15068).

Maybe something a little more constructive could be to tell me what I need to do to get this to process faster or maybe even to stop it altogether if its likely to be thrashing the drive. Could it be that I should reload the group headers one group at the time and wait for the cache to go 100/100 before started another group?

Don't get me wrong...I really appreciate the help from you and others on here I just thought you a little abrasive in your response to an actual physical problem seen by a relative noob. (see my no. posts to yours 38:5728 !)

Thnx

[beany]

I'd suggest Restarting Newsbin and just leaving it alone till that number goes to zero.

If you keep downloading headers, it'll never catch up. I've never seen a PC get this far behind. Mine doesn't start falling behing till I'm well over 100 Mbps on the header downloads.

Thanks Quade. Now that was constructive - you posted whilst I was still writing the last post.
My link is 100Mbps and its been running for days on the headers. Could this be the problem that they cam down faster than able to be processed? if so would it be advisable for me to throttle download speed when bringing down a large group like this?

[Quade]

I'd advise you to hold off downloading more headers until these headers are processed. I'm going to guess that you have a shitload of duplicate headers from downloading headers over and over again.

The dups don't really matter. Newsbin will sort then out but, it might be awhile before they're all fed in. You probably have a 30 gig backlog of headers waiting to be processed.

[DThor]

Yeah, there's nothing to make it faster, just needs catching up. On my computer, when newsbin is running and there's (a relatively modest) 200 groups to inject, my IO is solid. Not the occasional or constant 'blip blip' you tend to get on my media server when someone is watching something, but ON. Like I said, I'm OC about system resource monitoring. I'm not clear why your monitor suggests such a span of activity, assuming you're tracking it while newsbin is running, it should pretty well be maxed out. The page faults strike me as troublesome, I wonder if it's relevant?

DT

[saintsinner]

I will also side with the others, you'll not be able to process new headers until newsbin processes the old ones. Even if the new ones are different groups, etc. If you open the resource monitor and choose disk, then sort by newsbin only you'll see what files (headers) newsbin is reading/writing, you'll have to wait until it finishes before it will process new ones.

Case in point, I told it to download all of a.b.erotica, it finished doing so several hours ago, since then I've told it to download all of the a.b.multimedia and a.b.hdtv, both have finished (and respective files are NOT being accessed by newsbin),however newsbin will not let me view the new files in a.b.mm or a.b.hdtv. Once it's finally finished with A.b.erotica (at current pace, it looks like 4 or so hours to go) it will then tell me I have new posts in ABMM and ABHDTV. I got impatient last time I tried this (last night) so I deleted ab.erotica temporarily, and then all my tv headers instantly appeared. so it's in fact working, just be patient. Or you know, don't download all headers from a busy group anytime near when you might want to look at OTHER headers.

It's (more than a little) annoying, but it's the way Newsbin works.

[Quade]

In 6.50, import will be prioritized to import records for the currently visible tab before the rest. Code's mostly written.

[beany]

Thanks all for your input on this. Bottom line is I didn't know how it worked. I got to know how once i'd broken it. ; - )
NB is not an easy app to get to know if only used infrequently or as I have been doing with lightweight internet searches. I'm looking forward to more powerful usage now I'm using local headers.

Anyways...all is good now. The bracketed cache values are gone and I can see all the headers I expected. Yes I HAD royally screwed it up with my various trial and error learning exploits but now I'm older, wiser and a lot grayer !!!

btw what does cache 85/100 mean exactly (it is still at those values)? I read in the wiki that chunk cache had been disabled in 5.5 or something so I'm guessing its not that.

cheers all

[Quade]

It tells me you're running 15 connections to the server. When the chunk of a file downloads, the chunk will take up one of those slots. If the disk slows down, you might see all 100 get used. It's just a buffer for data Newsbin hasn't managed to write to disk yet.

I have mine set to 2000 but, I have 64 gigs of RAM so, I have room for it.

[beany]

It tells me you're running 15 connections to the server. When the chunk of a file downloads, the chunk will take up one of those slots. If the disk slows down, you might see all 100 get used. It's just a buffer for data Newsbin hasn't managed to write to disk yet.

I have mine set to 2000 but, I have 64 gigs of RAM so, I have room for it.

thnx

[saintsinner]

In 6.50, import will be prioritized to import records for the currently visible tab before the rest. Code's mostly written.

Woot! that will be very much appreciated

I have mine set to 2000 but, I have 64 gigs of RAM so, I have room for it.

Where can you change this? the only settings I see that seem to have any effect are the server connections, but that just changes the X part of X/Y? I would like to raise mine up a tad if it will in any way increase my header sorting time (as at the current rate of dropping my 85/100 (7905) is going to take another 34 hours, this is less than ideal obviously

[Quade]

It only impacts download. I have some code changes in 6.50 I hope to speed import but, it's a vast amount of data to go through.

[beany]

HI, I have all the headers downloaded now but I realize that this is not the pc to be doing such things on. So I want to move the headers to another box. Is it as simple as copying the group folders from one pc to another? or do I need to be aware of other things? Also I'd really like to keep the indication of which files have already been downloaded. What files do I need to transfer across to keep this? thnx

[Quade]

I'd move the entire data folder over. The tracker of downloaded status is tiny compared to header data so, moving it over is pretty much a freebie.

On the help section of the web page there's a section on moving the data folder. That's what I'd look at.

[beany]

I'd move the entire data folder over. The tracker of downloaded status is tiny compared to header data so, moving it over is pretty much a freebie.

On the help section of the web page there's a section on moving the data folder. That's what I'd look at.

thanks Quade

[beany]

As a matter of interest.... what is the ratio of raw downloaded header data to the processed database size? I've been downoading a.b.boneless for over 5 days now and its still only up to >500 days. The a.b.boneless db3 data file is around 40GB but my router is reporting that I have downlowded nearly 1/2TB to get this far - thats like 10:1. Not much other downloading has been occurring at the same time.
Does this all sound right? Should it really take this long and need so much raw download?. BTW its on a brand new i7 machine.
thnx

[Quade]

It's a function of your network speed, not your PC. You're talking about downloading 4-6 billion posts. At 200-250 bytes each, you can do the math. 10:1 compression isn't unreasonable. I'd actually expect it to be somewhat higher.

Not many people download boneless. I don't.

[beany]

Not many people download boneless. I don't.

Yes I can see why ; - ) but I've started now so may as well let it run. thnx