Headers are getting deleted as soon as they download

[aesopp]

I upgraded my paid full install earlier this week to the newest version, and now for some reason, no matter what I have selected in options, all my headers aren't showing.

I deleted my entire profile folder and re-started Newsbin again to recreate the profile folder and all files, and I started downloading headers form scratch. I have selected 9999 for download age, display age, and storage age. This always worked fine for me. Now, all the headers seem to download while the group is downloading (say 2-4 million headers, for example), but when the groups are done sorting, and all the headers are counted, only a small fraction is shown. The group might show 4 million headers downloading, but the total messages shown next to the group when it's all done is under 1 million. This never used to happen. It would always show the full message count. I even re-installed an older version (6.50) and it still acts the same way. So what happened all of a sudden? Why are most of the headers disappearing as soon as they are downloaded?

Also, the cache count at the bottom seems stuck on 198/200 (0), even though the profiles Import folder is empty. Can anybody help? Would a full uninstall and re-install help?

It also started showing "restartpausetime" when the group list first downloads, and a new server is automatically created called restartpausetime, which I have deleted. I have 2 different news servers currently and this behavior affects both servers, together and separately.

[aesopp]

I've just totally uninstalled the program, ran ccleaner, and re-installed 6.50. The same thing is still happening. Total headers are almost 6 million. I select "download all headers". "New files" next to the group name is now listing just a few hundred thousand. And it's not increasing like it used to do as it adds the messages. It's already done adding messages. What's wrong all of a sudden? I used to download and display several million messages easily.

Something is clearly broken now. And it somehow affects even a re-install of an older version. I've never had an issue like this before the update. Now, most of my headers are downloaded and immediately purged. I have no idea what's going on.

[Quade]

The group might show 4 million headers downloading, but the total messages shown next to the group when it's all done is under 1 million

Might be 100% normal. Normally, it's not even that many. A typical 50 meg file is 60 some odd posts so, if you download 6 million headers and the files are averaging 50 megs each, you're only going to see

6 mil/60 = 100,000 files.

Even image groups are often 2-5 posts per image. So, you almost never will see 1:1 files to posts. That's just how usenet works. You should look down at the status bar too. If you see a number in parens, then the header import isn't finished yet. Until that number hits zero, they aren't all available to see.

My guess is you have a bunch of records backlogged to import into the database, If you tried downloading headers multiple times on the same group, you may have a bunch of duplicate records being import (Newsbin will sort it out but, it's slow).

[aesopp]

Read again what I've done.

I've totally uninstalled the software. I've deleted the appdata Newsbin folder. I've re-installed the software. I've downloaded the same exact groups again that I've done many times before. There are millions of headers in each group I download. All headers have always displayed before.

Now, they do not. Right next to the group name, is the "new files" count. It's only a fraction of the total headers downloaded. This never used to happen, If I downloaded an image group from scratch with 5 million headers, it would list 5 million new posts roughly. Now, only a small fraction is being listed there. I've done nothing different with my settings. I've triple-checked everything. Something is wrong.

Some of the groups are binary image groups - which means each post is a single image. These always used to show up as millions of "new files" - now, they are just a few hundred thousand "new files". The last time I downloaded from the same groups, millions of "new files" appeared. i have been using this program for years - I'm familiar with how it works, and how usenet works. Headers are not being stored and displayd the way it always did before for me before the last couple weeks.

That's what I'm trying to say - just weeks ago I downloaded some groups from scratch, and the total new posts that were shown, not downloaded - shown - were several million. Now, it's just showing a few hundred thousand - same groups - same settings.

[Quade]

You should look down at the status bar too. If you see a number in parens, then the header import isn't finished yet. Until that number hits zero, they aren't all available to see.

My guess is you have a bunch of records backlogged to import into the database

I think you missed the most important part. 6 million headers = 18 gigs of data that needs to be packaged up and put into the database. Image groups import slower than large file groups. Before you toss everything out again, look for that number. It's next to "Cache".

[aesopp]

I know this.

I've done this exact same thing many times before.

The only difference is now Newsbin shows me several hundred thousand "new posts" when I first download a group - "download all headers".

It used to show me several million "new posts" when I first download the same exact groups.


Up until recently:

"download all headers" - when sorting is done, millions of "new posts" show next to group name.
(example: New posts: 3,542,623)

Now:

"download all headers" - when sorting is done, several hundred thousand "new posts" show next to group name.
(example: New posts: 453,225)

[Quade]

No matter how many times you tell me the same thing, until I know what the cache status is, we're dead in the water. I'm not going to stop beating that horse till you tell me what the cache status is. It's pointless to trouble shoot this until that number goes to zero.

This is like troubleshooting why your car won't start but not knowing how much fuel is in the tank. We're not even to the point where I can guess if this is a problem or not.

- I picked a picture group.
- I "Download all headers" on it. 6 million posts (pretty tiny group).
- I watched it import for 10 minutes till cache went to zero
- Said 1.4 million files in the groups list.
- I loaded the entire group, it was 1.4 million files.

Then I did a quick survey of the group so see if any files were missing. I didn't see any missing files. They were all sequential as you'd expect.

One thing I noticed is that the data from the server was bad. I had a bunch of 1.5K files (listed) that were actually 150K. If you were running some kind of header download size filter, these files would probably have been deleted. In fact the whole group is screwed up. There are 40 meg RAR files listed as 35K. This is from Astraweb. 2000 day old posts.

1.4 million files out of 6 million posts isn't unusual. Are you actually missing files or are you pinning everything on the new file count? In my survey, I see plenty of JPG and RAR files with between 4 and 40 posts per file. That means a 1:1 post to file count won't happen.

PM me the name of the group and I'll try it on a couple servers.