issues with many downloads and internet search subscription

[aardvark]

Hello,

I don't usually post because I'm able to figure things out or work around issues I encounter. This was really getting on my nerves though Image link not allowed for unregistered users

I searched for a generic term and got about 600 collections I added one at a time to download to a folder named after the subject. I learnt from last time when I did a select all, download (that was a mess!).

Oke. After a while it was sitting idle with no files failed but with a lot of stuff still under download. It appeared doing something (under progress some numbers were sometimes counting (N:xxx). When I went in the folders, most was not done, unchecked pars but Newsbin refused to finish them, even though there were plenty of pars available. Some of them were only missing 1 block. I tried to force it to assemble incompletes but nothing ever finished from what was left under downloads. But, it seemed not to do much, just counting some numbers ... I left it over night. This morning, these numbers were 3-number figures but nothing had changed, nothing else downloaded (same total amount was showing in the status bar).

So, I removed all remaining downloads, bought a search subcription and now I am manually checking each folder. When I miss par files, I search for them and redownload them. 600 folders is NOT FUN. But here's the next issue.

I had downloaded all the headers for several groups. This is where my downloads were found and added from. Since I got the search subscription, I figured I wouldn't need the local files and removed them because its very big and sitting on my SSD. But I can hardly find anything I found before, even literal names for the previous subjects are NOT found! I am simply unable to find and complete them.

So basically, I downloaded I-dont-know-how-many-gigabytes I might as well just delete it all and I wasted money on a subscription.

You know, if posts are simply incomplete with not enough par files, I can live with that. But this is just crap. When I do find what I found before, I can download the missing pars and complete them. The fact that I can't find what I found in the downloaded headers simply means your index is incomplete or incorrect, and its useless.

Needless to say I'm downloading headers again so I can provide my own search ... Image link not allowed for unregistered users

[DThor]

It's a little difficult to tell exactly what happened, but when you said 'generic term' that made me wonder. Not being a smartass, but a huge majority (in terms of posted items) is spam or worse. A sub-chunk of that are posts that have been hit with a dmca takedown. If you feed a large number of generic term search results to newsbin, I'm not surprised you would have trouble with a bunch of them. This doesn't make it your fault, but a little more prudence with the search process will probably make things smoother. Remember that you're searching raw usenet headers, everything shows up. As you probably know, there's no discussion of specific content on the forum, but if I wanted several chapters of AwesomeS01, I wouldn't just enter that term and add everything, I would try one specific post, if it worked, then I would add more from the same group (based on the subject formatting).

As far as something stuck that only needs one par and there's lots available - which version are you running? How many retries do you have set? How large is your download queue? Hundreds of entries - thousands? These can make a difference.

DT

[aardvark]

Thanks for your reply.

I hear what you're saying and I realize its impossible to tell what happened. I suspect it had to do with thousands of items in the download queue, hundreds of gigabytes. But as I added each group of items to download to their own folder, I didn't expect this to be a problem.

I can't better specify the search term as I don't know what I'm looking for until I see it. As far as I can tell it doesn't really return much spam. Imagine you're looking for collections of airplane pictures made by person A, but you don't know how he names his collections except he adds "PersonA" to the title. Your only option would be to search for "PersonA".

Take downs are a real problem. However, if that was the issue (it is in some cases), I would still find the item in the search but it would download incomplete, not be stuck in the queue with 1 block missing - I think it simply wasn't downloading the par files as it got stuck on something.

I'm running 6.42: 00 62 CF 7A A5 36 (from the about).

I asume you mean the max retries before files are assembled; its at its default of 2. Should I increase this?

The thing is, I've been at it most part of the day and I'm completing most of them using headers and local search. The ones that I still can't complete were probably taken down and miss like 50% or more, not just 1 or 2 blocks.

How does your index handle takedowns? Is the entire post removed or cripple it so it won't complete? I'm asking because I couldn't find the posts in your index but they are all still there now that I've downloaded all the headers.

Thanks.

[DThor]

If a post doesn't have enough pars to fix it, it's supposed to sit in the queue. It's newsbin's way of saying 'this post needs your attention'. AFAIK if there is a crapload of posts like this sitting in the queue, then newsbin can get 'stuck' - essentially there's so many backed up posts that Newsbin is waiting for(for example, they could conceivably be in-progress uploads as far as Newsbin is concerned) that it's tracking gets backed up. I'm not 100% on that, however, or if there's a workaround.

Part of the usefulness of search is that you can fine tune your searches before committing to download. Again, it's more about using the tool the best way to be the most helpful to you. I understand your PersonA example perfectly, but as long as he's not called 'the', I rarely have trouble seeing the pearls among the dross. I don't doubt it can happen, though. Keep in mind you can search by poster in that case. That would probably wipe out all spam.

Increasing retries might help, especially if they are recent posts.

The indexing with Newsbin, also AFAIK, doesn't deal with takedowns at all. Basically, they're paying to accrue virtually all binaries that are posted to usenet and offering access to that investment to it's users for a price - if something happens after that post, it's not updated or curated. That typically involves a lot of people with eyes on and with an incentive to add to a database, or an expensive proposition. Just to clarify, not 'my' index, I'm a user, like you.

Keep in mind the index is constantly updating, too. Again, hard to say, but not having part of a post, but later you manually download headers and find them, they might have updated since you checked. I've not experienced what you've described myself, but that doesn't mean it doesn't happen. If you have specific examples of posts that you cannot find in the index search but you can definitely download via spools, it's probably worth PMing Dexter about the details. It might be the indexing having trouble with a specific group.

DT

[aardvark]

Thanks for your help, appreciate it.

I guess that was it then, too many incompletes that Newsbin was trying to complete. Takes way too much time to fix it manually ... what a mess.

Oh well .... Il'l live. Image link not allowed for unregistered users

[Quade]

If you have specific cases where you're missing one block, has pars and won't assemble, you can PM them to me.

Newsbin won't assemble unless it has 90% of the file downloaded and what looks like enough pars to repair the file. It'll do N retries first, then assemble.