6.60 beta 8 still stops downloading

[RayMark]

the same still true for beta 8 as it was for beta 5 and beta 7, as I wrote before:

viewtopic.php?f=44&t=37836


Downloading just stops suddenly with cache: 0/200

I really don't know what to do:

If I use beta 3, it works fine and does not stop downloading, but it produces lots of incomplete files without marking them (with names in capital letters) as incompletes.
With beta 8, there are still suspiciously many incompletes (at least in book groups) but most of them are in capital letters. Still, not all of them - but perhaps some files cannot be recognized by NewsBin as incompletes without pars.

So, with beta 3 I have lots of garbage shown as successful downloads and have to spend hours checking each file manually, with beta 8, downloading just stops suddenly and does not continue.

I think I need to go back at least to 6.55 - is it possible after using 6.60? I think, download lists are incompatible, anything else?

[Quade]

I think I need to go back at least to 6.55 - is it possible after using 6.60? I think, download lists are incompatible, anything else?

You can switch back. 6.55 won't read your existing download, wish or failed lists though.

"0/200" tells me Newsbin is either leaking chunks or simply can't write to your disk for some reason. I'm not seeing any leaks so, I'd suspect a problem writing to disk. What happens if you "help/about"? This will normally flush the chunk cache to disk. If the cache goes back to "200/200" then the problem is assembling files. The final step during download. Click on "Open Data Folder" in the options and then look in the chunks folder. See how many chunks you have in there.

How long can you download before it stalls?

[RayMark]

I now tried to download 1400 books - no stalling.
But the very next attempt with 107 books - it stopped downloading when about 10 books were remaining. I noticed that the NewsBin stalled in the middle of downloading of couple of books, about 50%.

Anyway, Help/about worked - Newsbin suddenly was revived and finished downloading everything. After that the cache increased to 3/200 and stayed @ 3/200 with the download list empty.

As to chunks - yes, I see some chunks from my very recent downloads, just 65 files.

Trying to download those 107 files again: stalling immediately, on the very first files. Again 50% of the first 3 files downloaded. Cache 0/200.
Doing Help/About
downloaded about 30-40 files, stopped again. The cache stayed always at 0/200
Help/About - progress - 3 files downloaded only. Stalled again.
Help/About - progress: about 35 files downloaded, stalled again - this time larger files, so at 1/3
After about 10 more Help/About - all 107 files downloaded. The download list is empty, cache: 2/200.
Chunks: the same 65.
Now deleting those chunks and trying again.
Stalling on the 1st files. Deleting the chunks did not help. Number of chunks after stalling remains 0.

Ok, restarting NewsBin.
Newsbin started with those 107 books still in the download list, all downloaded in seconds without stalling. Cache after finishing: 193/200. Chunks: 0.


Perhaps the problem is with duplicated headers? In my books groups somehow I see two identical posts together all the time. It probably happened when I downloaded all the headers for those groups from another server later.
When downloading with 6.6x I always get one complete download and one incomplete. It does not seem to have anything to do with those post actually being incomplete, just because they are duplicated.

I sort of remember Newsbin stalling also on groups without such duplicated headers, but I may be wrong. I will try it on correct groups.

I think I will re-download all the headers for those groups again, this time with all four of my servers simultaneously, I had enough of those incomplete posts anyway.

[RayMark]

UPDATE:

No, my duplicated headers have nothing to do with it: NewsBin stalled while downloading much larger files not from local headers but from NewsBin Search.
So it probably should be occurring to others as well.


BTW, I re-downloaded headers for 23 e-book groups. The header download was fast, but the background processing is extremely slow. 16 hours later, still 66 remaining from the initial number of about 350.
About 300 in 15 hours or 20 per hour. Is it normal?

[Quade]

I'm trying it here. I have about 200 GZ files. It's about 40 gigabytes of data to process through.

On my machine it went through about 1/3 of the GZ's in 50 minutes.

[Quade]

Ok, took about 2 hours to import 40 gigs worth of headers. It might not be a fair comparison. My machine is pretty powerful.

I do think importing is too slow too but, it's a pretty massive amount of data.

[Quade]

Based on your comments I spent the day looking into this.

I think there's a good chance I've been able to pull better performance out. I won't set an expectation levels but the version you have isn't running as fast as it could run. I need to make sure the changes haven't broken anything. I have to make sure it's backwards compatible too.

[RayMark]

My machine is not very powerful but it was pretty decent 2 years ago: a quad-core i7-3770K CPU with 32 GB of memory and a video card with 1536 CUDA cores - in case NewsBin can use them (if not - why not???!)
Seriously, cuda cores could be used for par2 checking but perhaps for header processing too?

Anyway, I downloaded headers for 23 e-book groups and then they were processed in the background for more than a day.
But it finally ended. So now I just ran "Download latest headers" expecting some minimal header download and processing.
Turns out, almost all the headers were downloaded again almost as if from scratch - or at least it feels that way.

Actually, it seems that about 1/2 of headers were downloaded again, at least judging by the background count: yesterday (or was it 2 days ago already?) it was 355 or more. Today it was 185 and dropped to 154 while headers still were being downloaded. After all headers are downloaded, the Import folder now has 6 GB of headers in 154 files.
The two largest groups: alt.binaries.e-book and alt.binaries.ebook should have about 2 GB and 1.2 GB of headers. Maybe less, I am judging by the old version with duplicated headers and some already expired headers.

What is going on? Why again? I think the previous download of those headers was complete, not broken near the beginning - I watched the progress. And anyway, if header download breaks, an automatic retry occurs and download resumes where it left - I've seen this happen.

Is it because NewsBin cannot handle header download from 4 servers simultaneously?

What if it happens again? I want to reach the point where I can run "Download latest headers" and do it almost instantly.

It should be a whole new topic, not about stalled downloading....

[RayMark]

OK, finally the background processing reached 0 - tried "Download latest headers" again - this time, everything was ok: only newer headers downloaded, not half of the old ones as the last time.

BTW, this version of NewsBin still creates rar files using par2 blocks instead of downloading them. A typical situation: when 3 or so rar parts are left to download, instead, they are created from par2, then the whole rar is extracted and rars are deleted and, finally, the remaining 3 rar files are downloaded. So the result is the extracted files, unnecessary 3 or so rar parts and wasted time and CPU resources, because downloading is much much faster than restoring from pars.

[Quade]

1 - You know you're downloading the same headers 4 times? Most decent news servers have exactly the same headers. I'd suggest picking your best server and just downloading headers from it. You're doing 4 times as much processing with no real benefit. I mean if you want to download from all 4 you can but, it probably doesn't buy you anything but disk grinding.

BTW, this version of NewsBin still creates rar files using par2 blocks instead of downloading them.

2 - It doesn't for me so, I have no idea. It's as if you're not downloading the combined set but, are instead downloading everything individually. I could imagine triggering this if you download all the PARS first then download the rars. Newsbin will then start repairing immediately after it gets enough repair blocks. When the files are grouped, Newsbin knows they belong together so, the PARS aren't even downloaded till after the rars.

[RayMark]

beta 13 stops downloading too.

viewtopic.php?f=44&t=38175&p=209628#p209628


As I said there, I don't even try anymore above beta 3 for large files, because they always stalled.

Beta 13 is stalling on small files, so, I think, on large - too.

Previously I thought that perhaps corrupt/duplicated headers were to blame. But we saw that the same was with the internet search results.
Now I noticed it in two situations with book groups:

1. when downloading several siPDF files (they are larger than usual books, about 30 MB each - but still tiny compared to some other files)

2. when some suspicious rar files are cross-posted to various book groups (I am using a folder with 23 groups).
In this case, rar files with identical names but with different sizes are in different groups and are included into the download list multiple times (I just searched for/included all the files with the same last name of an author).
Seems to be spam - but still, NewsBin should be able to cope with it