newsbin.com

I have Newsbin set to automatically pull headers at midnight every night. Since updating to 6.50 beta 9 I have discovered a weird issue. Hopefully I can describe it correctly below.

I have 2 servers setup, Giganews and Astraweb. I use Giganews as my main and Astraweb was configured as a fill server before updating. After updating I set the priority on Astraweb to a lower number than Giganews. If I'm understanding things correctly this should set Giganews as the primary and use Astraweb if an item is not found on Giganews. After doing this, when the header download kicks off at midnight, the astraweb headers just sit there. They do not download. If I switch the priority back to 10 then they download but Newsbin also starts using Astraweb on my downloads instead of as a fill server. This is a problem for me since I pay for Astraweb on a download basis since it is just a fill server for me. I also assume this is happening because I have items in my DL queue and Newsbin is assuming those have priority over the headers. Since my download queue is usually pretty full this prevents me from ever getting headers from Astraweb.

How do I fix this or is this a bug?

Yeah bug. I already fixed it for B10.

Thanks for the report.

Thanks for the fast reply! Any word on when b10 will be out?

Dex is out of town. We'll probably start sampling when he gets back. Probably early next week.

You know, you don't HAVE to download headers from both servers in order for Newsbin to use both servers right? Unless AW has a group that Giga doesn't carry, just downloading headers from Giga should work just fine for both servers. I just download headers from AW but, about 50% of my downloads come off Giga.

If you have completion problems, Incomplete files or black blocks, then you might want to hit both servers for headers, otherwise one server's headers can typically work on any other server.

I don't know if this is related but:

I have 4 servers defined in 6.50B12 (build 2874):
1. active - master server - prio 0
2. active - slave server - prio 1
3. active - slave server - prio 2
4. disabled - slave server - prio 3

At first startup headers are downloaded from 1. Also other queued files are downloaded.
But under some unidentified circumstances the download stops.
Then I see the new update for the headers queued, and also other downloads are stopped.

The only way to resolve this, is to ENABLE server 4 for a moment, e.g. a minute. However this server is not used at all for resuming the download after enabling. And downloading continues even after disabling server 4 again, until the next update of headers is scheduled.

Weird...

Ge2rt wrote:I don't know if this is related but:

I have 4 servers defined in 6.50B12 (build 2874):
1. active - master server - prio 0
2. active - slave server - prio 1
3. active - slave server - prio 2
4. disabled - slave server - prio 3

At first startup headers are downloaded from 1. Also other queued files are downloaded.
But under some unidentified circumstances the download stops.
Then I see the new update for the headers queued, and also other downloads are stopped.

The only way to resolve this, is to ENABLE server 4 for a moment, e.g. a minute. However this server is not used at all for resuming the download after enabling. And downloading continues even after disabling server 4 again, until the next update of headers is scheduled.

Weird...

Not sure if it is relevant - but server priorities should be in the range 1 to 10 (as available from the drop down list when configuring).

itimpi wrote:Not sure if it is relevant - but server priorities should be in the range 1 to 10 (as available from the drop down list when configuring).

Thanks for the tip.
I guess it's still on prio 0, because I didn't change the priority for that server after the 6.50 beta-update.

I will try prio 1-4 later.

Using prio 1 to 4 doesn't help.

I'm still in the situation that downloading of headers and files stops after a variable number of minutes.
Shortly enabling server 4 (now prio 4, earlier prio 3), triggers Newsbin to continue downloading, but server 4 is not used because server 3 can provide the posts. Until it stops again.

I've experienced this behaviour at least 10 times today.

In the server options for each server, "really advanced" check "Use Body" then restart. Let me know if that fixes it for you.

Using the 'Use BODY command' (Really Advanced) for all four servers didn't help.

BTW.
I've used this four-server combination for 1 - 2 years, activating server 4 only if needed (because it's a paid one).
Server 4 is correctly configurated. I just used it for a minute, disabling server 1 to 3.
Server 4 is the only one with SSL enabled.
6.50B12 is the first beta of 6.50 I installed, so I don't know if this behaviour is from B1 or later.

If you look down at the cache line when it stops, what do the numbers say?

Quade wrote:If you look down at the cache line when it stops, what do the numbers say?

What do you mean exactly with the 'cache line'? The lines from the Logging tab?

Assuming you mean the logging tab: I have 430's from servers 1 and 2, followed by '222 0' from server 3. The last line is also a '222 0'.

Look all the way at the bottom of the screen. Should see (XXX/YYY) numbers.

430's are "post not here". If that's the case and the posts are there from server #4 then this might be normal.

Ow, that cache line

It says: 52 / 200

Now, after short enabling server 4 and after waiting for the download to stop: 172 / 200

And the next time: 109 / 200

If it stalls and it's a zero "(0/200)" that tells me something. It tells me that downloaded chunks aren't getting written to disk. In this case, it doesn't seem to be happening.

Server #3 seems to be the issue. Every 430 should bump the priority up so, the next server is tried.

430 from server 1 -> check server 2
430 from server 2 -> check server 3
430 from server 3 -> check server 4

This is with your setup. If you have multiple servers at the same priority they all get checked before moving to the next prioority. It's not designed for you to be enabling and disabling servers on the fly. You're supposed to keep the servers enabled and let it just work its priority magic so, I'm not sure if what you're reporting is to be expected or not. I'm not saying disabling the servers will break it, just that Newsbin assumes it has all servers that you want to download from enabled during the download.

If one of the servers is really slow, say server #3 to return results, the download can stall because it's not going to try #4 till #3 fails.

What I read is that turning #4 on and leaving it one works so, that suggests that the downloads are failing on the other 3 servers.

FWIW, I've seen similar behavior with the last couple betas. I'll have stuff in the download list happily downloading then the header downloads get added (I DL headers every hour, around 74 groups) and the file downloads stop until the header downloads are almost all completed. It used to keep downloading the files when the header downloads were happening. My cache is nowhere near 0/200 when this happens. It is almost as if there isn't a file download in the top 8 download slots (see my setup below to see why I suspect 8 ) in the download list then the files don't download. My setup is:

Server 1 : Master Priority 5, 1 connection
Server 2 : Slave Priority 1, 4 connections
Server 3 : Slave Priority 1, 4 connections

Server 1 and Server 3 are the same physical server, they just have different names. I set it up this way (1 header download connection) so the header downloads didn't swamp things on my old computer. I have a much faster computer now, but I still use this setup (too lazy to test if I need to keep the header download connections at 1).

I'm running the 64 bit version on Win7 pro 64 bit.

Here is a test I just did that reproduces it

Download List is empty
cache is at 192/200
add 6 50MB files to DL list
File downloads start and run normally
Download Latest Group Updates (74 group header downloads added at top of DL list) when first file is about 1/2 way done
cache is now 138/200
File downloads stop and header downloads continue normally
file downloads resume once the number of header downloads lines in DL list gets small

This happens every time there are files downloading and the header downloads happen. I usually run with the Speed Limit active via the scheduler, but this happens at full download speeds with the scheduler off also.

I'll have to try to reproduce it. The slaves should continue downloading but, only when the cache remains > 0. If the cache goes to zero because of disk IO, the slaves will sit there waiting for chunks to free up.