newsbin.com

Hi.

when downloading headers why does the program receive data only for about 3 seconds every 2 seconds? The other thing is that when changing download folder in the new beta client it crashes (works on the old one).


cli output from nload:
### ################ .################. #######. .#####| |## Curr: 9.71 MBit/s
### ################# ################## ######## ####### .| ### Avg: 3.13 MBit/s
### #################| ################## .######### ####### ## . ### Min: 0.00 Bit/s
#### ################## |################## ########## .#######. .#### .###||. .### Max: 9.75 MBit/s
#### .################## ###################. ###########....#########|##############. .|.. #### Ttl: 467.07 MByte

so.. there's 3 seconds of downloading headers and 2 seconds of doing nothing...

tnx 4 the reply

- I can change download folder with no crashing. I assume you mean from the options? Might try deleting the "Workspace.xml" and see if that fixes it.

- There's two header download modes, background and foreground mode (in the options). Background will download headers and write them to disk very quickly while a worker threads packs them away into the header database. Foreground mode, download headers, writes them to the DB, then downloads more headers. There are upsides and downsides to each mode but, you can pick which one you prefer. Background downloads faster but, it might take awhile before the new data shows up when you look at the groups.

While it was true before that sticking the data folder on a USB drive was a bad idea. It's actually worse now. Just a heads up.

i see that when the download pauses one thread jumps to 100% cpu usage. any chance to make this multithreaded?

It's heavily multi-threaded already. If you were updating multiple groups you'd see all of your cores working. It's kind of pointless to multi-thread something that's limited by disk IO though. You might be seeing something that simply doesn't happen in windows. We do zero linux testing with Newsbin.

I run Newsbin on linux most days, I have nothing like what you're experiencing. Lose any oddball things like usb drives and see if you can repro. Admittedly I run linux on a seriously hardassed machine, but there's a tendency to blame wine for everything, but it works surprisingly well, very robust, and many times it's tracked down problems that actually existed in windows but weren't trivial to catch. Having said that, you need to run Newsbin on a decent machine, regardless of OS, with a decent IO pipe. My system runs 64 bit Newsbin, it multithreads like mad when appropriate.

Once you get that out of the way, yes, if it misbehaves in wine only, it's a wine problem.

DT

@dthor
8 core opteron
3 disks in raid0
10gb ram
filesystem: btrfs
ubuntu 11.04 x64
nbpro6 build 1019
several 100gb in nbpro download list

worked with no problems till now - just experienced a crash:
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 11cd002
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, ab4c
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 29f8cd
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
fixme:file:SetFileValidData stub: 0x18c, 1388000
fixme:file:MoveFileWithProgressW MOVEFILE_WRITE_THROUGH unimplemented
err:seh:raise_exception Exception frame is not in stack limits => unable to dispatch exception.

and the program closes...

Fairly sure the 'movethrough' comments are just standard wine bitching, calls it doesn't implement so it falls back to another option. As far as to why it's crashed, can't really tell. I'd determine just how often it crashes, if we're talking occasionally, then perhaps a wine issue. Otherwise, start stripping out the variables.

DT

Hi.

Another question... nbpro doesn't seem to be doing anything - no cpu/net/disk usage (well about every 3sec some program writes about 50k to disk). The "log" tab doesn't say anything unusual, except maybe that autopar has 6x failed. i have some files in the "download list" that aren't par/rars and i don't understand why the program doesn't download them... The par/rar files that are in the "download list" have more or less finished the "current progress" bar which is purple if that means anything... so how can i check if nbpro is doing anything?
a lot of par2 files are paused. are they ever removed from the "download list" or are they there to stay for all the eternity?

edit: so now i selected a few items in "download list" and moved them to top. they are now being downloaded. what gives?

nbin is not responding again...
fixme:file:SetFileValidData stub: 0x374, e1a1ae
err:ntdll:RtlpWaitForCriticalSection section 0xa73850 "?" wait timed out in thread 0041, blocked by 0044, retrying (60 sec)

anybody else experienced similar problems?

The problem is you're using wine and linux so, anything you observe might be related to that. There's no way for me to tell. It's one reason I don't do much trouble-shooting in linux threads because I just don't know. You're starting from a different baseline and I have no experience with Wine at all.

I've not experienced that, sorry. FWIW I'm running wine wine-1.3.12.

DT

Quade wrote:The problem is you're using wine and linux so, anything you observe might be related to that. There's no way for me to tell. It's one reason I don't do much trouble-shooting in linux threads because I just don't know. You're starting from a different baseline and I have no experience with Wine at all.

looks like an opportunity for a little research and expansion to the linux market . besides, no support for a paying customer ?

DThor wrote:I've not experienced that, sorry. FWIW I'm running wine wine-1.3.12.

DT

can you tell me a bit more about your setup? please post the outputs of the following commands:
$cat /etc/*rel*
$uname -a

To clarify - Newsbin isn't supported on Linux. The fact you can get it working on there is just a cool extra. It's up to wine to be your chauffeur.

I've run many different configs with Newsbin, mostly without trouble, I can tell you right now that kernel versions and graphics hardware/driver combinations are not your enemy. It's mostly about wine, I'd upgrade it.

I'm running SUSE 11.4 on a fairly highend workstation, I've also run that config in a SUSE VM on Win7 without those sorts of problems.

DT

DThor wrote:To clarify - Newsbin isn't supported on Linux. The fact you can get it working on there is just a cool extra. It's up to wine to be your chauffeur.

I've run many different configs with Newsbin, mostly without trouble, I can tell you right now that kernel versions and graphics hardware/driver combinations are not your enemy. It's mostly about wine, I'd upgrade it.

I'm running SUSE 11.4 on a fairly highend workstation, I've also run that config in a SUSE VM on Win7 without those sorts of problems.

DT

I've upgraded to the latest - wine-1.3.23. didn't noticed any crash or threading problems yet but one question remains: having a bunch of par files on top in the "download list", other, non par files (like pdf etc.) that come after um.. about the 1000th par file don't get downloaded unless I select them by hand and click the "move to top" option. until I get them to the top when they get normally downloaded, all I see is a few kb/s traffic and about 50kb writing to disk every 3sec. Any idea what's that all about?

Not a clue. Do you need the par files? You know that v6 basically deals with all that, right? You shouldn't have to fiddle about with the order. If the post is incomplete and recent, by default Newsbin will wait for further posts. You can push it along if you think there's enough pars by Assemble Incompletes.

DT

DThor wrote:Not a clue. Do you need the par files? You know that v6 basically deals with all that, right? You shouldn't have to fiddle about with the order. If the post is incomplete and recent, by default Newsbin will wait for further posts. You can push it along if you think there's enough pars by Assemble Incompletes.

DT

hi.

transferred all files&cfgs to win xp machine. the "to top" problem remains. files at the top just sit there and those at the bottom aren't being downloaded.
i could run quickpar on one of the files that had half full purple progress bar and just sits there. quickpar said no repair needed.. so i guess there's sth up with autopar? any suggestions?
also there are full purple progress bars that i have to unrar by hand. most of them of the format "*.pdf.part01.rar". when extracted by hand (using the unrarar/join files option) they dissapear from the download list. can't this be automatized?

I'm suspecting the very specific posts you're talking about have no pars, or no pars that are matching up with the primary file.

DT

DThor wrote:I'm suspecting the very specific posts you're talking about have no pars, or no pars that are matching up with the primary file.

DT

well, as a matter of fact i have about 700 pars, don't know about the matching though but it seems to be a lot of non matched par files in a.b.e-books.technical group then. anyhow i need that data so i'm prepared to give teamviewer access to any nbpro dev.

edit2: ran quickpar on one of the pars in the download list that have empty progress bar. Quickpar says that the're 100% complete and that no repair is needed. So why are they still paused in the download list and how do i know which one is safe to remove without running quickpar on everyone?

Just a guess, but I'm betting this is one of those posts with a hundred files, not rars, and a parset for them. I'm unsure if Newsbin could ever work with those.

How could it possibly know that the foo.lit, bar.lit have anything whatsoever to do with the par files?

DT

DThor wrote:Just a guess, but I'm betting this is one of those posts with a hundred files, not rars, and a parset for them. I'm unsure if Newsbin could ever work with those.

How could it possibly know that the foo.lit, bar.lit have anything whatsoever to do with the par files?

DT

that's pretty much the case. there are hundreds of pdfs and some pars for the entire post. how could it know? well i dunno. but quickpar seems to have no troubles so i guess there is an algorithmic way to the solution. but that addresses just the several hundred paused pars problem and not the problem where files must be moved to top to be downloaded.
since the program now runs in winxp i suppose i should move this discussion out of the linux thread.

Well, if you think about it, it can't know. It's trivial for par because by you having the par files downloaded in the same directory as the files and you run the util, it simply finds them there. You're intervening. It needs to open the par file and process it to do that. When there's just a crapload of differently named files on usenet and some par files, I'm unclear how Newsbin can know to connect them. At that point, it means doing exactly what you're doing. I don't think it's ever behaved differently, unless I'm missing something.

Quade will probably comment, he's away for most of the day.

DT

DThor wrote:Well, if you think about it, it can't know. It's trivial for par because by you having the par files downloaded in the same directory as the files and you run the util, it simply finds them there. You're intervening. It needs to open the par file and process it to do that. When there's just a crapload of differently named files on usenet and some par files, I'm unclear how Newsbin can know to connect them. At that point, it means doing exactly what you're doing. I don't think it's ever behaved differently, unless I'm missing something.

Quade will probably comment, he's away for most of the day.

DT

well.. i mean.. couldn't nbpro then just open the files instead of me? i look fw to hearing from Quade on this one.

Newsbin works with split out pars and with non-rar files. I can't really figure out what you're expecting or what you're seeing. It's not going to try to repair until you've downloaded at least 90% of the data per set though.

I just downloaded a couple of sets of what I'm pretty sure you're referring to based on your description, it downloaded fine. No stopping, no errors, it cleaned up after itself when verifying the files and got rid of the pars. I just left it alone and it worked fine. At this point, I'm suspecting you're talking about either a bad post, a recent one(still in progress) or perhaps you got impatient and started intervening and it caused some problems.

If you have a specific example of something, send a PM describing how to get it.

DT

Quade wrote:Newsbin works with split out pars and with non-rar files. I can't really figure out what you're expecting or what you're seeing. It's not going to try to repair until you've downloaded at least 90% of the data per set though.

the problem is that the progress bar of some pars is empty whereas if I run quickpar on the same par it says 100% compleation and no repair needed. also i noticed that pars that are lower on queue don't get processed, i must move them to top.

So, PM an example as requested, I can try to repro. What I was saying was, I don't see the same behaviour.

DT