Stop Newsbin Assembling Incompletes, How to?

[Maxx]

I'm sorry if this is obvious, or if it's here somewhere, I have searched as best I can but either get nothing or pages and pages about other queries to do with assembling incompletes.

I want to STOP newsbin assembling incompletes at all, in fact I would have thought this should be the default.

I have tried setting "Max retries before .." to 0 but this doesn't work. In fact, when I go back to check it has not set it to 0 at all but is set at 1.
I could set this to 999 which would likely "delay" the assembly but dont really want to have it retrying all the time and again it would still seem to assemble an incomplete file without me knowing as I probably won't be watching at the time.

Again, sorry if it's obvious or in here somewhere.

M.

[DThor]

Newsbin's designed to 'get me my stuff' first and foremost. Stalling in the queue isn't the first choice.

To get some context, you have an incomplete post...does it have pars posted with it? What would you like it to do, exactly? Download but don't try to assemble or fix? Or just don't download anything at all? Do you have autopar enabled, and do you have Agressive Assembly enabled in the autopar options?

DT

[Quade]

Not knowing what version you're using, I don't actually know how much of this applies.

Are you trying to prevent repair? Most people want it to assemble incompletes if a repair of the files is likely to succeed. That's why it does what it does. The default is retry twice, then if PARS exist and 90% of the file set is present, assemble them and attempt the repair.


Setting the retries high should disable assembling incompletes.

[SETTINGS]
AutoAssembleTimer = 100000

Will disable time based assembly too. Basically files older than 48 hours (by default) are assumed to never complete. Setting this value large will basically disable the check.

[Maxx]

Sorry, I should have been way more precise in my query.

The files themselves are small files, typically 5Mb MP3s, there are no associated PARs as there rarely is with these type of posts.

Newsbin is 6.40, build 2059

I do have Autopar enabled but Aggressive Assembly mode is unchecked.

I'm pretty happy with how it deals with .rars

The issue with the MP3s is you don't know the resulting file is incomplete until you play it and notice the jumps and skips. If you knew it was incomplete you could try and get from another source or as in my case, query this with the host (giganews).

I'll try the AutoAssembleTimer option.

M.

[Quade]

Do they look incomplete? I mean do you have red or black blocks in the download progress bar? I don't think Giganews has recovered from the NZBMatrix shutdown and subsequent massive load from header downloaders. I find that I can download mostly 100% by combining giga and astra together but, I doubt I could have successful downloads with just one server at this point.

[Maxx]

Firstly, adding the [SETTINGS] AutoAssembleTimer = 100000 worked, my incompletes are not being assembled.

However ..

I was getting a bit confused here, seems to me though that what you might want for .rar's in not the same as you might want for single files such as MP3s.

For instance, obviously, if you download a rar set of which any of the segments are incomplete then you would want those assembled, automatically, and then newsbin will fetch the associated .par2 file and fix, if it can't 'fix' it, it will report as much.

However, if we take the case of MP3s. If we download a set of MP3s and there is either no par2 file associated with it (common) or you don't explicitly download it yourself then you would likely NOT want them to be assembled incomplete as you will have a track with skips and jumps and you would not know about it (unless you elected NOT to assemble incompletes).

If you don't assemble automatically, then you can then download the .par2 if available, or delete the downloads and look for those tracks elsewhere.

The reason I haven't come across this before is that I never realised newsbin would assemble 'broken' mp3s (not a big user) but I now know why some of my MP3s skip.

Plus, the reason it is such a BIG problem is that virtually ALL 2010 MP3s on giganews are incomplete.

I'm not sure what the answer is? I thought maybe indicating an MP3 was incomplete by maybe adding something to the filename (suffix etc.) but this would then stop par from working. Quite surprised this has not been raised before, or has it, as I said, my search returned too many unrelated results to be useful.

Malcolm

[Quade]

For instance, obviously, if you download a rar set of which any of the segments are incomplete then you would want those assembled, automatically, and then newsbin will fetch the associated .par2 file and fix, if it can't 'fix' it, it will report as much.

In my experience most posted files have PAR files. Even if they aren't combined with the files, they're still there. You just need to download them along with the non-rar files. These days if a file set doesn't have PARS, I look elsewhere before I attempt download. I'd suggest you might want to consider the same rule. Search for files and always prefer the ones with pars. I give you the tools to prevent autoassembly but, that's not what most people want.

It's still not clear to me if the files are really incomplete or not. During download you should be able to see they're incomplete. By default, Newsbin will upcase a filename for a file that assembled from incomplete blocks.

[Maxx]

In my experience most posted files have PAR files. Even if they aren't combined with the files, they're still there. You just need to download them along with the non-rar files. These days if a file set doesn't have PARS, I look elsewhere before I attempt download. I'd suggest you might want to consider the same rule. Search for files and always prefer the ones with pars. I give you the tools to prevent autoassembly but, that's not what most people want.

Thanks for the swift response, very much appreciated. My problem in truth is more with giganews. It seems that pretty much any binary file on there from around mid 2010 is incomplete, and that goes for .pars where they exist. I only use mid 2010 as an example as the files I originally wanted were posted then. I have since tried other files from between May 2010 and Dec 2010 and 95% are incomplete. I queried this with giganews, sent screengrabs of download screen, newsbin logs and created nzbs of all incompletes so they would have the message ID. Did this on 8 files from different months, different artists (they were mp3s) and they still say that they had been removed because of a DMCA takedown request.

I don't buy that as often 95% of the file is there, just one chunk missing, also earlier files (tried a few from 2009) work fine PLUS I have since gone into other newsgroups for pictures, videos, sound effects etc. etc. and have the same results, all files I have tried from around 07/2010 are incomplete.

It's still not clear to me if the files are really incomplete or not. During download you should be able to see they're incomplete. By default, Newsbin will upcase a filename for a file that assembled from incomplete blocks.

I can see they are incomplete as they are downloading but am on a 60Mb line so they disappear quite quick, plus of course I don't tend to look at it, plus ALL seem incomplete (red blocks).

I didn't know about the uppercase thing, is this the whole filename? or just the first letter?

If the first letter, on checking I can't see any MP3s that aren't listed in the actual newsgroup in lower case (unless numbered)
If fully then it doesn't seem to be working for me as I have dozen of incompletes but they are named exactly as the completes are. I do think this is a good idea though.

As I said, it has never occurred to me before as I have never come across such widespread data incompletes.

M.

[itimpi]

[quote="MaxxIf the first letter, on checking I can't see any MP3s that aren't listed in the actual newsgroup in lower case (unless numbered)[/quote]
It is the whole filename. They therefore tend to stand out.

[Maxx]

It is the whole filename. They therefore tend to stand out.

Ah, ok, mine doesn't seem to do that, is it an option somewhere? (I am currently on 6.40, build 2059)

TIA

Malcolm

[driverdude]

I want to STOP newsbin assembling incompletes at all, in fact I would have thought this should be the default.

Me TOO ! Absolutely.

side note about your original gripe root cause :
Keep in mind sometimes lots of "incomplete" damage is giganews punishing you for too much xfer from one ip address in one time period, changing your ip address (unplugging cable modem, rerolling MAC address with a MAC spoof in router or tool, and getting fresh new IP from a new lease when cable box turned on) will help with giganews. It will all work in that new header fetch. they also throttle your data and mess with it if the account is created from new giganews account with new credit card, and in first few hours, and also on occasions, if your monthly billing credit card is hung stalled, not going through for many days and you are looking like a freeloader. when that happened in astraweb, they just let it ride a couple weeks, and cut you off clean. giganews seemed to resort to odd things in the past.
secondly using a fill server account from astraweb is CRITICAL with mp3s on giganews.

=============

This (no corrupted mp3s) is my ultimate desire in accepting newsbin as my preferred download tool. I love the product and most of its design, but in prior years, my whole usenet existence with other tools, I have never ever ever downloaded and kept corrupted mp3 files. ever. not one file. I also honored and used Yenc with other now unused tools.

I agree with newsbin developers that if pars exist they should be utilized as part of download logic though allowing potentially repairable files to somehow be stored on disc. Using par2 , at least manually for me is part of usenet experience nowadays, even it not always part of mp3 world.

So far, ensuring setting the settings .ini text file up as :

[AutoRAR]
AggressiveAssembly=0
AutoDecode=0
AutoDelete=0
AutoFolder=0
AutoScratch=0
DisableAutoPAR=1

and using a fille server with giganews (astraweb) <<< VITAL and only 11 dollars a month

and leaving it as :
RetryBeforeAssemble=2

and also using :
AutoAssembleTimer=100000
(27 hours?)

seems to typically make me quite happy usually with actions of downloading.

I use my own specialized tools to manually sort the stuff i download later, compute and sort sfv checksummed files, compute and sort par2 clusters, repair par2, migrate unrepairable par2, unrar, etc.

I just want a simple usenet DOWNLOAD tool that supports at least one fill server and also tags each file with a ntfs datastream payload, (decript.ion not alone sufficient especially with some current renamed file bugs in newsbin stages, including users on rare occasions uploading decript.ion files themselves)

So far all is happy but I never update the tool often because I fear the writers of newsbin do not test the product often in modes with autoar disabled and auto unrar disabled, or care as much about mp3s as some users.

mp3 collecters suffer from 90 percent of mp3s being encoded WITHOUT internal crcchecks on frames despite part of original mp3 specs, and also pc users (cant blame mac users) from spreading corrupted mp3 files back up to usenet, but creating NEW corrupted mp3 files by usenet download errors from tools that keep damaged files without sorting, marking, flagging, or documenting them well as damaged... including not honoring Yenc in 2012 and 2013 uploads, when yenc can be trusted typically.

My life is mostly a very happy one with this tool, but fear of corrupted mp3 downloads for downloads lacking sfv, md5 , par2, and also lacking internal crc frame checks, is my horror and pain. My paranoia is so bad because I have been shocked too many times in my life to hear a corrupt or SEEMINGLY corrupt file play that I have to forensically validate, that I was driver to do two more things.... store all my files in two collections "mp3s" and "possibly corrupted mp3s''. Pathetic, i know. Bot there are no corrupt mps in my collection in the non corrupted collection.

A second sad result, is that I have to manually keep and scan the diagnostics log from newsbin to look for errors indicating bad yenc error failures, but the main programmer made me happy to provide a log mechanism to extend the log life duration across relaunches, though not fun to scan for yenc damage and i would LOVE LOVE LOVE if it still downloaded 90% complete files that have a valid par2 association of some form proven for future manual par2, but also add a new field to the descript.ion field or even better the NTFS Alternate data steam ted to the file marking it "suspect". Some users would probably love the file name suffixed with (-suspect-), as in "01-barking dog sfx march 1924.mp3" being renamed on disk as "01-barking dog sfx march 1924(-suspect-).mp3"

That is a third option for people who never want to download damaged incomplete mp3 files or even yenc corrupt files, but I would prefer ntfs ADS first (easiest I assume), descript.ion second alternate choice, and filename renaming with suffix (-suspect-) as third option.

Then upgrading new versions of newsbin will be less traumatic and less upsetting to switch to for file collectors that try very very hard to never ever allow a single damaged mp3 file to pollute their world.

All the various "aggressive" and "pause" and "retry" and possible par2 treatment, and such as welcome and flexible options and useful and cherished, but having ability to know which files are suspected as corrupt or KNOWN to have failed download due to missing parts, or bad yenc, is very very important to archivers that use their own tools for repair and analysis and just want to desire newsbin to do one thing : download large lists of files into one large directory with par2s and sfv to be handled in a later manual stage using other tools.

I mention all the above because I have seen the newsbin authors claim that they only really care about newsbin being written to be used with par2 and autorar enabled. EEEEEEKKS. That developer stance makes me understandably feel threatened with the way I will only ever use various builds of the product, and that is as a simple mp3 usenet file downloader that can mark files with ntfs ads database fields, and additionally use backup fill servers to combat the giganews 2010 fiasco. (its not giganews fault probably, and has a lot to do with half hearted takedown compliance steps I think) but I am not knowledgeable in this root matter, just suspecting.

A lot of people (myself too), just use the extremely famous "SABnzbd" for nzb of large assets, and have it do the par2, and unrar, and password spam sniff, even though most usenet download tools handle nzb files obviously. So the developers need to know that the people like me that have autopar off and autounrar off and file folder sort off are people that are not clueless newbies to mistreat, or not heed, but rather sophisticated fanatics who despise file corruption, and are well aware of the use of nzb files and nzb search tools for larger assets.

TLDR summary ': I wish newsbin had a way to mark suspected (known damaged) files by filename suffix or by metadata tied to files, especially including Yenc errors, which are indeed common in mp3 uploads. Yenc is not heeded in newsbin yet for past philosophical reasons, probably tangentially related to concept and emergence of advent of par2 for larger file assets. But Yenc is vital to heed for mp3s lacking par2 or sfv.


ITS AN ASTOUNDINGLY GREAT PROGRAM I LOVE THOROUGHLY, ALL THE SAME. thanks. you guys rock.

[BZee]

[SETTINGS]
AutoAssembleTimer = 100000

Will disable time based assembly too. Basically files older than 48 hours (by default) are assumed to never complete. Setting this value large will basically disable the check.

I added this to my .nbi file. Is there a way to add a comment in the .nbi so later I'll know it was my addition and why. For now I just made a text file with a similar filename as the .nbi.
As with driverdude, I don't want incomplete mp3s being assembled if there is no .par files (frequent). I did catch the use of upper case filenames and deleted the bad mp3s. I don't use descript.ion files with mp3s (I listen, not collect). I do use descript.ion files with .jpgs.

[Quade]

I'll add a one stop option to prevent assembly of parless files.

[nbwul62]

Newsbin is busy trying to repair
it will take a while (in total it is over 30GB of files)
still running in the background now

where can I find the stop button?

[09:53:37] ERROR AutoPARPlugin: Repair Failed: xxxxxbncxj.part002.rar PAR Repair: ProcessEx - Repair failed processing data blocks
[10:10:54] ERROR AutoPARPlugin: Repair Failed: xxxxxbncxj.part003.rar PAR Repair: ProcessEx - Repair failed processing data blocks
[10:21:27] ERROR AutoPARPlugin: Repair Failed: xxxxxbncxj.part004.rar PAR Repair: ProcessEx - Repair failed processing data blocks
[10:34:06] ERROR AutoPARPlugin: Repair Failed: xxxxxbncxj.part005.rar PAR Repair: ProcessEx - Repair failed processing data blocks

That aside, it is a bit of confusing thing, let's take part002.rar as an example.
- when opening the .nzb file, some of the set were marked as incomplete, but not 002 or 003 or any of the above
- as repaired failed, i gather that the set is useless and tfore tend to delete it, period.

as said, then again, how to stop this repair-process?

=
ps it is running about 2,5 hrs now checking/repairing, whatever, high disc activity, without me being able to stop it
and me being unable to work with my pc, every hit takes a minute or so..
=

[Quade]

1 - A repair shouldn't make your PC unusable. On my machine, I don't really even notice when repairs are processing. So, it might be worth exploring why your machine gets crushed during repair. I'm going to guess virus scanner because they seem to be the source of most bad behavior like this. Might want to make any V scanner ignore the download and data folder then, scan the download folder after download completes.

[09:53:37] ERROR AutoPARPlugin: Repair Failed: xxxxxbncxj.part002.rar PAR Repair: ProcessEx - Repair failed processing data blocks

2 - I've never actually seen this error before during repair so, I'm going to point to #1 again. Something seems to be screwing with file operations. You need to have plenty of disk space for this operation. If the download is 30 gigs. You really need 90 gigs free to be 100% sure everything will work and you won't run short of space.

That aside, it is a bit of confusing thing, let's take part002.rar as an example.

3 - The error message isn't saying there's anything wrong with the file when it downloaded. It's saying Newsbin is trying to scan it for PAR blocks and the scan failed. Just because it's complete in the NZB, that doesn't means the download was complete. The NZB just describes how to download. The server determines how much you can actually download though. An NZB with incomplete files in it is probably a bad NZB. They should always be pretty close to perfect. Most of the time the problem with files is that the server doesn't have the chunks so, even a perfect NZB can result in partial files.

4 - There currently isn't any way to abort a repair. I can look into it I guess. Normally this hasn't come up. If Newsbin thinks it can repair, the repair should work 100% of the time. It won't even start until it has enough blocks to actually succeed. Is this an AMD machine?

[nbwul62]

Thanks for the feedback. I donot know how to have Norton I.S. 2014 not scan my download folder. I checked the NIS log files and newsbin showed up twice only. Heavy disk/write entries.

A long shot, maybe something to do with NB Performance settings and I might improve those?
=

=

I really would not know.

The files were unpacked etc. at 12:14 hrs - below the log file
=

=

The total nzb was about 7.8mb, when opened it consisted of 576 .rar files and some 50 repair files (31gb)

You would not believe me, but I just re-opened it - to verify the number of 50mb files (576).
The moment I wanted to close NB something happened and I had a tremendous lot of disc activity.
I could not do anything at all, cud not bring the task manager, even not after two three minutes.
I tried ctrl-alt-del to get the taskmgr (and check the cause).
A black screen I got instead.
and
"the logon process was unable to display security and logon options when ctrl+alt+delete was pressed"
it disappeared, black screen and after a while I decided to reboot as the disc activity continued.

I can't tell what happened.

The Windows event log did not show any peculiar things, only error reported
"Time-out (30000 seconden) tijdens het wachten op het verbinden van deze service: Windows Error Reporting Service."

diskspace: sufficient (500GB).

Maybe I'll check it out again later.

=

[larrybud1320]

In my experience most posted files have PAR files. Even if they aren't combined with the files, they're still there. You just need to download them along with the non-rar files. These days if a file set doesn't have PARS, I look elsewhere before I attempt download. I'd suggest you might want to consider the same rule. Search for files and always prefer the ones with pars. I give you the tools to prevent autoassembly but, that's not what most people want.

Thanks for the swift response, very much appreciated. My problem in truth is more with giganews. It seems that pretty much any binary file on there from around mid 2010 is incomplete, and that goes for .pars where they exist. I only use mid 2010 as an example as the files I originally wanted were posted then. I have since tried other files from between May 2010 and Dec 2010 and 95% are incomplete. I queried this with giganews, sent screengrabs of download screen, newsbin logs and created nzbs of all incompletes so they would have the message ID. Did this on 8 files from different months, different artists (they were mp3s) and they still say that they had been removed because of a DMCA takedown request.

I don't buy that as often 95% of the file is there, just one chunk missing, also earlier files (tried a few from 2009) work fine PLUS I have since gone into other newsgroups for pictures, videos, sound effects etc. etc. and have the same results, all files I have tried from around 07/2010 are incomplete.

It's still not clear to me if the files are really incomplete or not. During download you should be able to see they're incomplete. By default, Newsbin will upcase a filename for a file that assembled from incomplete blocks.

I can see they are incomplete as they are downloading but am on a 60Mb line so they disappear quite quick, plus of course I don't tend to look at it, plus ALL seem incomplete (red blocks).

I didn't know about the uppercase thing, is this the whole filename? or just the first letter?

If the first letter, on checking I can't see any MP3s that aren't listed in the actual newsgroup in lower case (unless numbered)
If fully then it doesn't seem to be working for me as I have dozen of incompletes but they are named exactly as the completes are. I do think this is a good idea though.

As I said, it has never occurred to me before as I have never come across such widespread data incompletes.

M.

Check any files that end up with the filename in all-caps for skips...and I have a solution, but it requires opening a second account on a service without the file corruption problem.

The start and end dates for the file corruption on Giganews are 10/1/2009 and 4/1/2012 (April Fool's Day). Most binary files between those dates are so corrupted that even a generous .par2 posting is overwhelmed by the number of corrupted blocks in the download. The headers show these files as complete, but there are multiple missing segments in the download.

My solution was to add a Newshosting account (their prices are very competitive right now, retention matches Giganews and their completion rate is also very good. I am currently running the Giganews server as top priority, Newshosting as priority 2, and no headers downloaded from Newshosting. Newsbin does a nice job of picking up the corrupted/missing segments from Newshosting.

I was going to shut down the Giganews account completely, but that would force me to download a set of headers from Newshosting and I would lose any records of which files I'd already downloaded. I cut back to a cheaper plan and may go to their cheapest plan just to avoid downloading a new set of headers.

I also sent a full documentation of the problem, with .nzb files prior to, during, and after the start and end dates. Got the same BS answer, "DCMA

[dexter]

You will not have to re-download all your headers if you switch news servers. Just set your "Download Age" to the number of days since you last downloaded headers from the old server and the new server will pick up from there.

Headers you downloaded from the old server will still apply for the new server.