6.40b6 -bug -Self corruption of critical "DESCRIPT.ION" file

[driverdude]

6.40b6 -bug -Self corruption of critical "DESCRIPT.ION" file

CORRUPTION! 15,000 files lost data (subjectline, uploader, upload date) File today only held a couple hundred descriptions (it was a daylong batch test)

I deliberately use very few features of any usenet tool, disable almost every feature, and perform all my own par2, sfv, unzip, unrar and tag fix, and folder sort, and because of this.... the "DESCRIPT.ION" file generated by your amazing tool is CRITICAL to me, and as a backup plan I could finish a tool to use the critical data you optionally hide, similar to DESCRIPT.ION in Microsoft's NTFS Alternate Data Streams (ADS) , specifically the existing SummaryInformation comment field PID_COMMENTS, also known as "Comments" where I can direct your program to save usenet subject , in case DESCRIPT.ION is damaged or somehow lost. uploader name is critical too, and you preserve that. upload time would be nice too.

True, MOST users, will only enjoy NTFS ADS if booting Windows XP, unless they run 3rd party file browsing tools in Windows 7 or Vista, but the data is still vital for these situations.

In fact BOTH NTFS ADS and DESCRIPT.ION are critical to me, or I would need to go back to mac OSX to use usenet, and I have started to trust your amazing program more and more. But todays corruption was not too hard to diagnose because of your default settings for your diagnostics debug log. I have no proof of the solution, but I strongly suspect I found the issue and you could easily confirm it.

basically entries such as :

===========
[15:34:31] DEBUG - FileIOThread - AssembleFile: xxxxxxxxxxxxx- zzzzzzzzzzzzzzz- File 01 of 13 - yEnc "descript.ion" 2284 bytes
[15:34:31] DEBUG - AssembleFiles: descript.ion
[15:34:31] DEBUG - AssembleFiles: File already exists on disk: descript.ion
===============

and
"descript.ion" xxxxxxxxxxxxx- zzzzzzzzzzzzzzz- File 01 of 13 - yEnc "descript.ion" 2284 bytes , qqqqqqqqqqqq <No-one@nowhere.com>, alt.binaries.sounds.yyyyyyyyyyyyyyyyy, 3/12/2009 03:18:21

instead of preserving the critical and vital active EXISTING file called "DESCRIPT.ION" by renaming the new target as per my old setting I had selected [[[ FILENAME OPTIONS --> Duplicate Filename Settings --> Auto Rename : Rename a file if it already exists on Disk]]].... IT IGNORES THAT USER SELECTION OPTION and trashes the DESCRIPT.ION File, by renaming the incorrect file "DESCRIPT-(0001).ION"

The prior valid descript.ion, before erroneously being renamed -(0001), happily still has nearly all the data I thought lost for good, and combining the various differently named duplicate "descript-(000X).ion" files I can recreate a single large useable descript.ion, without having to run a custom tool to harvest the less similar data in the NTF ADS using custom C calls in a tool to reconstitute lost descript.ion.


==============

To test the serious bug in 6.40b6, use this in a browser :
'http://binsearch.info/?q=descript.ion"
It only shows SOME, not ALL the many evil hidden problematic files that poison newsbin functionality, but is a start to verify and fix this.

=============================
EASY TO FIX IN MINUTES?

2 possibly solutions :
1> QUIT RENAMING The EXISTING file on disk, and only rename the NEW duplicate file name about to save to disk. Besides, isn't renaming already saved files problematic to the concept of log files, database journals and even descript.ion functionality itself? That would fix the issue.

Though the above fix would work, a better one would be :
2> NEVER EVER EVER EVER EVER let ANYONE try to destroy or overwrite or rename the DESCRIPT.ION file, even as a deliberate side effect such as uploading a malicious or accidental file entitled "descript.ion" and treat the active existing file as a communally shared OS structure, temporarily in your trust and control. If you cannot trust 3rd party libraries, then at least monitor its filesize for SHRINKAGE and if file shrinks, post a prominent CRITICAL warning to user so they can be aware of the dataloss tragedy and take action.

=============================
Thank you !!! I love this amazing product. Its pretty much one of the the happiest things in my life.

[Quade]

For now, I'd probably just put a global filename and subject filter into place to prevent descipt.ion file downloads. Bizarre that people post them in the first place.

I'm thinking a better solution is "none of the above". Newsbin already has a common filename logging file that's out of band of the download folder. I was just checking it out on RC1 and it works, but not that well. I worked on it some and it'll be fully functional in the next RC. Basically, it stores the names and all the descript.ion information in the data folder instead of in the file download folder so, nothing can step on the file. Even if I was to implement what you're asking for, things like PAR files could still stomp on the descript.ion file if they were mentioned in a PAR set.

[driverdude]

Thanks for your quick reply!

eeeeks!

Don't change much. Please do not forget that some of your users have par and rar turned off and just want a single good program that can download a massive list of files from the internet. I absolutely need nothing done to interfere with merely selecting a huge list of files and having them download with trusted items associated such as uploader, subjectline, current filename, and upload date. I then run other tools after that part is safely performed.

Speaking of which, I certainly would not want descript.ion separated from the file destination, if it meant losing descript.ion the way it is. Its useful. In fact its why I use this program. It merely needs to be watched for size decreases to warn user of potential dataloss, and perhaps have option to prevent saving them to disc under that specific name, for people who have par and rar turned off. I also use the files to check if ANY files are missing that should be there, or any accidental extra files. Basically a 1:1 check. Your proposal to move it somewhere would be extra steps to triple check that it is purged and copied and repurged upon every large salvo of files to a target directory.

Pretty please, do not forget that some people just want to download files and use other fancier tools to do file repair, sfv, par2 and sorting and retagging. In fact if par is off, i wish you would not reorder all the par files to the top of a selection group. They should stay in the order selected and not move far ahead of the payload in the todo list. I understand why you do it when people have par enabled, but once again, i just want the ability to select a large list of files and download the list of files. Nothing else.

I guess outright blocking of files entitled " descript.ion " might be your best option, if you have no way to rename the incoming stream to anything else.

As for par2 or rar for other types of users, you should perhaps implement those yourself in your own code, or if using them in a library perform file actions to a target sandbox directory and filemove all results of tree contents back to destination but SKIP copying "descript.ion" back to maser destination directory or rename to noncolliding name.

====

I really wish it would also mark yenc corrupt files if upload date is 2011 or later. I suspect in 2012 Yenc can finally be trusted. Plus, marking them by filename alteration does not harm par2 uses. I want to trust that every yenc file i ever download is perfect and no internal corruption.