One Newbie's Guide to Handling Obfuscation and Flattening
Posted: Tue Mar 05, 2019 4:11 amFull congrats to Quade et al who are doing their best to battle the mindless beast that is obfuscation. Better than 90 per cent successful by my count. But it still takes a bit of effort to root out that extra one in ten, mostly 7zipped versions inside of rars. But Quade does the BIG job of still getting the correct folder name. This three-click routine also flattens the tree, bringing all the files from the sub-folders to the top level.
After Newsbin does the job it's designed to do, I'm three button clicks away from having a nice and tidy list of files to shovel into FileBot (https://www.filebot.net/).
First, I run the CleanNZBs batch file. It uses Sweep.exe (I found it at https://hydrogenaud.io/index.php/topic,36109.0.html), a utility that will repeat operations on each sub-folder from a starting point.
Now, you have to be wary of changing this batch file to handle where you have you DLs from Newsbin and where you are keeping the Sweep.exe file and where your NZBs are kept. Modify above to reflect that. Obviously, the file switches to the drive and specific folder of the downloads. It sweeps through it and all of it's sub-folders to remove par2, nzb and jpg/jpeg files. Finally, it switches over to where the NZBs are kept by Newsbin and gets rid of them. Optional of course.
The Next button goes hunting for those maddening 7z files that remain with the obfuscated name.
This time, you sweep through the sub-folders looking for 7zip files to extract. You could just as easily do unzipping or unraring if you find you have that problem by adding another line like the second one. Personally, I find I don't want any leftover rar files doing any damage accidentally if uncompressed. If they got left over by Newsbin, check your post-processing filter in AutoPAR options. My DeepSix filter does away with Rar, NFO, URL and other detritus. Some gets through, but that's because I'm watching and when an Undownloadable file makes itself evident I abort that process. Sometimes everything doesn't clean itself up, which is why the occasional Rar file gets left over. They are easily spotted post my clean up routine.
Finally, an AutoHotkey script that brings everything to the top in tip-top shape.
You will have to handle changing the script to make TheFolder point to where you do YOUR downloading. In the tenth line comparison, make sure JustDir is compared to just the last folder in your folder path. This to avoid renaming files that are already collected into your root download folder and that you haven't gotten around to processing with FileBot yet. You compile the script and run it from a shortcut and you will be happy with the moving of your files, with 100 per cent un-obfuscated names, to your root download folder. You DO HAVE TO DELETE THE EMPTY SUB-FOLDERS. I leave it to you to modify the AHK script to do so. I found it easier to do manually.
Now, I created a small test tree to work out the script and, indeed, this is not version one of the script. More like version six. But once you have it just right, you can handle future variations of Obfuscation designed to bedevil Quade, but still be accessible to manual work. I've taken the manual out of the work.
Hope this helps the community.
NOTE: My file manager lets me see a flattened view of any sub-folder, so I do check before starting my post-processing. If, as with 90 percent of the downloads, Newsbin has left you with no problems, simply use the AHK script turned compiled EXE to flatten your files. It's not a perfect flatten, but close enough for me.
After Newsbin does the job it's designed to do, I'm three button clicks away from having a nice and tidy list of files to shovel into FileBot (https://www.filebot.net/).
First, I run the CleanNZBs batch file. It uses Sweep.exe (I found it at https://hydrogenaud.io/index.php/topic,36109.0.html), a utility that will repeat operations on each sub-folder from a starting point.
- Code: Select all
C:
cd C:\DLs\NZBdls
C:\apps\Utils\Sweep.exe del *.par2
C:\apps\Utils\Sweep.exe del *.nzb
C:\apps\Utils\Sweep.exe del *.jp*
cd C:\Newsbin\NZBs\Shez
del *.nzb /Q
Now, you have to be wary of changing this batch file to handle where you have you DLs from Newsbin and where you are keeping the Sweep.exe file and where your NZBs are kept. Modify above to reflect that. Obviously, the file switches to the drive and specific folder of the downloads. It sweeps through it and all of it's sub-folders to remove par2, nzb and jpg/jpeg files. Finally, it switches over to where the NZBs are kept by Newsbin and gets rid of them. Optional of course.
The Next button goes hunting for those maddening 7z files that remain with the obfuscated name.
- Code: Select all
cd C:\DLs\NZBdls
C:\apps\Utils\Sweep.exe C:\apps\Utils\7zip\7z.exe x *.7z
This time, you sweep through the sub-folders looking for 7zip files to extract. You could just as easily do unzipping or unraring if you find you have that problem by adding another line like the second one. Personally, I find I don't want any leftover rar files doing any damage accidentally if uncompressed. If they got left over by Newsbin, check your post-processing filter in AutoPAR options. My DeepSix filter does away with Rar, NFO, URL and other detritus. Some gets through, but that's because I'm watching and when an Undownloadable file makes itself evident I abort that process. Sometimes everything doesn't clean itself up, which is why the occasional Rar file gets left over. They are easily spotted post my clean up routine.
Finally, an AutoHotkey script that brings everything to the top in tip-top shape.
- Code: Select all
#NoEnv ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn ; Recommended for catching common errors.
#SingleInstance,Force
TheFolder := "C:\DLs\NZBdls"
Loop,%TheFolder%\*.*,0,1
{
SplitPath,A_LoopFileLongPath,thisFile,thisDir,thisExt,thisName
JustDir := getDirName(thisDir)
if (JustDir <> "NZBdls")
FileMove,%A_LoopFileLongPath%,%thisDir%.%thisExt%
}
getDirName(_path)
{
return SubStr(_path,InStr(_path,"\",0,0)+1)
}
You will have to handle changing the script to make TheFolder point to where you do YOUR downloading. In the tenth line comparison, make sure JustDir is compared to just the last folder in your folder path. This to avoid renaming files that are already collected into your root download folder and that you haven't gotten around to processing with FileBot yet. You compile the script and run it from a shortcut and you will be happy with the moving of your files, with 100 per cent un-obfuscated names, to your root download folder. You DO HAVE TO DELETE THE EMPTY SUB-FOLDERS. I leave it to you to modify the AHK script to do so. I found it easier to do manually.
Now, I created a small test tree to work out the script and, indeed, this is not version one of the script. More like version six. But once you have it just right, you can handle future variations of Obfuscation designed to bedevil Quade, but still be accessible to manual work. I've taken the manual out of the work.
Hope this helps the community.
NOTE: My file manager lets me see a flattened view of any sub-folder, so I do check before starting my post-processing. If, as with 90 percent of the downloads, Newsbin has left you with no problems, simply use the AHK script turned compiled EXE to flatten your files. It's not a perfect flatten, but close enough for me.