bootblock / filesieve Goto Github PK
View Code? Open in Web Editor NEWThe official repository for the FileSieve application that sorts files using a variety of options.
Home Page: https://filesieve.com
The official repository for the FileSieve application that sorts files using a variety of options.
Home Page: https://filesieve.com
Add the option to the Source Item Editor to disable the encoding of any encountered %codes%
.
Even as the author, the Simulation view can be somewhat hard to understand at first glance as everything is shown in a flat, vertical listing.
Rewrite the Simulation view to optionally show folders/files using a hierarchical view.
Ensure there aren't any locations where an error can bring down the entire application.
Changing between standard and portable mode and having FileSieve automatically rearrange the settings won’t be implemented. This was enough of a nightmare with WoLEX!
Myfile
wouldn't match myfile
.Photos should contain all photo-related file extensions such as jpg/png/etc; Music should be mp3/ogg/etc.
The Directory.GetFiles() method supports a MaximumFileCount
property to limit the total number of files that are returned upon a directory scan. Expose that value to the user via the Source Item Editor window.
Being able to reference capture groups would be very useful.
After processing, show detailed information on what happened. For example, the Skipped Items stat doesn’t actually give you useful information on what was actually skipped. The Log is a pain in the arse for determining that information (if at all; had a quick look but couldn’t find anything, so don’t know if it actually says what was skipped or not!).
Give full info on what was skipped, errored, etc - and why.
Within Step 2 - Destination, allow the entering of %codes% that'll be substituted for their literal equivalents during processing.
For example, a destination of (date being the date the process was performed):
D:\Folders\Destination\{date}\
would be replaced with:
D:\Folders\Destination\2020-04-06\
Not a literal example, but that's how it would work.
Thanks to Phil P. for the idea!
The population code has been made modular in preparation for this addition.
When the Launcher performs its colour transition when a radio button at the bottom of the window is selected (Simulate, Copy, etc), the transition effect seems to "jump" part-way through.
This includes support for pasting (CTRL+V
) while the relevant control is active.
In the Launcher, selecting Copy As Command-Line in the Profiles list should insert the full path and filename of FileSieve.exe into the copy so the user doesn't have to discover it themselves.
To allow the simultaneous processing of multiple profiles and to allow the Trigger system to fire for more than one profile at a time, profile processing will need to be decoupled from the user interface.
Requires some thought, especially when it comes to showing the fact that multiple profiles are being processed at the same time.
There appear to be issues that may be a side-effect of the performance change made a while back.
When the Freeware Edition has reached the file limit, no message is shown indicating the fact; instead, the user needs to look at the Log to determine this is what happened.
In addition, it seems only one file gets processed - at least in the Simulation view; non-simulation hasn't been tested as of yet.
Add support to the Classification Method to also classify files by their content.
This was originally going to be its own plugin, but testing showed that almost everything would have to be duplicated from the Classification plugin; instead, the Content feature set has been rolled into the Classification plugin.
Currently, the full description just uses the summary text.
Allow the re-ordering of Nodes rather than just allowing Nodes to have their Parent changed.
This issue was manually imported from the old FileSieve tracker; not really sure what it's referring to. Maybe the Classification editor?
The UI for some plugin editors are a bit all-over-the-place and need cleaning up.
The UIs for regular expressions need some thought as the concept is confusing to users that have never used it before and FileSieve needs to offer as much help with them as possible. Also, regex capture groups will need to be implemented and the UI should account for that/offer as much help as possible.
Not sure if this just applies to the Checksum method, but performing a simulation run when the files already exist in the destination causes the NewItem names to be incorrect; that is, they're using their original file names rather than the new names they've been assigned.
Again, not sure if this is just the Checksum method plugin even though it does the same stuff as all the other methods.
Relates to #2.
The Tag plugin is where the issue was found; the EXIF plugin will be investigated later.
The Tag plugin doesn’t actually rename processed files, but instead just creates directories based on the entered Name Mask.
Was this the original intention? Doesn’t seem anywhere near as useful as also being able to rename the files.
For example, the original file:
This results in:
My Album\My Song.mp3\my_song.mp3
It should be:
My Album\My Song.mp3
The original filename for the file doesn’t get changed by FileSieve when it gets processed. FileSieve doesn’t seem to modify the filename at all.
That can’t be right?
Add file hashing to the %codes%
system; such as CRC, SHA-1, SHA-256, etc, including String.GetHashCode().
Example: Classification Method not finding any matches if no items are processed.
The problem appears to be related to there being items to be processed, but they have all been set to Item.IsProcessed = False
. This means that no items have been processed, which seems to cause a problem with the Windows Taskbar progress indicator.
Fixing this will have to be an ongoing process; in the meantime, EnableTaskbarProgress
has been added to Settings - Advanced and should be set to False should the issue of the FileSieve window staying disabled after processing has completed.
Within the Classification Editor: If the Files display contains no items, then the There are no files specified banner is correctly displayed. If then adding a file, the banner still stays visible. It's only hidden when the editor is opened and there are already items present; this check needs to be performed when adding or removing file items.
Changing a Modifier’s settings, Okaying the Modifier window, and reopening it shows the changed settings as expected; but restarting the application shows the original settings before the save.
Rather than just indicating what should happen to source items via the Copy Mode, plugins need to have the ultimate say in what happens to items/files during processing.
For example, the Archive Modifier compresses items into the destination location and the original file needs to either ignored (if Copy Mode is set to Copy), or deleted (if Copy Mode is set to Move).
The plugin(s) will then need to say “Okay, ignore/delete the original file but count it as being processed”.
The Delete functionality is implemented via the Item.DeletionMode
property that is set via plugins, but this more than likely needs to be rolled into a general ProcessMode
property to incorporate different types of outcomes that the plugin requires and FileSieve knows about.
Allow Start Points and End Points to be specified.
For example, setting a Start Point of HTTP will allow FileSieve to process items from an HTTP server and setting an End Point of FTP will allow FileSieve to use an FTP server as a destination.
This will probably require introducing an intermediate “language” that describes what FileSieve is trying to do and is interpreted by the respective Start/End Points into concrete actions.
This’ll allow Start/End Points of other services such as DropBox, Google Drive, Amazon S3, etc.
Definitely requires some planning.
The Overwrite Rules Editor may be a bit too complex for new users to understand, especially if all they want to do is tell FileSieve that it should rename files (current default behaviour) or just ignore already-existing files.
Sort files based on Smallest/Largest/Etc.
This will be a FileSieve v4 update of the v3 plugin, with additional features.
Allow the archiving of processed files.
As per the title.
If a detailed-enough log is kept of a full “processing”, then maybe that’ll allow the possibility of “undoing” a processing after-the-fact?
Would need proper planning and checks in place as it could get somewhat messy if anything unexpected happens.
This won’t be as easy as implementing a traditional plugin as it will possibly require core changes that touches lots of areas.
This plugin might not be appropriate for FileSieve.
Certain files seem to be ignored by either the engine scanner or somewhere else upstream.
I can’t seem to reproduce this with initial testing.
This also makes me think the Review Settings section should have a Debug/Verbose mode that contains additional information for cases such as these.
2 BIN
files, 2 IND
files, 2 BDM
files, 1 XML
file and 2 files with no
extension do not get copied even though .*
is set and and minimum filesize is 0
Thanks to Olivier C. for reporting
Threaded pre-scanning is partially in-place, but an effective thread initialiser is halting things. It needs to be intelligent so that it can determine the correct number of threads to be used in all cases, from number of files to if files reside on the same or different physical media.
This has the potential for big speed gains (potentially; a huge performance gain was obtained via defaulting to Get Files Mode to Burst
from Yield
).
The EXIFLib library that FileSieve uses hasn't been updated in years and refuses to work with TIFF files even though it's supposed to, so it's probably best that a replacement library is found and integrated instead.
Just like the destination path %codes%
implementation (#48), allow source items that have %codes%
in their path. If a source item path has a %
defined, then don't warn the user that it doesn't exist when processing as %codes%
suggests that the item may or may not exist at any moment.
If possible, extend this to Modifiers. Custom Modifers scripts were originally intended, but I can’t seem to remember why that’s a no-go.
Pressing Escape
in any non-code editor should give focus to the code editor. This is personally very important to me and why I hate most code editors.
ProcessItem
() returns a new object once it has finished processing (one for each file being processed), but the contents only differ in its Result
property - and there are only - currently - three different values.
Instead of creating a new object each time, pre-create all three objects (to each represent OK
, Skipped
, Error
) and instead just pass through back to the calling method.
To make the Enterprise Edition more suited for an enterprise environment, add support for storing settings/profiles/etc on a remote server that FileSieve can sync to that data.
When FileSieve is performing the initial scanning of the Source Items' files that should be processed, it should display something that assures the user that things haven't gotten "stuck".
This became especially required with the changes made to the Classification method as it can perform an enormous amount of work.
The Simulation window only shows the physical effects of processing but not metadata-style information.
Due to various changes to the core of FileSieve, the initial file scanning time can take longer than prior versions; because of this, the Pause
and Stop
buttons should work during the scanning. I thought it did, but apparently, that doesn't look like the case.
The internal regular expression engine in .NET uses the dollar symbol for referencing captures groups; change FileSieve so it uses that character and invokes the .NET Regex.Replace() method instead of using custom code.
Moving files to destination mirrors would probably be more useful than just copying them, but this will require significant core changes that it’ll take a while to implement.
Need to tread carefully as a lot of stuff could get broken.
Redesign the Simulation window so it presents results in a format that is analogous to a file browser (Windows Explorer, DOpus, etc).
In the past, I've been frustrated with the view showing results in a flat-list format as it requires some thought as to how that translates to an on-disk folder and file structure.
Instead, create a TreeView on the left containing the folder structure and create a ListView on the right which contains the files that are within the selected folder from the left.
Or add a checkbox to the Source Item advanced settings that say “Do not perform a pre-scan” as it’s not actually mutually-exclusive with the current Get Files Mode.
This needs to be thought out better from a code point-of-view.
Leaving for now.
Add a method that'll read a database file (json, text - whatever) and checksum files against it; any matches will then name those files to the names as specified within the database.
Kinda like how ROM renaming works.
This report has been created by the AppFramework framework on which the FileSieve application has been built and has been submitted with the user's express permission.
mscorlib
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089FileSieve4
, Version=4.57.0.0System.Windows.Forms
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System.Drawing
, Version=4.0.0.0, PublicKeyToken=b03f5f7f11d50a3aBiQubic.Framework
, Version=2.0.0.0FileSieve4.Plugins
, Version=4.57.0.0Microsoft.VisualBasic
, Version=10.0.0.0, PublicKeyToken=b03f5f7f11d50a3aSystem.Core
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System.Web
, Version=4.0.0.0, PublicKeyToken=b03f5f7f11d50a3aSystem.Xml
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System.Configuration
, Version=4.0.0.0, PublicKeyToken=b03f5f7f11d50a3aAccessibility
, Version=4.0.0.0, PublicKeyToken=b03f5f7f11d50a3aSystem.Management
, Version=4.0.0.0, PublicKeyToken=b03f5f7f11d50a3aNewtonsoft.Json
, Version=12.0.0.0, PublicKeyToken=30ad4fe6b2a6aeedSystem.Numerics
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System.Runtime.Serialization
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089System.Data
, Version=4.0.0.0, PublicKeyToken=b77a5c561934e089Method.Attributes
, Version=1.7.0.0Method.BatchedAtoZ
, Version=1.0.0.0Method.Checksum
, Version=1.0.0.0Method.Classification
, Version=1.10.0.0Method.Consolidate
, Version=1.5.0.0Method.DateStamp
, Version=1.8.0.0Method.Delimiter
, Version=1.0.0.0Method.Exif
, Version=1.9.0.0Method.Extension
, Version=1.5.0.0Method.FindReplace
, Version=1.8.0.0Method.Mirror
, Version=1.5.0.0Method.ParentRename
, Version=1.1.0.0Method.Quantity
, Version=1.6.0.0Method.StandardAtoZ
, Version=1.7.0.0Method.Substring
, Version=1.4.0.0Method.Tag
, Version=1.14.0.0Modifier.Attributes
, Version=1.7.0.0Modifier.Casing
, Version=1.6.0.0Modifier.DateStamp
, Version=1.2.0.0Modifier.FindReplace
, Version=1.8.0.0Plugin.Registered
, Version=4.54.0.10ExifLib
, Version=1.7.0.0MantisSharp
, Version=1.0.0.0Profiles.Count
= [1] '1
'Profile.Name
= [15] 'Default Profile
'Profile.ActiveMethod.Name
= [9] 'DateStamp
'Profile.ActiveMethod.NameExtended
= [53] 'DateStamp (Name Mask: %CREATED\_YEAR%-%CREATED\_MONTH%)
'Profile.ActiveModifiers.Count
= [1] '0
'Profile.AllowedModes
= [20] 'Simulate, Copy, Move
'Profile.CompleteAction
= [4] 'None
'Profile.CopyMode
= [4] 'Move
'Profile.EnsureSourceItemsExist
= [4] 'True
'Profile.OverwriteNoMatchAction
= [6] 'Ignore
'Profile.OverwriteOptions.HashAlgorithm
= [4] 'Sha1
'Profile.OverwriteOptions.MetadataFields
= [8] 'FileName
'Profile.OverwriteOptions.ComparisonMethod
= [8] 'Metadata
'Profile.OverwriteRules.Count
= [1] '1
'Profile.OverwriteRules[0].Action
= [15] 'RenameUsingMask
'Profile.OverwriteRules[0].Field
= [24] 'NewPathFilenameExtension
'Profile.OverwriteRules[0].IsEnabled
= [4] 'True
'Profile.OverwriteRules[0].RegEx
= [2] '.\*
'Profile.OverwriteRules[0].RenameMask
= [38] '%file\_name% (%file\_number%).%file\_ext%
'Profile.SourceItems[0].Name
= [13] 'DigitalCamera
'Profile.SourceItems[0].DeleteEmptySubFolders
= [5] 'False
'Profile.SourceItems[0].EnsureExists
= [4] 'Auto
'Profile.SourceItems[0].GetFilesMode
= [5] 'Burst
'Profile.SourceItems[0].IgnoreFilter
= [3] 'All
'Profile.SourceItems[0].Items.Count
= [1] '0
'Profile.SourceItems[0].MaximumFilesize
= [2] '-1
'Profile.SourceItems[0].MinimumFilesize
= [1] '0
'Profile.SourceItems[0].RegExOptions
= [22] 'IgnoreCase, Singleline
'Profile.SourceItems[0].RegexPatterns.Count
= [1] '1
'Profile.SourceItems[0].RulesType
= [9] 'Immediate
'Profile.SourceItems[0].SearchFilter
= [3] 'All
'Profile.SourceItems[0].SearchOption
= [14] 'AllDirectories
'Profile.SourceItems[0].SearchPatternFileItems
= [2] '.\*
'Profile.Triggers.Count
= [1] '0
'Profile.ShowInWindowsExplorer
= [4] 'True
'Profile.WasUpgraded
= [5] 'False
'Information
FileSieve v4.57.0.0Information
OS: Microsoft Windows 10 Pro 6.2.9200.0 (Win32NT; x64). CPUs: 12.Information
Memory: 53.47GB of 63.91GB free.Information
UI Culture: English (United States)Debug
Exception handler has been initialised to use MantisBT.Debug
Found plugins: Method.Attributes.dll, Method.BatchedAtoZ.dll, Method.Checksum.dll, Method.Classification.dll, Method.Consolidate.dll, Method.DateStamp.dll, Method.Delimiter.dll, Method.Exif.dll, Method.Extension.dll, Method.FindReplace.dll, Method.Mirror.dll, Method.ParentRename.dll, Method.Quantity.dll, Method.StandardAtoZ.dll, Method.Substring.dll, Method.Tag.dll, Modifier.Attributes.dll, Modifier.Casing.dll, Modifier.DateStamp.dll, Modifier.FindReplace.dll, Plugin.Registered.dllVerbose
Available Triggers: DateTime, Directory Watcher, User IdleVerbose
Attributes initialised OK.Verbose
Batched A-Z initialised OK.Verbose
Checksum initialised OK.Verbose
Classification initialised OK.Verbose
Consolidate initialised OK.Verbose
DateStamp initialised OK.Verbose
Delimiter initialised OK.Verbose
EXIF initialised OK.Verbose
Extension initialised OK.Verbose
Find Replace initialised OK.Verbose
Mirror initialised OK.Verbose
Parent Rename initialised OK.Verbose
Quantity initialised OK.Verbose
Standard A-Z initialised OK.Verbose
Substring initialised OK.Verbose
Tag initialised OK.Verbose
Attributes initialised OK.Verbose
Casing initialised OK.Verbose
DateStamp initialised OK.Verbose
Find Replace initialised OK.Verbose
Registered initialised OK.Information
Profile: Default ProfileInformation
Mode: MoveInformation
Method: DateStampInformation
Source Items: 1Information
Destination: \nas1\photoInformation
DestinationEncoded: \nas1\photoInformation
Overwrite Options: Metadata: FileNameInformation
Overwrite Rules: NewPathFilenameExtension -> RenameUsingMaskInformation
Overwrite Fallback: IgnoreInformation
Preparing to process items.Debug
Told Windows that it shouldn't shutdown while we're actively processing a Profile.Debug
Before Processing StatisticsDebug
Threads: 17Debug
Base Priority: 8Debug
Priority Class: NormalDebug
User CPU Time: 00:00:01.4218750Debug
Total CPU Time: 00:00:03.3593750Debug
Paged System Memory Size: 660 KBDebug
Non-paged System Memory Size: 43 KBDebug
Paged Memory Size: 60 MBDebug
Peak Paged Memory Size: 61 MBDebug
Working Set: 88 MBDebug
Peak Working Set: 88 MBDebug
Private Memory Size: 60 MBDebug
Virtual Memory Size: 4.78 GBDebug
Peak Virtual Memory Size: 4.78 GBInformation
Pre-scan Time: 68.2410133 seconds.Information
Found 110,682 items for processing.Information
Processing items...Information
The Processing Engine has been set to Stream mode by the DateStamp method.Debug
Created directory '\nas1\photo\2020-11'Debug
Created directory '\nas1\photo\2020-10'Debug
Created directory '\nas1\photo\2020-09'Debug
Created directory '\nas1\photo\2020-08'Debug
Created directory '\nas1\photo\2020-07'Debug
Created directory '\nas1\photo\2020-06'Debug
Created directory '\nas1\photo\2020-04'Debug
Created directory '\nas1\photo\2020-03'Debug
Created directory '\nas1\photo\2020-02'Debug
Created directory '\nas1\photo\2020-01'Debug
Created directory '\nas1\photo\2019-12'Debug
Created directory '\nas1\photo\2019-10'Debug
Created directory '\nas1\photo\2019-09'Debug
Created directory '\nas1\photo\2019-08'Debug
Created directory '\nas1\photo\2019-07'Debug
Created directory '\nas1\photo\2019-06'Debug
Created directory '\nas1\photo\2019-05'Debug
Created directory '\nas1\photo\2019-04'Debug
Created directory '\nas1\photo\2019-03'Debug
Created directory '\nas1\photo\2019-02'Debug
Created directory '\nas1\photo\2019-01'Debug
Created directory '\nas1\photo\2018-12'Debug
Created directory '\nas1\photo\2018-11'Debug
Created directory '\nas1\photo\2018-08'Debug
Created directory '\nas1\photo\2018-07'Debug
Created directory '\nas1\photo\2018-06'Debug
Created directory '\nas1\photo\2018-03'Debug
Created directory '\nas1\photo\2018-02'Debug
Created directory '\nas1\photo\2018-01'Debug
Created directory '\nas1\photo\2017-12'Debug
Created directory '\nas1\photo\2017-11'Debug
Created directory '\nas1\photo\2017-10'Debug
Created directory '\nas1\photo\2017-08'Debug
Created directory '\nas1\photo\2012-01'Debug
Created directory '\nas1\photo\2009-01'Fatal
A domain exception has been caught: sourceFileName does not exist.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.