lastbuild module not executed first in the CLI

[Johann-PLW]

lastbuild module not executed first in the CLI

[Johann-PLW]

@stark4n6 Thank you very much for your feedback. Adding multiple parsers at a time is now implemented.
@abrignoni Sure, it is obvious 🤪

[Johann-PLW]

The load profile file feature is now fully supported in the CLI.
Generated profile files have now the .ilprofile extension and are identified as iLEAPP profile files. Checking when it is loaded that this is a profile file for iLEAPP.
In the logs (CLI and GUI), it is indicated that a profile file has been loaded and the filename is displayed.
Info available in the Screen Output.html file.

[stark4n6]

@Johann-PLW would it make sense to be able to add multiple parsers at a time? Like separated by a comma or something? If you want to added a bunch you'd have to type a number and hit enter a lot haha. I know it could get a bit clustered

[Johann-PLW]

@abrignoni , @stark4n6
I was frustrated to have an 'Load Case Data' button that was useless in the GUI.
If you want to have a try and test it (https://github.com/Johann-PLW/iLEAPP/tree/case_data), I have implemented Load Case Data and Save Case Data in the GUI.
Input path, output path, profiles and timezone are saved in the .ilcasedata file.
Less click in the GUI.
Let me know if you are interested in having this feature (an maybe include it in other LEAPP tools).

[abrignoni]

Yes. Yes. Yes. Would be fantastic to add profile support to the CLI. As way of background the idea of running last build first was to have the iOS version available to artifacts in case version specific processing was needed the version number would be readily available. This is a feature that hasn't been used much because folks have been handling version differences in other ways. That being said I think it is good and necessary to have it. I am on board with the proposed changes.

…

On Fri, Nov 24, 2023, 4:20 PM Johann POLEWCZYK ***@***.***> wrote: When parsing a full file system extraction in the CLI, the lastBuild module is not executed first, whereas the code in ileappGUI.py is written in the way that this module must be executed before all the others. In my fork repository of iLEAPP, I've created the *cli_profiles* branch to be able to load a profile file to only execute selected modules in this profile. I didn't propose this update as a pull request because I wanted to get your opinion. However, in this update, the lastbuild module is executed first and it could easily be implemented in the actual code without profile support. The code was tested with Josh's iOS15 Public ffs image and backup, my own backup with and without a profile file loaded. As ileappGUI.py uses the *crunch_artifacts* function of ileapp.py file, I did the same tests with the GUI. I obtained the expected results. If you think this is a useful feature and would like to see it incorporated into iLEAPP, the next update will be to differentiate profile files of the different xLEAPP tools. Currently the verification process only checks if aleapp and 1 (for version) is present in the json file. We should use ileapp, rleapp and so on and maybe also distinct extensions for profile files to minimize the confusion. After that, I am also considering the possibility of generating a profile file from the command line. The list of all the available would be display on the screen. The user would enter the name(s) of desired module()s and the code would check if this module exists. Let me know if you think these are interesting improvements. — Reply to this email directly, view it on GitHub <#618>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3DPC7VV5N7JMOYOQ3ZEJDYGEFSZAVCNFSM6AAAAAA7ZSOUGOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYTAMRSHEZDSOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[abrignoni]

This is fantastic. Having profiles at the CLI enables faster testing of specific modules without having to click around in the GUI.

Thank you so much. This is fantastic.

[Johann-PLW]

If you agree, since load profile from the CLI and the .ilprofile files are fully functional, I can merge my cli_profiles branch with main and integrate these changes right away into iLEAPP by opening a pull request.
These features could be ported to other LEAPPs too.
I'll wait a bit before closing this issue in case there are any comments.

[Johann-PLW]

@abrignoni
I've just finished the implementation of creating an ilprofile file in the CLI.
The code is available in my cli_profiles branch: https://github.com/Johann-PLW/iLEAPP/tree/cli_profiles.
The -l argument is used to load a profile file and need to be used in conjonction with -t -i and -o parameters.
The -c argument is meant to be used alone, without any other arguments. You just have to provide the output_path of the ilprofile file.

In the profile file creation menu:
'l' display all the available plugins, sorted by category as in the GUI. Each plugin has a corresponding number

You can type the number to add a remove a plugin from the ilprofile file.

'p' displays the parsers added to the profile file

'q' is used to quit and save the profile file after choosing a filename. (ilprofile extension is automatically added).

This created profile file can also be used in the GUI.
I've chosen to use numbers to add or remove plugins from the profile file, rather than typing their names directly, because it's less prone to typos.
Tell me what do you think about this implementation or if you can suggest a better way to manage profile file creation.
This feature is easily removable if you only want to keep the load profile feature.

[abrignoni]

This looks fantastic. I dig it quite a lot. The main thing is being able apply a profile via the CLI. It will be awesome for automation. It will help also so much when developing artifacts. No more clicking around to just run one parser. This is really good stuff.

The edit via numbers is perfect btw.

[stark4n6]

@Johann-PLW Love this, let me know if you need me to test. This will be a huge time saver for running tests on new parsers.

[Johann-PLW]

@stark4n6
Thank you Kevin.
If you had time to test and give me your comments or suggestions, it would be great before merging to the main branch.

[abrignoni]

@Johann-PLW Just staying the obvious, all LEAPPs will benefit from profiles at the CLI. 🙏

[stark4n6]

@Johann-PLW worked like a charm, seems good to me to merge, much appreciated!

[JamesHabben]

sure would be nice if the modules had more details about them than just the filename and module name........

Nice add! @Johann-PLW I will be taking advantage of this as well in doing focused module testing.

[abrignoni]

Then load case data button allowed the user to set a case number and some other data points to be shown on the main report html. Will look at your fork shortly.

…

On Tue, Dec 5, 2023, 5:41 PM Johann POLEWCZYK ***@***.***> wrote: @abrignoni <https://github.com/abrignoni> , @stark4n6 <https://github.com/stark4n6> I was frustrated to have an 'Load Case Data' button that was useless in the GUI. If you want to have a try and test it ( https://github.com/Johann-PLW/iLEAPP/tree/case_data), I have implemented Load Case Data and Save Case Data in the GUI. Input path, output path, profiles and timezone are saved in the .ilcasedata file. Less click in the GUI. Let me know if you are interested in having this feature (an maybe include it in other LEAPP tools). — Reply to this email directly, view it on GitHub <#618 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3DPCYXAON6G3T37R7IWHTYH6PJJAVCNFSM6AAAAAA7ZSOUGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRG42DKOJVHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[abrignoni]

See the content of the zCaseDataExample.alprofile at the root of the repo. Being able to add a case number and other data points is necessary. Is that supported still? If no we need to add it again.

[Johann-PLW]

Ah OK!
In fact, I hadn't even noticed the 'zCaseDataExample.alprofile' file at the root of the repo.
So I didn't go in the right direction at all.
I finally replaced the 'Load Case Data' Button with a 'Add Case Data' Button opening a new windows.

From this window, we can load an existing case data file, checking that it is the right filetype, create case data by filling in the 'case number', 'Agency' and 'Examiner' fields, and save a case file.

If case data is existing, data is added to the index.html file.

Do you think it would be useful to add other information?
Would it be useful to add the paths (input/output) and the time zone, or could this confuse the user?

[abrignoni]

The add case data window is amazing. We need to add it for sure. If we were to add more data fields I think they should be things that are not in the interface already. This looks great.

…

On Wed, Dec 6, 2023, 11:31 AM Johann POLEWCZYK ***@***.***> wrote: Ah OK! In fact, I hadn't even noticed the 'zCaseDataExample.alprofile' file at the root of the repo. So I didn't go in the right direction at all. I finally replaced the 'Load Case Data' Button with a 'Add Case Data' Button opening a new windows. image.png (view on web) <https://github.com/abrignoni/iLEAPP/assets/27498476/74e1b95a-416e-4fc2-a573-fd134c03b2ca> From this window, we can load an existing case data file, checking that it is the right filetype, create case data by filling in the 'case number', 'Agency' and 'Examiner' fields, and save a case file. image.png (view on web) <https://github.com/abrignoni/iLEAPP/assets/27498476/c45fc83b-19c3-4092-9e7f-9df558d65977> If case data is existing, data is added to the index.html file. Do you think it would be useful to add other information? Would it be useful to add the paths (input/output) and the time zone, or could this confuse the user? — Reply to this email directly, view it on GitHub <#618 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3DPC3K5W3C2PPEMRJAU7LYICMVJAVCNFSM6AAAAAA7ZSOUGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBTGI2DQMBTGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Johann-PLW]

I agree to only add fields that are not already in the interface.

My first attempt with the "Case Data" button was to quickly get the input and output paths without having to click on the buttons to save time, but it wasn't a good idea.
I also tried to integrate them into the profile file, but this should only be used to select the desired modules.

We need to keep things simple and not confuse the user with our own needs as a developer.

So I'll stick to the current version, which lets you manage the 'Case Number', 'Agency' and 'Examiner' fields in the GUI, without having to manually edit the json file. I will have a try with the CLI too.

But perhaps these fields should appear in the upper part of the table, before Extraction location, extraction type, report directory and processing time rows.

[Johann-PLW]

Case Data is also available from the CLI (load/create a LEAPP case data file)
Also added a clear button in the case data window in the GUI
'Case Number', 'Agency' and 'Examiner' rows appear first in Index.html report if case data was loaded.

Note: A LEAP case data file can be used with any xLEAPP tool as fields are generic.

@abrignoni Maybe some suggestions about parameters chosen:
options:
-d LOAD_CASE_DATA, --load_case_data LOAD_CASE_DATA
Path to LEAPP Case Data file (.lcasedata).
-p LOAD_PROFILE, --load_profile LOAD_PROFILE
Path to iLEAPP Profile file (.ilprofile).
-cp CREATE_PROFILE, --create_profile CREATE_PROFILE
Generate an iLEAPP Profile file (.ilprofile) into the specified path.
This argument is meant to be used alone, without any other arguments.
-cd CREATE_CASE_DATA, --create_case_data CREATE_CASE_DATA
Generate a LEAPP Case Data file (.lcasedata) into the specified path.
This argument is meant to be used alone, without any other arguments.
-a, --artifact_paths Generate a text file list of artifact paths. This argument is meant to
be used alone, without any other arguments.

[abrignoni]

If I'm not mistaken the -p arguments stands for a lists of artifacts paths and names. This cannot be mapped to a different functionality since Autopsy and Paraben software depend on that - p argument for their LEAPP integrations.

[JamesHabben]

I haven't used Paraben or Autopsy integration with xLEAPP. do they rely on the table output in some way, or do they just generate the HTML report? @abrignoni

[abrignoni]

They take the paths and extract the files from unsupported extractions like EO1s or bin files. After the files are available the LEAPP runs against them and generates the html and TSVs. Paraben and Autopsy use the TSVs to populate their interfaces with the LEAPP parsed data.

…

On Sat, Dec 9, 2023, 9:19 PM James Habben ***@***.***> wrote: I haven't used Paraben or Autopsy integration with xLEAPP. do they rely on the table output in some way, or do they just generate the HTML report? @abrignoni <https://github.com/abrignoni> — Reply to this email directly, view it on GitHub <#618 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3DPC4ICEHGF3E2FFTOQ43YIUL35AVCNFSM6AAAAAA7ZSOUGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBYHAZTMMZZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Johann-PLW]

Good to know that the -p argument is used by Autopsy and Paraben.

So I decided to use only one argument for file creation. As it is used alone, you have the choice in the menu to create a profile file and/or a case data file:

I finally chose the following args:
-m LOAD_PROFILE, --load_profile LOAD_PROFILE
Path to iLEAPP Profile file (.ilprofile).
-d LOAD_CASE_DATA, --load_case_data LOAD_CASE_DATA
Path to LEAPP Case Data file (.lcasedata).
-c CREATE_PROFILE_CASEDATA, --create_profile_casedata CREATE_PROFILE_CASEDATA
Generate an iLEAPP Profile file (.ilprofile) or LEAPP Case Data
file (.lcasedata) into the specified path. This argument is
meant to be used alone, without any other arguments.
-p, --artifact_paths Generate a text file list of artifact paths. This argument is
meant to be used alone, without any other arguments.

m for profiles as we also speak about modules, d for data in case_data & c to create a profile/casedata file.

@abrignoni Please let me know if this suits you. Otherwise everything seems to work both in the GUI and the CLI.

[abrignoni]

This looks wonderful. Since the -p is there it won't break compatability. 👍

…

On Sun, Dec 10, 2023, 4:29 PM Johann POLEWCZYK ***@***.***> wrote: Good to know that the -p argument is used by Autopsy and Paraben. So I decided to use only one argument for file creation. As it is used alone, you have the choice in the menu to create a profile file and/or a case data file: image.png (view on web) <https://github.com/abrignoni/iLEAPP/assets/27498476/1fbfe237-3fc3-4183-a09d-6b034eedad8e> I finally chose the following args: -m LOAD_PROFILE, --load_profile LOAD_PROFILE Path to iLEAPP Profile file (.ilprofile). -d LOAD_CASE_DATA, --load_case_data LOAD_CASE_DATA Path to LEAPP Case Data file (.lcasedata). -c CREATE_PROFILE_CASEDATA, --create_profile_casedata CREATE_PROFILE_CASEDATA Generate an iLEAPP Profile file (.ilprofile) or LEAPP Case Data file (.lcasedata) into the specified path. This argument is meant to be used alone, without any other arguments. -p, --artifact_paths Generate a text file list of artifact paths. This argument is meant to be used alone, without any other arguments. m for profiles as we also speak about modules, d for data in case_data & c to create a profile/casedata file. @abrignoni <https://github.com/abrignoni> Please let me know if this suits you. Otherwise everything seems to work both in the GUI and the CLI. — Reply to this email directly, view it on GitHub <#618 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3DPCYKTLNQOKFOCHIHIWLYIYSTVAVCNFSM6AAAAAA7ZSOUGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGA4TANRTGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>