Better STIX Parsing

[misp]

Better STIX Parsing

[ag-michael]

@adulau it would be great if you can merge it. But I didn't know about the MISP feature for STIX import, do you know why MISP-Taxii-Server isn't using that? That's the only way to feed MISP with TAXII that I've found, can you point me in the direction of the docs that show how to import STIX directly into MISP

The STIX section here: https://pymisp.readthedocs.io/en/latest/tools.html uses pymisp.tools which in turn uses this project.

Edit:

I figured out the upload_stix() api, it works, but it doesn't parse confidence,title,information source,etc... I don't want to waste any effort, so can you tell me if MISP-Taxii-Server is maintained (have a pending PR there too), and if it is, I'd like to create a PR/FR to have additional metadata parsed by MISP, but in the meanwhile, it would be great if you can review the existing PR for this project.

[alatif113]

@FloatingGhost I added sample code but the formatting is not cooperating. Can you help me fix it please?

[iglocska]

Sounds like some very sane ideas!

[FloatingGhost]

o christ on a quadbike that's a lot of edited code

I might take a look when I'm feeling more masochistic than usual

[alatif113]

@FloatingGhost lol most of it is just the parsing of the different observable types taken directly from the existing code.

[FloatingGhost]

just

JUST

Nothing with STIX is ever "JUST"

It's always rooted to the hellish floor of the so-called standard and is never as simple as it seems. Inevitably there'll be a tendril of this eldritch horror that extends beyond the realm of human comprehension and into STIX world, in which it terminates in the STIXPackage of unimaginable terror

[iglocska]

@alatif113, could you do a pull request for the changes? It does sound like a clean approach indeed.

[alatif113]

@iglocska I'm not very git savvy. Don't really know how to do that.

[iglocska]

Should be simple enough, simply fork the project by clicking the "Fork" button on top, this will create a copy of the repository under your user name on github, so https://github.com/alatif113/MISP-STIX-Converter

You already have the code-base sitting locally on your machine, which is what you have modified. Create a commit with all of your changes by simply doing the following:

git add /path/to/your/changed/file

Repeat it for all of the files that you have modified (to see a list of all files that you've changed just type git status from within the MISP-STIX-Converter directory)

Once you are done it's time to commit the changes:

git commit -m "My STIX parsing improvements"

Once done, add your own github repository as a remote

git remote add myfork https://github.com/alatif113/MISP-STIX-Converter.git

Then push your committed changes to your fork:

git push myfork master

Once this is done, just go to

https://github.com/alatif113/MISP-STIX-Converter

and open up a pull request by clicking the "New pull request" button (upper left side, next to the branch name)

This should be it!

[alatif113]

@iglocska Ahh I see what you mean. I didn't edit any files directly (I didnt need the MISP to STIX part), but rather created my own 2 files (in the code above) solely for parsing and importing STIX to MISP, heavily using code that already existed within the project to parse the actual observable types.

It's just a proof of concept and is missing trivial things such as error checking and logging.

[iglocska]

Ah ok, I see. Any chance you could move that to MISP-STIX-Converter and integrate it directly? Or is it too different from how the converter works?

[alatif113]

@iglocska I think replacing the buildEvent function within the buildMISPAttribute file with the build_event function within the stix_to_misp.py file above should do it, barring the fact there would now be a lot of unused old functions.

There would also need to be some agreement on the taxonomy for tags. I use Confidence:<value> and TTP:<value> for mine, but I don't know if there is already some standard that exists out there.

Unfortunately I don't have the time to actually go through and do that (not at the moment at least). Just wanted to bring something to the dev's attention with sample code I utilized for my use case.

[iglocska]

Thanks a lot for the input, we'll keep this issue open until we can get around to implementing it. It indeed looks very promising!

[alatif113]

@iglocska No problem! Got the idea from how many of the SIEMs and commercial threat platforms parse STIX files.

[FloatingGhost]

Well if they parsed MISP everyone's life would be a lot easier :<

STIX needs to die.

[iglocska]

It looks like it's here to stay though, so we should make sure that the parser makes as much sense as possible - we'll definitely take a look at this too at some point, @FloatingGhost, to preserve some of your sanity ;))

[ag-michael]

Any updates on this? @iglocska , it seems you approve of the general idea and there is demand for this feature. Any chance of accepting PR's related to this soon?

[ag-michael]

@alatif113 I have a Pending PR #40 that addresses some of what you're wanting, care to take a look and comment? This is an important subject for me as well.

[adulau]

So we don't really maintain this as there is a full-blown STIX 1.x and 2.x import/export in MISP.

As the original maintainer is not maintaining this external package anymore, I can merge those. Just let me know if it works for you and I'll merge it.