Giter Site home page Giter Site logo

bluequartzsoftware / simplnx Goto Github PK

View Code? Open in Web Editor NEW
6.0 1.0 9.0 147.61 MB

The backend algorithms and framework associated with DREAM3DNX, a data analysis program for materials science data analytics

Home Page: http://www.dream3d.io/

License: Other

CMake 1.85% C++ 92.81% Python 4.62% Batchfile 0.08% Shell 0.36% Jupyter Notebook 0.28% Makefile 0.01%
analysis cpp data-analysis filter material-science materials-informatics microstructure multimodal-data 3d-reconstruction hacktoberfest

simplnx's Introduction

simplnx

windows linux macos clang-format PR's Welcome All Contributors

Multilanguage README

Language English PlaceHolder

Introduction

DREAM3D-NX is a user-friendly and versatile desktop application that leverages the open-source ‘simplnx’ software library to enable users to manipulate, analyze, and visualize multidimensional, multimodal data with ease. With its advanced reconstruction, quantification, meshing, data organization, and visualization capabilities, DREAM3D-NX has emerged as a go-to tool for the materials science community to reconstruct and quantify 3D microstructures. Its flexibility and adaptability make it suitable for a broad range of multidimensional data analysis applications beyond materials science and engineering domain.

simplnx is a rewrite of the SIMPL library upon which DREAM3D is written. This library aims to be fully C++17 compliant, removes the need for Qt5 at the library level and brings more flexible data organization and grouping options. The library is under active development by BlueQuartz Software at the current time.

Project Vision

When the DREAM3D project began in 2009, it was created with the goal of integrating the many disperate software codes in existance at the time under a single software framework. This software framework would allow these codes to define their requirements for input data, output data and required parameters in a common structure and thus enable the existing codes to ingest data from a previous code algorithm and produce output that the other code algorithms can readily utilize. By 2013 DREAM.3D had matured into a flexible desktop application that could peform a wide variety of materials science related processing. Over time, DREAM.3D’s capabilities expanded to include more generic data processing algorithms and image processing algorithms. The United States Government funded the entire development cycle for DREAM.3D and was gracious to allow the main project to be open sourced to allow it to penetrate into as many institutions as possible. In recent years the whole of DREAM3D and SIMPL were completely rewritten into DREAM3D-NX and the simplnx library. This new version focused on developing both maintainable and expandable software that enables a lower barrier to entry for contributions.

The vision for the DREAM.3D project is to produce accessible software tools that are professionally developed, relentlessly tested, and maintained into the future thus accelerating current and future R&D efforts by building upon the contributors to the DREAM3D project.

Public Release Notice

This software library is directly supported by USAF Contract FA8650-22-C-5290 and has been cleared as publicly releasable under the following:

AFRL has completed the review process for your case on 14 Sep 2022:

Subject: DREAM3D_Project (O) Shah #-11  (Software Code)

Originator Reference Number: RX22-0852
Case Reviewer: M. Allen
Case Number: AFRL-2022-4403

The material was assigned a clearance of CLEARED on 14 Sep 2022.

This clearance is in effect until 14 SEPT 2025 at which point any new additions created under USAF funding should be cleared. Public additions through the normal PR process will not be affected.

Get Help

The current documentation is hosted at dream3d.io docs.

The current python binding instructions and documentation can be viewed at dream3d.io python docs.

See our Support File for more ways to get help!

Building from Source

For instructions on building from source see our Guide to Building From Source.

Code of Conduct

By participating in this project, you are expected to uphold our Code of Conduct.

If you experience or witness unacceptable behavior—or have any other concerns—please report it by contacting the project maintainers at [email protected]. All reports will be handled with discretion.

Community

For those looking to engage with the DREAM3DNX community, see the discussions board of DREAM3DNX-Issues discussions.

Contact Us

You can talk directly with developers in the DREAM3DNX-Issues discussions.

If this doesn't work for you you can reach out to our team at our general address [email protected].

See our Support File for more ways to contact us!

Contribute

Thanks so much for your interest in contributing to simplnx!

The simplnx project needs all sorts of contributions, so don't be discouraged if you are unfamiliar with C++. We have all sorts of tasks that can be done without any prior coding knowledge. We need have non-coding tasks, computer power user tasks, security research tasks, Python tasks, and C++ tasks, all of which are outlined in our Contributing Guide.

We also understand the struggle involved with getting started on a new project and the frustration that can come with learning a new codebase. To help cut down on the stress of this we set up a discussion section specifically for Contributor Questions where you can quickly get a response directly from maintainers.

The average response time for new pull requests is same-day if it's Monday-Thursday (often times less than 1 hour), but Friday-Sunday response time is variable.

We deeply appreciate any and all contributions and that will absolutely be reflected in your interactions with active maintainers.

Contributors

Huge thanks go to these wonderful people (emoji key):

Michael Jackson
Michael Jackson

💻
Jared Duffey
Jared Duffey

💻
Joey Kleingers
Joey Kleingers

💻
Jessica Marquis
Jessica Marquis

💻
Matthew Marine
Matthew Marine

💻
Nathan Young
Nathan Young

💻
bpenniebq
bpenniebq

💻
john-stone-ics
john-stone-ics

💻
VKUDRI
VKUDRI

💻
Yang Liu
Yang Liu

🔬 🔣
Add your contributions

This project follows the all-contributors specification.

simplnx's People

Contributors

ajacksonbq avatar allcontributors[bot] avatar bpenniebq avatar imikejackson avatar jduffeybq avatar jmarquisbq avatar joeykleingers avatar john-stone-ics avatar mmarinebluequartz avatar nyoungbq avatar vkudri avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

simplnx's Issues

Empty JSON values will be allowed in Parameters

It is perfectly valid for the value of a json parameter to be empty, i.e., if the parameter is a DataPath and the value of the json is an empty string, this is perfectly valid and allowable to pass back an empty string.

DataStructure read/write test coverage

More test coverage is need for DataStructure related reading/writing HDF5 files. Specifically geometries are under tested since the changes that were made to the hierarchy.

Uninitialized memory in DataStore

DataStore doesn't initialize its memory when constructed. This can be desirable if that memory is filled from another source immediately, but currently some filters expect the memory to be 0 when it's not.

Specifically FindFeaturePhasesFilter's test fails in debug on Windows because the memory is not initialized to 0.

We should have two functions to create a DataStore, one initialized and one not, and initialized should be the default.

Create Data Array: Option to Create the array in an AttributeMatrix or just "Free".

This allows the user to not have to worry about putting in the correct dimensions, because the dimensions will come from the AttributeMatrix tuple shape.

Or have an option to copy dimensions from the Input Geometry, which again, will ensure they get the dimensions correct.

Also probably should have a sanity check if we create an array inside an attribute matrix, that the dimensions are correct.

PipelineRunner needs to have improved error reporting and progress reporting

Current pipeline runner does not report errors so that the user can effectively correct problems. Pipeline runner needs to do the following:

  • Properly load Plugins on all platforms and in both development and deployment scenarios.
  • Verbosely report errors to the user so that the user can take appropriate action to correct the errors.
  • Report the standard info and progress for each filter that is being executed.

DataObject::Type::AbstractGeometry will never be used

The derived classes of AbstractGeometry (and AbstractGeometry2D/AbstractGeometry3D) will have getDataObjectType() overridden by derived classes so that their associated DataObject::Type enum value will never be used.

Porting Filters from Core to ComplexCore

Filters to port over from complex_plugins/Core into ComplexCore

These filters have an Orientation Analysis Plugin dependency

In order to remedy this issue of cross dependency, one must rework the .dream3d input test file to accommodate the Orientation Analysis filters that are run to prep the test file for running the filter or analyzing the results. Here is the list of filters with said dependency:

  • AlignSectionsFeatureCentroidFilter
  • ErodeDilateBadDataFilter
  • FindNeighborhoodsFilter

Needs Further Filter Review

This has abnormalities that need to be patched before they can be ported.

  • RotateSampleRefFrameFilter - (Has missing actions in preflight)

Needs Further Test Case Review

  • FillBadDataFilter - (Needs test case implemented)
  • RemoveFlaggedFeaturesFilter - (Test the Neighborlist warning logic from preflight)
  • ResampleImageGeomFilter - (Extensive preflight, explore a Test Invalid Execution test case)

Need an Overarching Test Case for Instantiation

There needs a test case to instantiate all filters that are compiled - must be done dynamically

Also after dynamic test case completion, one must go through and strip out the instantiation test case from each test file.

If filter is not on this list but is in Core it is in the process of being ported already

Crop Image Geometry does not crop correctly

During test case runs I stumbled into a rabbit hole that showed out that Crop Image Geometry doesn't output data correctly. The test case for the file doesn't test rigorously enough, so it wasn't caught until today when the SmallIN100 was dropped into HDF5 view during a fix for ErodeDilateDataTest case. Upon further dive the the Crop Image Geometry filter itself seems to have an incorrect step.

Steps to replicate:

  1. Run CTest to get the CropImageGeometryTest to print a dream3d file containing the data
  2. Drop dream3d file from CropImageGeometryTest into H5Web on VS Code

InterpolatePointCloudToRegularGrid and Test

  • This filter is missing a few parameters from the original (Copied Array Suffix and Interpolated array suffix).

  • It is missing a check for boolean in the interpolated arrays (not sure how to interpolate booleans).

  • The test has boolean arrays to interpolate leading to nullpointers.

  • the filter does not take into consideration the interploated suffix that the user should be filling in (or have a default value assigned).

Export ASCII Data: Allow user to set the "Index" header name

If the user selects Header and Index Options (Both), then in the output file there is a new column "Feature_Ids". Add a new parameter (that is linked to this choice) that allows the user to set the name of that column, or change the name to "index" instead of "Feature_Ids".

Find Array Statistics: Show progress to the user

Maybe just state for each kind of statistic that is being calculated? Each kind of stat is on its own thread so you may need to do some sort of "ThreadSafeProgressMessage". See other filters that do something similar

Find Array Statistics: Too much memory is being used

Rework the algorithms to prefer compute time over memory usage. The "Feature to Voxels" map is taking way too much memory to calculate the statistics. We need to completely rework the "compute by feature index" algorithms to prefer gathering the data by feature "on-the-fly" as much as possible.

For example the Length, Min, Max, Mean, Summation, could all be found simultaneously during the loop.
Median is going to require all the voxels so that you can then sort and get the answer.
Find StdDev needs the summation and Mean anyways, so compute those and then compute the StdDev after those are computed.

DataArray<bool> round trip to/from HDF5 (.dream3d) changes type to uint8

Create a Pipeline with the following:

CreateDataArray: Int32, 100 tuples, init to '10', name it 'Data'

MultiThreshold Objects: Add a criteria for "Data" > 10.

Write DREAM3D File: write it out to 'test-1.dream3d'

run it.

Create a new pipeline.

Read DREAM3D File: read 'test-1.dream3d'
Write DREAM3D File: write out 'test-2.dream3d'

Now open test-1.dream3d and look at "Mask". in the attribute section it will be DataArray
Now open test-2.dream3d and look at "Mask". in the attribute section it will be DataArray instead of DataArray.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.