kitware/smqtk

Python toolkit for pluggable algorithms and data structures for multimedia-based machine learning.

SMQTK - Deprecated

Deprecated

As of Jan 2021, SMQTK v0.14.0 has been deprecated. The various interfaces and implementations have been broken out into the following distinct packages which will continue to be supported instead.

• SMQTK-Core provides underlying tools used by other libraries.

• SMQTK-Dataprovider provides data structure abstractions.

• SMQTK-Image-IO provides interfaces and implementations around image input/output.

• SMQTK-Descriptors provides algorithms and data structures around computing descriptor vectors.

• SMQTK-Classifier provides interfaces and implementations around classification.

• SMQTK-Detection provides interfaces and support for black-box object detection.

• SMQTK-Indexing provides interfaces and implementations around the k-nearest-neighbor algorithm.

• SMQTK-Relevancy provides interfaces and implementations around providing search relevancy estimation.

• SMQTK-IQR provides classes and utilities to perform the Interactive Query Refinement (IQR) process.

Intent

Social Multimedia Query ToolKit aims to provide a simple and easy to use API for:

• Scalable data structure interfaces and implementations, with a focus on those relevant for machine learning.
• Algorithm interfaces and implementations of machine learning algorithms with a focus on media-based functionality.
• High-level applications and utilities for working with available algorithms and data structures for specific purposes.

Through these features, users and developers are able to access various machine learning algorithms and techniques to use over different types of data for different high level applications. Examples of high level applications may include being able to search a media corpus for similar content based on a query, or providing a content-based relevancy feedback interface for a web application.

Documentation

Documentation for SMQTK is maintained at ReadtheDocs, including build instructions and examples.

Alternatively, you can build the sphinx documentation locally for the most up-to-date reference (see also: Building the Documentation):

# Navigate to the documentation root.
cd docs
# Install dependencies and build Sphinx docs.
pip install -r readthedocs-reqs.txt
make html
# Open in your favorite browser!
firefox _build/html/index.html

smqtk's Issues

Step by step instructions for computing an image feature set with the python library in README

Something like this (ok it will not look exactly like this but something similarly straightforward):

import smqtk

img = smqtk.load('myimage.png')
feature = img.get_feature('feature name')
print feature

In the README I see stuff about config files, etc. for setting up batch file ingests, but it seems something simple like the script above should be possible.

ImageSpace: text search on original ad text

Remove redundant descritpor generator configuration nexting in ``NearestNeighborServiceServer``

There's currently an extra layer example that serves no purpose.

Have FileDataSet use system configured data root when given relative paths

The other configuration chunks allow that, so it would make sense for the file data set constructor to do the same.

ImageSpace: Extraneous scrollbars under each image

Under each image, there is a div that holds the icons. It has scroll bars with nothing to scroll. This can be fixed by setting overflow:auto or deleting the overflow tag set on .im-caption in imagespace.min.css on line 38.

This appears in Firefox 38.0 on Ubuntu 15.04.

ImageSpace: clicking on image opens details window (maybe hover text for this)

ImageSpace: strip html from ad text

Fix travis yml

dateutil is not being installed.

Tests for IqrIndex and SimilarityIndex abstract/configuration/impls

Sorely lacking unit tests.

ImageSpace: Show image size while hovering over distribution graph

``ClassificationElement`` should not require classification values to sum to 1

Not all classifiers output classifications such that confidence values sum to 1. For example CNNs like AlexNet.

ImageSpace: search by ad id

Investigate / Add ``@numba.jit`` optimizations to ITQ model fitting functions

ITQ model fitting (here) makes heavy use of numpy functions for mathematical operations. Theoretically numba can Make This Better (tm). Investigate / Fix.

Reconnect search demo web-app to current interfaces

Since the various overhauls have occurred, the demo GUI no longer connects properly with the toolkit. This needs to get resolved in order to demonstrate IQR + modular framework functionality.

Migrate Image Space to SMQTK

Update IQR control structures

They are still using mechanisms from the old system: pure ID references, no new data representation use. This needs to be fixed before we can finish the IQR demo GUI again.

Add some tests for the revised implementations.

ImageSpace: Save search url with saved image

This issue is heading towards provenance and ability to determine if soft/hard connection between images

ImageSpace: connection to open other tools - TellFinder - DIG - HG

SVM indexer fails to auto-pick negatives with small datasets

AUTO_NEG_PERCENT could evaluate to less than 1 (cast down to 0) with small sets or small enough percents. This causes a situation where there are no negative adjudications, leading to SVM to fail to classify.

ImageSpace: icons for below images

Integrate skimage ORB detector/descriptor

This would add the ability to use the full stack without license concerns of the colorDescriptor software.

Fix ``get_plugins`` doc string

At least one input arg and arg documentation is inaccurate to its actual function (smqtk.utils.plugin.get_plugins line 69).

ImageSpace: Minor interface tweaks

I think these should be simple and can be handled with text and/or html, so I'm grouping them together:

1. A button to show all images. It's not too bad to click in the search bar and then hit enter, unless you're coming from a page that already has the search field populated. Then you have to remember how to do it.
2. A note on the Image Size Distribution and Serial Number Distribution graphs that says that they are only showing results from the current search.
3. If there are no results from the current search for these two graphs, let's say that: "No results found for current search terms." As it is, it's just a bit confusing.

Refactor out use of global configuration components

Since this is not intended to be a single application library, each application should have its own configuration. This would be useful for preventing preventing unrelated applications from having more configuration than they need, as well as preventing applications from accidentally having duplicate configurations, e.g. server ports, security keys, etc.

This should be easier to do since we now have the Configurable interface and mechanics for nested configuration of things.

Separate IQR Indexing and kNN similarity computation if possible

For scaling, we are going to want to use different nearest-neighbor implementations, like ITQ or other flavors of LSH. If we can hide the implementation behind an abstraction wall, pos/neg indexing will be more flexible.

Integrate use of DescriptorElement abstraction layer

in-memory and file-based implementations are already there, but are not currently being used anywhere in the system. It will be advantageous to use the abstraction for future scaling, i.e. able to store features in db, elastic cloud, etc.

Add multiple UUID specification option to ``DataSet.get_data`` method

Want to ability to specify multiple DataElement UUIDs to the getter to get multiple elements at the same time. This would also be desired for some implementations of DataSet that are better able to get fetch many elements on a in-batch basis vs. on a one-at-a-time basis.

Confidence Interval display is broken

The polygon being drawn to represent the CI for a PR/ROC curve doesn't always overlap the underlying curve. This basically means the CI as it is being displayed now is meaningless. Either fix current implementation/display of the Wilson's score implementation or find another CI method to use.

Add environment variable option for generic plugin method

Currently, the generic plugin function will only pick up available implementations within a single specific directory in the source tree. When SMQTK is installed somewhere, the source tree will mostly likely be un-modifiable, or should at least be assumed so. So, for this all to work in an actual "plugin" fashion, there should be a way to add other directories for the plugin methods to look at and pick implementations up from.

Tests for bin/ scripts

Might need/want a flag in CMake or otherwise to enable these since they may end up taking a bit of time to run, drawing from my experience with these kinds of tests on other projects.

ImageSpace: better initial page

Put instructions on home page. Enable navigation back to home page.

ImageSpace: dropping image opens the details window

Requirements files cleanup

There are numerous packages listed in requirements files that should probably be optional. Some are also only needed for web modules, which might also want separate requirements files. With many requirements files, might also want to put them into a sub-directory instead of making the source root messy.

ImageSpace: how do you show the full size image?

Previously clicking on the image would open it full size in a new browser window. Now, it doesn't seem possible to see the full size image at all.

I think this is important, especially if you're trying to see if it's the same person in two images - or get a better look at a tattoo, etc...

I would suggest another icon under the images (maybe the picture icon?) that when clicked opens it full size. This could be a browser window for now, but we should probably think about adding light box style functionality so that you could actually move through the images like you do through any web gallery.

Be able to score/validate multiple classifier models at the same time

Basically configure a list of classifier algorithm plugin configs instead of just one. Would expect on output graphs to display a curves for over-all / single-class for each configured classifier. Expect to be able to give a name for each classifier configured.

Add a high level data representation interface

So, at least, we can levy an abstract serialize and unserialize method instead of globally requiring the use of pickle for serialization, which is known to be sub-optimal for certain things (like numpy arrays).

ImageSpace: highlight text that was searched for

New ``DescriptorIndex`` Data structure

NearestNeighborIndex, RelevancyIndex and CodeIndex algorithms and structure use some sort of "index" of descriptor element instances. I think there could be a generalized DescriptorElement index that abstractly stores instances. Elements should be query-able by their type/UUID pair (true unique identifier).

ImageSpace: link below image to search for similar images

SVM Indexer will show inverted results sometimes

Negative results at the top and positive results at the bottom.

Error alert being shown when tagging element as explicit

Probably should be separate, but preview bluring doesn't work on all browsers.

ImageSpace: missing ads db data??

This may just be me being confused, but when I look at images such as this one:
id:"/data/roxyimages/5070f670039e500755fe7887d6f3d4a8f24e4a79.jpg"
I don't see an ad ID or the text from the ad that it was retrieved from.

Am I missing something or is something amiss?

ImageSpace: Image Size link produces no results

When I click on a dot in the Image Size Distribution graph, I get no results. The dot shouldn't show up if there are no results associated with it.

To reproduce:

1. Open ImageSpace, click in search bar (keep the *) and hit enter to bring up all images.
2. The first image (id: /data/roxyimages/cmuImages/Illinois_2012_11_12_1352748331000_6_0.jpg) is a woman in a green Vancouver t-shirt laying on her back. Open the details and click Find Similar Images.
3. When the results load, click Image Size Distribution in the left side bar.
4. Click the dot at width = 149, height = 166. This should probably have 4 results, according to its size.
5. See that it returns no results.

Parallel map function can hang during interruption or externally killed workers

When Ctrl-C'ing a parallel-map in progress, an dead-lock can occur.

It has also been seen that if the workers are doing web-requests, they can lock up, possibly due to an infinite wait issue with the request. Then the threads or processes are killed externally, the function dead-locks and can't clean itself up properly.

Find alternative to gmpy2 for fast popcount functionality

Want to remove dependency on anything GNU because we don't want GPL licensing.

Fix import tree such that lack of ColorDescriptor doesn't cause errors

Since the ColorDescriptor package is to be treated as an optional dependency, its lack of presence should not cause an error along any import tree.

Add VP-Tree HashIndex

Or somehow generalize this into the CodeIndex representation structure. Its pretty much the case where all LSH-based nearest neighbor algorithms will/can use the same nearest-code-neighbor strategy once hashs are generated.

Main REAME file links are broken

Links still point to files from before the documentation overhaul.

ImageSpace: Register/Login functionality

If I understand correctly, this functionality is not currently functioning. Can we disable it or comment it out then?

Similarity kNN server

Some server application that, given a label indicating the index to use, an example descriptor as a JSON list/vector, and N number of desired neighbors, return a JSON object that lists in order the [UUID, descriptor] pairs in distance order. Maybe include distance from exemplar value for each pair, too.