gchq/gaffer-tools

gaffer-tools is deprecated. Use https://github.com/gchq/gafferpy instead

Gaffer Tools

Gaffer Tools is deprecated.
Gafferpy has moved to its own repository: gchq/gafferpy.
The last release of the Gaffer UI was in gaffer-tools-1.23.0.

License

Copyright 2016-2023 Crown Copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

gaffer-tools's Issues

Update to Gaffer 0.6.3

Update release process to match main gaffer project

Update the release process for the gaffer-tools project by publishing artefacts using Travis.

Update to Gaffer 0.6.6

Cleaner example for your README.md.

In your README, you have:

start = '1'; end = '3'; g.V(start).hasLabel('id').store('v').repeat(bothE().where(without('e')).store('e').inV().hasLabel('id').where(without('v'))).until{it.get().id() == end || it.loops() == 6}.path().filter{it.get().last().id() == end}

This can be written without lambdas and bit cleaner as:

g.V('1').hasLabel('id').
  repeat(bothE().otherV().hasLabel('id').simplePath()).
    until(hasId('3').or().loops().is(6)).
  hasId('3').path()

Things to note:

1. You don't need to store "v" and "e" as simplePath() will ensure you don't go back to an already seen edge/vertex.
2. You don't need lambdas in your until().
3. If the end of the path must have an id of end, then simply hasId(end).
4. bothE().otherV() goes in both directions. If you don't care about the edges in your final path data structure, just use both().
5. If you want to be fancy, you could do ...path().by('name') and that would return the path where instead of the vertices, you would have name properties from those vertices. Easier for the human to read the result. http://tinkerpop.apache.org/docs/current/reference/#path-step

Federated rest service does not allow different FederatedExecutor implementations to be used

Update to gaffer version 0.4.7

Enable the TinkerPop implementation to connect via the REST API

Currently the tinkerpop implementation connects directly to Gaffer/Accumulo. It would be useful to be able to connect via the Gaffer REST API.

Fix new schema-builder lint errors

schema-builder add button for json import/export from/to file

On the 'schema' tab in the UI builder it would be good to have a 'save to file' button for each of the 'data schema', 'data types' and 'store types'

The user should be able to specify the path to the file

And similar for uploading a local file.

Mini accumulo cluster store properties template has incorrect groupId

Update Gaffer version to 0.7.4

Python shell does not allow a view to be provided in a GetAdjacentEntitySeeds operation

Make the python-shell more "pythonic"

The current python-shell project doesn't meet the python style guidelines and should be updated to meet PEP8 (https://www.python.org/dev/peps/pep-0008/)

Packaging for the mini accumulo jar should be set to the 'single' goal not 'assembly'

UI does not handle results with complex vertex types

This is linked to gchq/Gaffer#817

Fix Road Use Example walkthrough

Currently the last part of the walkthrough doesn't work.

This is due to the bug in Gaffer gh-642
error message is:

com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: org.apache.catalina.connector.CoyoteInputStream@15ca496; line: 1, column: 427] (through reference chain: uk.gov.gchq.gaffer.operation.OperationChain["operations"]->Object[][0]->uk.gov.gchq.gaffer.operation.impl.get.GetElements["view"]->uk.gov.gchq.gaffer.data.elementdefinition.view.View["edges"]->java.util.LinkedHashMap["RoadUse"]->uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition["preAggregationFilterFunctions"]->java.util.ArrayList[0]->uk.gov.gchq.gaffer.function.context.ConsumerFunctionContext["selection"]->java.util.ArrayList[0])

Develop a Slider application package for Gaffer

Develop an Apache Slider application add-on package to allow Gaffer instances to be easily deployed onto YARN-managed clusters.

Update group ID

Update group ID to uk.gov.gchq.gaffer to match the main project.

Python shell needs updating to use new EITHER flag instead of BOTH

The python shell needs updating due to changes in Gaffer 0.7.4.

Since Gaffer 0.7.4 the DirectedType enum and IncomingOutgoingType enum need updating to be EITHER instead of BOTH.

The EdgeSeed object now has the option for directed to be EITHER, this feature should also be added to the python shell.

See gchq/Gaffer#752.

Move tinkerpop to gaffer-experimental

Update to Gaffer 0.7.2

Random element generation

Develop a method of generating large amounts of random elements for testing purposes.

Move ui module from main gaffer project

As discussed in the main Gaffer project (issue gh-888).

Update to gaffer 0.6.1

Typo in SeedMatchingType in python shell

Add Slider cluster profile for Amazon EMR

Add gaffer-slider integration tests

Create Gaffer demo

This should use an in memory accumulo store.

Add post aggregation and post transform filter functions to python shell

Update to Gaffer 0.6.0

Migration Steps

See Gaffer 0.6.0.

Update to gaffer 0.7.3

Upgrade to work with Gaffer 0.4.6

Gaffer 0.4.6 has just been released.

Update schema builder UI to make stricter use of types

Using typescript type definitions properly rather than assigning everything to "any" should create more robust code. Warnings and errors should display if the variables are given values that don't match the interface they are given.

Update to gaffer 0.7.5

Add AccumuloStoreITs to run as part of gaffer-slider IT suite

#52 only runs AbstractStoreITs against a gaffer-slider instance.
We should also run AccumuloStoreITs.
Doing this requires gchq/Gaffer#686 to be fixed.

Update Slider package to allow Gaffer to be deployed onto HBase

Add tinkerpop implementation and python shell

These modules are currently in the main Gaffer repo and should be moved to this repository.

gaffer-slider should shade the accumulo iterator jar

The 'jar-shader' tool should be run as part of the build to avoid jackson version issues when Gaffer instances are deployed onto CDH clusters.

Create script for applying jackson shading to a jar

Fix slider scripts

Changing the artifactId from gaffer-slider to slider has broken the 'quick start' scripts

Add pylint/PEP8 checks to GitHub Actions to validate the python shell

Travis should run some code analysis checks to determine whether the Python shell is ready for release, e.g. pylint/PEP8.

Write an automated system test for the UI

We should use something like selenium to verify a basic query in the UI works. This can then be added to the travis ci scripts.

Update to Gaffer 0.6.2

UI needs updating to use new EITHER flag instead of BOTH

The UI needs updating due to changes in Gaffer 0.7.4.

Since Gaffer 0.7.4 the DirectedType enum and IncomingOutgoingType enum need updating to be EITHER instead of BOTH.

See gchq/Gaffer#752.

Add Python checks to Travis

Travis should run some checks to determine whether the Python shell is ready for release (unit tests, pylint/PEP8 checks etc)

Update to gaffer 0.7.1

Create a federated Gaffer REST API that delegates to multiple Gaffer REST APIs

             [ -------- Federated Gaffer REST API ----------  ]
                |                    |                     |
    [Gaffer REST API 1]    [Gaffer REST API 2]    [Gaffer REST API 3]
                |                    |                     |
    [Accumulo Store 1]     [Accumulo Store 2]      [HBase/Other Store]

Gaffer operations executed on this federated rest api would be delegated to multiple Gaffer REST APIs

Initially this should be implemented to assume that element groups are not shared across the different REST API. This means there is no need for summarising/merging the results across the REST APIs. The rest api would hold a map of element group to Gaffer url, which is used to delegate the operations to the REST API urls. This map of urls should be able to be updated without the rest api needing to be restarted.

Prior to delegating the operation chains to the URLs the requests need to be heavily validated - to check for things such as:
The Schema for each Store is compatible - e.g Element groups should not be shared across stores.
The functions used in an operation View are available for delegate store

Also, if an Operation View is provided it should be chopped up based on the Element groups in the View and the Element groups in the REST API schemas.

Deserialisation/Serialisation of json operation chains and results should be prevented. This will improve performance and mean the federated REST API will not require all custom Functions and Operations to be present on the classpath. E.g. if [Gaffer REST API 3] has a custom Filter function 'Function3' and a user executes an operation chain including Function3 then if the federated REST API tries to deserialise it into the java Function3 object it would fail if the Function3 wasn't on the class path.

Update schema builder to use newest angular-cli version

Also worth trying to remove the materialize css now that angular 2 material is more complete.

Create schema builder

Update to Gaffer 0.7.0