itemsapi/itemsapi

Blazingly fast, multi tenant, faceted search API

ItemsAPI 2.x - fast faceted search

Blazingly fast faceted search. It has Node.js simplicity and C++ speed.

Features

• fast faceted search
• full text search
• multi-tenancy
• data indexing
• pagination
• simple JSON API
• web ui for exploring / managing index
• dockerized with alpine

Performance

ItemsAPI has a great indexing performance. It can index 2M documents in less than 2 minutes on local computer. Searching seems to be 10x faster than Elasticsearch for similar use cases

Docker

docker pull itemsapi/itemsapi:latest
docker run --privileged -it -p 3000:3000 itemsapi/itemsapi

Getting started

Please look into full example here - index and search movies

Licence

ItemsAPI 2.x is created by Mateusz Rzepa. You can use ItemsAPI for free, if your project is Open Source. If you want to use ItemsAPI in to build commercial software, you need to buy a commercial license. Commercial editions can be used in closed source environment. More info soon

itemsapi's Issues

System statistics in CLI

Show system information in CLI like in Ionic Framework:

For now Node.js and elasticsearch version is enough.

Add default created_at and updated_at into each collections

Separate routes and controllers

Make routes small, lean and simple. Move code to the newly created controllers files

Search response with 400 if query contain the /, &

the problem solved if query enclosed with quotes like "goodyear R18 245/55"

Recent and saved searches

Save recent searches to improve user experience on mobile
https://uxplanet.org/mobile-ux-design-user-friendly-search-51e5f78f5a1e#.3nag8xpdn

(ctrl+f saved searches)

Manage collection from subdomain

User / developer should be able to manage collection / items by subdomain too:
i.e. currently we have POST customdomain.com/api/v1/items/restaurants, additionally it should be managed like POST restaurants.customdomain.com/api/v1/items

(good use case would be for itemsapi working as a service)

Implement range filters based on configurations

For such a configuration

"aggregations": {
  "rating_range": {
    "type": "range",
    "field": "rating",
    "title": "Rating range",
    "ranges": [
      {"lte": 1, "name": "0 - 1"},
      {"gte": 1, "lte": 2, "name": "1 - 2"},
      {"gte": 2, "lte": 3, "name": "2 - 3"},
      {"gte": 3, "lte": 4, "name": "3 - 4"},
      {"gte": 4, "lte": 5, "name": "4 - 5"},
      {"gte": 5, "lte": 6, "name": "5 - 6"},
      {"gte": 6, "lte": 7, "name": "6 - 7"},
      {"gte": 7, "lte": 8, "name": "7 - 8"},
      {"gte": 8, "lte": 9, "name": "8 - 9"},
      {"gte": 9, "name": "9 - 10"}
    ]
  }
}

It should generate range aggregation for the data out of the box.
It will look like this in the dashboard:

Create configurable logs

ItemsAPI should store logs out of the box and probably should be enough configurable to send it to external providers like logstash too

Timestamp, Client IP, Request URL, Request method, Request body, HTTP code should be good to store.
Inspiration from algolia (https://github.com/algolia/algoliasearch-client-js)

Make simple recommendation system

Documents should have options "similar documents" and algorithms should be based on mapping configuration.

response time for documents list

good for making optimization later and scalability

Docker integration

Tested docker cloud already with another project and works very cool.
Possible integration with amazon aws, digitalocean and even your own server. Project configurable by environment variables i.e. ELASTICSEARCH_URL, PORT etc

Would be great to integrate ItemsAPI with docker - won't be as easy as one-click-to heroku but very flexible though

Get list of all distinct aggregation values

Should return json list with distinct values like:

John Wayne (51)
Michael Caine (51)
Robert De Niro (47)
Bruce Willis (35)
Liam Neeson (33)
Christopher Walken (31)
Samuel L. Jackson (31)
Harvey Keitel (30)

and slugged for pretty url (if slugs option enabled)
john-wayne (51)
michael-caine (51)
samuel-l-jackson (31)

API keys - privileges

Make it possible to access itemsapi data and rest endpoints only by generated API keys.

Consider:

• one global key (token) for all actions in collection
• one key for only specific methods like (POST, PUT, DELETE)
• one key for specific actions (search, create_item, delete_item, get_item, get_collection, etc)

Ensure createProjectAsync add items

https://github.com/itemsapi/itemsapi/blob/master/src/services/project.js#L41

Show distance between geo field and my location

Collection list should have option showing distance between my current geo and geo field.

Aggregation parse wrong the string with the spaces

Catgory name -> kids clothes, ultimately looks like a

• kids (100)
• clothes (100)

make request statistics

i.e.

• requests count
• response time
  and write it to mongodb, redis or external saas

Ability to import / use mapping from existed elasticsearch index

Ignore non elasticsearch properties when saving mapping

elasticsearch > 2.0 is more restricted about non supported schema properties like "display: image"

error: Error: [mapper_parsing_exception] Mapping definition for [actors] has unsupported parameters: [display : array]

Extract `nconf` to separate file

Calling nconf.get multiple times to serialise all the elements into object may affect performance. Imo the better way would be to create something like this https://github.com/este/este/blob/master/src/server/config.js

So, ideally, this: https://github.com/itemsapi/item/blob/master/app.js#L14-L17 goes to config/index.js where the config is parsed according to current environment and then, exported with nconf.get().

Create changelog and roadmap

document add

Items enabled / disabled

Make option to enable / disable items
Good for items:

• added by users and waiting for activation process
• archiving items

Any other suggestions making it more generic very welcome

Make sorting by geo location

It is currently working but for hard coded geo location.

We should:

• get valid geo location from input
• make some tests

Ensure update and partial collection update works

Seems collection update works only partially and doesn't update the whole object

Besides:
exports.updateCollectionAsync

Add:
exports.partialUpdateCollectionAsync

Auto actualize documenation at the itemsapi.com from itemsapi.readme.io

document details

should take care about:

• text
• image
• gallery
• geo
• array (i.e. tags)

Store collections in mongodb

Currently we store collections / configurations in json file. This is good and easy for the start but it is out of sync if backend is growing and we have more server instances.

Probably mongodb could be solution here.

Don't wrap entire module in function call

https://nodejs.org/api/globals.html#globals_global

document edit

Take configurations from environment first

Inspired by http://12factor.net/ and required by cloud integration i.e. with heroku.

ItemsAPI should take options from environment first and from our json configurations later i.e.:

NODE_ENV=production PORT=3030 ELASTICSEARCH_HOST=http://domain.url:9200 node app.js

Heroku example: https://blog.heroku.com/archives/2014/8/7/heroku-button

Simplifying aggregations response

Currently looks like this:

Have a plan to share http://app.itemsapi.com source code?

Elasticsearch 2.1 version get issue with `display` mapping param

• has unsupported parameters: [display : image]
• has unsupported parameters: [display : array]

in schema expample from documentaion
"image": {"type": "string", "display": "image" },

btw what the preffer version of elastic for current documentation state?

Make data exporting

Make data exporting functionality with access rules which define if collection is exportable or not.

Collection validation

Make collection validator

Ensure:

• collection is json type (object)
• name exists and is unique
• schema, aggregations and sortings is an object type

It can be good start for next improvements i.e. if schema or aggregations fields are correct

document delete

Consider using YAML files instead JSON for configurations

YAML seems to be easier and is used in successful frameworks like symfony or ansible.

Check if node.js has good integration with yaml and if it supports importing other yaml files: i.e.:

collections:
- include: collections/movies.yml
- include: collections/songs.yml

Then configuration files would be much easier too read and maybe we should use it.

Refactor collections mapping / configuration

Refactor collections configuration:

• separate it from /config/root.json
• store it in database, firstly in json file is enough and make it flexible to store it in redis, mysql or mongodb later
• write RESTful endpoint for managing configurations (add, edit, delete, list)
• make async support (it will require code refactor in many places)

Benefits:

• more scalable because you can store even thousands collections in one application
• easier management (by dashboard or CLI)
• easier for writing instalation

Currently it looks like this btw:

Discuss overall app structure

This is actually a proposal rather than something to fix, mainly because I haven't dig into the architecture yet, but... the server.js is actually something we can improve in order to allow different backends (like elasticsearch) to be implemented in the future (the main I can think of is a graph database).

For now, we require the services explicitly https://github.com/itemsapi/item/blob/master/server.js#L28-L31 as well as the data and they do depend on each other (e.g. dataService requires some elastic elements - cross-folder requires), so we end up having a nice, not-that-small scope to lookup when handlers are invoked.

Imo, to improve that, we can make something like this:

-- backends
---- elasticsearch
-------db
-------dataService
-------index

and then, do:

app.use(elasticSearch({connectionString}));

ElasticSearch can then create a connection, attach it to request (it's a middleware!) and attach other dataServices to request as well, so in casual request handlers you just write:

router.use(elasticSearch(config));
router.get('/', function(req) {
   req.elastic.dataService(....);
});

All in all, you will end up with a nice layer on top of express that can be just dropped in to any existing app which surely will increase the possible adoption in the future :)

Another thing is the config, imo, with the middlewares pattern presented above, you can just allow user to pass config as an object to your middleware dropping the nconf dependency completely letting users handle the configuration on their own.

After the above are implemented, server.js can be moved to examples/elastic_search/index.js to present one of the possible ways of using the module.

Make default sorting options

Do you need MongoDB as a Dependency?

Hi @cigolpl are you using MongoDB for something?

itemsapi/package.json

Line 39 in 4db3b34

"mongodb": "^1.4.29",

Generate demo collection with CLI

Generate interactive demo collection with command line i.e. for:

• movies
• recipes, etc

It would be great for new users because they could create it only in few clicks.
We should inspire by great frameworks like symfony2, meteor, ionicframework.

Remove winston dependency

At least from core files, this can be easily replaced with an event emitter, or even this https://github.com/asyncly/EventEmitter2 and then, user can add in own middlewares by e.g.

api.on('error', ...)
api.on('elastic:query', ...)

Create configuration based on items (JSON)

System should be smart enough to create configuration (schema, sortings, aggregations) based on data provided by user / developer

It will significantly decrease learning curve, simplify creating project interface and generally improve user experience

i.e. for

[{
    name: 'Godfather',
    tags: ['Drama', 'Crime fiction'],
    rating: 10
}]

it generates:

{
    "name": "somename",
    "schema": {
        "name": {
            "type": "string",
            "store": true
        },
        "tags": {
            "type": "string",
            "display": "array",
            "index": "not_analyzed",
            "store": true
        },
        "rating": {
            "type": "string",
            "store": true
        },
    },
    "aggregations": {
        "tags": {
            "type": "terms",
            "field": "tags",
            "size": 10,
            "title": "Tags"
        },
    },
    "sortings": {
        "favorites": {
            "title": "Some rating title",
            "type": "normal",
            "order": "desc",
            "field": "rating"
        }
    }
}

Geo sorting via user IP

Currently if we sort by geolocation we use around_lat_lng. Would be great to sort by user IP too (if feature enabled) i.e. around_lat_lng_via_ip=true.

We can use that library to make it happen https://github.com/bluesmoon/node-geoip but it will require access to an external API (http://maxmind.com/)

Thanks algolia https://github.com/algolia/algoliasearch-client-js for inspiration.

autocomplete for collection

make research first how to make it simple and intuitive

Integrations with external providers

Consider collecting data from:

• google spreadsheet
• apifier.com
• google form
• external json file (url)
• mongodb / mysql
• shopify / prestashop / sylius / magento

Make different sorting and define it in the mapping

Each collections shoud have its own sorting types i.e. by relevance, distance, field, price, etc

Move hardcoded "project" index title to configuration