cnatale/ihtai2

ihtai2's Issues

Add caching for global points table nearest neighbor queries

Memcached seems like a good option. Will need to clear cache every time there time a global point is added or removed.

PatternRecognitionGroup: splitPatternRecognizer()

App crashes when told to create an n-dimensional point with decimals

This is because it creates action database tables using the point's component numbers, and you can't use dots in mysql table names. Add some type of dot replacement logic before assigning the n-dimensional point as a table name. Maybe just part of the "toActionString()" pattern-recognizer method.

multi-time periods: earlier time periods record first score until sliding window fills up

If I add multiple time periods to an Ihtai instance, it updates the score of all time periods with their first recorded score until the sliding window completely fills up. This is because sliding window decides the action/score combo based on index 0 in the sliding window array of instances.

To fix this, I could either not begin recording scores until the sliding window completely fills (not ideal), or start recording using offsets from the end of the sliding window array. This will result in more time-consuming/complex data insertion because the score from the different time periods won't all be inserted into the same actions table.

add error codes to api methods

Perhaps send an object with error_code and message props to client.

actions tables next_action should only contain action sub_array string

Right now a pattern tables next_action column contains a row for each entire state string. Instead turn it into a substring of only the actions dimensions.

splitting logic: ensure action row doesn't already exist before adding to table

Since each action row is no longer a unique combination, do an additional check in the pattern splitting logic to only add a new row if one doesn't exist with the same score n-dimensional number.

Occasionally get error 'Table 'ihtaidb.global_points_table' doesn't exist'

I'll occasionally get this error even though the global points table is already created. It crashed my automated training last night.

{"name":"Ihtai","hostname":"xxx","pid":11918,"level":50,"err":{"message":"select `point`, `first_action_index`, `first_drive_index` from `global_points_table` - ER_NO_SUCH_TABLE: Table 'ihtaidb.global_points_table' doesn't exist","name":"Error","stack":"Error: ER_NO_SUCH_TABLE: Table 'ihtaidb.global_points_table' doesn't exist\n    at Query.Sequence._packetToError (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)\n    at Query.ErrorPacket (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)\n    at Protocol._parsePacket (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:279:23)\n    at Parser.write (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Parser.js:76:12)\n    at Protocol.write (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:39:16)\n    at Socket.<anonymous> (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/Connection.js:103:28)\n    at emitOne (events.js:116:13)\n    at Socket.emit (events.js:211:7)\n    at addChunk (_stream_readable.js:263:12)\n    at readableAddChunk (_stream_readable.js:250:11)\n    at Socket.Readable.push (_stream_readable.js:208:10)\n    at TCP.onread (net.js:594:20)\n    --------------------\n    at Protocol._enqueue (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:145:48)\n    at Connection.query (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/Connection.js:208:25)\n    at /Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/dialects/mysql/index.js:152:18\n    at Promise._execute (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/debuggability.js:303:9)\n    at Promise._resolveFromExecutor (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:483:18)\n    at new Promise (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:79:10)\n    at Client_MySQL._query (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/dialects/mysql/index.js:146:12)\n    at Client_MySQL.query (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/client.js:197:17)\n    at Runner.<anonymous> (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/runner.js:146:36)\n    at Runner.tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)\n    at Runner.query (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/method.js:15:34)\n    at /Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/runner.js:65:21\n    at tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)\n    at /Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/using.js:185:26\n    at tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:512:31)\n    at Promise._settlePromise (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:569:18)\n    at Promise._settlePromise0 (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:614:10)\n    at Promise._settlePromises (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:693:18)\n    at Promise._fulfill (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:638:18)\n    at PromiseArray._resolve (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise_array.js:126:19)\n    at PromiseArray._promiseFulfilled (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise_array.js:144:14)","code":"ER_NO_SUCH_TABLE"},"msg":"select `point`, `first_action_index`, `first_drive_index` from `global_points_table` - ER_NO_SUCH_TABLE: Table 'ihtaidb.global_points_table' doesn't exist","time":"2018-03-13T05:11:39.731Z","v":0}```

PatternRecognitionGroup: deletePatternRecognizer()

PatternRecognitionGroup: Add dimensionality reduction utility api method

-make sure all dimensions are normalized
-split a point's dimensions into 2 subarrays (i guess just by middle of array)
-average each subbarray. that becomes the 2d point.

This functionality is useful for data visualization.

initialize() should add patternRecognizers based on existing patterns in global points db table

Currently pattern splitting functionality fails because /initialize only creates PatternRecognizers based on points passed into the constructor. Should start by doing a query of global points db table and adding PatternRecognizers for every row in the table. May need more information than schema currently provides, like which dimensions go in input, action, and drive sub-arrays.

Will need three new columns in global points table. One for starting index of input state, one for starting index of action state, and one for starting index of drive state. All sql queries that add rows to global points table will need to send this data as well.

PatternRecognitionGroup: Add de-normalize function

1. multiply normalize number by max de-normalized value (say, 5)
2. shift by the opposite value of normalization shift

refactor patternRecognitionGroup.getPatternRecognizer(req.body.patternString) to take O(1) space

Right now, PatternRecognitionGroup initializes one PatternRecognizer for each pattern in the global points db table. This doesn't scale well.

Instead, every call to PatternRecognitionGroup.getPatternRecognizer should directly instantiate a PatternRecognizer based on the pattern string passed in. Stop instantiating all patterns in the global points table as PatternRecognizers in memory, and only instantiate as they are used. This turns an O(n) memory requirement to O(1).

One downside is that this will probably increase garbage collection. I may want to implement an approach where only 1 PatternRecognizer is instantiated, and it gets its this.pattern property changed.

Feature: add ability to use a list of time periods instead of just one

I've been seeing very good results in the ball demo by creating a sliding window with a period of 30 steps, and assigning scores to action db tables based solely on the score of the last element in the window. This seems to allow for planning 1 second into the future in static environment.

There are a few problems with this approach. One is that if the environment is variable enough that the Ihtai agent is never able to settle on a score below the rubber banding cutoff, it'll perpetually cycle through all possible actions. The other is related: it has no way of handling unexpected "reflex" scenarios, like an object popping up and blocking it's path less than a second in advance. This will cause problem one.

A promising solution would be instead of only scoring against one time period, to score on a series of time periods. Start at a small period 3-5 steps ahead of time, and exponentially increase the time delta up to a maximum amount. Add them as extra columns to each action table. Then when selecting a best action, start with the farthest away time period and check if its best score is less than the rubber banding cutoff. If yes, select the action. If no, check the next-longest period, all the way down to the shortest period. If the shortest period is reached, always send its best score as the best action to take.

Theoretically, this could provide a way for an agent to perform long-term planning, as well as handle surprising "reflex" situations.

Add script to remove all patterns below a certain threshold

This script will be run when Ihtai instance isn't running.

Given a certain threshold, in decimal format (say .0001), remove associated global points table reference and actions table for all patterns that have less than the threshold update_count in relation to the update_count of pattern with top update_count.

Something like:
-select update_count from gobal_points_table ORDER BY update_count DESC LIMIT 1;
-use this value to select an array of global_points_table.point values that fall below the threshol
-select point from global_points_table where update_count <= highestUpdateCount*.0001;
-delete from gobal_points_table where update_count <= highestUpdateCount*.0001;
-drop table if exists B,C,A; (where B,C,A is a concatenated list of table names falling below the update_count threshold)

Force patternRecognitionGroup input to only use possibleActionValues

Dependent on issue #3

Right now, possibleActionValues are only used to create the next move rows for a specific patternRecognizer.

This means that programmers need to manually constrain pattern values before passing them into Ihtai. Add step so that the PatternRecognitionGroup constrains pattern values before doing anything with them.

add logic to pull all scores in an action table to near-ideal

One problem with the approach Ihtai takes towards acting on best-scoring action out of all available actions is that if the best available score isn't very good, but better than even worse scores, it'll still act on best available instead of periodically trying new things. A way to compensate for this is to, on score update, also run an operation on all of the selected action's table rows that pulls the score to something near ideal. I.e.: score = (score * 9 + .1) / 10. This will ensure that ihtai tries new behaviors in situations where its ideal score isn't very ideal.

Perhaps rubber banding is a good name for the feature.

bug: initializeFromDb fails if no drive scores are used by agent

{"name":"Ihtai","hostname":"Chriss-MacBook-Pro.local","pid":10083,"level":50,"msg":"FAILURE INITIALIZING PATTERN RECOGNITION GROUP FROM DB","time":"2018-03-15T15:56:45.673Z","v":0}
{"name":"Ihtai","hostname":"Chriss-MacBook-Pro.local","pid":10083,"level":50,"error":{"isJoi":true,"name":"ValidationError","details":[{"message":"\"0\" must be a number","path":[0,"driveState",0],"type":"number.base","context":{"key":0,"label":0}}],"_object":[{"inputState":[-40,40,0,0],"actionState":[0],"driveState":[null]},{"inputState":[-20,20,0,3],"actionState":[4],"driveState":[null]},{"inputStat....

... goes on like above for what i think is all pattern recognizers, then logs these errors...

    at Query.Sequence._packetToError (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
    at Query.ErrorPacket (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
    at Protocol._parsePacket (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:279:23)
    at Parser.write (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:39:16)
    at Socket.<anonymous> (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/Connection.js:103:28)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10)
    at TCP.onread (net.js:594:20)
    --------------------
    at Protocol._enqueue (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/protocol/Protocol.js:145:48)
    at Connection.query (/Users/natale/Projects/ihtai_2.0/node_modules/mysql/lib/Connection.js:208:25)
    at /Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/dialects/mysql/index.js:152:18
    at Promise._execute (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/debuggability.js:303:9)
    at Promise._resolveFromExecutor (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:79:10)
    at Client_MySQL._query (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/dialects/mysql/index.js:146:12)
    at Client_MySQL.query (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/client.js:197:17)
    at Runner.<anonymous> (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/runner.js:146:36)
    at Runner.tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)
    at Runner.query (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/method.js:15:34)
    at Runner.<anonymous> (/Users/natale/Projects/ihtai_2.0/node_modules/knex/lib/runner.js:193:19)
    at Runner.tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/reduce.js:157:18)
    at Object.gotAccum (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/reduce.js:144:25)
    at Object.tryCatcher (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/natale/Projects/ihtai_2.0/node_modules/bluebird/js/release/async.js:143:10)
  code: 'ER_DUP_KEYNAME',
  errno: 1061,
  sqlMessage: 'Duplicate key name \'id\'',
  sqlState: '42000',
  index: 0,
  sql: 'alter table `pattern_-40_40_0_0_0_` add unique `id`(`next_action`, `time_period`)' }
{"name":"Ihtai","hostname":"Chriss-MacBook-Pro.local","pid":10083,"level":50,"msg":"FAILURE INITIALIZING PATTERN RECOGNITION GROUP","time":"2018-03-15T15:56:45.823Z","v":0}
{"name":"Ihtai","hostname":"Chriss-MacBook-Pro.local","pid":10083,"level":50,"msg":"Error: PatternRecognitionGroup.addPatternRecogizer(): initializeTables() failed\n            on one or more PatternRecognizer","time":"2018-03-15T15:56:45.823Z","v":0}
TypeError: Cannot read property 'resetUpdateCount' of undefined
    at PatternRecognitionGroup.splitPatternRecognizer (/Users/natale/Projects/ihtai_2.0/server/pattern-recognition/pattern-recognition-group.js:323:64)
    at /Users/natale/Projects/ihtai_2.0/server/main.js:229:27
    at Layer.handle [as handle_request] (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/layer.js:95:5)
    at next (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/layer.js:95:5)
    at /Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/index.js:281:22
    at Function.process_params (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/index.js:335:12)
    at next (/Users/natale/Projects/ihtai_2.0/node_modules/express/lib/router/index.js:275:10)
    at /Users/natale/Projects/ihtai_2.0/node_modules/body-parser/lib/read.js:130:5
    at invokeCallback (/Users/natale/Projects/ihtai_2.0/node_modules/raw-body/index.js:224:16)
    at done (/Users/natale/Projects/ihtai_2.0/node_modules/raw-body/index.js:213:7)
    at IncomingMessage.onEnd (/Users/natale/Projects/ihtai_2.0/node_modules/raw-body/index.js:273:7)
    at emitNone (events.js:106:13)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1056:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
Error: PatternRecognitionGroup.getPatternRecognizer(): 
        no patternRecognizer found matching the input string pattern_-20_20_0_-1_3_

Add webdriver.io support

PatternRecognitionGroup: Add normalize function

1. shift so leftmost range value is 0
2. divide normalization max value (say, 100), by the number to be normalize after shifting

Explore better ways to tune hyperparameters

OriginalScoreWeight and rubberBandingDecay need to be in a tight range of values for a given agent so that behaviors streams of behavior are built, while at the same time new actions are tried often enough to allow for adaptive behaviors to emerge in response to the environment. For example, the balldemo currently uses a rubberBandingDecay of 0.025, and an originalScoreWeight of 16 to achieve pretty good results. These values took a lot of testing to come up with. Explore ways to automate the parameter tuning.

Add Joi object validation for PatternRecognizer objects

Required to complete issue #1

Add a script to calculate standard deviation of all scores in all action tables after n trials

This will help with parameter tuning. I'm seeing a general trend that using different time periods results in different deviations. Deviation is quite small using a time period like 5, gets larger at 30, and not sure what it does for numbers much larger than that. I would assume the difference gets smaller again as the action accounts for less of effect on state. Would be useful to quantify.