marg51/foul

tracker based on elasticsearch

foul's Issues

server: bulk operations should be smarter

draft

goal:

Improve performance and stability of /bulk endpoint, as well the architecture of the code. Readability is poor right now.

Stability:

Operations are currently not atomic and if one fails, it's reported as an error, as expected, but others will be successful. The integrity is compromised.

Each transaction should be saved in a log and kept if something went wrong

Performance:

We need to use the bulk endpoint from elasticsearch. It will be way faster than the current implementation. It's also easier to deal with errors. Since we generate IDs, everything can be flushed in the same transaction.

Architecture:

The code grew in size, let's put all managers into a new directory. server.coffee should have less logic in it and should only be a router.

retention: save dates

draft

Goal:

Cohort analysis -› retention rate …

• What percentage of users with the feature A enabled came back the next week of their signup
• After how many weeks our mobile users made their first purchase

---

We need to have access to two things : acquisition date and event date

• acquisition date can be sign up, first visit, first visit with a specific device, …
• event date can be purchases with date for each

We need to be able to filter events

• app versions
• feature enabled
• browsers
• location of user

---

The retention rate will be calculated this way :

• filter events
• aggregate events (for every browsers, app version ...)
• histogram on the diff between acquisition date and event date

---

Using flat document.

# we keep only events for the browser Chrome
query:
    filtered:
        filter:
            terms:
                browser.family: "Chrome"

# we split our results using the app version
aggs:
    appversion:
        terms:
            field: "appVersion"
       # and we calculate the number of days between the signup and the first buy
        aggs:
            firstBuy:
                histogram:
                    script: "Days.daysBetween(doc['signup_date'], doc['first_buy_date']).getDays()"
                    interval: "7"

Using a flat document has the advantages to be fast at execution time and easy to write the query (good point for Kibana)
However, it has a few drawbacks: slow to index, consume more memory and not really future proof.

The main problem is that it's not future proof. Since everything is stored inside a unique document, if you want to add useful data into a session, you would have to add it as well into this document and everything not in this document is not available at all.

---

Proposed schema for flat document

{
    "properties" : {
      "type": {
        "type": "string",
        "index": "not_analyzed"
      },
      "name": {
        "type": "string",
        "index": "not_analyzed"
      },
      "app": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "index": "not_analyzed"
          },
          "version": {
            "type": "string",
            "index": "not_analyzed"
          },
          "features": {
            "type": "object"
          }
        }
      },
      "user": {
        "type": "object",
        "properties": {
          "userId": {
            "type": "long",
            "index": "not_analyzed"
          }
        }
      },
      "device": {
        "type": "object",
        "properties": {
          "browser": {
            "type": "object",
            "properties": {
              "family": {
                "type": "string",
                "index": "not_analyzed"
              },
              ....
            }
          },
          "device": {
            "type": "object",
            "properties": {
              "family": {
                "type": "string",
                "index": "not_analyzed"
              },
              ....
            }
          },
          "os": {
            "type": "object",
            "properties": {
              "family": {
                "type": "string",
                "index": "not_analyzed"
              },
              ....
            }
          }
        }
      },
      "acquisitions": {
        "type": "object",
        "properties": {
          "first_visit": {
            "type": "date"
          }, "first_device_visit": {
            "type": "date"
          }, "signup": {
            "type": "date"
          },
          ...
        }
      }
    }
  }
}

server: authentication, public and private keys

draft

goal:

Every app should have a public key (tracker) and a private key (acquisition).

• rate limiting ?
• allow acquisitions from the tracker for a specific app ?

todo: how to implement ?