mrc0113/log-event-json-schema

A simple and straightforward JSON schema for logging structured events

Simple Log Event Schema

The purpose of this schema is to define a simple structure for log events. It solves the unpredictable, brittle nature of logs by creating a contract around its structure. This normalizes log data across applications and teams making it easy for downstream consumers of this data (graphs, alerts, etc) to use it effectively. It eliminates unexpected structure changes, improves data consumption reliability, and ultimately makes your logs reliable.

This schema is used internally at Timber and across thousands of companies with success. It's the foundational reason the Timber logging platform is able to provide a great user experience out of the box. It enables us to make assumptions about your data and work with predictable structures. There's no reason any consumer of this data, even your own internal ones, can't get the same benefit.

Implementation

You are welcome to log your own structured events that adhere to this schema, or you can use one of the Timber libraries to do it for you:

1. Timber for Elixir
2. Timber for Go
3. Timber for Node
4. Timber for Python
5. Timber for Ruby

The Schema

You can check out schema.json for the full schema definition, but here's a quick rundown:

{
  "dt": "2016-12-01T02:23:12.236543Z",           // Consistent ISO8601 dates with nanosecond precision
  "level": "info",                               // The level of the log
  "message": "POST /checkout for 192.321.22.21", // Human readable message
  "context": { ... },                            // Context data shared across log line, think of it like join data for your logs
  "event": { ... }                               // Structured representation of this log event
}

For actual events, see below.

Event Examples

1. Error Event

A structured event that represents an error:

{
  "dt": "2016-12-01T02:23:12.236543Z",
  "level": "error",
  "message": "(RuntimeError) MissingClass is undefined",
  "context": {
    "http": {
      "method": "GET",
      "path": "/checkout",
      "remote_addr": "123.456.789.10",
      "request_id": "abcd1234" // <------------- View all logs within a request!
    },
    "user": { // <------------------------------ Associate users with your log events!
      "id": 2,
      "name": "Ben Johnson",
      "email": "[email protected]"
    }
  },
  "event": {
    "error": {
      "name": "RuntimeError",
      "message": "MissingClass is undefined",
      "backtrace": [
        {
          "file": "/path/to/file",
          "function": "myFunc",
          "line": 45
        },
        {
          "file": "/path/to/file",
          "function": "myFunc",
          "line": 45
        },
        {
          "file": "/path/to/file",
          "function": "myFunc",
          "line": 45
        },
        {
          "file": "/path/to/file",
          "function": "myFunc",
          "line": 45
        },
        {
          "file": "/path/to/file",
          "function": "myFunc",
          "line": 45
        }
      ]
    }
  }
}

2. HTTP Server Request Event

An event that represents an incoming HTTP request to your application's HTTP server:

{
  "dt": "2016-12-01T02:23:12.236543Z",
  "level": "info",
  "message": "POST /checkout for 192.321.22.21",
  "context": {
    "http": {
      "method": "GET",
      "path": "/checkout",
      "remote_addr": "123.456.789.10",
      "request_id": "abcd1234" // <------------- View all logs within a request!
    },
    "user": { // <------------------------------ Associate users with your log events!
      "id": 2,
      "name": "Ben Johnson",
      "email": "[email protected]"
    }
  },
  "event": {
    "http_server_request": { // Event type
      "method": "GET",
      "scheme": "https",
      "host": "timber.io",
      "path": "/checkout",
      "port": 443,
      "headers": {
        "content_length": 894,
        "content_type": "application/json", // <- Example of data that wasn't in the log line itself
        "remove_addr": "192.321.22.21",
        "request_id": "gy23fbty523",
        "user_agent": "Mozilla/3.0 (Win95; U)"
      }
    }
  }
}

3. HTTP Server Response Event

An event that represents an outgoing HTTP response from your application:

{
  "dt": "2016-12-01T02:23:12.236543Z",
  "level": "info",
  "message": "Sent 200 OK in 117ms",
  "context": {
    "http": {
      "method": "GET",
      "path": "/checkout",
      "remote_addr": "123.456.789.10",
      "request_id": "abcd1234" // <------------- View all logs within a request!
    },
    "user": { // <------------------------------ Associate users with your log events!
      "id": 2,
      "name": "Ben Johnson",
      "email": "[email protected]"
    }
  },
  "event": {
    "http_server_response": { // Event type
      "request_id": "gy23fbty523",
      "status": 200,
      "time_ms": 117,
      "headers": {
        "content_length": 894,
        "content_type": "application/json",
        "x_request_id": "gy23fbty523"
      }
    }
  }
}

4. SQL Query Event

An event that represents a SQL query:

{
  "dt": "2016-12-01T02:23:12.236543Z",
  "level": "info",
  "message": "SELECT * FROM users WHERE id = 1 (54ms)",
  "context": {
    "http": {
      "method": "GET",
      "path": "/checkout",
      "remote_addr": "123.456.789.10",
      "request_id": "abcd1234" // <------------- View all logs within a request!
    },
    "user": { // <------------------------------ Associate users with your log events!
      "id": 2,
      "name": "Ben Johnson",
      "email": "[email protected]"
    }
  },
  "event": {
    "sql_query": { // Event type
      "sql": "SELECT * FROM users WHERE id = 1",
      "time_ms": 54
    }
  }
}

5. ...and many more, checkout the schema for a complete list.

Versioning & Releases

Timber follows the semver specification for versioning. Releases can be found in the releases sections. You can also watch this repo to be notified of any upcoming changes.

Validation

Data can be validated against the schema using any of these open source validators.

Contributing

Contributions are very much welcome, please submit a pull request.