enhancifire/dyte-sde1-assignment

Log Searcher 10k

Dyte SDE Assessment Project

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contact

About The Project

Log Ingester and Query Tool built using Django for Dyte SDE Assessment.

(back to top)

Built With

• Python
• Django
• PostgreSQL
• SQLite3
• Docker

(back to top)

Getting Started

The project can be run using a local SQLite database, or using Docker. Both methods are described below.

Prerequisites

• Python 3.10 or higher
• Docker (optional)

Installation

Without Docker

1. Clone the repo

   git clone https://github.com/dyte-submissions/november-2023-hiring-Enhancifire.git

2. Create virtual environment

   python -m venv .env

3. Activate the environment

   source .env/bin/activate.fish # for fish shell

4. Install requirements

    pip install -r requirements.txt

5. Run migrations

    python manage.py migrate

6. Run the server

    python manage.py runserver 3000

With Docker

1. Clone the repo

   git clone https://github.com/dyte-submissions/november-2023-hiring-Enhancifire.git

2. Switch branch to docker

   git checkout docker

3. Start DB and build docker image

   docker compose up db -d

4. Build and Start the server

   docker compose up server --build

5. Run migrations

   docker compose exec server python manage.py migrate

6. Access the server at localhost:3000

(back to top)

Usage

Access the server at localhost:3000

Endpoints:

• '/': Home page Query interface is accessible here

• '/logs/': Ingest point for logs POST request to add individual logs to the database

• '/logsearch/': Search endpoint for logs Query interface accesses this endpoint to search for logs

(back to top)

System Design

(back to top)

Features Implemented

Ingestor:

• Develop a mechanism to ingest logs in the provided format.
• Ensure scalability to handle high volumes of logs efficiently.
• Mitigate potential bottlenecks such as I/O operations, database write speeds, etc.
• Make sure that the logs are ingested via an HTTP server, which runs on port 3000 by default.

Query Interface:

• Offer a user interface (Web UI or CLI) for full-text search across logs.
• Include filters based on:
  • level
  • message
  • resourceId
  • timestamp
  • traceId
  • spanId
  • commit
  • metadata.parentResourceId
• Aim for efficient and quick search results.

Advanced Features:

• Allow combining multiple filters.
• Provide real-time log ingestion and searching capabilities.
• Distributed systems or cloud-based solutions can ensure robust scalability.
• Implement role-based access to the query interface.
• Implement search within specific date ranges.
• Utilize regular expressions for search.

Identified Issues

• Manual migration for Postgres required in docker usecase.
• Possible bottleneck of single threaded django instance.
• Static metadata field parameter. Adding additional data to metadata requires conversion from foreign key to BSON.

Contact

Faiz Saiyad - [phone] - [email protected]

Project Link: https://github.com/dyte-submissions/november-2023-hiring-Enhancifire

(back to top)