skelmis/logoo

A Python package for easy openobserve log ingestion

Logoo

A log ingestion engine for openobserve

Pronounced log-ew

pip install logoo

Examples

Primary file. One instance of PrimaryLogger is required per program.

import asyncio

from logoo import PrimaryLogger


async def main():
    logger: PrimaryLogger = PrimaryLogger(
        __name__,
        base_url="",
        org="",
        stream="",
        username="",
        password="",
        poll_time=5,
    )
    await logger.start_consumer()

    logger.info("Hello world!")
    logger.critical("Something went wrong!")
    await asyncio.sleep(10)


asyncio.run(main())

Any other file:

from logoo import Logger

logger = Logger(__name__)
logger.info("This comes from another file.")

Note that this supports argument interpolation and can be a fairly good drop in logging replacement:

from logging import Logger as built_in_logger
from logoo import Logger

log = built_in_logger(__name__)
logger = Logger(__name__)

log.info("%s", "Test")
logger.info("%s", "test")

Documentation is as follows and remarkably simple:

Parameters
----------
name: str
    The name of the logger
base_url: str
    Your base openobserve URL.

    E.g. `base_url="https://logs.example.com"`
org: str
    The org to make logs under
stream: str
    The stream for these logs to be ingested under
username: str
    The username to use for auth
password: str
    The password to use for auth
logs_per_call: int
    How many logs to send per request to your instance.
    Useful to set, so you don't hit things like WAF request
    limits in high throughput environments.
    
    Note logoo makes as many requests as required
    to exhaust the queue every `poll_time`.

    Defaults to `100`.
poll_time: datetime.timedelta | float
    How often to send all logs to your instance.

    Defaults to every `30` seconds.
extra_metadata: dict
    Extra metadata to add to all logs made by this class
global_metadata: dict
    Extra metadata to add to every log sent to your instance

---

Support

Want realtime help? Join the discord here.

---

License

This project is licensed under the MIT license

logoo's Issues

When logs fail to upload, requeue

Currently if an upload fails for any reason the logs that the API call attempted to upload are simply lost. This is not ideal. We should ideally throw them back onto the queue for the next round of requests