Giter Site home page Giter Site logo

siglens / siglens Goto Github PK

View Code? Open in Web Editor NEW
997.0 9.0 146.0 10.81 MB

100x Efficient Log Management than Splunk :rocket: Reduce your observability cost by 90%

Home Page: https://www.siglens.com

License: GNU Affero General Public License v3.0

Go 77.57% HTML 4.31% CSS 3.95% JavaScript 12.88% Makefile 0.01% Dockerfile 0.03% Shell 1.21% Python 0.05%
distributed-tracing go hacktoberfest log-management log-search logging logs monitoring newrelic observability

siglens's Introduction


Build Status Go Report Card GoDoc codecov

Twitter RSS LinkedIn

English | 简体中文

Open Source Observability that is 💥💥 100x 💥💥 more efficient than Splunk

Single binary for Logs 🎯, Metrics 🎯 and Traces 🎯.

Cut down your Splunk bill by ⚡ ⚡ 90% ⚡ ⚡

Why SigLens:

Our experience servicing 10,000+ engineers with Observability tools taught us a few things:

  • Developers have to jump through different tools for logs, metrics, traces
  • Splunk, DataDog, NewRelic are very expensive 💸 💸 💸
  • ElasticSearch takes too many machines, cluster maintenance is hard 👩‍💻👩‍💻
  • Grafana Loki has slow query performance 🐌🐌

Armed with decades of experience in monitoring domain, we set out to build a observability DB from the ground up, uniquely suited for logs, metrics and traces with zero external dependencies. A single binary that you can run on your laptop and process 8 TB/day.


Setup

Installation

Git   |   Docker  |   Helm

Documentation

Docs

Differentiators

SigLens v/s Splunk,Elastic,Loki

Check out this blog where SigLens ingested data at 1 PB/day rate for 24 hours on a mere 32 EC2 instances compared to 3000 EC2 instances required for Splunk, Elastic, Grafana Loki

SigLens v/s Elasticsearch

Check out this blog where SigLens is 1025x Faster than Elasticsearch 🚀🚀

SigLens v/s ClickHouse

Check out this blog where SigLens is 54x Faster than ClickHouse 🚀🚀


Features:

  1. Multiple Ingestion formats: Open Telemetry, Elastic, Splunk HEC, Loki
  2. Multiple Query Languages: Splunk SPL, SQL and Loki LogQL
  3. Simple architecture, easy to get started.

Join our Community

Have questions, ask them in our community Slack 👋


Contributing

Please read CONTRIBUTING.md to get started with making contributions to SigLens.

How-Tos

Searching Logs

Searching Logs

Tracing

Tracing

Creating Dashboards

Creating Dashboards

Creating Alerts

Creating Alerts

Live Tail

Live Tail

Minion Searches

Minion Searches

Code of Conduct

Please review our code of conduct before contributing.

Thanks to all contributors for their efforts

siglens's People

Contributors

andrewhess avatar anhlee66 avatar ashutoxh avatar davleen14 avatar dependabot[bot] avatar el-psy-kongrooo avatar girish332 avatar jiale-fang avatar kartikeyhebbar avatar kashmii avatar ksankeerth avatar macbeth98 avatar nagarjunma avatar nbhavana avatar nikitha-kambhampati avatar nkunal avatar parichaydidwania avatar pratikshapatil77 avatar sanchitd9 avatar saniya16 avatar shivasaicharanruthala avatar shyam2520 avatar smaftoul avatar sonamgupta21 avatar sriya03 avatar sunitakawane avatar tufank avatar venkateshamatam avatar xmh1011 avatar zhifei-li avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

siglens's Issues

SPL: fillnull

Description

SPL uses the fillnull command to replace null values with teh specified default value. See https://docs.splunk.com/Documentation/Splunk/9.1.0/SearchReference/Fillnull

You should make changes to the SPL parser in spl.peg, then install pigeon, and compile spl.go by running

pigeon -o pkg/ast/spl/spl.go pkg/ast/spl/spl.peg

You should update or create a struct in evaluationstructs.go to handle the fillnull command. Then your implementation of the fillnull logic should probably be inside segaggs.go

Testing

Manual Testing

Ingesting Data

For ingesting data with null values, you can follow the instructions in the description of #38 for ingesting traces from opentelemtry-demo. If you want to remove data ingested into siglens, you can do this by clearing the siglens/data/ directory--you should probably do this only when siglens is not running.

Querying data

After ingesting some data, you can run some basic manual tests in the siglens UI by searching for something like * | fillnull value="hello world" and checking the results.

Unit Testing

  • Add unit tests to splParser_test.go to ensure that this command gets parsed correctly. Make sure you handle all optional parameters.

Goals

  • Implement fillnull for Splunk QL and properly handle the optional value and field-list parameters (see the Splunk documentation linked above).
  • Implement tests mentioned above in the Unit Testing section. Feel free to add other unit tests as appropriate

Blog for auto-instrumenting a JavaScript app

Description

Using https://opentelemetry.io/docs/instrumentation/js/ as a reference, write a blog guiding someone with an existing JavaScript app to use OpenTelemetry to auto-instrument it and view the traces in siglens. Also provide a minimal app that someone could use to go through this tutorial if they don't have an existing app.

You may want to look at the description of PR #38 which explains how to properly setup the endpoint so opentelemetry-demo can send traces to siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

Blog for auto-instrumenting a Go app

Description

Using https://opentelemetry.io/docs/instrumentation/go/libraries/ as a reference, write a blog guiding someone with an existing Go app to use OpenTelemetry to instrument it and view the traces in siglens. OpenTelemetry doesn't yet have full auto-instrumentation for Go, so you'll have to guide them on how to auto-instrument certain libraries and point them to where the can find which Go libraries have auto-instrumentation. Also provide a minimal app that someone could use to go through this tutorial if they don't have an existing app.

You may want to look at the description of PR #38 which explains how to properly setup the endpoint so opentelemetry-demo can send traces to siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

Blog for auto-instrumenting a Java app

Description

Using https://opentelemetry.io/docs/instrumentation/java/ as a reference, write a blog guiding someone with an existing Java app to use OpenTelemetry to auto-instrument it and view the traces in siglens. Also provide a minimal app that someone could use to go through this tutorial if they don't have an existing app.

You may want to look at the description of PR #38 which explains how to properly setup the endpoint so opentelemetry-demo can send traces to siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

SQL: Add EXTRACT for year

We want to add support for SQL EXTRACT as described here: https://www.postgresqltutorial.com/postgresql-date-functions/postgresql-extract/ For this issue, we only need to implement it for extracting the year.

We do SQL parsing in pkg/ast/sql/astsql.go but you may also need to look into github.com/xwb1989/sqlparser as we use that to do some of the parsing. If #67 is done, you should look at how that was implemented. Otherwise, you may want to look at how we implemented some evaluation functions for Splunk QL; the parsing is in pkg/ast/spl/spl.peg and evaluation is done in pkg/segment/aggregations/segaggs.go with some lower-level parts implemented in pkg/segment/structs/evaluationstructs.go.

create install_with_binary.sh script

Create a installation script in bash. It should do the following steps:

  1. Detect the current OS and set the var OS to: ("linux" for linux and "darwin" for mac)
  2. Detect the arch type and set the arch to: (arm64 OR amd64)
  3. Download the "latest_version" file from https://github.com/siglens/siglens/blob/main/pkg/config/version.go This file will contain the version number of the latest release
  4. Construct the download url : <TBD-$latest_verion.$OS.$arch>
  5. Download the above url
  6. tar -xvf above url file name
  7. cd above url file name without the gz
  8. PORT=80 ./siglens --config server.yaml
  9. Display a msg that SigLens is running on port http://localhost:80
  10. If above port 80 is not accessible then ask the user to run PORT=8090 ./siglens --config server.yaml
  11. Before you execute each step, providing feedback on the terminal saying what you are going to do next.

Alert Silencing - Backend changes

We want to add functionality to silence alert for a certain number of hrs.
User can set silence period i.e. number of hrs that alert notification should not be sent. This can be set on UI and sent to backend using
/alerts/silenceAlert API. This will update the alert table.

  • Need to change Alert table definition to include silence_period column.
  • Add new API /alerts/silenceAlert

Alert table has a column, last_sent_time. Every time the alert is triggered , we should check if there is silence_period set for this alert and
has it elapsed i.e. if last_sent_time + silence_period < current_time , then send alert notification. Else alert state is firing but no notification is sent since silence_period is not over.

Make readme to run siglens on Heroku

This relies on #31 being completed because Heroku will set the PORT environment variable and forward http traffic to that port

The basic steps are:

  1. Make a Heroku account
  2. Install Heroku CLI
  3. Clone the siglens repository
  4. Make a new branch called main
  5. Run echo "web: ./bin/siglens --config server.yaml" > Procfile
  6. git add Procfile
  7. git commit
  8. git push heroku main
  9. From the siglens repository, run git push heroku main
  10. Run heroku open to open the siglens web app in your browser

TODO: You may also need to update the GetIngestPort() function to read the $PORT env variable. The end goal is that you can ingest data into siglens and then when you run heroku open you can search the data you ingested.

Comparison with ClickHouse is fake

I've read this article: https://www.sigscalr.io/blog/sigscalr-vs-clickhouse.html and made the following observations:

  1. You supposedly tested your system on im4gn.2xlarge, but you didn't test ClickHouse - the numbers are copied from the article from 2017: https://tech.marksblogg.com/billion-nyc-taxi-clickhouse.html and other articles on that website.
  2. You provide SQL queries for ClickHouse but don't provide the queries for SigLens, making the benchmark not reproducible.

Blog for auto-instrumenting a Python app

Description

Using https://opentelemetry.io/docs/instrumentation/python/ as a reference, write a blog guiding someone with an existing Python app to use OpenTelemetry to auto-instrument it and view the traces in siglens. Also provide a minimal app that someone could use to go through this tutorial if they don't have an existing app.

You may want to look at the description of PR #38 which explains how to properly setup the endpoint so opentelemetry-demo can send traces to siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

SQL: Add Round

We want to support using ROUND in SQL queries as described here: https://www.postgresqltutorial.com/postgresql-math-functions/postgresql-round/

We do SQL parsing in pkg/ast/sql/astsql.go but you may also need to look into github.com/xwb1989/sqlparser as we use that to do some of the parsing. You may also want to look at how we implemented round() for Splunk QL by parsing it in pkg/ast/spl/spl.peg and evaluating it in pkg/segment/structs/evaluationstructs.go, which we call in pkg/segment/aggregations/segaggs.go

Feature Enhancement : Add Column Value Filtering in Logs Table

Description:

Add the ability to filter column values in the logs table.

Feature Details:

  • Add a filter (funnel) icon next to a column title.
  • Clicking on the icon shows the filter options for that column.
  • Users can:
    • Click the checkbox next to the values they want to display.
    • Enter text in the search field at the top to filter values and show only those matching the entered text. This will make it easier for users to locate and select specific values without the need to scroll through the entire list.

We are using ag-Grid for the logs table, and the following link provides guidance on implementing similar features: ag-Grid Filtering Documentation.

Reference Screenshot:

image

UI : Show Contact Name in Delete Confirmation Popup

Description:

When deleting a contact point, the current confirmation popup provides a generic message for contacts used in alerts and a standard confirmation for others. To enhance user understanding, we want to show the name of the contact point being deleted.

Current Behavior:

  • If a contact point is used in alerts:
    Contact point cannot be deleted because it is used in more alerts. Please update or delete these alerts first.
  • Otherwise:
    Are you sure you want to delete this contact point?

Steps to Reproduce:

  1. Navigate to the Alerting screen and go to the Contact Point tab.
  2. Add a new contact point.
  3. Initiate the delete action.
  4. Observe the confirmation popup.

Expected Behavior:

The confirmation popup should display the name of the contact point being deleted, such as:

  • If a contact point is used in alerts:
    The contact point named 'ContactPointName' is currently in use by multiple alerts.
  • Otherwise
    Are you sure you want to delete the 'ContactPointName' contact point?

Screenshot:

image image

DataDog Ingest Interface

Implement DataDog ingestion method. Existing DD users can just point their DD agent to SigLens and SigLens must be able to ingest their logs/events/metrics

Alert History Table (with state changes, config changes)

When a user clicks on a specific alert, we should show a tab that shows "History". The history is just chronolgical sequence of:

  1. Every time the alert was fired or went normal
  2. Any config change in the alert

This table should show 3 columns:

  1. Timestamp of the event
  2. Event Description (alert fired, alert normal, config change)
  3. Username (user who changed the config, or show "system generated" when it is a firing/normal state change

Blog for auto-instrumenting a .NET app

Description

Using https://opentelemetry.io/docs/instrumentation/net/ as a reference, write a blog guiding someone with an existing .NET app to use OpenTelemetry to auto-instrument it and view the traces in siglens. Also provide a minimal app that someone could use to go through this tutorial if they don't have an existing app.

You may want to look at the description of PR #38 which explains how to properly setup the endpoint so opentelemetry-demo can send traces to siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

install_with_git.sh script

Create a installation script in bash. It should do the following steps:

  1. Check if git is installed (if not prompt the user to install git locally and have them create a GitHub account, point to this link to help them out: https://docs.github.com/en/get-started/onboarding/getting-started-with-your-github-account)
  2. then mkdir siglens-contrib
  3. then cd siglens-contrib
  4. Then perform git clone [email protected]:siglens/siglens.git
  5. then cd siglens
  6. then go mod tidy
  7. Check if the current user is allowed to open. port 80 (google how to do this)
  8. If yes then do go run cmd/siglens/main.go -config server.yaml
  9. if not then do PORT=8090 go run cmd/siglens/main.go -config server.yaml
  10. Display a msg that SigLens is running on port http://localhost:80 or port http://localhost:8090 depending on above 2 steps

Before you execute each step, providing feedback on the terminal saying what you are going to do next. for e.g: "Checking if git exists", "Creating local dirs and cloning git repo", "Downloading go dependencies", "Checking if port 80 is allowed", "Now runing server on port xx", etc...

Send SigLens internal metrics to itself.

Currently we use prometheus to scrape internal metrics at regular intervals. These are stored in prometheus time series database. OTEL collector is configured to forward these metrics from prometheus to siglens metrics ingest endpoint. This allows us to use these metrics in our self monitoring /default dashboards.
We want to add a cron job to send siglens metrics to itself without using prometheus/ OTEL collector.

Siglens instrumentation package collects internal metrics pkg/instrumentation/metrics.go
we use https://github.com/prometheus/client_golang

Prometheus set up:

  • Download prometheus https://prometheus.io/download/
  • Once Prometheus is started with default settings, its web UI will be listening at port 9090:
  • Promentheus will scrape metrics from local siglens instance. Configure that using /prometheus/prometheus.yml
  scrape_interval:     15s
  evaluation_interval: 15s

- job_name:       'siglens'
    static_configs:
      - targets: ['localhost:2222']
        labels:
          group: 'production'

OTEL collector set up

  • get the MacOS release using: curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.80.0/otelcol_0.80.0_darwin_amd64.tar.gz
  • They are packaged as gzipped tarballs (.tar.gz) and will need to be unpacked with a tool that supports this compression format: tar -xvf otelcol_0.80.0_darwin_amd64.tar.gz
  • Every Collector release includes an otelcol executable that you can run after unpacking.
  • Create a config.yaml file which will contain the configurations for the otelCol
  • Run otelCol using: ./otelCol --config config.yaml

The config file used:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'siglens'
          scrape_interval: 60s
          static_configs:
            - targets: ['localhost:2222']

processors:
  batch:

exporters:
  logging:
    loglevel: debug

  prometheusremotewrite:
    endpoint: 'http://localhost:8081/promql/api/v1/write'
    headers:
      Authorization: 'Bearer YOUR_TOKEN_HERE'  # Optional: Add any necessary headers

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]

Rename functions in segsearch and searchaggs

Description

The segsearch.go and searchaggs.go files define several functions named things like like applyXYZ but their naming is a bit inconsistent. Rename them so they are more clear and consistent. You should use the query type and flow when deciding on the name.

Goals

  • For each of the applyXYZ functions, propose a new name or to keep it as it is.
  • Once these proposed names are approved, change the function names.

Hide internal indexes when searching logs

Description

Update ExpandAndReturnIndexNames() in resolveIndexHandler.go so that the internal indexes are hidden when searching logs. These indexes are:

  • traces
  • red-traces
  • service-dependency

Goals

  • Update ExpandAndReturnIndexNames() as mentioned.
  • Update the unit tests in resolveIndexHandler_test.go to test your changes.
  • Verify that when you ingest some traces (see the description of PR #38 for instructions) and logs and then go to the siglens UI -> Logs, in the Index dropdown you see the indexes for the logs but don't see any of the internal trace indexes. If you see the internal indexes, you should make any additional changes so that those are hidden

Querying of boolean values

Description

Currently, any query with true will always consider it a string and will try to search accordingly.

We should also support for true and false queries to match boolean values as well. We can have true match both string and bool .

We should make necessary edits to segsearch so that we can query boolean values.

Handle columns with bool/integer/string values

Description

PR #73 fixed an issue where siglens would crash if we ingested strings and numbers into one column. However, we're still not properly handling cases where a column has both booleans and strings. You will address this in the following steps:

  1. During ingestion, track the types of values being stored in each column.
  2. When the wipBlock is flushed, check each column to determine which types it contains. If a column contains multiple types, consolidate them into a single type following these rules:
    • The null type (VALTYPE_ENC_BACKFILL) can coexist with any other type.
    • For columns with booleans and numbers: convert all values to strings.
    • For columns with booleans and strings: first, try to convert all values to booleans. If that fails, convert all values to strings.
    • For columns with numbers and strings: (this is already handled by PR #73) try converting all values to numbers; if any conversion fails, convert everything to strings.
    • For columns with booleans, numbers, and strings: convert all values to strings.

Testing

For quick testing during development, you can add something like the below somewhere it will be run (perhaps in main.go after setting up the server)

now := utils.GetCurrentTimeInMs()
indexName := "traces"
shouldFlush := false
localIndexMap := make(map[string]string)
orgId := uint64(0)

jsons = [][]byte{
	[]byte(`{"age": "171"}`),
	[]byte(`{"age": 103}`),
	[]byte(`{"age": 5.123}`),
	[]byte(`{"age": true}`),
	[]byte(`{"age": "181"}`),
	[]byte(`{"age": 30}`),
	[]byte(`{"age": 6.321}`),
	[]byte(`{"age": false}`),
	[]byte(`{"age": "hello"}`),
}

for _, jsonBytes := range jsons {
	err := writer.ProcessIndexRequest(jsonBytes, now, indexName, uint64(len(jsonBytes)), shouldFlush, localIndexMap, orgId)
}

Then you can run the server and query it (once the wipBlock gets flushed) with

curl -X POST -d '{"searchText": "*", "startEpoch": "now-15", "endEpoch": "now", "index": "traces", "queryLanguage": "Splunk QL"}' http://localhost/api/search

You may want to clear the data between runs; you can do this by deleting the data/ directory

Goals

  1. As described above, handle the cases where a column has values of multiple types
  2. Add unit tests

Alert Silencing - UI changes

We want to add functionality to silence alert for a certain number of hrs.
On AlertRules page, where there is list of alert rules, we have actions column in the end. On clicking

Add a new button before view button. On click on this button open a popup with heading (silence range) and show options like ( next 5 mins, 15 mins, 30mins, next hr, next 3hrs, next 12 hrs, next 24hrs).

On selecting #hrs , call /alerts/silenceAlert API to send this data to backend.

Add README for adding selenium tests

The README should give some instructions and/or references for how to get started with writing selenium tests for people who haven't done that before and also instructions on how to run the tests.

We have some selenium tests in the selenium-tests/tests folder.

How to run selenium tests:

  1. Prerequisite
    1. npm install --save selenium-webdriver chromedriver geckodriver
    2. npm install -g [email protected]
  2. To run individual test (All the test files are in the selenium-tests/tests folder)
    1. node selenium-tests/tests/
      for e.g. node selenium-tests/tests/panelEditScreen-test.js
  3. To run all the tests
    1. cd selenium-tests
    2. npm run test

UI : Show Alert Rule Name in Delete Confirmation Popup

Description

Currently, when attempting to delete an alert rule, the confirmation popup only displays a generic message:
Are you sure you want to delete this Alert?
To provide more context, we want to include the name of the alert rule in the confirmation popup.

Steps to Reproduce:

  1. Navigate to the Alerting screen.
  2. Add a new alert rule.
  3. Initiate the delete action.
  4. Observe the confirmation popup.

Expected Behavior:

The confirmation popup should display the name of the alert rule along with the generic confirmation message, like:
Are you sure you want to delete the "AlertRuleName" alert?

Screenshot:

image

Add selenium tests for alert pages

Similar to how the tests in selenium-tests/tests have tests for various pages of our site, we should add tests for the alert pages:

  • all-alerts.html
  • alert.html
  • contacts.html

See the selenium tests readme (which should be added by #33) for information on getting started with writing selenium tests.

LogQL- Implement Loki LogQL chained line/label queries

Currently, siglens supports LogQL queries like duration >= 20. One line & one label filter queries are also supported.
We need to add support for chained line/label queries, such as:
duration >= 20 or (method="GET" and size <= 20)
https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression

make sure all equivalent requests are supported as well
| duration >= 20 or size == 20 and method!"2.."
| duration >= 20 or size == 20 | method!"2.."
| duration >= 20 or size == 20 , method!"2.."
| duration >= 20 or size == 20 method!"2.."

We do LogQL parsing in pkg/ast/logql/logql.go
Other logQL query parsing is implemented in pkg/ast/logql/logql.peg

Tracing : Dependency Graph

When the user clicks on the tracing icon in the side navbar and navigates to the "Dependency Graph" tab, display the overall call paths of the spans.

Call an API (api/traces/dependencies) to get the matrix of service path counts, and then create a force-directed graph using a library like D3.js.
Root nodes should be positioned at the top of the graph.

endpoint: api/traces/dependencies
method: GET
Example:
request: http://localhost:80/api/traces/dependencies
response: 
    {[
		{
			"from": string (parent service),
			"to": []int (N elements, where N is the number of services. Each value is the number of calls from "from" to "to"
		},
		...
    ]}

Similar to below image -
image

SPL: limit head by condition

Description

SPL uses the head command to limit the number of results. See https://docs.splunk.com/Documentation/Splunk/9.1.0/SearchReference/Head

One way to limit results is to specify a boolean expression and then stop sending results once that expression becomes false.

When there's multiple segments being searched, we parallelize searching them. So to properly handle this head command, you can do the head logic on each segment (but keep one record past what head should return) and then merge the results and do the head logic on that.

Testing

Manual Testing

Ingesting Data

To ingest some fake data into siglens for testing. Clone https://github.com/sigscalr/sigscalr-client and then from within that repo run

go run main.go ingest esbulk -t 10_000 -d http://localhost:8081/elastic --processCount 1 -n 1 -b 500 -g dynamic-user

while siglens is running. You can also run

go run main.go ingest esbulk -n 10 -g benchmark -d http://localhost:8081/elastic -t 100_000

to ingest the data that is used for the CICD tests. You can clear the siglens/data/ folder to remove all the data from siglens (you should probably do this only when siglens is not running).

Querying data

After ingesting some data, you can run some basic manual tests in the siglens UI by searching for something like city=Boston | head weekday!=Friday and then comparing against a search that matches everything (a blank search will do this), and then checking that your head search stopped when it was supposed to.

Unit Testing

  • Add unit tests to splParser_test.go to ensure that this command gets parsed correctly.
  • Add CICD tests in cicd/ingest.csv and cicd/restart.csv

Goals

  • Implement head <eval-expression> for Splunk QL and properly handle the optional null and keeplast parameters (see the Splunk documentation linked above). You'll implement the parsing in spl.peg and may need to update or add a struct in evaluationstructs.go to handle head. Then the logic for performing head will probably be in segaggs.go, unless you find a more suitable place.
  • Implement tests mentioned above in the Unit Testing section. Feel free to add other unit tests as appropriate

Generate query audit log

Update the backend so that whenever a query is run, information is saved to the file access.log. This file will be similar to an Apache access log file and should include the following information:

  • current timestamp
  • logged in user
  • query URL
  • query post body params
  • query success/failure(status code)
  • time taken for query response

A log line will need to be added whenever the /api/search or /api/search/ws endpoints are accessed. The pkg/server/query/server.go file shows which functions are called when these endpoints are accessed.

UI : Remove "Eye Icon" from Actions Column in Alert Rules Table

Description:

The current implementation displays a detailed overview of a specific alert when the user clicks on the "eye icon" in the actions column. However, we have the same functionality when the user clicks on the specific alert row. Therefore, we want to remove the "eye icon" from the actions column.

Steps to Reproduce:

  1. Navigate to the Alerting screen.
  2. Add a new alert rule.
  3. Observe the alert rules table; the created alert is displayed with an "eye icon" in the actions column.
  4. Click on the "eye icon" to view the detailed overview.

Screenshot:

image

Note:

Remove the "eye icon" functionality from the actions column for newly created alerts. Ensure that the detailed overview functionality is still accessible by clicking on the specific alert row.

Blog for connecting OTEL collector to siglens

Description

Write a blog guiding someone on how to setup siglens and then connect their OTEL collector to siglens in order to send telemetry data to siglens. You can use the https://github.com/open-telemetry/opentelemetry-demo repo for setting up a test app and OTEL collector. Also look at the description in PR #38 as it has instructions on how to properly configure the collector to send traces to siglens. Your blog should provide detailed instructions as well as screenshots for how to view/search the data once it's in siglens.

Goals

  • Write the blog as a markdown document in the siglens-docs repo in the docs/ folder (or a subfolder of that)
  • Go through the steps yourself to ensure that they work and are easy to follow.

UI : Show Alerts Associated with Contact Point in Delete Confirmation Popup

Description:

When attempting to delete a contact point, we want to enhance the confirmation popup to include a button or text that allows users to view all alerts associated with the contact point.

Changes:

  1. Add a button or text in the confirmation popup: "View alerts associated with this contact point."
  2. When the user clicks on the button or text, display a scrollable container within the same popup showing all alerts associated with the contact point.

Confirmation Popup:

Cannot delete contact point 

The contact point named 'ContactPointName' is currently in use by multiple alerts.

[View alerts associated with this contact point]   [Close]

On Clicking "View alerts associated with this contact point":

[Alert 1]
[Alert 2]
[Alert 3]
[...]

Utilize the getAllAlertsWithSameContactPoint function from contact.js to retrieve the alerts associated with contact point.

ScreenShot:
image

UI : Colored Backgrounds for Alert States

Description:

We want to add colored backgrounds for different states of alerts.

Expected Behavior:

Normal State: Green background. (#53DB6E)
Firing State: Red background. (#F55759)
Pending State: Yellow background. (#F4BB20)

Changes:

Add colored backgrounds to the alert states in the alert table.
Implement the same color scheme for alert states in the alert details view.

Reference Screenshot:

image

Get query port from environment variable

Update GetQueryPort() in config.go. If the PORT environment variable is set, GetQueryPort() should return that value. Otherwise, it should return the value of the queryPort in the config file (e.g., server.yaml).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.