zinclabs / zinc Goto Github PK
View Code? Open in Web Editor NEWZincSearch . A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Home Page: https://zincsearch-docs.zinc.dev
License: Other
ZincSearch . A lightweight alternative to elasticsearch that requires minimal resources, written in Go.
Home Page: https://zincsearch-docs.zinc.dev
License: Other
Is there a more detailed explanation of the search types somewhere?
Are the named search types already implemented in a functional way?
https://github.com/johannesboyne/gofakes3
To allow testing
Add summary aggregation function for zinc, which can help summarize the number of certain keyword in the search results
Maybe this is more observation, but still - when particular value is null, I receive 500. I do not know what would be correct behaviour, ESS survives exactly this scenario.
When for particular key "null" value is assigned on indexing.
As:
185 3.377123217 127.0.0.1 127.0.0.1 HTTP 505 POST /es/product_v2/_doc HTTP/1.1 (application/json)
{"product_id":"eabf69fbe53f57a4ef6fabd6af8430ba","product_type":"car","car_manufacturer":null,"car_model":"\"VOLUNTEER 4X2\"","car_year":"2011","car_gear_type":null,"weight":0}
I received back: 500 Internal error with following output from zinc:
2021/12/28 18:00:38 [Recovery] 2021/12/28 - 18:00:38 panic recovered:
POST /es/product_v2/_doc HTTP/1.1
Host: localhost:4080
Accept: application/json
Authorization: *
Content-Length: 176
Content-Type: application/json
User-Agent: elasticsearch-php/6.7.1 (Linux 5.11.0-43-generic, PHP 7.4.3)
interface conversion: interface {} is string, not float64
/usr/local/go/src/runtime/iface.go:261 (0x40a374)
/Users/prabhat/projects/zinc/zinc/pkg/core/Index.go:77 (0x957df4)
/Users/prabhat/projects/zinc/zinc/pkg/core/UpdateDocument.go:5 (0x95a1a6)
/Users/prabhat/projects/zinc/zinc/pkg/handlers/UpdateDocument.go:47 (0x96de96)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x967c27)
/Users/prabhat/projects/zinc/zinc/pkg/auth/AuthMiddleware.go:21 (0x967a6e)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x891fa1)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x891f8c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x890e5d)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x890ae5)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x890644)
/usr/local/go/src/net/http/server.go:2879 (0x668eda)
/usr/local/go/src/net/http/server.go:1930 (0x664587)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x463580)
First thanks for creating Zinc. For now, its amazing for small applications and I am sure more features are coming.
I was playing with Zinc after reading your article. I tested with data with email addresses. There are no issues with writing/reading documents but on search it does not seems to consider email. I tried to encode email address and tried few ways and found that search does not work as soon as I add dot "." any data.
I tried to debug from source but my skill is limited with go lang so its not much help.
Thanks
Adding the "DELETE" option to the Zinc API would allow users to "clean up" unused/old indices. This should clean up the disk space and metadata around the indexes.
I'd love to be able to use the same ID between my SQL DB and zinc
. However, when I upload with the _id
field set, zinc
panics:
$ curl -u admin:Complexpass#123 -XPUT -d '{"author":"Prabhat Sharma", "_id": 1}' http://localhost:4080/api/myshinynewindex/document
2021/12/16 20:07:50 [Recovery] 2021/12/16 - 20:07:50 panic recovered:
PUT /api/myshinynewindex/document HTTP/1.1
Host: localhost:4080
Accept: */*
Authorization: *
Content-Length: 37
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.77.0
interface conversion: interface {} is float64, not string
/usr/local/go/src/runtime/iface.go:261 (0x1009f94)
/Users/prabhat/projects/raelabs/zinc/pkg/handlers/UpdateDocument.go:25 (0x1571566)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x156bbc7)
/Users/prabhat/projects/raelabs/zinc/pkg/auth/AuthMiddleware.go:21 (0x156ba0e)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x1497121)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x149710c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x1495fdd)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x1495c65)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x14957c4)
/usr/local/go/src/net/http/server.go:2878 (0x126e47a)
/usr/local/go/src/net/http/server.go:1929 (0x1269b27)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x10634a0)
I try to insert data (from a little file of 37M) but zinc get killed by OOM.
It eat more than 8GB before be killed by kernel.
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected],task=zinc,pid=129244,uid=1000
[13291.887950] Out of memory: Killed process 129244 (zinc) total-vm:12128240kB, anon-rss:11309292kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:22328kB oom_score_adj:0
[13292.196257] oom_reaper: reaped process 129244 (zinc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[GIN-debug] Listening and serving HTTP on :4080
{"level":"debug","time":"2021-12-15T18:49:58+01:00","message":"auth: success"}
Killed
Whenever I try to post data I get the message "404 page not found". This is with version 0.1.1 on Windows 10. I can access the web UI without problems.
This is what I did:
zinc.exe
set FIRST_ADMIN_USER=admin
set FIRST_ADMIN_PASSWORD=Pass123
data
subdirectoryzinc.exe
http://localhost:4080/ui/
I unsuccessfully tried to post data to the following URLs:
Am I missing something?
So far I've been playing with Zinc and it's been working great.
It would be cool if we could have a log system like Logtash for zinc.
I've always mainly been using Elastic for a log system for multiple machines.
https://github.com/opensearch-project/opensearch-go is an elastic alternaitve.
Seems perfect match ?
I save the data into zinc through a python script, and when I search for a complete word, the result is correct. But when searching for a part of the word, no results were found. Is this problematic?
In some cases it's required to have a string indexed as keyword
instead of just text
.
Could be solved like this in a single record:
data = {
"Athlete": "DEMTSCHENKO, Albert",
"Country": {
"mapping": "keyword",
"value": "RUS"
},
// ..
}
Or like this in a bulk ingestion file:
{ "index" : { "_index" : "olympics" } }
{"Athlete": "HAJOS, Alfred", "Country": {"mapping": "keyword", "value": "HUN"}, /* .. */}
Would it be possible to have an armv6 Docker image also? Thank you.
I wish I could mashup this project and quickwit:
Quickwit allows true decoupled compute and storage. We designed it to search straight from object storage like AWS S3 in a stateless manner.
I love that zinc
comes with a Kibana replacement -- a built in UI!
If zinc could also store all its data directly in S3 and decouple storage and compute, you'd have replaced the 99% percent use case of ES/Kibana with a vastly superior technology that is both free as in beer and tshirts, easy to operate, hard to lose data on, and effortless to deploy.
First of all, this is a really cool project, with really impressive response times in searches.
According to the docs there are currently a thew ES APIs implemented.
I'd like to ask if compatibility, I mean in regards of a drop-in replacement for workloads where ES is the backend, is intended?
Are there already experiences with deployments where ES is normally used as a backend like Jaeger, OpenCTI or other projects?
Are there any plans for implementation of relevance tuning?
Here's something similar with elastic to show what I'm talking about.
Is it possible to search in all indices?
and
is it possible to search without start and end date?
Login pages shows OK when accessing to http://192.168.56.112:4080, but API_ENDPOINT is configured to localhost (index.js:6) and pressing "Sing in" button can get the response.
Any way to modify this without recompiling?
Good job!!! Thank you in advanced
When starting zinc
I see the following:
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /healthz --> github.com/prabhatsharma/zinc/pkg/meta/v1.GetHealthz (3 handlers)
[GIN-debug] GET / --> github.com/prabhatsharma/zinc/pkg/meta/v1.GUI (3 handlers)
[GIN-debug] GET /ui/*filepath --> github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] HEAD /ui/*filepath --> github.com/gin-gonic/gin.(*RouterGroup).createStaticHandler.func1 (3 handlers)
[GIN-debug] POST /api/login --> github.com/prabhatsharma/zinc/pkg/handlers.ValidateCredentials (3 handlers)
[GIN-debug] PUT /api/user --> github.com/prabhatsharma/zinc/pkg/handlers.CreateUpdateUser (4 handlers)
[GIN-debug] DELETE /api/user/:userID --> github.com/prabhatsharma/zinc/pkg/handlers.DeleteUser (4 handlers)
[GIN-debug] GET /api/users --> github.com/prabhatsharma/zinc/pkg/handlers.GetUsers (4 handlers)
[GIN-debug] PUT /api/index --> github.com/prabhatsharma/zinc/pkg/handlers.CreateIndex (4 handlers)
[GIN-debug] GET /api/index --> github.com/prabhatsharma/zinc/pkg/handlers.ListIndexes (4 handlers)
[GIN-debug] PUT /api/:target/document --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST /api/:target/_search --> github.com/prabhatsharma/zinc/pkg/handlers.SearchIndex (4 handlers)
[GIN-debug] PUT /es/:target/_doc/:id --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST /es/:target/_doc --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] PUT /es/:target/_create/:id --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST /es/:target/_create/:id --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST /es/:target/_update/:id --> github.com/prabhatsharma/zinc/pkg/handlers.UpdateDocument (4 handlers)
[GIN-debug] POST /es/_bulk --> github.com/prabhatsharma/zinc/pkg/handlers.BulkHandler (4 handlers)
[GIN-debug] POST /es/:target/_bulk --> github.com/prabhatsharma/zinc/pkg/handlers.BulkHandler (4 handlers)
Does this mean zinc
doesn't support deleting documents or indices? I tried some curls modified from ElasticSearch tutorials I found online and had no success with those either
Hello, in the meantime, thank you for the constantly evolving product. Very useful for my projects. One question though: from version 0.1.3 onwards, I don't understand how to enable the limitation of 20 "records" on the "ZincSearch" web application. Thank you. I only ever see 20 records.
The short of the problem is that I can't get Zinc in a container to stay running without pre-existing indexes to load. It seems that without an existing index in your $DATA_PATH
directory, Zinc will panic and the docker container stops instantly before anyone would have the chance to have Zinc ingest any data.
I built an image from the Dockerfile in this repository using the command docker build -t test .
;
I ran the following command: docker run -d -v /server/zinc:/data -e DATA_PATH="/data" -e FIRST_ADMIN_USER=admin -e FIRST_ADMIN_PASSWORD=admin --name zinc test
I observed that the container had stopped near-instantly, and so I ran docker logs zinc
which returned the following:
{"level":"debug","time":"2021-12-03T20:09:04Z","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-03T20:09:04Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa299e4]
goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes(0xc0003f6180, 0x0, 0x0)
/go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:26 +0x1a4
github.com/prabhatsharma/zinc/pkg/core.init()
/go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3e
It appears you need to do more to start Zinc without pre-existing data that your documentation does not mention.
Since #16 is now merged, it would be good to register the zinc
name over at snapcraft.io and have a CI job to publish builds of zinc
automatically. Opening this to keep track! :)
Tried several options, but cat get it to go up and running from a command schell
$ FIRST_ADMIN_USER=admin FIRST_ADMIN_PASSWORD=Complexpass 123 zinc.exe
'FIRST_ADMIN_USER' is not recognized as an internal or external command,
operable program or batch file.
other try:
zinc.exe FIRST_ADMIN_USER=admin FIRST_ADMIN_PASSWORD=QNHadmin
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Loading system indexes..."}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Index loaded: _users"}
{"level":"debug","time":"2021-12-06T16:19:12+01:00","message":"Index loaded: _index_mapping"}
2021/12/06 16:19:12 FIRST_ADMIN_USER and FIRST_ADMIN_PASSWORD must be set on first start. You should also change the credentials after first login.
After staring zinc
on Windows 10, loading the data from the README python script, and choosing the games3
index, no results are returned - despite choosing 1 day
(or longer) as the time scale (the test was done an hour or so ago).
There are no logs (except a successful authentication).
The data is in because the available labels are displayed in the left drawer.
Hello, zinc uses GoReleaser, which offers excellent Homebrew integration. Would you be open to adding this integration? Here's what I do for reference
Would be great to have a cursor pagination for the search results.
This tool looks promising and it would be great to start planning SDK, so everyone can build tools on top of it.
I like to try installing on FreeBSD. How can I build from code?
When adding "_source" in query request, it still returns all the fields.
This is wonderful stuff ..
I was playing with Hey and GUI tools to test benchmarks and was thinking about a more solid way to get test data into the system.
I was wondering if you had any preferences / plans for this:
A faker like counterfeiter: https://github.com/maxbrunsfeld/counterfeiter
Example usage:
https://github.com/cloudfoundry/cf-networking-release/search?q=counterfeiter
It would be nice if there could be a discord server for zinc,
Plenty of OSS projects got a chat group somewhere.
It would be great if there was an option in the time settings for a search which would let you set the search time frame to all time.
curl
ing (or simply clicking in a browser) https://github.com/prabhatsharma/zinc/releases/download/v0.1.1/https://github.com/prabhatsharma/zinc/releases/download/v0.1.1/olympics.ndjson.gz returns a 404 Not Found
error
One of my indices is getting bigger and Zinc crashes every time i search anything using that index. It's the systemdlog index and it's ~560Mi on the filesystem
[root@lb02 tmp]# du -sch data/*
16K data/games3
116K data/_index_mapping
104K data/journal
46M data/syslog
558M data/systemdlog
12K da
Free memory before a search:
[root@lb02 tmp]# free -h
total used free shared buff/cache available
Mem: 7.5Gi 592Mi 6.1Gi 21Mi 843Mi 6.5Gi
Swap: ta/_users
After triggering a search on that index and just before the crash:
[root@lb02 tmp]# free -h
total used free shared buff/cache available
Mem: 7.5Gi 7.0Gi 158Mi 21Mi 298Mi 90Mi
Swap: 0B 0B 0B
Journal log after crash:
Dec 10 13:27:18 lb02.localdomain kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-12.scope,task=zinc,pid=59785,uid=0
Dec 10 13:27:18 lb02.localdomain kernel: Out of memory: Killed process 59785 (zinc) total-vm:8521368kB, anon-rss:7148568kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14596kB oom_score_adj:0
Is this expected behavior with such memory usage?
Would be lovely to have "how to contribute" section, a way to gain insights into your backlog and/or any working group meet up(s). So we can quickly identify opportunities to render our assistance.
Happy to get involved ~ 10hrs/week (go/vuejs)
dda7ec0 broke things becuase the default port changed to 8080
Might be best to make the test match the defaults here https://github.com/prabhatsharma/zinc/blob/main/main_test.go#L14
?
Good afternoon. Are there any plans to integrate with filebeat?
goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes()
/go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:25 +0x137
github.com/prabhatsharma/zinc/pkg/core.init()
/go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3c
{"level":"debug","time":"2022-01-04T14:23:59Z","message":"Loading indexes..."}
{"level":"debug","time":"2022-01-04T14:23:59Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x958bd7]
Refer to the Es-use-document to paging query,but it didn't work
Es-use-document: https://www.elastic.co/guide/cn/elasticsearch/guide/2.x/_empty_search.html
GET /_search
{ "from": 30, "size": 10 }
Thanks for your work on this!
It would be great to be able to specify a subpath, or path prefix to be used, for hosting behind a reverse proxy.
For example. if I wanted to proxy :4080/ui
, and have it accessible as /zinc/ui
This project looks promising. Any benchmark that compares ES? Thanks!
Hey, i've just recently stumbled across this project and before i invest time setting this up, i want to ask first: Is zinc a real logserver alternative? Like can i setup filebeat/whatever and push my logs to it or is zinc designed for other stuff?
I was in the process of installing ELK but its quite complex and there's no free user management so i was searching for other options.
Apologies if that's not the right place to ask those questions.
I would like to use Magento 2 open source ecommerce platform, which requires elasticsearch as a prerequisite for the installation. Would it be possible to swap ES with zinc?
Thank you in advance!
I started an instance on Windows 10 and used the exact example from the README (just adapting the admin password). When the data was sent (probably successfully - there is a successful auth message but nothing about data ingestion, and the script ran fine) and I pressed the Search button, zinc
panicked with
2021/12/04 10:03:52 [Recovery] 2021/12/04 - 10:03:52 panic recovered:
POST /api//_search HTTP/1.1
Host: localhost:4080
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,fr;q=0.8,pl;q=0.7,de;q=0.6
Authorization: *
Connection: keep-alive
Content-Length: 145
Content-Type: application/json
Dnt: 1
Origin: http://localhost:4080
Referer: http://localhost:4080/ui/
Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:221 (0x459ebc)
/usr/local/go/src/runtime/signal_windows.go:254 (0x459e8c)
/Users/prabhat/projects/raelabs/zinc/pkg/core/Search.go:61 (0x979af9)
/Users/prabhat/projects/raelabs/zinc/pkg/handlers/Search.go:23 (0x98d0c4)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x987e87)
/Users/prabhat/projects/raelabs/zinc/pkg/auth/AuthMiddleware.go:21 (0x987cce)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x8a5a81)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:99 (0x8a5a6c)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:165 (0x8a493d)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:489 (0x8a45c5)
/Users/prabhat/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:445 (0x8a4124)
/usr/local/go/src/net/http/server.go:2878 (0x67d35a)
/usr/local/go/src/net/http/server.go:1929 (0x678a07)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x4749c0)
mkdir data
docker run -v $(pwd)/data:/data -e DATA_PATH="/data" -p 4080:4080 -e FIRST_ADMIN_USER=admin -e FIRST_ADMIN_PASSWORD=Complexpass#123 -p 4081:4081 public.ecr.aws/m5j1b6u0/zinc:v0.1.1
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
{"level":"debug","time":"2021-12-07T03:39:24Z","message":"Loading indexes..."}
{"level":"debug","time":"2021-12-07T03:39:24Z","message":"Loading system indexes..."}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa299e4]
goroutine 1 [running]:
github.com/prabhatsharma/zinc/pkg/core.LoadZincSystemIndexes(0xc000402f00, 0x0, 0x0)
/go/src/github.com/prabhatsharma/zinc/pkg/core/LoadIndexes.go:26 +0x1a4
github.com/prabhatsharma/zinc/pkg/core.init()
/go/src/github.com/prabhatsharma/zinc/pkg/core/types.go:10 +0x3e
Please see
https://github.com/sbt/zinc
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.