Web based real-time log viewer. Stream ANY content to a web UI with autogenerated filters. Parse any format with TypeScript.
Home Page: https://logdy.dev
License: Apache License 2.0
go.mod defines the package via module logdy when it should be module github.com/logdyhq/logdy-core. Attempting to import it from another module then results in an error:
github.com/logdyhq/logdy-core/ring: github.com/logdyhq/[email protected]: parsing go.mod:
module declares its path as: logdy
but was required as: github.com/logdyhq/logdy-core
Hello.
I have launched logdy as:
pm2 logs my_app --json | logdy
In the frontend one log message is being split into chunks of 4096 characters max each. As result, json data is being split into several messages and could not be parsed, columns auto-generate doesn't work.
Also index.js file of the web based app is as big as 10.9MB. Can it be reduced?
Screenshots attached.
I am attempting to use bginx to proxy connections to this running in a Docker container and I keep getting a:
"Status: Not Connected"
Message when loading the UI. I am able to see and interact with the full UI when running with flag --ui-ip=0.0.0.0 but when accessing the same message appears. Additionally, in the STDOUT I see the error "Missing client id". I've tried with both --ui-pass and without. I also do not get a client connected in the STDOUT of the container this is running in at any point.
I am using Nginx Proxy Manager to serve the UI.
Curently the timestamp seems to be just text. This means no possibility to filter for time ranges.
Something like time >= 2024-07-11T00:00:00Z AND time <= 2024-07-12T00:00:00Z would be helpfull.
Cheers
👋 looks like announcement is down.
$ curl -I https://logdy.dev/blog/post/logdy-new-version-announcement-v12
HTTP/2 404
date: Fri, 28 Jun 2024 00:51:48 GMT
👋 looks like 0.12.1 got tagged but not get release completed, ie the release notes did not get created.
relates to Homebrew/homebrew-core#175870
Hey, not sure how high priority this is for you, but as I noted in #7 I am trying to replace a microservice runner with logdy for reading logs while developing services locally. This sometimes necessitates running a lot of services which would be nice to monitor through a single logdy UI.
However, there seems to be quite a bottleneck somewhere in logdy. Linked below is a screencap from me starting the services without some databases online, which causes a lot of errors that are badly formatted and verbose. You can see that from around 0:08 to 0:20 in the screencap logdy is stuck processing logs that all have their timestamp within 0.75 seconds. That's over 12 seconds to parse less than second of logs, which of there are quite a few, but not that many.
https://drive.google.com/file/d/14qLBAG-jQhaB3sK8RUZrCnfJvXQr0fT8/view
Again, not sure what the intended use-case for this is, but for this-use case it's just not performant enough yet.
EDIT: just wanted to note that I don't mean to come off harsh, the project seems super nice. I've actually been waiting for something like it for a long time since I don't have the time and motivation to do it myself. Great job so far!
my another application is running on port 8080; so I tried to forward to the 8081 port but it showing this error. can anyone help? my port 8081 is free. I checked.
command:
pm2 logs 1 | logdy forward 8081
error:
INFO[20:22:24.905] Accept stdin and forward to port port=8081
ERRO[20:22:24.905] Error while connecting to port error="dial tcp :8081: connect: connection refused"
panic: dial tcp :8081: connect: connection refused
goroutine 1 [running]:
logdy/modes.ConsumeStdinAndForwardToPort({0x0, 0x0}, {0x7ffffcf896ae, 0x4})
/Users/peter/projects/cf/logdyhq/logdy-core/modes/forward.go:20 +0x445
main.glob..func6(0xc0000c7100?, {0xc0000ae3b0, 0x1, 0x9109f2?})
/Users/peter/projects/cf/logdyhq/logdy-core/main.go:128 +0x49
github.com/spf13/cobra.(*Command).execute(0x26c0e00, {0xc0000ae380, 0x1, 0x1})
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:987 +0xaa3
github.com/spf13/cobra.(*Command).ExecuteC(0x26c0560)
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
main.main()
/Users/peter/projects/cf/logdyhq/logdy-core/main.go:195 +0x53
node:events:496
throw er; // Unhandled 'error' event
^
Error: write EPIPE
at afterWriteDispatched (node:internal/stream_base_commons:160:15)
at writeGeneric (node:internal/stream_base_commons:151:3)
at Socket._writeGeneric (node:net:952:11)
at Socket._write (node:net:964:8)
at writeOrBuffer (node:internal/streams/writable:564:12)
at _write (node:internal/streams/writable:493:10)
at Writable.write (node:internal/streams/writable:502:10)
at console.value (node:internal/console/constructor:304:16)
at console.log (node:internal/console/constructor:379:26)
at /usr/local/lib/node_modules/pm2/lib/API/Log.js:56:15
Emitted 'error' event on Socket instance at:
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}
Node.js v20.11.0
Thank you for the awesome logging tool.
I have a question as stated in the title.
is there any way to use other UI framework other than Vue (logdy-ui)?
I would like to use my own frontend to display PM2 logs.
I looked into the logdy-core to know how the logdy-core commuicates with the logdy-ui, but I could not find it out.
Could you explain how I can use my own frontend? To my knowledge, websocket should be involved. Any guide will be helpful.
I connected to th elogdy-core through ws://localhost:8080/ws
at first I got log_bulk messages and then client_msg_status periodically.
even though I created a message on the server, my own frontedend has not received the message except the 'client_msg_status'.
In contrast, the logdy-ui received the messages correctly (it seems that the 'client_msg_status' message is not displayed intentionally in the ui code)
It is strange that my own frontend has not received the log messages other than the 'client_msg_status'.
I took a look at the installer and i saw the following line
curl -sAx "https://notify.logdy.dev/download?version=${version}" > /dev/null
What does that do?
Thanks,
Chip
Hello,
i just downloaded latest 0.10.0 from Github. After start it shows an info message there is a neweer version available.
But looking at the version number its wrong - either version comparison does not work or external server does not show 0.10.0 as latest.
$ lodgy -v
DEBU[19:41:36.431] Setting verbose logger
INFO[19:41:36.431] Listen to stdin (from pipe)
WARN[19:41:36.431] No opt-out from analytics, we'll be receiving anonymous usage data, which will be used to improve the product. To opt-out use the flag --no-analytics.
INFO[19:41:36.431] WebUI started, visit http://127.0.0.1:8080 port=8080
DEBU[19:41:36.855] New version available current_version=0.10.0 latest_version=0.9.0 response="{0.9.0 04 April 2024 https://github.com/logdyhq/logdy-core/releases/tag/v0.9.0 https://logdy.dev/blog/post/logdy-new-version-announcement-v09 What's new? More useful UI enhancements, reading full file(s) with follow mode}"
Best regards
I'm using this command:
aws --profile=staging logs tail <log group name> --follow | logdy
And getting no output:
Is there a configuration step i'm missing?
❯ logdy
[1] 24462 segmentation fault logdy
❯ uname -a
Linux 210 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Dear team,
Thank you for your awesome work on this great tool.
We have only one blocking issue for both 0.11 and 0.12.
When logdy is running inside Docker container, UI does not work when we are accessing it from the host.
Inside container it is running fine (we can see with curl). No special port forwarding just 8080:8080.
Regarding installation we just copied binary file to container.
Best regards
Size returns an invalid size (larger than the buffer) in some cases.
Due to #27 , I created a new Ring queue implementation: https://github.com/arg0net/collections
As part of it, I added a Fuzzing test to look for edge cases that I might not have considered. I tried running the fuzzer on Logdy's RingQueue out of curiosity since I was seeing strange behavior. What it found is that the RingQueue Size function is incorrect and can report a size greater than the size of the ring. Since Size is called from other functions, those functions are also incorrect.
Reproduction case:
start is 1, and end is 0.Here's my hacked fix
func (r *RingQueue[T]) Size() int {
res := r.end - r.start
if res < 0 {
res = len(r.data) + res
} else if res == 0 && r.isFull {
return len(r.data)
}
return res
}After fixing that, the fuzzer next found a bug in PeekIdx. There's an off-by-one error which causes it to return a value for an empty ring. Since the ring values are not cleared, it is returning a previously used value from the queue. I believe the problem is that it isn't taking into account the isFull flag if start == end. I patched in a fix for that as well as the Size issue and then the fuzzer was still finding problems, so I've given up. I suggest implementing fuzzing to help track down the issues, or else consider importing the implementation I created. I'm happy to add additional methods if you need Scan or other methods.
FWIW, I also ran the benchmark on the arg0net and lodgy versions and found the arg0net is also 8x faster for repeating push/pops on a desktop server. I'm guessing that this is because there's no need to do modular arithmetic or the extra step of memory indexing (one is built into the array accessor) to determine an index.
Hope that helps!
in main.go before line 237, forwardCmd missing flag “ip”
ASCII characters displayed in Browser
tail -f /var/log/xlxd.log | logdy -p 8123 --ui-ip 0.0.0.0 --ui-pass Welcome41 -t
Result:
�[22m�[1m�[36mApr 22 13:36:35�[0m�[22m�[36m �[0m�[22m�[1m�[34mqra-team�[0m�[22m�[36m �[0m�[22m�[32mxlxd:�[0m�[22m�[36m �[0m�[22m�[36mDMRmmdvm�[0m�[22m�[36m �[0m�[22m�[36mconfiguration�[0m�[22m�[36m �[0m�[22m�[36mpacket�[0m�[22m�[36m �[0m�[22m�[36mfrom�[0m�[22m�[36m �[0m�[22m�[36mSASHKO�[0m�[22m�[36m �[0m�[22m�[36m�[0m�[22m�[36m �[0m�[22m�[36mB�[0m�[22m�[36m �[0m�[22m�[36mat�[0m�[22m�[36m �[0m�[22m�[1m�[34m83.220.237.124�[0m�[22m�[36m �[0m
In bash terminal it is displayed normal.
Host info:
Virtualization: kvm
Operating System: Ubuntu 22.04.3 LTS
Kernel: Linux 5.15.0-97-generic
Architecture: x86-64
Hardware Vendor: Red Hat
Hardware Model: KVM
bash --version
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: n/a
X11 Layout: us
X11 Model: pc105
Client browsers:
Microsoft Edge Version 125.0.2518.0 (Official build) stable app, dev channel (64-bit)
and
Google Chrome Version 123.0.6312.124 (Official Build) (64-bit)
As far as I'm understanding the documentation and examples, the following should work:
su # yes, this is to root the root user
$ logdy stdin `tail -f /var/log/messages`
But unfortunately this gives no output whatsoever. It "hangs" with nothing being emitted and seemingly no port access on 8080 of the server.
0.3.0 is installed in ~/bin/ (for the root user) and it executes:
$ logdy --version
logdy version 0.3.0
The following also does not work:
logdy stdin `tail -f /var/log/messages` -n -p 51235 -v
Changing the order does not work:
logdy -n -p 51235 -v stdin `tail -f /var/log/messages`
Neither of these work:
$ logdy --version stdin `tail -f /var/log/messages`
^C#
$ logdy stdin `tail -f /var/log/messages` --version
^C#
$ logdy stdin `tail -f /var/log/messages` -n -p 51235 -v --version
Oddly this does start the server, but has no output (eg, doesn't activate any kind of actual tailing in and of itself):
$ logdy -n -p 51235 -v /var/log/messages
INFO[2024-02-11T00:14:53.659103407-05:00] WebUI started, visit http://localhost:51235 port=51235
^C
But this does not work:
$ logdy -n -p 51235 -v
Error: requires at least 1 arg(s), only received 0
Usage:
logdy [command] [flags]
logdy [command]
...
Even if I attempt to tail another file such as a user's file, it fails:
$ logdy stdin `tail -f /home/username/public_html/error_log`
^C#
Any clarification as to how to call this with simply one file or multiple files?
The "Stick to bottom" button should have a pin/unpin or some kind of a visual indicator to demonstrate the current mode for visual clarity:
This presently does not change visual presentation when it is pinned/stuck or not.
When I run the command # logdy completion bash > logdy.bash, I expect to get just a shell script. But instead, I get a shell script and service startup. As a result, the file logdy.bash contains the following lines:
ex: ts=4 sw=4 et filetype=sh
time="13:31:05.676" level=info msg="Listen to stdin (from the pipe)"
time="13:31:05.676" level=warning msg="No opt-out from analytics, we'll be receiving anonymous usage data, which will be used to improve the product. To opt-out use the flag --no-analytics."
time="13:31:05.678" level=info msg="WebUI started, visit http://127.0.0.1:8080" port=8080
I'd prefer to install via https://brew.sh if possible.
https://docs.brew.sh/Acceptable-Formulae#requirements-for-homebrewcore just requires a single config file PR
I'm happy to help in what way I can, but the build steps to create this seem a bit complicated.
here's my log https://pastebin.com/9YjAviKU
Unable to use go install or go build because of this line: https://github.com/logdyhq/logdy-core/tree/a4fa83d/embed.go#L8-L9
Should assets be included in the repo — whether configured for LFS or not — so builds can be done by others?
Thank you <3 for the awesome project.
This would make it possible to create a config file for a project, add it to source control, and keep in sync with everyone as the project evolves.
Trying to get logdy to listen on "any" interface using --ui-ip=* panics:
`INFO[18:25:35.795] Listen to stdin (from pipe)
WARN[18:25:35.795] No opt-out from analytics, we'll be receiving anonymous usage data, which will be used to improve the product. To opt-out use the flag --no-analytics.
INFO[18:25:35.795] WebUI started, visit http://*:8080 port=8080
panic: listen tcp: lookup *: no such host
goroutine 1 [running]:
main.handleHttp({0x4d56ca, 0x4}, {0xffffd431e533, 0x1}, 0x1, {0x0, 0x0}, {0x0, 0x0}, 0x64)
/Users/peter/projects/cf/logdyhq/logdy-core/http.go:377 +0x670
main.glob..func3(0x4000128f00?, {0x4000030380?, 0x0, 0x4d56c6?})
/Users/peter/projects/cf/logdyhq/logdy-core/main.go:79 +0x408
github.com/spf13/cobra.(*Command).execute(0x227c620, {0x400001e0f0, 0x1, 0x1})
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:1005 +0x958
github.com/spf13/cobra.(*Command).ExecuteC(0x227c620)
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
/Users/peter/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
main.main()
/Users/peter/projects/cf/logdyhq/logdy-core/main.go:250 +0x60`
An option to make it listen on any available interface could be helpful.
Hi. I am downloading the latest version 0.10.0 from Github and trying to run the tests: # go test -v ./...
I am getting an error:
=== RUN TestGenerateTextRandomData
--- FAIL: TestGenerateTextRandomData (0.00s)
panic: slice is empty [recovered]
panic: slice is empty
goroutine 6 [running]:
testing.tRunner.func1.2({0x783d60, 0x8cdb20})
/usr/lib/golang/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
/usr/lib/golang/src/testing/testing.go:1634 +0x377
panic({0x783d60?, 0x8cdb20?})
/usr/lib/golang/src/runtime/panic.go:770 +0x132
logdy/utils.PickRandom[...](...)
/root/test/logdy-core/utils/utils.go:53
logdy/modes.generateTextRandomData()
/root/test/logdy-core/modes/demo.go:86 +0x427
logdy/modes.TestGenerateTextRandomData(0xc000185380)
/root/test/logdy-core/modes/demo_test.go:13 +0x18
testing.tRunner(0xc000185380, 0x868da8)
/usr/lib/golang/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
/usr/lib/golang/src/testing/testing.go:1742 +0x390
FAIL logdy/modes 0.013s
P.S. All tests passed successfully in version 0.9.0
When accessing the web ui from another server the settings (and columns) are not synchronized. It might be nice save them on disk so they are not tied to a browser.
What would be the right trick to read an entire existing file full of JSON log messages and feed it into Logdy for viewing? Tried a couple variations of cat some-file.log | logdy and didn't immediately get anything working. Any suggestions here?
Thanks!
I think have an http API to add log is useful.
An ASDF plugin would be sweet. Are you open to a PR?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.