package main
import (
"fmt"
"log"
"os"
"strings"
"github.com/nxadm/tail"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
fmt.Printf("PID: %v\n", os.Getpid())
logfile := os.Args[1]
t, err := tail.TailFile(logfile, tail.Config{Follow: true, ReOpen: true})
if err != nil {
return err
}
for line := range t.Lines {
fmt.Printf("line: %v; newline ending: %v\n", line.Text, strings.HasSuffix(line.Text, "\n"))
}
return nil
}
And run this against a random text file, go run repro.go tmp.log
and then write to it:
echo -n "foo" >> tmp.log
echo "foo" >> tmp.log
echo "foofoo" >> tmp.log
This creates two lines, both containing foofoo
, but we can't tell from the reader's point of view:
$ go run repro.go tmp.log
PID: 57959
2021/03/29 11:17:07 Waiting for tmp.log to appear...
line: foo; newline ending: false
line: foo; newline ending: false
line: foofoo; newline ending: false
System information
- tail version v1.4.8
- OS: macOS
- Arch: amd64
Additional context
This was reported back in 2014 in the original repo #35 and "fixed" without tests in #40, which deadlocked the process if the tailed file didn't end with a newline. It was then reverted in this repo by merging in #126.
My proposal doesn't affect the library's behaviour, just adds a flag to inform the user of partial rows. I wasn't super sure how to handle splitting or large lines, so please review my code.
- Loss situation: When the file flush and file renaming occur very close to each other, part of the content of the old file may be lost.
- Duplication situation: If a write event and a rename event for a file occur close in time, and the write notification is marked as a truncated event, then if collection starts from the beginning, the first part of the new file will be collected twice.
Hi - I made use of a fork of the original hpcloud/tail
repo in termshark to tail a pcap (packet capture) file. The output from the tail process is fed into some tshark
(CLI tool) processes which provide data for termshark's UI. Because pcaps are a binary format, I couldn't rely on a line-based tail to provide timely output for tshark
's input, so I made some cheesy modifications to hpcloud/tail
to tail in "chunks" of bytes. These chunks are not necessarily line-terminated, and possibly even 1-byte long if the pcap file being tailed grows very slowly. Would you be interested in these sorts of changes in nxadm/tail
? If so, I could make a PR - and if you find it acceptable, I'd then switch termshark over to use nxadm/tail
. Thanks!
Describe the bug
log-parser.go:7:2: package nxadm/tail is not in GOROOT (/usr/local/go/src/nxadm/tail)
Expected behaviour
log-parser.go runs like the wind
To Reproduce
go get github.com/nxadm/tail/...
(import library in logparser.go)
go run logparser.go
System information
- tail version 8.32
- WSL ubuntu
- amd64
Additional context
Hey just trying to install this library, I do have other libraries installed with go get
that import correctly. I'm wondering if I'm missing something I should know before I dig into my Go environment, thanks!
Hi,
I have the issue when I trying to tail the file which was tailed before. The channel tail.Lines doesn't return anything.
Steps to reproduce.
- open and tail file A with
tail.TailFile()
- receive some lines using Lines channel.
- close file with
.Stop()
and .Cleanup()
.
... everything work as intended.
- open and tail file B with
tail.TailFile()
- receive some lines using Lines channel.
- close file with
.Stop()
and .Cleanup()
.
... everything work as intended.
- repeat tailing for file A, open it with
tail.TailFile()
- attempt to receive some lines using
Lines
channel will not work. No errors no panics, select
is just silently waiting for new lines.
If file A contains some content before attempt 2, lines are returned, but new lines appended later are not returned from channel.
Next attempts to tail another new files will work correctly.
The usecase is related to tailing PostgreSQL logs, which could be configured with "-%a" suffix, which means a weekday. After one week the program will have to return to tailing previously tailed (and truncated) file.
hpcloud/tail#159
the changes were already done when the fork was migrated to modules.
Describe the bug
Tailing files that do not exist causes the go channel to close almost immediately. I've also noticed that the go channel closes (sometimes) when a file rotates; I think that happens when the file does not get re-created right away. I hope i'm just missing something and there's a workaround to this issue, or maybe some obvious tactic I'm not seeing.
Also, calling .Stop()
on a closed tail
produces an error (on windows) that indicates it's trying to open the file:
Unable to open file C:\ProgramData\file.path: The process cannot access the file because it is being used by another process.
Maybe this is expected, and I'm going to avoid calling stop when the channel is closed, but my first iteration did not and I noticed this. Wanted to share.
Expected behavior
It's not clear why the go channel closes. Once it's closed, it cannot be reused. I would expect the channel not to be closed until .Stop()
is called.
To Reproduce
Try to tail a file that does not exist. The channel closes, and when you create the file there is no channel to read lines from.
package main
import (
"log"
"os"
"github.com/nxadm/tail"
)
func main() {
log := log.New(os.Stdout, "", 0)
t, err := tail.TailFile("/tmp/does.not.exist", tail.Config{
Follow: true,
ReOpen: true,
Logger: log,
})
if err != nil {
log.Println(err)
}
defer func() {
if err := t.Stop(); err != nil {
log.Println(err)
}
}()
for line := range t.Lines {
log.Println(line.Text)
}
log.Println("channel closed")
}
// Output:
// Waiting for /tmp/does.not.exist to appear...
// channel closed
// Failed to detect creation of /tmp/does.not.exist: bad file descriptor
(the last line is likely OS dependent, the above error is from macos x86)
System information
Happens on macos arm&x86 and windows x86 using v1.4.8.
Additional context
🤞
Branch: https://github.com/nxadm/tail/tree/noTomb
Tail depends on Tomb v1, a version of the library not updated updated in the last 5 years. The next version of Tomb, v2, is not API compatible with v1 and hasn't been updated for 2 years. A better solution is using the core coroutine workflow around context.Context. This branch implements this workflow and passes all the tests. However, because the API of Tomb v1 is very different to the one of context.Context, the changes in this branch need to be well tested in order to make sure no threading issues occur.
This branch also removes internal packages that resulted in many exported functions and structs. Most of these, probably all, were only useful within tail and their public nature was just a consequence of being in a package. This methods were removed from the public API. Also, the OpenFile function was removed from the API as it doesn't do anything special and is not related to tailing. The documentation was updated and public functions and structs were documented. Programs that use the "useful" API of previous releases will not be impacted:
- the variables DefaultLogger and DiscardingLogger.
- the structs Config, Line, Seekinfo and Tail.
- the functions TailFile, Tell, Stop and StopAtEOF.
This branch, once ready, will be released as a major release, v2.
Describe the bug
We have a file that we're tailing where we're also interested in the file offset location. However, when we do t.Tell()
, we sometimes run into the exception "err":"seek <filename>: file already closed"
Code:
t, err := tail.TailFile(
path,
tail.Config{
Follow: true,
ReOpen: true,
})
for line := range t.Lines {
// Fetch the current offset in the tailed file
offset, err := t.Tell()
// consume data
}
Has anyone else seen a similar issue with the file being closed? We've kind of solved this by adding a small wait and a retry in the t.Tell()
function call, and that's resolved the issue, but we'd like to understand if this is a bug in the underlying tailing code where it's prematurely closing files.
System information
- tail version: v1.4.8
- OS: Ubuntu: 18.04
I used tail.TailFile
and t.Lines
happily. Until one day I had a UTF16-LE encoded file.
Since golang string
type supports utf-8, it is not capable to throw a UTF-16 string correctly.
It will be nice if a tail.Line
can return a []byte. What do you think?
Describe the bug
If two (or more) goroutines are tailing the same file and one of them stops it impacts the other goroutine(s).
Expected behaviour
Other goroutines should keep on tailing the file.
To Reproduce
See this test.
It runs three goroutines, one will continuously write to a file every 500 milliseconds, the other two will each tail the same file. The writing goroutine runs for 5 seconds, the first tailing goroutine runs for 2 seconds and the second tailing goroutine runs for 4 seconds.
The expectation is that the second goroutine would keep on tailing the file after the first one stops, but that is not the case.
System information
- tail version 1.4.8
- OS: macOS Monterey 12.3.1
- Arch: amd64
Additional context
The issue is (presumably) that both goroutines use the same underlying watcher and after one stops the watcher is removed.
If you StartTail(file_x) following from end, and then append "hello\n" to file_x, there's a race that makes it uncertain whether your appended line will come out of the tail operation or not; the StartTail operation seeks the end of the file in its own goroutine, and provides no signal that it has done so, so there's no way to reliably force the ordering of the operations.
#48 adds a config option "SyncOpen" that allows StartTail to perform any requested seek synchronously before it returns, thereby guaranteeing an order of operations.
If you sent an issue/PR to hpcloud, you can have a look at
this meta issue tracking issues
and PRs of the dormant upstream. However, as the code bases diverge the
issue may have been solved already.
Describe the bug
The StopAtEOF
function can sometimes stop tailing the file before we actually encounter an EOF. In particular, when using the polling function for checking for file changes, we might select this tail.Dying channel here first before the poller gets another chance to notify us of new changes, which I believe makes us then return here, even though we should probably be waiting for another poll cycle.
Expected behaviour
The library checks again for new changes in the file after StopAtEOF
is reached.
To Reproduce
This is a bit awkward to reproduce, since it's based on racyness. However tailing a file that's being written to in short intervals, and calling StopAtEOF
might not return all data before exiting.
Additional context
I'd be happy to contribute a PR for this, but probably won't have the time to do so before next week.
Describe the bug
When errors occur, it is important to get the error message logged out. Otherwise it can get very annoying to debug. I have for example hit the following error:
|
util.Fatal("failed to create Watcher") |
I don't know why fsnotify.NewWatcher()
would fail, but since the error message isn't included in the error log, it is almost impossible to debug.
Expected behaviour
I would expect the error message is a part of the logs using something like the following:
util.Fatal(fmt.Sprintf("failed to create Watcher: %v", err))
Additional context
I actually came onto this error by using https://github.com/hpcloud/tail which is used in https://github.com/kubeflow/katib.
If you start tailing the same file twice (by calling TailFile
and supplying an identical filename), and are using the inotify backend, then only one of the Tail
objects is notified when a line is appended to the file. Which one of the two objects gets notice is chosen at random, and the one which does then catches up on lines it has slept on in previous appends.
I would expect both of the objects to be notified at each append.
Describe the bug
I’ve encountered an intermittent issue with log collection loss during log rotation when using the [hpcloud/tail v1.0.0] library, tail is not stop or kill.
I’ve reviewed previous issues, namely issue 94[https://github.com/hpcloud/tail/issues/94] and issue 97[https://github.com/hpcloud/tail/issues/97], and after analyzing the source code of tail, I’m unable to consistently reproduce the problem. I don’t understand how the ‘deleted’ signal could be getting lost.
The configuration for tail is as follows, using the polling mechanism:
tailFile, err := tail.TailFile(logPath, tail.Config{
ReOpen: true,
Follow: true,
Location: &tail.SeekInfo{Offset: 0, Whence: 2},
MustExist: false,
Poll: true,
})
Expected behaviour
I’m wondering if the nxadm/tail has addressed this issue, and I would appreciate it if the experts could help explain the above-mentioned problem.
Describe the bug
Simple code:
func main() {
if len(os.Args) != 2 {
panic("not enough arguments")
}
filename := os.Args[1]
t, err := tail.TailFile(filename, tail.Config{Follow: true, ReOpen: true})
if err != nil {
panic(err)
}
for line := range t.Lines {
fmt.Println(line.Text)
}
}
When used in conjunction with continuous truncation (>
) as opposed to appending (>>
) only reacts to the first event, ignore all following ones. Simple test case:
$ while true; do echo "$(date)" > watched.log; sleep 1; done
It will truncate file every second before writing a single line into it.
Expected behaviour
The code correctly detects the truncation events and re-opens the file accordingly. So the output would look something like this:
Mon Dec 18 04:23:44 PM UTC 2023
2023/12/18 16:23:45 Re-opening truncated file watched.log ...
2023/12/18 16:23:45 Successfully reopened truncated watched.log
Mon Dec 18 04:23:45 PM UTC 2023
2023/12/18 16:23:46 Re-opening truncated file watched.log ...
2023/12/18 16:23:46 Successfully reopened truncated watched.log
Mon Dec 18 04:23:46 PM UTC 2023
...
Actual behaviour
It detects first event, and reads a single line after that ignoring everything else:
2023/12/18 16:31:45 Waiting for watched.log to appear...
Mon Dec 18 04:31:48 PM UTC 2023
To Reproduce
See steps above. I've tried with Poll:true
to no avail.
System information
Additional context
Recommend Projects
-
-
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
An Open Source Machine Learning Framework for Everyone
-
The Web framework for perfectionists with deadlines.
-
A PHP framework for web artisans
-
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
Some thing interesting about web. New door for the world.
-
A server is a program made to process requests and deliver data to clients.
-
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Some thing interesting about visualization, use data art
-
Some thing interesting about game, make everyone happy.
-
Recommend Org
-
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Open source projects and samples from Microsoft.
-
Google ❤️ Open Source for everyone.
-
Alibaba Open Source for everyone
-
Data-Driven Documents codes.
-
China tencent open source team.
-