hillu / go-yara Goto Github PK
View Code? Open in Web Editor NEWGo bindings for YARA
License: BSD 2-Clause "Simplified" License
Go bindings for YARA
License: BSD 2-Clause "Simplified" License
So, I made a very rough draft of a program. And I have the following code:
func printMatches(m []yara.MatchRule, err error) {
if err == nil {
if len(m) > 0 {
for _, match := range m {
log.Printf("- [%s] %s ", match.Namespace, match.Rule)
}
} else {
log.Print("no matches.")
}
} else {
panic(err.Error())
}
}
func main() {
compiler, err := yara.NewCompiler()
if err != nil {
panic(err.Error())
}
file, err := os.Create("/home/user/Documents/Golang/code/Go-Away-AV/ShadowTech.yar")
if err != nil {
panic(err.Error())
}
compiler.AddFile(file, "Infinity.yar")
scanner, err := compiler.GetRules()
if err != nil {
panic(err.Error())
}
scanned, err := scanner.ScanFile("/home/user/Documents/SecurityUSB - Backup/Windows/Anti malware", 0, 0)
printMatches(scanned, err)
}
However, It for some reason completely deletes everything in the .yar file, while keeping the file itself. It is also throwing the error: "could not open file" at the 'ScanFile()' section
I'm not sure if this is of any interesting, but this warning is raised when cross-compiling for Windows on Go 1.11:
GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc PKG_CONFIG_PATH=${YARA_SRC}/x86_64-w64-mingw32/lib/pkgconfig go install -ldflags '-extldflags "-static"' github.com/hillu/go-yara
# github.com/hillu/go-yara
../../.gvm/pkgsets/go1.11/global/src/github.com/hillu/go-yara/rules_yara34.go: In function ‘_yr_rules_scan_fd’:
../../.gvm/pkgsets/go1.11/global/src/github.com/hillu/go-yara/rules_yara34.go:30:34: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
return yr_rules_scan_fd(rules, (YR_FILE_DESCRIPTOR)fd, flags, callback, user_data, timeout);
I'm trying to build a statically linked Linux binary using on Go 1.11:
GOOS=linux GOARCH=amd64 go build --ldflags '-s -w -extldflags "-static"' -o agent
But I'm having some difficulties. I have compiled Yara from the sources with --without-crypto
, but I get the following errors:
/usr/bin/ld: /tmp/go-link-968020938/000011.o: in function `unixDlOpen':
/home/user/.gvm/pkgsets/go1.11/global/pkg/mod/github.com/mattn/[email protected]/sqlite3-binding.c:37945: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000019.o: in function `mygetgrouplist':
/home/user/.gvm/gos/go1.11/src/os/user/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetgrgid_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetgrnam_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetpwnam_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000018.o: in function `mygetpwuid_r':
/home/user/.gvm/gos/go1.11/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-968020938/000016.o: in function `_cgo_18049202ccd9_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:49: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: i386 architecture of input file `/tmp/go-link-968020938/000000.o' is incompatible with i386:x86-64 output
/usr/bin/ld: //usr/local/lib/libyara.a(math.o): in function `string_entropy':
math.c:(.text+0xea7): undefined reference to `log2'
/usr/bin/ld: //usr/local/lib/libyara.a(math.o): in function `data_entropy':
math.c:(.text+0x1126): undefined reference to `log2'
Using the thor-webshells.yar
rules file from Neo23x0 rules repository I got the fatal error when there is a match.
fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x31 pc=0x7c53d7]
However, if I comment line no.123 keepAlive(r)
, it works okay, can u explain what keepAlive
function is needed for?
I'm not good at understanding usage of C in Golang.
No way to get defined Variables
only can define variables by using DefineVariable
I have been attempting to deploy a web service in Go within a docker container (ubuntu 19.04) that accept a file and returns the names for each rule that matches. When testing this on my local machine (macOS) with yara 3.11 and yarac 3.11 I am able to scan the wannaCry sample [ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa](https://www.virustotal.com/gui/file/ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa/detection)
. Although when I go deploy this service within the docker container I get the following errors.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2f pc=0xcd3e97]
runtime stack:
runtime.throw(0xf1dee4, 0x2a)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:378 +0x47c
goroutine 61 [syscall, locked to thread]:
runtime.cgocall(0xcd3e60, 0xc000344d20, 0x1)
/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000344cf0 sp=0xc000344cb8 pc=0x40b96b
github.com/hillu/go-yara._Cfunc_string_matches(0x7fb6c544ec05, 0x0, 0xc0004c9bfc)
_cgo_gotypes.go:1121 +0x45 fp=0xc000344d20 sp=0xc000344cf0 pc=0xb07415
github.com/hillu/go-yara.(*String).Matches(0xc000344e48, 0x0, 0x0, 0x0)
/go/src/github.com/hillu/go-yara/rule.go:225 +0x6b fp=0xc000344db8 sp=0xc000344d20 pc=0xb0c2ab
github.com/hillu/go-yara.(*Rule).getMatchStrings(0xc0000a8ce0, 0x0, 0x0, 0x0)
/go/src/github.com/hillu/go-yara/rule.go:244 +0x2c2 fp=0xc000344ec0 sp=0xc000344db8 pc=0xb0c722
github.com/hillu/go-yara.(*MatchRules).RuleMatching(0xc000324220, 0xc0000a8ce0, 0xc000324220, 0x7fb6ec1219f0, 0xc000324220)
/go/src/github.com/hillu/go-yara/rules_callback.go:151 +0x224 fp=0xc000345020 sp=0xc000344ec0 pc=0xb0e3e4
github.com/hillu/go-yara.scanCallbackFunc(0x1, 0x7fb6c3d6916d, 0x21bb7e0, 0xf395a8)
/go/src/github.com/hillu/go-yara/rules_callback.go:90 +0x220 fp=0xc000345100 sp=0xc000345020 pc=0xb0ddf0
github.com/hillu/go-yara._cgoexpwrap_08a63f7e5a95_scanCallbackFunc(0x7fb600000001, 0x7fb6c3d6916d, 0x21bb7e0, 0x0)
_cgo_gotypes.go:1621 +0x3d fp=0xc000345130 sp=0xc000345100 pc=0xb08aad
runtime.call32(0x0, 0x7fb6cdffa3f0, 0x7fb6cdffa480, 0x20)
/usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc000345160 sp=0xc000345130 pc=0x461f3b
runtime.cgocallbackg1(0x0)
/usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc000345248 sp=0xc000345160 pc=0x40bd17
runtime.cgocallbackg(0x0)
/usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0003452b0 sp=0xc000345248 pc=0x40bac1
runtime.cgocallback_gofunc(0x40b98f, 0xcd40b0, 0xc000345340, 0xc000345330)
/usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0003452d0 sp=0xc0003452b0 pc=0x46350b
runtime.asmcgocall(0xcd40b0, 0xc000345340)
/usr/local/go/src/runtime/asm_amd64.s:640 +0x42 fp=0xc0003452d8 sp=0xc0003452d0 pc=0x4633a2
runtime.cgocall(0xcd40b0, 0xc000345340, 0x10279e1)
/usr/local/go/src/runtime/cgocall.go:131 +0x7f fp=0xc000345310 sp=0xc0003452d8 pc=0x40b98f
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x7fb6d4000eb0, 0xc001c5d19a, 0x35a000, 0x1, 0xcd3790, 0x21bb7e0, 0x0, 0x0)
_cgo_gotypes.go:1524 +0x4d fp=0xc000345340 sp=0xc000345310 pc=0xb0855d
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback.func1(0xc0000a8960, 0xc001c5d19a, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x21bb7e0, 0x0, 0x7523cf)
/go/src/github.com/hillu/go-yara/rules.go:91 +0xfc fp=0xc0003453a0 sp=0xc000345340 pc=0xb1099c
github.com/hillu/go-yara.(*Rules).ScanMemWithCallback(0xc0000a8960, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x1, 0x0, 0xdc17a0, 0xc000324220, 0x0, 0x0)
/go/src/github.com/hillu/go-yara/rules.go:91 +0x1b7 fp=0xc000345488 sp=0xc0003453a0 pc=0xb0cb27
github.com/hillu/go-yara.(*Rules).ScanMem(...)
/go/src/github.com/hillu/go-yara/rules.go:68
github.secureserver.net/threat/util/yara.(*Rules).ScanBuffer(0xc0001559e0, 0x1054b20, 0xc000022080, 0xc001c5d19a, 0x35a000, 0xd4a656, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.secureserver.net/threat/util/yara/yara.go:84 +0x184 fp=0xc000345660 sp=0xc000345488 pc=0xb142d4
project/classification.(*Plugin).classifySamples(0xc0002b6c80, 0x1054b20, 0xc000022080, 0xc0001f5000, 0xc00014c660, 0x0, 0x17, 0xc000575960)
/go/src/project/classification/classification.go:176 +0x1dd fp=0xc000345800 sp=0xc000345660 pc=0xb16b0d
This stack trace basically shows that we're taking in a sample, attempting to classify it, using ScanBuffer as the mechanism for scanning and returning the matching rules. The top function within the stack trace points to github.com/hillu/go-yara._Cfunc_string_matches
so I've been trying to figure out how this function could cause issues within a docker a container but nowhere else.
As for the ruleset, I have been using a relatively large ruleset for testing and when using rules that just check for the PE header or the Cannot Run in DOS mode string, there are no errors so this could also be the source of error here.
With regards to how I am installing Yara in the docker container, following is a snippet for how I am installing yara on the host that contains the go built application as well
# steps required to install Yara and remove the source files
RUN git clone https://github.com/VirusTotal/yara.git /tmp/yara
WORKDIR ${YARA_SRC_PATH}
RUN \
apt update && \
apt install automake libtool make pkg-config libyara-dev libc-dev -y && \
./bootstrap.sh && \
./configure && \
make && \
make install && \
rm -rf ${YARA_SRC_PATH}
Hi,
go install -tags no_pkg_config github.com/hillu/go-yara
error msg is:
../../github.com/hillu/go-yara/config_yara38.go:12:39: could not determine kind of name for C.YR_CONFIG_MAX_MATCH_DATA
Hi,
Currently we want to use Go-Yara V4 in a Go Modules project. However, due to the go.mod in the master branch not having v4 in the file the tooling is fetching v1.3.0. It can't fetch v4.0.0 as this is disallowed (since the go.mod doesn't have v4, but the fallback to v0.0 or v1.x ..).
Could you bump the go.mod file to v4?
Thanks!
When scanning a byte slice with ScanMem go panics with:
panic: runtime error: cgo argument has Go pointer to Go pointer
goroutine 7 [running]:
panic(0x5fbd00, 0xc82000e8b0)
/usr/lib/go/src/runtime/panic.go:481 +0x3e6
github.com/hillu/go-yara._cgoCheckPointer0(0x5a9be0, 0xc820016fe4, 0x0, 0x0, 0x0, 0x54417e)
??:0 +0x4d
github.com/hillu/go-yara.addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
/home/tom/src/github.com/hillu/go-yara/rules.go:117 +0xce
github.com/hillu/go-yara._cgoexpwrap_f3bc5924bb1c_addString(0xc82000e888, 0x11b3dd4, 0x0, 0xc820016fe4, 0xc800000030)
??:0 +0x47
github.com/hillu/go-yara._Cfunc_yr_rules_scan_mem(0x12360e0, 0xc820016fe4, 0x3b, 0x1, 0x58a900, 0xc82000e888, 0x1, 0x0)
??:0 +0x44
github.com/hillu/go-yara.(*Rules).ScanMem(0xc82002c068, 0xc820016fe4, 0x3b, 0x40, 0x1, 0x3b9aca00, 0x0, 0x0, 0x0, 0x0, ...)
/home/tom/src/github.com/hillu/go-yara/rules.go:147 +0x23d
Using a bytes.Buffer in rules_test.go/TestReader seems to trigger the above error
diff --git a/rules_test.go b/rules_test.go
index ca59966..83b912d 100644
--- a/rules_test.go
+++ b/rules_test.go
@@ -109,7 +109,10 @@ func TestReader(t *testing.T) {
if err != nil {
t.Fatalf("ReadRules: %+v", err)
}
- m, err := r.ScanMem([]byte(" abc "), 0, 0)
+
+ buf := &bytes.Buffer{}
+ buf.Write([]byte(" abc "))
+ m, err := r.ScanMem(buf.Bytes(), 0, 0)
if err != nil {
t.Errorf("ScanMem: %s", err)
}
I use Go v1.5.3 and YARA v3.4 on Ubuntu Linux. I compiled YARA from source (current github master). I also used the --without-crypto --disable-cuckoo and --disable-magic options.
However when I try to install the module I get some errors. (its not a missing yara.h file)
I am new to golang, so it's difficult for me to understand this compiler error.
root@Ronin:/mnt/go-projects/src/github.com/hillu/go-yara# go install github.com/hillu/go-yara
# github.com/hillu/go-yara
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_add_file':
./compiler.go:106: undefined reference to `yr_compiler_add_file'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_add_string':
./compiler.go:118: undefined reference to `yr_compiler_add_string'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_create':
./compiler.go:128: undefined reference to `yr_compiler_create'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_boolean_variable':
./compiler.go:140: undefined reference to `yr_compiler_define_boolean_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_float_variable':
./compiler.go:152: undefined reference to `yr_compiler_define_float_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_integer_variable':
./compiler.go:164: undefined reference to `yr_compiler_define_integer_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_define_string_variable':
./compiler.go:176: undefined reference to `yr_compiler_define_string_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_destroy':
./compiler.go:185: undefined reference to `yr_compiler_destroy'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_get_error_message':
./compiler.go:197: undefined reference to `yr_compiler_get_error_message'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_get_rules':
./compiler.go:208: undefined reference to `yr_compiler_get_rules'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/compiler.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_compiler_set_callback':
./compiler.go:219: undefined reference to `yr_compiler_set_callback'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc__yr_rules_scan_fd':
./rules.go:60: undefined reference to `yr_rules_scan_fd'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_initialize':
./rules.go:69: undefined reference to `yr_initialize'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_boolean_variable':
./rules.go:81: undefined reference to `yr_rules_define_boolean_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_float_variable':
./rules.go:93: undefined reference to `yr_rules_define_float_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_integer_variable':
./rules.go:105: undefined reference to `yr_rules_define_integer_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_define_string_variable':
./rules.go:117: undefined reference to `yr_rules_define_string_variable'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_destroy':
./rules.go:127: undefined reference to `yr_rules_destroy'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_load':
./rules.go:138: undefined reference to `yr_rules_load'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_load_stream':
./rules.go:149: undefined reference to `yr_rules_load_stream'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_save':
./rules.go:160: undefined reference to `yr_rules_save'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_save_stream':
./rules.go:171: undefined reference to `yr_rules_save_stream'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_file':
./rules.go:186: undefined reference to `yr_rules_scan_file'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_mem':
./rules.go:202: undefined reference to `yr_rules_scan_mem'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules.cgo2.o: In function `_cgo_a3213eabeb19_Cfunc_yr_rules_scan_proc':
./rules.go:217: undefined reference to `yr_rules_scan_proc'
/tmp/go-build542103796/github.com/hillu/go-yara/_obj/rules-callback.o: In function `rules_callback':
./rules-callback.c:39: undefined reference to `yr_get_tidx'
collect2: error: ld returned 1 exit status
Thanks, Florian
While cross-compiling from a Debian dev machine to other platforms, building for Darwin and FreeBSD raises this issue instead:
go build github.com/hillu/go-yara/internal/callbackdata: build constraints exclude all Go files in $GOPATH/pkg/mod/github.com/hillu/[email protected]/internal/callbackdata
I've not been able to triage the issue entirely, although it sounds like it could be related to this? golang/go#24068
The flag
argument for ScanProc
function is int
but ScanMem
and ScanFile
use the ScanFlags
type. ScanProc
should expect the ScanFlags
type for keeping the API consistency.
While investigating some other issue I accidentally hit a bug that seems related to the mechanism for passing data to callback functions. By adding the following test case to rules_test.go
you should be able to trigger the bug:
func TestBug(t *testing.T) {
ch := make(chan bool, 0)
go func() {
for i := 0; i < 10000; i++ {
makeRules(t, "rule test { strings: $a = /a.*a/ condition: $a }")
time.Sleep(1 * time.Millisecond)
}
ch <- true
}()
Loop:
for {
select {
case <-ch:
break Loop
default:
runtime.GC()
fmt.Println("GC")
}
}
}
The bug can manifest itself in two ways, by panicing in line 44 of callback-util.go or by raising the following error:
runtime: nelems=512 nfree=501 nalloc=11 previous allocCount=10 nfreed=65535
fatal error: sweep increased allocation count
runtime stack:
runtime.throw(0x4174515, 0x20)
/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0x70000e1b3cd8 sp=0x70000e1b3cb8
runtime.(*mspan).sweep(0x44042f8, 0x4404200, 0x70000e1b3d00)
/usr/local/go/src/runtime/mgcsweep.go:288 +0x786 fp=0x70000e1b3da8 sp=0x70000e1b3cd8
runtime.sweepone(0x0)
/usr/local/go/src/runtime/mgcsweep.go:110 +0x1ad fp=0x70000e1b3e10 sp=0x70000e1b3da8
runtime.gcSweep(0x2)
/usr/local/go/src/runtime/mgc.go:1786 +0xd4 fp=0x70000e1b3e40 sp=0x70000e1b3e10
runtime.gcMarkTermination.func2()
/usr/local/go/src/runtime/mgc.go:1261 +0x62 fp=0x70000e1b3e60 sp=0x70000e1b3e40
runtime.systemstack(0xc42001e600)
/usr/local/go/src/runtime/asm_amd64.s:327 +0x79 fp=0x70000e1b3e68 sp=0x70000e1b3e60
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1132 fp=0x70000e1b3e70 sp=0x70000e1b3e68
In my case commenting out the line fmt.Println("GC")
switch from one error to the other, but your mileage may vary. It was the panic in callback-util.go
what lead me find a possible explanation for what's happening.
Take a look at this:
C.yr_compiler_set_callback(c.cptr, C.YR_COMPILER_CALLBACK_FUNC(C.compilerCallback), unsafe.Pointer(&id))
This passes to the callback the address of id
, as an unsafe.Pointer. My theory is that the garbage collector is moving id
around before compilerCallback
is called. By the time compilerCallback
is called the address where id
was previously stored contains different data.
Thoughts?
I ran some fuzz testing and uncovered this assertion fail:
scan.c:521: int _yr_scan_match_callback(const uint8_t *, int32_t, int, void *): Assertion match_offset + match_length <= callback_args->data_size failed.
This is technically libyara related, but I felt like posting here too.
Fuzzer code:
package main
// commit as of file creation (13.02.2019): 4cc0204f70e49688c0f42637dda8326344c64325
// https://github.com/hillu/go-yara/tree/4cc0204f70e49688c0f42637dda8326344c64325
import "github.com/hillu/go-yara"
//we need import c!
import "C"
//replace original main with dummy main()
func main(){}
//export Fuzz
func Fuzz(data []byte){
c, err := yara.NewCompiler()
if err != nil {
return
}
defer c.Destroy()
err = c.AddString(string(data[:]), "")
if err != nil {
return
}
r, err := c.GetRules()
if err != nil {
return
}
_, _ = r.ScanFile("testfile", 0, 0)
}
Testfile is a PNG file.
Crasher file is attached. crash-0de6903a06f42da60003232dd862790a7bd5ade3.zip
Trying to cross compile from OS X to Linux using Yara 3.8.1. Following appears on build:
[REDACTED]/go/src/github.com/hillu/go-yara/util.go:9:20: undefined: makecbPool
Compiles fine for OS X.
My project ran fine with Go 1.10.2. Since upgrading to Go 1.11.2, I see random crashes, after 30-60 minutes of runtime (several million scans):
exception.h:118: exception_handler: Assertion `FALSE' failed.
SIGABRT: abort
PC=0x80d248 m=16 sigcode=18446744073709551610
Using Yara 3.7.1. Line 118 reads:
assert(FALSE); // We should not reach this point.
Shall I report this with Yara instead? My build script:
export YARA_SRC=/home/willem/src/yara-3.7.1
export CGO_CFLAGS="-I${YARA_SRC}/libyara/include"
export CGO_LDFLAGS="-L${YARA_SRC}/libyara/.libs -lyara -lm" #
go build --ldflags '-w -s -extldflags "-static"' -tags 'yara_static yara3.7 no_pkg_config'
Thanks!
Hey!
We're trying to use go-yara in Google Cloud Functions. Currently it does not work due to pkg_config not being available in Google Cloud Functions
Build failed: # pkg-config --cflags -- yara
Package yara was not found in the pkg-config search path.
Perhaps you should add the directory containing `yara.pc'
to the PKG_CONFIG_PATH environment variable
No package 'yara' found
pkg-config: exit status 1
My guess is that we need to pre-compile Yara and ship it while deploying for it to work?
What do you think?
go test fails with the following panic
panic: runtime error: cgo argument has Go pointer to Go pointer
goroutine 1 [running]:
panic(0x5913e0, 0xc82000a630)
/usr/lib/go/src/runtime/panic.go:464 +0x3e6
github.com/hillu/go-yara._cgoCheckPointer0(0x553360, 0xc820010980, 0x0, 0x0, 0x0, 0x540060)
??:0 +0x4d
github.com/hillu/go-yara.NewCompiler(0x0, 0x0, 0x0)
/home/alexander/go/src/github.com/hillu/go-yara/compiler.go:76 +0x2f7
github.com/hillu/go-yara.Compile(0x60fca0, 0x5d, 0x0, 0x0, 0x0, 0x0)
/home/alexander/go/src/github.com/hillu/go-yara/compiler.go:189 +0x44
github.com/hillu/go-yara.TestMain(0xc820053ef8)
/home/alexander/go/src/github.com/hillu/go-yara/main_test.go:9 +0x39
main.main()
github.com/hillu/go-yara/_test/_testmain.go:100 +0x114
exit status 2
FAIL github.com/hillu/go-yara 0.006s
Apparently go 1.6 enforces the rules documented at golang .
I've detected a strange issue that seems related to the freeing of resources during garbage collection. For reproducing the issue run the following program:
package main
import (
"fmt"
"log"
"net/http"
"os"
yara "github.com/hillu/go-yara"
)
func compile() {
for i := 0; ; i++ {
rules, err := yara.Compile("rule t {condition:true}", nil)
if err == nil {
log.Printf("%d\n", i)
rules.Destroy()
} else {
fmt.Println(err)
os.Exit(1)
}
}
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Println("ok")
})
go compile()
log.Fatal(http.ListenAndServe(":8080", nil))
}
Once it's running do a bunch of requests to the HTTP server:
while true ; do curl localhost:8080 ; done
It can take a while before failing, but after a 5-10 minutes I get an error like this:
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff9d067f44]
runtime stack:
runtime.throw(0x42a4e05, 0x2a)
/usr/local/go/src/runtime/panic.go:596 +0x95
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:274 +0x2db
goroutine 5 [syscall, locked to thread]:
runtime.cgocall(0x420ce10, 0xc420221e38, 0xc4204ca000)
/usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420221e08 sp=0xc420221dc8
github.com/hillu/go-yara._Cfunc_yr_compiler_get_rules(0x30f95e00, 0xc4204ca000, 0xc400000000)
github.com/hillu/go-yara/_obj/_cgo_gotypes.go:790 +0x4d fp=0xc420221e38 sp=0xc420221e08
github.com/hillu/go-yara.(*Compiler).GetRules.func1(0x30f95e00, 0xc4204ca000, 0x0)
/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:181 +0xfc fp=0xc420221e80 sp=0xc420221e38
github.com/hillu/go-yara.(*Compiler).GetRules(0xc4204c85c0, 0x429f4b5, 0x17, 0x0)
/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:181 +0x5b fp=0xc420221ec8 sp=0xc420221e80
github.com/hillu/go-yara.Compile(0x429f4b5, 0x17, 0x0, 0x1, 0x1, 0x0)
/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:205 +0x183 fp=0xc420221f78 sp=0xc420221ec8
main.compile()
/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:12 +0x4a fp=0xc420221fe0 sp=0xc420221f78
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420221fe8 sp=0xc420221fe0
created by main.main
/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:26 +0x5e
goroutine 1 [runnable]:
net.runtime_pollWait(0x4769f70, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005a148, 0x72, 0x0, 0xc4202d5060)
/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005a148, 0xffffffffffffffff, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005a0e0, 0x0, 0x43a53c0, 0xc4202d5060)
/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e030, 0xc4201f3200, 0x4247dc0, 0xffffffffffffffff)
/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e030, 0xc42003ae18, 0xc42003ae20, 0xc42003ae10)
/usr/local/go/src/net/tcpsock.go:215 +0x49
net/http.tcpKeepAliveListener.Accept(0xc42000e030, 0x42a95e8, 0xc4201f3180, 0x43a9240, 0xc420018e10)
/usr/local/go/src/net/http/server.go:3044 +0x2f
net/http.(*Server).Serve(0xc4200a0370, 0x43a8dc0, 0xc42000e030, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2643 +0x228
net/http.(*Server).ListenAndServe(0xc4200a0370, 0xc4200a0370, 0xc42004ff38)
/usr/local/go/src/net/http/server.go:2585 +0xb0
net/http.ListenAndServe(0x429a220, 0x5, 0x0, 0x0, 0x29, 0xc420018c90)
/usr/local/go/src/net/http/server.go:2787 +0x7f
main.main()
/Users/vmalvarez/go/src/github.com/VirusTotal/poc/main.go:27 +0x89
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1
goroutine 4 [runnable, locked to thread]:
github.com/hillu/go-yara._Cfunc_yr_compiler_destroy(0x154dd200)
github.com/hillu/go-yara/_obj/_cgo_gotypes.go:762 +0x41
github.com/hillu/go-yara.(*compiler).finalize.func1(0x154dd200)
/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:80 +0x60
github.com/hillu/go-yara.(*compiler).finalize(0xc4204c1ee0)
/Users/vmalvarez/go/src/github.com/hillu/go-yara/compiler.go:80 +0x2e
I was able to solve this issue by explicitly destroying the compiler in yara.Compile
. I inserted defer c.Destroy()
in https://github.com/hillu/go-yara/blob/master/compiler.go#L196. However I still don't know why this happens. I'm completely new to Go internals, but I've studied your code and everything looks to me, specifically everything related to finalizers.
I think that destroying the compiler explicitly makes sense anyways, but I don't want to send a PR without fully understanding the issue and its possible ramifications.
I'm using go1.8.3 darwin/amd64
with the head version of go-yara
.
Hi there,
It seems that a previously working behaviour, being the ability to pull warnings/errors from the compiler, no longer works.
OSX 10.11.6
$ go version
go version go1.8 darwin/amd64
$ yara -v
yara 3.5.0
$ cd $GOPATH/src/github.com/hillu/go-yara && git rev-parse HEAD
f0a137cfb146d381a6cca4ba9797251bfc5b06f1
To observe this behaviour, run the following code:
package main
import (
"fmt"
"github.com/hillu/go-yara"
)
func main() {
// Build a Yara Compiler
compiler, err := yara.NewCompiler()
if err != nil {
panic(err)
}
// Add an obviously broken rule
compiler.AddString("rule foo { bar }", "default")
// Debug the Compiler Errors/Warnings
fmt.Println(compiler.Errors)
fmt.Println(compiler.Warnings)
}
$ go run main.go
[]
[]
I have found that userData
, when checked at https://github.com/hillu/go-yara/blob/master/compiler.go#L30, is appearing as nil. I am not too familiar with how this should work to offer a fix.
Following the steps on the readme:
# pkg-config --cflags yara
Package yara was not found in the pkg-config search path.
Perhaps you should add the directory containing `yara.pc'
to the PKG_CONFIG_PATH environment variable
No package 'yara' found
pkg-config: exit status 1
A lot of the yr_?_foreach()
macros that are used in the C interface have changed in recent YARA commits expecting an additional YR_SCAN_CONTEXT
parameter.
The same is true for the callback functions given to the yr_?_scan_mem()
functions:
int callback_function(
YR_SCAN_CONTEXT* context,
int message,
void* message_data,
void* user_data);
As it is, go-yara does not compile with VirusTotal/yara@6c91c5d.
Not sure if this is supported but the documentation suggests that it should be possible:
The build tag yara_static can be used to tell the Go toolchain to run pkg-config with the --static switch.
Apparently this is not enough:
mic@elite:~/go/src/github.com/hillu/go-yara$ go build -tags yara_static _examples/simple-yara/simple-yara.go
mic@elite:~/go/src/github.com/hillu/go-yara$ ldd simple-yara
linux-vdso.so.1 (0x00007fffa8162000)
libyara.so.3 => /usr/local/lib/libyara.so.3 (0x00007f24916c3000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f24914a4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f24910b3000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f2490c3b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f249089d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f249192c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2490699000)
Should I be able to build without having to ship the shared object?
I am trying to cross compile go-yara for x86 and x64 and I am getting some errors.
The tool set for cross compiling is xgo.
I can cross compile correctly for darwin
or windows
, but I am unable to do it for linux
.
/usr/local/lib/libyara.a(math.o): In function `string_entropy':
math.c:(.text+0xecc): undefined reference to `log2'
/usr/local/lib/libyara.a(math.o): In function `data_entropy':
math.c:(.text+0x1150): undefined reference to `log2'
collect2: error: ld returned 1 exit status
/usr/bin/ld: skipping incompatible /usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: skipping incompatible //usr/local/lib/libyara.a when searching for -lyara
/usr/bin/ld: cannot find -lyara
collect2: error: ld returned 1 exit status
I am not sure that the errors came either from go-yara or yara itself.
If you need more datils, please ask them.
I'm trying to use go-yara
for a project but found an issue with rules that have long strings, like this one:
rule longstring
{
strings:
$a = { 558bec81ec58010000a1[1]00410033c58945fc568b75085733ff5757576a018d85b4feffff68[1]d24000c785c4feffff0000408489bdbcfeffffc785b4feffff2a2f2a0089bdacfeffffc685c8feffff008985a8feffffff1544d140008985b8feffff3bc775125f33c05e8b4dfc33cde8[2]00008be55dc35357576a0368[1]d2400068[1]d24000575650ff1548d140008bd83bdf0f84010100008b8dc4feffff57518d95a8feffff5268[1]d2400068[1]d240008d85c8feffff5068[1]d2400053ff154cd140008bf03bf70f84cb000000575757578b3d50d1400056ffd785c07524818dc4feffff80f100006a048d8dc4feffff516a1f56ff1554d1400085c00f84960000006a006a006a006a0056ffd785c00f84830000006a008d95c0feffff528d85b0feffff50681300002056c785c0feffff04000000ff1558d1400085c0745981bdb0feffffc800000075268b3d5cd140008d8dc0feffff516a328d55c85256ffd783bdc0feffff0075e7bf01000000eb068bbdbcfeffff568b3560d14000ffd685c0741453ffd685c0740d8b85b8feffff50ffd685c075135b5f33c05e8b4dfc33cde8[1]0e00008be55dc38b4dfc33c083ff015b0f94c05f33cd5ee8[1]0e00008be55dc3 }
condition:
$a
}
Although the string is accepted by YARA command-line tool, in go-yara
it causes a stack overflow:
$ go run main.go
ASAN:DEADLYSIGNAL
=================================================================
==5131==ERROR: AddressSanitizer: stack-overflow on address 0x70000711cb80 (pc 0x00000427c60e bp 0x70000711d3d0 sp 0x70000711cb80 T4)
#0 0x427c60d in _yr_re_emit (libyara.3.dylib+0xe460d)
#1 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#2 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#3 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#4 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#5 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#6 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#7 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#8 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#9 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#10 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#11 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#12 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#13 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#14 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#15 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#16 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#17 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#18 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#19 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#20 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#21 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#22 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#23 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#24 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#25 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#26 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#27 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#28 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#29 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#30 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#31 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#32 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#33 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#34 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#35 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#36 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#37 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#38 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#39 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#40 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#41 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#42 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#43 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#44 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#45 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#46 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#47 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#48 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#49 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#50 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#51 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#52 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#53 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#54 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#55 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#56 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#57 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#58 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#59 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#60 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#61 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#62 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#63 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#64 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#65 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#66 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#67 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#68 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#69 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#70 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#71 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#72 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#73 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#74 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#75 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#76 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#77 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#78 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#79 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#80 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#81 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#82 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#83 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#84 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#85 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#86 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#87 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#88 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#89 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#90 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#91 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#92 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#93 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#94 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#95 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#96 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#97 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#98 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#99 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#100 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#101 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#102 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#103 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#104 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#105 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#106 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#107 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#108 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#109 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#110 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#111 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#112 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#113 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#114 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#115 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#116 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#117 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#118 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#119 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#120 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#121 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#122 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#123 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#124 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#125 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#126 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#127 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#128 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#129 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#130 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#131 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#132 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#133 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#134 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#135 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#136 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#137 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#138 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#139 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#140 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#141 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#142 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#143 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#144 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#145 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#146 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#147 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#148 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#149 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#150 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#151 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#152 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#153 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#154 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#155 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#156 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#157 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#158 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#159 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#160 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#161 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#162 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#163 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#164 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#165 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#166 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#167 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#168 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#169 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#170 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#171 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#172 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#173 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#174 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#175 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#176 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#177 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#178 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#179 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#180 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#181 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#182 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#183 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#184 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#185 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#186 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#187 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#188 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#189 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#190 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#191 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#192 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#193 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#194 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#195 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#196 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#197 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#198 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#199 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#200 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#201 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#202 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#203 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#204 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#205 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#206 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#207 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#208 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#209 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#210 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#211 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#212 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#213 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#214 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#215 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#216 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#217 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#218 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#219 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#220 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#221 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#222 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#223 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#224 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#225 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#226 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#227 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#228 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#229 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#230 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#231 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#232 0x427d046 in _yr_re_emit (libyara.3.dylib+0xe5046)
#233 0x4276142 in yr_re_ast_emit_code (libyara.3.dylib+0xde142)
#234 0x426d9c8 in _yr_parser_write_string (libyara.3.dylib+0xd59c8)
#235 0x426ee81 in yr_parser_reduce_string_declaration (libyara.3.dylib+0xd6e81)
#236 0x41eac60 in yara_yyparse (libyara.3.dylib+0x52c60)
#237 0x425f735 in yr_lex_parse_rules_string (libyara.3.dylib+0xc7735)
#238 0x4218357 in yr_compiler_add_string (libyara.3.dylib+0x80357)
#239 0x409cff4 in _cgo_01fa3c9c21b0_Cfunc_yr_compiler_add_string (main+0x409cff4)
#240 0x405265f in runtime.asmcgocall (main+0x405265f)
Apparently the problem has to do with the stack size of golang
programs. Have you experienced this issue? Do you know how to increase the stack size?
When someone clones this repository the path github.com/hillu/go-yara/internal/callbackdata
imported from utils.go
causes the following error:
util.go:10:2: use of internal package not allowed
As the path contains internal
it can only be imported from code inside the same module (ie: github.com/hillu/go-yara
) the cloned repo has a different path and can't use this import path.
All users cloning this repository for experimentation will be faced with this error and may think that the code in the original repository is broken.
Hi!
Can't seem to load files that has include statements like this:
include "subdir/SomeGreatYaraRules.yar"
I load these files with err := compiler.AddFile(f, "ns")
Hi,
I tried to compile go-yara on AIX 7.2 POWER7 (ppc64) but I get the following error:
bash-5.0$ CGO_LDFLAGS="-L/usr/local/lib -lyara" CGO_CFLAGS="-I/usr/local/include" go.gcc build --gccgoflags="-Xlinker -bbigtoc" -o out -mod vendor -tags="no_pkg_config" aixtest/main.go
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_add_string
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_create
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_boolean_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_float_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_integer_variable
ld: 0711-317 FEHLER: Nicht definiertes Symbol: .yr_compiler_define_string_variable
...
Using -Xlinker -bnoquiet
shows me the following information on top of the error:
(ld): setopt 64
(ld): halt 4
(ld): setopt r/o->w
(ld): setopt bigtoc
(ld): setfflag 4
(ld): savename $WORK/b001/exe/a.out
(ld): filelist 12 1
(ld): i /lib/crt0_64.o
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/crtcxa.o
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/crtdbase.o
(ld): i $WORK/b001/_pkg_.a
(ld): i $WORK/b001/_pkg1_.a
(ld): lib /usr/local/lib/libyara.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgobegin.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgo.a
(ld): lib /usr/lib/libm.a
(ld): lib /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgcc_s.a
(ld): i /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/9.1.0/ppc64/libgcc.a
(ld): lib /usr/lib/libc.a
LIBRARY: Gemeinsam genutztes Objekt libgo.a[libgo.so.14]: Es wurden 44098 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libgcc_s.a[shr.o]: Es wurden 151 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[shr_64.o]: Es wurden 3148 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[posix_aio_64.o]: Es wurden 20 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[aio_64.o]: Es wurden 18 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[pse_64.o]: Es wurden 8 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[dl_64.o]: Es wurden 4 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[pty_64.o]: Es wurden 1 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[cthread_64.o]: Es wurden 25 Symbole importiert.
LIBRARY: Gemeinsam genutztes Objekt libc.a[uchar_64.o]: Es wurden 4 Symbole importiert.
FILELIST: Anzahl der zuvor eingefügten Dateien, die verarbeitet wurden: 12
(ld): resolve
RESOLVE: 1680 von 55377 Symbolen wurden beibehalten.
(ld): addgl /usr/lib/glinkl64.o
ADDGL: Es wurde Glink-Code für 131 Symbole hinzugefügt.
(ld): er full
ld: 0711-318 FEHLER: Es wurden nicht definierte Symbole gefunden.
Die folgenden Symbole sind fehlerhaft:
libyara (3.11.0) was installed with ./bootstrap.sh && ./configure --disable-shared --disable-magic --disable-cuckoo --enable-dotnet && make && make install
It was installed to:
ls /usr/local/lib /usr/local/include
/usr/local/include:
yara yara.h
/usr/local/lib:
libyara.a libyara.la pkgconfig
/usr/local/bin/yara --version
3.11.0
ar -t /usr/local/lib/libyara.a
tests.o
pe.o
elf.o
math.o
time.o
pe_utils.o
hash.o
dotnet.o
grammar.o
ahocorasick.o
arena.o
atoms.o
bitmask.o
compiler.o
endian.o
exec.o
exefiles.o
filemap.o
lt1-hash.o
hex_grammar.o
hex_lexer.o
lexer.o
libyara.o
mem.o
modules.o
object.o
parser.o
proc.o
re.o
re_grammar.o
re_lexer.o
rules.o
scan.o
scanner.o
sizedstr.o
stack.o
stopwatch.o
strutils.o
stream.o
threading.o
none.o
There are no more libyaras installed:
find / -name "libyara.a"
/home/user/yara-3.11.0/libyara/.libs/libyara.a
/usr/local/lib/libyara.a
find / -name "yara.h"
/home/user/yara-3.11.0/libyara/include/yara.h
/usr/local/include/yara.h
I also tried using pkg-config
but this results in same error.
go.gcc version
go version go1.12.2 gccgo (GCC) 9.1.0 aix/ppc64
git -C vendor/github.com/hillu/go-yara/ log | head -n 1
commit 510cc23f05d05782d7f600ee26154542127e6281
Do you know what I am doing wrong here?
Thanks.
It is common to add hashes to test rules on in the meta variables like so:
rule test
{
meta:
// The files that should match this rule. It will be tested against these.
hash = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
hash = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
condition:
true
}
The current API loses the first hash
, because the result is a map keyed by the variable name.
Possible fix, which will put []interface{}
into the metas map[string]interface{}
map:
diff --git a/rule.go b/rule.go
index e0b7e7d..ef0966c 100644
--- a/rule.go
+++ b/rule.go
@@ -132,15 +132,30 @@ func (r *Rule) Metas() (metas map[string]interface{}) {
for _, m := range mptrs {
var id, str *C.char
C.meta_get(m, &id, &str)
+ goId := C.GoString(id)
+
+ var val interface{}
switch m._type {
case C.META_TYPE_NULL:
- metas[C.GoString(id)] = nil
+ val = nil
case C.META_TYPE_STRING:
- metas[C.GoString(id)] = C.GoString(str)
+ val = C.GoString(str)
case C.META_TYPE_INTEGER:
- metas[C.GoString(id)] = int(m.integer)
+ val = int(m.integer)
case C.META_TYPE_BOOLEAN:
- metas[C.GoString(id)] = m.integer != 0
+ val = m.integer != 0
+ default:
+ continue
+ }
+
+ if oldval, prs := metas[goId]; prs {
+ if sl, ok := oldval.([]interface{}); ok {
+ metas[goId] = append(sl, val)
+ } else {
+ metas[goId] = []interface{}{oldval, val}
+ }
+ } else {
+ metas[goId] = val
}
}
return
I'm not too happy with it though, too many interface{}
:/
Not an error, just a warning.
# github.com/hillu/go-yara
../../hillu/go-yara/rules-callback.c: In function 'rules_callback':
../../hillu/go-yara/rules-callback.c:44:61: warning: passing argument 4 of 'addString' discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
addString(user_data, string->identifier, m->offset, m->data, (int)m->data_length);
^
In file included from ../../hillu/go-yara/rules-callback.c:10:0:
cgo-gcc-export-header-prolog:54:13: note: expected 'void *' but argument is of type 'const uint8_t * {aka const unsigned char *}'
https://github.com/malice-plugins/yara/blob/master/Dockerfile
thank you for all your great work!
I have been struggling to get this compiled for windows using MinGW.
It would be good to have more instructions on how to proper compile the Yara project and generate libyara for windows 32 and 64bit. Even thought Yara has the Visual Studio solutions and I can easily generate libyara32.lib and libyara64.lib.... Not sure if these are compatible with MinGW and CGO
And how to have this package use these libs with CGO.
As per the title, would it be possible to provide a scanning interface that accepts an io.Reader
. The implementation could be based on a streaming interface by implementing the YR_MEMORY_BLOCK_ITERATOR*
parameter for yr_scanner_scan_mem_block()
?
With an io.Reader
it'd be much easier to integrate go-yara
in a clean and Go-like manner. Unless I missed something of course - happy to hear about alternatives (currently I believe there's either ScanMem()
, which takes a singular []byte
, ScanFile
, which takes a filename, and ScanFileDescriptor
, which takes a fd) :-)
Hi hillu,
It seems that the (*yara.Rules).Destroy()
function does not free all the memory allocated in C.
I wrote an example that reproduces the problem:
package main
import (
"fmt"
"os"
"runtime"
"strconv"
"time"
yara "github.com/hillu/go-yara"
)
const YaraRule = `rule MemoryLeak_%d {
strings:
$s1 = "memory"
$s2 = "leak"
condition:
1 of them
}`
var variables = map[string]interface{}{
"filename": "",
"filepath": "",
"filetype": "",
"extension": "",
}
func memoryLeak(i int64) {
defer runtime.GC()
r, err := yara.Compile(fmt.Sprintf(YaraRule, i), variables)
if err != nil {
panic(err)
}
defer r.Destroy()
if _, err := r.ScanMem([]byte(fmt.Sprintf("memory leak %d", i)), 0, time.Second); err != nil {
panic(err)
}
}
func main() {
n, err := strconv.ParseInt(os.Args[1], 10, 64)
if err != nil {
panic(err)
}
var i int64
for ; i < n; i++ {
memoryLeak(i)
}
time.Sleep(time.Second)
}
This program compiles rule by rule and destroys the compiled rule before initializing the next one. If Destroy()
would work correctly, the memory usage for n=1
should be approximately the same as for n=10000
.
Here is an program that uses the above compiled program to measure the memory usage peek:
package main
import (
"fmt"
"os/exec"
"syscall"
)
func main() {
for _, n := range []int64{
1, 10, 100, 1000,
} {
cmd := exec.Command("./memoryleak", fmt.Sprint(n))
if err := cmd.Run(); err != nil {
panic(err)
}
kb := cmd.ProcessState.SysUsage().(*syscall.Rusage).Maxrss / 1024
fmt.Printf("%.4d: %.5d KB\n", n, kb)
}
}
This produces the following output on macOS:
0001: 02612 KB
0010: 04704 KB
0100: 06156 KB
1000: 06872 KB
Do you know, weather this is a memory leak or not? I tried the same with a similar program that uses strings.Contains(...)
instead of YARA to exclude that this is a Golang problem:
package main
import (
"fmt"
"os"
"runtime"
"strconv"
"strings"
"time"
)
const ContainsString = "No memory leak %d"
func noMemoryLeak(i int64) {
defer runtime.GC()
if !strings.Contains(fmt.Sprintf(ContainsString, i), fmt.Sprintf("leak %d", i)) {
panic("assertion error")
}
}
func main() {
n, err := strconv.ParseInt(os.Args[1], 10, 64)
if err != nil {
panic(err)
}
var i int64
for ; i < n; i++ {
noMemoryLeak(i)
}
time.Sleep(time.Second)
}
and the measurements were:
0001: 01328 KB
0010: 01480 KB
0100: 01488 KB
1000: 01484 KB
Kind Regards,
Marcel Gebhardt
When a rule fails to compile it causes a crash
ERRO[0000] failed to add rule to compiler: /rules/yara/yara_mixed_ext_vars.yar: undefined identifier "filename" animal=walrus
Assertion failed: compiler->errors == 0 (compiler.c: yr_compiler_get_rules: 825)
SIGABRT: abort
PC=0x7fe370f11f1b m=0 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7fe370f11f1b
stack: frame={sp:0x7ffe3a5bee58, fp:0x0} stack=[0x7ffe3a5ac0f8,0x7ffe3a5bf120)
00007ffe3a5bed58: 00353238203a7365 0000564c0e1b74a0
00007ffe3a5bed68: 00007ffe3a5bf070 0000564c0d6e60c5 <runtime.gentraceback+6117>
00007ffe3a5bed78: 0000564c0df763e0 0000564c0e1b74a0
00007ffe3a5bed88: 0000564c0d6f0da1 <runtime.goexit+1> 00007ffe3a5bee80
00007ffe3a5bed98: 0000002200000000 00000000ffffffff
00007ffe3a5beda8: 00007ffe3a5bedf8 0000564c0d6b03f3 <runtime.sysMap+211>
00007ffe3a5bedb8: 000000c41fff0000 0000000000000001
00007ffe3a5bedc8: 0000002200000003 0000564c0d9bfbcc <github.com/maliceio/malice-yara/vendor/github.com/pkg/errors.Wrapf+236>
00007ffe3a5bedd8: 000000c42005ffe0 0000000000000000
00007ffe3a5bede8: 0000000000000000 00007ffe3a5bee38
00007ffe3a5bedf8: 0000564c0d6ac107 <runtime.persistentalloc1+439> 0000000000000007
00007ffe3a5bee08: 000000c42005f7e8 00007ffe3a5bf0a0
00007ffe3a5bee18: ffffffffffffffff 000000000000015e
00007ffe3a5bee28: 000000000000015d 0000000000000200
00007ffe3a5bee38: 00007fe370f1456f 0000564c0d6eb647 <runtime.persistentalloc.func1+71>
00007ffe3a5bee48: 0000003000000010 0000000000000006
00007ffe3a5bee58: <00007fe370f12035 0000000000000000
00007ffe3a5bee68: 0000000000000000 0000000000000000
00007ffe3a5bee78: 00007fe370cbd628 00007fe370cbdb10
00007ffe3a5bee88: 0000000000000339 0000564c0d6c660a <runtime.main+538>
00007ffe3a5bee98: 0000005000048176 0000564c0dae699e <main.main.func2+382>
00007ffe3a5beea8: 000001f00027dd40 0000000000000000
00007ffe3a5beeb8: 0000000000000000 00007fe37115a2c0
00007ffe3a5beec8: 0000000000000000 ffffffffffffffff
00007ffe3a5beed8: 00007fe370f138fc 000000c42005f7e8
00007ffe3a5beee8: 00007fe370f1543c ffffffffffffffff
00007ffe3a5beef8: 000000c42005f7e8 00007ffe3a5bf0a0
00007ffe3a5bef08: 00007fe370ee9fb4 ffffffffffffffff
00007ffe3a5bef18: 00007fe370eea004 0000564c0d6f0da1 <runtime.goexit+1>
00007ffe3a5bef28: 00007fe370c95163 0000000000000000
00007ffe3a5bef38: 0000000000000000 0000564c0dad2410 <github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run+1520>
00007ffe3a5bef48: 000001d000271e03 0000000000000000
runtime: unknown pc 0x7fe370f11f1b
stack: frame={sp:0x7ffe3a5bee58, fp:0x0} stack=[0x7ffe3a5ac0f8,0x7ffe3a5bf120)
00007ffe3a5bed58: 00353238203a7365 0000564c0e1b74a0
00007ffe3a5bed68: 00007ffe3a5bf070 0000564c0d6e60c5 <runtime.gentraceback+6117>
00007ffe3a5bed78: 0000564c0df763e0 0000564c0e1b74a0
00007ffe3a5bed88: 0000564c0d6f0da1 <runtime.goexit+1> 00007ffe3a5bee80
00007ffe3a5bed98: 0000002200000000 00000000ffffffff
00007ffe3a5beda8: 00007ffe3a5bedf8 0000564c0d6b03f3 <runtime.sysMap+211>
00007ffe3a5bedb8: 000000c41fff0000 0000000000000001
00007ffe3a5bedc8: 0000002200000003 0000564c0d9bfbcc <github.com/maliceio/malice-yara/vendor/github.com/pkg/errors.Wrapf+236>
00007ffe3a5bedd8: 000000c42005ffe0 0000000000000000
00007ffe3a5bede8: 0000000000000000 00007ffe3a5bee38
00007ffe3a5bedf8: 0000564c0d6ac107 <runtime.persistentalloc1+439> 0000000000000007
00007ffe3a5bee08: 000000c42005f7e8 00007ffe3a5bf0a0
00007ffe3a5bee18: ffffffffffffffff 000000000000015e
00007ffe3a5bee28: 000000000000015d 0000000000000200
00007ffe3a5bee38: 00007fe370f1456f 0000564c0d6eb647 <runtime.persistentalloc.func1+71>
00007ffe3a5bee48: 0000003000000010 0000000000000006
00007ffe3a5bee58: <00007fe370f12035 0000000000000000
00007ffe3a5bee68: 0000000000000000 0000000000000000
00007ffe3a5bee78: 00007fe370cbd628 00007fe370cbdb10
00007ffe3a5bee88: 0000000000000339 0000564c0d6c660a <runtime.main+538>
00007ffe3a5bee98: 0000005000048176 0000564c0dae699e <main.main.func2+382>
00007ffe3a5beea8: 000001f00027dd40 0000000000000000
00007ffe3a5beeb8: 0000000000000000 00007fe37115a2c0
00007ffe3a5beec8: 0000000000000000 ffffffffffffffff
00007ffe3a5beed8: 00007fe370f138fc 000000c42005f7e8
00007ffe3a5beee8: 00007fe370f1543c ffffffffffffffff
00007ffe3a5beef8: 000000c42005f7e8 00007ffe3a5bf0a0
00007ffe3a5bef08: 00007fe370ee9fb4 ffffffffffffffff
00007ffe3a5bef18: 00007fe370eea004 0000564c0d6f0da1 <runtime.goexit+1>
00007ffe3a5bef28: 00007fe370c95163 0000000000000000
00007ffe3a5bef38: 0000000000000000 0000564c0dad2410 <github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run+1520>
00007ffe3a5bef48: 000001d000271e03 0000000000000000
goroutine 1 [syscall]:
runtime.cgocall(0x564c0dae81c5, 0xc42005f7e8, 0x564c0de01b60)
/usr/lib/go/src/runtime/cgocall.go:128 +0x66 fp=0xc42005f7b8 sp=0xc42005f780 pc=0x564c0d69c756
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara._Cfunc_yr_compiler_get_rules(0x564c10139020, 0xc42000ec80, 0xc400000000)
_cgo_gotypes.go:1015 +0x4f fp=0xc42005f7e8 sp=0xc42005f7b8 pc=0x564c0d9aed1f
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara.(*Compiler).GetRules.func1(0x564c10139020, 0xc42000ec80, 0x0)
/go/src/github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara/compiler.go:192 +0xd9 fp=0xc42005f820 sp=0xc42005f7e8 pc=0x564c0d9b57a9
github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara.(*Compiler).GetRules(0xc42006a840, 0x0, 0x0, 0x0)
/go/src/github.com/maliceio/malice-yara/vendor/github.com/hillu/go-yara/compiler.go:192 +0x51 fp=0xc42005f860 sp=0xc42005f820 pc=0x564c0d9b1001
main.scanFile(0xc420024500, 0x16, 0x564c0daeb7b3, 0x6, 0x3c, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/maliceio/malice-yara/yara.go:116 +0x8c fp=0xc42005f930 sp=0xc42005f860 pc=0x564c0dae4f1c
main.main.func2(0xc4200c4580, 0x0, 0x0)
/go/src/github.com/maliceio/malice-yara/yara.go:296 +0x17e fp=0xc42005fb28 sp=0xc42005f930 pc=0x564c0dae699e
github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.HandleAction(0x564c0de355a0, 0xc420050df0, 0xc4200c4580, 0x0, 0x0)
/go/src/github.com/maliceio/malice-yara/vendor/github.com/urfave/cli/app.go:490 +0xca fp=0xc42005fb50 sp=0xc42005fb28 pc=0x564c0dad415a
github.com/maliceio/malice-yara/vendor/github.com/urfave/cli.(*App).Run(0xc42009e820, 0xc42001e180, 0x3, 0x3, 0x0, 0x0)
/go/src/github.com/maliceio/malice-yara/vendor/github.com/urfave/cli/app.go:264 +0x5f0 fp=0xc42005fd28 sp=0xc42005fb50 pc=0x564c0dad2410
main.main()
/go/src/github.com/maliceio/malice-yara/yara.go:344 +0x89b fp=0xc42005ff88 sp=0xc42005fd28 pc=0x564c0dae64fb
runtime.main()
/usr/lib/go/src/runtime/proc.go:198 +0x21a fp=0xc42005ffe0 sp=0xc42005ff88 pc=0x564c0d6c660a
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42005ffe8 sp=0xc42005ffe0 pc=0x564c0d6f0da1
rax 0x0
rbx 0x6
rcx 0x7fe370f11f1b
rdx 0x0
rdi 0x2
rsi 0x7ffe3a5bee70
rbp 0x7ffe3a5bee70
rsp 0x7ffe3a5bee58
r8 0x2000
r9 0x3
r10 0x8
r11 0x246
r12 0xffffffffffffffff
r13 0x15e
r14 0x15d
r15 0x200
rip 0x7fe370f11f1b
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
I am scanning files against a large collection of compiled yara rules. I am using yara-4.0.0
I am encountering this error when I try to run the cross-compiled version of the same go code on windows OS.
When I scan the same files with the same yara ruleset on linux, everything goes smoothly. I already searched a bit for this issue and I was hoping getting rid of the hash
import in yara would solve the problem. Unfortunately it did not.
Any idea what might be causing this?
Any ideas? Thanks for making this available!
$ go install -tags yara3.4
# github.com/hillu/go-yara
./rules_callback.go:89:7: could not determine kind of name for C.CALLBACK_MSG_MODULE_IMPORTED
$ dpkg -l | grep yara
ii libyara-dev:amd64 3.4.0+dfsg-2build1 amd64 YARA development libraries and headers
ii libyara3:amd64 3.4.0+dfsg-2build1 amd64 YARA shared library
ii yara 3.4.0+dfsg-2build1 amd64 Pattern matching swiss knife for malware researchers
$ go version
go version go1.10.1 linux/amd64
On 32-bit platforms, the TestWriter test fails as both compiled rule binaries seem to differ:
(sid_armhf-dchroot)satta@abel:~/golang-github-hillu-go-yara$ go test -v
=== RUN TestCompiler
--- PASS: TestCompiler (0.00s)
compiler_test.go:15: expected error: syntax error, unexpected _IDENTIFIER_
=== RUN TestPanic
--- PASS: TestPanic (0.00s)
compiler_test.go:25: Everything ok, MustCompile panicked: syntax error, unexpected _IDENTIFIER_
=== RUN TestWarnings
--- PASS: TestWarnings (0.00s)
compiler_test.go:37: Recorded Errors=[]yara.CompilerMessage{yara.CompilerMessage{Filename:"", Line:1, Text:"syntax error, unexpected _IDENTIFIER_, expecting _CONDITION_"}}, Warnings=[]yara.CompilerMessage(nil)
=== RUN TestCompilerFinalizer
Create compiler
Trigger GC
Trigger Gosched
Manually call destructure on copy
--- PASS: TestCompilerFinalizer (0.00s)
crash_test.go:23: Did not crash due to yr_*_destroy() being called twice. Yay.
=== RUN TestRulesFinalizer
Create rules
Trigger GC
Trigger Gosched
Manually call destructure on copy
--- PASS: TestRulesFinalizer (0.00s)
crash_test.go:40: Did not crash due to yr_*_destroy() being called twice. Yay.
=== RUN TestBooleanOperators
--- PASS: TestBooleanOperators (0.01s)
=== RUN TestComparisonOperators
--- PASS: TestComparisonOperators (0.03s)
=== RUN TestArithmeticOperators
--- PASS: TestArithmeticOperators (0.02s)
=== RUN TestBitwiseOperators
--- PASS: TestBitwiseOperators (0.01s)
=== RUN TestStrings
--- PASS: TestStrings (0.04s)
=== RUN TestWildcardStrings
--- PASS: TestWildcardStrings (0.00s)
=== RUN TestHexStrings
--- PASS: TestHexStrings (0.02s)
=== RUN TestEntrypoint
--- PASS: TestEntrypoint (0.00s)
=== RUN TestFilesize
--- PASS: TestFilesize (0.00s)
=== RUN TestExternals
--- PASS: TestExternals (0.02s)
=== RUN TestModules
--- PASS: TestModules (0.02s)
=== RUN TestIntegerFunctions
--- PASS: TestIntegerFunctions (0.01s)
=== RUN TestSimpleMatch
--- PASS: TestSimpleMatch (0.00s)
rules_test.go:35: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN TestSimpleFileMatch
--- PASS: TestSimpleFileMatch (0.00s)
rules_test.go:50: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN TestSimpleFileDescriptorMatch
--- PASS: TestSimpleFileDescriptorMatch (0.00s)
rules_test.go:65: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN TestEmpty
--- PASS: TestEmpty (0.00s)
rules_test.go:71: Scan of null-byte slice did not crash. Yay.
=== RUN TestLoad
--- PASS: TestLoad (0.00s)
=== RUN TestReader
--- PASS: TestReader (0.01s)
rules_test.go:116: Matches: [{Rule:test Namespace:default Tags:[tag1] Meta:map[author:Hilko Bengen] Strings:[{Name:$a Offset:1 Data:[97 98 99]}]}]
=== RUN TestWriter
--- FAIL: TestWriter (0.25s)
rules_test.go:142: buffers are not equal
=== RUN TestReaderBZIP2
--- PASS: TestReaderBZIP2 (8.93s)
=== RUN TestScanMemCgoPointer
--- PASS: TestScanMemCgoPointer (0.00s)
=== RUN TestFileWalk
--- SKIP: TestFileWalk (0.00s)
stress_test.go:16: Set TEST_WALK to enable scanning files from user's HOME with a dummy ruleset.
(Setting -test.timeout may be a good idea for this.)
FAIL
exit status 1
FAIL _/home/satta/golang-github-hillu-go-yara 9.440s
Dumping both buffers to files, it seems like 4 bytes are different:
(sid_armhf-dchroot)satta@abel:~$ xxd /tmp/f1 > f1.hex
(sid_armhf-dchroot)satta@abel:~$ xxd /tmp/f2 > f2.hex
(sid_armhf-dchroot)satta@abel:~$ diff f1.hex f2.hex
5c5
< 00000040: 0000 c90d 320d b001 0000 e8b8 afb6 161d ....2...........
---
> 00000040: 0000 c90d 320d b001 0000 00c9 0d32 161d ....2........2..
I'm not enough of an expert on Yara to guess what the reason might be, but maybe it helps. libyara 3.6.0 was used here.
This seems to happen at least on armhf and i386, see https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/golang-github-hillu-go-yara.html.
I'm testing your library, and at this attempt:
rules, err := yara.LoadRules(rules_path)
if err != nil {
fmt.Println("Something failed loading rules!");
panic(err)
}
I always get:
Something failed loading rules!
panic: invalid file
goroutine 1 [running]:
panic(0x4a79a0, 0xc420012310)
/usr/lib/go-1.7/src/runtime/panic.go:500 +0x1a1
main.main()
/whatever/whatever/main.go:19 +0x5a7
exit status 2
I am using Yara 3.5.0 and I have tested with pretty much whatever rule.
Am I doing something wrong?
It would be useful to have a way to convert or get a Rules object from a Rule object or a set of Rule objects.
Something like this:
func LoadRule(rule *Rule) (*Rules, error)
In the same line, adding Rule objects to a already Rules object, like this:
func (r *Rules) AddRule(rule *Rule) error
+ export CGO_CFLAGS=-I/usr/local/includeBuilding info Go binary...
+ export CGO_LDFLAGS=-L/usr/local/lib
+ go version
go version go1.6.3 linux/amd64
+ go get
# github.com/hillu/go-yara
../../hillu/go-yara/rules-callback.c: In function 'rules_callback':
../../hillu/go-yara/rules-callback.c:40:76: error: 'YR_MATCH {aka struct _YR_MATCH}' has no member named 'length'
addString(user_data, string->identifier, m->offset, m->data, (int)m->length);
At least on Mac os x when using go-yara with a Yara being installed using brew trhough an error.
# github.com/hillu/go-yara
../../../github.com/hillu/go-yara/compiler.go:15:10: fatal error: 'yara.h' file not found
#include <yara.h>
I found a solution for this error, but I am not sure whether it will work on others systems as well.
The solution is adding this lines on compiler.go
diff --git a/compiler.go b/compiler.go
index 4c5a493..30ddbf8 100644
--- a/compiler.go
+++ b/compiler.go
@@ -5,6 +5,8 @@
package yara
/*
+#cgo pkg-config: --libs-only-L yara
+#cgo pkg-config: --cflags-only-I yara
#ifdef _WIN32
#define fdopen _fdopen
#define dup _dup
Hope this helps.
I copied _examples/simple-yara/simple-yara.go to another directory for testing.
uname -a
Linux test 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux
apt-get install libyara3
apt-get install libyara-dev
ls /usr/lib/x86_64-linux-gnu/libyara* -l
-rw-r--r-- 1 root root 802980 Feb 23 2019 /usr/lib/x86_64-linux-gnu/libyara.a
lrwxrwxrwx 1 root root 16 Feb 23 2019 /usr/lib/x86_64-linux-gnu/libyara.so -> libyara.so.3.9.0
lrwxrwxrwx 1 root root 16 Feb 23 2019 /usr/lib/x86_64-linux-gnu/libyara.so.3 -> libyara.so.3.9.0
-rw-r--r-- 1 root root 373496 Feb 23 2019 /usr/lib/x86_64-linux-gnu/libyara.so.3.9.0
export CGO_CFLAGS="-I/usr/include/yara"
export CGO_LDFLAGS="-L/usr/lib/x86_64-linux-gnu/ -lyara -lm"
go build simple-yara.go
# github.com/hillu/go-yara
In file included from /usr/include/yara.h:33,
from ../../../go/pkg/mod/github.com/hillu/[email protected]/compiler.go:17:
/usr/include/yara/types.h:530:27: error: 'CHAR_BIT' undeclared here (not in a function)
unsigned char tidx_mask[YR_BITARRAY_NCHARS(YR_MAX_THREADS)];
^~~~~~~~~~~~~~~~~~
/usr/include/yara/types.h:530:27: note: 'CHAR_BIT' is defined in header '<limits.h>'; did you forget to '#include <limits.h>'?
/usr/include/yara/types.h:41:1:
+#include <limits.h>
/usr/include/yara/types.h:530:27:
unsigned char tidx_mask[YR_BITARRAY_NCHARS(YR_MAX_THREADS)];
^~~~~~~~~~~~~~~~~~
/usr/include/limits.h exists, I checked /usr/include/yara.h, it includes #include "yara/utils.h"
/usr/include/yara.h includes #include <limits.h>
I can't find the reason.
I found that if I define the same variable twice in the compiler, scanning gives incorrect results.
Here's a minimal testcase:
package main
import (
"fmt"
"github.com/hillu/go-yara"
)
func errcheck(err error) { if err != nil { panic(err) } }
func main() {
c, err := yara.NewCompiler()
errcheck(err)
err = c.DefineVariable("category", "")
errcheck(err)
err = c.DefineVariable("category", "")
errcheck(err)
err = c.AddString(`rule category_is_empty { condition: category == ""}`, "")
errcheck(err)
r, err := c.GetRules()
errcheck(err)
r.DefineVariable("category", "http")
z, err := r.ScanMem([]byte{}, 0, 0)
fmt.Printf("%#v\n", z)
}
One would expect that the rule category_is_empty
wouldn't be matched, since category
was defined to be equal to "http"
. However, the rule does get matched.
If you remove the second definition, the code works as intended.
Hi,
When I try to build the most recent version of go-yara against yara 3.5 it fails with the following error:
go build -tags yara3.5
vendor/github.com/hillu/go-yara/config.go:17:39: could not determine kind of name for C.YR_CONFIG_MAX_STRINGS_PER_RULE
It seems that the config.go
is not correctly excluded for yara 3.5. Building it against yara 3.7 works fine. Previous versions of go-yara also worked fine; building v1.0.8 against yara 3.5 does not generate errors.
Thank you!
Can you try to compile this rules? I get an error unknown module "magic"
, and also undefined identifier "filepath"
import "magic"
rule EXE_cloaked_as_TXT {
meta:
description = "Executable with TXT extension"
license = "https://creativecommons.org/licenses/by-nc/4.0/"
author = "Florian Roth"
condition:
uint16(0) == 0x5a4d // Executable
and magic.mime_type() matches /\.txt$/is // TXT extension (case insensitive)
}
rule Cloaked_RAR_File {
meta:
description = "RAR file cloaked by a different extension"
license = "https://creativecommons.org/licenses/by-nc/4.0/"
author = "Florian Roth"
condition:
uint32be(0) == 0x52617221 // RAR File Magic Header
and not magic.mime_type() matches /(rarnew.dat|\.rar)$/is // not the .RAR extension
and not filepath contains "Recycle" // not a deleted RAR file in recycler
}
I installed yara with all the flags needed, and file
library is present in my system (Fedora XFCE 28)
I have a fresh Debian box with the latest version of Yara (3.11) as well as Golang which is 1.14.1. When running go get github.com/hillu/go-yara
I get the following output:
go install github.com/hillu/go-yara
# github.com/hillu/go-yara
../go/src/github.com/hillu/go-yara/rule.go:144:8: could not determine kind of name for C.META_TYPE_NULL
../go/src/github.com/hillu/go-yara/rule.go:193:27: could not determine kind of name for C.RULE_GFLAGS_GLOBAL
../go/src/github.com/hillu/go-yara/rule.go:188:27: could not determine kind of name for C.RULE_GFLAGS_PRIVATE
cgo:
gcc errors for preamble:
../go/src/github.com/hillu/go-yara/rule.go: In function 'string_matches':
../go/src/github.com/hillu/go-yara/rule.go:81:36: error: macro "yr_string_matches_foreach" requires 3 arguments, but only 2 given
yr_string_matches_foreach(s, match) {
^
../go/src/github.com/hillu/go-yara/rule.go:81:2: error: 'yr_string_matches_foreach' undeclared (first use in this function); did you mean 'string_matches'?
yr_string_matches_foreach(s, match) {
^~~~~~~~~~~~~~~~~~~~~~~~~
string_matches
../go/src/github.com/hillu/go-yara/rule.go:81:2: note: each undeclared identifier is reported only once for each function it appears in
../go/src/github.com/hillu/go-yara/rule.go:81:27: error: expected ';' before '{' token
yr_string_matches_foreach(s, match) {
^ ~
;
By running pkg-config
I get the following:
pkg-config --libs yara
-L/usr/local/lib -lyara
pkg-config --cflags yara
-DUSE_LINUX_PROC -pthread -DMAGIC_MODULE -DHASH_MODULE -I/usr/local/include
Any ideas?
Hey,
I managed to successfully compile go-yara for windows and I am trying to run the example script that was provided. However, when I attempt to read and parse in the rules from a file, I would get Could not parse rule file yara.yar: The storage control block address is invalid.
The rule file yara.yar
contains a sample rule: rule dummy { condition: true }
I am calling my script using go run scanner.go -rule "yara.yar" test.go
Is it possible that I compiled the go-yara binary wrongly or am I using the example script wrongly?
Thank you.
I download and installed the library and wrote a quick script to test if it works.
package main
import (
"os"
"path"
"path/filepath"
log "github.com/Sirupsen/logrus"
"github.com/hillu/go-yara"
)
func CompileAllRules() (*yara.Compiler, *yara.Scanner, error) {
log.Info("Start")
var rule_count = 0
var invalid_rules = 0
compiler, err := yara.NewCompiler()
if err != nil {
log.Info(err)
os.Exit(0)
}
current_path, cerr := os.Executable()
if(cerr != nil){
log.Info(cerr)
os.Exit(0)
}
rules_path := filepath.Dir(current_path)
log.Info("[COMPILER] Looking for Rules in ", rules_path)
_ = filepath.Walk(rules_path, func(filePath string, fileObj os.FileInfo, ferr error) error {
fileName := fileObj.Name()
if (path.Ext(fileName) == ".yar") || (path.Ext(fileName) == ".yara") {
rulesObj, _ := os.Open(filePath)
defer rulesObj.Close()
if(compiler.AddFile(rulesObj, "") != nil){
compiler.Destroy()
a, ferr := yara.NewCompiler()
compiler = a
invalid_rules+=1
if ferr != nil {
log.Info(ferr)
os.Exit(0)
}
}else{
rule_count+=1
}
}
return nil
})
rules, err := compiler.GetRules()
if(err != nil || rules == nil){
log.Info("[COMPILER] Could not get the rules")
os.Exit(0)
}
scanner, err := yara.NewScanner(rules)
if(err != nil){
log.Info("[SCANNER] Could not generate a scanner")
os.Exit(0)
}
log.Info("[COMPILER] Compiled: ", rule_count, " Invalid: ", invalid_rules)
log.Info("[SCANNER] New scanner initiated: ", scanner)
return compiler, scanner, cerr
}
var (
compiler *yara.Compiler
scanner *yara.Scanner
)
func main() {
compiler, scanner, _ = CompileAllRules()
var matches []yara.MatchRule
_ = filepath.Walk("/root", func(filePath string, fileObj os.FileInfo, ferr error) error {
fileName := fileObj.Name()
if (path.Ext(fileName) == ".yar") || (path.Ext(fileName) == ".yara") {
//log.Info("[scanner] Scanning file: ", fileName)
matches, _ = scanner.ScanFile(fileName)
if (len(matches) != 0) {
log.Info("[SCANNER] Mathes found: ", len(matches))
}
}
return nil
})
}
The code simply loops through the current directory looking for valid rules, Compiles them and returns a scanner to scan the files with. However when I try to scan the files which I know for a fact that the YARA rules have a match on, it returns an empty list.
Sample output:
INFO[0000] Start
INFO[0000] [COMPILER] Looking for Rules in /root/yarax-scanner
INFO[0001] [COMPILER] Compiled: 465 Invalid: 67
INFO[0001] [SCANNER] New scanner initiated: &{0x7ffaa842be50 0xc0000a1360 <nil>}
It might be my mistake rather than the library, nevertheless I wanted to ask.
It appears that there is a typo in error.go...
C.ERROR_INSUFICIENT_MEMORY: "insufficient memory",
should be
C.ERROR_INSUFFICIENT_MEMORY: "insufficient memory",
This does not make it work for me yet, but at least now I am getting errors that make some sense...
I've run into a weird issue while playing with go-yara in a test project. After further testing I've found that the same issue occurs in the simple-yara
test thats provided with this repo.
The yara rule I'm using in my testing is a very basic detection for Mimikatz:
rule MimikatzMem
{
strings:
$s1 = "sekurlsa::msv" fullword ascii
$s2 = "sekurlsa::wdigest" fullword ascii
$s4 = "sekurlsa::kerberos" fullword ascii
$s5 = "sekurlsa::tspkg" fullword ascii
$s6 = "sekurlsa::livessp" fullword ascii
$s7 = "sekurlsa::ssp" fullword ascii
$s8 = "sekurlsa::logonPasswords" fullword ascii
$s9 = "sekurlsa::process" fullword ascii
$s10 = "sekurlsa::minidump" fullword ascii
$s11 = "sekurlsa::pth" fullword ascii
$s12 = "sekurlsa::tickets" fullword ascii
$s13 = "sekurlsa::ekeys" fullword ascii
$s14 = "sekurlsa::dpapi" fullword ascii
$s15 = "sekurlsa::credman" fullword ascii
condition:
1 of them
}
As this rule will match on itself the error I'm seeing can be triggered by scanning the rule itself (i.e. ./simple-yara -rule rules.yar rules.yar
)
Doing so produces the following crash:
2020/01/02 14:43:37 Scanning file rules.yar...
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4b4687]
runtime stack:
runtime.throw(0x4fcbd1, 0x2a)
/home/user/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
/home/user/usr/local/go/src/runtime/signal_unix.go:378 +0x47c
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4b4650, 0xc0000bd678, 0x1)
/home/user/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc0000bd648 sp=0xc0000bd610 pc=0x405beb
github.com/hillu/go-yara._Cfunc_string_matches(0x27fa400, 0x0, 0xc00009c0bc)
_cgo_gotypes.go:1070 +0x45 fp=0xc0000bd678 sp=0xc0000bd648 pc=0x4a4e85
github.com/hillu/go-yara.(*String).Matches(0xc0000bd7a0, 0x0, 0x0, 0x0)
/home/user/go/src/github.com/hillu/go-yara/rule.go:225 +0x6b fp=0xc0000bd710 sp=0xc0000bd678 pc=0x4a82eb
github.com/hillu/go-yara.(*Rule).getMatchStrings(0xc00009e050, 0x0, 0x0, 0x0)
/home/user/go/src/github.com/hillu/go-yara/rule.go:244 +0x2c2 fp=0xc0000bd818 sp=0xc0000bd710 pc=0x4a8742
github.com/hillu/go-yara.(*MatchRules).RuleMatching(0xc0000940c0, 0xc00009e050, 0xc0000940c0, 0x7f507c7e6fa8, 0xc0000940c0)
/home/user/go/src/github.com/hillu/go-yara/rules_callback.go:151 +0x224 fp=0xc0000bd978 sp=0xc0000bd818 pc=0x4a95b4
github.com/hillu/go-yara.scanCallbackFunc(0x1, 0x27f9020, 0x278e7e0, 0x40e4b6)
/home/user/go/src/github.com/hillu/go-yara/rules_callback.go:90 +0x220 fp=0xc0000bda58 sp=0xc0000bd978 pc=0x4a8fc0
github.com/hillu/go-yara._cgoexpwrap_08a63f7e5a95_scanCallbackFunc(0x1, 0x27f9020, 0x278e7e0, 0xee341bf499c3200)
_cgo_gotypes.go:1570 +0x3d fp=0xc0000bda88 sp=0xc0000bda58 pc=0x4a595d
runtime.call32(0x0, 0x7ffeaaa3b570, 0x7ffeaaa3b600, 0x20)
/home/user/usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc0000bdab8 sp=0xc0000bda88 pc=0x455abb
runtime.cgocallbackg1(0x0)
/home/user/usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc0000bdba0 sp=0xc0000bdab8 pc=0x405f97
runtime.cgocallbackg(0x0)
/home/user/usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0000bdc08 sp=0xc0000bdba0 pc=0x405d41
runtime.cgocallback_gofunc(0x405c0f, 0x4b4860, 0xc0000bdc98, 0xc0000bdc88)
/home/user/usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0000bdc28 sp=0xc0000bdc08 pc=0x45708b
runtime.asmcgocall(0x4b4860, 0xc0000bdc98)
/home/user/usr/local/go/src/runtime/asm_amd64.s:640 +0x42 fp=0xc0000bdc30 sp=0xc0000bdc28 pc=0x456f22
runtime.cgocall(0x4b4860, 0xc0000bdc98, 0xc0000bdc90)
/home/user/usr/local/go/src/runtime/cgocall.go:131 +0x7f fp=0xc0000bdc68 sp=0xc0000bdc30 pc=0x405c0f
github.com/hillu/go-yara._Cfunc_yr_rules_scan_file(0x27f8ed0, 0x27cc090, 0x0, 0x4b3f00, 0x278e7e0, 0x0, 0x0)
_cgo_gotypes.go:1455 +0x4d fp=0xc0000bdc98 sp=0xc0000bdc68 pc=0x4a542d
github.com/hillu/go-yara.(*Rules).ScanFileWithCallback.func2(0xc00009e038, 0x27cc090, 0x0, 0x278e7e0, 0x0, 0xc0000bddb8)
/home/user/go/src/github.com/hillu/go-yara/rules.go:121 +0xe6 fp=0xc0000bdcf0 sp=0xc0000bdc98 pc=0x4aade6
github.com/hillu/go-yara.(*Rules).ScanFileWithCallback(0xc00009e038, 0x7ffeaaa3d520, 0x9, 0x0, 0x0, 0x4d3240, 0xc0000940c0, 0x0, 0x0)
/home/user/go/src/github.com/hillu/go-yara/rules.go:121 +0x1ce fp=0xc0000bddf8 sp=0xc0000bdcf0 pc=0x4a89de
github.com/hillu/go-yara.(*Rules).ScanFile(...)
/home/user/go/src/github.com/hillu/go-yara/rules.go:100
main.main()
/home/user/go-yara/_examples/simple-yara/simple-yara.go:117 +0x729 fp=0xc0000bdf60 sp=0xc0000bddf8 pc=0x4b3889
runtime.main()
/home/user/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc0000bdfe0 sp=0xc0000bdf60 pc=0x42f0be
runtime.goexit()
/home/user/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x4577d1
As best I can tell this is crashing when trying to get the matches from some C binding related to getting string matches.
Any ideas on how to fix this?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.