egirna/icapeg

Open Source ICAP server

icapeg's Issues

Error: Empty ICAP HEADERS

I tried to run the below command 10 times and I found that the output was different in each time

• some outputs have an empty icap headers
• other commands have correct icap header with 404 error code

The command:
c-icap-client -i 127.0.0.1 -p 1344 -s fakeService -f "./input.pdf" -o ./output -v

Clamav with Google drive

When trying to open google drive this link " https://myaccount.google.com/?utm_source=sign_in_no_continue&pli=1"
with icapeg in Response mode using Clamav services, the icapeg server is down, and the server is being down till restart again.

invalid memory address or nil pointer dereference

testing echo services on the default configuration browsing this url (by squid version 5.7)
https://www.google.com/search?client=firefox-b-d&q=amazon

2022/10/19 13:35:46 icap: panic serving 127.0.0.1:53036: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/root/icapeg/icapeg/icap/server.go:106 +0xdd
panic({0x877520, 0xc45790})
/usr/local/go/src/runtime/panic.go:884 +0x212
io.(*multiReader).Read(0xc00049f068, {0xc0001b1600, 0x200, 0x200})
/usr/local/go/src/io/multi.go:26 +0x95
bytes.(*Buffer).ReadFrom(0xc0001e5ec0, {0x828e89f40, 0xc0004a70d0})
/usr/local/go/src/bytes/buffer.go:202 +0x98
icapeg/api.(*ICAPRequest).preview(0xc0000b86e0?)
/root/icapeg/icapeg/api/icap-request.go:398 +0x217
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc00006acc0, 0xc0?)
/root/icapeg/icapeg/api/icap-request.go:215 +0x5ab
icapeg/api.(*ICAPRequest).RequestProcessing(0xc00006acc0)
/root/icapeg/icapeg/api/icap-request.go:159 +0x7d8
icapeg/api.ToICAPEGServe({0x9a8d38, 0xc0001e5dd0}, 0xc000490c60?)
/root/icapeg/icapeg/api/icap.go:21 +0x85
icapeg/icap.HandlerFunc.ServeICAP(0xc000095ed8?, {0x9a8d38?, 0xc0001e5dd0?}, 0xc00031b90e?)
/root/icapeg/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc0000140b8, {0x9a8d38, 0xc0001e5dd0}, 0xc00012a510)
/root/icapeg/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc0001ea9c0, 0x0?)
/root/icapeg/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/root/icapeg/icapeg/icap/server.go:207 +0x205
2022/10/19 13:35:46 icap: panic serving 127.0.0.1:5990: runtime error: invalid memory address or nil pointer dereference
goroutine 14 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/root/icapeg/icapeg/icap/server.go:106 +0xdd
panic({0x877520, 0xc45790})
/usr/local/go/src/runtime/panic.go:884 +0x212
io.(*multiReader).Read(0xc00049f0b0, {0xc0001b1a00, 0x200, 0x200})
/usr/local/go/src/io/multi.go:26 +0x95
bytes.(*Buffer).ReadFrom(0xc00025c1e0, {0x828e89f40, 0xc0004a7180})
/usr/local/go/src/bytes/buffer.go:202 +0x98
icapeg/api.(*ICAPRequest).preview(0xc0000b88f0?)
/root/icapeg/icapeg/api/icap-request.go:398 +0x217
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc00006ae40, 0xc0?)
/root/icapeg/icapeg/api/icap-request.go:215 +0x5ab
icapeg/api.(*ICAPRequest).RequestProcessing(0xc00006ae40)
/root/icapeg/icapeg/api/icap-request.go:159 +0x7d8
icapeg/api.ToICAPEGServe({0x9a8d38, 0xc00025c0f0}, 0xc000490c60?)
/root/icapeg/icapeg/api/icap.go:21 +0x85
icapeg/icap.HandlerFunc.ServeICAP(0xc000055ed8?, {0x9a8d38?, 0xc00025c0f0?}, 0xc00031bc9e?)
/root/icapeg/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc0000140b8, {0x9a8d38, 0xc00025c0f0}, 0xc00012a750)
/root/icapeg/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc0001eaa80, 0x0?)
/root/icapeg/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/root/icapeg/icapeg/icap/server.go:207 +0x205
https://www.google.com/search?client=firefox-b-d&q=amazon

Error page with reqmod

Does icapeg has dependency on Squid? Can it work with other load balancers like F5, for example?

Disable resp mode in echo servoce

I Disabled the resp mode of echo service in config.toml file:resp_mode=false
then tested icapeg with this command : c-icap-client -i 127.0.0.1 -p 1344 -s echo -f "./in.pdf" -o ./out.pdf -v
the output was empty icap header
the output should be :
405 - Method not allowed for service (e.g., RESPMOD requested for service that supports only REQMOD).

ICAGeg has it difficult to process Multipart/form-data containing multiple uploaded files

If the multipart/form-data post request contains multiple file uploads, our WAF failed to report the file upload containing the EIcar testing attack signature under some scenarios:

when the concerned malicious file upload is NOT the first Content-Disposition

POST /fileupload/ HTTP/1.1
Host: my.hostname.com
Content-Length: 906
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120"
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryroEHrYoGcEwB3fY5

------WebKitFormBoundaryroEHrYoGcEwB3fY5
Content-Disposition: form-data; name="dataService"; filename="blob"
Content-Type: application/json

{"delivery":"EMAIL","exampleReportName":null,"frequency":"WEEKLY","hasExampleReport":false,"hasProductSheet":false,"hasTermsFile":false,"hasImage":false,"id":null,"identification":null,"priceRange1":"45","priceRange2":"454","priceRange3":"454","priceRange4":"445","productInformation":"asdf","productName":"fas","productSheetFileName":"","imageFileName":"image-2023-01-30-08-00-51-575.png","termsFileName":"","types":["ALL_ISINS"],"status":"NOT_ACTIVE","isins":[],"hasImageFile":true}
------WebKitFormBoundaryroEHrYoGcEwB3fY5
Content-Disposition: form-data; name="image"; filename="image-2023-01-30-08-00-51-575.png"
Content-Type: image/png

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
------WebKitFormBoundaryroEHrYoGcEwB3fY5--

However, if I put the same PNG file upload containing the EICar attack signature in the first Content-Disposition or I manually changed filename="blob" into filename123="blob" with the Burp Suite tools, our WAF can successfully block this malicious file upload with Multipart/form-data. I had thought that our WAF has problem to handle the multipart/form-data. All of sudden, I realized that I'm wrong, because our WAF as an ICAP client only encapsulates the income multipart/form-data as a REQMOD message and sends it to the ICAP server who has responsibilities to parse the mulipart/form-data post data, extracting the uploaded files and ask its associated AV scanning service to scan these uploaded files.

After code analyzing your implementation https://github.com/egirna/icapeg/blob/master/service/services-utilities/ContentTypes/multipartForm.go#L45-52, I think I've found the explanations for our observed problems #153

        var theFile FormPart
	for i := 0; i < len(formParts); i++ {
		if formParts[i].FileName != "" {
			theFile = formParts[i]
			break
		}
	}
	return formParts, theFile, boundary

As the above code snippet shows, your implementation is only interested in the firstly found filename specified in the Content-Disposition, which fits well with our observations.

How do you think of collecting all file uploads in the multipart/form-data rather than break after the first matched filename, and then looping them at https://github.com/egirna/icapeg/blob/master/service/services-utilities/ContentTypes/contentType.go#L24 ?

If you think it makes sense, refactoring your implementation in time will be really appreciated!

Preview is needed to be controlled by the service

I test ICAPeg with echo service by squid version 5.7 browsing this url "https://www.google.com/search?client=firefox-b-d&q=lenovo"

2022/10/24 09:11:06 icap: panic serving 127.0.0.1:65183: runtime error: invalid memory address or nil pointer dereference
goroutine 12 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/root/icapeg/icapeg/icap/server.go:106 +0xdd
panic({0x877520, 0xc45790})
/usr/local/go/src/runtime/panic.go:884 +0x212
io.(*multiReader).Read(0xc000286c00, {0xc0001df600, 0x200, 0x200})
/usr/local/go/src/io/multi.go:26 +0x95
bytes.(*Buffer).ReadFrom(0xc000260570, {0x828e89740, 0xc0000a2f70})
/usr/local/go/src/bytes/buffer.go:202 +0x98
icapeg/api.(*ICAPRequest).preview(0xc0000b84d0?)
/root/icapeg/icapeg/api/icap-request.go:398 +0x217
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc00006aba0, 0xc0?)
/root/icapeg/icapeg/api/icap-request.go:215 +0x5ab
icapeg/api.(*ICAPRequest).RequestProcessing(0xc00006aba0)
/root/icapeg/icapeg/api/icap-request.go:159 +0x7d8
icapeg/api.ToICAPEGServe({0x9a8d38, 0xc000260480}, 0xc0003ebd40?)
/root/icapeg/icapeg/api/icap.go:21 +0x85
icapeg/icap.HandlerFunc.ServeICAP(0xc000093ed8?, {0x9a8d38?, 0xc000260480?}, 0xc00002954e?)
/root/icapeg/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc0000140b8, {0x9a8d38, 0xc000260480}, 0xc00012a240)
/root/icapeg/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc000450540, 0x0?)
/root/icapeg/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/root/icapeg/icapeg/icap/server.go:207 +0x205
2022/10/24 09:11:07 icap: panic serving 127.0.0.1:21578: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/root/icapeg/icapeg/icap/server.go:106 +0xdd
panic({0x877520, 0xc45790})
/usr/local/go/src/runtime/panic.go:884 +0x212
io.(*multiReader).Read(0xc000286c48, {0xc0001dfa00, 0x200, 0x200})
/usr/local/go/src/io/multi.go:26 +0x95
bytes.(*Buffer).ReadFrom(0xc000260870, {0x828e89740, 0xc0000a3020})
/usr/local/go/src/bytes/buffer.go:202 +0x98
icapeg/api.(*ICAPRequest).preview(0xc0000b8630?)
/root/icapeg/icapeg/api/icap-request.go:398 +0x217
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc00006ad20, 0xc0?)
/root/icapeg/icapeg/api/icap-request.go:215 +0x5ab
icapeg/api.(*ICAPRequest).RequestProcessing(0xc00006ad20)
/root/icapeg/icapeg/api/icap-request.go:159 +0x7d8
icapeg/api.ToICAPEGServe({0x9a8d38, 0xc000260780}, 0xc0003ebd40?)
/root/icapeg/icapeg/api/icap.go:21 +0x85
icapeg/icap.HandlerFunc.ServeICAP(0xc000055ed8?, {0x9a8d38?, 0xc000260780?}, 0xc00002987e?)
/root/icapeg/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc0000140b8, {0x9a8d38, 0xc000260780}, 0xc00012a480)
/root/icapeg/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc0001e4b00, 0x0?)
/root/icapeg/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/root/icapeg/icapeg/icap/server.go:207 +0x205

Clamav Response mode

When downloading “eicar” virus files using icapeg in response mode with Clamav service
This site ”https://www.eicar.org/download-anti-malware-testfile/”
The value of File Hash didn’t appear in the exception page

Confusing ADDING-NEW-VENDOR.md manual

Hello there.

Is it okay that echo and abc module are mixed here https://github.com/egirna/icapeg/blob/master/ADDING-NEW-VENDOR.md? According to the manual xyz service can't be founded.

Generalize the error page

Description

Making one error page for all the services

Error: txt extension not recognized

testing with echo services
I tried to add "txt" in process_extensions array and value 10 in max_filesize then test 1k txt file, the file passed and it should not.
the txt file only processed when I add "*" to process_extensions

Clamav uploading files

In the case of putting [ pdf, doc, docx, etc. ] in reject extensions, Using icapeg in request mode with Clamav service, They uploaded normally to google drive.
They must be rejected.

Preview not working with zero file byte

I tested Preview exceeding the test file (zero byte file ), the server didn't respond

Error: Try squid with icapeg

I tried to connect squid with icapeg
icapeg running in docker container
squid is running in docker container
icapeg from restructure branch with original config.toml
squid version 5
in squid :

• I Added these lines to squid.conf file

icap_enable on
adaptation_send_username on
adaptation_send_client_ip on
icap_service srv_resp respmod_precache 0 icap://127.0.0.1:1344/echo
#icap_service srv_req reqmod_precache 0 icap://localhost:1344/echo
adaptation_access srv_resp allow all
#adaptation_access srv_req allow all
icap_service_failure_limit -1
icap_preview_enable off

I tested the proxy with curl command

curl -I --proxy "http://localhost:5002" http://www.example.com
result from squid :
HTTP/1.1 500 Internal Server Error
Server: squid/5.4.1
Mime-Version: 1.0
Date: Tue, 28 Jun 2022 14:25:43 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3141
X-Squid-Error: ERR_ICAP_FAILURE 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from 193a33666dbf
Via: 1.1 193a33666dbf (squid/5.4.1)
Connection: keep-alive

result of icapeg logs :

2022/06/28 14:15:41 icap: panic serving 172.18.0.4:37520: runtime error: invalid memory address or nil pointer dereference
goroutine 10 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/home/icapeg/icap/server.go:106 +0xdd
panic({0x804be0, 0xb73ec0})
/usr/local/go/src/runtime/panic.go:838 +0x207
compress/gzip.(*Reader).Read(0x0?, {0xc000149400?, 0x0?, 0x0?})
/usr/local/go/src/compress/gzip/gunzip.go:247 +0x2e
io.ReadAll({0x914100, 0x0})
/usr/local/go/src/io/io.go:645 +0xfe
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
icapeg/service/services-utilities/general-functions.(*GeneralFunc).DecompressGzipBody(0x9142e0?, 0xc00010abd0?)
/home/icapeg/service/services-utilities/general-functions/general-functions.go:125 +0x35
icapeg/service/services/echo.(*Echo).Processing(0xc0002540c0, 0xc5?)
/home/icapeg/service/services/echo/echo.go:67 +0x6a5
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc0000648a0, 0x1e?)
/home/icapeg/api/icap-request.go:146 +0x2b9
icapeg/api.(*ICAPRequest).RequestProcessing(0xc0000648a0)
/home/icapeg/api/icap-request.go:110 +0xbe
icapeg/api.ToICAPEGServe({0x9172a8?, 0xc0002522d0?}, 0xc00049efc0?)
/home/icapeg/api/icap.go:20 +0x48
icapeg/icap.HandlerFunc.ServeICAP(0xc000193ed8?, {0x9172a8?, 0xc0002522d0?}, 0xc00010abea?)
/home/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc000010058, {0x9172a8, 0xc0002522d0}, 0xc000100120)
/home/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc000067400, 0x0?)
/home/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/home/icapeg/icap/server.go:207 +0x205
2022/06/28 14:15:59 icap: panic serving 172.18.0.4:37522: runtime error: invalid memory address or nil pointer dereference
goroutine 52 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/home/icapeg/icap/server.go:106 +0xdd
panic({0x804be0, 0xb73ec0})
/usr/local/go/src/runtime/panic.go:838 +0x207
compress/gzip.(*Reader).Read(0x0?, {0xc00009a200?, 0x0?, 0x0?})
/usr/local/go/src/compress/gzip/gunzip.go:247 +0x2e
io.ReadAll({0x914100, 0x0})
/usr/local/go/src/io/io.go:645 +0xfe
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
icapeg/service/services-utilities/general-functions.(*GeneralFunc).DecompressGzipBody(0x9142e0?, 0xc00010a240?)
/home/icapeg/service/services-utilities/general-functions/general-functions.go:125 +0x35
icapeg/service/services/echo.(*Echo).Processing(0xc0002540c0, 0xc5?)
/home/icapeg/service/services/echo/echo.go:67 +0x6a5
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc0000686c0, 0x1e?)
/home/icapeg/api/icap-request.go:146 +0x2b9
icapeg/api.(*ICAPRequest).RequestProcessing(0xc0000686c0)
/home/icapeg/api/icap-request.go:110 +0xbe
icapeg/api.ToICAPEGServe({0x9172a8?, 0xc0004821b0?}, 0xc00049efc0?)
/home/icapeg/api/icap.go:20 +0x48
icapeg/icap.HandlerFunc.ServeICAP(0xc00004eed8?, {0x9172a8?, 0xc0004821b0?}, 0xc00010a25a?)
/home/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc000010058, {0x9172a8, 0xc0004821b0}, 0xc00016e120)
/home/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc000066000, 0x0?)
/home/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/home/icapeg/icap/server.go:207 +0x205
2022/06/28 14:17:14 ICAP server gracefully shut down
2022/06/28 14:17:29 starting the ICAP server
2022/06/28 14:17:29 ICAP server is running on localhost: 1344
2022/06/28 14:21:43 icap: panic serving 172.18.0.4:37528: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/home/icapeg/icap/server.go:106 +0xdd
panic({0x804be0, 0xb73ec0})
/usr/local/go/src/runtime/panic.go:838 +0x207
compress/gzip.(*Reader).Read(0x0?, {0xc000234200?, 0x0?, 0x0?})
/usr/local/go/src/compress/gzip/gunzip.go:247 +0x2e
io.ReadAll({0x914100, 0x0})
/usr/local/go/src/io/io.go:645 +0xfe
io/ioutil.ReadAll(...)
/usr/local/go/src/io/ioutil/ioutil.go:27
icapeg/service/services-utilities/general-functions.(*GeneralFunc).DecompressGzipBody(0x9142e0?, 0xc00010a270?)
/home/icapeg/service/services-utilities/general-functions/general-functions.go:125 +0x35
icapeg/service/services/echo.(*Echo).Processing(0xc00021c0c0, 0xc5?)
/home/icapeg/service/services/echo/echo.go:67 +0x6a5
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc0005861e0, 0x1e?)
/home/icapeg/api/icap-request.go:146 +0x2b9
icapeg/api.(*ICAPRequest).RequestProcessing(0xc0005861e0)
/home/icapeg/api/icap-request.go:110 +0xbe
icapeg/api.ToICAPEGServe({0x9172a8?, 0xc0003dc330?}, 0xc0004b8fc0?)
/home/icapeg/api/icap.go:20 +0x48
icapeg/icap.HandlerFunc.ServeICAP(0xc00004ded8?, {0x9172a8?, 0xc0003dc330?}, 0xc00010a28a?)
/home/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc000010058, {0x9172a8, 0xc0003dc330}, 0xc00016e240)
/home/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc000066000, 0x0?)
/home/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/home/icapeg/icap/server.go:207 +0x205

Transformers unit tests

In this pr I created 3 files in transformers directory which contains unit tests for vmray, metadefender and virustotal files

Uploaded “eicar” files

Uploading “eicar” files to Google Drive normally using icapeg in request mode using clhashlookup service. It must be rejected because it virus file.

these files "https://www.eicar.org/download-anti-malware-testfile/"

Clamav with doc files

In the case of putting “doc” in reject extensions. try to download it with icapeg with Clamav service in response mode, It downloaded normally instead of a rejection

Adding embedded Yara rules service

Adding an embedded Yara rules vendor.

How ICAPeg will inform our ICAP client of the AV scanning results of an infected file

When the AV Scanning results of a posted file, e.g ZIP or PDF with ClamAV is that this file is not safe, that is, infected, how does the ICAP response from ICAPeg look like and how our ICAP client can interpret the ICAP response that this uploaded file is infected?

2023-02-22T15:36:16.203Z info general-functions/general-functions.go:49 extracting the body of HTTP message
2023-02-22T15:36:16.204Z info general-functions/general-functions.go:262 getting the file name
2023-02-22T15:36:16.204Z info general-functions/general-functions.go:442 getting the mime extension of the HTTP message body
2023-02-22T15:36:16.204Z debug general-functions/general-functions.go:462 HTTP message body mime extension is zip
2023-02-22T15:36:16.204Z info general-functions/general-functions.go:71 checking the extension (reject or bypass or process)
2023-02-22T15:36:16.204Z debug general-functions/general-functions.go:75 extension is process
2023-02-22T15:36:16.204Z debug clamav/clamav.go:84 sending the HTTP msg body to the ClamAV through antivirus socket
2023-02-22T15:36:21.205Z debug clamav/clamav.go:111 clamavFile is not safe
2023-02-22T15:36:21.207Z debug api/icap-request.go:189 adding the headers which the service wants to add them in the ICAP response
2023-02-22T15:36:21.207Z debug api/icap-request.go:198 checking if shadow service mode is enabled to add logs instead of returning another
2023-02-22T15:36:21.207Z debug api/icap-request.go:237 clamav returned ICAP response with status code 200

Bug: filetype pkg makes a mistake in getting the file type of some files

The pkg gets the file type of a file with the name eicar.com.txt com which is a wrong type, the right type is txt.

go report badge Error

when trying to get a report from https://goreportcard.com/ I got this error

There was an error processing your request: Could not analyze the repository: could not download repo: could not get latest module version from https://proxy.golang.org/icapeg/@latest: bad request: invalid escaped module path "icapeg": malformed module path "icapeg": missing dot in first path element

Known issue: error page in reqmod

Issue: test method in req mode

I tried to test request mode with CONNECT method with this command
c-icap-client -i 127.0.0.1 -p 1344 -s echo -f ./input.pdf -o ./toutput -req http://www.example.com -method CONNECT -no204 -v

the return header is ICAP/1.0 204 No modifications needed and there is no file returned

Also when I used HEAD method the icap header was empty and empty file created

Note: Other methods works fine .

`

icapeg different behavior when running as non-root

Hello, I’ve experienced different icapeg behavior depending on the mode application is running. When I run icapeg as a root user application runs correctly, and doesn’t show any errors in the console. When testing with c-icap-client it correctly detects malicious files, but when icapeg is run as non-root user I got the following error in the console also application treat malicious file as healthy one:

Icapeg environment

Openshift 4.12.14v

Application deployed as pods, both tested pods are run with the same image. Differences are security contexts

• non-root icapeg security context
  
• root icapeg security context
  

Icapeg behavior

When icapeg is running as root user

Logs

When icapeg is running as non root user

Logs

Current icapeg configuration

title = "ICAP configuration file"

[app]
port = 1344
log_level="debug"
write_logs_to_console= true
services= ["echo", "virustotal", "clamav", "cloudmersive", "grayimages", "hashlookup"]
debugging_headers=true
web_server_host = "$_WEB_SERVER_HOST"
web_server_endpoint = "$_WEB_SERVER_ENDPOINT"

[echo]
vendor = "echo"
service_caption= "echo service"   #Service
service_tag = "ECHO ICAP"  #ISTAG
req_mode=true
resp_mode=true
shadow_service=false
preview_bytes = "1024" #byte
preview_enabled = true# options send preview header or not
process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = []
bypass_extensions = ["*"]
#max file size value from 1 to 9223372036854775807, and value of zero means unlimited
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=false
return_400_if_file_ext_rejected=false

[clamav]
vendor = "clamav"
service_caption= "clamav service"   #Service
service_tag = "CLAMAV ICAP"  #ISTAG
req_mode=true
resp_mode=true
shadow_service=false
preview_bytes = "1024" #byte
preview_enabled = true# options send preview header or not
process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = ["docx"]
bypass_extensions = ["*"]
socket_path = "/var/tmp/clamd.sock"
timeout = 10 #seconds, the time upto which the server will wait for clamav to scan the results
#max file size value from 1 to 9223372036854775807, and value of zero means unlimited
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=false
return_400_if_file_ext_rejected=false

[virustotal]
vendor = "virustotal"
service_caption= "virustotal service"   #Service
service_tag = "Virustotal ICAP"  #ISTAG
req_mode=true # enable support for ICAP request mode
resp_mode=true # enable support for ICAP response mode
shadow_service=false
preview_bytes = "1024" #byte
preview_enabled = true# options send preview header or not
process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = ["docx"]
bypass_extensions = ["*"]
scan_url = "https://www.virustotal.com/vtapi/v2/file/scan" #
report_url = "https://www.virustotal.com/vtapi/v2/file/report" #
api_key = "$_VIRUSTOTAL_API_KEY"
timeout  = 1000 #seconds , ICAP will return 408 - Request timeout
#max file size value from 1 to 9223372036854775807, and value of zero means unlimited
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=false
return_400_if_file_ext_rejected=false


[cloudmersive]
vendor = "cloudmersive"
service_caption= "cloudmersive service"   #Service
service_tag = "CLOUDMERSIVE ICAP"  #ISTAG
req_mode=true
resp_mode=true
shadow_service=false
preview_enabled = false# options send preview header or not
preview_bytes = "1024" #byte
timeout  = 300 #seconds , ICAP will return 408 - Request timeout
#max file size value from 1 to 9223372036854775807, and value of zero means unlimited
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=false
return_400_if_file_ext_rejected=false
process_extensions = ["pdf", "zip", "com"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = ["docx"]
bypass_extensions = ["*"]
verify_server_cert=false
base_url = "$_CLOUDMERSIVE_BASE_URL" #
scan_endpoint = "/virus/scan/file/advanced"
api_key = "$_CLOUDMERSIVE_API_KEY"
allow_executables = true
allow_invalid_files = false
allow_scripts = true
allow_password_protected_files = false
allow_macros = false
allow_xml_external_entities = false
allow_html = true
allow_insecure_deserialization = false

[grayimages]
vendor = "grayimages"
service_caption= "grayimages service"   #Service
service_tag = "GRAYIMAGES ICAP"  #ISTAG
req_mode=true
resp_mode=true
shadow_service=false
preview_bytes = "1024" #byte
preview_enabled = true# options send preview header or not
bypass_extensions = ["*"]
process_extensions = ["png","jpg","jpeg","webp"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = [""]
base_url = "" #
scan_endpoint = "<grayimages>"
api_key = "<api key>"
timeout  = 300 #seconds , ICAP will return 408 - Request timeout
fail_threshold = 2
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=true
return_400_if_file_ext_rejected=false

[hashlookup]
vendor = "hashlookup"
service_caption= "hash lookup"   #Service
service_tag = "HASHLOOKUP ICAP"  #ISTAG
req_mode=true
resp_mode=true
shadow_service=false
preview_bytes = "1024" #byte
preview_enabled = true# options send preview header or not
bypass_extensions = ["*"]
process_extensions = ["pdf","exe", "zip"] # * = everything except the ones in bypass, unknown = system couldn't find out the type of the file
reject_extensions = ["docx"]
scan_url = "http://23.106.36.101:5555/is-malicious" #
timeout  = 300 #seconds , ICAP will return 408 - Request timeout
fail_threshold = 2
max_filesize = 0 #bytes
return_original_if_max_file_size_exceeded=true
return_400_if_file_ext_rejected=false

Upload files

In the case of putting [ pdf, doc, docx, etc. ] in reject extensions, Using icapeg in request mode with clhashlookup service, They uploaded normally to google drive.
They must be rejected.

we can not get the file name scanned, why?

when download virus file from remote, with this icap and squid both deployed.
why the block page dont show the file name

thank you

Clamav Service uploading virus file

Uploading “eicar” files to Google Drive normally using icapeg in request mode using Clamav service. It must be rejected because it virus file.

these files "https://www.eicar.org/download-anti-malware-testfile/"

Test ICAPeg with OPNsense in response mode and preview size=0 ,by browsing this site " https://file-examples.com/index.php/sample-documents-download/sample-doc-download/" gives me this error

2022/10/31 09:54:59 icap: panic serving 127.0.0.1:2834: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
icapeg/icap.(*conn).serve.func1()
/root/icapeg/icapeg/icap/server.go:106 +0xdd
panic({0x877520, 0xc45790})
/usr/local/go/src/runtime/panic.go:884 +0x212
io.(*multiReader).Read(0xc000013980, {0xc0001b1e00, 0x200, 0x200})
/usr/local/go/src/io/multi.go:26 +0x95
bytes.(*Buffer).ReadFrom(0xc000162a50, {0x828e88f18, 0xc0001969d0})
/usr/local/go/src/bytes/buffer.go:202 +0x98
icapeg/api.(*ICAPRequest).preview(0xc0000b8580?)
/root/icapeg/icapeg/api/icap-request.go:398 +0x217
icapeg/api.(*ICAPRequest).RespAndReqMods(0xc00006b200, 0xc0?)
/root/icapeg/icapeg/api/icap-request.go:215 +0x5ab
icapeg/api.(*ICAPRequest).RequestProcessing(0xc00006b200)
/root/icapeg/icapeg/api/icap-request.go:159 +0x7d8
icapeg/api.ToICAPEGServe({0x9a8d38, 0xc000162960}, 0xc0000ab200?)
/root/icapeg/icapeg/api/icap.go:21 +0x85
icapeg/icap.HandlerFunc.ServeICAP(0xc000093ed8?, {0x9a8d38?, 0xc000162960?}, 0xc00047646e?)
/root/icapeg/icapeg/icap/server.go:39 +0x2f
icapeg/icap.(*ServeMux).ServeICAP(0xc0000140b8, {0x9a8d38, 0xc000162960}, 0xc00012a3f0)
/root/icapeg/icapeg/icap/mux.go:94 +0x448
icapeg/icap.(*conn).serve(0xc000198680, 0x0?)
/root/icapeg/icapeg/icap/server.go:127 +0x7f
created by icapeg/icap.(*Server).Serve
/root/icapeg/icapeg/icap/server.go:207 +0x205

Fixing multiple bugs and adding some features