A fast Golang library for media type and file extension detection, based on magic numbers
Home Page: https://pkg.go.dev/github.com/gabriel-vasile/mimetype#pkg-overview
License: MIT License
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Attach the file for which the detection is inaccurate
You can use the file sharing site of your choice, just ensure the file will not expire or get deleted.
Expected mime type
text/xml; charset=utf-8
Returned mime type
text/plain
Output of go version
go 1.12
Additional context
<resources>
<string name="scan_tabs_left_button">Scan code</string>
<string name="scan_tabs_right_button">My address</string>
<string name="my_address">My Address</string>
<string name="addresses_copied">Copied to clipboard.</string>
<string name="copy_address">Copy Address</string>
</resources>
I have started using your fine library for the detection of mimetypes, but I ran into a panic the other day in ms_office.go.
Here is the trace:
runtime error: slice bounds out of range [995:989]
/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/gabriel-vasile/mimetype/internal/matchers.msoXML(0xc018d6e000, 0x3fb, 0x80000, 0xc00dc3fa2d, 0x3, 0x3, 0xc00dc3fa70)
/go/pkg/mod/github.com/gabriel-vasile/[email protected]/internal/matchers/ms_office.go:27 +0x2bf
github.com/gabriel-vasile/mimetype/internal/matchers.Xlsx(0xc018d6e000, 0x3fb, 0x80000, 0x44ef4e)
/go/pkg/mod/github.com/gabriel-vasile/[email protected]/internal/matchers/ms_office.go:43 +0x67
github.com/gabriel-vasile/mimetype.(*MIME).match(0xc000133080, 0xc018d6e000, 0x3fb, 0x80000, 0xc000133080, 0xc00dc3faf8)
/go/pkg/mod/github.com/gabriel-vasile/[email protected]/mime.go:79 +0x90
github.com/gabriel-vasile/mimetype.(*MIME).match(0xc0001336e0, 0xc018d6e000, 0x3fb, 0x80000, 0xc0001336e0, 0xc008289280)
/go/pkg/mod/github.com/gabriel-vasile/[email protected]/mime.go:80 +0xc8
github.com/gabriel-vasile/mimetype.Detect(0xc018d6e000, 0x3fb, 0x80000, 0x0)
/go/pkg/mod/github.com/gabriel-vasile/[email protected]/mimetype.go:26 +0xaf
From what I can see its taking a byteslice at line 27 without checking the slices capacity:
for i := 0; i < 4; i++ {
in = in[lastCheckedIndex:]It seems to me that should be fixed :). I'm willing to submit a PR if that helps.
Just leaving some thoughts here about a use-case that we have...
We use mimetype library to detect if declared (via Content-Type header) MIME type actually matches the real content type. However, one media file can map to multiple MIME types, which are often alternative IANA names of the same type. Public APIs of mimetype library only returns one MIME type.
For example, Content-Type may be set to audio/x-wav and the type detected by mimetype shall be audio/wav
We internally maintain aliases as shown below, which works for us.
// NOTE: aliases for MIME type detected by
// https://github.com/gabriel-vasile/mimetype/blob/master/supported_mimes.md
// mimeAliases is map in which key is a MIME type detected by
// gabriel-vasile/mimetype and value is list of alternative names
// for same MIME type
var mimeAliases = map[string][]string{
"audio/wav": {"audio/wave", "audio/x-wav", "audio/vnd.wave"},
"audio/midi": {"audio/mid", "audio/sp-midi", "audio/x-mid", "audio/x-midi"},
"audio/amr": {"audio/amr-nb"},
"audio/mpeg": {"audio/mp3"},
"video/x-msvideo": {"video/avi"},
"video/3gpp": {"video/3gp", "audio/3gpp"},
"video/3gpp2": {"video/3g2", "audio/3gpp2"},
"video/x-ms-asf": {"video/asf", "video/x-ms-wmv"},
"video/webm": {"audio/webm"},
}Is this something mimetype wants to handle/implement ? This could look like:
IsWav(in []byte, mimetype string) boolor via a single API:
Matches(in []byte, mimetype string) boolIf we use matchers.ReadLimit then the test fails with use of internal package github.com/gabriel-vasile/mimetype/internal/matchers not allowed. matchers.ReadLimit is needed to implement a custom readers.
When detecting from a reader, the caller will need to seek to the beginning of a file prior to calling
mimetype.DetectReader(file)
How to rewind an os.File
// Reset file pointer to start of file
_, err = file.Seek(0, 0)
if err != nil {
return nil, err
}
This should be noted in the Readme
application/ventuz-scene & .vzsapplication/ventuz-scene-archive & .vzaIs that something feasible?
<?xml version="1.0" encoding="UTF-8"?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.2" xsi:schemaLocation="urn:oasis:names:tc:xliff:document:1.2 http://docs.oasis-open.org/xliff/v1.2/os/xliff-core-1.2-strict.xsd">
<file original="App/Base.lproj/CBMain.storyboard" source-language="en" target-language="en" datatype="plaintext">
<header>
<tool tool-id="com.apple.dt.xcode" build-num="10E1001"/>
</header>
<body>
<trans-unit id="JPg-N1-33I.title">
<source>Prices</source>
<target>Prices</target>
<note>Class = "UITabBarItem"; title = "Prices"; ObjectID = "JPg-N1-33I";</note>
</trans-unit>
<trans-unit id="Rl0-UQ-u6L.title">
<source>Settings</source>
<target>Settings</target>
<note>Class = "UITabBarItem"; title = "Settings"; ObjectID = "Rl0-UQ-u6L";</note>
</trans-unit>
<trans-unit id="Vvv-j3-51d.title">
<source>Invite</source>
<target>Invite</target>
<note>Class = "UITabBarItem"; title = "Invite"; ObjectID = "Vvv-j3-51d";</note>
</trans-unit>
<trans-unit id="xxg-wY-IeR.title">
<source>Alerts</source>
<target>Alerts</target>
<note>Class = "UITabBarItem"; title = "Alerts"; ObjectID = "xxg-wY-IeR";</note>
</trans-unit>
<trans-unit id="zV5-gU-jfH.title">
<source>Accounts</source>
<target>Accounts</target>
<note>Class = "UITabBarItem"; title = "Accounts"; ObjectID = "zV5-gU-jfH";</note>
</trans-unit>
</body>
</file>
</xliff>
Shapefile is a little bit complex because you need to manage multiples files. For a Shapefile you have 3 mandatory files :
I think it could be a good thing to add this 3 types of files in mimetype.
You can find examples here
https://file-examples.com/wp-content/uploads/2017/02/file_example_XLS_10.xls
application/vnd.ms-excel
application/msword
https://github.com/gabriel-vasile/mimetype master branch
go version go1.11.2 linux/amd64
.
Attach the file for which the detection is inaccurate
while call DetectReader or DetectFile while the in 2048（matchers.ReadLimit） bytes starts with '{' and ends with "type" like in[0] = '{' and in[2042:] = "type" or call Detect like this case, panic~~
Expected mime type
application/json
Returned mime type
panic
Output of go version
go1.12.4
Additional context
.
function GeoJson looks like it should check slice bounds is discussed，and node.match to recovery panic, Instead of throwing exceptions directly to the caller
Attach the file for which the detection is inaccurate
You can use the file sharing site of your choice, just ensure the file will not expire or get deleted.
Expected mime type
text/csv
Returned mime type
text/plain
Output of go version
go1.12
Additional context
Here's the CSV file I'm trying to validate:
key,context,string
identity_verifications.legacy_complete.Success,,Success
identity_verifications.legacy_complete.Your information has been recorded. Thank you.,,Your information has been recorded. Thank you.
identity_verifications.legacy_complete.Return to Home Page,,Return to Home Page
identity_verifications.rate_limit.Try again later,,Try again later
identity_verifications.rate_limit.You can try again in %{time_from_now}.,,You can try again in %{time_from_now}.
"identity_verifications.rate_limit.We located your ID, but your question answers were wrong. You can try again in %{time_from_now}.",,"We located your ID, but your question answers were wrong. You can try again in %{time_from_now}."
"identity_verifications.rate_limit.We couldn't locate your ID, but you can try again in %{time_from_now}.",,"We couldn't locate your ID, but you can try again in %{time_from_now}."
"identity_verifications.rate_limit.We couldn't verify your identity right now, but you can try again in %{time_from_now}.",,"We couldn't verify your identity right now, but you can try again in %{time_from_now}."
shared.general.save,,Save
Line 110 in f4632f4
Depending on git settings the beginning of foobar.fb will be foobar\n or foobar\r\n, causing the test to fail in the latter case.
Would it suffice to just have it match "foobar"?
test.xlsx
Attach the file for which the detection is inaccurate
You can use the file sharing site of your choice, just ensure the file will not expire or get deleted.
**Expected mime
xlsx
**Returned mime
zip
Version of the library you are using
the semver version or the hash of the commit
1fe33ab
Output of go version
go 1.13
test file
{
"AllowIP": [
"127.0.0.1"
],
"AllowRawReadingFromPath": [
"c:/j(/|/.*)?$",
"./.*"
],
"ArgsLoggingExclude": [],
"Binding": "0.0.0.0:53638",
"DebugLevel": "debug",
"DebugLevel1": "error",
"EnableAuthentication": true,
"EnableIPAcl": false,
"LogMaxAge": 10,
"LogMaxBackups": 3,
"LogMaxSize": 5,
"RawReadingFilesEnabled": false,
"RawReadingMaxKbytes": 4096,
"SQL": {
"BackupCount": 5,
"BackupDir": "backups",
"BackupPeriod": 1,
"Queries": [
{
"Name": "migrations",
"Query": "SELECT * FROM ALALA;"
}
]
},
"SvcDescription": "testing environment",
"SvcDisplayName": "testing",
"SvcNameSuffix": "",
"TemplateData": {},
"WorkerPool": {
"Max": 5,
"Timeout": 5
}
}
test code
package main
import (
"fmt"
"io/ioutil"
"github.com/gabriel-vasile/mimetype"
)
func main() {
data, _ := ioutil.ReadFile("test.json")
fmt.Println(mimetype.Detect(data))
fmt.Println(mimetype.DetectFile("test.json"))
}
Result
application/json json
text/plain txt <nil>
Currently there are several matching functions not implemented, ex: doc, xls, flac, wav. Because of this there are file formats which are not detected by the library.
A good resource for file signatures is https://www.garykessler.net/library/file_sigs.html
Attach the file for which the detection is inaccurate
Tricky, as the file is copyrighted material, however the problem is fairly obvious.
Expected mime type
application/zip (.zip)
Returned mime type
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet (.xlsx)
Version of the library you are using
0.3.21
Output of go version
go version go1.13.1 darwin/amd64
Additional context
It looks like the detection code is looking for the string/bytes "xl/" anywhere in the file when I'm guessing the intent is to look for a file path in the TOC with that directory, I can see in the problem file that it just happens to contain those 3 bytes as part of the compressed data.
Hi,
what about "image/svg"? I think it's pretty important and simple at the same time, just consider <svg> tag.
Specify the MIME type and extension for which to add support
application/x-msaccess, .accdb
Share an example file
can be found online
Optionally, add a reference to the specification of the file format.
magic
Hi,
The documentation suggests that text/plain or similar is returned by the String() method, but I got text/plain; charset=utf-8 when parsing some bytes with plaintext.
Is there a way to return just the charset, or just the mimetype?
Attach the file for which the detection is inaccurate
https://github.com/strider2038/mimetype/commit/4678076d99a9a462c42069644906225a63a5f57d
Expected mime type
application/msword
Returned mime type
application/octet-stream
Output of go version
go version go1.12.3 linux/amd64
Looks like you've made some progress adding SVG etc matchers for text files -- what about JSON? That would be very helpful! Thanks for what looks like a really nice package.
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1]
},
"properties": {
"name": "Dinagat Islands"
}
}Example of the official website.
TIA ;)
Go built-in mime package has ExtensionsByType, but not very complete,
is this package has ExtensionsByType-like function ?
xz command.application/x-yaml or text/yamlOther repos that might have useful information in them (if possible turn these into JSON format so that they can be ported into Python and JS)
I noticed all of the matchers had their file signatures defined inline and was curious if hoisting them to global scope would make any difference. Turns out for huge files it makes a significant difference. Of course would have to take into consideration whether to sacrifice memory over speed since all signatures stored in global vars would be stored on the heap.
...
var dataPdf []byte
func init() {
dataPdf, _ = ioutil.ReadFile("../testdata/a.pdf")
}
var sigPdf = []byte{0x25, 0x50, 0x44, 0x46}
func PdfAlt(in []byte) bool {
return bytes.Equal(in[:4], sigPdf)
}
func Benchmark_Pdf(b *testing.B) {
for i := 0; i < b.N; i++ {
Pdf(dataPdf)
}
}
func Benchmark_PdfAlt(b *testing.B) {
for i := 0; i < b.N; i++ {
PdfAlt(dataPdf)
}
}goos: darwin
goarch: amd64
pkg: github.com/gabriel-vasile/mimetype/matchers
Benchmark_Pdf-8 200000000 6.27 ns/op
Benchmark_PdfAlt-8 500000000 3.56 ns/op
PASS
ok github.com/gabriel-vasile/mimetype/matchers 4.075s
...
var dataPng []byte
func init() {
dataPng, _ = ioutil.ReadFile("../testdata/a.png")
}
var sigPng = []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A}
func PngAlt(in []byte) bool {
return bytes.Equal(in[:8], sigPng)
}
func Benchmark_Png(b *testing.B) {
for i := 0; i < b.N; i++ {
Png(dataPng)
}
}
func Benchmark_PngAlt(b *testing.B) {
for i := 0; i < b.N; i++ {
PngAlt(dataPng)
}
}goos: darwin
goarch: amd64
pkg: github.com/gabriel-vasile/mimetype/matchers
Benchmark_Png-8 500000000 3.13 ns/op
Benchmark_PngAlt-8 500000000 3.11 ns/op
PASS
ok github.com/gabriel-vasile/mimetype/matchers 3.767s
I have a (generated) whitelist of allowed MIME types and want to check whether an incoming MIME type is allowed, but apparently it's not possible to create a list of MIME types with some kind of factory like mimetype.Of("image/jpeg"). There's only an indirect way to access an existing, well-known MIME type via mimetype.DetectFile(), but it seems a bit odd to create pseudo files just to get the proper MIME type, and in my case its not even feasible.
Some support to access known MIME types (in tree.go) would be great!
it will return application/octet-stream as samilar to standard libararay, if I use "github.com/h2non/filetype" check the movie , it will give me correct result.
func Mp4(buf []byte) bool {
return len(buf) > 11 &&
(buf[4] == 'f' && buf[5] == 't' && buf[6] == 'y' && buf[7] == 'p') &&
((buf[8] == 'a' && buf[9] == 'v' && buf[10] == 'c' && buf[11] == '1') ||
(buf[8] == 'd' && buf[9] == 'a' && buf[10] == 's' && buf[11] == 'h') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == '2') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == '3') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == '4') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == '5') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == '6') ||
(buf[8] == 'i' && buf[9] == 's' && buf[10] == 'o' && buf[11] == 'm') ||
(buf[8] == 'm' && buf[9] == 'm' && buf[10] == 'p' && buf[11] == '4') ||
(buf[8] == 'm' && buf[9] == 'p' && buf[10] == '4' && buf[11] == '1') ||
(buf[8] == 'm' && buf[9] == 'p' && buf[10] == '4' && buf[11] == '2') ||
(buf[8] == 'm' && buf[9] == 'p' && buf[10] == '4' && buf[11] == 'v') ||
(buf[8] == 'm' && buf[9] == 'p' && buf[10] == '7' && buf[11] == '1') ||
(buf[8] == 'M' && buf[9] == 'S' && buf[10] == 'N' && buf[11] == 'V') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'A' && buf[11] == 'S') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'S' && buf[11] == 'C') ||
(buf[8] == 'N' && buf[9] == 'S' && buf[10] == 'D' && buf[11] == 'C') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'S' && buf[11] == 'H') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'S' && buf[11] == 'M') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'S' && buf[11] == 'P') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'S' && buf[11] == 'S') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'X' && buf[11] == 'C') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'X' && buf[11] == 'H') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'X' && buf[11] == 'M') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'X' && buf[11] == 'P') ||
(buf[8] == 'N' && buf[9] == 'D' && buf[10] == 'X' && buf[11] == 'S') ||
(buf[8] == 'F' && buf[9] == '4' && buf[10] == 'V' && buf[11] == ' ') ||
(buf[8] == 'F' && buf[9] == '4' && buf[10] == 'P' && buf[11] == ' '))
}// Mp4 matches an MP4 file.
func Mp4(in []byte) bool {
if len(in) < 12 {
return false
}
mp4ftype := []byte("ftyp")
mp4 := []byte("mp4")
boxSize := int(binary.BigEndian.Uint32(in[:4]))
if boxSize%4 != 0 || len(in) < boxSize {
return false
}
if !bytes.Equal(in[4:8], mp4ftype) {
return false
}
for st := 8; st < boxSize; st += 4 {
if st == 12 {
// minor version number
continue
}
if bytes.Equal(in[st:st+3], mp4) {
return true
}
}
return false
}I found the two core code, Can I Use the way to fix？
According to this answer in SO: https://stackoverflow.com/questions/29838185/how-to-detect-additional-mime-type-in-golang
it says that github.com/gabriel-vasile/mimetype is extendable.
However, as far as I can see it is not extendable (other than modifying the source code).
Is this a planned feature?
I have some local file formats, that I want to detect, that is not suitable for inclusion in the library.
Specify the MIME type and extension for which to add support
application/vnd.ms-publisher, .pub
Share an example file
samples can be found online
Optionally, add a reference to the specification of the file format.
magic
Instead of using regex to find the 2 files the regex searches for, use the regular zip header parsing.
https://github.com/gabriel-vasile/mimetype/blob/pprof/internal/matchers/ms_office.go#L13
With regex, this function is the slowest when it is fed inputs which will fail to be detected as office files.
go test -bench=FailingXlsx -cpuprofile pprof.out
go tool pprof pprof.outSpecify the MIME type and extension for which to add support
text/x-gimp-ggr, .ggr
text/x-gimp-gpl .gpl
image/x-xcf .xcf
image/x-gimp-pat .pat
image/x-gimp-gbr .gbr
image/x-gimp-gih .gih
Share an example file
Can be generated with GIMP.
Optionally, add a reference to the specification of the file format.
https://github.com/file/file/blob/master/magic/Magdir/gimp
Is it possible to wrap this library in Python for ease of usage?
From specification :
date,temperature,place
2011-01-01,1,Galway
2011-01-02,-1,Galway
2011-01-03,0,Galway
2011-01-01,6,Berkeley
2011-01-02,8,Berkeley
2011-01-03,5,Berkeley
TIA ;)
svg files should be image/svg+xml, but is just text/xml now.
I noticed that DetectReader is calling ReadAll -- could get expensive -- per the filetype package, https://github.com/h2non/filetype, shouldn't you be able to just do something like this:
// We only have to pass the file header = first 261 bytes
head := make([]byte, 261)
file.Read(head)
Right now the order of the matching funcs is not optimal. More common formats should be checked before the less used ones.
{"some":"thing"}
{"foo":17,"bar":false,"quux":true}
{"may":{"include":"nested","objects":["and","arrays"]}}
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.