transloadit/transloadify

🤖☁️📽📦 Transloadit's cloud encoding in a box

transloadify's Issues

Consider using fancy email prompt for account creation

https://github.com/zeit/email-prompt

Crash on empty input dir

When input dir is empty, and you run transloadify like so:

TRANSLOADIT_KEY=x \
TRANSLOADIT_SECRET=y \
./transloadify \
  -input="./input" \
  -output="./output" \
  -template-file="./template.json"

it crashes with:

2015/10/01 09:47:49 Converting all files in '/home/kvz/dennis/input' and putting the result into '/home/kvz/dennis/output'.
2015/10/01 09:47:49 Using template file '/home/kvz/dennis/template.json' (read 1 steps).
fatal error: all goroutines are asleep - deadlock!

goroutine 16 [select]:
main.main()
    /gopath/src/github.com/transloadit/transloadify/transloadify.go:104 +0xc57

goroutine 19 [finalizer wait]:
runtime.park(0x417e50, 0x935df0, 0x934089)
    /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x935df0, 0x934089)
    /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445

which seems a harsh way of letting the user know its dir is empty :)

Gobuild.io builds broken after move to separate repo

http://v1.gobuild.io/github.com/transloadit/go-sdk/transloadify/master/darwin/amd64

Locally it works

$ make build
go get .
go build -o bin/transloadify transloadify.go
bin/transloadify -h || true
Usage of bin/transloadify:
  -input=".": Input directory

build a single transloadify binary

We could use e.g. https://github.com/vercel/pkg to build a single binary, this way, people from outside the js ecosystem will have an easier time wielding it.

This was an idea by @ethanwillis - we thought about porting to Go. This would result in a tinier and likely faster binary, but also in a complete rewrite, and less of our team being able to help out. So perhaps building the existing codebase has more viability.

We should first do #25 though

Got an OOM when uploading 6GB file

To reproduce:

mkdir input output

cd  input
wget http://fixture.transloadit.com.s3.amazonaws.com/file/dennis-large.mp4
cd ..

curl http://releases.transloadit.com/transloadify/transloadify-linux-amd64-latest \
  -o ./transloadify && chmod 755 $_

echo '{
  "read": {
    "use": ":original",
    "robot": "/meta/read"
  }
}' > template.json

TRANSLOADIT_KEY=x \
TRANSLOADIT_SECRET=y \
./transloadify \
  -input="./input" \
  -output="./output" \
  -template-file="./template.json"

This will throw:

2015/10/01 09:48:58 Converting all files in '/home/kvz/dennis/input' and putting the result into '/home/kvz/dennis/output'.
2015/10/01 09:48:58 Using template file '/home/kvz/dennis/template.json' (read 1 steps).
fatal error: runtime: out of memory

goroutine 20 [running]:
runtime.throw(0x9306f7)
    /usr/local/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x7ff9f70bd538 sp=0x7ff9f70bd520
runtime.SysMap(0xc3097c0000, 0x101680000, 0x427f00, 0x93bf58)
    /usr/local/go/src/pkg/runtime/mem_linux.c:147 +0x93 fp=0x7ff9f70bd568 sp=0x7ff9f70bd538
runtime.MHeap_SysAlloc(0x947f40, 0x101680000)
    /usr/local/go/src/pkg/runtime/malloc.goc:616 +0x15b fp=0x7ff9f70bd5c0 sp=0x7ff9f70bd568
MHeap_Grow(0x947f40, 0x80b40)
    /usr/local/go/src/pkg/runtime/mheap.c:319 +0x5d fp=0x7ff9f70bd600 sp=0x7ff9f70bd5c0
MHeap_AllocLocked(0x947f40, 0x80b3c, 0x0)
    /usr/local/go/src/pkg/runtime/mheap.c:222 +0x379 fp=0x7ff9f70bd640 sp=0x7ff9f70bd600
runtime.MHeap_Alloc(0x947f40, 0x80b3c, 0x10100000000)
    /usr/local/go/src/pkg/runtime/mheap.c:178 +0x7b fp=0x7ff9f70bd668 sp=0x7ff9f70bd640
largealloc(0xc200000001, 0x7ff9f70bd718)
    /usr/local/go/src/pkg/runtime/malloc.goc:224 +0xa2 fp=0x7ff9f70bd6b0 sp=0x7ff9f70bd668
runtime.mallocgc(0x101678000, 0x67fd81, 0x1)
    /usr/local/go/src/pkg/runtime/malloc.goc:169 +0xb6 fp=0x7ff9f70bd718 sp=0x7ff9f70bd6b0
cnew(0x67fd80, 0x101678000, 0x7ff900000001)
    /usr/local/go/src/pkg/runtime/malloc.goc:836 +0xc1 fp=0x7ff9f70bd738 sp=0x7ff9f70bd718
runtime.cnewarray(0x67fd80, 0x101678000)
    /usr/local/go/src/pkg/runtime/malloc.goc:849 +0x3a fp=0x7ff9f70bd758 sp=0x7ff9f70bd738
makeslice1(0x671940, 0x101678000, 0x101678000, 0x7ff9f70bd7b8)
    /usr/local/go/src/pkg/runtime/slice.goc:55 +0x4d fp=0x7ff9f70bd770 sp=0x7ff9f70bd758
runtime.makeslice(0x671940, 0x101678000, 0x101678000, 0x0, 0x101678000, 0x101678000)
    /usr/local/go/src/pkg/runtime/slice.goc:36 +0xb3 fp=0x7ff9f70bd7a0 sp=0x7ff9f70bd770
bytes.makeSlice(0x101678000, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/bytes/buffer.go:191 +0x64 fp=0x7ff9f70bd7d8 sp=0x7ff9f70bd7a0
bytes.(*Buffer).grow(0xc2080285b0, 0x8000, 0xc208098000)
    /usr/local/go/src/pkg/bytes/buffer.go:99 +0x204 fp=0x7ff9f70bd878 sp=0x7ff9f70bd7d8
bytes.(*Buffer).Write(0xc2080285b0, 0xc208098000, 0x8000, 0x8000, 0xc20803a058, 0x0, 0x0)
    /usr/local/go/src/pkg/bytes/buffer.go:127 +0x56 fp=0x7ff9f70bd8e0 sp=0x7ff9f70bd878
mime/multipart.(*part).Write(0xc208042980, 0xc208098000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/local/go/src/pkg/mime/multipart/writer.go:179 +0x148 fp=0x7ff9f70bd958 sp=0x7ff9f70bd8e0
io.Copy(0x7ff9f7245da0, 0xc208042980, 0x7ff9f72443f0, 0xc20803a058, 0x80b30000, 0x0, 0x0)
    /usr/local/go/src/pkg/io/io.go:355 +0x27b fp=0x7ff9f70bda20 sp=0x7ff9f70bd958
github.com/transloadit/go-sdk.(*Assembly).makeRequest(0xc20801a780, 0x42664f, 0x0, 0x0)
    /gopath/src/github.com/transloadit/go-sdk/assembly.go:202 +0x440 fp=0x7ff9f70bdbe0 sp=0x7ff9f70bda20
github.com/transloadit/go-sdk.(*Assembly).Upload(0xc20801a780, 0xc20803a068, 0x0, 0x0)
    /gopath/src/github.com/transloadit/go-sdk/assembly.go:156 +0x4a fp=0x7ff9f70bdca8 sp=0x7ff9f70bdbe0
github.com/transloadit/go-sdk.(*Watcher).processFile(0xc20801a6e0, 0xc208024de0, 0x23)
    /gopath/src/github.com/transloadit/go-sdk/watch.go:172 +0x5c1 fp=0x7ff9f70bdf90 sp=0x7ff9f70bdca8
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445 fp=0x7ff9f70bdf98 sp=0x7ff9f70bdf90
created by github.com/transloadit/go-sdk.(*Watcher).processDir
    /gopath/src/github.com/transloadit/go-sdk/watch.go:139 +0x267

goroutine 16 [select]:
main.main()
    /gopath/src/github.com/transloadit/transloadify/transloadify.go:104 +0xc57

goroutine 19 [finalizer wait]:
runtime.park(0x417e50, 0x935df0, 0x934089)
    /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x935df0, 0x934089)
    /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 17 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445

TypeError: Cannot read property '0' of undefined

npm install -g [email protected]

with steps.json being:

{
  "webp": {
    "use": ":original",
    "robot": "/image/resize",
    "result": true,
    "imagemagick_stack": "v2.0.3",
    "format": "webp"
  }
}

and the following command:

env \
  TRANSLOADIT_KEY=xx \
  TRANSLOADIT_SECRET=yy \
transloadify \
  --steps ./steps.json < assets/images/logo_milan_white_10.png > output.webp

I'm getting:

ERROR   TypeError: Cannot read property '0' of undefined
    at callback (../src/assemblies-create.js:430:25)
    at _fulfilled (/usr/local/lib/node_modules/transloadify/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/usr/local/lib/node_modules/transloadify/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/usr/local/lib/node_modules/transloadify/node_modules/q/q.js:796:13)
    at /usr/local/lib/node_modules/transloadify/node_modules/q/q.js:604:44
    at runSingle (/usr/local/lib/node_modules/transloadify/node_modules/q/q.js:137:13)
    at flush (/usr/local/lib/node_modules/transloadify/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Allow multiple files for single assembly

As of now, I'm not sure about this but it seems passing multiple files to an assembly command creates a separate assembly for each file input passed.

I believe there are use cases where one may want to pass multiple files to a single assembly.

upgrade node-sdk

Now that there is a new version of node-sdk coming, it may be a good idea to upgrade this CLI also. I'll put this here as a todo

transloadit/node-sdk#87

Also currently used node-sdk version 2.0.0 doesn't include Transloadit-Client header.

Allow to change endpoints

During an aws us-east-1 outage it was asked how to change endpoints in this tool. I could not answer this, hence this issue.

Also noticed we are still on v2, so likely we first want to upgrade to v3, and then use the endpoint property vs region