transloadit/go-sdk

Transloadit's official Go SDK, maintained by the community

go-sdk's Issues

`AssemblyInfo.bytes_expected` can be null

Hey!

WaitAssembly fails for me with the following error: failed unmarshal http request: json: cannot unmarshal string into Go struct field AssemblyInfo.bytes_expected of type int

The corresponding assembly response is below. You can clearly see that bytes_expected is null.

{
    "ok": "ASSEMBLY_COMPLETED",
    "http_code": 200,
    "message": "The Assembly was successfully completed.",
    "assembly_id": "10182eee988945da98a455cc1e3d1319",
    "parent_id": null,
    "account_id": "***REDACTED***",
    "template_id": "229a0e60629241e3848af05038322ee6",
    "instance": "rang.transloadit.com",
    "assembly_url": "http://api2.rang.transloadit.com/assemblies/10182eee988945da98a455cc1e3d1319",
    "assembly_ssl_url": "https://api2-rang.transloadit.com/assemblies/10182eee988945da98a455cc1e3d1319",
    "uppyserver_url": "https://api2-rang.transloadit.com/companion/",
    "companion_url": "https://api2-rang.transloadit.com/companion/",
    "websocket_url": "https://api2-rang.transloadit.com/ws20034",
    "tus_url": "https://api2-rang.transloadit.com/resumable/files/",
    "bytes_received": 118604,
    "bytes_expected": null,
    "upload_duration": 0.886,
    "client_agent": null,
    "client_ip": null,
    "client_referer": null,
    "transloadit_client": "go-sdk:v1.1.1",
    "start_date": "2019/11/15 13:51:53 GMT",
    "upload_meta_data_extracted": true,
    "warnings": [],
    "is_infinite": false,
    "has_dupe_jobs": false,
    "execution_start": "2019/11/15 13:51:54 GMT",
    "execution_duration": 1.727,
    "queue_duration": 0.317,
    "jobs_queue_duration": 0.01,
    "notify_start": null,
    "notify_url": null,
    "notify_status": null,
    "notify_response_code": null,
    "notify_duration": null,
    "last_job_completed": "2019/11/15 13:51:56 GMT",
    "fields": {},
    "running_jobs": [],
    "bytes_usage": 763547,
    "executing_jobs": [],
    "started_jobs": [
        ":original:::original",
        "original:::original",
        "resize_full1080::original",
        "resize_thumb360::original",
        "export::original",
        "thumb360::resize_thumb360",
        "full1080::resize_full1080",
        "export::thumb360",
        "export::full1080"
    ],
    "parent_assembly_status": null,
    "params": "{\"auth\":{\"key\":\"****\",\"expires\":\"2019/11/15 14:51:53+00:00\"},\"template_id\":\"229a0e60629241e3848af05038322ee6\"}",
    "template": "{\"steps\":{\":original\":{\"robot\":\"/upload/handle\"},\"original\":{\"use\":\":original\",\"robot\":\"/file/filter\",\"accepts\":[[\"${file.mime}\",\"regex\",\"image\"]],\"error_on_decline\":true},\"resize_full1080\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":1080,\"height\":1080,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"resize_thumb360\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":360,\"height\":360,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"full1080\":{\"robot\":\"/image/optimize\",\"use\":\"resize_full1080\",\"priority\":\"compression-ratio\"},\"thumb360\":{\"robot\":\"/image/optimize\",\"use\":\"resize_thumb360\",\"priority\":\"compression-ratio\"},\"export\":{\"use\":[\"original\",\"full1080\",\"thumb360\"],\"robot\":\"/s3/store\",\"credentials\":\"s3_user_content\",\"path\":\"${previous_step.name}/${file.original_id}_${previous_step.name}.${file.ext}\",\"acl\":\"private\"}}}",
    "merged_params": "{\"steps\":{\":original\":{\"robot\":\"/upload/handle\"},\"original\":{\"use\":\":original\",\"robot\":\"/file/filter\",\"accepts\":[[\"${file.mime}\",\"regex\",\"image\"]],\"error_on_decline\":true},\"resize_full1080\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":1080,\"height\":1080,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"resize_thumb360\":{\"use\":\"original\",\"robot\":\"/image/resize\",\"resize_strategy\":\"min_fit\",\"format\":\"jpg\",\"quality\":90,\"width\":360,\"height\":360,\"strip\":true,\"zoom\":true,\"background\":\"#ffffff\",\"rotation\":true,\"imagemagick_stack\":\"v2.0.7\"},\"full1080\":{\"robot\":\"/image/optimize\",\"use\":\"resize_full1080\",\"priority\":\"compression-ratio\"},\"thumb360\":{\"robot\":\"/image/optimize\",\"use\":\"resize_thumb360\",\"priority\":\"compression-ratio\"},\"export\":{\"use\":[\"original\",\"full1080\",\"thumb360\"],\"robot\":\"/s3/store\",\"credentials\":\"s3_user_content\",\"path\":\"${previous_step.name}/${file.original_id}_${previous_step.name}.${file.ext}\",\"acl\":\"private\"}},\"allow_steps_override\":true,\"auth\":{\"key\":\"****\",\"expires\":\"2019/11/15 14:51:53+00:00\"},\"template_id\":\"229a0e60629241e3848af05038322ee6\"}",
    "uploads": [
        {
            "id": "dece9a004dda4a3ebbeb65eada3ddadf",
            "name": "xxx.jpg",
            "basename": "xxx",
            "ext": "jpg",
            "size": 117953,
            "mime": "image/jpeg",
            "type": "image",
            "field": "asset",
            "md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
            "original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
            "original_basename": "xxx",
            "original_name": "xxx.jpg",
            "original_path": "/",
            "original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
            "from_batch_import": false,
            "is_tus_file": false,
            "tus_upload_url": null,
            "url": "http://tmp-eu-west-1.transloadit.com.s3-eu-west-1.amazonaws.com/dece9a004dda4a3ebbeb65eada3ddadf.jpg",
            "ssl_url": "https://s3-eu-west-1.amazonaws.com/tmp-eu-west-1.transloadit.com/dece9a004dda4a3ebbeb65eada3ddadf.jpg",
            "meta": {
                "width": 341,
                "height": 341,
                "date_recorded": null,
                "date_file_created": null,
                "date_file_modified": "2019/11/15 13:51:54 GMT",
                "title": null,
                "description": null,
                "duration": null,
                "location": null,
                "aspect_ratio": "1.000",
                "city": null,
                "state": null,
                "country": null,
                "country_code": null,
                "keywords": null,
                "aperture": null,
                "exposure_compensation": null,
                "exposure_mode": null,
                "exposure_time": null,
                "flash": null,
                "focal_length": null,
                "f_number": null,
                "iso": null,
                "light_value": null,
                "metering_mode": null,
                "shutter_speed": null,
                "white_balance": null,
                "device_name": null,
                "device_vendor": null,
                "device_software": "Picasa",
                "latitude": null,
                "longitude": null,
                "orientation": "Horizontal (normal)",
                "has_clipping_path": false,
                "creator": null,
                "author": null,
                "copyright": null,
                "copyright_notice": null,
                "frame_count": 1,
                "colorspace": "RGB",
                "has_transparency": false,
                "dominant_colors": null,
                "average_color": "#886a54",
                "xp_title": null,
                "xp_comment": null,
                "xp_keywords": null,
                "xp_subject": null
            }
        }
    ],
    "results": {
        "original": [
            {
                "id": "dece9a004dda4a3ebbeb65eada3ddadf",
                "name": "xxx.jpg",
                "basename": "xxx",
                "ext": "jpg",
                "size": 117953,
                "mime": "image/jpeg",
                "type": "image",
                "field": "asset",
                "md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
                "original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
                "original_basename": "xxx",
                "original_name": "xxx.jpg",
                "original_path": "/",
                "original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
                "from_batch_import": false,
                "is_tus_file": false,
                "tus_upload_url": null,
                "url": "***REDACTED***",
                "ssl_url": "***REDACTED***",
                "meta": {
                    "width": 341,
                    "height": 341,
                    "date_file_modified": "2019/11/15 13:51:54 GMT",
                    "aspect_ratio": "1.000",
                    "device_software": "Picasa",
                    "orientation": "Horizontal (normal)",
                    "has_clipping_path": false,
                    "frame_count": 1,
                    "colorspace": "RGB",
                    "has_transparency": false,
                    "average_color": "#886a54"
                },
                "cost": 23591,
                "queue": "live",
                "queueTime": 0,
                "execTime": 0.58
            }
        ],
        "thumb360": [
            {
                "id": "17c477802c884ea39b768ce873443125",
                "name": "xxx.jpg",
                "basename": "xxx",
                "ext": "jpg",
                "size": 17366,
                "mime": "image/jpeg",
                "type": "image",
                "field": "asset",
                "md5hash": "04185b2e84f28e56c30318a6143dc2e7",
                "original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
                "original_basename": "xxx",
                "original_name": "xxx.jpg",
                "original_path": "/",
                "original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
                "from_batch_import": false,
                "is_tus_file": false,
                "tus_upload_url": null,
                "url": "http***REDACTED***",
                "ssl_url": "***REDACTED***",
                "meta": {
                    "width": 360,
                    "height": 360,
                    "date_file_modified": "2019/11/15 13:51:54 GMT",
                    "aspect_ratio": "1.000",
                    "has_clipping_path": false,
                    "frame_count": 1,
                    "colorspace": "sRGB",
                    "has_transparency": false,
                    "average_color": "#886a54"
                },
                "cost": 55424,
                "queue": "live",
                "queueTime": 0,
                "execTime": 0.43
            }
        ],
        "full1080": [
            {
                "id": "165b7965da2c4751a38d5a1c38b21b82",
                "name": "xxx.jpg",
                "basename": "xxx",
                "ext": "jpg",
                "size": 77887,
                "mime": "image/jpeg",
                "type": "image",
                "field": "asset",
                "md5hash": "f754a69ad50523e674a484a8c0bdcfee",
                "original_id": "dece9a004dda4a3ebbeb65eada3ddadf",
                "original_basename": "xxx",
                "original_name": "xxx.jpg",
                "original_path": "/",
                "original_md5hash": "fdc26c4c343d129b8568f8f1c45a16d2",
                "from_batch_import": false,
                "is_tus_file": false,
                "tus_upload_url": null,
                "url": "***REDACTED***",
                "ssl_url": "***REDACTED***",
                "meta": {
                    "width": 1080,
                    "height": 1080,
                    "date_file_modified": "2019/11/15 13:51:55 GMT",
                    "aspect_ratio": "1.000",
                    "has_clipping_path": false,
                    "frame_count": 1,
                    "colorspace": "sRGB",
                    "has_transparency": false,
                    "average_color": "#886954"
                },
                "cost": 250125,
                "queue": "live",
                "queueTime": 0.01,
                "execTime": 0.55
            }
        ]
    },
    "build_id": "300"
}

The code producing the error is really nothing special:

	assembly := transloadit.NewAssembly()
	assembly.TemplateID = "229a0e60629241e3848af05038322ee6"

	data, err := ioutil.ReadAll(body)
	if err != nil {
		return nil, err
	}

	err = ioutil.WriteFile("/tmp/upload.jpg", data, 0644)
	if err != nil {
		panic(err)
	}
	log.Printf("assets: upload saved as /tmp/upload.jpg")

	assembly.AddReader("asset", "xxx.jpg", ioutil.NopCloser(bytes.NewReader(data)))
	log.Printf("assets: uploading...")
	info, err := as.trans.StartAssembly(ctx, assembly)
	if err != nil {
		log.Printf("WARNING: assets: uploading failed: %+v", err)
		return nil, err
	}

	log.Printf("assets: uploading complete")
	spew.Dump(info)

	log.Printf("assets: checking %v", info.AssemblySSLURL)

	info, err = as.trans.WaitForAssembly(ctx, info)
	if err != nil {
		log.Printf("WARNING: assets: processing failed: %+v", err)
		return nil, err
	}

	log.Printf("assets: processing complete")
	spew.Dump(info)

and it fails at WaitForAssembly call.

Note: I think I've redacted all private data. The image itself isn't private, just my userpic.

Improve documentation in the Readme

I think the readme lacks quite a bit of documentation for all the available public methods. An example would be https://github.com/transloadit/node-sdk

It would be nice if the Readme also included where a template is used.

Bonus: coveralls.io integration

Cleanup templates after tests

Remove the test template we create, so the test account does not end up with unnecessarily many templates.

Parameters don't understand `~`

E.g.

kevin at kvz-mbp in ~
$ ./transloadify \
→   -template-file=./hls.json \
→   -input=~/Downloads \
→   -watch=true \
→   -preserve=true \
→   -output=~/Encoded
2014/09/25 12:53:35 Converting all files in '~/Downloads' and putting the result into '~/Encoded'.
2014/09/25 12:53:35 Using template file './hls.json' (read 16 steps).
2014/09/25 12:53:35 Watching directory '~/Downloads' for changes...
fatal error: all goroutines are asleep - deadlock!

goroutine 16 [chan send]:
github.com/transloadit/go-sdk.(*Watcher).processDir(0x2084f8640)
    /home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:87 +0xdc
github.com/transloadit/go-sdk.(*Watcher).start(0x2084f8640)
    /home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:60 +0x27
github.com/transloadit/go-sdk.(*Client).Watch(0x208554b40, 0x2084e4300, 0x0)
    /home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/watch.go:52 +0x13b
main.main()
    /home/gobuild/goproj/src/github.com/codeskyblue/gobuild/gopath/src/github.com/transloadit/go-sdk/transloadify/transloadify.go:94 +0x75d

goroutine 19 [finalizer wait]:
runtime.park(0x15950, 0x476da8, 0x475889)
    /home/gobuild/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x476da8, 0x475889)
    /home/gobuild/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /home/gobuild/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /home/gobuild/go/src/pkg/runtime/proc.c:1445

Bonus: Docs up on godoc.org

Bonus: Build (binaries of the) cli tool

While discussing how to implement #5 to support the studio/producer use-case it slowly became clear this should be a separate small cli utility, albeit supported by the lib for most part.

TRANSLOADIT_ACCOUNT=abc TRANSLOADIT_SECRET=def ./transloadify -template_id=ghi -dir_in=./incoming -dir_out=./var/www/carshow.tv/results

I would like it if

• we can build this utility via a simple make build

I would love it if

• this would produce binaries for all main platforms
• runs on Travis automatically

I would lose my mind if

• In case of master, these binaries are versioned & stored on s3

`-watch` should remove incoming files upon successful conversion

To avoid

$ ./transloadify \
→   -key=xxxxxxxx \
→   -secret=xxxxxxxx \
→   -template=xxxxxxxx \
→   -input=./incoming \
→   -watch=true \
→   -output=./converted
2014/09/17 12:44:25 Converting all files in './incoming' using template 'xxxxxxxx' and putting the result into './converted'.
2014/09/17 12:44:25 Watching directory './incoming' for changes...
2014/09/17 12:47:04 Successfully converted 'clip-2012-12-29 17;39;19.mov'.
2014/09/17 12:48:05 Successfully converted 'clip-2012-12-29 17;39;19.mov'.

failed execute http request:

Auth key & secret should be obtained from ENV variables

Currently they can be specified as cli args, but that is less than ideal as you can see the credentials in e.g. ps auxf

$ ./transloadify -h
Usage of ./transloadify:
  -input=".": Input directory
  -key="": Auth key
  -output="": Output directory
  -secret="": Auth secret
  -template="": Template's id to create assemblies with
  -watch=false: Watch input directory for changes

Make release-(major|minor|patch)

For node modules, we have

release-major: build test
    npm version major -m "Release %s"
    git push
    npm publish

I would like something similar for this Go project. So e.g. make release-minor:

• Aborts unless working tree is clean
• Bumps appropriate semver part
• Saves version to file
• Commits the file with msg "Release v"
• Creates a Git tag with this version
• Runs gobuild on this tag for most platforms
• Saves them to S3 as transloadify-<paltform>-<arch>-<version>
• Pushes commit & tag to GitHub

Have transloadify support local templates

I think developers would love to point transloadify to a ./some/path/in/git/encode-to-ipad.json with instructions.

This way they can directly hack on templates (or just overriding parameters) without going back & forth to the crm.

Here's what I'm envisioning:

./transloadity -template=<id> # just a stored template
./transloadity -template=<id> -parameters=./params.json # a stored template, with tweaked parameters found in the json
./transloadity -parameters=./params.json # everything is coming out of the json, no stored template was used

Up for discussion

User agent

The user agent of transloadify should be "Transloadit Go SDK {{version}}".

Bonus: More resilient Bored Instance logic (with cached fallback)

For Bonus points: You'll see in the node-sdk that the getBored function can fall back on an infra- file on S3 that contains all uploaders and picks a random one: https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.coffee#L266. This is to shield the client from the unlikely event that the machine you got routed to (via dns, so it's sticky for a while), is unable to provide a bored instance due to high load or defects.

As discussed here: 2c5b62c#commitcomment-7580913

`-watch` should announce it found a file before, not after conversion

So a log entry, the moment it discovers a new file. We can keep

2014/09/17 12:47:04 Successfully converted 'clip-2012-12-29 17;39;19.mov'.

as well.

Complete docs

As you mentioned, @Acconut, the godocs aren't fully finished yet. We would like that ofc before we release.

Bonus: Optionally Block until done encoding

Polling assembly status so you can block locally, until we have finished encoding. Useful for piping/chaining CLI tools.

We have a blocking: true option, so that we keep the connection with the client open until all encoding/processing is done. But it's undocumented and we'll probably phase it out. A better way would be for the client to periodically check the assembly status, and release control once it sees ASSEMBLY_COMPLETED, or an error.

This allows Tranloadit server processes to crash, or the network path to fail, making it less prone to breakage than keeping 1 connection open.

Utilize batch queue

Discussed this with @tim-kos, and the Go SDK should add "queue": "batch", to every assembly.
That way the right queue can be selected for all steps. Wether coming from disk, or template_id.
It does not make sense if steps can have different queues. That will just make everything as slow as the weakest link

Felix' feedback

Felix' Feedback

1. Don't pass options as a reference or someone might change it
2. As a static binary, ./VERSION won't be there. Use constants in a .go instead
3. In doRequest: call close() on res
4. WaitForAssembly can have a timeout, then you can remove stop() on the AssemblyWatcher (?)
5. Read go memory model: There is no guarantee that watchers.stopped will ever be observed by the go routine. Might as well be dead code as tyhere is no synchronization -- always use a log or a channel to communicate
   • This is behavior replicated in a few other places too
6. Document if AssemblyUpload err != null, you can also check info
7. When assembly is not blocking, err isn't checked
8. LimitReader vs ReadAll would protect against giant buffers. E.g. 128mb goes a long way in protecting a server and still handle crazy assemblies

Bonus: Scan local folder for incoming files

Scanning a local directory for new/incoming files, that will be automatically uploaded to Transloadit, converted by us using a template_id, and the results then saved to another local directory. The usecase here being people running their own ftp/upload system, and interested in converting every video to HLS for instance.

Fix tests

A readme upgrade somehow broke the tests. There must be a bug/race lurking somewhere. This should be addressed before we can confidently release this project