Is it possible to pipe the downloaded file to a write stream?

[jcohenho]

Is it possible to pipe the downloaded file to a write stream?

[Kikobeats]

Thanks for this great suggestion!

I added the .raw method to make possibleto access stream objects at [email protected]:

const youtubedl = require('youtube-dl-exec')
const fs = require('fs')

const subprocess = youtubedl.raw('https://example.com', { dumpJson: true })

console.log(`Running subprocess as ${subprocess.pid}`)

subprocess.stdout.pipe(fs.createWriteStream('stdout.txt'))
subprocess.stderr.pipe(fs.createWriteStream('stderr.txt'))

setTimeout(subprocess.cancel, 30000)

[jcohenho]

@Kikobeats This is great, thank you! I wrote a script to pipe the stream object to a stream passthrough in order to directly upload to S3, but the resulting video file is corrupt. Am I misunderstanding something here? I can successfully download the Twitter link in this code example using the normal youtubedl method.

const youtubedl = require('youtube-dl-exec')
const AWS = require('aws-sdk');
const stream = require('stream');

const s3 = new AWS.S3();
const uploadStream = ({ Bucket, Key, Metadata }) => {
  const pass = new stream.PassThrough();
  return {
    writeStream: pass,
    promise: s3.upload({ Bucket, Key, Body: pass }).promise(),
  };
}

const { videoId, filename } = {
  "videoId": "732194327145906176",
  "filename": "sample_video",
}
const url = `https://twitter.com/i/status/${videoId}`
const { writeStream, promise } = uploadStream({ Bucket: process.env.S3_BUCKET, Key: `${filename}.mp4`});

youtubedl.raw(url, {
  noCallHome: true,
  noCheckCertificate: true,
  preferFreeFormats: true,
  format: 'mp4',
  youtubeSkipDashManifest: true,
  output: '%(id)s.%(ext)s'
}).stdout.pipe(writeStream);

const data = await promise;
return {
  statusCode: 200,
  body: data,
};

[Kikobeats]

The script looks nice to me but I can't say too much there since I didn't have a use case like that before.

What I can recommend you to do is get the streaming source video URL via Microlink API:

https://api.microlink.io/?url=https://twitter.com/eejohnnnn/status/732194327145906176&video

It uses youtube-dl (and this library) under the hood, then you can do a simple HTTP GET and pipe to your s3 bucket.

[jcohenho]

@Kikobeats amazing! Wasn't aware of this API, thank you!