asiniy / download Goto Github PK
View Code? Open in Web Editor NEWDownloads remote file and stores it in the filesystem
Downloads remote file and stores it in the filesystem
Is it possible to add an HTTPoison's options to the Download.from opts argument? E.g
options = [
path: "./file1",
http_opts: [
headers: [
"user-agent": "XXX",
"referer": "YYY"
]
]
]
Download.from(address, options)
The options[:http_opts]
is HTTPoison's options.
I'm seeing this error if the upstream closes the connection (or times out?)
17:11:16.935 [error] Process #PID<0.4030.2> raised an exception
** (FunctionClauseError) no function clause matching in Download.handle_async_response_chunk/2
(download) lib/download.ex:92: Download.handle_async_response_chunk(%HTTPoison.Error{id: #Reference<0.2630262677.3125280769.67801>, reason: {:closed, :timeout}}, %{controlling_pid: #PID<0.4028.2>, downloaded_content_length: 170004, file: #PID<0.4029.2>, max_file_size: 1048576000, path: "/tmp/foo"})
** (EXIT from #PID<0.91.0>) an exception was raised:
** (FunctionClauseError) no function clause matching in Download.handle_async_response_chunk/2
(download) lib/download.ex:92: Download.handle_async_response_chunk(%HTTPoison.Error{id: #Reference<0.2630262677.3125280769.67801>, reason: {:closed, :timeout}}, %{controlling_pid: #PID<0.4028.2>, downloaded_content_length: 170004, file: #PID<0.4029.2>, max_file_size: 1048576000, path: "/tmp/foo"})
Would it be possible to add a flag to follow_redirect for downloading certain AWS S3 files.
curl s3-url
does a 302 on quite a few AWS URLs, so at this time, I have to make an HTTP request and then use the location
value in the response header as the actual download URL with Download
Thanks for writing this module. I was considering writing such mechanism myself but I was delighted to find out I don't need to shave that yak :)
Currently I have file download as a fist stage in a pipe, passing the downloaded file path to the next stage. I suspect it could be a common pattern much like HTTPoison.request/5
vs HTTPoison.request!/5, so it may be worth adding to your module.
I would happily supply a PR but I'm about to leave for computer-free holidays :)
I was using this with a genserver, and it didn't handle calls well while the downloader was active so I put together my own genserver implementation of your library. We could probably add a couple calls to make the API compatible with yours, if you were interested in entertaining a PR
defmodule Local.Downloader do
use GenServer
alias HTTPoison.{AsyncHeaders, AsyncStatus, AsyncChunk, AsyncEnd}
def start_link(default) when is_map(default) do
GenServer.start_link(__MODULE__, default)
end
@impl true
def init(state) do
{:ok, state}
end
def start_download(pid), do: GenServer.cast(pid, :start_download)
@impl true
def handle_cast(:start_download, %{url: url, path: path} = state) do
with {:ok, file} <- do_start_download(url, path) do
{:noreply, Map.merge(state, %{file: file, downloaded: 0})}
end
end
@impl true
def handle_cast(:status, %{downloaded: size, content_length: total} = state) do
{:reply, size / total, state}
end
@impl true
def handle_info(%AsyncHeaders{headers: headers}, state) do
{_, content_length} = Enum.find(headers, fn({ header_name, _value }) ->
header_name == "content-length" || header_name == "Content-Length"
end)
{:noreply, Map.put(state, :content_length, content_length)}
end
def handle_info(%AsyncStatus{code: 200}, state) do
{:noreply, state}
end
def handle_info(%AsyncStatus{code: _}, state) do
finish_download({:error, :unexpected_status_code}, state)
{:noreply, state}
end
def handle_info(%AsyncChunk{chunk: data}, %{downloaded: size, file: file} = state) do
IO.binwrite(file, data)
{:noreply, Map.put(state, :downloaded, size + byte_size(data))}
end
def handle_info(%AsyncEnd{}, state) do
finish_download({:ok, state})
{:stop, :normal, state}
end
defp do_start_download(url, path) do
with {:ok, file} <- create_file(path),
{:ok, _result} <- HTTPoison.get(url, %{}, stream_to: self()) do
{:ok, file}
else
{:error, _reason} -> File.rm!(path)
end
end
defp finish_download({:error, _}, %{path: path}), do: File.rm!(path)
defp finish_download({:ok, %{notify: pid}}) do
send(pid, {:download_complete})
end
defp create_file(path) do
if File.exists?(path), do: File.rm(path)
with {:ok, file} <- File.open(path, [:write, :exclusive]) do
{:ok, file}
end
end
end
Thanks for the helpful mod!
Is it possible to have some options passed to HTTPoison, for example headers like user agent name.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.