Comments (2)
You're right - the existing implementation does do something that I explicitly warn against. Note that hypothetically speaking, including cache-busting headers in the responses can help.
The main reason that I don't do it is that it's quite prescriptive to decide to use HTTP1.0 and come up with an appropriate scheme for chopping up the payload. The ideal situation would often be to match the granularity to your units of change (as described a bit lower down).
However, at least theoretically, it should not be too difficult to provide a different implementation of HttpRequester that could source block ranges from potentially multiple files, and fallback to downloading and caching an entire file if a ranged request failed. The work that would be required would largely be to decide how to chop up the input, lay that out as URIs and if that should be handled as a dynamic concern by a webserver or a static one (say, S3).
In order to achieve this, it would probably be best to embed the Verifier into the requester so that it can handle and respond to more cases internally (See BlockSourceBase).
The intent is that the use of Go's interfaces allows delegation of responsibility - the HttpRequester and BlockSourceBase should be almost entirely pluggable, and how you would get the information about the file splits could be an external concern.
from go-sync.
I appreciate your input. I've been digging and starting to draw diagrams to picture how the code is designed. It seems, though, writing test cases against possible failures, especially related to http connections, could render a feasible leverage to catch up the design in time and build the HTTP 1.0 fallback measure.
Coincidentally, you've stated that Network Error handling
is one of the improvements you've sought for, and it seems to be the top priority to production readiness mentioned in #15.
I have had very pleasant experience from network simulator such as Network Link Conditioner, albeit only on OSX. I don't believe the very utility could be deployed on such test cases, but the notion of having artificial conditioner to stimulate network issues should stand valid for some degree.
In search of similar tools, I've encountered tylertreat/comcast and Shopify/toxiproxy.
Both projects are highly oriented toward CLI usage, but I think it should be possible to incorporate one of them into tests within affordable time window. I'll start building test cases, and make PRs. I hope you can save some time to review additional tests, and guide me to cover corners properly.
from go-sync.
Related Issues (15)
- failed to understand http_test.go HOT 1
- Patch <reference index> requires local file HOT 4
- Detailed comparison to rsync/librsync/zsync HOT 1
- Efficient sparse file handling HOT 6
- Assign severities to TODO items HOT 1
- Support exclude patterns
- How do you host a tls webserver hosting go-sync diff files? HOT 1
- Trailing zeroes in patched file HOT 1
- json files
- Can you sync a directory? HOT 15
- Possible to stream files through an xz algorithim? HOT 6
- Clarification on example HOT 4
- what is the file format of the .gosync file ? HOT 2
- example for using as a library HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-sync.