solus-project / ferryd Goto Github PK
View Code? Open in Web Editor NEWFast, safe and reliable transit for the delivery of software updates to users.
Home Page: https://solus-project.com/
License: Apache License 2.0
Fast, safe and reliable transit for the delivery of software updates to users.
Home Page: https://solus-project.com/
License: Apache License 2.0
it's returning an error, Cannot proxy the given URL
.
One of the last standing issues with getting packages to users is now security updates outside of the standard repo syncs, or quick fixes to problems not spotted prior to sync e.g. https://dev.solus-project.com/T4575
We should have a simple system in place that returns a connection handle for a given duration (i.e. batched functions) and is then later closed. When the last connection is closed, set the connection count to 0, and after some timeout, close the underlying database connection again. This will help to ensure that leveldb memory is reclaimed and ensure recovery, etc, all work properly.
Without this, our memory usage is going to keep growing unbounded.
Due to the sync/async/sync -> sync/async changes we have a lot of specialist code within the delta job.
This means we're always creating new deltas - we first need to check if the delta ID exists within the pool, and ref that. Otherwise we can just create and then ref.
The delta
repo job is fairly slow as it has to check every package in the repo, and is only really intended to be used after initial import. During a pull
operation, track the names in the repo that are being modified, and then spawn a delta job for each one of them on the async queue.
This will greatly reduce the time it takes to delta the repo after sync windows.
tldr boltdb is really, really bad for concurrency.
Establish a simpler model that gets rid of all the bucket-orientated logic, and rely on many to many relationships:
struct RepoEntry {
Repository Repository
Published Package
Packages []Package ..
Deltas []Package
}
Basically rip out all the current boltdb crap, start a new subproject to test the data storage.
Long story short I'm pulling me hair out with how long each repo settle is taking. This needs implementing immediately ..
In order of Things To Make It All Work:
.eopkg
add support (cache into pool
and repo
storage)eopkg-index.xml
write support (stable sort + constant time emission)components.xml
distribution.xml
groups.xml
merge support.xz
.sha1sum
)If we do a pull that results in 320 jobs, and average each sync job as ~11s, this is an hour added for no reason. Sure, it might not always index because deltas existing or not needed, but still, optimise it.
Just swap it out for a DeltaJob and then we can manually index it once status shows it done.
Marking it here so i don't actually forget..
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.