Comments (13)
It is a thing that has been discussed but I can't say I've got any schedule for implementing it.
from plotman.
Would it be possible to run two instances of plotman archive each with its own config.yaml, and a slightly different target definition?
from plotman.
Yes, sorry I didn't think more to mention that. It detects based on the site root so if you just mount to two different directories that would cut it. I don't think we actually have a configuration path override though... Definitely a missing feature. I should also have mentioned https://github.com/rjsears/chia_plot_manager. The author uses plotman to plot and their own tooling to do "higher end" archiving (plus whatever other features it has).
from plotman.
Thanks for the info. I just need to double the plotman archive throughput at this point so will take a look at the chia plot manager later. I am able to run two instances of plotman archive now each working off a different tmp dir (dst drive), and to a different site root.
from plotman.
Yes it'll be more convenient to allow config file override.
from plotman.
Actually I think there is some issue there. When 1 instance is running rsync, the other instance doesn't start rsync. But perhaps since I changed the archive sleep time to 10s, each instance from time to time starts rsync around the same minute. I recall the archive.py code is actually checking for the transfer script name and argument list. Any suggestions?
from plotman.
plotman does check for the site_root
or URL to be in the options of existing rsync processes. Part of the suggestion was to have the different drives under a different site_root
so they wouldn't detect rsync processes from the other plotman archiving instance.
To be clear, yes, we are talking about an annoying hacky way to get to what you want (sort of). I'm not suggesting this is a good way for plotman to work.
from plotman.
That's exactly how I set this up. Two config files with different site_roots (and different buffer drive paths, and different log directories). The inconvenience is I had to copy the the desired config file to the only location plotman is looking before starting that plotman instance, but the real problem is, it seems, plotman still detects the rsync process run by the other instance, most of the time. This test seems to only fail when the other rsync is started within the same minute. Therefore, I either only see 1 rsync running (most of the time), or see 2 rsync processes that are started at the same hour and minute. Is the code checking for the command_name (rsync in both instances) AND site_root (different by 1 character in both instances)?
So with this setup and the problem, I have increased my archive throughput but by nowhere close to doubling it. The buffer drive Use% is still growing, although at a much slower rate than when only 1 instance was running.
from plotman.
plotman/src/plotman/archive.py
Lines 191 to 195 in 77f85e3
What are the actual site roots? Is one just the other one plus a character? Perhaps just share both complete config config files.
from plotman.
Yes. The second site_root is first $site_root}1. So it's getting a partial match by using startwith I suppose.
from plotman.
Is there a function that does exact match? Or I can change site_root to something like appending 2 to it I suppose.
from plotman.
Yeah, for now, making it so that neither starts with the other seems best. I'm sure the code could change as well.
from plotman.
Ok. So after naming 2 site_roots neither of which starting with the other, both plotman archive instances are firing up 1 rsync regardless of the other instance. To summarize I guess a few things are good to have:
- allow config file override
- make archive mode polling period configurable, and
- make string match exact in the test for transfer script dest
FYI I'm using plotman archive only and it's been working well.
from plotman.
Related Issues (20)
- Allow Randomization of Archive Drive Destinations HOT 3
- Configuration file does not show by default the pool_contract_address HOT 1
- Plotman v0.5 on mac osx creates new plot every 20 secs HOT 10
- how to create new plots
- phase stagger doesn't work with madmax? HOT 18
- log file stops at [P4] Writing C2 table . madmax HOT 2
- Fixed secondary temp -2 folder is inflexible tbh HOT 8
- allow passing of k setting with madmax HOT 4
- M1 Mac Mini, fresh install errors HOT 2
- finding unused dst dirs fails because of trailing slash "/" mismatch HOT 1
- v0.5.2 breaks with config file used in v0.5.1 HOT 4
- Plotman (k33) keeps making plots for a -d storage disk that's already full - it even reports it as full! HOT 1
- ParserError first time running plotman HOT 1
- Install fails when upgrading to chia 1.3, Debian 11, Python 3.9 HOT 6
- archiving issue HOT 1
- Error in resources/target_definitions.yaml - rsyncd - transfer_script HOT 6
- archiving does not pick up compressed plots HOT 3
- Allow Bitblade CUDA to create more than 1 plot per run (-n) HOT 2
- pip install does not work on Ubuntu 22.04 LTS HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from plotman.