Comments (11)
I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.
I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.Thanks for writing this up and share your valuable experience!
I am evaluating Metaflow for my team but am not very familiar with Terraform, do you know how much work it make take to create the "professional".tf?
It probably depends on how much, and what kind of, existing infrastructure is assumed. The complexity of Terraform starts to increase significantly when it is expected to handle many different situations.
For example, the official CloudFormation template for Metaflow is opinionated and creates a VPC, an ECS task definition/service/managed cluster, API Gateway, a network load balancer, and at least one Lambda function. For my professional environments, the API gateway wouldn't be need, since there is no need to expose the service publicly and there is a VPN connection to the VPC. We also use Kubernetes with existing ingress setups, and would probably just deploy the service with that infrastructure.
The official CloudFormation stack seems like a perfectly reasonable setup to me, and probably meets many use cases. If that is a good assumption, then porting it to Terraform isn't too challenging. A slight increase in complexity would give you the ability to selectively disable certain components, or allow existing infrastructure components as input. Is there interest in that?
from metaflow.
@sandman21dan it's still a work in progress (finding time after work), but I pushed the repo yesterday: https://github.com/zcking/metaflow-terraform
I'm having a 403 Unauthorized permission when trying to run the hello-aws tutorial right now. Made some tweaks to it last night that I haven't tried yet, but will pick it back up this evening.
from metaflow.
I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.
I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.
from metaflow.
I also found a tf PR for this but in a different repo: Netflix/metaflow-tools#14
from metaflow.
@zcking is working on a terraform template - https://gitter.im/metaflow_org/community?at=5dee749f8e906a1c8d9afc49
from metaflow.
@zcking any progress on this?
from metaflow.
I modified it for my own purpose. Thanks for the good work @zcking . I primarily wanted to separate networking, since ours is already created.
The final piece that seems to be missing is the load balancer or cname.
from metaflow.
I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.
I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.
Thanks for writing this up and share your valuable experience!
I am evaluating Metaflow for my team but am not very familiar with Terraform, do you know how much work it make take to create the "professional".tf?
from metaflow.
@mtx2d If you are okay with cloudformation, we have an official template for that.
from metaflow.
@mtx2d If you are okay with cloudformation, we have an official template for that.
@savingoyal, thanks for the reference to cloudformation template.
Unfortunately, the company I work for has invested in terraform heavily, so continue to use terraform is the only way forward for my current situation.
from metaflow.
Thanks to the excellent work by @kldavis4, @GregHilstonHop and @queueburt, you can now deploy all the cloud resources for Metaflow through a Terraform template 🎉
from metaflow.
Related Issues (20)
- Suggest replacing `pull_request_target` in branches as well as `main`
- Bug: Passing an `S3PutObject` to `s3.put_files` treated as `tuple` of key path values.
- Allow passing of `trusted-host` parameter to `@pypi` decorator.
- Unable to utilise pytest unit test for metaflow HOT 1
- Update extras_require for tracing dependencies?
- R tests on Github fail with macos-latest runner
- Argo-workflows with name > 52 characters fail if run as CronWorkflows HOT 1
- Introduce support for private registry for AWS Batch
- Write metadata about log scrubbing events
- Language-agnostic API and/or other language SDKs for the client API HOT 1
- Performance Degradation When Running PyCaret Model with n_jobs > 1 Inside Metaflow
- Question on Executing Metaflow Workflow from Python Script Without 'run' Argument HOT 4
- Resume Doesn't Resume from the Last Failed Step HOT 1
- Argo Events: trigger sensors are not deleted
- Allow option to set labels on pods using `kubernetes` decorator HOT 1
- Bug: Runner API doesn't pass parameters to flows as expected HOT 4
- Log failure reason in slack alerts for deployed flow failures
- Display full trigger event payload in metaflow ui HOT 1
- ArgoEvents.publish attempts to make requests even when url is None HOT 1
- @conda/@pypi appears to remove CUDA instructions for GPU HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metaflow.