Terraform support for infra configuration

[netflix]

Terraform support for infra configuration

[wtanner]

I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.
I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.

Thanks for writing this up and share your valuable experience!
I am evaluating Metaflow for my team but am not very familiar with Terraform, do you know how much work it make take to create the "professional".tf?

It probably depends on how much, and what kind of, existing infrastructure is assumed. The complexity of Terraform starts to increase significantly when it is expected to handle many different situations.

For example, the official CloudFormation template for Metaflow is opinionated and creates a VPC, an ECS task definition/service/managed cluster, API Gateway, a network load balancer, and at least one Lambda function. For my professional environments, the API gateway wouldn't be need, since there is no need to expose the service publicly and there is a VPN connection to the VPC. We also use Kubernetes with existing ingress setups, and would probably just deploy the service with that infrastructure.

The official CloudFormation stack seems like a perfectly reasonable setup to me, and probably meets many use cases. If that is a good assumption, then porting it to Terraform isn't too challenging. A slight increase in complexity would give you the ability to selectively disable certain components, or allow existing infrastructure components as input. Is there interest in that?

[zcking]

@sandman21dan it's still a work in progress (finding time after work), but I pushed the repo yesterday: https://github.com/zcking/metaflow-terraform

I'm having a 403 Unauthorized permission when trying to run the hello-aws tutorial right now. Made some tweaks to it last night that I haven't tried yet, but will pick it back up this evening.

[wtanner]

I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.

I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.

[mtx2d]

I also found a tf PR for this but in a different repo: Netflix/metaflow-tools#14

[savingoyal]

@zcking is working on a terraform template - https://gitter.im/metaflow_org/community?at=5dee749f8e906a1c8d9afc49

[notadamking]

@zcking any progress on this?

[dr3s]

I modified it for my own purpose. Thanks for the good work @zcking . I primarily wanted to separate networking, since ours is already created.

The final piece that seems to be missing is the load balancer or cname.

[mtx2d]

I have been evaluating Metaflow for several use cases, and I recently wrote about my personal setup, which includes a Terraform module that meets my requirements for my personal projects. In this setup, I opted to not run a Metadata service, since I don't yet have a use for it. It also is opinionated about networking, since I run this in a separate VPC and decided to bundle it all together. There are several other features that are likewise opinionated and not generalized.

I am beginning the process of evaluating Metaflow for professional infrastructure, the outcome of which I expect to be a more general Terraform module. This would include the Metadata service, and probably an optional network setup, since I suspect many users have existing AWS infrastructure.

Thanks for writing this up and share your valuable experience!
I am evaluating Metaflow for my team but am not very familiar with Terraform, do you know how much work it make take to create the "professional".tf?

[savingoyal]

@mtx2d If you are okay with cloudformation, we have an official template for that.

[mtx2d]

@mtx2d If you are okay with cloudformation, we have an official template for that.

@savingoyal, thanks for the reference to cloudformation template.
Unfortunately, the company I work for has invested in terraform heavily, so continue to use terraform is the only way forward for my current situation.

[savingoyal]

Thanks to the excellent work by @kldavis4, @GregHilstonHop and @queueburt, you can now deploy all the cloud resources for Metaflow through a Terraform template 🎉