aws-samples/aws-serverless-nextcloud

moved - https://github.com/f7o/aws-serverless-nextcloud

Nextcloud Container Deployment on AWS - Serverless

Welcome

This repository provides AWS CloudFormation templates to deploy NextCloud on AWS completely. No need to manage servers or manually react to monitoring events with adding capacity manually.

AWS Services & How they match the NextCloud design

• Elastic Container Service - Fargate
  • Running and scaling the official nextcloud docker container (Apache, PHP)
• Elastic Filesystem - NFS
  • Persist basic settings and configuration, support official nextcloud upgrade mechanism
• Amazon S3
  • Primary data storage for cloud native data handling (archiving, tiering, versioning)
• RDS Aurora Serverless - RDS (Postgres)
  • Cloud native full managed auto-scaled database system backing the nextcloud installation
• ElastiCache - Redis
  • Handles PHP Sessions to enable container cluster to scale easily without interruption for end-users
• Application Load Balancer, Route53, Amazon Certificate Manager
  • Secures the application with HTTPS, balances load, performs health checks, auto-certificate renewal

Quickstart

You can use the following link to deploy this solution directly into your AWS account. Ensure you are logged into the AWS Console before following it.

Quickstart CloudFormation Link

Deployment

This project assumes a familiarity with AWS CLI and AWS CloudFormation. Additional reference can be found at the following links: https://aws.amazon.com/cli/ https://aws.amazon.com/cloudformation/

• If you have never used AWS ECS within your AWS Account before, create the Service-Linked Role before going on. See the official AWS Documentation
• aws cloudformation package --template-file ecs-nextcloud.yml --s3-bucket <cfn-artifact-bucket-name> --output-template-file packaged.yaml
• Deploy packaged CloudFormation file (packaged.yaml) with appropriate parameters
  • For example using aws cloudformation deploy --template-file packaged.yaml --stack-name nextcloud-test-env --parameter-overrides DbPassword=<DB-Admin-Password> NextCloudAdminPassword=<Nextcloud-Admin-Password> IsolationLevel=Public --capabilities CAPABILITY_IAM
• The VPC setup has three levels of isolation
  • Public places containers within the public subnets
  • Private places containers into private subnets, deploys one NAT Gateway for outbound internet access
  • PrivateHA same as privat, but deploys two NAT Gateways
• After the CloudFormation Stack is deployed successfully it takes up to 15min for the frontend being available initially
• Find the Nextcloud URLs within the CloudFormation Outputs

Architecture

Sizing

The recommendation is to use the at least the default values to get decent performance (cpu: 1024, mem: 2048). A desired container capacity of 2 allows scaling and re-deployment without downtime. For initialization go with a single container to avoid clashes when copying files during the setup phase.

The baseline cost drivers are the AWS RDS database, ElastiCache Redis and Fargate task costs. Find the hourly charged costs for your AWS region within the AWS pricing pages:

• Check per ACU (Auora Capacity Unit) costs https://aws.amazon.com/rds/aurora/pricing/
• Fargate Pricing https://aws.amazon.com/fargate/pricing/
• ElastiCache Redis Pricing https://aws.amazon.com/elasticache/pricing/

How to upgrade Nextcloud to newer version

1. Create backups of RDS and EFS
2. Suspend AutoScaling using the CloudFormation parameter
3. Scale in to 1 task (set desired ECS capacity to 1)
4. Update CFN stack with new version number
5. Wait for Nextcloud to become available
6. Verify the upgrade was successful
7. Scale out service to desired size and disable AutoScaling suspension

Future Work

• Enhanced monitoring for fine granular auto-scaling (ECS)
• Enable WebCron
• Whenever Nextcloud supports it
  • Use short-term credentials instead of IAM User for S3 access
  • Redis Cluster Support

Monitoring

Sample CloudWatch Dashboard pre-configured with basic metrics will be deployed within CloudFormation

Note:

• While code samples in this repository has been tested and believe it works well, as always, be sure to test it in your environment before using it in production!
• It is highly recommended to change the administrator password after initial deployment
• While the Public VPC isolation level is the most cost efficient one, the suggestion is, for production workloads keep instances and containers in private subnets. This lowers risk, e.g. opening up ports with wrongly configured inbound port on VPC security groups.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

This deployment references the official Nextcloud Docker image which is published under AGPL-3.0 License.

aws-serverless-nextcloud's Issues

Instructions for importing into a fresh AWS account

Saw the entire account loses access and the user needs to restart the nextcloud from a fresh instance. How would one do that? I see the data is split between s3/efs and postgresql. Perhaps a short section on backing this whole thing up and restoring it in the readme would be helpful?

Is the quick link still recent?

Hi, I see that the CloudFormation quicklink is trying to get a 21.01 next cloud version setup in CF? Is this quicklink still valid/maintained?

[BUG] ELB throws 500 bad GW

Tested this with defaults apart from image (I used 23.0.1) and while everything provisions correct, NC never passed the health check and is in a 500 loop.
I unfortunately don't have time to investigate any further today but can follow up as needed over the weekend. I just wanted to flag this to see if anyone else is in the same boat?

I have not tried this with the default image from the template as I want to use the latest available

Suddenly 504 Gateway Time-out after half year without any problems

We have a setup with about 30 users, calendar integration and a few gigabytes data running for about half an year now without any problems, but suddenly the container is very slow and in 95% of times responds with a 504 Gateway Time-Out.

It seems that it for some reason is totally busy with something like when there would be a ddos attack (what is not, I blocked all requests in the LB to verify). Sometimes it even restarts the ECS task because the health checks also run into timeouts.

The only thing noticeable in the logs are a lot of entries like this:

{"reqId":"R88qYRLYFO3NKB6IneRT","level":3,"time":"2022-03-04T15:53:23+00:00","remoteAddr":"10.192.10.76","user":"xxxx","app":"PHP","method":"PUT","url":"/remote.php/dav/files/xxxx/abcd.csv","message":{"Exception":"Error","Message":"fclose(): supplied resource is not a valid stream resource at /var/www/html/3rdparty/icewind/streams/src/Wrapper.php#96","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::","args":[2,"fclose(): supplied resource is not a valid stream resource","/var/www/html/3rdparty/icewind/streams/src/Wrapper.php",96,[]]},{"file":"/var/www/html/3rdparty/icewind/streams/src/Wrapper.php","line":96,"function":"fclose","args":[null]},{"file":"/var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php","line":117,"function":"stream_close","class":"Icewind\\Streams\\Wrapper","type":"->","args":[]},{"function":"stream_close","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php","line":108,"function":"fclose","args":[null]},{"file":"/var/www/html/3rdparty/guzzlehttp/psr7/src/Stream.php","line":74,"function":"close","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[]},{"function":"__destruct","class":"GuzzleHttp\\Psr7\\Stream","type":"->","args":[]}],"File":"/var/www/html/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Macintosh) mirall/3.3.2git (build 7106) (Nextcloud, osx-21.3.0 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"21.0.1.1"}

What I already tried:

• Increasing task computing power to 8gb memory and 4vcpu (before 2gb/1vcpu)
• Spawning 8 parallel instances (normally only 1 running)
• Blocking all requests in LB
• Deleting 100k small files
• Restarting Redis

Database load seems normal.

I don't know what else I can do. Is it somehow possible to connect via ssh to the Nextcloud instance? Or do you have any other suggestions? What could be the problem here?

ECS not taged

When deploying the stag it seems that the Fargate tasks are not tagged and thus do not show up in the cost explorer.

The bucket you are attempting to access must be addressed using the specified endpoint

When trying to deploy the stack I get this error message in the VpcStack deployment:

S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. 
Please send all future requests to this endpoint. For more information check 
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

I am using the following commands to package and deploy.

aws cloudformation package \
      --template-file ecs-nextcloud.yml --s3-bucket <my-bucket> \
      --output-template-file packaged.yaml

and deploy

aws cloudformation deploy --template-file packaged.yaml --stack-name nextcloud-test-env \
      --parameter-overrides DbPassword=<dbpass> NextCloudAdminPassword=<adminpass> \
       Route53Zone=<zoneid> Domain=nc.<example.com> IsolationLevel=Public --capabilities CAPABILITY_IAM

Not sure what I am missing here. I would like to deploy 22.0.0. I figure hat the error might stem from my AWS_REGION selection, which is eu-west-1.

Upgrade pass

I deployed with 21.0.1 and now I want to upgrade to a new version.

I did this after the initial deployment worked:

aws cloudformation deploy \
    --template-file ${OUTPUT_TEMPLATE_FILE} \
    --stack-name ${STACK_NAME} \
    --parameter-overrides \
        SuspendAutoScaling=true --capabilities CAPABILITY_IAM \
    --tags env=${TAG_ENV} service=${TAG_SERVICE}

Afterwards:

aws cloudformation deploy \
    --template-file ${OUTPUT_TEMPLATE_FILE} \
    --stack-name ${STACK_NAME} \
    --parameter-overrides \
        NextCloudVersion=${NC_VERSION} --capabilities CAPABILITY_IAM \
    --tags env=${TAG_ENV} service=${TAG_SERVICE}

That worked for me - I wonder if that is the recommended way and if so - can we add it to the README.md?

Internal Server Error

After fresh install (stable/ 23.0.0 /22.2.3/ 21.0.7/ 21.0.1) via AWS Console CFN GUI (with default config), above error show up when going to the url on output. I have tried installing the 5 build mentioned to us-east-1 and stopping task in ECS (to let ECS auto re-provision), but with no luck. May I ask if any idea/ where I could locate more log and info pls (I am new to AWS)? Thanks a lot.

Yaml to json?

Filing mainly to have a discussion around reasons to refactor the current template notation from yaml to json?

Documentation issue

Great Project!
The documentation assumes that the user has the AWS CLI installed and is familiar with it as well as CloudFormation using the CLI.
Please consider adding the following links in the documentation as reference . . .

This project assumes a familiarity with AWS CLI and AWS CloudFormation. Additional reference can be found at the following links:
https://aws.amazon.com/cli/
https://aws.amazon.com/cloudformation/

Elaboration on the deploy process and price

According to the aws-cli manual the command --output-file is not available.

aws cloudformation package --template-file ecs-nextcloud.yml --s3-bucket <cfn-artifact-bucket-name> --output-file packaged.yaml

I reckon for novice Cloudformation users it would be helpful to elaborate on the deployment process. How do I get the parameters configured and how do I actually deploy the stack at the end?

Also, a price estimate for different deployments would be awesome. Like:

• what is and how much costs the smallest possible deployment (I guess public and everything configured with minimum values) for testing?
• what is and how much costs the smallest production ready deployment?
• what is and how much costs the recommended production deployment?