Giter Site home page Giter Site logo

matthewjohn / terrareg Goto Github PK

View Code? Open in Web Editor NEW
240.0 6.0 19.0 10.41 MB

Open source Terraform module registry with UI, optional Git integration and deep analysis

Home Page: https://gitlab.dockstudios.co.uk/pub/terrareg

License: GNU General Public License v3.0

Dockerfile 0.11% Shell 0.04% Python 82.77% JavaScript 12.48% HTML 4.32% Mako 0.01% CSS 0.07% SCSS 0.19%
terraform-modules terraform-registry terraform python

terrareg's Introduction

Terrareg

Open source Terraform Registry.

Provides features to aid usage and discovery of modules, providing:

  • Fully implemented Terraform modules API
  • Completely API driven for automating all features
  • Host Terraform modules internally or from an external Git source
  • Analytics about the usage of modules
  • All information about a module - README, inputs, outputs, provider requirements and managed resources
  • Security alerts for each module, submodule and examples
  • Cost estimation for each module example
  • Module example source code within the UI, with automatic rewriting of 'source' arguments
  • Interactive 'Usage builder', helping users build terraform to use the terraform modules
  • Hooks for git SCM applications to automatically index modules
  • Authentication via SSO (OpenIDConnect/SAML2) and GitHub
  • Terraform provider support (very early alpha version)

If you like and use this project and are happy to let us know, please raise a GitHub issue, create a PR or contact MatthewJohn so it can be added to the README :)

For a full list of issues and pull requests, please see https://gitlab.dockstudios.co.uk/pub/terrareg

Screenshots

Homepage

Search

Module Provider

Example

Getting started

# Clone the repository
git clone https://github.com/matthewJohn/terrareg
cd terrareg

# Builder docker image
docker build . -t terrareg:latest

# Create secret key for session data
export SECRET_KEY=$(python -c 'import secrets; print(secrets.token_hex())')

# Run container, specifying secret key and admin password
docker run -ti -p 5000:5000 -e PUBLIC_URL=http://localhost:5000 -e MIGRATE_DATABASE=True -e SECRET_KEY=$SECRET_KEY -e ADMIN_AUTHENTICATION_TOKEN=MySuperSecretPassword terrareg:latest

The site can be accessed at http://localhost:5000, which will provide a 'Initial Setup' guide for getting started.

For information about running with docker-compose, SSL certificate generation or running without docker, see docs/LOCAL_DEVELOPMENT.md.

Upload a terraform module:

From a git repo

From source

terrareg_root=$PWD

cd source/of/my/module

# OPTIONAL: Create an terrareg meta-data file
echo '{ "description": "My first module", "owner": "ME!", "source": "https://github.com/me/my-tf-module" }' > ./terrareg.json

# Zip up module
zip -r ../my-tf-module.zip *

# Upload to terrareg
bash $terrareg_root/scripts/upload_module.sh http://localhost:5000 helloworld my-tf-module aws 1.0.0 ../my-tf-module.zip

Navigate to http://localhost:5000 to get started, or http://localhost/modules/helloworld/my-tf-module to see the uploaded example!

NOTE: To use modules from the registry in Terraform, a valid SSL certificate is incredibly highly recommended. Terraform will not work if you supply http:// as a protocol for the module URL, as it will treat this as a direct HTTP download, rather than treating it as a registry. Whilst Terrareg does handle these types of requests, you will not be able to the module version ranges that Terraform provides.

Additional configuration

For a full list of environment variables, please see Deployment Guide and docs/CONFIG.md

User Guide

For documentation about features, deployment, configuration and using Terrareg, please see: https://matthewjohn.github.io/terrareg

Terraform Provider

The Terrareg Terraform provider can be used to create namespaces, modules and manage other aspects of Terrareg:

Upgrading

For notes on upgrading, please see: https://matthewjohn.github.io/terrareg/UPGRADE

API Documentation

See the API Docs for a list of all API endpoint.

Changelog

See CHANGELOG.md

Local development

For all installation about local development, see https://matthewjohn.github.io/terrareg/LOCAL_DEVELOPMENT.

Since terraform requires HTTPS with a valid SSL cert, this must be provided in local development

On linux, by default, non-privileged users cannot listen on privileged ports, so the following can be used to route requests locally to port 5000:

sudo iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 443 -j REDIRECT --to-ports 5000

Example to run in local development environment:

virtualenv -ppython3.8 venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Without SSL cert
PUBLIC_URL=http://localhost:5000 ALLOW_CUSTOM_GIT_URL_MODULE_PROVIDER=False ALLOW_CUSTOM_GIT_URL_MODULE_VERSION=False GIT_PROVIDER_CONFIG='[{"name": "GitHub", "base_url": "https://github.com/{namespace}/{module}", "clone_url": "ssh://[email protected]/{namespace}/{module}.git", "browse_url": "https://github.com/{namespace}/{module}/tree/{tag}/{path}"}, {"name": "Bitbucket", "base_url": "https://bitbucket.org/{namespace}/{module}", "clone_url": "ssh://[email protected]/{namespace}/{module}-{provider}.git", "browse_url": "https://bitbucket.org/{namespace}/{module}-{provider}/src/{tag}/{path}"}, {"name": "Gitlab", "base_url": "https://gitlab.com/{namespace}/{module}", "clone_url": "ssh://[email protected]/{namespace}/{module}-{provider}.git", "browse_url": "https://gitlab.com/{namespace}/{module}-{provider}/-/tree/{tag}/{path}"}]' SECRET_KEY=ec9b8cc5ed0404acb3983b7836844d828728c22c28ecbed9095edef9b7489e85 ADMIN_AUTHENTICATION_TOKEN=password ANALYTICS_AUTH_KEYS=xxxxxx.atlasv1.zzzzzzzzzzzzz:dev,xxxxxx.atlasv1.xxxxxxxxxx:prod VERIFIED_MODULE_NAMESPACES=hashicorp TRUSTED_NAMESPACES=test DEBUG=True AUTO_PUBLISH_MODULE_VERSIONS=False LISTEN_PORT=5001 python ./terrareg.py

# With SSL Cert
# Add the following argument
#  --ssl-cert-private-key ./example/ssl-certs/private.pem --ssl-cert-public-key ./example/ssl-certs/public.pem

Committing

See CONTRIBUTING.md

License

This project and all associated code is covered by GNU General Public License v3.0.

For full license, see LICENSE.

For licenses of third party software and libraries used in this project, see LICENSE.third-party.

Contributors

  • @mholttech for usage builder improvements and docker-compose dev environment
  • @Davidsoff for implementing GitHub hook support
  • @0xken for impleenting a custom Cherry Dark theme
  • @chriscunningham-trivago for fixing tfswitch bug in docker build
  • @bmaximuml for fixing typos and Dockerfile improvements
  • @cayce-hengeveld for grammar fixes

terrareg's People

Contributors

0xken avatar alexanderwiechert avatar bmaximuml avatar cayce-hengeveld avatar chriscunningham-trivago avatar davidsoff avatar matthewjohn avatar mholttech avatar semantic-release-bot avatar snyk-bot avatar virtualroot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

terrareg's Issues

Feature Request: Allow for Simplified Version Tagging

Hello!

Our internal versioning scheme is set up to use a simplified format, such as "0.2", it would be incredibly useful if we could have the flexibility to use a simplified version rather than SemVer.

Proposed solution:
Introduce the option to use a two-digit version tagging format (major.minor), alongside the existing three-digit format. As i understand the current version format is enforced through RegEx - If needed, we can certainly provide the specific RegEx string that would accommodate this proposed change. By adjusting the RegEx, we can ensure that the two-digit format remains valid while preserving the integrity of our versioning system.

Thank you for your time and consideration.

dowload module fail

Hi John

I have a problem about download module file when run 'terraform init' ,the console shows that the url of module source.zip is invaild
image

however , I have found the source.zip in docker contianer
image

the docker logs shows this may be an 204 http error
image

how should I fix it
thanks a lot

Completely disable analytics?

Is there a way to completely disable analytics? We don't want to use it at all since the extra field in the Usage Builder is confusing as is the custom url it generates. We already have issues with people copy/pasting terraform from other teams and this will only lead to incorrect analytics and users confused by URLs.

Feature req: Specify modules / namespaces as code

I have terrareg deployed in multiplie environments. It would be nice to be able to specify modules and namespaces to create (if they don't exist) in a configuration file (JSON/YAML maybe?) which I could then load in. This owuld allow me to define the modules in one place and then easily copy them between environments. It also effectively self documents which modules are present, and how to add new ones

Git clone is using the wrong URL

Trying to create a module using the required ssh://[email protected]:org/terraform-s3-bucket.git format but it consistently fails to index in the UI. It gives this error message:
Error occurred during git clone: fatal: Could not read from remote repository.

I went in and added a print to GitModuleExtractor._clone_repository() and the ssh command looks like this:

terrareg_1  | ['git', 'clone', '--single-branch', '--branch', 'v6.0.5', 'ssh://[email protected]:org/terraform-s3-bucket.git', '/tmp/tmpo1yx9kao']

The ssh:// prefix is messing up the command. If I exec into the container and run the command manually I see this error which matches

git clone --single-branch --branch v6.0.5 ssh://[email protected]:org/terraform-s3-bucket.git
Cloning into 'terraform-s3-bucket'...
ssh: Could not resolve hostname github.com:org: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I confirmed I am mounting my ssh key and I can manually clone in the container

Upgrading to v3 - failed migrations

Hello 👋

Hope you're doing well @MatthewJohn 😁

When upgrading from v2.81.4 to v3.0.4 I get the following error with MIGRATE_DATABASE set to True:

Log snippet (MIGRATE_DATABASE: True) terrareg-6c56759c95-fgz25 terrareg INFO [alembic.runtime.migration] Context impl MySQLImpl. terrareg-6c56759c95-fgz25 terrareg INFO [alembic.runtime.migration] Will assume non-transactional DDL. terrareg-6c56759c95-fgz25 terrareg INFO [alembic.runtime.migration] Running upgrade fb6a94791a14 -> ee14b27baeb1, Add tables for provider sources, providers and tables for supporting data for provider versions terrareg-6c56759c95-fgz25 terrareg Traceback (most recent call last): terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context terrareg-6c56759c95-fgz25 terrareg self.dialect.do_execute( terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute terrareg-6c56759c95-fgz25 terrareg cursor.execute(statement, parameters) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/cursor.py", line 615, in execute terrareg-6c56759c95-fgz25 terrareg self._handle_result(self._connection.cmd_query(stmt)) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 1046, in cmd_query terrareg-6c56759c95-fgz25 terrareg result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query)) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 824, in _handle_result terrareg-6c56759c95-fgz25 terrareg raise get_exception(packet) terrareg-6c56759c95-fgz25 terrareg mysql.connector.errors.ProgrammingError: 1050 (42S01): Table 'provider_analytics' already exists terrareg-6c56759c95-fgz25 terrareg terrareg-6c56759c95-fgz25 terrareg The above exception was the direct cause of the following exception: terrareg-6c56759c95-fgz25 terrareg terrareg-6c56759c95-fgz25 terrareg Traceback (most recent call last): terrareg-6c56759c95-fgz25 terrareg File "/usr/local/bin/alembic", line 8, in terrareg-6c56759c95-fgz25 terrareg sys.exit(main()) terrareg-6c56759c95-fgz25 terrareg ^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/config.py", line 591, in main terrareg-6c56759c95-fgz25 terrareg CommandLine(prog=prog).main(argv=argv) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/config.py", line 585, in main terrareg-6c56759c95-fgz25 terrareg self.run_cmd(cfg, options) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/config.py", line 562, in run_cmd terrareg-6c56759c95-fgz25 terrareg fn( terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/command.py", line 378, in upgrade terrareg-6c56759c95-fgz25 terrareg script.run_env() terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/script/base.py", line 576, in run_env terrareg-6c56759c95-fgz25 terrareg util.load_python_file(self.dir, "env.py") terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file terrareg-6c56759c95-fgz25 terrareg module = load_module_py(module_id, path) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py terrareg-6c56759c95-fgz25 terrareg spec.loader.exec_module(module) # type: ignore terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "", line 994, in exec_module terrareg-6c56759c95-fgz25 terrareg File "", line 488, in _call_with_frames_removed terrareg-6c56759c95-fgz25 terrareg File "/app/terrareg/alembic/env.py", line 82, in terrareg-6c56759c95-fgz25 terrareg run_migrations_online() terrareg-6c56759c95-fgz25 terrareg File "/app/terrareg/alembic/env.py", line 76, in run_migrations_online terrareg-6c56759c95-fgz25 terrareg context.run_migrations() terrareg-6c56759c95-fgz25 terrareg File "", line 8, in run_migrations terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/runtime/environment.py", line 868, in run_migrations terrareg-6c56759c95-fgz25 terrareg self.get_context().run_migrations(**kw) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/runtime/migration.py", line 622, in run_migrations terrareg-6c56759c95-fgz25 terrareg step.migration_fn(**kw) terrareg-6c56759c95-fgz25 terrareg File "/app/terrareg/alembic/versions/ee14b27baeb1_add_tables_for_provider_sources_.py", line 21, in upgrade terrareg-6c56759c95-fgz25 terrareg op.create_table('provider_analytics', terrareg-6c56759c95-fgz25 terrareg File "", line 8, in create_table terrareg-6c56759c95-fgz25 terrareg File "", line 3, in create_table terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/operations/ops.py", line 1255, in create_table terrareg-6c56759c95-fgz25 terrareg return operations.invoke(op) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/operations/base.py", line 401, in invoke terrareg-6c56759c95-fgz25 terrareg return fn(self, operation) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/operations/toimpl.py", line 114, in create_table terrareg-6c56759c95-fgz25 terrareg operations.impl.create_table(table) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/ddl/impl.py", line 354, in create_table terrareg-6c56759c95-fgz25 terrareg self._exec(schema.CreateTable(table)) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/alembic/ddl/impl.py", line 193, in _exec terrareg-6c56759c95-fgz25 terrareg return conn.execute( # type: ignore[call-overload] terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1380, in execute terrareg-6c56759c95-fgz25 terrareg return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection terrareg-6c56759c95-fgz25 terrareg return connection._execute_ddl( terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1472, in _execute_ddl terrareg-6c56759c95-fgz25 terrareg ret = self._execute_context( terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context terrareg-6c56759c95-fgz25 terrareg self._handle_dbapi_exception( terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception terrareg-6c56759c95-fgz25 terrareg util.raise_( terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ terrareg-6c56759c95-fgz25 terrareg raise exception terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context terrareg-6c56759c95-fgz25 terrareg self.dialect.do_execute( terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute terrareg-6c56759c95-fgz25 terrareg cursor.execute(statement, parameters) terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/cursor.py", line 615, in execute terrareg-6c56759c95-fgz25 terrareg self._handle_result(self._connection.cmd_query(stmt)) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 1046, in cmd_query terrareg-6c56759c95-fgz25 terrareg result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query)) terrareg-6c56759c95-fgz25 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-6c56759c95-fgz25 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 824, in _handle_result terrareg-6c56759c95-fgz25 terrareg raise get_exception(packet) terrareg-6c56759c95-fgz25 terrareg sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) 1050 (42S01): Table 'provider_analytics' already exists terrareg-6c56759c95-fgz25 terrareg [SQL: terrareg-6c56759c95-fgz25 terrareg CREATE TABLE provider_analytics ( terrareg-6c56759c95-fgz25 terrareg id INTEGER NOT NULL AUTO_INCREMENT, terrareg-6c56759c95-fgz25 terrareg provider_version_id INTEGER NOT NULL, terrareg-6c56759c95-fgz25 terrareg timestamp DATETIME, terrareg-6c56759c95-fgz25 terrareg terraform_version VARCHAR(128), terrareg-6c56759c95-fgz25 terrareg namespace_name VARCHAR(128), terrareg-6c56759c95-fgz25 terrareg provider_name VARCHAR(128), terrareg-6c56759c95-fgz25 terrareg PRIMARY KEY (id) terrareg-6c56759c95-fgz25 terrareg ) terrareg-6c56759c95-fgz25 terrareg terrareg-6c56759c95-fgz25 terrareg ] terrareg-6c56759c95-fgz25 terrareg (Background on this error at: https://sqlalche.me/e/14/f405)

and the following with it set to False:

Log snippet (MIGRATE_DATABASE: False) terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,284 INFO sqlalchemy.engine.Engine SELECT DATABASE() terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,284 INFO sqlalchemy.engine.Engine [raw sql] {} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,289 INFO sqlalchemy.engine.Engine SELECT @@sql_mode terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,289 INFO sqlalchemy.engine.Engine [raw sql] {} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,293 INFO sqlalchemy.engine.Engine SELECT @@lower_case_table_names terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,293 INFO sqlalchemy.engine.Engine [raw sql] {} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,301 INFO sqlalchemy.engine.Engine SELECT git_provider.id, git_provider.name, git_provider.base_url_template, git_provider.clone_url_template, git_provider.browse_url_template terrareg-ddcd67f55-n2c87 terrareg FROM git_provider terrareg-ddcd67f55-n2c87 terrareg WHERE git_provider.name = %(name_1)s terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,301 INFO sqlalchemy.engine.Engine [generated in 0.00237s] {'name_1': 'GitHub'} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,310 INFO sqlalchemy.engine.Engine SELECT git_provider.id, git_provider.name, git_provider.base_url_template, git_provider.clone_url_template, git_provider.browse_url_template terrareg-ddcd67f55-n2c87 terrareg FROM git_provider terrareg-ddcd67f55-n2c87 terrareg WHERE git_provider.id = %(id_1)s terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,310 INFO sqlalchemy.engine.Engine [generated in 0.00173s] {'id_1': 1} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,317 INFO sqlalchemy.engine.Engine UPDATE git_provider SET base_url_template=%(base_url_template)s, clone_url_template=%(clone_url_template)s, browse_url_template=%(browse_url_template)s WHERE git_provider.id = %(id_1)s terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,317 INFO sqlalchemy.engine.Engine [generated in 0.00174s] {'base_url_template': 'https://github.com/{namespace}/{module}', 'clone_url_template': 'ssh://[email protected]/{namespace}/{module}.git', 'browse_url_template': 'https://github.com/{namespace}/{module}/tree/{tag}/{path}', 'id_1': 1} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,319 INFO sqlalchemy.engine.Engine COMMIT terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,325 INFO sqlalchemy.engine.Engine BEGIN (implicit) terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,330 INFO sqlalchemy.engine.Engine SELECT provider_category.id, provider_category.name, provider_category.slug, provider_category.user_selectable terrareg-ddcd67f55-n2c87 terrareg FROM provider_category terrareg-ddcd67f55-n2c87 terrareg WHERE provider_category.id = %(id_1)s terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,330 INFO sqlalchemy.engine.Engine [generated in 0.00273s] {'id_1': 1} terrareg-ddcd67f55-n2c87 terrareg 2024-02-12 17:05:56,359 INFO sqlalchemy.engine.Engine ROLLBACK terrareg-ddcd67f55-n2c87 terrareg Traceback (most recent call last): terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context terrareg-ddcd67f55-n2c87 terrareg self.dialect.do_execute( terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute terrareg-ddcd67f55-n2c87 terrareg cursor.execute(statement, parameters) terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/cursor.py", line 615, in execute terrareg-ddcd67f55-n2c87 terrareg self._handle_result(self._connection.cmd_query(stmt)) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 1046, in cmd_query terrareg-ddcd67f55-n2c87 terrareg result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query)) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 824, in _handle_result terrareg-ddcd67f55-n2c87 terrareg raise get_exception(packet) terrareg-ddcd67f55-n2c87 terrareg mysql.connector.errors.ProgrammingError: 1146 (42S02): Table 'terrareg-database.provider_category' doesn't exist terrareg-ddcd67f55-n2c87 terrareg terrareg-ddcd67f55-n2c87 terrareg The above exception was the direct cause of the following exception: terrareg-ddcd67f55-n2c87 terrareg terrareg-ddcd67f55-n2c87 terrareg Traceback (most recent call last): terrareg-ddcd67f55-n2c87 terrareg File "/app/./terrareg.py", line 20, in terrareg-ddcd67f55-n2c87 terrareg s = Server(ssl_public_key=args.ssl_pub_key, ssl_private_key=args.ssl_priv_key) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/app/terrareg/server/__init__.py", line 139, in __init__ terrareg-ddcd67f55-n2c87 terrareg terrareg.provider_category_model.ProviderCategoryFactory.get().initialise_from_config() terrareg-ddcd67f55-n2c87 terrareg File "/app/terrareg/provider_category_model.py", line 192, in initialise_from_config terrareg-ddcd67f55-n2c87 terrareg existing_provider_category = self.get_provider_category_by_pk(pk=pk) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/app/terrareg/provider_category_model.py", line 126, in get_provider_category_by_pk terrareg-ddcd67f55-n2c87 terrareg if instance.exists: terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/app/terrareg/provider_category_model.py", line 43, in exists terrareg-ddcd67f55-n2c87 terrareg return bool(self._get_db_row()) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/app/terrareg/provider_category_model.py", line 70, in _get_db_row terrareg-ddcd67f55-n2c87 terrareg res = conn.execute(select) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1380, in execute terrareg-ddcd67f55-n2c87 terrareg return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection terrareg-ddcd67f55-n2c87 terrareg return connection._execute_clauseelement( terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement terrareg-ddcd67f55-n2c87 terrareg ret = self._execute_context( terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context terrareg-ddcd67f55-n2c87 terrareg self._handle_dbapi_exception( terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception terrareg-ddcd67f55-n2c87 terrareg util.raise_( terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ terrareg-ddcd67f55-n2c87 terrareg raise exception terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context terrareg-ddcd67f55-n2c87 terrareg self.dialect.do_execute( terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute terrareg-ddcd67f55-n2c87 terrareg cursor.execute(statement, parameters) terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/cursor.py", line 615, in execute terrareg-ddcd67f55-n2c87 terrareg self._handle_result(self._connection.cmd_query(stmt)) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 1046, in cmd_query terrareg-ddcd67f55-n2c87 terrareg result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query)) terrareg-ddcd67f55-n2c87 terrareg ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ terrareg-ddcd67f55-n2c87 terrareg File "/usr/local/lib/python3.12/site-packages/mysql/connector/connection.py", line 824, in _handle_result terrareg-ddcd67f55-n2c87 terrareg raise get_exception(packet) terrareg-ddcd67f55-n2c87 terrareg sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) 1146 (42S02): Table 'terrareg-database.provider_category' doesn't exist terrareg-ddcd67f55-n2c87 terrareg [SQL: SELECT provider_category.id, provider_category.name, provider_category.slug, provider_category.user_selectable terrareg-ddcd67f55-n2c87 terrareg FROM provider_category terrareg-ddcd67f55-n2c87 terrareg WHERE provider_category.id = %(id_1)s] terrareg-ddcd67f55-n2c87 terrareg [parameters: {'id_1': 1}] terrareg-ddcd67f55-n2c87 terrareg (Background on this error at: https://sqlalche.me/e/14/f405)

Any way I can upgrade without losing data?

Provide method to specify Terraform version per module

At the moment, each module needs a constraint, but ideally in the root module, each sub-module and each example.

Maybe a configuration on the module (or specified in the terrareg.json file) that determines the verison of Terraform that should be used for extraction could be useful?
Alternatively, could the entire code-base be scanned for any terraform version constraints and be used as a fallback?

Enforce authentication to be able to use Terrareg

I've realised quite a few people are using OpenIDC and SAML within corporate environments.
There are a couple of tickets on the backlog (https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/392, https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/289) to enforce authentication (basically removing anonymous read access to Terrareg).
This would also mean adding authentication when using Terraform.

I'm creating this ticket as a place for people to vote on it, to see if it's something that would be useful - so please give it a thumbs up, if it's something that's wanted :)

Matt

API URL's

Hi,
I am unable to find the API URL's for creating namespaces and the like. Can you point me in the right direciton?

Feature Request: terraform provider

Are there any plans for a terraform provider for terrareg?

I am currently debating whether to deploy terrareg in my org to move away from our current GitLab module registry.

Since we create our module repositories via terraform anyways, it would be nice to shine up our repo modules by automatically creating the corresponding terrareg module and the namespaces along with our gitlab groups.

If this is not currently in the pipeline, would you be amendable to support me if I find the time to create a provider for this?

Feature Request: Update Module Version Archive to Include Only Specified Path Contents

Hey,

I've noticed that currently when specifying a git path for creating a module, the archive generated includes the entire repository's contents, not just the specified path.

Current Behavior

When specifying a git path for a module, the archive created by Terrareg contains the whole repository's contents. For example, with repo Structure:

.
├── Dockerfile
├── README.md
└── my-module
    ├── main.tf

If the specified git path is /my-module, Terrareg currently generates an archive reflecting the entire folder structure as shown above.

Proposed Change

Update the module version archive creation process to only include the contents of the specified path. For example if the specified git path is /my-module, the archive would contain:

└── my-module
    ├── main.tf

Benefits

  • Less Storage Capacity Used: The archive now contains only the necessary parts of the module
  • Faster Downloads: Faster Module Downloads
  • Cheaper Cloud Storage Costs: Lower Storage = less costs
  • Privacy Concerns: Sometimes we don't want to share all the contents of our repo
  • Easier for Debug modules: It would be easier for a Dev working locally to debug the contents of the module being used .terraform folder as there is less clutter within the files.

Allow numbers on module name

Hi,

Feature request

I would like to name my module S3. Right now that is throwing an error. It doesn't say specifically the problem but I assume is the number.

Module name is invalid

Best practices for git repository containing multiple modules

I've got a single repository containing multiple modules at different distances down the tree (structure below). I'm trying to work out the best values to supply to GIT_PROVIDER_CONFIG to make it as easy as possible to set up these modules. So far I have something like what's below, but terrareg won't accept it because it wants both {namespace} and {module} in the clone_url.

Have I got the right idea here, or am I missing something obvious?

{"name": "multiple modules", "base_url": "https://github.com/example/{namespace}/tree/main/dir1/{module}/terraform", "clone_url": "ssh://[email protected]/vivacitylabs/{namespace}.git", "browse_url": "https://github.com/vivacitylabs/{namespace}/tree/{tag}/cloud/{module}/{path}", "tag_format": "v{major}.{minor}"}

Repo layout:

/
==> /dir1
=======> /dir1/nest1/module1
=======> /dir1/nest1/module2
==> /dir2
=======> /dir2/module3

Use tenv instead of tfswitch

Hi,
I've noticed that you're using tfswitch for version management. It's sad, but tfswitch does not support OpenTofu.
My team created a project tenv that supports Terraform, OpenTofu and Terragrunt in one tool. I suggest to migrate to it, because it allow you to manage everything related to Terraform just by one tool.

You're also welcome to open any issues or contribute to tenv.

Migrate to tenv from tfswitch

Hi! I saw that you're using tfswitch in this repository. It's a really good tool, but it does not support OpenTofu and Terragrunt. My team designed a successor of tenv that support Terraform, Terragrunt and OpenTofu. It will be a good idea to migrate into it in due to the growing popularity of OpenTofu.
url: https://github.com/tofuutils/tenv

I'll be welcomed to help with such migration.

OIDC Login error: (insecure_transport) OAuth 2 MUST utilize https.

So, I built myself a terrareg container today from current HEAD on main branch.

It's deploy with the following ENV vars via docker compose:

version: '3'

networks:
  gitea:
    external: false

services:
  terrareg:
    image: git.home.lynxcore.org/homelab/terrareg/terrareg:latest
    container_name: "terrareg"
    restart: unless-stopped
   environment:
      - MIGRATE_DATABASE=True
      - SECRET_KEY=YEAHNO
      - PUBLIC_URL=https://terraform.home.lynxcore.org
      - DOMAIN_NAME=https://terraform.home.lynxcore.org
      - OPENID_CONNECT_CLIENT_ID=STILLNO
      - OPENID_CONNECT_CLIENT_SECRET=NICETRY
      - OPENID_CONNECT_ISSUER=https://auth.home.lynxcore.org/application/o/terrareg/
      - OPENID_CONNECT_SCOPES=openid,profile,email,groups
      - AUTO_CREATE_NAMESPACE=False
      - AUTO_CREATE_MODULE_PROVIDER=False
      - UPLOAD_API_KEYS=ASIF
      - PUBLISH_API_KEYS=INYOURDREAMS
    ports:
      - "127.0.0.1:5000:5000"
    volumes:
      - /srv/docker/code/terrareg:/app/data
    networks:
      - gitea
    healthcheck:
      test: curl --fail http://localhost:5000 || exit 1
      interval: 5s
      retries: 30
      start_period: 30s
      timeout: 10s

When trying to log in, my IdP (authentik) authenticates me just fine, but I get an Invalid response from SSO error.

Now, after digging into this for a few hours with DEBUG=true etc, I managed to coax an error message by changing Line 39 (error_description) in terrareg/server/api/open_id_callback.py to

            res = make_response(
                render_template(
                    "error.html",
                    error_title="Login error",
                    error_description=f"""Invalid response from SSO for url {request.url} | {exc}""",   # this is what I changed
                )
            )

With that I get the following error:
Invalid response from SSO for url http://terraform.home.lynxcore.org/openid/callback?code=REDACTED&state=REDACTED | (insecure_transport) OAuth 2 MUST utilize https..

I then proceeded to see where the hell the "http" comes from - no dice. I just can't find it.
I verified that everything works IdP-Wise by setting OAUTHLIB_INSECURE_TRANSPORT=1 in my docker-compose.yml, and I can log in just fine.

Something in the depths of the OIDC implementation here overwrites the protocol to http, and I can 't find it.

Setup-wise I have HAProxy before terrareg, and HAProxy terminates SSL for me.

Complete HAProxy config if it's relevant:

global
  chroot  /usr/share/haproxy
  daemon  
  group  haproxy
  log  127.0.0.1 local0
  ssl-default-bind-ciphers  ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-bind-options  no-sslv3 no-tls-tickets
  ssl-default-server-ciphers  ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-server-options  no-sslv3 no-tls-tickets
  stats  timeout 30s
  tune.ssl.default-dh-param  2048
  user  haproxy

defaults
  log  global
  maxconn  2000
  mode  http
  option  http-server-close
  option  httplog
  option  dontlognull
  option  redispatch
  option  srvtcpka
  option  clitcpka
  retries  3
  timeout  http-request 10s
  timeout  client 20s
  timeout  connect 4s
  timeout  server 2m
  timeout  http-keep-alive 4s
  timeout  tunnel 2m
  timeout  client-fin 1s
  timeout  server-fin 1s
  timeout  check 10s
  timeout  queue 1m

frontend main
  bind :443 ssl crt /etc/haproxy/ssl
  bind :80 
  mode http
  acl terraform        hdr(host)  -i terraform.home.lynxcore.org
  default_backend terraform
  http-request set-header X-Forwarded-Proto https if  { ssl_fc }
  http-request set-header X-Forwarded-Proto http if  !{ ssl_fc }
  http-request redirect scheme https code 301 unless { ssl_fc }
  log-format %ci\ "%r"\ %ST\ %B\ "%hr"
  option forwardfor except 127.0.0.1/8
  option httplog
  option dontlognull
  use_backend terraform        if terraform


listen stats
  bind 127.0.0.1:9001 
  stats enable
  stats show-legends
  stats uri /
  stats realm haproxy\ statistics
  stats admin if TRUE

backend terraform
  balance roundrobin
  mode http
  server terraform 127.0.0.1:5000 check

I'm at my wits end. Any takers?

GitHub Webhook Validation issues

There seems to be some issue with the hash validation on GH release events.

SHA-256 seems to come in over the 'X-Hub-Signature-256' header, and is calculated by a combination of the request body and the secret key, as per: https://docs.github.com/en/webhooks/webhook-events-and-payloads . Right now Terrareg appears to be passing an empty byte string to the HMAC function and is attempting to pull the SHA-256 hash from the 'X-Hub-Signature' header, which is a SHA-1 hash. I've tried a few local changes, but am struggling to get the auth to work. I'll post if I find any solution here.

Bug: Namespaces incorrectly showing as empty

I have some modules defined in a namespace where both the namespace and the modules are defined using the terraform provider. This namespace is hidden in the list of namespaces in the gui, and only shows up when I click on the Show empty namespaces button. Clicking on the namespace does then show a list of modules.

about configure ssl key

I have deploy terrareg with docker and create ssl key with following command:

  1. openssl genrsa -out rsa_private_key.pem 1024
  2. openssl rsa -in rsa_private_key.pem -pubout -out rsa_public_key.pem

however when I recreate container with -e SSL_CERT_PRIVATE_KEY=pathincontainer/rsa_private_key.pem & -e SSL_CERT_PUBLIC_KEY=pathincontainer/rsa_public_key.pem , it show error in docker logs "ssl.SSLError: [SSL] PEM lib (_ssl.c:3900)"

How should I fix it ,if you have any demo for ssl configuration will be better.
thanks a lot

Optional module provider_name

Hello 👋 ,

All of my modules are being deployed internally on a GKE cluster, and therefore provider_name is always set to null for all the modules I create. It doesn't cause problems, but it feels a bit untidy having it there, especially as it's shown in so many places.

Is it possible to make this optional? Or have a config flag to disable it entirely?

Thanks!

Bug: server gets into weird state when doing massive parallel version loads

I've set up an instance of Terrareg with ~15 different modules backed by Git repositories. When attempting to load versions, one at a time per-module but in parallel across all modules, the server eventually gets into a state where it cannot do any version loads, throwing the following error for each attempt.

Using a MySQL DB backend, and EFS for the module data directory.

  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2548, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 271, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/views.py", line 107, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/app/terrareg/auth_wrapper.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/app/terrareg/server/error_catching_resource.py", line 41, in post
    return self._post(*args, **kwargs)
  File "/app/terrareg/server/api/module_version_create.py", line 27, in _post
    previous_version_published = module_version.prepare_module()
  File "/app/terrareg/models.py", line 2990, in prepare_module
    self.create_data_directory()
  File "/app/terrareg/models.py", line 2874, in create_data_directory
    if not os.path.isdir(self._module_provider.base_directory):
  File "/app/terrareg/models.py", line 1582, in base_directory
    return safe_join_paths(self._module.base_directory, self._name)
  File "/app/terrareg/models.py", line 891, in base_directory
    return safe_join_paths(self._namespace.base_directory, self._name)
  File "/app/terrareg/models.py", line 756, in base_directory
    return safe_join_paths(terrareg.config.Config().DATA_DIRECTORY, 'modules', self._name)
  File "/app/terrareg/config.py", line 82, in DATA_DIRECTORY
    return os.path.join(os.environ.get('DATA_DIRECTORY', os.getcwd()), 'data')
FileNotFoundError: [Errno 2] No such file or directory

Looking at an strace, it seems like os.getcwd is getting called on a directory which no longer exists. The issue seems to go away with a server re-deploy. I'm still digging into the issue, but wanted to file this.

The loads to work for some time, until this error happens, then they all fail.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.