Giter Site home page Giter Site logo

tk-cpenv's Introduction

cpenv

Manage software plugins, project dependencies and environment variables using Modules.

Installation

The recommended method of installing cpenv is via pipx. Pipx is used to install python cli applications in isolation.

pipx install cpenv
pipx upgrade cpenv
pipx uninstall cpenv

Overview

Cpenv is a cli tool and python library used to create, edit, publish, and activate Modules. A Module is a folder containing a dependency, like Arnold for Maya, and a module file that configures it.

Environment Variables

Variable Description Default
CPENV_HOME Customize path to cpenv home
CPENV_DISABLE_PROMPT Disable prompt when modules activated 0
CPENV_ACTIVE_MODULES List of activated modules
CPENV_SHELL Preferred subshell like "powershell"

Example Modules

Create a Module

Use cpenv create <module> to use a guide to create a new Module. This will create a new folder in your current working directory with a module file in it.

Edit a Module

Each Module contains a module.yml file, referred to as a module file. A module file contains metadata like the name and version of a module, as well as configuration, like environment variables.

# Variables
# $MODULE - path to this module
# $PLATFORM - platform name (win, mac, linux)
# $PYVER - python version (2.7, 3.6...)

# Wrap variables in brackets when they are nested within a string.
#     DO 'this${variable}isnested/' NOT 'this$variableisnested'

name: 'my_module'
version: '0.1.0'
description: 'My first module.'
author: 'Me'
email: '[email protected]'
requires: []
environment:
    MY_MODULE_VAR: 'Test'

Environment key

Setting a value will insert a key or overwrite it's existing value.

SOME_VAR: 'SOME_VALUE'

Use the $MODULE variable for module relative paths.

RELATIVE_VAR: $MODULE/bin

Use lists to prepend values to a key.

PATH:
    - $MODULE/bin

Use win, linux and mac or osx keys to declare platform specific values. If you leave out a platform, the variable will not be included on that platform.

PLATFORM_VAR:
    mac: /mnt/some/path
    linux: /Volumes/some/path
    win: C:/some/path

You can also use platform keys when prepending values to a variable.

PATH:
    - mac: $MODULE/macattack
      linux: $MODULE/penguin
      win: $MODULE/squares

Reuse environment variables to simplify things.

BASE: $MODULE/$PLATFORM/base
PATH:
    - $BASE/bin
PYTHONPATH:
    - $BASE/python

Advanced

The implicit set and prepend operations above cover the most common use cases when modifying environment variables. For more advanced use cases you can use the following explicit operation keys.

SVAR:
    set:
        - Value0
        - Value1
PVAR:
    prepend:
        - X
RVAR:
    unset: 1
PATH:
    remove:
        - C:/Python27
        - C:/Python27/Scripts
PYTHONPATH:
    append:
        - $MODULE/python
        - $MODULE/lib

You can also uses lists of opreations to perform complex modifications.

PATH:
    - remove: /some/file/path
    - append: /some/other/path
    - prepend: /one/more/path

One workflow that this enables is the use of modules solely for the purpose of overriding environment variables. Imagine you have a module my_tool and it uses a variable MY_TOOL_PLUGINS to lookup plugins.

name: my_tool
...
environment:
    MY_TOOL_PLUGINS:
        - //studio/dev/my_tool/plugins
        - //studio/projects/test_project/plugins

Now imagine you have a new project and you want my_tool to look at a different location for plugins just for that project. Rather than create a new version of the my_tool module, create a override module. We might name this module after our project, project_b.

name: project_b
...
environment:
    MY_TOOL_PLUGINS:
        set:
            - //studio/prod/my_tool/plugins
            - //studio/projects/project_b/plugins

All we have to do is activate my_tool and project_b in that order to make sure our overrides are used.

> cpenv activate my_tool project_b

Requires key

The requires key is a list of dependencies that a module needs to function. Currently this is only used for reference, these modules will not be activated automatically.

Test a Module

When you're working on a module navigate into it's root directory. Then you can activate it using cpenv activate .. This is the best way to validate your module prior to publishing.

Publish a Module

Once you're Module is ready for production, use cpenv publish . to publish it. Publishing a Module uploads it to a Repo of your choosing.

Repos

Repos are storage locations for Modules that support finding, listing, uploading, and downloading Modules via requirements like my_module-0.1.0. Cpenv is configured with the following LocalRepos by default:

  • cwd - Your current working directory
  • user - A user specific repo
  • home - A machine wide repo

Use cpenv repo list to display your configured Repos. LocalRepos point directly to folders on your local file system. Customize the home Repo by setting the CPENV_HOME environment variable.

When you activate a module using a requirement, all configured Repos are searched and the best match is used. If the resolved module is not in a LocalRepo it will be downloaded to your home Repo then activated. This is one of the key features of cpenv and allows for strong distributed workflows. For example, you can configure a remote repo like the ShotgunRepo and store your modules directly in a Shotgun studio database. Visit the tk-cpenv repository for more info on using cpenv with Shotgun

Requirements

Requirements are strings used to resolve and activate modules in Repos. They can be versionless like my_module or require a version like my_module-0.1.0. Cpenv supports semver/calver, simple versions (v1), and what I like to call weird versions like 12.0v2 (The Foundry products). In the future cpenv may support more complex requirements by utilizing resolvelib.

tk-cpenv's People

Contributors

danbradham avatar fabiangeisler avatar ricardomusch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

tk-cpenv's Issues

Store module requirements in CustomProjectEntity

Create a new CustomProjectEntity named Environment to store module requirements per engine.

Environment Fields:

project (Entity)
engine (Text) - Name of engine this Environment applies to
requires (Text) - Space separated list of requirements

Example:

project: {'id': <project_id>}
engine: 'tk-maya'
requirements: 'studio_maya-0.1.0 mtoa-4.0.3 studiolibrary-2.7.1'

[BUG] Empty environment causes GUI to break

When an empty environment for the engine to be loaded exists in the current project the choice menu breaks.

Steps to reproduce:

  1. Make two environments for a software, example Nuke and Nuketest - tk-nuke
  2. Add modules to Nuke but none to Nuketest
  3. Launch Nuke
  4. The Popup UI will break with the following error:
Traceback (most recent call last):
  File "C:\Users\ricardom\AppData\Roaming\somepath\p815c2015.basic.desktop\cfg\install\core\python\tank\platform\engine.py", line 1686, in _create_widget
    widget = derived_widget_class(*args, **kwargs)
  File "S:\_cache\shotgrid\bundle_cache\github\cpenv\tk-cpenv\v0.5.10\python\cpenv_ui\env_selector.py", line 42, in __init__
    + '\n'.join(environment['sg_requires'].split())
AttributeError: 'NoneType' object has no attribute 'split'

image

software_entity can be None in before_app_launch

In some setups, maybe with shotgun configs that do not use the newer software discovery mechanism for Shotgun, software_entity was None in before_app_launch.

2020-08-07 12:01:21,109 [   ERROR] [PROXY] Failed to activate cpenv modules.
Traceback (most recent call last):
  File "...\tk-multi-launchapp\before_app_launch.py", line 68, in execute
    **kwargs
  File ".../_bundle_cache\github\cpenv\tk-cpenv\v0.3.2\app.py", line 171, in before_app_launch
    engine = software_entity.get('engine', engine_name)
AttributeError: 'NoneType' object has no attribute 'get'

Removing a module from ShotGrid causes GUI to break

If you remove a module from ShotGrid and that is referenced in the environment that is about to be loaded in the tk-cpenv GUI, you will get an error dialog and the GUI wont show.

Manually fixing the environment in ShotGrid does work but is not particularly elegant. Would be nice to have some nice feedback about that in the interface, like missing modules marked in red or something.

Flow Desktop v1.9.1 Admin UI crash on startup

Traceback (most recent call last):
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\nodeslayers\p686c703.basic.desktop\cfg\install\core\python\tank\platform\engine.py", line 1704, in _create_widget
    widget = derived_widget_class(*args, **kwargs)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_selector.py", line 213, in __init__
    self.set_state_from_context(app.context)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_selector.py", line 246, in set_state_from_context
    self.update_widgets()
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_selector.py", line 277, in update_widgets
    self._update_available_list()
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_selector.py", line 306, in _update_available_list
    self.available_list.add_spec_set(spec_set)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_list.py", line 140, in add_spec_set
    item = self.create_item(spec_set)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\module_list.py", line 99, in create_item
    item = QtGui.QTreeWidgetItem(parent=self)
AttributeError: PySide2.QtWidgets.QTreeWidgetItem.__init__(): unsupported keyword 'parent'

Add support for Software Entities that do not have Engines.

During the application launch process an Environment lookup is performed using the engine_name provided as an argument to the before_app_launch hook. This should be extended to lookup Environments using the code field of the Software entity so that cpenv could be used for ALL software in ShotGrid, not just software with an associated toolkit engine.

Originally raised by @fabiangeisler in cpenv/cpenv#52.

Add support for multiple environments per engine.

If there are multiple Environments for the same toolkit Engine, show a selector dialog before launching an application. To make this even more useful, add a user / group multi-entity attribute to the Environment entity. When listing available Environments for an Engine use the user / group to restrict access to Environments. This would allow a developer to create test Environments for their access alone or just the dev team. To finish this off, add a permissions field to the Set Modules dialog!

Allow user defined ordering of modules

An environment's modules are currently sorted alphabetically and activated in that order.

But what we really want is for the user (TD/Job lead) to define the order in which modules are activated, so that we ensure modules that are meant to override some Environment Variables are loaded when the user expects.

Add pipeline configuration as additional filter

Currently environments can be user restricted, it would be a great option to add a restriction per Pipeline Configuration in cases where studios run a Stable and Beta/Alpha configuration as Pipeline Configuration.

Add additional tools to Module Selector dialog

Currently the ModuleSelector allows a user to select modules to activate for a project. It would be
nice if the dialog provided an interface to use more of cpenv's functionality like:

  • Preview Combined Environment Variables
  • Publishing modules
  • Cloning modules
  • Localizing modules

Publish to sg zip package seems to exclude .pyc files?

I noticed that when publishing to SG the zip files do not include .pyc files.
Not sure if this is by design but a plugin like animbot seems to break because of this.
I manually fixed it by replacing the zip file with one that has all files.

User Restriction dialog broken

When I try to click on user restriction in the GUI I get this broken dialog displayed:
image
and this error log in the SGTK console:

2022-08-23 13:06:22,212 [   ERROR] [PROXY] Traceback (most recent call last):
  File "/home/fabian.geisler/.shotgun/bundle_cache/github/cpenv/tk-cpenv/v0.5.10/python/cpenv_ui/env_permissions.py", line 66, in on_initialized
    except Exception:
  File "/home/fabian.geisler/.shotgun/bundle_cache/github/cpenv/tk-cpenv/v0.5.10/app.py", line 151, in debug
    self.logger.debug('tk-cpenv: %s' % (message % args))
TypeError: not enough arguments for format string

which is actually a bug in the except block of the initialization of the user widget.
By changing env_permissions.py line 66 from:

app.debug(traceback.format_exc())

to

app.logger.exception('Error on initialize.')

I get this more helpful error:

2022-08-23 13:13:42,014 [   ERROR] [PROXY] Error on initialize.
Traceback (most recent call last):
  File "/home/fabian.geisler/.shotgun/bundle_cache/github/cpenv/tk-cpenv/v0.5.10/python/cpenv_ui/env_permissions.py", line 62, in on_initialized
    parent=self,
  File "/home/fabian.geisler/.shotgun/bundle_cache/app_store/tk-framework-qtwidgets/v2.10.3/python/shotgun_fields/shotgun_field_manager.py", line 326, in create_widget
    sg_entity_type, field_name, entity, parent, **kwargs
  File "/home/fabian.geisler/.shotgun/bundle_cache/app_store/tk-framework-qtwidgets/v2.10.3/python/shotgun_fields/shotgun_field_manager.py", line 499, in _create_editor_widget
    if not shotgun_globals.field_is_editable(sg_entity_type, field_name):
  File "/home/fabian.geisler/.shotgun/bundle_cache/app_store/tk-framework-shotgunutils/v5.8.5/python/shotgun_globals/cached_schema.py", line 898, in field_is_editable
    "Could not find the schema for %s.%s" % (sg_entity_type, source_field_name)
ValueError: Could not find the schema for CustomEntity03.sg_permissions_users

This seems to indicate that some local ShotGrid schema cache is not updated correctly and I don't know why that is. Restarting SG Desktop does not help. I will dig some more to try to find the cause of this.

Any hint or fix is much appreciated!

PS: No promises, but if time allows I will submit a PR for the broken debug statement in the except block.

Show a progress dialog when downloading modules

Currently when modules are localized from a shotgun site, there is no visualization of notification that modules are being downloaded. This will cause users to think that Shotgun Desktop has frozen when launching applications.

This feature goes hand in hand with improved reporting in cpenv.

Overflow Error on large package

Archive is a Nuke Installation and is 4311799850 bytes (4.11GB).
Already had to change the SG field to a text field to be able to store the bytesize.

Traceback (most recent call last):
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\app.py", line 461, in before_app_launch
    self._before_app_launch(
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\app.py", line 529, in _before_app_launch
    modules = self.activate(requires)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\app.py", line 427, in activate
    return self.cpenv.activate(requirements)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\api.py", line 94, in activate
    modules = activator.activate(module_specs)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\resolver.py", line 112, in activate
    modules = self.localizer.localize(module_specs)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\resolver.py", line 233, in localize
    module = module_spec.repo.download(
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\repos\shotgun.py", line 178, in download
    with progress_bar as progress_bar:
  File "C:\Program Files\Shotgun\Python3\lib\contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\reporter.py", line 84, in progress_bar
    bar.start()
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv\reporter.py", line 35, in start
    self.reporter.start_progress(self.label, self.max_size, self.data)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\reporter.py", line 14, in run
    return fn(self, *args, **kwargs)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\reporter.py", line 115, in start_progress
    self.dialog.set_progress(max_size=max_size)
  File "C:\Users\sysadmin\AppData\Roaming\Shotgun\bundle_cache\sg\nodeslayers\v4864\bundle_cache\git\tk-cpenv.git\v0.5.12\python\cpenv_ui\dialogs.py", line 80, in set_progress
    self.progress.setRange(0, max_size)
OverflowError

Allow version-less modules in environments

It would be nice to keep some modules always at the "latest" version. For example the Maya scanner plugin which scans Maya scenes for malware content. At the moment I would need to update this module for all environments individually.

Proposed changes

Adding an extra entry to each module drop down called "latest", which in turn would save the module as version-less in the environment .

Notes

Removing the version from a module in an environment requires string, already resolved to the latest version in the GUI. But saving the environment adds the version again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.