armmbed/yotta

DEPRECATED: yotta build; better software

Deprecation note!

Please note: This repository is deprecated and it is no longer actively maintained.

yotta: Build Software with Reusable Components

yotta is a tool from ARM mbed, to make it easier to build better software with C++ and C by re-using modules. Publish your own modules to the yotta registry to share them with other people, or re-use them privately in your own projects.

Whenever you build a project with yotta, you first select a yotta target. Targets describe the platform that you're building for (such as an embedded IoT development board, or natively for Mac or Linux), and provide all the information that yotta and modules you're using need to configure themselves correctly for that platform.

Installation

yotta is written in python, and is installed using pip. Install yotta itself by running:

pip install yotta

Note that yotta needs several non-python dependencies to be installed correctly (such as a C++ compiler). The detailed installation instructions include a full guide.

Exactly which other dependencies (such as compilers and other build tools) are required will also depend on the yotta target description that you intend to use, so please be sure to also check the target description's own documentation.

Get Started!

The best way to get started is to follow the tutorial, or if you have questions/feedback please create an issue!

How yotta works

Every yotta module or application includes a module.json file, which lists the other modules that it needs (amongst other information like the module's license, and where to submit bug reports).

When you run yotta build to build your project, yotta downloads your dependencies, and makes them available to your project. It's similar in concept to npm, pip or gem: although because C and C++ are compiled languages, yotta also controls the build of your software in order to ensure downloaded modules are available to use in your code.

To add a new module to your program run yotta install <modulename>. yotta will install both the module you've specified and any of its dependencies that you don't already have. It will also update your module.json file to reflect the new dependency.

The best way to really understand how yotta works is to follow the tutorial.

Further Documentation

For further documentation see the yotta docs website.

Tips

• yt is a shorthand for the yotta command, and it's much quicker to type!
• yotta is strongly influenced by npm, the awesome node.js software packaging system. Much of the syntax for module description and commands is very similar.

License

yotta is licensed under Apache-2.0

yotta's Issues

yotta unlink doesn't exist

(as title) You can't unlink linked packages (easily).

The project isn't available as a dependency

If dependencies of the current component depend on the current component, install, update and build fail because they use/install a duplicate of the current component.

yotta publish to a non-owned package needs a clear error message

Currently the 403 (permission denied) causes a re-auth attempt with github. That's wrong.

Should check the owner first. In fact, the registry (and client) should check that the owner in the package/target description is the person who's publishing.

Option to `yotta up` that also updates in any linked components.

`yotta ninja` command that clones the source repo for a snapshot-only dependency

This would let you easily push fixes to a dependency.
If the source URL is a github one, it should also create a fork of the repo for you, if you don't have one already.

Ill-formed dependency list needs better error message

In the package.json file, dependency statements with invalid syntax produces unhandled python exception (see below) instead of outputting more informative error messages.

Traceback (most recent call last):
  File "/usr/local/bin/yt", line 9, in <module>
    load_entry_point('yotta==0.0.18', 'console_scripts', 'yt')()
  File "/Library/Python/2.7/site-packages/yotta/main.py", line 87, in main
    status = args.command(args)
  File "/Library/Python/2.7/site-packages/yotta/link.py", line 51, in execCommand
    dropSudoPrivs(lambda: install.execCommand(args))
  File "/Library/Python/2.7/site-packages/yotta/link.py", line 30, in dropSudoPrivs
    r = fn()
  File "/Library/Python/2.7/site-packages/yotta/link.py", line 51, in <lambda>
    dropSudoPrivs(lambda: install.execCommand(args))
  File "/Library/Python/2.7/site-packages/yotta/install.py", line 31, in execCommand
    installDeps(args)
  File "/Library/Python/2.7/site-packages/yotta/install.py", line 63, in installDeps
    available_components = [(c.getName(), c)]
  File "/Library/Python/2.7/site-packages/yotta/lib/component.py", line 426, in satisfyDependenciesRecursive
    provider = provider
  File "/Library/Python/2.7/site-packages/yotta/lib/component.py", line 266, in __getDependenciesRecursiveWithProvider
    provider = provider
  File "/Library/Python/2.7/site-packages/yotta/lib/component.py", line 174, in __getDependenciesWithProvider
    satisfyDep, self.getDependencySpecs(target)
  File "/Library/Python/2.7/site-packages/yotta/lib/component.py", line 167, in satisfyDep
    update_installed
  File "/Library/Python/2.7/site-packages/yotta/lib/component.py", line 413, in provider
    r = access.satisfyVersionByInstalling(name, version_req, working_directory)
  File "/Library/Python/2.7/site-packages/yotta/lib/access.py", line 211, in satisfyVersionByInstalling
    return _satisfyVersionByInstallingVersion(name, version_required, install_into, v)
  File "/Library/Python/2.7/site-packages/yotta/lib/access.py", line 219, in _satisfyVersionByInstallingVersion
    assert(version)
AssertionError

Need login/logout commands

Should be able to do something like:
$ yotta login headlessme
> Enter password:

and

$ yotta logout

Test directory is (accidentally?) installed to site packages.

This is probably just a case of me misunderstanding, but yotta probably shouldn't install the test/ directory to site-packages. Obviously pip knows these files are owned by yotta, but none the less... and it's confusing to have a directory called "test" in site-packages that does not correspond to the Python-internal regression test module (https://docs.python.org/2/library/test.html).

/usr/local/lib/python2.7/site-packages/test$ ls
__init__.py      download.py      hg_access.py     vcs.py
blobs.py         git_access.py    registry.py      versions.py
components.py    github_access.py validation.py

--save flag

If you do

npm install some-package --save

then it patches your package.json to include the new dependency (marked with the ~ semver modifier, like ~0.1.2).

Publishing a package without permissions sends user to login flow

This should probably just print a message saying they don't have access if they are logged in, but direct them to the login flow if they're not. Should be able to tell the different by the response code from the registry (401 vs 403 I think)

When you're in a package, "yotta install" should place in package directory

It should install the same way as a "natural" dependency would.

Must use https to talk to registry (and verify cert).

install/build doesn't warn about missing target-specific dependencies

Login failure needs better error message

Andys-MacBook-Air-2:yotta andy$ yotta login headlessme
Enter the password for github user headlessme:
Traceback (most recent call last):
  File "/usr/local/bin/yotta", line 9, in <module>
    load_entry_point('yotta==0.0.11', 'console_scripts', 'yotta')()
  File "build/bdist.macosx-10.9-x86_64/egg/yotta/main.py", line 87, in main
  File "build/bdist.macosx-10.9-x86_64/egg/yotta/login.py", line 10, in execCommand
  File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/github_access.py", line 126, in authorizeUser
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/restkit/resource.py", line 144, in post
    headers=headers, params_dict=params_dict, **params)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/restkit/resource.py", line 210, in request
    response=resp)
restkit.errors.RequestFailed: {"message":"Validation Failed","documentation_url":"https://developer.github.com/v3/oauth_authorizations/#create-a-new-authorization","errors":[{"resource":"OauthAccess","code":"already_exists","field":"description"}]}

When the target is changed, Yotta should delete the CMake cache.

Signed/authenticated releases

We may eventually need to support a SHA (or similar) checksum in the yotta package metadata, and/or GPG signatures by the "owner". Not a priority right now, but maybe something to consider for the package.json schema.

Dependency version is not enforced for sub-modules

Example, the UIKit module depends on yottos-platform version 0.2.1 or higher, which Yotta enforces with this line in the dependency list:

"yottos-platform": "ARM-RD/yottos-platform >=0.2.1"

However, when UIKit is included in modules that also includes yottos-platform (through other modules) the version requirement is ignored if yottos-platform is already installed.

Workaround:
Manually tweak the dependency list in module.json to install modules in the right order.

Header-only modules cause linker errors in auto generated CMakeLists

(as there is no library to link against)

The best solution for this is probably to auto generate a dummy library when we're auto generating CMakeLists and find no source files.

Static analysis support

Either targets need to always respect the CC/CXX environment variables, so that we can use scan-build to run the clang static analyser (they probably should anyway), or we need to expose another way for targets to provide static analysis support – and yotta should run this by default as part of building, if it is available on a specific target.

Yotta doesn't install properly with Pip

When installing Yotta with Pip, it leaves a "src" directory lying around afterwards, containing "pip-delete-this-directory.txt" and the Yotta source code. If you follow the instructions (which note that the "src" directory is an installation artefact that can be deleted) and go ahead and delete this directory, yotta no longer works. If nothing else, the instructions should be updated to put that directory somewhere sensible and not that you're not actually supposed to follow the deletion instructions.

(I'm using brew python and pip, not system python and manually-installed pip).

`yotta init` should create skeleton CMakeLists

The CMakeLists should pick up new files automatically (use globs).

This will require a question about whether the component is an application or a library – to determine whether the CMakeLists should add_library or add_executable.

Yotta doesn't like it when there are .hg files in a directory

bobbys-air:frdm_gpio 16.49.33 taylorr$ yotta link-target frdm-k64f
info: /Users/taylorr/Desktop/frdm_gpio 16.49.33/yotta_targets/frdm-k64f -> /usr/local/lib/yotta_targets/frdm-k64f -> /Users/taylorr/Code/GitHub/target-frdm-k64f
bobbys-air:frdm_gpio 16.49.33 taylorr$ yotta target frdm-k64f
bobbys-air:frdm_gpio 16.49.33 taylorr$ yotta build
error: The current directory does not contain a valid module.
bobbys-air:frdm_gpio 16.49.33 taylorr$ yotta init
Enter the module name:
Enter the initial version: <0.0.0>
Short description:
Keywords: <>
Author:
Repository url:
Homepage:
What is the license for this project (ISC, MIT, Apache-2 etc.)?
Is this module an executable? yes
Traceback (most recent call last):
File "/usr/local/bin/yotta", line 9, in
load_entry_point('yotta==0.0.28', 'console_scripts', 'yotta')()
File "/usr/local/lib/python2.7/site-packages/yotta/main.py", line 117, in main
status = args.command(args)
File "/usr/local/lib/python2.7/site-packages/yotta/init.py", line 132, in execCommand
c.writeDescription()
File "/usr/local/lib/python2.7/site-packages/yotta/lib/pack.py", line 167, in writeDescription
self.vcs.markForCommit(self.description_filename)
File "/usr/local/lib/python2.7/site-packages/yotta/lib/vcs.py", line 171, in markForCommit
self.repo.hg_add(os.path.join(self.worktree, relative_path))
File "/usr/local/lib/python2.7/site-packages/hgapi/hgapi.py", line 144, in hg_add
self.hg_command("add", filepath)
File "/usr/local/lib/python2.7/site-packages/hgapi/hgapi.py", line 118, in hg_command
return Repo.command(self.path, self._env, *args)
File "/usr/local/lib/python2.7/site-packages/hgapi/hgapi.py", line 89, in command
stdout=PIPE, stderr=PIPE, env=env)
File "/usr/local/Cellar/python/2.7.7_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File "/usr/local/Cellar/python/2.7.7_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

`yotta list`, and make `yotta install` quieter

non-registry specification support for `yotta install <spec>`

Currently only individual access modules know how to parse these specs, that either needs exposing or re-implementing.

need devDependencies / testDependencies for dependencies only required when running tests

Think I prefer testDependencies – the usage is quite different to npm's devDependencies.

`yotta up` doesn't check for updates to the target description

yotta link should highlight if the symlink created is broken.

e.g. instead of printing:

/Volumes/Work/synced/Dev/IoT/objc/libobjc2-svn/yotta_modules/foundation -> /usr/local/lib/yotta_modules/foundation -> /usr/local/lib/yotta_modules/foundation

for a broken symlink, it should print:

/Volumes/Work/synced/Dev/IoT/objc/libobjc2-svn/yotta_modules/foundation -> /usr/local/lib/yotta_modules/foundation -> BROKEN

And why not use colours to make it obvious, too.

Docs link in readme.md is broken

Multiple Client Support?

While I was trying to get yotta work on my Macbook Pro, when I do yt install, it shows this error

Enter the password for github user dorafmon:
Traceback (most recent call last):
File "/usr/local/bin/yt", line 9, in
load_entry_point('yotta==0.0.21', 'console_scripts', 'yt')()
File "build/bdist.macosx-10.9-x86_64/egg/yotta/main.py", line 87, in main
File "build/bdist.macosx-10.9-x86_64/egg/yotta/install.py", line 31, in execCommand
File "build/bdist.macosx-10.9-x86_64/egg/yotta/install.py", line 63, in installDeps
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/component.py", line 438, in satisfyDependenciesRecursive
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/component.py", line 281, in __getDependenciesRecursiveWithProvider
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/component.py", line 189, in __getDependenciesWithProvider
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/component.py", line 182, in satisfyDep
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/component.py", line 427, in provider
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/access.py", line 209, in satisfyVersionByInstalling
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/access.py", line 79, in latestSuitableVersion
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/github_access.py", line 189, in availableVersions
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/github_access.py", line 59, in wrapped
File "build/bdist.macosx-10.9-x86_64/egg/yotta/lib/github_access.py", line 127, in authorizeUser
File "/usr/local/lib/python2.7/site-packages/restkit/resource.py", line 144, in post
headers=headers, params_dict=params_dict, **params)
File "/usr/local/lib/python2.7/site-packages/restkit/resource.py", line 210, in request
response=resp)
restkit.errors.RequestFailed: {"message":"Validation Failed","documentation_url":"https://developer.github.com/v3/oauth_authorizations/#create-a-new-authorization","errors":[{"resource":"OauthAccess","code":"already_exists","field":"description"}]}

I assume this is because I have an Oauth token installed for my Mac mini so I am not allowed to have another one, right?

yotta build should run installl

checking out a fresh project you should be able to do yotta build and it will run the install step to grab dependencies for you

yotta link should install deps before linking

npm does this (and you have to do this for anything to be able to use the link since shared dependencies may be resolved into the yotta_modules of the linked package), so yotta should.

"yotta install" doesn't re-attempt properly on failure

I got an error installing ARM-RD/libobjc2, but when I fixed the underlying issue and re-ran yotta install, it just said

info: [email protected]

instead of attempting re-install (so I guess it thought it was already complete?). I had to delete yotta_modules/ to force it to re-install.

2-factor authentication on Github not supported

Title.

Workaround:

• Create personal access token on Github
• insert in file .yotta/config:

[github]
user = username
authtoken = personal_access_token

Build with debug symbols by default.

There is almost no reason not to, since we don't load debug sections onto the device, and it's a pain when you forget to add -d while building.

Yotta should cache downloads (like NPM)

Don't touch unmodified cmakelists

When yotta writes cmakelists files it shouldn't touch files that haven't changed.

ctest integration

running yotta test should run ctest in the build/$TARGET/test directory

Display friendly error message if cmake is not installed (instead of Errno 2 backtrace)

Linking the right things should be automatic

Possibly components should specify the name of libraries they export in their package description.

yotta could then maintain a YOTTA_LINK_DEPENDENCIES variable with the names of the components that the current component depends on.

Challenge of this is dealing with yotta components that don't export a library at all.

It should be possible to specify github tags and branches in module.json

i.e.: name/repo#branchname name/repo#tag or name/repo@tagname (and possibly name/repo@branchname) instead of just name/repo@version.

This will require versions & specifications that can wrap arbitrary strings when they don't parse as actual semantic versions.

Raw hg url support

yotta list doesn't display all linked dependencies

For example, if a dependency is linked at the top level, but not depended on directly by the top-level component, then list doesn't display it.

Yotta should enforce that a module builds without errors before publishing to the registry

Title.

When doing 'yt publish', Yotta should first install the module candidate in a clean directory and do a 'yt install' and 'yt build' on it. Only if the build is successful should the module be uploaded to the registry.

Raw git url support.

Downloads should be unpacked into a temporary directory then moved into place.

Currently if yotta is interrupted then partially-unpacked modules can be left in place, causing build errors.

OAuth Token already exists

There's a problem when yt install is run on more than one computer when the same github username and password are used, as github complains that OAuth Tokens already exist. I'm not sure whether a fix is possible, and a work around is just to delete the keys in user settings on Github, however it would be nice to just work.

Yotta doesn't like spaces in paths

bobbys-air:frdm_gpio 16.49.33 taylorr$ yotta build
info: generate for target: frdm-k64f 0.0.1 at /Users/taylorr/Desktop/frdm_gpio 16.49.33/yotta_targets/frdm-k64f
CMake Error at /usr/local/Cellar/cmake/3.0.0/share/cmake/Modules/CMakeDetermineSystem.cmake:104 (message):
Could not find toolchain file:
/Users/taylorr/Desktop/frdm_gpio;16.49.33/yotta_targets/frdm-k64f/CMake/toolchain.cmake
Call Stack (most recent call first):
CMakeLists.txt:21 (project)

CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER
CMake Error: Could not find cmake module file: /Users/taylorr/Desktop/frdm_gpio 16.49.33/build/frdm-k64f/CMakeFiles/3.0.0/CMakeCCompiler.cmake
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CXX_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CXX_COMPILER
CMake Error: Could not find cmake module file: /Users/taylorr/Desktop/frdm_gpio 16.49.33/build/frdm-k64f/CMakeFiles/3.0.0/CMakeCXXCompiler.cmake
CMake Error at CMakeLists.txt:21 (project):
No CMAKE_C_COMPILER could be found.

Tell CMake where to find the compiler by setting the CMake cache entry
CMAKE_C_COMPILER to the full path to the compiler, or to the compiler name
if it is in the PATH.

CMake Error at CMakeLists.txt:21 (project):
No CMAKE_CXX_COMPILER could be found.

Tell CMake where to find the compiler by setting the CMake cache entry
CMAKE_CXX_COMPILER to the full path to the compiler, or to the compiler
name if it is in the PATH.

CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
error: command ['cmake', '-D', 'CMAKE_BUILD_TYPE=Debug', '-G', 'Unix Makefiles', '.'] failed
bobbys-air:frdm_gpio 16.49.33 taylorr$ ls /Users/taylorr/Desktop/frdm_gpio;16.49.33/yotta_targets/frdm-k64f/CMake/toolchain.cmake
ls: /Users/taylorr/Desktop/frdm_gpio: No such file or directory
-bash: 16.49.33/yotta_targets/frdm-k64f/CMake/toolchain.cmake: No such file or directory

yotta should provide the version of the component being built to cmake

Propagate arguments to make

It should be possible for 'yotta build' to propagate some command line arguments to 'make' (such as '-j' for parallel builds and 'VERBOSE=1' to see what make is actually executing)

`yotta link` doesn't replace a symlink in the global modules folder if it already exists