apache/openwhisk-catalog

Curated catalog of Apache OpenWhisk packages to interface with event producers and consumers

Openwhisk Catalog

This openwhisk-catalog maintains the package catalogs of openwhisk. In OpenWhisk, the catalog of packages gives you an easy way to enhance your app with useful capabilities, and to access external services in the ecosystem. Examples of external services that are OpenWhisk-enabled include the Slack and GitHub.system packages and sample packages.

The catalog is available as packages in the /whisk.system namespace. See Browsing packages for information about how to browse the catalog by using the command line tool.

How to install openWhisk-catalog

Pre-requisites

• openwhisk is installed.
• Environment variable OPENWHISK_HOME is configured as the path to openwhisk source code directory.

Install openwhisk-catalog

We should be able to run the script packages/installCatalogUsingWskdeploy.sh to install the catalog like:

./packages/installCatalogUsingWskdeploy.sh [catalog_auth_key] [api_host] [cli_path]

The first argument catalog_auth_key, defines the secret key used to authenticate the openwhisk service. The second argument api_host, determines the location, where the openwhisk edge host is running, in the format of IP or hostname. The third argument cli_path is the full path to the wsk cli executable.

Existing packages in catalog

For more details about how to use packages in the catalog, you can go to the README.md under each package subfolder.

Package	Description
/whisk.system/github	offers a convenient way to use the GitHub APIs.
/whisk.system/slack	offers a convenient way to use the Slack APIs.
/whisk.system/websocket	Package to send messages to Web Socket server

How to create top-level packages

If your package is more involved you may want to create your own top-level packages repository using the openwhisk-package-template. It will provide a good example on what a good template should include (i.e., sufficient help build, test and integrate into OpenWhisk).

openwhisk-catalog's Issues

move hellopromise from openwhisk/catalog to openwhisk-catalog

hellopromise is recently added, it depends on greeting.

installGit.sh failed

I try to run command "ansible-playbook postdeploy.yml " but got error as follows.

open catalog ERRORS = 6
/home/vagrant/openwhisk-catalog/packages/installGit.sh: line 16: -a: command not found

I found that need to modify installGit.sh to add escape character \ in the end of 15th line like this

createPackage github
-p endpoint "openwhisk.ng.bluemix.net"
-a description "Package which contains actions and feeds to interact with Github"

Please check and verify this bug.

Thank you.
yoshitan

Add test cases for github

This targets for webhook

Move "slack" package from openwhisk to openwhisk-catalog

@houshengbo

Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…

Issue #4 is a prereq.

Move sample "slack" in current repo./directory:

• /openwhisk-openwhisk/catalog/slack

to the "openwhisk-catalog" repo. under directory structure:

• /openwhisk-catalog/packages/slack

including action files

• post.js

Update install script (in /openwhisk-catalog/packages directory, after Issue #4 is completed) :

• installSlack.sh

New package for the catalog, like websocket plus test cases should come to this repo

websocket is something new after the catalog movement is done, so it should be put into this repo, wih test cases.

This also applies to any new package for catalog.

make openwhisk-catalog repo support packages or tests cases involved with credentials.

Now if we want to move watson packages and corresponding test cases under this repo, we need some adjustment in our repo build script to make our test could read the credentials files.

We could reference this repo.
https://github.com/openwhisk/openwhisk-package-template

Github hooks appear to not be deleting when a trigger is deleted

Recently I was running the UI tests and one of the tests started failing. It was trying to create a Github trigger with the "push" event. I kept getting a 502 error back.

I tried another account and everything worked fine. I then tried to use the "delete" event and it worked.

So I was told to run a curl command against the hooks endpoint to see what hooks existed against that particular account and I saw a bunch with the "push" event and the one I created with the delete event.

I then tried to do a wsk trigger delete of the trigger with the "delete" event and when I inspected that hooks endpoint again I still saw the hook for it present.

I ran wsk activation get ## against the activation id that was returned in the wsk trigger delete command and saw the following:

"result": {
            "error": "Found no existing webhooks for trigger URL https://9aac5786-5cd8-4eed-bde3-8d0b202ccb89:[e-mail]/api/v1/namespaces/eweiter%40us.ibm.com_dev/triggers/myGitTrigger"
        }

OPENWHISK_HOME env variable is not recongnized in mac system

OPENWHISK_HOME is retrieved in the way like
def whiskhome = "$System.env.OPENWHISK_HOME"

in the gradle.build.

It works for linux, but not mac.
windows needs to be verified as well.

Change test cases for slack from java to scala

this is for post.js

Move install scripts and util.sh from openwhisk to openwhisk-catalog

@houshengbo

Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…

Note: This issue is pre-req for other issues for "move" of catalog.

Please note that other issues (e.g., Issue 2, Issue 3) which are moving "utils" and "samples" package actions to new locations (and adjusting namespaces) will be updating these files. It may be a good idea to address (i.e., commit and merge) this issue before other issues titled "Move".

Please move the following files from the current repo. / directory "openwhisk-openwhisk/catalog" to this repo. under "openwhisk-catalog/packages":

• installCatalog.sh
• installGit.sh
• installSlack.sh
• installSystem.sh
• installWatson.sh
• installWeather.sh
• util.sh

add field level descriptions and doclink to github webhook

in addition to adding some simple but helpful field-level descriptions. i suggest that we pioneer here a doclink field. this will be particularly helpful for the accessToken and events fields, which are quite cryptic on their own.

also, the main description field inaccurately renders GitHub as "github".

compare with: apache/openwhisk#1102

Catalog travis CI failed

See https://travis-ci.org/openwhisk/openwhisk-catalog

Move the websocket from openwhisk/catalog to openwhisk-catalog

websocket is newly added in openwhisk, need to move this package.

Current build.gradle doesn't work for IDE environment

According to PR #68 comment, current build.gradle doesn't work for Eclipse. We shall improve it in the future.

Openwhisk and openwhisk-catalog sync issue.

I noticed people are still updating packages in openwhisk repo, for example this one:
https://github.com/openwhisk/openwhisk/pull/948/files#diff-1

How shall we keep repo openwhisk-catalog synced with openwhisk, or we ask people to contribute
their updates here? To avoid we maintained a obsolete one?

@houshengbo @daisy-ycguo @mrutkows @csantanapr

Dependency of whisk.properties and auth.whisk.system in OpenWhisk

Although the install scripts are moved in openwhisk-catalog, installCatalog.sh still needs two file from OpenWhisk to launch the installation. One is the whisk property file located at $OPENWHISK_HOME/whisk.properties, the other is the system authentication file located at $OPENWHISK_HOME/config/keys/auth.whisk.system.
Each time to run the install script, we need to specify these two files as the inputs. Currently we pass in the authentication file auth.whisk.system and the path of $OPENWHISK_HOME.
Are we going to leave it this way or give the files to openwhisk-catalog? Need an agreement.

Circular dependency of openwhisk and openwhisk-catalog

Since the packages and install scripts are moved into openwhisk-catalog, how shall we cope with the dependencies between openwhisk and openwhisk-catalog?

If we still keep an install script in openwhisk, each time we run it, it git clone the openwhisk-catalog repo and install the packages.
If we run the install script in openwhisk-catalog, it will clone the openwhisk repo and install the packages.

This situation sustains in the Travis CI as well. openwhisk and openwhisk-catalog both need Travis CI to run the tests on each PRs to ensure the code quality. However, each of them needs one another's repo to install the catalog.

We need to break the circular dependency in either of the projects.

Installing catalog is ready, how about uninstalling the catalog

I personally believe this is necessary, from both users' and developers' perspective.
It does not make sense to allow folks to install only, but not the opposite.

This may extend to a broad life cycle management of all the actions and packages.

Switch from python client to go client.

This two pull request.

apache/openwhisk#882
apache/openwhisk#883

slack.post should support passing through access token

right now, slack.post only supports the incoming webhook style of posting a slack message. in order to support the chat.postMessage API https://api.slack.com/methods/chat.postMessage, we will need to update slack.post in order to pass through a token field (the access token, probably acquired via a separate oauth exchange)

the rest of slack.post should handle the webhook-versus-postMessage distinction with grace. observe that both protocols require channel and text parameters, and both taken an optional icon_emoji parameter. i.e. the only distinction between the two is username versus token.

parallels the main repo issue apache/openwhisk#1091

Wordcount needs test cases for creation in nodejs and nodejs6 containers

Move "weather" package from openwhisk to openwhisk-catalog

@houshengbo

OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx

Issue #4 is a prereq.

Move sample "weather" in current repo./directory:

• /openwhisk-openwhisk/catalog/weather

to the "openwhisk-catalog" repo. under directory structure:

• /openwhisk-catalog/packages/weather

including action files

• forecast.js

Update install script (in /openwhisk-catalog/packages directory, after Issue #4 is completed) :

• installWeather.sh

Please note, there 2 other places where we have "weather" related actions/feeds. This issue only seeks to move the forecast.js for now. Other issues will be opened to address these other actions/feeds later.

Is it allowed to depend on binaries, which are pre-built in openwhisk

Currently this openwhisk-catalog gradle build is dependent on openwhisk binaries as it is written in build.gradle:

def test_common = whiskhome + '/tests/build/classes/test'
...
...
...
then:  testCompile files(test_common)

Since openwhisk-catalog needs to be installed on openwhisk. Each time users want to build openwhisk-catalog, they have to manually build the openwhisk binaries first prior to build the openwhisk-catalog. Is this something we can accept in openwhisk-catalog?

What is the optimal way to resolve the build dependencies on another project for gradle?

Make sure newly added test code and legacy tests are converted from java to scala.

There are now some testing code available for Java on BlueWhisk repo, we need to convert them to Scala. For newly added test code, let's also use scala instead of java.

Move /whisk.system/util from openwhisk to openwhisk-catalog as /whisk.system/utils (plural)

@houshengbo

OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx

Issue #4 is a prereq.

Please note that as part of this move, there are 3 special changes that need to be made apart from the move of the action files.

1. The current JavaScript (JS) actions in the "openwhisk-openwhisk" repo. are installed into the catalog with the namespace "/whisk.system/util" which is incorrect. It should be plural "utils" with an "s". Please fix the install script "installSystem.sh" to reflect this name change.
2. The action file "pipe.js" is not supposed to be in the "utils" directory; it is actually installed (by "installSystem.sh) into the "/whisk.system/system" namespace. As part of this move, we will need to create a "system" directory under the top-level "packages" directory with the name "system" and update the "installSystem.sh" script to look for it there.
3. This issue should move the " openwhisk/catalog/installSystem.sh" script to the "openwhisk-catalog/packages/" directory (with the updates mentioned above).

The action files affected by this issue currently reside under repo. "openwhisk-openwhisk" in the "/openwhisk/catalog/utils". They will now live in the "openwhisk-catalog" repo. in the "openwhisk-catalog/packages/utils" directory; these include:

• cat.js
• date.js
• head.js
• sort.js

except for "pipe.js" which will reside in the "openwhisk-catalog/packages/system" directory.

Add the test cases for printparams

This is for printparams.js

Add the test cases for countdown

countdown.js is the target action

Move /whisk.system/samples from openwhisk to openwhisk-catalog

@houshengbo

Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…

Issue #4 is a prereq.

In general, we are are moving actions that appear in the "/whisk.system/samples" namespace form the current repo. "openwhisk-openwhisk" to this repo. "openwhisk-catalog".

By convention, we will:

• Create top-level “package” named directories (1-per logical sample, same functional sample)
• For each unique language implementation for sample package, create a subdirectory named for the language in which that language impl. will reside (e.g., “javascript”, "swift", “python”, java”, etc.)
  • so it would look like: openwhisk/catalog/packages/samples/ <action name> / <language name> / <xxx>.<language extension>
  • for example: "openwhisk/catalog/packages/samples/hello/swift/hello.swift"

Specifically, create known sample directories (and language specific subdirs.) for the following packages:

• “countdown”
  • create “javascript” -> “countdown.js”
  • Rodric indicates code is not working properly but could and needs to be fixed.
• "curl"
  • create “javascript” -> “curl.js”
  • May need testing/fixing?
• "echo"
  • move echo.js to /whisk.system/utils package, remove from “samples” directory, move to “utils” directory.
  • move catalog install logic to change package namespace and directory locations.
• "greeting"
  • create “javascript” - > “echo.js”
• “hello”
  • Needs subdirs, for javascript, python, java, swift (others?)
  • rename impl. files for actions/feeds to remove any language references (e.g., “helloJava.java” should ne just “hello.java”) as a new convention
  • Python “hello.py” would come from the "blue" sample named “helloPython.py”…
  • we will do this on a different issue # after this initial move.
• “helloAsync”
  • create “javascript” -> “helloAsync.js”
• “httpGet”
  • create subdir. “swift” to host “httpGet.js”
• “invoke”
  • create subdir “swift” to host invoke.swift
• "printParams"
  • create “javascript” - > “printParams.js”
• "trigger"
  • create ”swift” -> “trigger.swift”
  • Note: may need testing/fixing
• "wordcount"
  • create “javascript” - > “wc.js”
  • rename wc.js to wordcount.js
  • move wcbin.js to wordcount javascript directory
  • merge wcbin.js (i.e., wordcount binary) into wordcount.js
  • provide new optional parameter to return binary result

Please also move the following actions currently in "openwhisk-openwhisk/catalog/utils" to their own directories under "openwhisk-catalog/packages/samples/<action name>".

• cat.swift
• head.swift
• split.swift

Be sure to coordinate "delete" from the current repo. once these are moved (see Issue #2) as this move is noted there is well.

Move "watson" package from openwhisk to openwhisk-catalog

@houshengbo

Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…

Issue #4 is a prereq.

Move sample "watson" in current repo./directory:

• /openwhisk-openwhisk/catalog/watson

to the "openwhisk-catalog" repo. under directory structure:

• /openwhisk-catalog/packages/watson

including action files

• languageId.js
• speechToText.js
• textToSpeech.js
• translate.js

Update install script (in /openwhisk-catalog/packages directory, after Issue #4 is completed) :

• installWatson.sh

Update contributing.md

1. to clarify the contribution process
2. to update dev steps based on the update of PR #68.
3. to document IDE has problem

Add test cases for weather

This is for forecast

Add test cases for watson

This is for all the js under watson folder.

translate action not working with new bluemix service Language Translator

The language translation is deprecated, there is a new service language translate service

Using the new service the action doesn't work.
We should update the translate action to work with the new service

Create a new service in Bluemix using the new service "Language Translator"

Bind the package with the username and password

wsk package bind /whisk.system/watson myWatson --param username MY_USERNAME --param password MY_PASSWORD

Then try to run the action

$ wsk action invoke myWatson/translate --blocking --result --param payload "Blue skies ahead" --param translateParam payload --param translateFrom en --param translateTo fr
error: Unable to invoke action 'myWatson/translate': Request failure: Request failed (status code = 502). Error details: [123 10 32 32 34 110 97 109 101 34 58 32 34 116 114 97 110 115 108 97 116 101 34 44 10 32 32 34 115 117 98 106 101 99 116 34 58 32 34 99 115 97 110 116 97 110 97 64 117 115 46 105 98 109 46 99 111 109 34 44 10 32 32 34 97 99 116 105 118 97 116 105 111 110 73 100 34 58 32 34 55 51 49 57 50 53 48 52 102 54 55 102 52 99 102 56 97 102 48 54 50 51 51 101 54 98 49 55 102 53 48 101 34 44 10 32 32 34 112 117 98 108 105 115 104 34 58 32 102 97 108 115 101 44 10 32 32 34 97 110 110 111 116 97 116 105 111 110 115 34 58 32 91 123 10 32 32 32 32 34 107 101 121 34 58 32 34 108 105 109 105 116 115 34 44 10 32 32 32 32 34 118 97 108 117 101 34 58 32 123 10 32 32 32 32 32 32 34 116 105 109 101 111 117 116 34 58 32 54 48 48 48 48 44 10 32 32 32 32 32 32 34 109 101 109 111 114 121 34 58 32 50 53 54 44 10 32 32 32 32 32 32 34 108 111 103 115 34 58 32 49 48 10 32 32 32 32 125 10 32 32 125 44 32 123 10 32 32 32 32 34 107 101 121 34 58 32 34 112 97 116 104 34 44 10 32 32 32 32 34 118 97 108 117 101 34 58 32 34 119 104 105 115 107 46 115 121 115 116 101 109 47 119 97 116 115 111 110 47 116 114 97 110 115 108 97 116 101 34 10 32 32 125 93 44 10 32 32 34 118 101 114 115 105 111 110 34 58 32 34 48 46 48 46 55 55 34 44 10 32 32 34 114 101 115 112 111 110 115 101 34 58 32 123 10 32 32 32 32 34 114 101 115 117 108 116 34 58 32 123 10 32 32 32 32 32 32 34 101 114 114 111 114 34 58 32 123 10 32 32 32 32 32 32 32 32 34 99 111 100 101 34 58 32 52 48 49 44 10 32 32 32 32 32 32 32 32 34 101 114 114 111 114 34 58 32 34 85 110 97 117 116 104 111 114 105 122 101 100 58 32 65 99 99 101 115 115 32 105 115 32 100 101 110 105 101 100 32 100 117 101 32 116 111 32 105 110 118 97 108 105 100 32 99 114 101 100 101 110 116 105 97 108 115 34 10 32 32 32 32 32 32 125 10 32 32 32 32 125 44 10 32 32 32 32 34 115 117 99 99 101 115 115 34 58 32 102 97 108 115 101 44 10 32 32 32 32 34 115 116 97 116 117 115 34 58 32 34 97 112 112 108 105 99 97 116 105 111 110 32 101 114 114 111 114 34 10 32 32 125 44 10 32 32 34 101 110 100 34 58 32 49 52 55 53 56 48 55 55 55 48 54 56 49 44 10 32 32 34 108 111 103 115 34 58 32 91 93 44 10 32 32 34 115 116 97 114 116 34 58 32 49 52 55 53 56 48 55 55 55 48 52 52 48 44 10 32 32 34 110 97 109 101 115 112 97 99 101 34 58 32 34 99 115 97 110 116 97 110 97 64 117 115 46 105 98 109 46 99 111 109 34 10 125]

Discussion: test files structure

The test codes shall be kept in folder /tests/src. And every package shall have enough test files. A good practice is to have a test file for a single source file. But it's not a mandatory rule. If you want to put all test codes of a package in a single file, that's fine. So our tests shall look like

tests/src/package
                            - samples
                                    |- GreetingTest
                                    |- CatTest
                                    |- CountdownTest
                                    |- CurlTest
                                    |- ...
                            - github
                                    |- WebhookTest
                            - watson
                                    |- LanguageIDTest
                                    |- SpeechtoTextTest
                            - utils
                                    |- CatTest
                                    |- DateTest

Here is a question and I'd like to get your inputs:

1. shall we keep "package" folder ? or shall we just remove it ?
   My opinion is to keep it, just in order to keep the consistency of openwhisk project.

Please input your opinion to the file structure. If we all agree, let's use it as a rule and write it down in CONTRIBUTING.md file.

github package should specify accessToken field as type:password

Issues moved from openwhisk repo apache/openwhisk#698

Original open by @starpit
In general, any protected/confidential fields should be marked with type:password, so that the UI can occlude entered text ******

README.md should have content about catalog

Take the content from catalog.md located here https://github.com/openwhisk/openwhisk/blob/master/docs/catalog.md

And paste in README.md but only the parts that are located in this repo

• weather
• watson
• slack
• github
• new section for websockets
• new section describing the packages utils and samples
  this should be very generic short description of the action and the parameters

Update the Github repository param documentation

When browsing the documentation for how to use a Github package binding and a trigger set up for the push event, a user will see the following information
"wsk package bind /whisk.system/github myGit --param username myGitUser --param repository myGitRepo --param accessToken aaaaa1111a1a1a1a1a111111aaaaaa1111aa1a1a"

The "--param repository ..." is too vague and not clear in what needs to be entered. Some users may think that they need to enter an entire url when really it is just the Github org/repo.

Discussion: files structure of a single package

I think, we can put all related files of a single package in the package folder ( test files are not included ).
A package folder shall contain:

• source code files
• README.md to describe actions/feeds in the package and how to invoke them
• install.sh to install the package

If we agree with this proposal, we can:

• move installCatalog.sh to the root folder
• move installXXX.sh to each package folder.
• make installCatalog.sh more dynamic, calling every install.sh under the sub folder of openwhisk-catalog/packages, other than hard coding the file name of every package install script.

Daisy

Split the watson package into three with one package per service

We currently support three different Watson services and currently install all of them under one Watson package. The credentials (Username and password) are bound to the package. However, each Watson service has different credentials so we need to create three unique packages. Listed below are the three services we currently support and the actions belonging to each.

watson-translate/
    translate
    languageid
watson-speechToText/
    speechToText
watson-textToSpeech/
    TextToSpeech

Change the name of the tests in openwhisk-catalog to remove naming conflicts

The tests under openwhisk-catalog is also named tests, the same to to the test under openwhisk.
This will lead to naming conflicts, if openwhisk and openwhisk-catalog are located and imported in the same workspace.

hello needs test cases for creation in nodejs and nodejs6 containers

General directions: 1. create new test cases for creation nodejs and nodejs6? 2. adding new test if missing?

These issues we are facing now is:

1. Many test cases in openwhisk/tests/src/packages/UtilsTests.scala . It seems we have got all the test cases regarding the "kind: nodejs and nodejs6". Some action like greeting in samples did not have the test cases for "kind:nodejs and nodejs 6". Shall we also modify the greeting test cases by adding the tests for "kind:nodejs and nodejs 6"?
2. Another thing is that there are test cases missing in openwhisk for many sample actions like countdown, wordcount, printparam, etc. Shall we add them during the openwhisk-catalog transition this time? Or we can leave them still empty and add them later?

@rabbah @markusthoemmes I need your folks' comments on this.

@daisy-ycguo @daisy-ycguo

Move the test case for greeting in sample from openwhisk to openwhisk-catalog

Move the greeting in sample from openwhisk to openwhisk-catalog:

• Move tests/src/packages/samples/GreetingTests.java from openwhisk to openwhisk-catalog

Add the test cases for helloasync

target at helloasync.js

Move "github" package from openwhisk to openwhisk-catalog

@houshengbo

Uploading OpenWhisk-Repo-Proposal-2016-06-14 - Open Catalog only.pptx…

Issue #4 is a prereq.

Move sample "github" in current repo./directory:

• /openwhisk-openwhisk/catalog/github

to the "openwhisk-catalog" repo. under directory structure:

• /openwhisk-catalog/packages/github

including action files

• webhook.js

Update install script (in /openwhisk-catalog/packages directory, after Issue #4 is completed) :

• installGit.sh

Consolidate greeting, echo and hello, after all the test cases are moved.

Three actions greeting, echo and hello are similar, we can consolidate all of them into one after all the test cases work is done.

Add the the Travis CI to test the PR

We need travis CI to test future PRs.
It should include git clone the openwhisk repo, run the installCatalog.sh against the new PR, and check if the result is good or not.

Add test cases for utils

Add test cases for openwhisk-catalog/packages/utils

Document how to write automation test

There is no documents about automation test code. We need to document it.

Delete template.json or Add documentation for the package-template repo.

After discussed with Daisy, she thoughts might we don't have to put the template_credentials.json in our repo, we need to add in our documentation to let user read repo https://github.com/openwhisk/openwhisk-package-template
The repo also introduced the encrypt and decrypt stuff.
So we need to add a link in our doc.