pipedreamhq / pipedream Goto Github PK
View Code? Open in Web Editor NEWConnect APIs, remarkably fast. Free for developers.
Home Page: https://pipedream.com
License: Other
Connect APIs, remarkably fast. Free for developers.
Home Page: https://pipedream.com
License: Other
It might make sense to separate the streams of output, for example in { stdout, stderr }.
Puppeteer provides a Node API to drive a Chrome headless browser. This allows you to programmatically visit sites, take screenshots, and more.
PIpedream currently doesn't support Puppeteer, but I'd like to run workflows that use it.
I'd like to programmatically create a workflow via API, passing the relevant data (title, description, trigger, steps, etc.) as parameters.
I'd like to access a log of all workflow executions (my event history) or errors so I can run analysis on them and observe patterns. The log would need to contain metadata like:
timestamp of execution
duration of workflow execution
success / failure state
workflow id
deployment id
(any other relevant data from steps.trigger.context
)
timestamp of error (if error)
error object (if present)
I'd like a status page for pipedream.com services so I can monitor the health of the service and understand if there's a system issue affecting my workflow.
I want to run a workflow every time a new row is added to a Google Spreadsheet
Some possible game APIs to integrate against:
We'd love feedback on others that you'd find helpful.
I'd like to run a set of steps or actions within a loop construct. For example, I might receive an array of 100 events in my HTTP payload, and I need to run an action on each. I can do this with code, but I have no way to loop through the elements of this array using Pipedream's actions. I'd like a "loop" action that can handle this use case.
I star repos on Github to 1) give kudos to the author and 2) bookmark the repo so I can find it later. I'd like a similar feature on Pipedream.
I'd like to send data to Amplitude using the API documented here: https://help.amplitude.com/hc/en-us/articles/360032842391-HTTP-API-V2
Currenty, I can require()
an npm package within a code step, but I cannot specify the version of the package I'd like to use. I'd like support for this.
I might enable a large stream of events via webhook, for example, that hit my workflow. But I might only need to process a subset of the event types. I'd like a way to filter those events before my workflow runs via some set of rules.
I can add a code step to perform this filtering today, but my workflow still executes for every event. I'd like my workflow to execute only on the events I care about.
e.g. via the CLI, if I have a .env
file in a local project, I'd like Pipedream to read and apply the values of those environment variables, using them for the relevant workflow.
I want to connect to the SQL Service and run queries using a JDBC interface.
I'd like to author TypeScript code steps on Pipedream, similar to how I can choose to "Run Node.js code" today.
I'd like a way to configure automatic retries for a workflow so that, if it throws an error, I can run it again N number of times with a specified delay.
I need a way to store data in Pipedream in one workflow and read it in another. Currently, data saved using $checkpoint
is scoped to a workflow, so I don't have an easy way to share that data across workflows.
To make this useful, the delay couldn't count against the execution time limit. The workflow would have to suspend during the delay, and resume when the delay is done.
I'd like a way to group related workflows in the Pipedream UI, for example by adding a tag to workflows or putting them into folders.
I'd like a way to upload a file that I can reference in a workflow, for example a machine learning model I've trained offline and want to load into Pipedream. Or I might have a simple text file I need to reference as metadata in my workflow.
Instead of making HTTP requests to https://myendpoint.m.pipedream.net
, I'd like to couple a custom domain to a workflow and make requests to https://my.domain.com
, instead.
In code steps, I'm presented with an auto-complete menu that gives me the property at the next "level" in a given object, e.g. I type steps.
and get the properties of steps
that I can select from:
Selecting one of these properties presents yet another "level" of properties, if there are more to show, and I can continue until I select the value I want.
I like this form of auto-complete and would like it in actions forms, as well. I still like the sample values presented next to the values in the "action-style" autocomplete, so ideally I'd like to keep those, too.
When a workflow needs a long debugging stage, it is highly probable that it will accumulate a lot of log entries. At the moment such entries can be deleted only one by one.
It would be nice to have the option to delete all entries realtive to a day (an [x] button near the labels days ago)
See attached screenshot
Because my workflows are getting bigger and bigger, I ended up with lot of environment variables (API keys, external services end-points urls, and so on).
I would suggest, within "Settings" section, to have such env variables sorted by name (ascending and/or descending). Because it is getting really hard to find the right env variable while I'm writing new code, it would be really nice to have an arrow icon (up/down) that implements the sorting, like in the workflows list.
Many thanks!
For HTTP or Email triggers, anyone who knows the destination URL or address of the workflow can send events to it. I can manually add validation in a code step to ensure workflow events come from some source (e.g. only run the workflow when a header is set to a specific value), but I'd like a built-in way to restrict how a workflow is triggered, via API key or some other mechanism.
Some workflows trigger errors with a high frequency, since they hit unreliable APIs. Or I might receive messy source data, which triggers errors because it's not of the expected format.
I already get notified of these errors, but when I visit my workflow, I need to quickly filter on errors so I can see them in one place. This lets me easily replay them one by one.
I'd like a way to search through and discover all public workflows on Pipedream.
I'd like to send AMQP messages to a workflow.
I'd like to run workflows on Gmail events, e.g. when a new email arrives in my Inbox.
Currently, workflows are a linear sequence of steps. I'd like support to branch a workflow in a conditional manner (if condition is true, run this step, else run this other step).
I'd like to trigger a workflow from another workflow via a built-in action.
Workaround: I can create an HTTP trigger for the second workflow and send an HTTP request from the first, but this isn't as clean.
Describe the bug
When executing :w
to save in Vim mode, the app loses focus on the step I'm currently editing. I have to click into the step again to start editing.
To Reproduce
Steps to reproduce the behavior:
:w
Expected behavior
Ideally, I'd like to keep the browser's focus in the editor for the current step.
Desktop (please complete the following information):
This doc articulates what's public and what's private in a workflow, but as a new user this isn't clear to me when I'm using the product. I'd like more visual clarity on this.
Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
Just like Pipedream exposes a way to "Run Node.js code", I'd like a way to "Run Python code". I'd expect support for Python 3 only since Python 2 will be EOL in 2020.
Do you have a workaround?
I need to use specific packages like pandas
for which there is no npm equivalent, so even though I can write Node.js for certain things, I need Python.
I want to write and test workflows on my local machine, storing the workflow's code in my existing version control system.
I also want to programmatically deploy workflows using a CLI, for example using a command like pipedream deploy
. This way, I can deploy workflows locally or as part of my CI/CD pipeline.
Related: #1139
When my workflow yields an error, sometimes it's because my source data was messy. I want to modify the event payload slightly before replaying that event through my workflow.
Other tools โ for example integrator.io โ have an optimized flow for this use case.
As a workaround, I can edit the test event for my workflow, add the modified payload there, and send the event through the workflow. But this is time consuming when I get a lot of errors in my workflow.
Similar to how I can "Run Node.js code" in a step, I'd like to "Run Golang code" so I can write Go on Pipedream.
That is: if I'm pre-processing data in one workflow, then sending the data to another workflow, I'd like to see how they're logically connected (data flows from A -> B).
For example, I'd like to query a Postgres DB from a workflow, and I need to add a network ACL to expose port 5432 to a limited set of IPs. So I'd like to know the IP address(es) from which the request will come.
Ideally, this would be provided in a publicly-accessible data exchange format (for example, AWS provides theirs as a JSON doc), so I can update firewall or network rules programmatically.
I'd like to see common Slack events as triggers, for example I want to trigger a workflow every time a new message is posted to a channel.
Right now the list of actions can be quite long for some apps, and I don't have a way to open that list in fullscreen mode like I do for editing code. I'd love to expand the app / action search list so it's easier to see everything in one place.
Is your feature request related to a problem? Please describe.
I want to rename a workflow during the forking process.
Describe the solution you'd like
After hitting "fork this workflow" I'd expect to see a modal with a text field that allows me to either enter a new name or modify the existing workflow name (potentially with "copy" appended to the end or something similar.
Do you have a workaround?
Forking then renaming afterward.
Comparable features in other tools?
Yea probably somewhere, but I don't have a reference handy : )
Additional context
Add any other context or screenshots about the feature request here.
Currently, workflows are a linear sequence of steps. I'd like a way to spawn N instances of a step, each of which operates on a subset of the input data.
For example, if I have 100 URLs and want to make HTTP requests to each, I have to write a Node.js code step using something like Promise.all
to make those requests in parallel. I'd prefer to use the Send HTTP Request action, feeding in each of the 100 URLs to that action, and run them in parallel.
I also need a way to process the results of these parallel operations once each of the tasks has completed.
Think: map / reduce semantics, or something like JavaScripts Promise.all
.
I'd like to get a regular email with a summary of my workflow executions, similar to Zapier or other services.
Like the Global Error Workflow, it'd be great if this were a workflow I could modify, so that I could send the summary to Slack or some other destination, or modify the data sent in the summary.
"If I change the name of a step, I need to manually change it if I used it in other steps"
I'd like to run Gmail actions as workflow steps, e.g. I'd like to send a new email or attach a label to an email using Pipedream.
Currently, Pipedream requests broad OAuth scopes for many apps so that, upon connecting your account, you're able to perform any of our listed actions.
But sometimes I just want to perform a very narrow task, and allow Pipedream access to a subset of the listed scopes.
Other times, I might want to authorize one step of a workflow with access to one scope, and another step access to another, so I can tightly control what steps have access to what auth. This is especially important if I'm running a third party npm package that interacts with a service, and I want to carefully control the access that code has to my account.
Overall, some mechanism for selecting my own scopes when I authorize a connected account in Pipedream feels like it would solve these use cases, but I'm open to any other ideas.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.