meltano/files-superset

Meltano project [file bundle](https://meltano.com/docs/plugins.html#file-bundles) for [Apache Superset](https://superset.apache.org/).

The declarative code-first data integration engine

Say goodbye to writing, maintaining, and scaling your own API integrations.
Unlock 600+ APIs and DBs and realize your wildest data and ML-powered product ideas.

---

Integrations

Meltano Hub is the single source of truth to find any Meltano plugins as well as Singer taps and targets. Users are also able to add more plugins to the Hub and have them immediately discoverable and usable within Meltano. The Hub is lovingly curated by Meltano and the wider Meltano community.

Installation

If you're ready to build your ideal data platform and start running data workflows across multiple tools, start by following the Installation guide to have Meltano up and running in your device.

Documentation

Check out the "Getting Started" guide or find the full documentation at https://docs.meltano.com.

Contributing

Meltano is a truly open-source project, built for and by its community. We happily welcome and encourage your contributions. Start by browsing through our issue tracker to add your ideas to the roadmap. If you're still unsure on what to contribute at the moment, you can always check out the list of open issues labeled as "Accepting Merge Requests".

For more information on how to contribute to Meltano, refer to our contribution guidelines.

Community

We host weekly online events where you can engage with us directly. Check out more information in our Community page.

If you have any questions, want sneak peeks of features or would just like to say hello and network, join our community of over +2,500 data professionals!

👋 Join us on Slack!

Responsible Disclosure Policy

Please refer to the responsible disclosure policy on our website.

License

This code is distributed under the MIT license, see the LICENSE file.

files-superset's Issues

Update readme after transfer to GitLab - [merged]

Merges update_readme -> main

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/merge_requests/1

Metadata.yaml timestamp should be removed

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/1

Originally created by @pnadolny13 on 2022-01-03 15:33:12

---

Created by: pnadolny13

When exporting zip files (charts and databases) a metadata.yaml is included in the bundle and a timestamp field is included based on the export time. It would be better if it only changed when the asset is updated. Maybe we parse those files and determine if anything changed before updating the TS.

Template Schema Names

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/3

Originally created by @pnadolny13 on 2022-01-03 15:41:30

---

Created by: pnadolny13

Schema names are hard coded in the export files. They should be templated so the environment settings can be injected at import time.

Import datasources from dbt - [merged]

Merges import_models_from_dbt -> main

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/merge_requests/2

---

• replace API scripts with CLI scripts
• data source import script using dbt manifest.json
• add ui that runs in the foreground and imports datasources/dashboards automatically
• refresh readme

Consider single-container version of Superset for local devtest use cases

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/6

Originally created by @aaronsteers on 2022-01-27 20:32:24

---

The current implementation takes a docker-compose.yml file which runs several containers.

Our initial spec in https://gitlab.com/meltano/meltano/-/issues/2205+ will expect a single container definition.

Following from the alternate instructions at https://superset.apache.org/docs/installation/installing-superset-from-scratch, what do you think about us having a version of this that runs in a single container? Assumption being that the PostgresDB is launched in a separate process, and does not need to be part of the execution command.

In theory a bootstrap script in the docker container could import the file definitions and then export after execution completes (assuming it gets a graceful shutdown, or assuming ctrl+c is only hit once and enough time is allowed before a hard process kill).

@pnadolny13, @pandemicsyn - Curious as to your thoughts.

Import/Export Scripts CSRF Issue

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/7

Originally created by @pnadolny13 on 2022-02-01 22:33:53

---

With CSRF tokens enabled, which we want, the import/export scripts dont work. This was an issue that other people ran into in slack. Theres a way to retrieve a token but it still doesnt work for me.

Potentially passing the token around got fixed, so look into it.

Or explore using the CLI instead of the API for importing/exporting to avoid it.

Inject Passwords Into Databases

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/2

Originally created by @pnadolny13 on 2022-01-03 15:37:09

---

Created by: pnadolny13

When databases are exported it masks the password but when importing the passwords parameter isnt respected so imported databases need to be manually changed in the UI.

Database Imports Fail

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/8

Originally created by @pnadolny13 on 2022-02-01 22:35:52

---

Importing databases worked at one point but now fail every time I try. Instead I just manually create my database, comment out the database import line in the script, then it works.

Either make it a requirement to manually add databases before an import and remove the database method or get it fixed somehow.

Consider `superset:ui` as (blocking) version of `superset:up` and `superset:down`

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/5

Originally created by @aaronsteers on 2022-01-27 19:27:25

---

@pnadolny13 - In the meltano project, we considered a generic ui command for any utility that has a web UI. As we discussed it, that would be a blocking command, which when killed would also kill the UI.

I see here in this package we have up and down as commands. What do you think about feasibility and practicality of having ui as a command - either in place of or in addition to the up and down commands?

Linked issues:

• https://gitlab.com/meltano/meltano/-/issues/2606+
• https://gitlab.com/meltano/meltano/-/issues/2944+

Import Details Should Be Removed

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/4

Originally created by @pnadolny13 on 2022-01-03 15:43:49

---

Created by: pnadolny13

For yamls like charts a section is added with import details after someone loads them into their local superset instance so the git diff gets noisy. Import details should be excluded from the files so if a chart is unchanged it shouldnt have a git diff after exporting.

Consider access to localhost for database connections

Migrated from GitLab: https://gitlab.com/meltano/files-superset/-/issues/9

Originally created by @andrewstewart on 2022-03-17 20:11:42

---

For local dev/test purposes, it could be useful if the database connections in superset could access databases running on the localhost (the docker host, not the container itself, just to clarify).