simonw/datasette-app-support

Part of https://github.com/simonw/datasette-app

datasette-app-support

Part of https://github.com/simonw/datasette-app

Installation

Install this plugin in the same environment as Datasette.

$ datasette install datasette-app-support

Using this outside of the context of Datasette.app probably won't work.

API endpoints

This plugin exposes APIs that are called by the Electron wrapper.

All plugins are protected by authentication: they need to be called with a Authorization: Bearer xxx token here the xxx matches the value of the DATASETTE_API_TOKEN environment variable.

/-/auth-app-user

POST /-/auth-app-user
{"redirect": "/-/metadata"}

If a valid Authorization header is passed, sets a signed cookie identifying the user as {"id": "admin"} and redirects them to the specified page.

/-/open-database-file

POST /-/open-database-file
{"path": "/path/to/file.db"}

Attaches a new database file to the running Datasette instance - used by the "Open Database..." menu option.

Returns HTTP 200 status with {"ok": True, "path": "/file"} if it works, 400 with an "error" JSON string message if it fails.

/-/new-empty-database-file

POST /-/new-empty-database-file
{"path": "/path/to/file.db"}

Creates a brand new empty SQLite database file at the specified path and attaches it to the Datasette instance. Used by the "Create Empty Database..." menu option.

Returns HTTP 200 status with {"ok": True, "path": "/file"} if it works, 400 with an "error" JSON string message if it fails.

/-/open-csv-file

POST /-/open-csv-file
{"path": "/path/to/file.csv"}

Imports a CSV or TSV file into the default /temporary in-memory database. Used by the "Open CSV..." menu option.

Returns HTTP 200 status with {"ok": True, "path": "/temporary/table"} if it works, 400 or 500 with an "error" JSON string message if it fails.

/-/import-csv-file

POST /-/import-csv-file
{"path": "/path/to/file.csv", "database": "database_name"}

Permanently imports a CSV or TSV file into the specified database. Used by the "Import CSV..." menu option.

Returns HTTP 200 status with {"ok": True, "path": "/database_name/table"} if it works, 400 or 500 with an "error" JSON string message if it fails.

/-/open-csv-from-url

POST /-/open-csv-from-url
{"url": "https://example.com/file.csv", "table_name": "My_suggested_table_name"}

Imports a CSV file into the default /temporary in-memory database. Used by the "Open CSV from URL..." menu option. table_name is optional - if omitted the name to use will be derived from the URL.

Returns HTTP 200 status with {"ok": True, "path": "/temporary/table"} if it works, 400 or 500 with an "error" JSON string message if it fails.

/-/dump-temporary-to-file

POST /-/dump-temporary-to-file
{"path": "/path/to/backup.db"}

Dumps the contents of the temporary in-memory database to the specified file. This is intended to be used to take a temporary backup when the Datasette server is restarted after a plugin has been installed, see datasette-app/issues/42.

Returns HTTP 200 status with {"ok": True, "path": "/path/to/backup.db"} if it works, 400 or 500 with an "error" JSON string message if it fails.

/-/restore-temporary-to-file

POST /-/restore-temporary-from-file
{"path": "/path/to/backup.db"}

Restores the temporary in-memory database to the contents of the specified file.

Returns HTTP 200 status with {"ok": True, "path": "/path/to/backup.db"} if it works, 400 or 500 with an "error" JSON string message if it fails.

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd datasette-app-support
python3 -mvenv venv
source venv/bin/activate

Or if you are using pipenv:

pipenv shell

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

datasette-app-support's Issues

Don't allow double-imports of example CSVs

Bug: clicking the "Import" button on Squirrel Census a second time doubles the number of rows in that example table! I should hide it entirely if the "temporary" database already has a table with that suggested name in it.

Originally posted by @simonw in simonw/datasette-app#91 (comment)

startup() hook firing twice

I'm not sure why this is happening, but in some circumstances the startup() hook fires twice, resulting in the following:

This also explains the previous issue:

• #24

Upgrade and uninstall buttons

Refs #11 and simonw/datasette-app#67 and simonw/datasette-app#100 and simonw/datasette-app#74

API endpoints in `datasette-app-support` for dump/restore of temporary in-memory database

This is going to need some new API endpoints in datasette-app-support:

• /-/dump-temporary-to-file
• /-/restore-temporary-from-file

I'll leave it to the Electron app to define the file - I thought about using tempfile in Python but I don't think that will survive the server restart.

Originally posted by @simonw in simonw/datasette-app#42 (comment)

Tests fail due to allow_redirects

https://github.com/simonw/datasette-app-support/actions/runs/2658851433

datasette-app-support/tests/test_dump_restore_temporary.py

Lines 21 to 26 in af34d67

	response = await datasette.client.post(
	"/-/dump-temporary-to-file",
	json={"path": backup_path},
	headers={"Authorization": "Bearer fake-token"},
	allow_redirects=False,
	)

datasette-app-support/tests/test_auth_app_user.py

Lines 9 to 14 in d8eb4c3

	response = await datasette.client.post(
	"/-/auth-app-user",
	json={"redirect": "/-/metadata"},
	headers={"Authorization": "Bearer fake-token"},
	allow_redirects=False,
	)

Search no longer shows up on plugin directory

Custom template displaying "Import CSV" menu option on database page

Also need the custom template that displays the "Import CSV" button on the database page - though I'm not 100% sure this is the right way to go, I think a "File -> Import CSV" option would be more consistent with the rest of the interface.

Originally posted by @simonw in simonw/datasette-app#48 (comment)

`/-/open-csv-from-url` endpoint

Need a /-/open-csv-from-url endpoint.

Originally posted by @simonw in simonw/datasette-app#107 (comment)

Survive `_memory` KeyError

Spotted this when the app failed to start:

{
  message: '  File "/Users/simon/.datasette-app/venv/lib/python3.9/site-packages/datasette/utils/__init__.py", line 111, in await_me_maybe',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: '    value = await value',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: '  File "/Users/simon/.datasette-app/venv/lib/python3.9/site-packages/datasette_app_support/__init__.py", line 51, in inner',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: '    datasette.remove_database("_memory")',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: '  File "/Users/simon/.datasette-app/venv/lib/python3.9/site-packages/datasette/app.py", line 456, in remove_database',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: '    new_databases.pop(name)',
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}
{
  message: "KeyError: '_memory'",
  type: 'stderr',
  ts: 2023-02-17T02:02:36.938Z
}

I'm not sure under what circumstances that would happen, but this fixes the problem and getting out a fast fix is a good idea:

        try:
            datasette.remove_database("_memory")
        except KeyError:
            pass

/-/auth-app-user API endpoint

Setting cookies from Electron code looks too complicated - I'll go with the simpler mechanism where every new BrowserWindow hits /-/auth-app-user?redirect=/-/plugins which accepts the Authorization: Bearer xxx token from #53 and redirects with the newly set cookie.

Originally posted by @simonw in simonw/datasette-app#37 (comment)

Importing this CSV from a URL only gets 13 rows, not 596

https://raw.githubusercontent.com/okfn/dataportals.org/master/data/portals.csv

The "Open CSV from URL..." menu option only produced 13 rows - but using sqlite-utils insert portals.db portals portals.csv --csv on the command-line got all 596.

Custom table interface for browsing and installing plugins

Here's a fun idea: what if the list of plugins you could install was a very lightly customized Datasette table? Would get search and faceting for free, and help people further understand what the tool can be used for.

An in-memory plugins database which is deliberately excluded from the homepage.

Originally posted by @simonw in simonw/datasette-app#74 (comment)

Handle example CSVs with source but no about

e.g. simonw/datasette.io@ad6321a

open-csv-from-url should follow redirects

Refs:

• simonw/datasette-app#146

"Clear filters" link in plugin directory to recover from getting stuck

If you upgrade the last plugin on the "upgrade available" list, the page will refresh and there will be nothing on it with no mechanism to return to the full list of plugins.

API endpoints should check Bearer against DATASETTE_APP_TOKEN env variable

Token is now being generated and passed to the Datasette server - next step is to upgrade the plugin to only accept calls if they are accompanied by a Bearer token that matches the DATASETTE_API_TOKEN environment variable.

Originally posted by @simonw in simonw/datasette-app#53 (comment)

Error loading CSV should return JSON error

Got an HTML 500 opening a file with invalid UTF-8 - that should have been JSON.

Typo in CSS

datasette-app-support/datasette_app_support/static/sticky-footer.css

Lines 1 to 4 in 639d8bf

	shtml,
	body {
	height: 100%;
	}

Option to specify table_name for /-/open-csv-from-url

Need to be able to influence the table name used for these examples, otherwise that Squirrel one gets a table name of "rows" which is a bit rubbish.

Originally posted by @simonw in simonw/datasette-app#91 (comment)

Server should start even if internet is not available

It hangs on the loading screen and the console log shows this:

{
  message: 'httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known',
  type: 'stderr',
  ts: 2021-09-13T18:24:53.484Z
}

Originally posted by @simonw in simonw/datasette-app#115 (comment)

Sign user in as root but hide the _internal database

This needs a better solution in the longer-term, but a hack will do for the moment.

Plugins such as datasette-edit-schema have a default permission check for the "root" user. To avoid confusion, the app doesn't sign users in as "root", it uses "admin" instead.

But... https://github.com/simonw/datasette-edit-schema/blob/9b7a139c541d8b44aca77fde9745edc82c527cd7/datasette_edit_schema/__init__.py#L8-L11 currently does this:

@hookimpl
def permission_allowed(actor, action):
    if action == "edit-schema" and actor and actor.get("id") == "root":
        return True

We want that permission check to pass for the "admin" user too.

`/-/import-csv-file` API

Originally posted by @simonw in simonw/datasette-app#48 (comment)

CSS file is not included in package on PyPI

More plugin directory facet values: upgradable, default

Plugins that are not at their most recent version should show a "upgrade" button and have a installation status of "upgradable"

Default plugins cannot be uninstalled, so should have a status of "default" and not offer the forthcoming uninstall button.