simonw/datasette-publish-fly

Datasette plugin for publishing data using Fly

datasette-publish-fly's Issues

Support configuration directory mode

App published to fly.io with volume is losing data after `fly apps restart`

I successfully deployed datasette to fly.io using the command below. I am trying to keep the .db in a persisted volume so that I can add and update data periodically but am running into some problems including:

• After creating a new table, then running fly apps restart, the database is missing the table. Steps to create a new table were:
  • fly ssh console -a some-project
  • apt update && apt install sqlite3
  • sqlite3
  • >.open some.db
  • > .tables
  • some_table
  • > create table test_table as select 1 col1 union select 2;
  • > .tables
  • some_table test_table
• The --install datasette-saved-queries flag doesn't do anything. If I install it locally first before deploying, then it works.
• Created a file with echo some_data > /app/vol1/some_file.txt and it was gone after fly apps restart

Here's the full command I'm using

datasette publish fly data/outputs/sqlite/some.db \
	--app="some-project" \
	--create-volume 2 \
	--volume-name vol1 \
	--static vol1:data/outputs/test/ \
	--install datasette-saved-queries \
	--setting sql_time_limit_ms 25000

I'm not sure if this is a bug or user error, and appreciate any guidance. There's a good chance I'm missing something fundamental about how fly.io volumes work.

`--setting force_https_urls 1` fails due to missing imports

Traceback (most recent call last):
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/datasette_publish_fly/__init__.py", line 62, in convert
    return name, value_as_boolean(value)
NameError: name 'value_as_boolean' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cldellow/src/dux-publish/venv/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 1655, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 920, in make_context
    self.parse_args(ctx, args)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 1378, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 2360, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 2316, in process_value
    value = self.type_cast_value(ctx, value)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 2302, in type_cast_value
    return tuple(convert(x) for x in check_iter(value))
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/core.py", line 2302, in <genexpr>
    return tuple(convert(x) for x in check_iter(value))
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/click/types.py", line 82, in __call__
    return self.convert(value, param, ctx)
  File "/home/cldellow/src/dux-publish/venv/lib/python3.8/site-packages/datasette_publish_fly/__init__.py", line 63, in convert
    except ValueAsBooleanError:
NameError: name 'ValueAsBooleanError' is not defined

Adding the two missing imports seems to resolve it, #27 has a fix

Tool now asks interactive questions, it did not before

% datasette publish fly fixtures.db -a datasette-fly-test-december-2020

Selected App Name: datasette-fly-test-december-2020

Automatically selected personal organization: Simon Willison

? Select builder:  [Use arrows to move, type to filter]
> Dockerfile
    (Do not set a builder and use the existing Dockerfile)

Don't break if upgrade message shown

Just saw this error:

  File "/usr/local/lib/python3.7/site-packages/datasette_publish_fly/__init__.py", line 106, in fly
    apps = existing_apps()
  File "/usr/local/lib/python3.7/site-packages/datasette_publish_fly/__init__.py", line 123, in existing_apps
    assert lines[0].startswith("NAME")
AssertionError
~/Downloads $ flyctl apps list
Update available 0.0.108 -> 0.0.109
  NAME                 OWNER            LATEST DEPLOY         
  datasette-demo       simon-willison   2020-03-19T01:59:40Z  
  squirrel-map         simon-willison   1m19s ago             
  summer-violet-8396   simon-willison   2020-03-19T22:10:45Z  

Parameter --plugin-secret is not working

Deploying an app with

datasette publish fly \
    data.db \
    --app my-app

works and I can access the data.

But using the script with --install ... and --plugin-secret ... flags, e.g.

datasette publish fly \
   data.db \
   --app my-app \
   --install datasette-auth-passwords \
   --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha2 ...'

this error shows up:

./start-fly.sh                                                                                                ─╯
Error: Error calling 'flyctl secrets set':

Error: unknown flag: --json

Is there a change in the flyctl secret ... command?

Return code not non-zero on errors

I have the following in my GHA configuration:

name: Fly Deploy
on: [push]
env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
jobs:
  deploy:
      name: Deploy app
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v3
        - uses: actions/setup-python@v4
          with:
            python-version: '3.10'
            cache: 'pip'
        - run: pip install -r requirements.txt
        - run: yaml-to-sqlite wfms.db wfms wfms.yaml
        - uses: superfly/flyctl-actions/setup-flyctl@master
        - run: datasette publish fly wfms.db --app workflows --metadata metadata.json --plugins-dir=plugins/ --static assets:static-files/

which failed in the last step with this:

Run datasette publish fly wfms.db --app workflows --metadata metadata.json --plugins-dir=plugins/ --static assets:static-files/
  datasette publish fly wfms.db --app workflows --metadata metadata.json --plugins-dir=plugins/ --static assets:static-files/
  shell: /usr/bin/bash -e {0}
  env:
    FLY_API_TOKEN: ***
    pythonLocation: /opt/hostedtoolcache/Python/3.10.5/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.5/x64/lib/pkgconfig
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.5/x64/lib
==> Verifying app config
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-black-cloud-1435...
Error failed to fetch an image or build from source: error connecting to docker: failed building options: lookup dfw2.gateway.6pn.dev on 127.0.0.53:53: server misbehaving

but GHA did not fail the job, which usually happens when a job returns a non-zero return/error code.

TypeError: __init__() got an unexpected keyword argument 'capture_output'

Hi,
I am following instructions in this blog post:

https://fly.io/blog/making-datasets-fly-with-datasette-and-fly/

but have hit this error:

$ datasette publish fly squirrels.db --app squirrelsx
Traceback (most recent call last):
File "/home/darreng/code/flyio_datasette_example/env/bin/datasette", line 8, in
sys.exit(cli())
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/datasette_publish_fly/init.py", line 106, in fly
apps = existing_apps()
File "/home/darreng/code/flyio_datasette_example/env/lib/python3.6/site-packages/datasette_publish_fly/init.py", line 119, in existing_apps
process = run(["flyctl", "apps", "list"], capture_output=True)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: init() got an unexpected keyword argument 'capture_output'

But it appears that capture_output is new in Python 3.7.

Should the README be updated to indicate that 3.7 in required? or could you accomodate older versions using the PIPE workaround detailed here:
https://stackoverflow.com/questions/53209127/subprocess-unexpected-keyword-argument-capture-output

Thanks
Darren

Let users specify the instance size to be used

https://fly.io/docs/pricing/ lists four options:

It would be great if you could run datasette publish fly data.db --size=cpu1 to deploy with a larger instance.

`--generate-dir` works confusingly

I tried out --generate-dir and ran into two problems with the code that's responsible:

datasette-publish-fly/datasette_publish_fly/__init__.py

Lines 313 to 322 in 564a216

	if generate_dir:
	dir = pathlib.Path(generate_dir)
	if not dir.exists():
	dir.mkdir()
	# Copy files from current directory to dir
	for file in pathlib.Path(".").glob("*"):
	shutil.copy(str(file), str(dir / file.name))
	(dir / "fly.toml").write_text(fly_toml, "utf-8")
	return

• It can only copy files, not directories (e.g. the plugin directory!)
• If you give a relative path, the directory is generated not in the current working directory, but in the temporary directory which is cleaned up afterwards!

As a workaround I had to give an absolute path, but that was after some confusion and digging into the code.

--setting option

As seen in datasette-publish-vercel - easier than using --extra-options

Also test against Python 3.10

And drop test for Python 3.6 since Datasette won't support that version going forward.

Allow setting `force_https = true` in `fly.toml`

I would like to force http -> https redirects, but as far as I see it's currently not possible to set force_https = true for the generated fly.toml file.

Am I missing something or is this an option that could be exposed to the user? (Personally, I wouldn't mind to hard code force_https = true in FLY_TOML though.)

Support Fly volumes

Especially interesting given the 3GB available now in the free tier: https://fly.io/blog/free-postgres/

Research how to deploy with latest flyctl and with a volume

Just got "Error: unknown flag: --builder" from flyctl apps create while trying to run a deploy.

Large sqlite db, 10min threshold guess

Tried:

datasette publish fly x.db \
--app "test" \
--region "sin" \
--extra-options="--setting sql_time_limit_ms 15000" \
--metadata metadata.yml \
--install pysqlite3-binary \
--install datasette-auth-tokens

The ph.db sqlite3 file is 2gb.

After ~10 minutes it stops with an error.

The deployment used to be work well. I'm guessing it was because it hit the threshold of inactivity for the remote builder, i.e. as a feature: "They turn off automatically after 10 minutes of inactivity"

==> Verifying app config
→ Verified app config
==> Building image
Remote builder fly-builder-white-wood-1586 ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.12 linux x86_64
[+] Building 596.1s (0/1)
=> [internal] load remote build context 596.1s
*ERRO[0622] Can’t add file /private/var/folders/7g/qc77bj6j63q0gr0mrzvbmxkc0000gn/T/tmph1n8z2gw/test/x.db to tar: io: read/write on closed pipe *
*ERRO[0622] Can’t close tar writer: io: read/write on closed pipe *
Error failed to fetch an image or build from source: error building: unexpected EOF

Assuming the guess is correct, is any any workaround possible? Will building the docker image locally help in not consuming the full 10 minutes?

Allow loading SQLite extensions

I tried adding a custom SQLite extension like this:

--extra-options '--load-extension ./text.so'

but it fails to deploy. For now I am using the --generate-dir flag, copying the .so manually and then run flyctl deploy myself.

`--generate-dir` option

I'm going to imitate datasette publish vercel which has this option:

  --generate-dir DIRECTORY        Output generated application files and stop
                                  without deploying

Originally posted by @simonw in #12 (comment)

Add output of `datasette publish fly --help` to the README

Using cog.

Make --app optional

Fly can generate random names for applications for you. I should make it an optional argument and let fly generate you a random name.

I should also have datasette publish fly output a command-clickable URL when it finishes (with an https:// prefix).

Refs superfly/flyctl#82

Take advantage of new --json option

The code here parses the output of flyctl apps list:

datasette-publish-fly/datasette_publish_fly/__init__.py

Lines 118 to 131 in 990fdf8

	def existing_apps():
	process = run(["flyctl", "apps", "list"], capture_output=True)
	output = process.stdout.decode("utf8")
	all_lines = [l.strip() for l in output.split("\n")]
	# Skip lines until we find the NAME line
	lines = []
	collect = False
	for line in all_lines:
	if collect:
	lines.append(line)
	elif line.startswith("NAME"):
	collect = True
	apps = [l.strip().split()[0] for l in lines if l.strip()]
	return apps

Fly now has a --json flag which would make this more robust: https://fly.io/blog/flyctl-meets-json/

`--plugin-secret` should use Fly secrets

In writing the documentation for the new volumes stuff - #12 (comment)_ (also refs #10) - I realized that if I'm going to show people how to use volumes I need them to deploy writable plugins. But if I show them that, it's not responsible to show them how to do it without authentication. So I should include datasette-auth-passwords - but that requires them to set a password secret using --plugin-secret...

... and I don't want them to have to set the same plugin secret every time they run a deploy to update an existing instance!

So, instead, I'm going to see if I can get --plugin-secret to set a Fly secret, which will persist for the lifetime of the application across multiple deploys.

"Can't close tar writer: io: read/write on closed pipe"

(datasette-demo) datasette-demo % datasette publish fly fixtures.db -a datasette-demo
Update available 0.0.200 -> 0.0.229
Update with flyctl version update

Deploying datasette-demo
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Remote builder fly-builder-blue-surf-155 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
ERRO[0007] Can't add file /private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq6ofghqk/datasette-demo/Dockerfile to tar: io: read/write on closed pipe 
ERRO[0007] Can't close tar writer: io: read/write on closed pipe 
Error error building: error building with docker: error during connect: Post "https://fly-builder-blue-surf-155.fly.dev:10000/v1.41/build?buildargs=%7B%7D&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&platform=linux%2Famd64&rm=0&shmsize=0&t=registry.fly.io%2Fdatasette-demo%3Adeployment-1627679663&target=&ulimits=null&version=": read tcp 10.0.0.8:62141->213.188.211.42:10000: read: connection reset by peer

Deployment fails with "incompatible types: TOML value has type string; destination has type integer"

I didn't change anything in the configuration, yet redeployment started to fail:

 datasette publish fly wfms.db --app workflows --metadata metadata.yaml --plugins-dir=plugins/ --static assets:static-files/
  shell: /usr/bin/bash -e {0}
  env:
    FLY_API_TOKEN: ***
    pythonLocation: /opt/hostedtoolcache/Python/3.10.5/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.5/x64/lib/pkgconfig
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.5/x64/lib
  
Error failed loading app config from fly.toml: toml: line 18 (last key "services.ports.port"): incompatible types: TOML value has type string; destination has type integer

`*.db` database being created

Running this command:

datasette publish fly \                        
--app my-new-fly-application-2 \
--volume-name vol1 \
--create-volume 1 \
--create-db tiddlywiki \
--install datasette-auth-passwords \
--install datasette-tiddlywiki \
--plugin-secret datasette-auth-passwords root_password_hash $ROOT_PW

Resulted in a database called * being created. This is using the released versions of everything.

fly() got an unexpected keyword argument 'secret'

Downgrading to datasette 0.43 fixed it.

Document usage with datasette-scale-to-zero

Following #29 I decided to try this:

datasette publish fly \
   fixtures.db \
   --app datasette-publish-fly-issue-29 \
   --install datasette-auth-passwords \
   --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$480000$9ce99372d1fa079f770d4e2245bcf335$zJjskTDc6M8sxmEUYZBr/EC0e730Q9pzcF8RJB43c/c=' \
   --install datasette-scale-to-zero \
   --plugin-secret datasette-scale-to-zero duration 5m

Using https://datasette.io/plugins/datasette-scale-to-zero ... and it worked!

I watched the Fly Dashboard and the app at https://datasette-publish-fly-issue-29.fly.dev/ now successfully scales to zero if it receives no hits for 5 minutes, then starts running again when traffic arrives:

Here's it exiting 5m after the last request:

And here's it rebooting when a new request comes in:

Error org slug must be specified when not running interactively

~ % datasette publish fly --create-volume 20 --app laion-aesthetic2 --create-db data
Error: Error calling 'flyctl apps create':

Error org slug must be specified when not running interactively

I'm going to add a --org option.