simonw/sphinx-to-sqlite

Create a SQLite database from Sphinx documentation

sphinx-to-sqlite

Create a SQLite database from Sphinx documentation.

Demo

You can see the results of running this tool against the Datasette documentation at https://latest-docs.datasette.io/docs/sections

Installation

Install this tool using pip:

$ pip install sphinx-to-sqlite

Usage

First run sphinx-build with the -b xml option to create XML files in your _build/ directory.

Then run:

$ sphinx-to-sqlite docs.db path/to/_build

To build the SQLite database.

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd sphinx-to-sqlite
python -mvenv venv
source venv/bin/activate

Or if you are using pipenv:

pipenv shell

Now install the dependencies and tests:

pip install -e '.[test]'

To run the tests:

pytest

sphinx-to-sqlite's Issues

Preserving code blocks

Looking at example provided, it looks like code fencing is stripped? If code blocks were preserved, then they could also be rendered using styled markup, eg prism.js.

It may also be possible to generate additional structure by running text through Jupytext to split out code cells from markdown cells, then you can search over code and/or markdown? I've started exploring things relating to this in ouseful-testing/nbsearch (repo is likely to move at some point...).

Reasoning behind project

Hi Simon!

This isn't really an issue so feel free to close this. I am just curious behind the motivation for this project. What is the usecase and why would anyone want this? :)

Have a good day and best of luck with the project!

~ Yasoob

Some content gets concatenated with no spaces

e.g. https://latest-docs.datasette.io/docs/sections/index%3Acontents

Getting startedPlay with a live demoTry Datasette without installing anything using GlitchUsing Datasette on your own computerdatasette --getdatasette serve --helpInstallationBasic installationUsing HomebrewUsing pipAdvanced installation optionsUsing pipxUsing DockerThe Datasette EcosystemTools for creating SQLite databasescsvs-to-sqlitesqlite-utilsdb-to-sqlitedbf-to-sqlitemarkdown-to-sqlitegeojson-to-sqliteshapefile-to-sqlitesocrata2sqlDatasette Pluginsdatasette-graphqldatasette-cluster-mapdatasette-vegadatasette-auth-githubdatasette-auth-tokensdatasette-permissions-sqldatasette-upload-csvsdatasette-json-htmldatasette-atomdatasette-icsdatasette-initdatasette-writedatasette-mediadatasette-jellyfishdatasette-doublemetaphonedatasette-jqdatasette-ruredatasette-render-imagesdatasette-render-binarydatasette-render-markdowndatasette-render-htmldatasette-leaflet-geojsondatasette-pretty-jsondatasette-saved-queriesdatasette-haversinedatasette-sqlite-fts4datasette-bplistdatasette-corsdatasette-template-sqldatasette-mask-columnsdatasette-auth-existing-cookiesdatasette-sentrydatasette-publish-flyPages and API endpointsTop-level indexDatabaseTableRowPublishing datadatasette publishPublishing to Google Cloud RunPublishing to HerokuPublishing to VercelPublishing to FlyCustom metadata and pluginsdatasette packageDeploying DatasetteDeployment fundamentalsRunning Datasette using systemdDeploying using buildpacksJSON APIDifferent shapesPaginationSpecial JSON argumentsTable argumentsColumn filter argumentsSpecial table argumentsExpanding foreign key referencesRunning SQL queriesNamed parametersViewsCanned queriesCanned query parametersSetting a default fragmentWritable canned queriesMagic parametersJSON API for writable canned queriesPaginationAuthentication and permissionsActorsUsing the "root" actorPermissionsDefining permissions with "allow" blocksThe /-/allow-debug toolConfiguring permissions in metadata.jsonControlling access to an instanceControlling access to specific databasesControlling access to specific tables and viewsControlling access to specific canned queriesControlling the ability to execute arbitrary SQLChecking permissions in pluginsactor_matches_allow()The permissions debug toolThe ds_actor cookieIncluding an expiry timeThe /-/logout pageBuilt-in permissionsview-instanceview-databaseview-database-downloadview-tableview-queryexecute-sqlpermissions-debugPerformance and cachingImmutable modeUsing "datasette inspect"HTTP cachingHashed URL modeCSV ExportStreaming all recordsA note on URLsFacetsFacets in querystringsFacets in metadata.jsonSuggested facetsSpeeding up facets with indexesFacet by JSON arrayFacet by dateFull-text searchThe table page and table view APIAdvanced SQLite search queriesConfiguring full-text search for a table or viewSearches using custom SQLEnabling full-text search for a SQLite tableConfiguring FTS using sqlite-utilsConfiguring FTS using csvs-to-sqliteConfiguring FTS by handFTS versionsSpatiaLiteInstallationInstalling SpatiaLite on OS XInstalling SpatiaLite on LinuxBuilding SpatiaLite from sourceSpatial indexing latitude/longitude columnsMaking use of a spatial indexImporting shapefiles into SpatiaLiteImporting GeoJSON polygons using ShapelyQuerying polygons using within()MetadataPer-database and per-table metadataSource, license and aboutSpecifying units for a columnSetting a default sort orderSetting a custom page sizeSetting which columns can be used for sortingSpecifying the label column for a tableHiding tablesUsing YAML for metadataConfigurationUsing --configConfiguration directory modeConfiguration optionsdefault_page_sizesql_time_limit_msmax_returned_rowsnum_sql_threadsallow_facetdefault_facet_sizefacet_time_limit_msfacet_suggest_time_limit_mssuggest_facetsallow_downloaddefault_cache_ttldefault_cache_ttl_hashedcache_size_kballow_csv_streammax_csv_mbtruncate_cells_htmlforce_https_urlshash_urlstemplate_debugbase_urlConfiguring the secretUsing secrets with datasette publishIntrospection/-/metadata/-/versions/-/plugins/-/config/-/databases/-/threads/-/actor/-/messagesCustom pages and templatesCustom CSS and JavaScriptCSS classes on the <body>Serving static filesPublishing static assetsCustom templatesCustom pagesPath parameters for pagesCustom headers and status codesReturning 404sCustom redirectsCustom error pagesPluginsInstalling pluginsOne-off plugins using --plugins-dirDeploying plugins using datasette publishSeeing what plugins are installedPlugin configurationSecret configuration valuesWriting pluginsWriting one-off pluginsStarting an installable plugin using cookiecutterPackaging a pluginStatic assetsCustom templatesWriting plugins that accept configurationBuilding URLs within pluginsPlugin hooksprepare_connection(conn, database, datasette)prepare_jinja2_environment(env)extra_template_vars(template, database, table, columns, view_name, request, datasette)extra_css_urls(template, database, table, columns, view_name, request, datasette)extra_js_urls(template, database, table, columns, view_name, request, datasette)extra_body_script(template, database, table, columns, view_name, request, datasette)publish_subcommand(publish)render_cell(value, column, table, database, datasette)register_output_renderer(datasette)register_routes()register_facet_classes()asgi_wrapper(datasette)startup(datasette)canned_queries(datasette, database, actor)actor_from_request(datasette, request)permission_allowed(datasette, actor, action, resource)register_magic_parameters(datasette)forbidden(datasette, request, message)Testing pluginsUsing pytest fixturesInternals for pluginsRequest objectThe MultiParams classResponse classSetting cookies with response.set_cookie()Datasette class.plugin_config(plugin_name, database=None, table=None)await .render_template(template, context=None, request=None)await .permission_allowed(actor, action, resource=None, default=False).get_database(name).add_database(name, db).remove_database(name).sign(value, namespace="default").unsign(value, namespace="default").add_message(request, message, message_type=datasette.INFO)datasette.clientdatasette.urlsDatabase classawait db.execute(sql, ...)Resultsawait db.execute_fn(fn)await db.execute_write(sql, params=None, block=False)await db.execute_write_fn(fn, block=False)Database introspectionCSRF protectionContributingGeneral guidelinesSetting up a development environmentDebuggingEditing and building the documentationRelease processAlpha and beta releasesUpgrading CodeMirrorChangelog0.51a0 (2020-10-19)0.50.2 (2020-10-09)0.50.1 (2020-10-09)0.50 (2020-10-09)0.49.1 (2020-09-15)0.49 (2020-09-14)0.48 (2020-08-16)0.47.3 (2020-08-15)0.47.2 (2020-08-12)0.47.1 (2020-08-11)0.47 (2020-08-11)0.46 (2020-08-09)0.45 (2020-07-01)Magic parameters for canned queriesLog outBetter plugin documentationNew plugin hooksSmaller changes0.44 (2020-06-11)AuthenticationPermissionsWritable canned queriesFlash messagesSigned values and secretsCSRF protectionCookie methodsregister_routes() plugin hooksSmaller changesThe road to Datasette 1.00.43 (2020-05-28)0.42 (2020-05-08)0.41 (2020-05-06)0.40 (2020-04-21)0.39 (2020-03-24)0.38 (2020-03-08)0.37.1 (2020-03-02)0.37 (2020-02-25)0.36 (2020-02-21)0.35 (2020-02-04)0.34 (2020-01-29)0.33 (2019-12-22)0.32 (2019-11-14)0.31.2 (2019-11-13)0.31.1 (2019-11-12)0.31 (2019-11-11)0.30.2 (2019-11-02)0.30.1 (2019-10-30)0.30 (2019-10-18)0.29.3 (2019-09-02)0.29.2 (2019-07-13)0.29.1 (2019-07-11)0.29 (2019-07-07)ASGINew plugin hook: asgi_wrapperNew plugin hook: extra_template_varsSecret plugin configuration optionsFacet by dateEasier custom templates for table rows?_through= for joins through many-to-many tablesSmall changes0.28 (2019-05-19)Supporting databases that changeFaceting improvements, and faceting pluginsdatasette publish cloudrunregister_output_renderer pluginsMedium changesSmall changes0.27.1 (2019-05-09)0.27 (2019-01-31)0.26.1 (2019-01-10)0.26 (2019-01-02)0.25.2 (2018-12-16)0.25.1 (2018-11-04)0.25 (2018-09-19)0.24 (2018-07-23)0.23.2 (2018-07-07)0.23.1 (2018-06-21)0.23 (2018-06-18)CSV exportForeign key expansionsNew configuration settingsControl HTTP caching with ?_ttl=Improved support for SpatiaLitelatest.datasette.ioMiscellaneous0.22.1 (2018-05-23)0.22 (2018-05-20)0.21 (2018-05-05)0.20 (2018-04-20)0.19 (2018-04-16)0.18 (2018-04-14)0.17 (2018-04-13)0.16 (2018-04-13)0.15 (2018-04-09)0.14 (2017-12-09)0.13 (2017-11-24)0.12 (2017-11-16)0.11 (2017-11-14)0.10 (2017-11-14)0.9 (2017-11-13)0.8 (2017-11-13)

That's no good for building search! I need to make sure those tokens have at least one space between them.

"breadcrumbs" column should be a JSON array of links, like "references"

https://latest-docs.datasette.io/docs/sections/authentication%3Aallowdebugview is currently:

id	page	ref	title	content	breadcrumbs	references
authentication:allowdebugview	authentication	allowdebugview	The /-/allow-debug tool	The /-/allow-debug tool lets you try out different "action" blocks against different "actor" JSON objects. You can try that out here: https://latest.datasette.io/-/allow-debug	["Authentication and permissions", "Permissions"]	[{"href": "https://latest.datasette.io/-/allow-debug", "label": "https://latest.datasette.io/-/allow-debug"}]

Breadcrumbs should be something like this instead:

[
  {
    "href": "authentication.html",
    "label": "Authentication and permissions"
  }, {
    "href": "authentication.html#permissions",
    "label": "Permissions"
  }
]