barryanders/grav-plugin-blackhole

The static site generator for Grav CMS

The ⚫ Blackhole Plugin is for Grav CMS.

Description

Why Blackhole? Grav is a space term, so I think this plugin should follow suit. Time stops at the event horizon of a black hole, which is exactly what this plugin does to your website. It freezes it in a state. By Increasing gravity to infinity you get a static black hole, or in this case you generate a static html copy of your Grav website.

Installation

GPM Installation

The simplest way to install this plugin is via the Grav Package Manager (GPM). From the root of your Grav install type: bin/gpm install blackhole

Manual Installation

If you can't use GPM you can manually install this plugin. Download the zip version of this repository and unzip it under /your/site/grav/user/plugins, then rename the folder to blackhole.

Usage

Absolute URLs must be enabled in Grav System Configuration.

Generate Command

The generate command can be used from the command line or directly in the Grav Admin Panel. Generate your static site. generate can also be written as gen or g.

• Input URL (required) - Enter the URL to your live Grav site.

bin/plugin blackhole generate http://localhost/grav

Options

• Output URL --output-url or -d The URL of your static site. This determines the domain used in the absolute path of your links.

  --output-url https://website.com

• Output Path --output-path or -p The directory to which your static site will be written (relative to Grav root).

  --output-path ../build

• Routes --routes or -r Limit generation to a select list of page routes.

  --routes home,about,about/contact

• Use Sitemap --use-sitemap or -m Instead of listing all pages use a sitemap generated by the grav Sitemap plugin. Usefull when your site has multiple languages and/or different URL extension (e.g. .html) When this option is enabled the --routes option will be ignored.

  --use-sitemap /sitemap

• Simultaneous Limit --simultaneous or -s Determine how many files will generate at the same time (default: 10).

  --simultaneous 10

• Assets --assets or -a Copy assets to the output path.

• Force --force or -f Overwrite previously generated files.

• Verbose Mode --verbose-mode Enable verbose mode.

Author

Barry Anders

grav-plugin-blackhole's Issues

Navigation links not rewritten

Hello - I am super excited about this plugin. Thanks for making it!

I have it successfully running from the command line and generating all of the static files and folders, but it is not changing the links.

I'm running Grav in a sub-folder (e.g., sitename.com/gravcms) and using the command

php bin/plugin blackhole generate https://sitename.com/gravcms/ --output-url https://sitename.com/ --output-path ../ --assets --verbose-mode --force

But when I look at the HTML it saves, I still have things like this:

<a href="/gravcms/">Home</a>
<a href="/gravcms/page-two">Page Two</a>

Am I doing something wrong, or is there a bug? Thank you!

Auto-generate on page save [Feature Request]

Hi,

Would it be possible, like the Git-Sync plugin does, to trigger the generate action when saving a "page" file?

It would be quite nice!

Many thanks in advance,

Multithreading

As was requested. I previously said I didn't know how or that I didn't think you could, but one search proved that it is possible. I'll at least look into it. I don't want to have to rely on a PHP extension for a Grav plugin though. I want it to work out of the box.

Incorrect release name

Because your most recent version is named "v1.0.0 beta" it cannot be picked up by the GPM, as Grav resolves this as "https://getgrav.org/download/plugins/blackhole/1.0.0 beta". To adhere to SemVer, it needs to be "v1.0.0-beta" or similar.

Feature request: generate static page from a browser

I don't have shell access on my shared webhosting.

Generating with assets

Just an idea. I won't implement this if I get a lot of negative feedback, or if it slows down generation too much. It was suggested in #12, so I thought I would give it its own issue.

The idea would be to copy theme related assets to the generated site, so that users don't have to manually do this. This could be done in a few ways, but in short, I would presume that all users are putting their assets in their theme folder. With that in mind, the entire theme directory could be scraped for JS and CSS, then copied to a directory such as /assets/js and /assets/css. Then the links on each page could be changed during generation time to reflect the new asset locations without messing with your live Grav site. The processing could be taken a step forward by minifying the files if people are interested in that.

Not installable atm

Hi,

The issue I'm having with updating/installing the plugin is that it isn't doing either of the 2.
On trying to install it from the terminal it gives a 'page not found' error, wich it does in the admin plugin too.

Versions I'm running are:

• Grav v1.5.0-beta.2
• Admin v1.8.6

Call another plugin [Feature Request]

Hi!

Would it be possible to intertwine seamlessly BlackHole with Git-Sync?

Here is how Flex Objects does it https://github.com/trilbymedia/grav-plugin-flex-objects/blob/develop/classes/Controllers/ObjectController.php#L133

It would be super nice that, upon finishing generation of static files, we could call a hook (e.g. git-sync)

This way, it would require zero human interaction to:

• Generate the static files
• Push the files to Git
• Have Netlify detect the changes, and deploy the site

(provided that BlackHole can generate the static files upon save, as I suggested in another Feature Request)

Thank you in advance!

Best,

No longer working with Grav CMS 1.6.28

The plugin doesn't seem to be working anymore. It does generate the static html files, and changes the URL, but the css files for example are no longer moved and linked. This used to just work, like just a few grav versions ago.

Here is what I used:
bin\plugin blackhole generate http://localhost -p static -s 2 -d https://example.com -a -f

I also installed a fresh Grav 1.6.28 with Quark theme (default). CSS etc are in user/themes/quark/css

Pages are generated, but the css and folders are not.

Could you look into this?

Add a button on top of left menu [Feature Request]

Hi,

Would it be possible to add a "generate" button inside the top area of the left menu?

The same kind that exists for "clear cache" and "git sync" plugin

I would not want the CMS users to go to the plugins page each time they change some article.

Thank you kindly for any feedback on this :)

Best,

No support for servers that don't include the domain name in the file tree

Hey, are you getting this error too? It's from the latest commit.

root@GravServer:/srv/users/serverpilot/apps/grav/public# bin/plugin blackhole generate static
PHP Warning:  Invalid argument supplied for foreach() in /srv/users/serverpilot/apps/grav/public/user/plugins/blackhole/cli/GenerateCommand.php on line 59

Generate button does nothing

Running a local fresh Docker Grav install, from inside the Grav Admin, I fill in:

• Output URL : https://mysite.tld
• Output Path : ../static (I also tried static to prevent a possible permissions problem)
  And hit Generate. Browser asks if I want to leave the page as though I have unsaved data, I hit yes, I'm back on the page, no static folder has been generated outside my Grav install - or anywhere on my laptop really.

What am I doing wrong?
PS: cannot use command line as I'm not on a PHP server (hence the Docker usage), and didn't find any logs
PPS : Found this thread with the exact same problem, but couldn't find any related issue in here.
(v1.1.0)

Quickdraw

Feature request: --copy-assets

Can blackhole modify paths when generating html in a subdir using the -p param?

My use case here is that I'd like to generate a static version of the site on the master branch and put it into a /docs dir so that I can utilize GitHub pages to build it. Here's the command I'm using:
bin/plugin blackhole generate http://localhost:8000/ -d http://[my.domain] -p docs

However, this results in static html that references paths such as /user/themes... but since it's being served out of the docs directory, these don't work. It seems like a lot to ask, but I wanted to at least give you a use case that's not working well.

Need help?

This isn't an issue, just wanted to see if you have any todo items you would liked banged out? I am a JS dev, but interested in looking at some PHP for this!

Feature Idea: Minify The HTML

Just an idea. I won't do it if I get a bunch of negative feedback. I just thought it might be a nice touch, but you may not like this idea. I wouldn't do it if it showed huge differences in generation time, of course.

No output generated too

I'm facing a similar issue to #1 where no output is generated. I've tried a few variations:

#assuming it generates relative to the root folder of Grav
bin/plugin blackhole gen /new

#assuming it generates from anywhere in the server
bin/plugin blackhole gen /srv/users/serverpilot/apps/grav-site/public/new

Would there be a log that I can check from?

css not exported

using Gantry 5 framework plugin (v5.5.0) and Helium theme, css is not correctly exported.

example of output: http://fnaah.com

screenshot of source site attached

Error After Installing Blackhole

After installing website and admin stop working with the below error

Stacktrace: #14 Whoops\Exception\ErrorException in /****/user/plugins/blackhole/blackhole.php:23 #13 Whoops\Run:handleError in /****/user/plugins/blackhole/blackhole.php:23 #12 Grav\Plugin\BlackholePlugin:onPageInitialized in /****/vendor/symfony/event-dispatcher/EventDispatcher.php:184 #11 call_user_func in /****/vendor/symfony/event-dispatcher/EventDispatcher.php:184 #10 Symfony\Component\EventDispatcher\EventDispatcher:doDispatch in /****/vendor/symfony/event-dispatcher/EventDispatcher.php:46 #9 Symfony\Component\EventDispatcher\EventDispatcher:dispatch in /****/vendor/rockettheme/toolbox/Event/src/EventDispatcher.php:23 #8 RocketTheme\Toolbox\Event\EventDispatcher:dispatch in /****/system/src/Grav/Common/Grav.php:277 #7 Grav\Common\Grav:fireEvent in /****/system/src/Grav/Common/Processors/PagesProcessor.php:25 #6 Grav\Common\Processors\PagesProcessor:process in /****/system/src/Grav/Common/Grav.php:126 #5 Grav\Common\Grav:Grav\Common\{closure} in /****/system/src/Grav/Common/Grav.php:368 #4 Grav\Common\Grav:Grav\Common\{closure} in /****/system/src/Grav/Common/Grav.php:344 #3 call_user_func_array in /****/system/src/Grav/Common/Grav.php:344 #2 Grav\Common\Grav:__call in /****/system/src/Grav/Common/Grav.php:127 #1 Grav\Common\Grav:measureTime in /****/system/src/Grav/Common/Grav.php:127 #0 Grav\Common\Grav:process in /****/index.php:52

the only way the grav worked again is by deleting the plugin

pages html empty - SKIPPING No changes

Hi,

When I generate pages like this bin/plugin blackhole generate http://mydomain.fr -p ../static

It displays SKIPPING No changes ➜....... on all pages who it geenrates, and so when I read each index.html they are empty with just the comment with Generated the date etc..

Did I do domething wrong ?

Doesn't seem to publish to site root?

First, thank you for making Blackhole, it is the reason I choose to use Grav for my site as I was searching for a lightweight CMS that can publish to static.

I have a vanilla Grav + Admin install on a typical LAMP stack. PHP 5.6.38
Grav is installed in a subdirectory leaving site root blank.
Site root intended for the static version of the site from Blackhole.
I put ../ in Blackhole's Output Path via Grav Admin > Plugins > Blackhole > field box for Output Path.
When clicking generate, it redirects to what seems like a cronjob url but, nothing is produced in the site root from blackhole.

All pages generated are the home page

Just logging this one. Will fix it tonight.

ERROR No pages were found

When I do a command in my localhost: bin/plugin blackhole generate http://localhost:8080/mypagename, I get this error: ERROR No pages were found.
The website exists and works otherwise.

Do you know what the problem could be?

How I grab my taxonomies

Just thought I share how I currently use blackhole to generate taxonomies, by exposing them in an array on a page.

{% if config.plugins.taxonomylist.enabled %}
{% set taxlist = taxonomylist.get() %}
{% if taxlist %}
[
{% for tax,value in taxlist['taxonomy1']|ksort %}"/taxonomy1{{ config.system.param_sep }}{{ tax|e('url') }}", {% endfor %}
{% for tax,value in taxlist['taxonomy2']|ksort %}"/taxonomy2{{ config.system.param_sep }}{{ tax|e('url') }}", {% endfor %}
{% for tax,value in taxlist['taxonomy3']|ksort %}"/taxonomy3{{ config.system.param_sep }}{{ tax|e('url') }}", {% endfor %}
{% for tax,value in taxlist['taxonomy4']|ksort %}"/taxonomy4{{ config.system.param_sep }}{{ tax|e('url') }}", {% endfor %}
{% for tax,value in taxlist['taxonomy5']|ksort %}"/taxonomy5{{ config.system.param_sep }}{{ tax|e('url') }}"{% if not loop.last %}, {% endif %}{% endfor %} 
]
{% endif %}
{% endif %}

The taxonomies are hardcoded because I couldn't find a way in my time to grab the name of the taxonomies. If you are able to, it might be a good idea to integrate into the plugin. Though I would suggest being very selective to what you would support and feature.

This just happens to be required for my use case and extending blackhole is easy using my current implementation of creating what are maps for blackhole to grab.

I also started doing this for my pages instead of the "?pages=all", because I wanted to account for pagination as well as custom pages such as /sitemap. The code below is currently untested.

content:
    items: '@root.descendants'
---
{% set collection = page.collection() %}
[
{% for post in collection %}"{{ post.url }}"
{% if config.plugins.pagination.enabled and collection.params.pagination %} 
{% set pagination = collection.params.pagination %}
{% if pagination.hasNext %}
, "{% set url = (pagination.params ~ pagination.nextUrl)|replace({'//':'/'}) %}"
{% if not loop.last %}, {% endif %}
{% endif %}
{% else %}
{% if not loop.last %}, {% endif %}
{% endif %}
{% endfor %}
]

That said, I'm curious to see how taxonomies and pagination will work for blackhole. It will be the next step to figure out.

New release does not generate Assets

Assets are being generated outside the build directory, names are being output with the entire route.

Idea: rewrite using wget

This plugin could be rewritten to use wget. It would then account for things like pagination which Blackhole does not catch right now. Most of the features could be implemented and this would not require nearly as much code upkeep as Grav continues to develop and change. I can't think of any reason why this wouldn't work well, but maybe you can. To me, it seems too easy.

shell_exec("wget ...");

Since there would be user input, it would just need to have all options sanitized.

escapeshellarg($input);

Multisite setup compatibility

Hi, I was wondering if this plugin is compatible with a multisite setup: https://learn.getgrav.org/16/advanced/multisite-setup

Blackhole multithreading causing crashes on large websites

Discussed in #12 and #15

RollingCurl appears to not be implemented correctly, because there is a reported bug that blackhole is causing crashes on large websites.

Proposed solution will be implemented in v0.8.1. If you would be so kind, let me know if it's solved after the update, @msharron. Thanks and hope it helps you with your websites.

Hmm... Don't seem to be picking up all the routes - what am I doing wrong?

Hi

First, many thanks for such a great tool!

I have a real "newbie" question.

When generating the static files by issuing the command:

bin/plugin generate http://localhost:8080 -d https://example.com -p blackhole -f

I'm only finding ten pages get written into the "blockhole" folder. By manually specifying a route to other pages then I can get a static file for them generated.

Am I missing something obvious here? Do I need to manually specify the route to each page to be extracted, or should the tool be able to crawl all of them?

Feature request: Extract only what is needed

Greetings.
I like the idea of Blackhole, more than any performance considerations the security it can bring.
For that latter, it would be nice if Balckhole was isolating only the parts that are needed. Ie, such that you end up with a copy of html and css files (ie, no php, no yaml...) in a separate location. Basically, it should lead to the possibility to run grav "inhouse" and create a readonly copy on a public server where theres no php installed.
Thanks and best regards.

Fonts are not build when assets is call

Description

Hello there, i just found your plugins to export grav to static HTML.

And i just realize that fonts are not compile in the build folder when we call the generate function
bin/plugin blackhole generate http://127.0.0.1:8000/ --assets --output-path ./build

Reproduce

1. Install the default version of Grav theme with the default page
2. execute the generate command
3. visit the build folder \build\user\themes\quark\fonts is missing

thinking

I think all fonts is an assets to ? or im wrong ^_^ im not sure

thanks for all your work body 🙌

Tag links not working

Article tags don't work. I'm using Gantry 5 framework and Helium theme.

The link generated is <a href="[/tag:projects](https://domain.com/tag:projects)" class="p-category">projects</a>, clicking on this generates a 404.

generate php files?

While most of the websites I work on are very low maintenance and don't change much, occasionally a simple event announcement thing is useful. Working with Jekyll I just put a bit of php code in that checks the date and only displays the announcement if the event is still in the future. However, I can't seem to do this when using Blackhole. How feasible would it be to incorparate a check for such code in Blackhole and generate php files instead of html?

Again, I may be able to contribute to this but I'm not yet entirely sure how. I love to learn though.
Anna~

new feature idea: upload to another server

I have recently discovered Grav and I'm loving it to bits. And Blackhole makes it even better for me, because I mostly work on small to tiny websites that don't change all that much (I have mostly used Jekyll in the past). So in my workflow it would be absolutely perfect if I could host a client's Grav site under a subdomain of my own website, let them make their edits there and check what the result looks like before telling Blackhole to generate the files and push them directly to their own domain. This would in one fell swoop also solve the problem of not being able to preview unpublished changes (my clients are generally not tech savvy, and preview is good).

Does this sound doable and like a good idea to you? I may be able to contribute this bit myself, though that'd involve a bit of a learning curve. Your opinion on this is much appreciated.

Thanks for your time and for making Blackhole!
Anna~

Possible to generate without access to the running Grav site?

Just wondering if the plugin NEEDS access to an actual running Grav site to work, or if there's any chance that with some tweaking it could just run against the codebase without Grav ever actually running.

I'm assuming that there's a good reason for needing a running website, and that it'd be really difficult if you didn't have that, but wanted to verify that before I went down the path of trying to make that work.

Thoughts?

Validation failed: enabled is not defined in blueprints

Getting this message from within the Admin plugin when I try to save blackhole's settings.

Got this both on localhost and on the deploy server.

I'm assuming there is some configuration problem within the blackhole blueprints.yaml.

Few questions/issues for publishing static site to Netlify

Hey, this is looking really good. Very close I think to a good production V1. I tested it out yesterday taking a GRAV site that uses GANTRY, to generate a static site that I could host on netlify. Couple blocking issues and a few "nice to have's" below. If anyone is interested and able to help with this I would be happy to pay for the time or contribute something as compensation.

Blocking Issues

• Images and assets (JS, CSS) are being exported with the cached ?5897135 appended to the files. I dont know if there is a setting or something I could do in Gantry or Grav (like turn cache off maybe, will test that) but all static content and files need to terminate in a filetype, like .css, .js, .png, etc
• Absolute URL is used to link all images and assets. So let's say my site is "site.test" everything in the static site right now becomes http://site.test/user/plugins/gantry5/theme/filename.css?59483. This really should be relative, so the result would be /path/to/file.css
• Only the first generate command words. If I try to generate a static site after the build path is already established, it doesnt add/update existing files even if the FORCE option is selected. it starts creating new folders with random names.

Nice To Haves

• Organized Assets. Would be nice if we could have one assets folder with CSS, JS, FONTS resources all aggregated into one spot rather than copying all of the paths verbatim.
• HTML Formatter / Cleaner - the output of the files is a bit messy. Would be great to integrate a library that cleans the HTML, CSS and JS so everything is tabbed nicely.

no output generated…

i ran bin/plugin blackhole gen ./static but nothing is generated… any ideas?

i am running grav 1.1.8 and the latest plugin version?

Question: Running on Localhost?

Hi

I am trying to run this via localhost.
C:\wamp64\www\htdocs1>php bin/plugin blackhole generate localhost/htdocs1 static
ERROR Blackhole failed to start.
ERROR The website must match the location of your Grav installation.
ERROR You must have at least one page.

Any ideas? I can in my browser go to localhost/htdocs1 and see the homepage.

case "asset was changed: copy the new one" does not work

error:

Warning: filemtime(): stat failed for UsersgnaranjoDocuments…
cms/user/plugins/blackhole/cli/GenerateCommand.php on line 215

commenting the case and starting from scratch worked for me

Usage in Admin

Hi, this looks useful but I'm on shared hosting so I can't run it directly. You say "The generate command can be used ..... in the Grav Admin Panel." but I don't see how to do it in the Admin panel, please could you guide me?

blackhole produces empty index.html

generic brand new installation with test page theme skeleton. trying to export delivers empty /build index.html

PHP Fatal Error with Option -a

I'm currently trying to get this running in a local grav docker instance (grav 1.7.44).
However, I did not get a usable Output so far.
I tried
bin/plugin blackhole gen http://localhost -p ./scripts/ -f -a from the grav root, then I get a whole tree with index.html files in folder scripts, gut all are empty (0 bytes size).
In Addition, generation ends with
PHP Fatal error: Uncaught Error: Class "DOMDocument" not found in /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/functions.php:42 Stack trace: #0 /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/functions.php(85): tidal_disruption() #1 /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/cli/GenerateCommand.php(200): assets() #2 /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/vendor/RollingCurl/RollingCurl.php(267): Grav\Plugin\Console\GenerateCommand->Grav\Plugin\Console\{closure}() #3 /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/cli/GenerateCommand.php(206): RollingCurl\RollingCurl->execute() #4 /mnt/docker/docker/Grav/Playground/web/system/src/Grav/Console/ConsoleCommand.php(33): Grav\Plugin\Console\GenerateCommand->serve() #5 /mnt/docker/docker/Grav/Playground/web/vendor/symfony/console/Command/Command.php(255): Grav\Console\ConsoleCommand->execute() #6 /mnt/docker/docker/Grav/Playground/web/vendor/symfony/console/Application.php(1039): Symfony\Component\Console\Command\Command->run() #7 /mnt/docker/docker/Grav/Playground/web/vendor/symfony/console/Application.php(275): Symfony\Component\Console\Application->doRunCommand() #8 /mnt/docker/docker/Grav/Playground/web/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun() #9 /mnt/docker/docker/Grav/Playground/web/system/src/Grav/Console/Application/PluginApplication.php(80): Symfony\Component\Console\Application->run() #10 /mnt/docker/docker/Grav/Playground/web/bin/plugin(43): Grav\Console\Application\PluginApplication->run() #11 {main} thrown in /mnt/docker/docker/Grav/Playground/web/user/plugins/blackhole/functions.php on line 42
which seems weird, but even if I omit the '-a' Flag, I still get only empty index.html files.
and, yes, all involved directories are writable.
is this known behavior, or am I just doing something wrong ?

Feature wishlist from msharron

I'm carrying this list over from #11 as that issue will be closed pending @msharron's review. Anyway, I'm currently working on these.

• Render via user specified urls (quite helpful ;) )
• Only render pages that have changed/been modified.

I think the second one there was fulfilled in v0.6.0. Correct me if I'm wrong.

How to "blackhole" (make static) particular folders only?

Probably there is already a solution for this, otherwise please consider this as an idea.

So, how about an option to make not the whole website static, but just a particular folder(s) in user/pages? I.e. by checking --output-url not only for a domain name, but also for a path or by accepting another setting with a list of folders somehow...

Here is the example of a use case:
Let's say, there is a knowledge base website. And there is is a folder /user/pages/locations with 30000+ of pages inside it with hamlets' names and some static info (map, coordinates etc). It is used a bit like a taxonomy (some other pages just have static links over here), but location pages themselves are upadated very rarely. From the other hand, Grav have been reported to have performance slow downs because of caching a lot of files. So, performance-wise it might be reasonable to make such a rarely updated pages static (and i.e. just put them into the website root as /locations/ to override Grav's preccessing), but keep the rest of the website "alive" with Grav.

This example might look pretty tricky, but the general idea is to have an option to have rarely updated pages as static ones for better performance. Hope you get it. Thanks.