Long build times - what could be the reason?

[stalkerGH]

Long build times - what could be the reason?

[stalkerGH]

Thank you for the hint. Maybe I can try, just for testing purposes, on the site copy. But first I should convert all .webp to .png. I chose this format because of smaller files. I think I can withstand a longer build time - until it reaches an hour or so :)

[stalkerGH]

Side effect is that content directory swelled from 66 to 507 MB :)

[welpo]

Build time is stunning

Yay! 🎉

Maybe I could optimize my PNGs

I've had great (lossless) results with oxipng. pngquant looks promising if you want lossy compression as well.

[stalkerGH]

Can I do something more yet?

[welpo]

Thanks again for helping with this. Let's try this:

If you are using absolute paths:

{%- set read_metadata = read_metadata | default(value=true) -%}

{#- Set paths based on whether the src is remote or local -#}
{%- if src is starting_with("http") -%}
    {%- set image_url = src -%}
{%- elif read_metadata -%}
    {%- set colocated_path = page.colocated_path | default(value="") -%}
    {%- set relative_path = colocated_path ~ src -%}
    {%- set meta = get_image_metadata(path=relative_path, allow_missing=true) -%}

    {#- Fallback to absolute path if relative path doesn't work -#}
    {%- if not meta -%}
        {%- set meta = get_image_metadata(path=src, allow_missing=true) -%}
        {%- set image_url = get_url(path=src) -%}
    {%- else %}
        {%- set image_url = get_url(path=relative_path) -%}
    {%- endif -%}
{%- else -%}
    {%- set image_url = get_url(path=src) -%}
{%- endif -%}

{%- set lazy_loading = lazy_loading | default(value=true) -%}

<div class="full-width">
    <img src="{{ image_url }}"{% if alt %} alt="{{ alt }}"{% endif %}{% if read_metadata and meta.width %} width="{{ meta.width }}"{% endif %}{% if read_metadata and meta.height %} height="{{ meta.height }}"{% endif %}{% if lazy_loading %} loading="lazy"{% endif %}/>
</div>

If you want to test with relative paths:

{%- set read_metadata = read_metadata | default(value=true) -%}

{#- Set paths based on whether the src is remote or local -#}
{%- if src is starting_with("http") -%}
    {%- set image_url = src -%}
{%- else -%}
    {%- set colocated_path = page.colocated_path | default(value="") -%}
    {%- set relative_path = colocated_path ~ src -%}

    {%- if read_metadata -%}
        {%- set meta = get_image_metadata(path=relative_path, allow_missing=true) -%}

        {#- Fallback to absolute path if relative path doesn't work -#}
        {%- if not meta -%}
            {%- set meta = get_image_metadata(path=src, allow_missing=true) -%}
            {%- set image_url = get_url(path=src) -%}
        {%- else %}
            {%- set image_url = get_url(path=relative_path) -%}
        {%- endif -%}
    {%- else -%}
        {%- set image_url = get_url(path=relative_path) -%}
    {%- endif -%}
{%- endif -%}

{%- set lazy_loading = lazy_loading | default(value=true) -%}

<div class="full-width">
    <img src="{{ image_url }}"{% if alt %} alt="{{ alt }}"{% endif %}{% if read_metadata and meta.width %} width="{{ meta.width }}"{% endif %}{% if read_metadata and meta.height %} height="{{ meta.height }}"{% endif %}{% if lazy_loading %} loading="lazy"{% endif %}/>
</div>

Both shortcode do the same, except one ONLY works with full paths, and the other with relative paths. Use whichever requires less work for you.

This is the current shortcode for full width with an extra option: read_metadata. If set to false, it will skip reading the metadata.

If possible, can you test this with webp?

1. Change the code to your shortcode
2. Check how much time it takes to build with render_metadata=false (should be super fast)
3. Check how much time it takes to build with render_metadata=true (should be slow)

Make sure images render in both cases. The shortcode should work with both absolute and relative paths.

An example of calling the shortcode (first version, with full paths) with this option:

{{ full_width_image(src="blog/shortcodes/img/amsterdam_by_oskerwyld.png", read_metadata=false) }}

[welpo]

Thank you for reporting back!

In conclusion: WebP is the culprit.

I'm not sure what to do. I could add the read_metadata options to all image shortcodes, but it feels like a hack and not a proper solution. I've replied to the Zola repo discussion in hopes of figuring out how how this could really be fixed.

I'll wait for an answer before deciding. If fixing this on Zola/a library Zola uses is not feasible, I'll probably add the feature to the shortcodes and document it. I'm thinking a general config.toml setting would help make things simpler if I add this.

[welpo]

Thank you so much for reporting back!

I tested my site (with a bunch of webp files) and it's much faster building as well. Yay! 🎉

[welpo]

Hi!

Please ensure you're using the latest version, as a recent update (#325) greatly improved build times.

Let me know whether you're indeed using tabi with the fix, please.

[stalkerGH]

Yes, I got the latest version, with changes in templates/macros/settings.html.

[welpo]

Thanks for confirming.

Is it using image processing macros? Those tend to increase build times.

[stalkerGH]

Hmmm, I don't know... How can I check it? Most of my images are .webp plus some .png.

[welpo]

Are you using the image shortcodes? Or image resizing?

[stalkerGH]

Yes, I'm using image shortcodes, mostly image_toggler and full_width_image. Some images have full_width and inline. Do you think this is the reason?

[welpo]

Image processing is costly. I'm not finding the exact discussion where I learnt this, but this is related: getzola/zola#2106

Do you think this is the reason?

It's likely. Try to remove the image shortcodes and we'll see the difference.

[stalkerGH]

Thanks for advice. I have plenty of images (~100 currently). So maybe it is the case. I'll try to mark some pages as draft and try to build page.

[welpo]

Let me know how it goes!

[stalkerGH]

I disabled (by draft = true) two pages with about 30 images each and build time dropped from 15 to 5 seconds. So it's not a bug - the images are "the problem".

Thank you!

[stalkerGH]

My "problem" is solved so I close the issue.

[welpo]

I'm glad you figured it out! Thanks for reporting back.

If build time is important, you could try using png instead of webp; the Zola discussion I linked mentions that might be faster (as it can read only metadata, not the whole file).

[stalkerGH]

OK, I was so curious about this thing that I:

1. made a backup of whole project,
2. batch converted all .webp to .png,
3. removed all .webp,
4. changed all extensions .webp to .png in .md files,
5. build the page again.

But I got the message:

Error: Failed to serve the site
Error: Failed to render content of /path/to/index.md
Error: Reason: Failed to render image_toggler shortcode
Error: Reason: Failed to render 'shortcodes/image_toggler.html'
Error: Reason: Function call 'get_image_metadata' failed
Error: Reason: `resize_image`: Failed to read image: /path/to/image.png

But the file exists and I can read/display it. pnginfo shows it is valid.

[welpo]

I've had issues with seemingly OK images not working well with Zola. I would try changing the problematic png with another one, preferrably one that you haven't converted yourself.

[stalkerGH]

I've had issues with seemingly OK images not working well with Zola. I would try changing the problematic png with another one, preferrably one that you haven't converted yourself.

I've done as you recommended. There were three problematic PNGs. I converted them from PNG to PNG ;) and Zola started it's work. Build time is stunning:

Building site...
Checking all internal links with anchors.
Successfully checked 1 internal link(s) with anchors.
-> Creating 12 pages (0 orphan) and 3 sections
Done in 226ms.

Maybe I could optimize my PNGs. Will try if it is worth of effort.
`

[stalkerGH]

I've had great (lossless) results with oxipng. pngquant looks promising if you want lossy compression as well.

Thank you very much! :)

[stalkerGH]

If build time is important, you could try using png instead of webp; the Zola discussion I linked mentions that might be faster (as it can read only metadata, not the whole file).

I reopen the issue. If I understand properly, the problem is in WEBP files. When I use PNG, site is build instantly after change detection. Is there some method or hack to read WEBP metadata as it is done for PNG files?

[welpo]

Is there some method or hack to read WEBP metadata as it is done for PNG files?

That would need to be on the Zola side of things. You might want to ask on their issue tracker.

On the tabi side, we could try to modify the image shortcodes. At the cost of potential rendering issues, you could try editing the shortcodes so they don't read image metadata.

For example, a simplified dimmable image shortcode that doens't read metadata (and also doesn't work with relative paths):

{%- set lazy_loading = lazy_loading | default(value=true) -%}

{% if full_width | default(value=false) %}
    <div class="full-width">
{% endif %}
<img class="dimmable-image" src="{{ get_url(path=src) }}"{% if lazy_loading %} loading="lazy"{% endif %}{% if alt %} alt="{{ alt }}"{% endif %}/>
{% if full_width | default(value=false) %}
    </div>
{% endif %}

For the older version of image shortcodes, prior to the relative path feature (which forces metadata reading), see this.

If you edit the shortcode you're using to not use metadata, and it speeds up the webp situation, we might consider what to do next. I'm thinking a ignore_metadata option could be added to shortcodes, for example.

[stalkerGH]

Thanks for in-depth explanation. I made some experiments. First, I changed the shortcodes removing this part:

{% if default_meta.width %} width="{{ default_meta.width }}"{% endif %}{% if default_meta.height %} height="{{ default_meta.height }}"{% endif %}

But it changed nothing - at least nothing what I could catch.

Then I downloaded older version of full_width.html and image_toggler.html which I use in my code. This improved things a lot - site builds in miliseconds.

Because both - old and current (latest) - .html files include metadata reading code, in my opinion problem is not related to metadata itself. It has something to do with checking the paths to images. But if you don't have time to investigate it or just don't think it is some crucial thing to solve, it's OK. Maybe this is only my "problem". If this "hack" - reverting to old version of .hmtl files - works, it is sufficient for my needs (as long as it works).

[stalkerGH]

It was more complicated that I thought in the very beginning. I didn't realized that after reverting to old versions of .html files images stopped being rendered. So I had to adapt the code to my needs and remove everything which I don't use - for example remote URLs for images.

PS. I'd like to attach my patches but don't know how to do that - I don't know if they are usable but maybe you will be curious to know what I messed up in your code :)

[welpo]

It's not clear to me what the culprit is, then. Is it the relative path handling? Did you figure out a way to speed it up WITH metadata reading (and image rendering)?

I'd like to attach my patches but don't know how to do that

You could link to your repo files (if public) or just paste the code here.

[stalkerGH]

OK, I will paste whole files (not the patches) in separate messages. I don't know if it has or not something in common with metadata - I suspect rather relative path.

[stalkerGH]

full_width_image.html

{%- set colocated_path = page.colocated_path | default(value="") -%}
{%- set relative_path = colocated_path ~ src -%}
{%- set image_url = get_url(path=relative_path) -%}

{%- set lazy_loading = lazy_loading | default(value=true) -%}

<div class="full-width">
    <img src="{{ image_url }}"{% if alt %} alt="{{ alt }}"{% endif %}{% if lazy_loading %} loading="lazy"{% endif %}/>
</div>

[stalkerGH]

image_toggler.html

{# The `random_id` ensures that each instance of the shortcode has a "unique" id #}
{# allowing individual interactive elements (like toggles) to function correctly. #}
{# This avoids conflicts when multiple instances of the shortcode are used. #}
{%- set random_id = get_random(end=100000) -%}
{%- set colocated_path = page.colocated_path | default(value="") -%}
{%- set lazy_loading = lazy_loading | default(value=true) -%}
{%- set inline = inline | default(value=false) -%}

{#- Determine the class for the images -#}
{#- Necessary for inline images -#}
{%- set tag = "div" -%}
{%- if inline -%}
    {%- set tag = "span" -%}
{%- endif -%}

{%- set img_class_list = "" -%}
{%- if inline -%}
    {%- set img_class_list = img_class_list ~ " inline" -%}
{%- endif -%}

{%- set relative_default_path = colocated_path ~ default_src -%}
{%- set default_image_url = get_url(path=relative_default_path) -%}

{%- set relative_toggled_path = colocated_path ~ toggled_src -%}
{%- set toggled_image_url = get_url(path=relative_toggled_path) -%}

<{{ tag }} class="image-toggler-container {% if full_width %}full-width{% endif %}">
    <input type="checkbox" id="toggle-img-{{ random_id }}" class="image-toggler-toggle">
    <label for="toggle-img-{{ random_id }}" class="image-label">
        <{{ tag }} class="image-default">
            <img class="{{ img_class_list }}" src="{{ default_image_url }}"{% if lazy_loading %} loading="lazy"{% endif %}{% if default_alt %} alt="{{ default_alt }}"{% endif %}>
        </{{ tag }}>
        <{{ tag }} class="image-toggled">
            <img class="{{ img_class_list }}" src="{{ toggled_image_url }}"{% if lazy_loading %} loading="lazy"{% endif %}{% if toggled_alt %} alt="{{ toggled_alt }}"{% endif %}>
        </{{ tag }}>
    </label>
</{{ tag }}>

[welpo]

I don't know if it has or not something in common with metadata - I suspect rather relative path

Neither of the two files reads the metadata, so we can't tell.

Try this for image toggler and this for the full width.

Those versions do read the metadata, but don't try to load relative paths. Note: they require you use absolute paths on the shortcodes, though.

Can you test whether this changes build times?

[stalkerGH]

I tested both files now and in the morning - they don't render images. Build seems to go but images are broken in the webpage.

[welpo]

The shortcodes did work, so there's most likely a problem with the path. Are you using an absolute path? E.g. blog/your_post/graph.webp

[welpo]

If you're hosting the site on a public repo (or wouldn't mind sharing it with me/the world), feel free to drop a link; I could test this stuff myself. I appreciate your time~

[stalkerGH]

The shortcodes did work, so there's most likely a problem with the path. Are you using an absolute path? E.g. blog/your_post/graph.webp

No, i use relative path: just image.webp. But when I use the .html files which you pointed me earlier to, with absolute path, the site is built as expected - the images are rendered.

[stalkerGH]

If you're hosting the site on a public repo (or wouldn't mind sharing it with me/the world), feel free to drop a link; I could test this stuff myself. I appreciate your time~

Unfortunately I don't use a repo for my website - it is public domain but no source available.

[welpo]

But when I use the .html files which you pointed me earlier to, with absolute path, the site is built as expected - the images are rendered.

And how's the speed with webp files?

[stalkerGH]

And how's the speed with webp files?

Very fast. 15 pages, about 150 images (including several PNGs) - build time is 250-370 ms.

[welpo]

And if you switch from the old shortcodes to the current version, without changing the png/webp files or paths, it slows down?

[stalkerGH]

If you mean current as latest in repo then answer is - yes, it is much slower, about 15 seconds.

[welpo]

Just to triple check: it's slow even when using absolute paths? (with current shortcodes)

[welpo]

Actually, I'm still confused: current shortcodes were fast with PNG files, right? Have we isolated all variables?

[stalkerGH]

Actually, I'm still confused: current shortcodes were fast with PNG files, right? Have we isolated all variables?

Yes, build time was much faster with PNGs, with latest versions of .html shortcodes.

[stalkerGH]

Just to triple check: it's slow even when using absolute paths? (with current shortcodes)

Yes, I just checked it out. I disabled most of files with draft = true and build time was 2.4 seconds. Then I enabled one file with ~40 WEBP images, added absolute paths to all images and build time is 3 times longer - 7.6 seconds.

[welpo]

Thank you for all this testing.

Can you do one more? Let's try this full width:

{%- set meta = get_image_metadata(path=src, allow_missing=true) -%}
{%- set image_url = get_url(path=src) -%}

{%- set lazy_loading = lazy_loading | default(value=true) -%}

<div class="full-width">
    <img src="{{ image_url }}"{% if alt %} alt="{{ alt }}"{% endif %}{% if meta.width %} width="{{ meta.width }}"{% endif %}{% if meta.height %} height="{{ meta.height }}"{% endif %}{% if lazy_loading %} loading="lazy"{% endif %}/>
</div>

This removes the relative path logic. My hypothesis:

• It will be slow with webp (slower than this)
• It will be fast with png

[stalkerGH]

I used above code, disabled all pages (draft = true) and build time was ~250 ms. Then I enabled one page with many images and build time was ~2.5 seconds. All with WEBP images.

[stalkerGH]

When I converted all WEBPs to PNGs in a directory which belongs to one and only file which was enabled to build, build time is ~250 ms. So something is "broken" with WEBPs.

EDIT: no so fast... No images are rendered. Just for case I made this operation: oxipng -o 4 --strip safe --alpha *.png - no render at all. Coming back to WEBP - rendering as expected.

EDIT no. 2: images not render if path is relative but rendering is OK whet path is absolute.

[welpo]

No images are rendered

Did you fix this? I can render PNGs with the shortcode I posted.

images not render if path is relative but rendering is OK whet path is absolute.

Ah, yes, that's expected.

---

So, to clarify: did you try rendering many PNGs?

[stalkerGH]

I converted WEBPs to PNGs only in the directory where the site enabled for rendering is seated. All other pages are disabled by draft = true so I didn't converted these WEBPs to PNGs. Is that what you are asking for?

[welpo]

Yes, that's fine. I'm trying to verify the hypotheses:

• It will be slow with webp (slower than this)

• It will be fast with png

We confirmed it's slow with webp, and I'm 99.9% sure it's fast with PNG, but wanted to confirm.

[stalkerGH]

Thank you for the effort. If you allow, I will conduct tests tomorrow. Sometimes you need to sleep :)

[welpo]

Of course :) No rush at all~

[stalkerGH]

Good morning! I'm back, ready to testing.

Good news: your change works!

With read_metadata = true option it takes 10.5 seconds to render 15 pages.

With the option read_metadata = false it takes 3.5 seconds to build. It's still a long way from milliseconds, but that's probably because in addition to the full_width_image code I'm still using image_toggler, which requires analogous changes.

[stalkerGH]

I see no reaction on Zola's issue tracker. Maybe because it's very old thread - over 1 year? Maybe I should start new one?

EDIT: after some digging, I got great news. Something has changed in Zola's internals because rendering is incredibly fast even for WEBP images. I installed 0.19.0 but zola serve wasn't working. I've noticed that it is a bug, fixed in 0.19.1. I've installed it and my page is rendered in milisecongs. I've reverted to original shortcodes (vanilla) of tabipackage and it works like a charm. Also with versions with read_metadata = false but it is not necessary now, I think.

So, it seems the "the bug" or whatever it was is repaired. Great :)

[stalkerGH]

Thank you for great work and your willingness to help!