kiliman/tailwindui-crawler

tailwindui-crawler downloads the component HTML files locally

tailwindui-crawler

This script will crawl the tailwindui.com website and download all the components to the ./output folder.

🛠 How to use

To install, clone this repo and run yarn or npm install to pull down the dependencies.

Then create a .env file with your email, password, and optional output folder.

EMAIL=youremail
PASSWORD=yourpassword
# OUTPUT optional, defaults to ./output
OUTPUT=/path/to/output
# LANGUAGES defaults to html
LANGUAGES=html,react,vue,alpine
# COMPONENTS commma-delimited list (defaults to all)
COMPONENTS=(all|marketing,application-ui,ecommerce)
# BUILDINDEX generate index file to view components offline
BUILDINDEX=(0 | 1)
# TEMPLATES download template files
TEMPLATES=(0 | 1)

NOTE: The tool uses dotenv-expand to support variable expansion like $HOME/path/to/output so if your password or any other value includes a $, make sure you add a \ (backslash) to escape the $. For example, PASSWORD=p@\$\$w0rd

Also, dotenv does not support inline comments, so do not do something like LANGUAGES=html,react,vue # some comment as it will not get the correct values.

Then finally, run yarn start or npm start

The script will login to tailwindui.com with your credentials, and download all the components as individual files in the ./output folder.

🤔 What's it for?

The benefit of pulling down all the components is that you can commit them to a local or private repo, and by running this periodically, you can see exactly which files were added or changed. Hopefully, some time in the future, they will open up a private repo for those that have purchased the library.

🚀 New v4.0

The crawler has been updated to support the new Tailwind UI site as of 2022-06-27. You can also download the new Page Templates. Add TEMPLATES=1 to your .env file. If you are using the GitHub action to crawl, you will need to update your deploy.yml. See instructions below.

The crawler now supports the new Tailwind UI site and can download HTML, React and Vue versions of the components.

You can also download the "alpine" version of the components. ⚠️ WARNING: the alpine code is NOT production ready. It does not support accesibility and is used to show a preview of how the component interactivity can be implemented.

It also adds the ability to generate an index page that emulates the tailwindui.com website so you can browse components offline.

🗂 Preview page

You can set the .env key BUILDINDEX=1 to have the crawler generate an index file similar to the components page on tailwindui.com. Install and run the serve package to view the index.

NOTE: The HTML Preview does not apply transformers. It's a copy of the component site on tailwindui.com.

yarn global add serve
cd $OUTPUT/preview # change to your preview folder
serve

⚙️ Example .env file

EMAIL=******
PASSWORD=******
OUTPUT=$HOME/Projects/tailwindui
LANGUAGES=html,react,vue,alpine
COMPONENTS=marketing,application-ui
BUILDINDEX=1
TEMPLATES=1

🤖 Automatically keep a private GitHub Repository up-to-date

NOTE: GitHub action has been updated in v4.0.0. Please make sure your default.yml file is updated with the latest actions.

You can automatically keep a private GitHub repository up-to-date with component changes from TailwindUI by using this tool with GitHub Actions.

1. Create a private GitHub repository.

2. Add TAILWINDUI_EMAIL and TAILWINDUI_PASSWORD secrets to the GitHub repository.

3. Optionally create a .env file with additional settings for the crawler.

4. Create a new file .github/workflows/default.yml:

   name: Update
   on:
     schedule:
       - cron: '0 0 * * *' # Every day at midnight
   
   jobs:
     update:
       name: Update
       runs-on: ubuntu-latest
       steps:
         - name: Checkout
           uses: actions/checkout@v2
         - name: Run crawler
           uses: kiliman/[email protected]
           with:
             email: ${{ secrets.TAILWINDUI_EMAIL }}
             password: ${{ secrets.TAILWINDUI_PASSWORD }}

   NOTE: Make sure to update to the latest action v1.1.0 to support the crawler v3+

   Read more about the schedule cron syntax in the official GitHub Actions documentation.

   NOTE: if you're creating a new repository or have updated your default branch from master, you will have to specify with branch (and/or current_branch) like so:

   # ...
         - name: Run crawler
           uses: kiliman/[email protected]
           with:
             email: ${{ secrets.TAILWINDUI_EMAIL }}
             password: ${{ secrets.TAILWINDUI_PASSWORD }}
   +         branch: main
   +         current_branch: main

Email Notifications

To be emailed whenever there is a change to a component, simply setup GitHub Notifications on your repository.

🚦 Upgrading to v3.

This is a major change. Unfortunately, v2 will no longer work with the existing site due to the updates they may to add support for React/Vue components. Please also note that the GitHub Action has been updated from v1.0.0 to v1.1.0.

Currently, there is no support for transformers, as the need for them is not as critical since the components don't need to be converted to React or Vue.

NOTE: Since this script is essentially screen scraping, there's the potential of it breaking if the HTML structure changes. I will do my best to keep it in sync with the website.

😍 Thank you

Thanks to Adam and Steve for making an amazing library. This has definitely made creating a UI for my applications a pleasant experience.

Enjoy and let me know if you have any questions.

Kiliman

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Kiliman 💻	Simon Waloschek 💻	Pavel Fomchenkov 💻	Robin Malfait 💻	Miguel Piedrafita 💻 📖 🤔	Vlad Dumitrescu 📖	C-Bass 💻
Greg Brimble 📖 🔧	Yagnik 💻	idebeijer 💻

This project follows the all-contributors specification. Contributions of any kind welcome!

tailwindui-crawler's Issues

Update GitHub Action to support new crawler v3

Make sure the action uses the latest version of the crawler, as v2 will not properly parse the updated Tailwind UI site.

/cc @GregBrimble

Use local build / tailwind and postcss config file instead of CDN

After I run serve from within the output directory, everything magically works!

As a suggestion, I don't see a tailwind config file anywhere...and it definitely seems like that is the recommended option:
https://tailwindui.com/documentation#add-the-tailwindcss-ui-plugin
https://tailwindui.com/documentation#update-your-purgecss-configuration

Would it be possible to get this working with a local tailwind config file and PurgeCSS configuration, so we eliminate the CDN?
https://cdn.jsdelivr.net/npm/@tailwindcss/ui@latest/dist/tailwind-ui.min.css

Once again, many thanks for the repo!

index.html links are absolute (2.3)

This makes it hard to just browse the files, a server is needed...

Change in HTML layout of component page causes filenames to be wrong

Example:

Application UI = > Application Shells should have the folders inside

1. Stacked Layouts (9 components)
2. Sidebar Layouts (8 components)
3. Multi-Column Layouts (7 components)

But instead of this folders, crawler returns just three html files

https://prnt.sc/1qqd47x

[Bug] Crawling seems broken (empty body tags)

HTML output without using any transformer:

<html>
<head /><body /></html>

GitHub Action

Hello! Thank you for creating this excellent tool. It has already saved me many hours!

I have created a GitHub Action to automatically run this project, and push the output to a GitHub repo. This allows users to maintain a git version controlled copy of all the components, and for it to keep itself up-to-date.

It also supports the .env file for this project, so you can run all your transformers, and login info is kept securely in secrets.

I'd appreciate any feedback/input/questions you might have, and would love to put this somewhere on your README.md, if you think it could help others.

Preview Header on Safari Broken Height

Not sure if this would even be the right place for this, but I'm seeing some strange behavior when browsing the index pages for components after generating.

When using Safari there seems to be some really strange behavior going on with the h-full class on the vertical spacer.

When comparing against the elements on the actual TailwindUI site I'm seeing the same classes, but the behavior is correct.

It also seems to be just in Safari, as the local pages look just fine when using Brave.

Any thoughts? I feel like I'm missing something obvious but can't seem to pin it down.

New Version 3 ideas

Just posting this here to put down my thoughts on what the next version of the crawler will be. Calling this v3, since it will have some breaking changes.

• Always store the original interactive (alpine JS) versions and HTML comment version regardless of transformers. This is to ensure that we can still display the component properly locally (offline). Also, Adam adds additional markup around the component to make it look better in the preview. This markup is not present in the HTML comments version, so it will not look correct in that mode. Transformers will store their output independent of these original versions.
• Ability to apply transformers from offline viewer. That is, the user can apply the changeColor transformer while viewing the component in Preview mode. This will apply the transformer on the cached alpine version.
• Update config to use JSON format. Still keep .env for credentials, but the rest should be in a more readable and extensible format.
• Add a configurable workflow that can script out tasks, like generating the tailwind CSS from the config. Generating a list of class names for use by the prefixClasses transformer, etc.
• Finish up transformers for adding transition support for Vue and React. Make it extensible enough that it would be relatively easy to customize for Angular and other frameworks.

If anyone has any comments, please feel free to post.

thankyou X 10 million! 🤩🙏🏾

i too just wanted to say thank you for making this, and for making it available... 🤩🙏🏾

CSS not applied

Thanks for this great contribution.
When yarn is done, the output looks pretty okay regarding file-structure, however,This something is missing I guess:

This is the .env file: https://d.pr/i/dnmiw8

This is the output: https://d.pr/i/n6TnuX

This is the output in browser (using Serve): https://d.pr/i/cnBQwm

It looks like the CSS is not applied. - As I understand, it defaults to TailwindCSS CDN if I'm not specifying local CSS.

Can anyone point me in the right direction here? - Thanks

Crawler hangs on download or throws ETIMEDOUT error (REVISITED)

Today, the ETIMEDOUT error was back again. Difference is that I crawled all languages today (instead of just HTML), but I don't think it has anything to do with the timeout, as a lot of directories were processed correctly before:

‼️ FetchError: request to https://tailwindui.com/img/category-thumbnails/sections/content-sections.png failed, reason: connect ETIMEDOUT 172.67.217.20:443
at ClientRequest. (D:\laragon\www\xxx-tailwindui-crawler\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (events.js:315:20)
at TLSSocket.socketErrorListener (_http_client.js:469:9)
at TLSSocket.emit (events.js:315:20)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
type: 'system',
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT'
}

(Maybe it would be helpful to show the timeout value as part of the error output by default?)

My actions started failing a few days ago, thoughts? I ran it manually with debug

[main] Update tailwindui.com
145 files changed, 878 insertions(+), 878 deletions(-)
Pushing changes...
error: RPC failed; HTTP 408 curl 18 HTTP/2 stream 7 was reset
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Run crawler

Cannot read property 'replace' of undefined

Vue Transformer

See #7

Some components' pages dont match tailwind UI

After installing and running the tailwindui-crawler, some components' pages dont match those of tailwindui. For example:
http://localhost:5000/components/application-ui/data-display/description-lists
Looks very different from
https://tailwindui.com/components/application-ui/data-display/description-lists

Error: ENOENT: no such file or directory, open 'outputmailto:[email protected]?subject=Upgrade to get Marketing/index.html

FOR FUTURE REFERENCE:
Package seem to be broken when your license doesn't have the marketing components as well.
To quickly fix it in those situations modify:index.js:167 to:
const links = $('#product-application-ui .grid a.block.group')

Not logged in error

This is what I am getting when I run this.

npm start

[email protected] start
node index.mjs

🔐 Logging into tailwindui.com...
🔍 Fetching https://tailwindui.com/login
⏱ 326ms (200)
✅ Success!

🗂 Output is ./output
🔍 Fetching https://tailwindui.com/components
⏱ 687ms (200)
⏳ Processing /components/marketing/sections/heroes...
🔍 Fetching https://tailwindui.com/components/marketing/sections/heroes
⏱ 579ms (200)
🚫 Not logged in

Syntax error?

yarn run v1.22.15
$ node index.mjs
file:///Users/xxx/workspace/tailwindui-crawler/index.mjs:79
      port: uri.port ?? 443,
                      ^

SyntaxError: Unexpected token '?'
    at Loader.moduleStrategy (internal/modules/esm/translators.js:140:18)
    at async link (internal/modules/esm/module_job.js:42:21)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Integrate with vue-tailwind-prefix-applicator to customize the tailwind prefix

This is not my idea and it sounds like you've already got the integration with vue-tailwind-prefix-applicator working:
vesper8/vue-tailwind-prefix-applicator#1

Can't wait to have this functionality become available so I can add a prefix to the components automatically. I imagine this will be another transformer and .env configuration option?
Once again thank you!

Show component images on the Index.html page

Another one (nice to have) that builds on #12

It would be nice if we could download and show component images on the Index.html page (like the tailwindui.com index page). That would make it easy to (visually) tell what each component is about.

Examples:
https://tailwindui.com/img/category-thumbnails/sections-heroes.svg
https://tailwindui.com/img/category-thumbnails/sections-pricing.svg
etc

Thanks for considering!

Are you sure this package still works?

Hi,

I tried to download the Alpine js versions:

OUTPUT optional, defaults to ./output

OUTPUT=./output
LANGUAGES=alpine
BUILDINDEX=1

But all the layouts are broken. Any index file I click bring me to "Your file couldn’t be accessedIt may have been moved, edited, or deleted."

And any index.html files with a component brings me to a broken layout.
ERR_FILE_NOT_FOUND

Thanks

auto-convert templates from React/Next to Vue.. feasible ?

I just found out that the new templates are React only and all components are written in JSX. I find this really annoying since all my projects are in Vue and thus integrating the templates means a lot of adaptation.

I'm sure you've had a look at the new templates code.. how difficult do you think it might be to transform the .jsx files to .vue SFCs and if not handle a fully working conversion.. at least save some time by handling the repetitive tasks?

fails while fetching ecommerce components - forEach on null

Writing vue with_image_grid.vue
‼️ TypeError: Cannot read properties of null (reading 'forEach')

npm start error

Hey guys,

Thanks for this amazing project. I'm not able to start it and check it out. On npm start I'm getting the following error:

Any suggestion is appreciated.

Thanks

Craweler not fetching all available components

Thank you for developing this tool! Great idea and work for sure.

I noticed that the crawler is not pulling all available snippets/components.

➜  tailwindui-crawler node index.js
Processing /components/marketing/sections/heroes...
* Found 1 snippet
Writing ./output/components/marketing/sections/heroes/with_angled_image_on_right.html...

There are 3 snippets not just 1:

Thank you.

TypeError: Cannot read properties of undefined (reading 'replace')

I got this error message when running the script - why is that?

tailwindui-crawler-master % npm start

[email protected] start
node index.mjs

file:///Users/name/Downloads/tailwindui-crawler-master/index.mjs:71
const regexEmail = new RegExp(process.env.EMAIL.replace(/[.@]/g, '\$&'), 'g')
^

TypeError: Cannot read properties of undefined (reading 'replace')
at file:///Users/name/Downloads/tailwindui-crawler-master/index.mjs:71:49
at ModuleJob.run (node:internal/modules/esm/module_job:194:25)

Node.js v19.1.0

Does not load the application & ecommerce files on the editor

Thanks,
After the last tailwind update when they released the Ecommerce page the theme editor seems to only open the Marketing files. It does not load the application & e-commerce files on the editor.

Yes, I have followed the instructions keenly not sure what I missed because last time it just worked. thanks

Styling issues and inconsistencies after crawler completes

I'm not sure if it's just me, but I'm seeing quite a few inconsistencies between the tailwindui code and the one the crawler creates.

1. Comparing https://tailwindui.com/components/application-ui/forms/form-layouts
   What I SHOULD be seeing: https://www.webpagescreenshot.info/#v2=2W7aOzUVq
   What I am seeing (on localhost): https://www.webpagescreenshot.info/#v2=eKFvEvtOY

2. Comparing https://tailwindui.com/components/marketing/elements/flyout-menus
   What I SHOULD be seeing: https://www.webpagescreenshot.info/#v2=uSqQgzqDM
   What I am seeing (on localhost): https://www.webpagescreenshot.info/#v2=OSF15rftp

Here's my .env file:

EMAIL=
PASSWORD=
OUTPUT=../tailwindui_output
BUILDINDEX=1
HTMLMODE=alpine
TRANSFORMERS=addTailwindCss,prefixSrc,useInter,prefixClasses,convertVue,stripAlpine
VUE_OUTPUT=$OUTPUT/vue
STRIPALPINE_OUTPUT=$OUTPUT/no-alpine

# changeColor
#CHANGECOLOR_TO=red

# changeLogo
#CHANGELOGO_URL=https://upload.wikimedia.org/wikipedia/commons/thumb/e/e7/Instagram_logo_2016.svg/1200px-Instagram_logo_2016.svg.png

# addTailwindCss. This setting is needed if you have prefixClasses enabled
# Note: This config file does not get auto-generated. Please make sure it already exists.
ADDTAILWINDCSS_URL=/tailwind.css

# prefixClasses
PREFIXCLASSES_PREFIX=tw-

Btw @kiliman I did compare the source code, and it looks like the tailwindui code includes styles from this extra css file. Not sure what to make of it.
https://tailwindui.com/css/iframe.css?id=a05cfc4bbf6c64e1422e

Add .env.example file with all possible config options

This is not an issue for me but thought I'd log it anyway! Adding a .env.example file with all possible config options would make it easier for new users.
It would be nice to have:

• everything pre-configured with defaults (we would leave out the username and password of course!)
• options in the .env.example file commented (like you have on the main github page)

As an aside, only through trial and error did I realize that in order to get the app running and see the index page, you have to cd into the output folder first before you run "serve" / do this: https://github.com/kiliman/tailwindui-crawler#-index-page

Optionally parse HTML to create React-ready output

I've integrated @RobinMalfait's React find/replace tool for TUI code.

https://gist.github.com/RobinMalfait/a90e8651196c273dfa51eec0f43e1676

Would you be interested in a PR to merge this? I've added an env variable to enable it, and it's still optional in index.js.

Github Action not working

The Github action to crawl and add it to a private repo isn't working. I copied the yml file exactly. I set the secrets for TAILWINDUI_EMAIL and TAILWINDUI_PASSWORD. Two days in a row now the actions tab showed it ran for about 40s but nothing was added to the repo.

This is what I put in my .env

LANGUAGES=html,react,vue,alpine
BUILDINDEX=1

underscores galore

After the latest Tailwind UI components update, the crawler has produced files with a ton of underscores left and right of the template name. A snippet of the output is below.

Processing /components/application-ui/layout/panels...
* Found 10 snippets
Writing ../tailwindui/components/application-ui/layout/panels/_____________basic_card___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card__edge_to_edge_on_mobile___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card_with_header___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card_with_footer___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card_with_header_and_footer___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card_with_gray_footer___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________card_with_gray_body___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________well___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________well_on_gray___________.html...
Writing ../tailwindui/components/application-ui/layout/panels/_____________well__edge_to_edge_on_mobile___________.html...

As a result, there's nothing to diff against as the filenames are different.

No prefixes are added for space-y-* and space-x-*

See the "Avatar group stacked bottom to top" component at http://localhost:5000/components/application-ui/elements/avatars

Notice that no prefixes are added for: space-y-6 sm:space-y-0

            <div class="space-y-6 sm:space-y-0 sm:tw-flex tw-justify-around tw-items-end tw-w-full tw-max-w-lg tw-mx-auto">
              <div class="tw-flex tw-relative tw-z-0 tw-overflow-hidden">
                <img class="tw-relative tw-z-30 tw-inline-block tw-h-8 tw-w-8 tw-rounded-full tw-text-white tw-shadow-solid" src="https://images.unsplash.com/photo-1491528323818-fdd1faba62cc?ixlib=rb-1.2.1&amp;ixid=eyJhcHBfaWQiOjEyMDd9&amp;auto=format&amp;fit=facearea&amp;facepad=2&amp;w=256&amp;h=256&amp;q=80" alt="">
                <img class="tw-relative tw-z-20 tw--ml-2 tw-inline-block tw-h-8 tw-w-8 tw-rounded-full tw-text-white tw-shadow-solid" src="https://images.unsplash.com/photo-1550525811-e5869dd03032?ixlib=rb-1.2.1&amp;auto=format&amp;fit=facearea&amp;facepad=2&amp;w=256&amp;h=256&amp;q=80" alt="">
                <img class="tw-relative tw-z-10 tw--ml-2 tw-inline-block tw-h-8 tw-w-8 tw-rounded-full tw-text-white tw-shadow-solid" src="https://images.unsplash.com/photo-1500648767791-00dcc994a43e?ixlib=rb-1.2.1&amp;ixid=eyJhcHBfaWQiOjEyMDd9&amp;auto=format&amp;fit=facearea&amp;facepad=2.25&amp;w=256&amp;h=256&amp;q=80" alt="">
                <img class="tw-relative tw-z-0 tw--ml-2 tw-inline-block tw-h-8 tw-w-8 tw-rounded-full tw-text-white tw-shadow-solid" src="https://images.unsplash.com/photo-1472099645785-5658abf4ff4e?ixlib=rb-1.2.1&amp;ixid=eyJhcHBfaWQiOjEyMDd9&amp;auto=format&amp;fit=facearea&amp;facepad=2&amp;w=256&amp;h=256&amp;q=80" alt="">
              </div>
            </div>

This probably also is an issue with https://github.com/vesper8/vue-tailwind-prefix-applicator/

BUILDINDEX=1 not working?

Not sure what I am doing wrong.

I had BUILDINDEX=1 but there is no index.html generated.

Did I do anything wrong?

Update okay i figured it out. Please delete this issue.

Abort with "RangeError: Maximum call stack size exceeded"

When I run the crawler I now get the following error:

⏳  Processing /components/application-ui/data-display/calendars...
⏱   233ms (200) https://tailwindui.com/components/application-ui/data-display/calendars
🔍  Found 8 components
📝  Writing html small_with_meetings.html
📝  Writing react small_with_meetings.jsx
📝  Writing vue small_with_meetings.vue
📝  Writing alpine small_with_meetings.html
📝  Writing html month_view.html
📝  Writing react month_view.jsx
📝  Writing vue month_view.vue
📝  Writing alpine month_view.html
📝  Writing html week_view.html
📝  Writing react week_view.jsx
📝  Writing vue week_view.vue
‼️   RangeError: Maximum call stack size exceeded
    at LoadedCheerio.Cheerio (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\cheerio.js:24:21)
    at new LoadedCheerio (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\load.js:32:46)
    at LoadedCheerio.Cheerio (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\cheerio.js:35:26)
    at new LoadedCheerio (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\load.js:32:46)
    at LoadedCheerio.Cheerio._make (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\cheerio.js:93:23)
    at LoadedCheerio.children (D:\dev\utils\tailwindui-crawler-local\node_modules\cheerio\lib\api\traversing.js:78:25)
    at findFirstElementWithClass (D:\dev\utils\tailwindui-crawler-local\index.js:146:42)
    at findFirstElementWithClass (D:\dev\utils\tailwindui-crawler-local\index.js:146:10)
    at findFirstElementWithClass (D:\dev\utils\tailwindui-crawler-local\index.js:146:10)
    at findFirstElementWithClass (D:\dev\utils\tailwindui-crawler-local\index.js:146:10)

Is there anything I can do here?

[Feature Request] Convert transitions when generating vue components

This is a feature request to handle transition comment blocks when generating vue components

So this:

  <!--
    Background overlay, show/hide based on modal state.

    Entering: "ease-out duration-300"
      From: "opacity-0"
      To: "opacity-100"
    Leaving: "ease-in duration-200"
      From: "opacity-100"
      To: "opacity-0"
  -->
  <div class="fixed inset-0 transition-opacity">
    <div class="absolute inset-0 bg-gray-500 opacity-75"></div>
  </div>

Would become this:

  <transition
    enter-class="opacity-0"
    enter-active-class="ease-out duration-300"
    enter-to-class="opacity-100"
    leave-class="opacity-100"
    leave-active-class="ease-in duration-200"
    leave-to-class="opacity-0"
  >  
    <div class="fixed inset-0 transition-opacity">
      <div class="absolute inset-0 bg-gray-500 opacity-75"></div>
    </div>
  </transition>

Basically this would entail converting the comment block to a transiton tag, and then figuring out the next child below the block and adding a closing transition tag on the other end of that immediate children tag

If this is too complicated, then at least replacing the comment block by a commented-out vue transition tag would also be very helpful, and it would be up to the developer to add the closing tag and if necessary figure out the correct position of the opening transition tag

Download all tailwindui.com images locally

Thank you for this amazing repo, huge time saver!

One quick suggestion: It would be great it we could add an option to download all tailwindui.com images locally so there are no dependencies.

Perhaps this could be a .env file setting like
IMAGES_OUTPUT=$OUTPUT/images

Thoughts?!

Error: ENOENT: no such file or directory, mkdir ...

This ran good the first time, now it fails every time with this error ...

% yarn start
yarn run v1.3.2
$ node index.js
🔐  Logging into tailwindui.com...
✅  Success!

🗂   Output is ./output
⏳  Processing /components/marketing/sections/heroes...
🔍  Found 6 components
‼️   { Error: ENOENT: no such file or directory, mkdir './output/components/marketing/sections/heroes'
    at Object.fs.mkdirSync (fs.js:885:18)
    at module.exports.ensureDirExists.dir (/Users/mikeyrab/Projects/tailwindui-crawler/utils.js:23:8)
    at processComponentPage (/Users/mikeyrab/Projects/tailwindui-crawler/index.js:109:5)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)
  errno: -2,
  code: 'ENOENT',
  syscall: 'mkdir',
  path: './output/components/marketing/sections/heroes' }
✨  Done in 1.10s.

The output folder is empty.
The dir permissions are 777 so that is not the issue.

419 error on login

hey guys, I am getting 419 error on the response from login
any idea how to fix that?

Components.popover?

I just synced TailwindUI for a fist time in a while it looks like I am missing a reference for Components.popover (at least in flyover's simple)

Is there a library I need to add now in addition to Aplinejs?

Thanks.

Option download figma file

add option for download de figma file https://tailwindui.com/download-figma

Just want to say Thank YOU!!!!!

Saves a lot of time. Thank you for making this package.

Always hangs up in different sections without any error

Is there a trick to run it stable? Or is the crawler outdated meanwhile?
Has anyone tried with current version of tailwind ui site?

Running with

EMAIL=yyy
PASSWORD=xxxx
OUTPUT=./output
HTMLMODE=comments 
TRANSFORMERS=prefixSrc

Process hangs up always in one of the first 4 marketing sections.

Blank vue components

removed credentials..
OUTPUT=./output
VUE_OUTPUT=./vue
HTMLMODE=alpine|comments
TRANSFORMERS=convertVue
BUILDINDEX=1

Is generating blank vue components as:

<template>
  
</template>

<script>
export default {
  data: () => ({
	
  })
}
</script>

If you enjoy this project, say thank you!

Here is the place to say thank you to folks who are working hard on this project.

« One of the greatest ways to show your appreciation to open source projects you enjoy is to open an issue that let people say thank you »

Disclaimer: I don't have any kind of connexion or personal interest with maintainers(s) of this project. Pure gratitude here.

Update crawler to support new TailwindUI site

It's previously worked, but ran it today and am getting:

hasn't found a single component.

TailwindUI pushed some big updates to their site today, so maybe related to changes they made that broke the crawler? The URL paths seem to still be the same though.

"Automatically keep private github repository up-to-date" does not work (no tailwind.css file created) with prefixClasses enabled

No css file is created if you have the tw- prefix enabled and use the github action to automatically keep a private github repository up-to-date.

It does build periodically, and the code gets checked into the private git repo, but there is no .css file created, so when you run npx serve, the pages look broken.

The Main page at http://localhost:5000/ looks ok:
https://www.webpagescreenshot.info/#v2=QXGpcmiSB

However, the rest of the pages look broken.
Screenshots:
https://www.webpagescreenshot.info/#v2=3S7WF3he-
https://www.webpagescreenshot.info/#v2=xOuZUmwkY

These are the steps I took:
https://github.com/kiliman/tailwindui-crawler#automatically-keep-a-private-github-repository-up-to-date

.env file:

OUTPUT=./tailwindui_output
BUILDINDEX=1
HTMLMODE=alpine
TRANSFORMERS=addTailwindCss,prefixSrc,useInter,prefixClasses,convertVue,stripAlpine
VUE_OUTPUT=$OUTPUT/vue
STRIPALPINE_OUTPUT=$OUTPUT/no-alpine
ADDTAILWINDCSS_URL=/tailwind.css
PREFIXCLASSES_PREFIX=tw-

# .github/workflows/default.yml

name: Update
on:
  push:
  schedule:
    - cron: "*/5 * * * *" # Every 5th minute

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/[email protected]
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}
          

Latest version of crawler broken

After a fresh git pull, this is what I see:

$ yarn start
yarn run v1.22.0
$ node index.js
🔐 Logging into tailwindui.com...
✅ Success!

🗂 Output is ../tailwindui_output
⏳ Processing /components/marketing/sections/heroes...
‼️ /TAILWINDUI/tailwindui_crawler_kiliman/crawler/transformers/prefixClasses/tailwind-classes.js:2
export default {
^^^^^^
SyntaxError: Unexpected token 'export'
at wrapSafe (internal/modules/cjs/loader.js:1067:16)
at Module._compile (internal/modules/cjs/loader.js:1115:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
at Module.load (internal/modules/cjs/loader.js:1000:32)
at Function.Module._load (internal/modules/cjs/loader.js:899:14)
at Module.require (internal/modules/cjs/loader.js:1040:19)
at require (internal/modules/cjs/helpers.js:72:18)
at Object. (/TAILWINDUI/tailwindui_crawler_kiliman/crawler/transformers/prefixClasses/index.js:3:21)
at Module._compile (internal/modules/cjs/loader.js:1151:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
✨ Done in 1.78s.

Components dont get rendered if you use prefixClasses for tailwind prefix replacement

@kiliman I just tested #15 out, and the prefix replacement works beautifully. Really neat, and this will save a good chunk of time as well, so I no longer have to manually do it using
https://github.vue.tailwind-prefix.cbass.dev/

There is a problem, though. Without #16, this is what happens:
https://www.webpagescreenshot.info/#v2=4CUgNYg52

Here's my env file:

OUTPUT=../tailwindui_output
BUILDINDEX=1
HTMLMODE=alpine
TRANSFORMERS=addTailwindCss,prefixSrc,useInter,prefixClasses,convertVue,stripAlpine
VUE_OUTPUT=$OUTPUT/vue
STRIPALPINE_OUTPUT=$OUTPUT/no-alpine

# prefixClasses
PREFIXCLASSES_PREFIX=tw-

And this is what gets output in the HTML:

<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@tailwindcss/ui@latest/dist/tailwind-ui.min.css" />
<link rel="stylesheet" href="https://rsms.me/inter/inter.css" />

<style type="text/css">
html, .font-sans{font-family:"Inter var",system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}
.sm\:font-sans{font-family:"Inter var",system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}
.md\:font-sans{font-family:"Inter var",system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}
.lg\:font-sans{font-family:"Inter var",system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}
.xl\:font-sans{font-family:"Inter var",system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}
</style>
</head>
<body>
<div class="tw-bg-white">
  <div class="tw-max-w-7xl tw-mx-auto tw-py-8 sm:tw-px-6 lg:tw-px-8">
    <div class="tw-border-t tw-border-gray-200 tw-px-4 tw-flex tw-items-center tw-justify-between sm:tw-px-0">
      <div class="tw-w-0 tw-flex-1 tw-flex">
        <a href="#" class="tw--mt-px tw-border-t-2 tw-border-transparent tw-pt-4 tw-pr-1 tw-inline-flex tw-items-center tw-text-sm tw-leading-5 tw-font-medium tw-text-gray-500 hover:tw-text-gray-700 hover:tw-border-gray-300 focus:tw-outline-none focus:tw-text-gray-700 focus:tw-border-gray-400 tw-transition tw-ease-in-out tw-duration-150">
          <svg class="tw-mr-3 tw-h-5 tw-w-5 tw-text-gray-400" fill="currentColor" viewBox="0 0 20 20">
            <path fill-rule="evenodd" d="M7.707 14.707a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414l4-4a1 1 0 011.414 1.414L5.414 9H17a1 1 0 110 2H5.414l2.293 2.293a1 1 0 010 1.414z" clip-rule="evenodd" />
          </svg>
          Previous
        </a>
        

Since none of the tw- styles are defined on the CDN version, none of the components are styled. Is there a setting I am missing, or is this still something you're working on?
Once again, many thanks for all the work you're doing on this!

Crawler hangs on download or throws ETIMEDOUT error

Hello,

I have the issue that the crawler randomly stops forever after crawling a couple of components. What is the issue?

Regards,