kbravh/tweet-to-markdown

A command line tool to convert Tweets to Markdown.

Tweet to Markdown

A command line tool to quickly save tweets as Markdown.

Report a Bug · Request a Feature

⚠️ Heads up! ⚠️

If your API key is not working, you will need to apply for a Twitter bearer token.

Due to recent changes to the Twitter API, the free access method listed below has stopped working as of April 27, 2023. You must sign up for your own Twitter bearer token to use this application.

About The Project

This command line tool allows you to quickly save a tweet in Markdown format. This is great for Zettelkasten note-taking or any other commonplace notebook, vade mecum, Obsidian, Roam, Foam, &c. It is built on the new Twitter v2 API.

Installing

⚠ You'll need to have Node.js of at least v12.x to use this tool.

You can install this CLI tool by running

yarn global add tweet-to-markdown

or

npm install --global tweet-to-markdown

You can also run it without installing:

npx tweet-to-markdown

Setup

To use this tool, you have two options:

• Sign up for a free API key from https://ttm.kbravh.dev (new in v2.0.0)
• Sign up for a bearer token through the Twitter Developer dashboard

Free TTM API key (❌ disabled)

You can sign up for a free API key at https://ttm.kbravh.dev by signing in with either your GitHub or Twitter account and heading to your account page. Once you sign in and retrieve your API key from your account page, either store it in the environment variable TTM_API_KEY or pass it to the command line tool with the -b (--bearer) flag with each call.

Twitter Developer bearer token

Nota bene: You need at least a Basic plan in order to look up tweets. The Free plan is not sufficient.

To get a bearer token from Twitter, you'll need to set up an application on the Twitter developer dashboard. For a guide on doing so, see Getting a bearer token. Once you have the bearer token, either store it in the environment variable TWITTER_BEARER_TOKEN or pass it to the command line tool with the -b (--bearer) flag with each call.

Usage

Grabbing a tweet is as easy as calling the ttm command and passing in the tweet URL.

ttm -b "<bearer token>" https://twitter.com/JoshWComeau/status/1213870628895428611
# Tweet saved as JoshWComeau - 1213870628895428611.md

Nota bene: If passing your bearer token as an argument instead of an environment variable, wrap your bearer token or API key in quotes to prevent any special symbols from being misinterpreted by the command line. Also, if using a TTM API key, be sure to include the entire key, from the TTM at the beginning all the way to the end with the alphanumeric characters (e.g. "TTM>asdf1123" or "TTM_ghjk4567").

The tweet will be saved to a Markdown file in the current directory. Here's how the tweet will look:

Any attached images, polls, and links will also be linked and displayed in the file.

Options

There are many options to customize how this tool works. It is highly recommended to find the options you need and set up an alias in your terminal.

For example on Mac or Linux, you can define an alias with your options like so:

alias ttm="ttm -p $HOME/notes --assets --quoted"

For Windows, have a look at DOSKEY.

Copy to Clipboard

What if you want to just copy the Markdown to the clipboard instead of saving to a file? Just pass the -c (--clipboard) flag.

ttm -c https://twitter.com/JoshWComeau/status/1213870628895428611
#Tweet copied to the clipboard.

Quoted tweets

If you would like to include quoted tweets, pass the -q (--quoted) flag. This is disabled by default because a separate request has to be made to fetch the quoted tweet.

ttm <tweet url> -q

Tweet threads

To capture an entire tweet thread, use the -t (--thread) flag and pass the URL of the last tweet in the thread.

Nota bene: this will make a separate network request for each tweet.

ttm <last tweet url> -t

Condensed threads

Instead of showing complete, individual tweets with profile picture, date, etc. when downloading a thread, this option will show the header once and then only show the tweet bodies, representing tweet threads as a cohesive body of text. A header will be shown if a different author appears in the thread, for example if you're downloading a conversation between various authors.

ttm <last tweet url> -T

Semicondensed threads

These follow the same rules as condensed threads, but each tweet will still be separated by ---.

ttm <last tweet url> -s

Text only

With this flag, only the text of the tweet itself will be included. No author tags, frontmatter, or other information will be attached.

Nota bene: This has not been tested well with threads. Please use at your own risk. Condensed threads may be a better fit for you.

Custom File Name

In order to save the tweet with a custom filename, pass the desired name to the --filename flag. You can use the variables [[name]], [[handle]], [[text]], and [[id]] in your filename, which will be replaced according to the following chart. The file extension .md will also be added automatically.

| Variable | Replacement | | :--------: | ------------------------------------------------------------------------------- | --- | | [[handle]] | The user's handle (the part that follows the @ symbol) | | [[name]] | The user's name | | [[id]] | The unique ID assigned to the tweet | | [[text]] | The entire text of the tweet (truncated to fit OS filename length restrictions) | . |

ttm <tweet url> --filename "[[handle]] - Favicon versioning".
# Tweet saved to JoshWComeau - Favicon versioning.md

If the file already exists, an error will be thrown unless you pass the -f (--force) flag to overwrite the file.

Custom File Path

To save the tweet to a place other than the current directory, pass the location to the -p (--path) flag. If this path doesn't exist, it will be recursively created.

ttm <tweet url> -p "./tweets/"
# Tweet saved to ./tweets/JoshWComeau - 1213870628895428611.md

Tweet Metrics

If you'd also like to record the number of likes, retweets, and replies the tweet has, pass the -m (--metrics) flag. This will save those numbers in the frontmatter of the file.

ttm <tweet url> -m

---
author: Josh ✨
handle: @JoshWComeau
likes: 993
retweets: 163
replies: 24
---

Save Images Locally

Want to really capture the entire tweet locally? You can pass the -a (--assets) flag to download all the tweet images as well, instead of just linking to the images on the web. If the tweet is ever deleted or Twitter is unavailable, you'll still have your note.

ttm <tweet url> -a

Tweet images will be automatically saved to ./tweet-assets. If you'd like to save the assets to a custom directory, use the --assets-path flag and pass in the directory.

ttm <tweet url> -a --assets-path "./images"

Nota bene: Unfortunately, there is currently not a way to save gifs or videos from tweets using the v2 API.

Date locale

Using the --date_locale option, you can pass a BCP 47 language code to have the date displayed in a specific locale. If you don't pass this option (or if you're using lower than Node 13), your computer's default locale will be used.

Finding a complete list of these language codes is difficult, but here's a short list of examples:

Tag	Language
en	English
en-US	American English
es-MX	Mexican Spanish
ta-IN	Indian Tamil

A more complete list can be found here: https://gist.github.com/typpo/b2b828a35e683b9bf8db91b5404f1bd1

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch ( git checkout -b feature )
3. Commit your Changes ( git commit -m "Add a cool feature" )
4. Push to the Branch ( git push origin feature )
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details

Contact

Karey Higuera - @kbravh - [email protected]

Project Link: https://github.com/kbravh/tweet-to-markdown

tweet-to-markdown's Issues

No user feedback given while pulling Thread

This is more of a usability request rather than a bug.

Basically, when you call a long thread with the -t flag, ttm sits there, presumably doing work, but not giving any user feedback.

So a quick way to perhaps, "workaround," this problem might be to add to the documentation and instruct users how to check whether a PID is running, just open up a new tab and check, ps and you should see that the process is still ongoing.

Another solution would maybe to allow the user to enable logging output with perhaps an -l flag.

Of course a more universal user-centric approach might be to add something like a progress bar, or a spinning, "working," icon.

Progress bar - perhaps more technically difficult for this tool, because you can't necessarily predict the total size of a Tweet thread at the start, I am not sure.

https://stackoverflow.com/questions/238073/how-to-add-a-progress-bar-to-a-shell-script

Spinning ball - at least shows the user that a process is still in operation, if not predicting when it might be done.

https://stackoverflow.com/questions/10470139/creating-rotating-circle-using-characters-in-shell-script

Possible @mention parsing bug

Possible edge case. Version 2.1.1 ttm does not appear to parse the @mentions in this tweet correctly.

https://twitter.com/jose_edil/status/1538271708918034433

This is the resulting md:

Notice the double square brackets following the 1st @mention. Also, viewing the original tweet, "@Its_Prasa" is not a current twitter.com user.

---
author: "Edil Medeiros 🚢 ✏️🏴‍☠️ 🧩"
handle: "@jose_edil"
source: "https://twitter.com/jose_edil/status/1538271708918034433"
likes: 14
retweets: 0
replies: 4
---
![jose_edil](https://pbs.twimg.com/profile_images/669315274353561600/eHheoab4_normal.jpg)
Edil Medeiros 🚢 ✏️🏴‍☠️ 🧩 ([@jose_edil](https://twitter.com/jose_edil))

[@visualizevalue](https://twitter.com/visualizevalue) [[@EvansNifty](https://twitter.com/EvansNifty)](https://twitter.com/EvansNifty) [[@OzolinsJanis](https://twitter.com/OzolinsJanis)](https://twitter.com/OzolinsJanis) [[@milanicreative](https://twitter.com/milanicreative)](https://twitter.com/milanicreative) [[@design_by_kp](https://twitter.com/design_by_kp)](https://twitter.com/design_by_kp) [[@victor_bigfield](https://twitter.com/victor_bigfield)](https://twitter.com/victor_bigfield) [[@StartupIllustr](https://twitter.com/StartupIllustr)](https://twitter.com/StartupIllustr) [[@tracytangtt](https://twitter.com/tracytangtt)](https://twitter.com/tracytangtt) [[@AlexMaeseJ](https://twitter.com/AlexMaeseJ)](https://twitter.com/AlexMaeseJ) [[@ash_lmb](https://twitter.com/ash_lmb)](https://twitter.com/ash_lmb) [[@moina_abdul](https://twitter.com/moina_abdul)](https://twitter.com/moina_abdul) @Its_Prasa [[@elliottaleksndr](https://twitter.com/elliottaleksndr)](https://twitter.com/elliottaleksndr) [[@aaraalto](https://twitter.com/aaraalto)](https://twitter.com/aaraalto) [[@tanoseihito](https://twitter.com/tanoseihito)](https://twitter.com/tanoseihito) [[@jeffkortenbosch](https://twitter.com/jeffkortenbosch)](https://twitter.com/jeffkortenbosch) [[@FerraroRoberto](https://twitter.com/FerraroRoberto)](https://twitter.com/FerraroRoberto) [[@eneskartall](https://twitter.com/eneskartall)](https://twitter.com/eneskartall) [[@SachinRamje](https://twitter.com/SachinRamje)](https://twitter.com/SachinRamje) [[@AidanYeep](https://twitter.com/AidanYeep)](https://twitter.com/AidanYeep) [[@jozzua](https://twitter.com/jozzua)](https://twitter.com/jozzua) Here they are:

@EvansNifty
@OzolinsJanis
@milanicreative
@design_by_kp
@victor_bigfield
@StartupIllustr
@tracytangtt
@AlexMaeseJ
@ash_lmb
@moina_abdul
@Its_Prasa
@elliottaleksndr
@aaraalto
@tanoseihito
@jeffkortenbosch
@FerraroRoberto
@eneskartall
@SachinRamje
@AidanYeep
@jozzua

[Tweet link](https://twitter.com/jose_edil/status/1538271708918034433)

I've tested w/ single tweet, thread, and condensed_thread.

ttm -b '<TWITTER_BEARER_TOKEN>' --filename '[[handle]]-[[id]]' --path 'markdown/' -fqm https://twitter.com/i/status/1538271708918034433

# Tweet saved to markdown/jose_edil-1538271708918034433.md

Tweet "date" information

I noticed that tweet-to-markdown does not have the "date" information, which the Obsidian plugin has.

I edited the main.js directly. Here is the diff for anyone that might find this useful or if someone wants to upgrade the source code to add this feature.

17808c17808
<         'tweet.fields': 'attachments,public_metrics,entities,conversation_id,referenced_tweets',
---
>         'tweet.fields': 'attachments,public_metrics,entities,conversation_id,referenced_tweets,created_at',
18042a18043,18045
>   // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleDateString
>   // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/DateTimeFormat
>   const cdate_options = { weekday: 'short', year: 'numeric', month: 'short', day: 'numeric', hour12: true,hour: 'numeric', minute: 'numeric', timeZone: 'America/Los_Angeles', timeZoneName: 'short' };
18043a18047,18048
>   let created = new Date(tweet.data.created_at);
>   let fetched = new Date();
18045a18051,18052
>           `date: ${created.toLocaleDateString('en-US', cdate_options)}`, // Tue, Aug 16, 2022, 2:41 PM PDT
>           `fetched: ${fetched.toLocaleDateString('en-US', cdate_options)}`,
18089a18097
>   const date_options = { year: 'numeric', month: 'short', day: 'numeric' };
18095c18103
<         `${user.name} ([@${user.username}](https://twitter.com/${user.username}))`, // name and handle
---
>         `${user.name} ([@${user.username}](https://twitter.com/${user.username})) - ${created.toLocaleDateString('en-US', date_options)}`, // name, handle and date (Aug 10, 2022)

app produces invalid YAML

Hey! I noticed that the frontmatter that is produced by the app is actually invalid, because @ is a special character in YAML and text that starts with an @ must be quoted (see here).

For example:

---
author: Josh ✨
handle: @JoshWComeau <-- should be "@JoshWComeau"
---

Will this tool still work under Twitter new API policy?

Free access will stop from Feb-9.

#hashtag parsing bug

If a tweet contains hashtags like this:

#abc
#ab

The parsing result will be:

[[#ab](https://twitter.com/hashtag/ab) c](https://twitter.com/hashtag/abc)  
#ab

Each tag link is added to the first matched phrase it came across, which may not be the correct one. I think this is a similar issue to the @mention bug #9.

TypeError: Cannot read property 'statusText' of undefined

Type Error

I attempted to run this command on a long thread:

$ ttm --bearer "TTM>XXXXX" https://twitter.com/iotmpls/status/1555601890116788225 -a -t

• So as you can see, I'm saving the images locally and attempting to run through and write up as long of a markdown file that I could. However, partway through (118 tweets through), it failed with:

`$ ttm --bearer "TTM>XXXXX" https://twitter.com/iotmpls/status/1555601890116788225 -a -t
/Users/patrick/.config/yarn/global/node_modules/tweet-to-markdown/dist/main.js:14961
panic(source.red(error.response.statusText));
^

TypeError: Cannot read property 'statusText' of undefined
at /Users/patrick/.config/yarn/global/node_modules/tweet-to-markdown/dist/main.js:14961:41
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)`

• Moreover, while I did get a large number of images downloaded, no markdown file was authored.

A couple points:

1. I believe the expected functionality from a user standpoint would be that even if there is an error, the markdown file should have been at least created. Now what I have is a folder full of images, with no markdown file. So perhaps the user ran out of credits, that would be an acceptable outcome, because it's up to the user to have enough credits to finish the tweet string. However then the error handling should say, "not enough credits! Finishing thread."

I don't know much about how node works, but the equivalent behavior I would expect in bash psuedocode would be something like:

# start a file
touch tweetstringfile.md

# for each tweet starting at tweet 0 through n
for TWEET in seq 0 n
do
    # if ran out of credits
    echo "you ran out of credits! Closing and saving markdown file thread."
    # pull each tweet
    echo $TWEET >> tweetstringfile.md
done

Where essentially the >> is "appending" to the bottom of the file.

Better yet, prompt the user ahead of time to understand that the number of the tweets vs. remaining credits is acceptable. Warning! This will use up all of your remaining credits and will not finish the .md file!

2. All of the above being said, I believe the error was actually just an API error, which came from here in your code:

https://github.com/kbravh/tweet-to-markdown/blob/main/src/util.ts#L141

So basically, the TTM API didn't respond in that once instance, and without the proper error handling and logic built in to your program, it appears that your program collects everything in, "string"

https://github.com/kbravh/tweet-to-markdown/blob/main/src/util.ts#L153

and then writes it to a markdown file all at once in the end:

https://github.com/kbravh/tweet-to-markdown/blob/main/src/util.ts#L332

So basically if any error occurs along the way, then a markdown file never gets written.

Does that sound correct to you? What are your thoughts?

API Does Not Seem to Be Working

Greetings! Thanks for building this.

I have attempted to use this tool on both zsh and bash with no results.

• I set the alias to the exact location have installed ttm.
• Using node 16.0.0
• Using asdf
• Using your API

When running the command:

ttm --bearer <key> https://twitter.com/JoshWComeau/status/1213870628895428611

• As a sample, I see nothing in the current directory. I have tried this with a variety of different tweets, options, custom file name --filename, copied to clipboard, -c, etc. and I see no results.
• Also interestingly I see no usage on the API toward my limit, though I am not seeing a, "not authorized" message from ttm.

I'm wondering if the API may take a while to be activated?

Allow users to define date format

Currently, the date format is hardcoded for the frontmatter and the tweet text. Ideally, the user should be able to pass in a format string, such as LLL or YYYY-MM-DD.

Unfortunately, the built-in toLocaleDateString does not accept those format strings, and passing a JSON options object is unwieldy for a CLI. I'd switch to Day.js, but packaging all of the i18n languages in will make it much heavier than needed, when each user will probably only use one of those locales.

[feature request] option to strip out metadata and save only content

For example right now the content saved is

![championswimmer](../images/championswimmer-63066473.jpg)
Arnav Gupta 💉💉💉 ([@championswimmer](https://twitter.com/championswimmer))

I don't think doing artificially created puzzles like LeetCode should be involved in early stage learning to code. 

Learning to build a website or app first, and coming to algo-gyms later in life is a better approach.

Or else people miss the forest for the trees.

[Tweet link](https://twitter.com/championswimmer/status/1548562277502046208)

IF there is a flag to download only the content, i.e.

- ![championswimmer](../images/championswimmer-63066473.jpg)
- Arnav Gupta 💉💉💉 ([@championswimmer](https://twitter.com/championswimmer))

I don't think doing artificially created puzzles like LeetCode should be involved in early stage learning to code. 

Learning to build a website or app first, and coming to algo-gyms later in life is a better approach.

Or else people miss the forest for the trees.

- [Tweet link](https://twitter.com/championswimmer/status/1548562277502046208)

Final result

I don't think doing artificially created puzzles like LeetCode should be involved in early stage learning to code. 

Learning to build a website or app first, and coming to algo-gyms later in life is a better approach.

Or else people miss the forest for the trees.

Why ?

Helps me when building a substack article out of my Twitter threads :)

Which ZSH theme are you using?

ZSH theme you used on gif on README looks cool, could you tell me which one are you using?

Forbidden error when downloading a tweet and having a valid bearer token

Hi, im getting this error when trying to pull a tweet

2023-05-07XXXXXXZ: Forbidden; [object Object]
Error: Request failed with status code 403
    at createError (/usr/lib/node_modules/tweet-to-markdown/dist/main.js:7854:15)
    at settle (/usr/lib/node_modules/tweet-to-markdown/dist/main.js:7870:12)
    at IncomingMessage.handleStreamEnd (/usr/lib/node_modules/tweet-to-markdown/dist/main.js:10226:11)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

the bearer token should be valid as i just made it and if it was invalid the error would be Unauthorized, not Forbidden

UNKNOWN_OPTION: Unknown option: -T

Hey, sorry I have not gotten back to try out this tool since almost a month ago after you had addressed the last issue, I was busy.

Fairly self-explanatory, the -T flag does not appear to be working.

Wondering if the order of the flag matters?

$ ttm -b "TTM>BEARERINFO" https://twitter.com/iotmpls/status/1565387911264075776 -T -a
/Users/username/.config/yarn/global/node_modules/tweet-to-markdown/dist/main.js:2040
        throw err
        ^

UNKNOWN_OPTION: Unknown option: -T
    at commandLineArgs (/Users/username/.config/yarn/global/node_modules/tweet-to-markdown/dist/main.js:2037:21)
    at Object.<anonymous> (/Users/username/.config/yarn/global/node_modules/tweet-to-markdown/dist/main.js:15367:17)
    at Module._compile (node:internal/modules/cjs/loader:1108:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Function.Module._load (node:internal/modules/cjs/loader:828:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47 {
  optionName: '-T'
}

get multiple tweets (by date range)

Thanks for this project, it's useful!

Is there a way to get this tool to pull a range of tweets from a given account? Something like:

npx tweet-to-markdown --since https://twitter.com/example/status/14542 --until https://twitter.com/example/status/9999

where the tool would pull all tweets starting from the --since tweet until the --until tweet.

Alternatively, between a date range:

npx tweet-to-markdown --since 2022-01-1 --until 2022-07-1 https://twitter.com/example