browsh-org/browsh

A fully-modern text-based browser, rendering to TTY and browsers

A fully interactive, real-time, and modern text-based browser rendered to TTYs and browsers

Why use Browsh?

Not all the world has good Internet.

If you only have a 3kbps internet connection tethered from a phone, then it's good to SSH into a server and browse the web through, say, elinks. That way the server downloads the web pages and uses the limited bandwidth of an SSH connection to display the result. However, traditional text-based browsers lack JS and all other modern HTML5 support. Browsh is different in that it's backed by a real browser, namely headless Firefox, to create a purely text-based version of web pages and web apps. These can be easily rendered in a terminal or indeed, ironically, in another browser. Do note that currently the browser client doesn't have feature parity with the terminal client.

Why not VNC? Well VNC is certainly one solution but it doesn't quite have the same ability to deal with extremely bad Internet. Terminal Browsh can also use MoSH to further reduce bandwidth and increase stability of the connection. Mosh offers features like automatic reconnection of dropped or roamed connections and diff-only screen updates. Furthermore, other than SSH or MoSH, terminal Browsh doesn't require a client like VNC.

One final reason to use terminal Browsh could be to offload the battery-drain of a modern browser from your laptop or low-powered device like a Raspberry Pi. If you're a CLI-native, then you could potentially get a few more hours of life if your CPU-hungry browser is running somewhere else on mains electricity.

Installation

Download a binary from the releases (~11MB). You will need to have Firefox already installed.

Or download and run the Docker image (~230MB) with: docker run --rm -it browsh/browsh

Usage

Most keys and mouse gestures should work as you'd expect on a desktop browser.

For full documentation click here.

Contributing

Note that some of these instructions may be out of date (July 2022)

For generic Linux systems you can follow this guide on how to setup a build environment, that you may be able to adapt for other systems as well.

Windows users can follow this guide in order to set up a build environment.

Mac users may follow this guide that goes through the steps of setting up a build environment.

Communication

Questions about Brow.sh? Stuck trying to resolve a tricky issue? Try:

• Github Discussions
• Gitter

Tests

For the webextension: in webext/ folder, npm test
For CLI unit tests: in /interfacer run go test src/browsh/*.go
For CLI E2E tests: in /interfacer run go test test/tty/*.go
For HTTP Service tests: in /interfacer run go test test/http-server/*.go

Special Thanks

• @tobimensch For essential early feedback and user testing.
• @arasatasaygin For the Browsh logo.

Donating

Please consider donating: https://www.brow.sh/donate

License

GNU Lesser General Public License v2.1

browsh's Issues

Experiment with Fast DOM

https://github.com/wilsonpage/fastdom The idea is that batched reads and writes to the DOM significantly reduce page reflow and thus CPU load. I'm not totally sure that Browsh does enough writes, but it certainly does a lot of reads and could benefit from doing more reads.

Bold, underline and italic in terminals

Bold, underlined and italic text can add a lot to reading quality (if they are used correctly).

Sub/headings can provide orientation for the reader.

Currently browsh produces pure text both in tty and http-server mode, without any special formattings applied. (The only exception being color)

It will be hard(read: next to impossible) to display different text sizes in the terminal, but terminals do support bold, underlined and italic text:
https://askubuntu.com/questions/528928/how-to-do-underline-bold-italic-strikethrough-color-background-and-size-i#528938

You can use this to emulate a heading, for example through bold and underlined text.

The http-server can use the usual HTML tags of course.

To clarify I'm not advocating for emulating all text characteristics of the original source website, I'm just of the opinion that some basic formatting does indeed improve the reading flow and therefore browsh should try to retain a minimal amount of formatting.

URL bar dropdown of matching previously visited sites

As we all already know and love from our GUI browsers.

Make the character density changable

[Feature request]

For those moments of "unusually fast" slow unstable internet connections, add the possibly to increase the character density (even smaller characters) to improve the resolution?

(Thanks for making texttop, it's awesome!)

Loading https://www.welt.de causes a screenshot failure

Relevant log message:
["_getPixelData@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18180:12\n_getScreenshot@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18133:12\n_getScreenshotWithText@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18075:29\ngetScreenshotWithText/<@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18058:7\nlogPerformance@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:17228:5\ngetScreenshotWithText@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18057:5\nsendRawText@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:18261:5\nsendRawText@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:17372:5\n_setupMode@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:17572:7\n_handleBackgroundMessage@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:17528:9\n_listenForBackgroundMessages/<@moz-extension://a268f53e-0352-4b9c-815c-cf1d00d4c150/content.js:17483:9\n"]

Documentation (ELinks)

Your link to elinks is dead.

ELinks itself appears dead as well.

Consider the following:

• 1. Latest stable is 0.11.7 (22 August 2009; 8 years ago), latest preview is 0.12pre6 (30 October 2012; 5 years ago);
• 2. On 17 March 2017, OpenBSD removed ELinks from its ports tree, citing concerns with security issues and lack of responsiveness from the developers;
• 3. Some support for ECMAScript by using Mozilla's SpiderMonkey JavaScript engine;
• 4. SpiderMonkey surely had security fixes since 30 October 2012; 5 years ago;

5 years is a long time in software land. I don't think this software should be used anymore, by you or anyone else. I don't think this software should be recommended anymore. If you recommend it, at least be clear about the above.

What I would recommend as alternative? I'm not aware of a CLI browser supporting JavaScript. Links removed their SpiderMonkey code from its tree because it was out of date.

Sources used: Wikipedia

Ability to add any of Firefox's addons/extensions

For example being able to add uBlock Origin, etc.

This is so almost possible already, it's just that when you go to an addon's page and try to add it, there's an actual browser GUI prompt that Browsh is unable to see and therefore render. It's actual really easy to use the existing Marionette API that Browsh already uses to add its own rendering webextension. So there just needs to be some domain-specific code that can detect or override the normal behaviour on individual addon pages.

But then there's also the problem that Browsh's can't see any non-web URLS like about:config, so extensions can't be configured. However, I wonder if Firefox's other RDP API can see these pages? This would relate to: #27

For now you can at least follow instructions like these: https://stackoverflow.com/questions/37728865/install-webextensions-on-firefox-from-the-command-line to manually download and install an extension via the CLI.

Inefficiencies in text_builder's sendRawText method

I looked through the code for a bit and this code in
webext/src/dom/text_builder.js

    // TODO:
    //   The presence of the `getScreenshotWithText()` and `setTimeout()` calls are a hack
    //   that I am unable to understand the reasoning for - unfortunately they came about
    //   by trial and error :( Without them the the returned raw text is largely empty.
    this.graphics_builder.getScreenshotWithText();
    setTimeout(() => {
      this.buildFormattedText();
      this._sendRawText();
    }, 200);
  }

I'm not sure how much this does slow down rendering pages, however the timeout makes it look like it's a slowdown by 200 ms at a minimum, which would be pretty significant, if true.

Since you didn't understand this behaviour, might this point to a bug in firefox? Did you try this mechanism in Chrome?

Configurable browser font size

Interesting and potentially useful things happen when you change the font size of the browser. At the moment it's just universally fixed to 15px. This is a good balance, but some web pages are designed for much bigger or smaller sizes, so being able to accommodate that will improve the final quality of the TTY-rendered page.

But also just being able to manually play with the font-size might be advantageous in some circumstances as well.

Select, copy and paste

Firstly it'd be good to have these internally, so at least for example you can copy a paragraph from Wikipedia into an email your composing. But also it'd be nice to try to copy the internal clipboard into the host's clipboard.

It appears there's already a nice cross-platform clipboard Golang lib: https://github.com/atotto/clipboard

Automatically try to use existing headlesss Firefox instance if detected

There's still a bit of work to get the webextension to constantly retry connecting to the TTY when it detects a closed socket. At the moment you have to somehow refresh the webpage to get the TTY to load the frame again. During development this happens automatically when you save a change to a source file.

Hiptext build is arch dependant

AMD users using the Docker Hub version get an "Illegal instruction" error when running hiptext. Either find a way of avoiding hiptext's arch dependency or provide multiple builds.

docker is difficult to install

If you can remove the docker dependency #ThatWouldBeGreat

I've tried to install it on two platforms now (my Ubuntu Chroot on a SD card and my Nitrous.io box), neither work. It always says docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.

The reason I'm posting this issue is that I'm genuinely excited about this project and have been wondering for a while whether something is possible.

Shortcut to output the contents of the clipboard to the TTY

Because you can select text and send a CTRL+C to copy, then it might be a handy thing to have a texttop-only shortcut to output the contents of the clipboard to the TTY using real characters. But where would the text be output?

Config file

So that user's preferences can be saved and Browsh can be started in particular state, eg; starting in Monochrome mode.

Issue: There are no issues

This is a really cool project, but it doesn't have any issues

Windows: "dial tcp 127.0.0.1:2828: connectex: No connection could be made"

I don't how popular running Browsh on Windows is? But we might as well have it working if it's possible. At the moment Firefox crashes during boot, I don't know why.

Of course Windows users, as I suspect most users, will use Browsh over Mosh/SSH, where Browsh runs on a Linux VM.

Stub `alert()`

Alerts are fairly common, but they are not rendered or interactable in the TTY.

Hidden text over animated graphics (eg video) is incorrectly made visible

This is most obviously seen on Youtube.

Because Browsh uses a method of turning the text on and then off and seeing if the average pixel value changes in order to detect if the text is visible, then text over moving images creates false positives. The only thing I can think of at the moment is to resort to CSS visibility, but that can be expensive.

Tabbing should navigate input boxes

Installed properly but not working for me. Any missing step?

Hi,
I found this project interesting but unfortunately couldn't get it to work

Yesrerday I setup everything on my Debian 8 Xen VPS exactly as described without any installation error.

( First, when I tried to run it directly on VPS I got an error 'Bad signal name "-" ' on Docker shell - but I understand it's not designed to work this way. )

When I tried to connect using MobaXterm (as Mosh client for Windows) I got an error which said something like "not working ..Is SSH proxycommand enabled? "

I suspect problem has something to do with port forwarding part which blocks my Mosh access to the Docker container. (no firewall on that fresh VPS I was using).

I'm no way an expert with Docker, maybe some additional settings are required apart from those commands provided in description?

Thank you for any help

Zoom anchor doesn't seem quite right

Using the Tilda terminal. When I first zoom in, the zoom centre doesn't lock to where my mouse is. However it does after a few zooms, maybe level 4 or 5? But even then it seems a little out.

Support all browsers that support web extensions

The biggest hurdle is using a standardised RDP. This; https://remotedebug.org/ isn't yet supported by all browsers!

• Chrome doesn't support extensions in headless mode! So we need to find a way to stop Chrome stealing focus on things like window resizing. Only relevant if using Browsh locally of course.
• Chrome also has a different API for snapshoting the rendered tab.
• Firefox still doesn't have support for the standardised RDP. It'd be nice to not have to rewrite the RDP commands for every browser. Although theoretically the RDP commands should be minimal, eg; resizing the browser window and automatically installing the extension.

In-browser help

I think this can just be a shortcut to load https://www.brow.sh/docs/introduction/ in a new tab?

Provide alternatives to CTRL+click

Some terminals, like xterm, already intercept CTRL+click. Perhaps look into using CTRL+ALT+click or CTRL+arrows.

Uncaught mouse reporting sent to desktop as text

Sometimes you find stuff like M[?1015h gets typed into text inputs every time you move the mouse. I suspect this is because some mouse reporting codes aren't being picked up by termbox.

Config to turn images off

On critically slow connections, images can add unneeded bandwidth to frames. So have an option to stay in colour mode but not use images.

Stops after displaying sh: bad signal name '-'

Arch x86_64
Used the command to test on my local machine. It stops after the bad signal and leaves the terminal in a corrupt state. I am using xterm.

Text rendering lags, or doesn't render until after some scrolling

Firstly there's the unavoidable fact that text takes longer to parse and render than graphics, because each character receives special treatment; for a start text needs 2 full sized screenshots with and without the browser's rendered text, in order to deduce whether a character is hidden.

However there seem to be some situations where a little bit of text is rendered but text below the fold isn't. This could be to do with the fact that there are 2 text rendering parses, 1 frequent parse to render the currently viewed TTY screen and a less frequent parse to render 6 times the TTY's height. Maybe the larger parse is buggy?

Link clicking feedback

Links can fail to load for a number of reasons;

1. The terminal doesn't support mouse clicks.
2. Normal web page bugs; the page is busy, the link isn't really a link, etc.
3. The new page causes Browsh's renderer to crash. This needs to be reported at the bottom of the screen with the other HTTP status update messages.

To help with 1 and 2 a nice feature that would help a little here is that if a link is clicked then it might flash briefly, acknowledging at least that user did something.

Raw text pages should be incrementally parsed

Since the fix in 9ce338f to restrict large pages to a certain size, there now needs to be a way to limit the size of canvas screenshots. So in order to render pages larger than the limit pages need to be parsed incrementally.

Vim mode

We can take the lead from something like: https://vimium.github.io/

Many Chinese characters are not rendered, or appear with blurred background

This is because the custom fonts aren't generated to include the full range of CJK characters. There's a bug in the webext/contrib/font_maker.py script, for glyphs above about 0x7f00, we get this error:
Internal Error: Attempt to output 81854 into a 16-bit field. It will be truncated and the file may not be useful.

Native copying and pasting don't work

I'm pretty sure this is because the native cursor is constantly moving due to the screen updates. I'm reluctant to automatically turn off frame updates when an input box is selected as there are some nice features like Googl'e autosuggestions that would be lost. So is the only solution is to have a shortcut key that a user can press to pause frame updates so they can copy and paste?

What to do about TTY resizing?

It'd be good if you could dynamically resize your terminal, because:

1. It's just convenient when rearranging your windows.
2. You get an increase in resolution when the TTY is bigger.
3. You get a reduction in bandwidth when the TTY is smaller.

The main obstacle I see is how to negotiate the dimensions with Xvfb. When texttop first starts it assumes a desktop width of 1200 and decides the height depending on the TTY width/height ratio. So Xvfb is locked into that resolution and xrandr (which can dynamically resize the desktop) doesn't work with Xvfb, besides it would be clunky anyway I reckon. Also those desktop dimensions are passed onto ffmpeg, so it knows which part of the desktop to stream.

So. My first thought is just to retain this initial size and aspect ratio for the duration of the session. If the TTY size changes then scaling can happen but the aspect ratio is static. The only downside is that it's then possible to have large, unused areas of the TTY.

I cant use texttop

I've read some of the other issues but even there is some similar one ( #6 ) i can't launch correctly this (maybe some i forget some step... i dunno).

What i do, and what i get is:

Execute:
docker run --rm -it tombh/texttop sh
Then i execute the run.sh:
/app # ./run.sh

In the screen we can read:

Starting Xvfb...
Starting Firefox...
Starting ffmpeg video stream of desktop...
Starting mouse and keyboard interface...
/app # Killall: hiptext: no process killed
/app #

An nothing more happens.
I've copied and attached the different logs

ffmpeg.log:

ffmpeg version 2.8.5 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 5.3.0 (Alpine 5.3.0)
configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libmp3lame --enable-librtmp --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-x11grab --disable-stripping --disable-static --enable-vaapi --enable-libopus --disable-debug
libavutil 54. 31.100 / 54. 31.100
libavcodec 56. 60.100 / 56. 60.100
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 40.101 / 5. 40.101
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100
[x11grab @ 0x564a4de1e3e0] 16 bits per pixel screen is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
Input #0, x11grab, from ':0.0+1600':
Duration: N/A, start: 1470317325.273694, bitrate: N/A
Stream #0:0: Video: rawvideo (RGB[16] / 0x10424752), rgb565le, 1600x835, 12 fps, 12 tbr, 1000k tbn, 12 tbc
Output #0, mpegts, to 'udp://127.0.0.1:1234':
Metadata:
encoder : Lavf56.40.101
Stream #0:0: Video: mpeg2video, yuv420p, 1600x835, q=2-31, 200 kb/s, 12 fps, 90k tbn, 12 tbc
Metadata:
encoder : Lavc56.60.100 mpeg2video
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg2video (native))
Press [q] to stop, [?] for help
frame= 8 fps=0.0 q=2.0 size= 26kB time=00:00:00.50 bitrate= 433.2kbits/s

hiptext.log:

Illegal instruction (core dumped)
The processor is a Intel(R) Xeon(R) CPU 3075, so #4 shoud not be applicable.

input.log:

Starting...
Desktop dimensions: W: 1600, H: 835
Term dimensions: W: 184, H: 48

interfacer.log:
(It's empty)

xvfb.log:

Xlib: extension "RANDR" missing on display ":0".

ERROR: unsatisfiable constraints: xdotool (missing)

I'm trying to rebuild to make this work on my server with an Intel E5520 because I get "Illegal instruction" when running hiptext. I assume the build for that package uses some new features that my processor doesn't have. However, it looks like Alpine Linux has dropped xdotool?

fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.3/main/x86_64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.3/community/x86_64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/testing/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
  xdotool (missing):
    required by: world[xdotool]
The command '/bin/sh -c apk add --no-cache bc xvfb ttf-dejavu xdotool@testing ffmpeg openssh mosh firefox dbus' returned a non-zero code: 1

Magnifying glass

I'm thinking of a mode where clicking creates a slightly zoomed box of the page under the browser.

Doesn't seem to work on Mac with Iterm

I am trying the quickstart and it does show the firefox launch screen, it does not respond to any input or clicks, neither keyboard shortcuts.

What could I be doing wrong?

Implement a client side caching mechanism

Generate a md5sum for every full page of text that is intended to being sent to the client. Send md5sum and the full page text/data, with the md5sum in the front.
When the client detects it already displayed the text with the same md5sum before, because it is in its cache, it can cancel loading the rest of the text/data and therefore lower the total bandwith required.
Now the client can simply display the text/data from its cache.
Possibly there are more elegant ways to implement a mechanism like that, this is just food for thought.

I'm an addict to multiple websites and in reality they change a lot less often than I'm frequenting them.

And then there are webpages that don't change at all over long periods of time like documentation for APIs and so forth.

The cache on the client side (be it the CLI or a future GUI) should be compressed to save space, of course.

Doubts on how to use, can't run it successfully

I issued the following command on the server

docker pull tombh/texttop

After that The following commands on server

docker run -d \
  -p 7777:7777 -p 60000-60020:60000-60020/udp \
  -v ${HOME}/.ssh:/root/.ssh:ro \
  tombh/texttop

Everything worked fine with no errors on the server so far

After issuing the following commands on the terminal of the local machine, it keeps on watitng for input with no error message or prompt

mosh --ssh="ssh -p 7777" user@yourserver

I wrote the following commands

cd /app
./run.sh

Still no response from the server, Can you please tell me what I am doing wrong

Although I am able to ssh into the server with the normal ssh command.

Rendering sanity check

Unfortunately sometimes pages get rendered wrongly by browsh's webext.

To the human eye this is obvious with missing text or distorted graphics.

Browsh should try to perform a sanity check on the rendered output.

Red flags are:

• No text, or very little text especially on websites with a lot of text nodes. If there's text in the source one can assume it is supposed to be rendered somewhere.
• Instead of boxes/rectangles being in straight lines, they are distorted. It's probably not easy to implement an efficient, reliable algorithm for this check though. For example instead of this:

|  b  |
|  o  |
|  x  |

you get:

|  b  |
  |   o  |
    |   x  |

• Very little text with a huge variety in character coloring instead of the more expected regular colors(usually black) for text. I've experienced this a few times.

If the sanity check fails, Browsh should not send anything to the client and instead try to rerender the page until it succeeds. If rerending fails n times, it should rather send a failure page/report to the client instead of corrupted renders.

It's better to prevent corrupted renders in the first place by fixing browsh's rendering mechanism. However I'm assuming this will not always be possible and therefore a sanity check would save the user's time and bandwith.

Strange glyphs in normal Firefox after starting browsh

First: Thank you for this nice project!

I wanted to try it and installed the v0.2.5 Arch Linux package. I already have Firefox 58.0.1 installed. So I started browsh and got this output:

Starting Browsh...
Starting...

Then it exited with error code 1. I then tried to start my normal Firefox again and now I get this glyphs instead of a normal font:

They're even there after a reboot.

ENTER and ESC keys don't always work

For instance on Google News the search box doesn't register the ENTER key.

The ESC can't ever be used to close things like image viewer overlays.

Wide characters need to be CSS rendered as wide characters

Wide characters like 大 take up 2 cells in a TTY, so they also need to take up 2 'cells' in the browser grid.

Texttop on ARM

Hi Thomas,
I am really excited about Texttop. It would be great supporting ARM. I tried building it on Archlinx armv7l, but I receive an error message.

Here it is, if you want to investigate. If not, it is an amazing tool anyway;-)

Thanks & Regards,
Jens

docker build -t texttop .
Sending build context to Docker daemon 179.7 kB
Step 1 : FROM alpine
latest: Pulling from library/alpine

fae91920dcd4: Pull complete 
Digest: sha256:ea0d1389812f43e474c50155ec4914e1b48792d420820c15cab28c0794034950
Status: Downloaded newer image for alpine:latest
 ---> f70c828098f5
Step 2 : COPY . /app
 ---> a5db764d3a31
Removing intermediate container a42ad72418af
Step 3 : RUN echo "http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.3/main" > /etc/apk/repositories
 ---> Running in 9014dc5c1e50
rpc error: code = 2 desc = "oci runtime error: exec format error"

Will Browsh support browsers other than Firefox?

First renders are sometimes incorrect

When this happens it kind of reminds me of old cathode ray TVs turning on/off.

It's problematic for slow connections that have to send the incorrect rendered data over the wire, only to later settle on the final frames. Ideally we need to figure out why these incorrect renderings happen. If not then there needs to be a grace period where unstable frames are not sent.

My only clue is that it is caused by inconsistent dimensions and unfinished browser resizes.

Completely filling the TTY

Firstly, Firefox doesn't completely fill the desktop at first load. That could be because of run.sh not correctly calculating values, or xzoom not correctly grabbing the correct area of the left half of the desktop. You can see that, upon zooming, Firefox does fill the TTY (apart from a single col and row) which means ffmpeg is at least setup correctly.

The single row and col gap is do to with hiptext, maybe it can be forced.

Scrollbars

Just little single pixel-wide darkened bars should do the job.