There are dozens of Open Source TCP proxies available, written in close to a dozen languages, many of them capable of load balancing. Many of them would work with "distcc". Why write yet another TCP proxy? Why do it in Erlang?
All of the TCP proxies I found, none appeared to have the following combination of features:
1. Not be too HTTP-centric to not be able to work with "distcc".
2. Be aware that some back-end hosts may be faster than other
hosts. For each client connection, the proxy should choose the
fastest back-end host that is currently idle.
3. Be aware of back-end hosts with multiple CPUs.
4. When all back-end hosts are busy, make the client wait for the
next available back-end host when it is available, rather than
giving a back-end host more work than it is configured to
handle.
5. Detect when a back-end host is down and do something sane (like
avoid giving future jobs to the dead machine).
6. Permit an administrator to put back-end hosts back in service,
take them out of service, as well as add and remove hosts from
the pool without adversely affecting clients using the proxy.
7. Keep basic statistics about back-end hosts and make them
available via HTTP or Telnet.
Features 1-5 were mandatory. Features 6-7 would be nice. In a couple of hours of Web surfing, I didn't find a TCP proxy that was capable of doing 1-5, so I decided to write my own.
I knew it would be fairly easy to implement features 6-7 as well as 1-5 in Erlang (see http://www.erlang.org/), so that's what I used.
This proxy has been in use at Caspian Networks for over two months. It's pretty solid.
This README file is quite long. Sorry about that. However, much of it is a tutorial for Erlang newbies ... and perhaps a bit of evangelism. :-) I'll try to keep things straightforward, but I will also demonstrate some of the nifty communication, fault-tolerance, and hot code upgrade features of Erlang.
See the file "LICENSE", at the top of the tcpbalance source distribution, for full licensing terms.
Oh boy, yet another programming language development environment to compile and install. Well, if you're still reading this, you're interested enough in tcpbalance's feature set to try it out.
The current release of Erlang (as of 15 January 2003) is R9B-0. All of the source and pre-compiled packages mentioned below can be obtained from http://www.erlang.org/download/.
- UNIX and Linux users:
Erlang runs quite well under: Linux, BSD flavors, Solaris, and others.
Although it's slow, I recommend obtaining the source and compiling from scratch. The distribution is pretty big (8MB compressed) and contains a kajillion files ... but the Erlang programming & runtime environment contains many useful tools (including a nearly-complete CORBA implementation) that are themselves written in Erlang. Tcpbalance doesn't use most of them, but you get to compile all of them. :-)
Follow the included directions, but the simple instructions are:
1. Extract the source package and change directory to its root.
2. Run "./configure" to use the default installation root
("/usr/local" on most platforms) or
"./configure --prefix=/path/to/install/root" if you wish to use
another installation root path.
3. Run "make".
4. Go do something else for a while. The Erlang VM is implemented
in C, which compiles in a few minutes, but most of the run-time
environment & misc tools are implemented in Erlang itself, and
there's a *lot* of it.
5. Run "make install".
6. Add /path/to/install/root/bin, whatever it is, to your shell's
program path.
- MacOS X users:
I haven't used it, but there's a pre-compiled, disk image-style installation thingie available. Give it a try if you wish.
- Microsoft Windows users:
Your only option is installing a pre-compiled package. Whoo hoo!
When I installed Erlang R9B-0 on a Windows NT 4.0 machine, I discovered that "C:\Program Files\erl5.2\bin" was already added to my program path. What a deal.
One of the really nice things about Erlang is that communication (via message passing) between threads inside an Erlang virtual machine is exactly the same as message passing between threads on different Erlang virtual machines.
An instance of the Erlang virtual machine is called a "node". Erlang threads are called "processes", which can really confuse UNIX geeks if you're not careful. In this document, if I want to refer to a UNIX or NT process/task, I'll call it an "operating system process" or "OS process" to avoid confusion.
Erlang's inter-node message passing relies on a simple shared secret mechanism similar to the X11's "MIT magic cookie" authentication scheme. Erlang stores a "cookie" in $HOME/.erlang.cookie (where $HOME is your account's home directory) on UNIX boxes and in C:.erlang.cookie on NT boxes. To communicate with each other, all Erlang nodes must share the exact same cookie.
(There are other ways to configure cookies, but I'm only going to describe one.)
For nodes running on the same machine, there's no problem: everybody is sharing the same file system(s).
For nodes running on different UNIX machines, you'll need to: 1. Use NFS or another shared file system for your $HOME directory. 2. Copy the .erlang.cookie file by hand to the $HOME directory of each machine you wish to run an Erlang node.
For nodes running on different NT machines, you'll need to copy the .erlang.cookie to each machine.
If you're mixing NT and UNIX machines (which is certainly possible), make certain the exact same .erlang.cookie file is used on all of them.
To make life easier on yourself, make certain that all of the machines involved are present in DNS. E.g. if you're playing with "davinci" and "munch", make certain that "ping davinci" and "ping munch" works on both machines.
If a machine has a hyphen, "-", in its DNS name ... choose another machine. Erlang's syntax requires treating hyphens specially. This example is complicated enough as it is. (See footnote [1] below.)
An Erlang node name looks like: foo@hostname or: [email protected] For simplicity, we'll use the former.
Each node must have a unique node name. For nodes running on the same machine, the lefthand side of the "@" must, therefore, be unique. If you attempt to start two nodes on the same machine with the same name, the second VM will spit out a very long and cryptic error message (including the string "Kernel pid terminated").
For my example, I'm going to use the nodes 'foo@davinci' and 'bar@rover'.
NOTE: Erlang commands are case-specific!
On each machine, start the Erlang VM and interactive shell: 1. UNIX: run "erl -sname NODE_LHS". E.g. "erl -sname foo" 2. NT: run "werl -sname NODE_LHS" E.g. "werl -sname foo" 3. Mac OS X: I've never used it, so you're on your own.
You will see something like:
Erlang (BEAM) emulator version 5.2 [source] [hipe] [threads:0]
Eshell V5.2 (abort with ^G)
(foo@davinci)1>
Note that the node name is included in the prompt string.
Type the command "erlang:get_cookie()." and press Enter. You should see:
(foo@davinci)1> erlang:get_cookie().
'JQHZIQLDNQGUSZRAJXHB'
You should see the same cookie on each node. If not, go back and fix it.
Now, we'll test the inter-node message passing capability. One one of your nodes, type the following:
(foo@davinci)2> register(test, self()).
true
(foo@davinci)3> receive Msg -> Msg end.
You won't get see a prompt right away: you're blocked waiting for a message to arrive.
On the other machine, type the following, substituting the other machine's node name:
(bar@rover)1> {test, 'foo@davinci'} ! hello_world.
hello_world
On the first machine, you should see:
(foo@davinci)3> receive Msg -> Msg end.
hello_world
(foo@davinci)4>
If you run the function "nodes()", you should see the name of all other nodes that your node is aware of.
(bar@rover)3> nodes().
[foo@davinci]
(bar@rover)4>
Congratulations! It's now time to play with the tcpbalance application. Use the command "q()." to exit the shell, or press Control-c.
I haven't spent any time trying to auto-magically edit the files that will need editing before you can use tcpbalance. I've made life "easier" by using relative, not absolute paths. Therefore, no extra configuration or file editing should be necessary.
This means that you must change the current working directory exactly as described below (or else things won't work). This is not how a "real" Erlang application would be installed & run, but I haven't taken the time to do that. Sorry.
If you want the built-in HTTP server to use a port other than port 8080, edit the file "priv/inets.conf" and modify the "Port" directive.
The file "src/Makefile" requires GNU Make. To use it:
% cd src
% make
If you do not have GNU Make, or if you're using NT and don't have GNU Make available, execute the commands found in the file "Make.all.out".
The file src/balance.rel contains version numbers that are specific to a particular Erlang/OTP release version. If you see error messages like this:
stdlib: No valid version ("1.11.0") of .app file found. Found file
"/usr/local/lib/erlang/lib/stdlib-1.11.4.1/ebin/stdlib.app" with
version "1.11.4.1"
kernel: No valid version ("2.8.0") of .app file found. Found file
"/usr/local/lib/erlang/lib/kernel-2.8.1.1/ebin/kernel.app" with
version "2.8.1.1"
gmake: *** [balance.boot] Error 1
... then you have a different version of Erlang/OTP than I originally used for tcpbalance. Edit the file src/balance.rel to replace the invalid version numbers with the version numbers mentioned at the end of each error message.
Run the following:
% cd src ... if you haven't already
% erl -sname bal -pz ../ebin -boot balance -config ../priv/be-list -noshell
You will see a whole bunch of diagnostic messages, labelled "PROGRESS REPORT". The last one will say something like:
=PROGRESS REPORT==== 16-Jan-2003::13:59:45 ===
application: balance
started_at: bal@davinci
Congratulations. This means that the application is running successfully. If this isn't what you see, and you're certain that you've followed all the directions, cut-and-paste the output and email it to me. (See footnote [2] below.)
The example configuration file, "../priv/be-list.config", is a proxy for two SMTP servers, mx1.mail.yahoo.com and mx1.hotmail.com. The proxy is listening to local port 2525.
% telnet davinci 2525
Trying 10.10.10.10...
Connected to localhost.localdomain.
Escape character is '^]'.
220 YSmtp mta614.mail.yahoo.com ESMTP service ready
quit
221 mta614.mail.yahoo.com
Connection closed by foreign host.
See the "be-list.config" file for full details of the configuration. To summarize:
1. The proxy's local TCP port is 2525.
2. The back-end connection timeout is 10 seconds.
3. The back-end connection activity timeout is 2 minutes.
4. The back-end host list:
a. mx1.mail.yahoo.com, TCP port 25, 2 simultaneous sessions.
b. bogus-demo, TCP port 25, 1 simultaneous sessions.
c. mx1.hotmailcom, TCP port 25, 1 simultaneous sessions.
Use a Web browser to connect to the Web server running on TCP port 8080 on the machine running the balancer, e.g. http://davinci:8080/ and follow the link there. You'll see something like (edited to fit in 80 columns):
Proxy start time: 2003/1/16 13:59:44
Current time: 2003/1/16 15:39:21
Local TCP port number: 2525
Connection timeout (seconds): 10.0000
Activity timeout (seconds): 120.000
Length of wait list: 0
Name Port Status MaxConn ActConn ActiveCount ActiveTime
mx1.mail.yahoo.com 25 up 2 0 1 2.94391
bogus-demo 25 up 2 0 0 0
mx1.hotmail.com 25 up 1 0 0 0
Now, do the following:
1. Open four windows: xterm, terminal, Telnet application, or
whatever.
2. Use those windows to create four simultaneous TCP connections
to the proxy. E.g. "telnet davinci 2525".
3. The first two clients should see greetings from a Yahoo mail
exchanger.
4. The third client should see a greeting from a HotMail mail
exchanger.
5. The fourth client should connect but otherwise see nothing.
6. Type "QUIT" in the second client to terminate the session.
7. The fourth client should then be connected to an available
back-end host, namely a Yahoo server.
8. In the second window, connect to the proxy again.
9. Retrieve (or reload) the balancer's stats via its HTTP server.
You should see that all three back-end sessions are busy, that
there's one client in the "wait list", and that the status of
the "bogus-demo" server has been changed to "down".
I haven't extended the balancer's HTTP server to be able to change the balancer's config on-the-fly, but it's easy enough to do using Erlang's native message passing mechanism. It's clunkier than "click here to change BE's status to 'down'", but hey, this was an afternoon's hack!
Run "cd src" (if you aren't already there) and "erl -sname foo -pz ../ebin" on any machine that you've verified that the Erlang cookies are correct & message passing works, then run the following Erlang shell commands (changing the node name as appropriate):
1. bal_proxy:get_state({balance, 'bal@davinci'}).
2. bal_proxy:reset_host({balance, 'bal@davinci'}, "mx1.mail.yahoo.com", down).
3. bal_proxy:get_state({balance, 'bal@davinci'}).
The first and third commands return raw state data maintained by the balancing process: the HTTP server simply pretty-prints this data. The second command sets the state of the "mx1.mail.yahoo.com" back-end host to 'down'. ('up' is the other valid state)
Other commands to experiment with are (you don't have to type them on a single line, but you should, unless you're familiar with Erlang syntax):
bal_proxy:get_host({balance, 'bal@davinci'}, "mx1.hotmail.com").
bal_proxy:reset_all({balance, 'bal@davinci'}).
bal_proxy:del_be({balance, 'bal@davinci'}, "bogus-demo").
bal_proxy:add_be({balance, 'bal@davinci'}, {be,"mx-ca-1.pobox.com",25,up,1,0,0,no_error,0,0,0,[]}, "").
bal_proxy:add_be({balance, 'bal@davinci'}, {be,"smtp.TheWorld.com",25,up,1,0,0,no_error,0,0,0,[]}, "mx1.hotmail.com").
Use the bal_proxy:get_state() function to see how these functions affect the state of the balancer.
NOTE: You probably shouldn't delete a back-end host unless you've marked its status as 'down' first ... and then waited for all active sessions to finish. :-)
NOTE: The proxy has a bug (one of several, see comments in src/bal_proxy.erl) that happens if: 1. All back-end hosts are status 'down' 2. A proxy client connects. 3. A back-end host is marked status 'up' Work-around: Don't allow this to happen.
One of the many applications distributed with Erlang is called "appmon", the application monitor. To start it, run "erl -sname something" on any machine that you've verified that the Erlang cookies are correct & message passing works, then run:
appmon:start().
A GUI box should pop up that displays a tree of the applications running on the local node. In your case, "kernel" is probably the only application running.
If you haven't already done so, run this command in the Erlang shell (using the balancer's node name, of course):
bal_proxy:get_state('bal@davinci').
Then pull down the "Nodes" pull-down menu. Both the local node and 'bal@davinci' should be listed. Select the balancer's node. You should then see a tree of three applications running on the balancer: kernel, sasl, and balance. Click on the "balance" box.
Another window should appear. This window displays the tree of processes (Erlang threads, remember!), including "supervisor" processes, used by the balancer application.
* A top-level supervisor, named 'balance_sup'.
* The 5 processes used by the "inets" HTTP server.
* A variable number of processes used by the TCP balancer portion
of the application: the socket listener process will always
appear under the 'balance' process, as will the transient
per-TCP-session processes.
Now, create a TCP connection to the balancer's port. The tree will be updated to show a second process underneath 'balance'.
A basic programming philosophy behind Erlang is "code only for the common case". If there's an error, e.g. divide by zero or an uncaught exception, the default action is to kill the process. You rely on supervisor processes to restart abnormally-terminated processes. Very nice.
You can use "appmon" to send kill signals to any process it displays, thus simulating a bug/software failure. Do the following:
1. Make a connection to the proxy's local port.
2. Note which process appears under 'balance' when the appmon
window is updated. This is the process used to copy data back
and forth between your client and the back-end host.
3. Click on the "Kill" button.
4. Click on the process box representing the process found in step
number 2.
This will terminate your client's TCP connection, as well as the proxy's connection to the back-end host.
You can have more fun with this fault injection by:
* Killing the socket accept()ing process underneath 'balance'.
* Killing 'balance' or any of the HTTP server's processes.
If you do any of those things, the 'balance_sup' supervisor will kill all remaining application processes and restart them. It may happen so quickly that the appmon window doesn't appear to change. However, look carefully at the process ID numbers of the socket listener underneath 'balance' and the process underneath 'httpd_acc_sup_8080': both will change, indicating that they aren't the same process that used to be there. That's the supervisor in action.
If you kill the 'balance_sup' process or any of its parents, the proxy will crash and exit. That's because 'balance_sup' doesn't have a parent supervisor to restart it. However, the only things that supervisors do are:
1. Start child processes.
2. Monitor those children
3. Restart any of those children, if they should be restarted.
4. Exit if there are too many child failures within a configured
amount of time.
The supervisor's code is assumed to be bug-free. It probably is. :-)
Typical Erlang application design uses a tree of supervisors. If there's a low level problem severe enough that the immediate supervisor cannot deal with it, that supervisor will exit ... in effect, passing the problem up the supervisor chain. In the worst case, the top-level supervisor can kill everything and restart from scratch. This kind of deterministic application startup and fault handling is quite nice.
The 'balancer_sup' supervisor is configured to tolerate up to 5 child deaths within a 30 second time period. That probably isn't realistic for real-world use, but it's fun for demonstration purposes.
For more detail on the process supervisor scheme of OTP, the Open Telecom Platform, see http://www.erlang.org/doc/r9b/doc/system.html, in particular the "Design Principles" document.
Like several other functional programming languages, Erlang permits on-the-fly, hot code update. The Erlang VM supports the notion of two simultaneous loads of any particular module, current and new. It's way beyond the scope of this README to describe how this works or how the Erlang OTP release handler can upgrade (or downgrade!) the running code (and associated data structures) of one or more applications. It's a nifty, if complex, feature.
This example will be much more basic: we'll add a line of output to the HTTP server's status overview. We would like the summary at the top to include the balancer's Erlang node name. That's easy enough to do. Follow these steps:
1. Edit the file "src/bal_proxy.erl" with your favorite text
editor.
2. Locate the word "README", about 90% of the way from the top.
You will insert code on the line immediately after this
comment. (The "%" character denotes the start of an Erlang
comment; they continue to the end of the line.)
3. Add the following line after the comment you found in step #2:
io_lib:format("Proxy's Erlang node name: ~w\n", [node()]),
4. Change working directory to "src", if you haven't already.
5. Run "make", otherwise run
"erlc -bbeam +debug_info -o../ebin bal_proxy.erl"
Now that you've made the code change and recompiled it, we just need to tell the balancer to load the new code.
First, just to make it really obvious what's going on, use your Web browser to retrieve the balancer's current stats.
The balancer node was started using the "-noshell" command line flag, so there is no Erlang shell available for us to modify the balancer's internals. So, we'll start a second node, then create a shell session on the balancer's node, then use the second node's shell to communicate with the balancer's node.
Run "cd src" (if you aren't already there) followed by "erl -sname foo" on any machine that you've verified that the Erlang cookies are correct & message passing works, then type the following things (changing the node name as appropriate):
1. Control-g
2. r bal@davinci ENTER
3. j ENTER
4. c 3 ENTER
You should see something like this:
% erl -sname foo
Erlang (BEAM) emulator version 5.2 [source] [hipe] [threads:0]
Eshell V5.2 (abort with ^G)
(foo@rover)1>
User switch command
--> r bal@davinci
--> j
1 {}
2 {shell,start,[]}
3* {bal@davinci,shell,start,[]}
--> c 3
Eshell V5.2 (abort with ^G)
(bal@davinci)1>
Any command you type in this command shell will be executed on the 'bal@davinci' node, not the local one. Pretty slick, huh?
Type the command "l(bal_proxy)." into this shell, and you'll see:
(bal@davinci)1> l(bal_proxy).
{module,bal_proxy}
(bal@davinci)2>
Now, tell your Web browser to reload the stats page. Notice that your new code has indeed been executed!
See the file "priv/sample-distcc.config" for an example config for 6 back-end machines with different numbers of CPUs and different CPU speeds.
If you have questions, bug reports, etc., please email them to me. My email address is in footnote [2] below. Tcpbalance isn't meant to be a 100% bulletproof, full-featured distcc application proxy ... but that doesn't mean that I'm not willing to help out or perhaps fix bugs.
Martin Pool, distcc's maintainer, suggested that this Erlang proxy could be a model for a "real" bulletproof, full-featured distcc application proxy that someone might write someday. I think that is a great idea! In the meantime, I'll continue using this proxy....
-Scott Lystig Fritchie
[1] If you really want to use a machine with a hyphen in its DNS hostname, you can do it. You just need to put single quotes around the node name whenever you use it. For example, if you run "erl -sname foo" on a machine called "nt-regal", then whenever this document asks you to type a node name into the Erlang shell, you must type: 'foo@nt-regal' ... instead of: foo@nt-regal The latter is incorrect Erlang syntax.
Technically, a node name is treated as an Erlang atom, a primitive data type, much like an atom in Lisp or Scheme. An atom typically starts with a lowercase letter and may be alphanumeric or underscore ("_"). However, if you want an atom to contain other characters or to start with an upper-case letter, it can be surrounded by single quotes. For example, 'This_is_an_atom' is valid syntax for an atom.
[2] My Internet email address: the lefthand side of the "@" is "slf". The righthand side is "caspiannetworks.com".
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent