chan-kelv/gosixdegrees

gosixdegrees's Issues

Add article not found page

Recursively distribute Wiki links amongst nodes

Crawl page by title

Add option to crawl endpoint by title

Node to Node RPC

Multiple instances of the server will be spun up to share the work of web crawling. One is designated as the master (with a command prompt flag -m) and the others as slaves (-s). On startup, the master node begins an rpc listen on a port to allow for the connection of slave nodes. Slave nodes must connect to the master to connect to the master.

• Master must keep track of all nodes connected to it
• May need to add command arg to let slaves know of master's IP:port
• RPC setup needs to occur BEFORE http requests can be made.

Listening example:
https://parthdesai.me/articles/2016/05/20/go-rpc-server/
https://golang.org/pkg/net/rpc/

go func() {
//turn serverIp into tcp address
listen, err := net.Listen("tcp", serverIp)
//Register new server with ability to handle MServer RPC methods
mServer := new(MServer)
server := rpc.NewServer()
server.Register(mServer)
for {
conn, err := listen.Accept()
//TODO - save the conn
}
}()

Slave connection example:

/*
Connect to a server. The server will keep a record of the worker who dialed
TODO - save the client that is returned
*/
func createClientToServer(serverIp string) (client *net.TCPConn, err error) {
address, err := net.ResolveTCPAddr("tcp", serverIp)
if err != nil {
fmt.Println("Worker client error", err)
err = fmt.Errorf("Worker client error", err)
}
client, err = net.DialTCP("tcp", nil, address)
if err != nil {
fmt.Println("Worker client error", err)
err = fmt.Errorf("Worker client error", err)
}
return client, nil
}

func setWorkerListenToServer(conn *net.TCPConn) {
server := rpc.NewServer()
workerRpc := new(WorkerRpc)
server.Register(workerRpc)
server.ServeConn(conn) //this hangs while connecting to the conn obj
fmt.Println("Server connection broken") //this executes after connection breaks
}