abingham/traad

An JSON+HTTP server for the rope Python refactoring library

Traad: A Python refactoring server

Traad is an refactoring server for Python code. It listens for HTTP requests to perform refactorings, performs them, and allows clients to query for the status. There is also talk from EuroPython 2014 with design description and some live demo: https://www.youtube.com/watch?v=NvV5OrVk24c

Setup

To use traad you'll normally need both the server and a client.

Emacs

The only client code (that I know of) is the emacs-traad package for Emacs. emacs-traad is able to install the server for you, so if you're using it you should just need to follow its setup instruction.

Installing the server with pip

If you just want to install the most recent release of the Python server components (i.e. this project), you can use pip:

pip install traad

Installing the server from source

If you want to install the server from source - perhaps because you're doing development on it - you should first clone the repository:

git clone https://github.com/abingham/traad

Then you can install everything with setup.py:

cd traad
python setup.py install

Python 2 vs. Python 3

Note that if you install traad into a Python 3 environment, the server name will be traad3. If you install it into a Python 2 environment it will simply be traad. This makes it somewhat simpler to have system-wide installations of both on systems with both Python 2 and 3.

Tests

traad has a suite of tests in the tests directory. They are based on pytest. In order to run them, go to the project's root directory (i.e. the one containing this README) and use:

pytest tests

Rationale

I (the author of traad) use emacs for most of my Python development, and I've often been jealous of the cool refactoring tools that my colleagues get with their fancy IDEs. Not jealous enough to actually switch, of course. I'm way to stubborn for that. But I was jealous enough that I investigated the options available for emacs.

One of the best options available is the rope Python refactoring library. Rope is very powerful and does all of the things I'd like. Unfortunately, I could never find a satisfactory way of integrating it into emacs. The pymacs-based approaches never quite worked for me, and in any case that approach struck me as somehow incorrect.

So, in the spirit of open-source development, I decided to try my own approach. I wanted a way to access rope functionality without having to contort either emacs or Python in unnatural ways. Thus the idea of using a client-server approach was born. It may strike others as odd or incorrect, but it works well for me.

Design

Traad is a client-server approach to using the rope Python refactory library. It involves two basic components:

1. A HTTP server exposing the rope API via JSON, and
2. Client libraries (currently just emacs lisp) for talking to the server

The hope is that this approach will make it easier - at least in some cases - to use rope from various other tools.

Since rope is written in Python, any tool that wants to use rope needs to either embed Python, or it needs to find a way to communicate with a Python process running rope. The embedding approach is difficult if the target environment is not written in Python, and it also faces challenges when dealing with more than one Python version.

So traad aims to simplify communication with rope running in an independent process. HTTP communication and JSON data handling is well supported in many, many languages, so any environment that wants to use rope should be able to easily communicate with traad.

traad's Issues

Add support for code completion

Not sure how much this involves, but it seems like a really nice thing to have.

Please stop bundling third-party libraries

traad is mirrored on the Emacsmirror, which is a large up-to-date collection of Emacs packages.

As the maintainer of the mirror I am trying to resolve feature conflicts that result from one package bundling libraries from another package. I suspect in most cases these libraries were included so that users would not have to find, download and install each dependency manually.

Unfortunately bundling also has negative side-effects: if the bundled libraries are also installed separately, then it is undefined which version actually gets loaded when the respective feature is required.

Initially that isn't a big problem but in many cases upstream changes are not included or only after a long delay. This can be very confusing for users who are not aware that some of the installed packages bundle libraries which are also installed separately. In other cases bugs are fixed in the bundled versions but the fixes are never submitted to upstream.

Also now that Emacs contains the package.el package manager there is a better way to not require users to manually deal with dependencies: add the package (and when that hasn't been done yet the dependencies) to the Melpa package repository. If make is required to install your make you might want to add it to the el-get (another popular package manager) package repository instead.

Alternatively if you want to keep bundling these libraries please move them to a directory only containing bundled libraries and add the file ".nosearch" to that directory. You can then load the library using something like this:

(or (require 'bundled nil t)
    (let ((load-path
           (cons (expand-file-name "fallback-libs"
                                   (or load-file-name buffer-file-name)
                                   load-path))))
      (require 'bundled)))

Of course if your version differs from the upstream version this might not be enough in which case you should make an effort to get your changes merged upstream.

traad bundles at least the following libraries:

• xml-rpc

Best regards,
Jonas

Support the rope.refactor.move refactorings

Since trace now raises exceptions, we may need to fix network interface

The trace decorator used to catch exceptions and return them as strings. But I recently changed it to reraise the exceptions so that the code was more testable. This will probably mean that diagnostics from the client end will be more difficult, so we may need to introduce a layer between RopeInterface and the xmlrpc server which translates exceptions into strings.

Findit buffer should not require user to specify path

In the findit buffer emacs will prompt you for "which directory contains this file", but we shouldn't require that. In a grep buffer you'll see that the header contains a root directory, so maybe we can leverage something like that.

Really, we just need to understand the whole "clickable file locations in a buffer" protocol/API.

Add file/module renaming function

Something like traad-rename-file. This will determine the current file and call traad-rename appropriately.

Look into how we might provide progress feedback

A failed extrat can block the processing queue.

If you do something like an extract with an empty name, this will cause the server to hand. Presumably we don't handle the resulting rope error properly.

Switch back to synchronous RPC calls until xml-rpc gets fixed.

Currently we don't seem to be getting return values properly from xml-rpc when we use async calls. This is bad because we can't give good feedback to users when things go wrong. Need to consider going back to sync calls, or perhaps finding a new xml-rpc lib (or jsonrpc lib?) for emacs.

look into using start-process et. al for managing process

See e.g. clang-completion-mode.

FYI I made virtualenv wrapper for Emacs Lisp. It could help Traad installation problems.

I made virtualenv wrapper for Emacs Lisp to help installation of Emacs packages relying on Python modules: python-environment.el. I think it would help Traad installation.

You can just define a command like the following so that user can just do M-x traad-make-env to setup Python side of Traad.

(defun traad-make-env ()
  (interactive)
  (python-environment-run (list "pip" "install" traad-path-to-source)))

I am requesting for comment before fixing the API. If you are interested, please come in: tkf/emacs-python-environment#1

Server should print/log information about python failures rather than (apparently) swallowing them.

It seems that the server will silently swallow errors in the underlying Python code. We should probably be printing some diagnostics, at least at certain verbosity levels. Otherwise it can make diagnosis difficult when someone encounters a bug when using the server.

Add support for "unspecified port" when starting the server.

This would then pick some unused port on the system instead of forcing the user to find one. Not sure if there's an easy way to do this in elisp or if I need to resort to platform-specific stuff.

Investigate code-completion performance on larger projects

It seems that performance on larger projects (e.g. ipython) is not as good as it needs to be. There are reports of noticeable lag when typing. So, we need to look into that, see what might be done, etc. It could be that rope is the slow-down.

Disallow adding main project as a cross project.

Consider how to deal with unsaved files

What should we do if someone tried to refactor a buffer which is not saved? Performing the refactoring can lead to a messy situation. Perhaps we should disallow it, or perhaps have a variable that controls what to do.

Think about how the elisp interface should behave when no server is found

For things like auto-completion, we want the elisp interface to be very responsive (i.e. fail-fast) when no server is found.

Look at webtest for JSON api testing

setting traad-server-progam to a non-string should fail immediately

If you set traad-server-program to a non-string, it won't fail on the setting. It will fail later when you try to use the value (e.g. in traad-open). It should fail immediately.

Catch exceptions from rope API to avoid crashing server.

code-assist, for example, will throw exceptions from time to time. We should catch those and continue on when at all possible. We definitely shouldn't let the server crash.

Investigate getting more error/success information across the XMLRPC boundary

Right now there's not much information getting from the server across to e.g. emacs. For example, if an extraction fails, no information about the failure makes it over to emacs, making it difficult for users to figure out what's wrong. See what can be done about this.

Update elisp comments to match standard tool-ese

See prosjekt patch.

Support for undo/redo history

Users should be able to view their undo/redo history. They should also be able to undo/redo selected elements, and perhaps even ranges.

Investigate communicating edits to rope

I've heard that rope has support for telling it about changes made in the editor. Investigate whether this is true, and what we might do to support it.

Work on "extract method".

Finish findit functionality.

Do find_implementations() and find_definition(). These should be essentially the same as find_occurrences(), so refactor as needed.

traad-host: localhost.localdomain over 127.0.0.1?

In my Emacs, setting "localhost.localdomain" or "localhost" for traad-host does not work. I need to set it to "127.0.0.1". I don't know why so, but changing it to "127.0.0.1" may save other people like me.

I am using GNU Emacs 24.1.50.1 in Ubuntu 11.10.

Refresh traad-history buffer when history changes

When someone runs a refactoring, undo, redo, or whatever, we should refresh the history window when it's open.

Add support for looking up docstrings for user-entered strings

Right now we only look up docstrings for text in a buffer. It would be nice to let the user request a docstring for something not in a buffer, e.g. (traad-get-docstring "np.zeros")

Undo of a traad-rename-current-file fails.

If I rename a file to "yaks.py" and then undo that change, I get an error like this:

error in process filter: File /home/abingham/projects/udcheck/udcheck/yaks.py no longer exists!

We're probably just making some assumption about the existence of the un-renamed file (yaks.py in this case) somewhere in a buffer-refresh call or something.

Investigate support for multi-project refactorings

Just understand the issues a bit more. This is a bit fringe-feeling right now, but I'd like to avoid going too far down one design path that makes multi-project support difficult in the future.

When undoing a file rename, try to revert to old file.

Currently, when we undo a file rename we simply leave an open buffer on the file that no longer exists (i.e. the one that was undone.) We do this because it's simple.

A smarter solution might be to see if we can examine the results of the undo and determine which file we should be visiting instead. If the undo results include a description like "undid rename of foo.py to bar.py", then we can try visiting foo.py and removing the bar.py buffer.

I don't know if this information actually comes out of undo. I haven't looked into it yet.

traad-rename fails with time error

Emacs 23.1, Ubuntu 11.04

M-x traad-rename gives the following:
let*: Wrong number of arguments: #[(time) "����A�@���� :�& [...MORE GARBAGE] [time time high low micro nil 0 + * 1.0 ...] 6 ("/usr/local/share/emacs/23.1/lisp/calendar/time-date.elc" . 2655)], 0

Investigate "validate" cost and options

It looks like validate might be fairly expensive, and probably more so with larger projects. If so, how to we address this?

One option is to make validation an explicit option. In fact, this is probably a good idea in any event. We should expose validate as a server command.

But we could also allow users to set up a timer that calls validate every so often. Though perhaps this is best left to the client environment...if we expose validate, then e.g. emacs can be instructed to call validate every so often. Think it over...

Investigate "similar" and "global_" for extract methods

In extract operations (and maybe others) you can specify the "similar" and "global_" argument to get_changes():

extractor.get_changes(
name,
similar=True,
global_=True)

Investigate and understand these a bit better. We should consider supporting them in the API.

Request: make everything asynchronous

Isn't it better to use xml-rpc-method-call-async instead of xml-rpc-method-call? Asynchronous programming is bit hard but otherwise you can't use the advantage of having multiple processes. I prefer not waiting during traad does the job in background.

Allow async-call handler to properly revert buffers

The switch to asynchronous calls has broken the support for reverting buffers after changes have been made. It would be nice if a buffer could be associated with an async call, and that buffer reverted when the call finishes. I'm not certain, but I think that this will require a macro since the async handler (likely a lambda) will need some way to "remembering" the buffer to revert. Need to ponder this...

Add "drop" support to undo/redo.

These functions support an optional "drop" argument that determines if the change gets moved to the other (redo or undo) list after being run. We should consider supporting them in the elisp and server API.

Investigate the "importutils" functions

These might be nice to have. The investigation should happen soon, though the actual work may not happen in milestone 1.

Investigate server performance

Is it slow? The emacs client seems a bit sluggish, and it may be the server. (or it may be any number of things.) We should get some idea of how the server performs, and possible change over to something faster if possible.

Not working when compiled

I get the following traceback when tried M-x traad-rename-object.
This happens only when traad.el is compiled.

Debugger entered--Lisp error: (invalid-function traad-call)
  traad-call(rename "glob_list" "/home/takafumi/repos/watch/ipython/IPython/utils/path.py" 12110)
  traad-rename("glob_list" "/home/takafumi/repos/watch/ipython/IPython/utils/path.py" 12110)
  traad-rename-object("glob_list")
  call-interactively(traad-rename-object)
  (if (or (stringp (symbol-function cmd)) (vectorp (symbol-function cmd))) (execute-kbd-macro (symbol-function cmd)) (setq this-command cmd) (call-interactively cmd))
  (let* ((cmd (anything (if (and anything-execute-extended-command-use-kyr (require (quote anything-kyr-config) nil t)) (cons anything-c-source-kyr anything-execute-extended-command-sources) anything-execute-extended-command-sources)))) (unless (and cmd (commandp (intern-soft cmd))) (error "No command: %s" cmd)) (setq extended-command-history (cons cmd (delete cmd extended-command-history))) (setq cmd (intern cmd)) (if (or (stringp (symbol-function cmd)) (vectorp (symbol-function cmd))) (execute-kbd-macro (symbol-function cmd)) (setq this-command cmd) (call-interactively cmd)))
  anything-execute-extended-command()
  call-interactively(anything-execute-extended-command nil nil)

I don't see any reason why you want make it as a macro. The following should do the same job.

(defun traad-call (func &rest args)
  "Make an XMLRPC to FUNC with ARGS on the traad server."
  (apply
   #'xml-rpc-method-call
   (concat
    "http://" traad-host ":"
    (number-to-string traad-port))
   func args))

Use asynchronous calls where possible/reasonable.

Refactorings can take a long time on large code bases, so we should run them asynchronously when possible. This is probably all of the time, but I need to investigate the issue.

Try without system installation

Hi, this is awesome project! I feel limitation on the Pymacs's serial execution nature, too. This is great you are breaking the limitation.

BTW, I want install traad using el-get without installing it system-wide (or using virtualenv) because it is easy to sync elisp and python parts this way. The problem is that there is no easy way to get executable without installing it. I suggest two solutions:

1. Put traadserver.py on the top level which contains

   #!/usr/bin/env python
   from traad.server import main
   main()

2. Allow traad-host to take a list. This way, I can do:

   (setq traad-host '("python" "-m" "traad.server"))

Investigate contrib.finderrors

This could be a nice to have.

Task information should include a description of the task

Right now we just return the completion status of a task. We should also include information on what it was...rename X to Y, etc.

Investigate contrib.autoimport

Also nice.

Do we need a "traad-reopen" function?

It seems that sometimes we need to "reboot" a rope project so that it returns the correct information.

ExtractVariable does not replace extracted code with new variable.

It appears that extract-variable will indeed extract the variable, but it doesn't seem to replace the extracted code with the new variable. I think this is a short-coming in rope itself, so we need to investigate. Maybe this just results in a bug report to rope.

Support for ExtractVariable

Should be simple.

traad simply does not work for me

I don't really know how to give a test case for this, because nothing at all works. I specifically tried a lot to get traad-code-assist to work; everytime I would set "is the server running?". Well, the server is running, and it can contact it. I

I poked around in xml-rpc, and received various errors: such as "can't find " (I just created a new file, hadn't saved. ropemacs doesn't have this problem), some generic error when trying to code assist on

import sys
sys.|

where | is caret, and even at one point an error saying the xml was malformed (I wish I saved that one, I haven't been able to get it again). I did remember to open a project.

I'm running Emacs HEAD, traad HEAD, rope HEAD. It is running traad, not traad3. All I did was (require 'traad) to setup. Am I doing something wrong? Is there something I can do to help diagnose this problem? I am getting pretty fed up with ropemacs.