miversen33 / netman.nvim Goto Github PK

View Code? Open in Web Editor NEW

325.0 6.0 1.0 1.12 MB

Neovim (Lua powered) Network Resource Manager

License: MIT License

Lua 99.92% Shell 0.08%

lua neovim netman nvim

netman.nvim's Introduction

TODO: Mike: Make this pretty 👍

netman.nvim's People

Contributors

Stargazers

Watchers

Forkers

temcom

netman.nvim's Issues

Create Unload Provider Function in Netman.api

In order to better facilitate developers life while developing plugins for Netman, we need to consider an "unload" and "reload" provider function inside Netman.api. The current workflow for testing providers is to load up neovim, require your provider, then restart neovim to import new changes. That is not ideal

Create function to interface with optional "set_metadata" function in api

It would probably be smart to allow file explorers to perform metadata updates (chmoding, chowing, etc), but only if the provider has implemented an optional set_metadata function.

Refactor netman.utils

It so desperately needs it and before we move forward on more documentation, we probably need to clean up this poor helper file

Notify doesn't appear to actually notify the user, except for notify.error

The title says it all. There are 5 different levels of notify, and only notify.error actually propagates up to :messages. Investigate why this is. A way to test this

:lua require('netman.utils').notify.info("Some cool log")

This will not display anything in your messages area. However

:lua require('netman.utils').notify.error("Some cool log")

This will display a more traditional vim error

SSH Provider Doesn't Prevent Overwritting "Newly Modified" Files

As we have removed the locking system (which was terse and tough to integrate with) with PR 35, we need to ensure that the ssh provider does not overwrite files that have been modified after we thought they have been modified (IE, a file that was modified by another process and not us). Consider adding this after issue 38 is complete

Create Metadata Gathering (Required) Function for Provider Integration

I know I said API was stable 🙃 but there is 1 more function that I believe needs to be required by providers.
After looking into how to "shim" file explorers, it looks like masking libuv is going to be the first approach. libuv has a variety of stat commands that can be used to gather various pieces of metadata on call and we currently dont have anything to support that. This was brought up in conversation with a friend but had not yet been implemented.

I will need to put another pair of functions in API that allows shims to fetch metadata from the API and the API to fetch metadata from the providers in order to properly mask libuv.

Note: Consider how to do this in such a way so as to account for weirdness with future metadata requests (EG, use the netman.options table for the Metadata)

Create Standard For Provider Configuration

As providers are created for netman, we need to have a standard way to feed the provider its configuration. This is critical to help ensure a cohesive development experience, as well as a simple API to develop against

Consider creating an interface for diffing files from various providers

When working with remote files, the ability to diff between any mixture of local/remote files would be incredibly helpful. Consider something like

Nmdiff provider1://some_user@some_host///some_file.txt provider2://some_user@some_other_host///some_file.txt

Where Nmdiff simply performs an async read of the 2 files and then feeds their contents to vims built in diff tool which can then be displayed in the buffer

Remote Absolute Path (with user) appears to be broken

When attempting to open a remote path via sftp (while specifying the user), the ssh provider appears to misread the path and thus wont open the file (it is unable to download the remote path). Log attached

absolute-path-broken

Consider optional API for shell integration

Considering that certain use cases of netman will consist of (effectively) operating on remote file systems, it may make sense to expose some interface that a provider can implement, and when implemented the provider will allow the API to open "terminals" in the remote system.

This would be especially useful in ssh and docker, though it may have uses elsewhere

Builtin Explorer for Netman creates a new buffer for each entry into a directory

Additionally, it looks like there is some shenangians happening with init functionality on how buffers are being created. Probably worth revisiting this as this is definitely a bug.
Note:
Steps to recreate this

Open neovim
:edit sftp://somehost/somedir/
Open another directory in the displayed explorer
Watch as new buffers open for each directory
Logs: buffermadness

Make logs stop being displayed wrapped in double quotes

A simpler request but still an annoying one. When you use require('netman.utils').log.$SOMELEVEL$("my message"), this will log out to the file (and notify window if you used notify instead of log) wrapped in double quotes.

For those with prying eyes, this is happening due to how we are merging the log info table (found here)

Some day I will get around to this (when the annoyance eclipses the amount of work I have to do on other issues).

Netman Options (`netman.options`) Needs documentation

Title is self explanatory. Netman Options are used everywhere and have effectively no documentation.
We should absolutely have a vim doc for this, and probably also worth throwing something in the wiki for it.

Perl LSP fails to intialize

When attempting to edit a remote perl file, perlpls fails to initialize on the file.
Netman log
lsp.log

It looks like the suggested change from this commit didn't quite work.

Error:

[ERROR][2022-03-28 07:57:04] .../vim/lsp/rpc.lua:420    "rpc"   "pls"   "stderr"    "Can't locate object method \"file\" via package \"URI::sftp\" at /usr/local/share/perl/5.30.0/PLS/Parser/Document.pm line 68, <DATA> line 1.\n"

Netman _might_ have a memory leak?

I have noticed that when Neovim is open for a long time (more than a few hours), my memory usage begins to creep up. I haven't recently made any changes to anything outside Netman so this might be caused by it. I am unsure how this could happen as netman doesn't save much in memory. But its best to verify that we are clearing all caches correctly and aren't accidentally leaving things in memory forever that don't need to be there.

The new cache module should be helping with this

Create function to print out debug information for issues

In order to gather accurate and useful information for issues, we need to have the following details provided

Core Version:
Remote Protocol:
Protocol Provider Version:
Protocol Version:
Log:

To better gather this information, Netman should have a globally exposed function that can be called to fetch this. Something like
:NmGetInfo (for example)

Create File Explorer Consumer Plugin System

Consider ways to handle registering and feeding "File Explorers" that already exist (such as nvim-tree, chadtree, telescope-file-browser, as well as an internally provided file browser).

This is a deliberately vague issue that is meant more as a placeholder for Milestone 1.1

Undoing on initial remote file open deletes contents of buffer

After a remote file is opened into buffer, executing an undo command will remove the contents of the buffer. If you then execute a "save" command, the buffers now empty contents will be written to the remote file effectively removing the contents of the remote file.

Consider this... Not great.

Consider removing lock system

As the idea is to push for development of plugins, having a half baked lock system in place that sits under vim's default lock system, and doesn't truly prevent editing of remote files via different uris/access paths, the locking system will likely end up being alot more work than it's worth, with us getting at best, little use of it, and at worst, it being an active hindrance to provider developers.

Consider removing it and allowing providers to handle file locking in their own way

Opening remote file under different user fails

Plugin Info:
Log: logs.txt

Issue:
When opening an existing "relative" file over ssh with a different user than the current neovim user, the SSH provider fails to pull the file properly.

Note: When opening an existing "absolute" file over ssh, the provider operates correctly.

Desired Behavior:
When opening a remote file, regardless of it being relative or absolute, it should be pulled down and opened (and pushed back up) correctly

Edit: Updated log

SSH Provider appears to fail on missing file pull

When trying to pull a file that doesn't exist, the ssh provider fails with the following error

Error detected while processing BufReadCmd Autocommands for "sftp://*":
"Fetching file: sftp://miversen-dev-desktop/non-existent-file.txt"
E5108: Error executing lua ...m/site/pack/packer/start/netman.nvim/lua/netman/init.lua:30: Vim(read):E484: Can't open file /home/miversen/.cache/nvim/netman/remote_files/lmovyrbxlrp
stack traceback:
        [C]: in function 'nvim_command'
        ...m/site/pack/packer/start/netman.nvim/lua/netman/init.lua:30: in function 'read'
        [string ":lua"]:1: in main chunk

Error Log

Figure out how to "override" `cwd`

Alot of file managers rely on cwd to figure out what to open. If we are not setting this, they will ignore the remote file/directory we are in. Consider how to implement this safely without breaking other plugins outside the Netman system

Create Metadata Flags that associate with functions in `libuv`

Once issue 38 is complete, we will want to get proper options added to netman.options.METADATA that reflect items that can be pulled from libuv.

Create Docker File Manager provider

Editing files in docker containers is not easy if the filesystem is not mounted. Files could be accessed "remotely" via docker file commands. Consider creating a docker URI for this

utils.notify logging out incorrect source information

When calling utils.notify the source file and line number are incorrect. This is due to us calling log from notify, which adds 1 call to the call stack and thus we are "off" by 1 in the stack search. Im not quite sure the best way to fix this just yet. Probably by migrating the logging section to an actual function vs an anon function and calling it from both log and notify to offset the varying stack depths

Create Actual Documentation

Target this for 1.1

We need 2 major things in order to move forward and attempt any support for 3rd party developers
Provider Creation Documentation
Vim Help Pages

Below is a breakdown of the required documentation for this issue to be resolved

Research the efficacy of having the docker provider utilize the docker socket instead of commands over shell out

Title

Right now, the docker provider talks to the docker binary to do stuff and consumes (and processes) the standard out to determine if the container is alive (in order to execute commands inside of it, browse its directory structure, etc). It might be worth seeing if we can instead just talk to the socket for more structured responses.

Or it might now be. Who knows.

Create Neovim Compat Layer

As Neovim evolves, we want to continue being able to support previous versions of Neovim while simultaneously supporting whatever the current version of Neovim is (including leveraging whatever new features are currently available).
To do this, we will want to establish a set number of versions we will support and a layer that we can provide to support compatibility between the versions.

Alternatively we only guarantee support for whatever the current point version is, though I don't much like this.

Have Netman (Utils/API) lock local files into a "session directory" for easy later cleanup

Currently netman.utils has an attribute that is exposed for developers called files_dir. This is simply located in ~/.cache/nvim/netman/remote_files/ but would be better served to have a directory within this structure for each running process. Something like
~/.cache/nvim/netman/remote_files/PID1 (for example).

This would make orphaned file clean up easy as we could just check to see if PID1 is still running on Netman.utils init and if its not, clean up the orphaned files. Right now we rely on the API being able to cleanly clean up after itself on buffer close which may result in inconsistent behavior (eg, if all of vim is shutdown with a kill -9 command, or the machine itself was shutdown, etc).

SSH Remote Directory Access Crashes

When attempting to read a remote directory, it looks like the SSH provider dies.
Logs Attached

SSH Keyphrase/password appears to be ignored during initial tunnel connection

Currently researching but wanted to get an issue opened for it for visibility.

It appears that there is an issue with connection to secured ssh connections (secured via keys that have a passphrase or password verification).

The issue is that we are able to capture the STDERR output indicating that we need to enter credentials, but it appears that the credentials never make it to the ssh connection (or they are ignored). Further investigation needs to be done here. It might be worth digging into the pty functionality of jobstart to see if we can't "mock" a pty for this

Consider Making Netman Logging an Asynchronous function

As more logs are added to netman, its definitely worth considering having Netman Logs be an asynchronous function so as to avoid blocking the main processing thread while logs are flushed to the file (or displayed to the user).

Remote files don't connect to the LSP

It appears that the LSP attempts to connect to the empty buffer before its attributes are set and there is no clean way to force connect the LSP after we have populated it with content and attributes.

Issue filed with Neovim here

The correct way to deal with this is to

Find the appropriate lspclient id(s) related to the buffer's filetype
Associate the buffer with each lsp client manually via

vim.lsp.buf_attach_client(buffer_id, client_id)

This would be done at the end of the read function in the init for netman

Netman a core and provider unit testing and profiling framework

See title.

As it is, netman does not have any sort of unit testing built into it currently and this leads to issues with regression bugs cropping up as development moves forward. Additionally, not being able to profile netman and establish pain points in file loading/management is continuing to make it more difficult to continue optimizing netman

Figure out how to call lua functions asynchronously

In order to better support the user experience, we have a handful of functions that should be run asychronously as they should be background tasks (write, delete, etc)

Remove netrw override

While this tool is meant to be a drop in replacement for netrw, it probably shouldn't disregard netrw altogether. As long as it can be verified that netman runs when its supposed to instead of netrw (for now, test against the ssh provider to ensure netman provides it and not netrw), we can safely remove the allow_netrw flag and allow them to exist together in peace and harmony.

Have core providers verify they _can_ be used

Currently the core providers (at this time ssh is the only one) simply assume the resources they need (ssh in this case) exist and are available within path. This is incorrect behavior and the provider should instead verify it can be used during its initialization. Likely means we need to change the init to be required and have it return a boolean for is_usable (as an example)

Using Netman with Missing Provider Dependencies causes Errors on Neovim Startup

When you open neovim while missing a dependency (for example, docker), Neovim explodes. This is due to the provider throwing a notify.error if it can't find docker in its path. I am unsure what the best path forward is for Netman on this (if we should silence all notification errors during a provider's init, or if I should push the issue to a provider to not cause firey explosions on failure during init). For Docker, we will fix the issue in the plugin itself, reducing notifications to logs and reducing errors to warnings.

For reference, code block in question

function M:init(config_options)
    local command = 'command -v docker'
    local command_options = {}
    command_options[command_flags.IGNORE_WHITESPACE_ERROR_LINES] = true
    command_options[command_flags.IGNORE_WHITESPACE_OUTPUT_LINES] = true
    command_options[command_flags.STDERR_JOIN] = ''
    command_options[command_flags.STDOUT_JOIN] = ''

    local command_output = shell(command, command_options)
    local docker_path, error = command_output.stdout, command_output.stderr
    if error ~= '' or docker_path == '' then
        notify.error("Unable to verify docker is available to run!") -- This is bad
        if error ~= '' then log.warn("Found error during check for docker: " .. error) end
        if docker_path == '' then log.warn("Docker was not found on path!") end
        return false
    end

    local docker_version_command = "docker -v"
    command_output = shell(docker_version_command, command_options)
    if command_output.stdout:match(invalid_permission_glob) then
        notify.error("It appears you do not have permission to interact with docker on this machine. Please view https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user for more details") -- This is also bad
        log.info("Received invalid docker permission error: " .. command_output.stdout)
        return false
    end
    if command_output.stderr ~= '' or command_output.stdout == '' then
        notify.error("Invalid docker version information found!") -- Really bad
        log.info("Received Docker Version Error: " .. command_output.stderr)
        return false
    end
    log.info("Docker path: '" .. docker_path .. "' -- Version Info: " .. command_output.stdout)
    return true
end

Create Rysnc Provider

Implement Chunk Reading

The libuv library supports reading chunks of a file (by specifying the start byte and the amount of bytes to read). This would be incredibly useful to duplicate for remote resource reading.

Consider that not all resources will support reading chunks of data (they may not be able to read only a section and simply discarding the excess is wasteful) for a variety of reasons. Consider this an optional (but recommended) feature for providers to implement in their read implementations.

That said, this will likely need to be a post 1.1 Milestone idea (getting File explorer integration setup is more important than previewing resources currently)

API Overhaul

As core approaches 1.0, there are a few last changes that should be made to the interface API in the Remote Tools Interface.

Of these changes, the following proposals should help streamline integration of 3rd party providers with netman, as well as allow a more open approach for edgecase style usage of the provider system.

The API requirements should be reduced to the following required variables/functions

write (replaces write_file)
read (replaces read_file and read_directory)
delete (replaces delete_file and delete_directory)
name (no change)
protocol_patterns (no change)
version (no change)
parse_uri (replaces get_details)
init_connection (new, but will also replace get_unique_name)

Adding optional functions that can be implemented, and will be called in the lifecycle of a remote buffer connection

init (change the signature to return a boolean for successful initialization if implemented)
close_connection (new, can be used to perform any sort of cache cleanup after a remote buffer is closed)

Justification

Renaming read_file/read_directory and reducing it to just read helps make the function more concise and eliminates the potential for developers to believe netman is purely file based. think outside the box for providers

Reduce end user confusion
Make reading clear and concise
Reduce complexity for integration

Renaming write_file and removing create_directory, reducing to just write helps to reduce confusion on what exactly should be called when you wish to make changes. There is no spec for create, as creating a new file is simply writing that new file to the filesystem

Reduce end user confusion
Make writing clear and concise
Reduce complexity for integration

Renaming delete_file/delete_directory falls into the same idea as read_file/directory in that it helps reduce confusion on which to call. Delete deletes. It doesn't care if its a file or directory. It just expects it to be gone

Reduce end user confusion
Make deletion clear and concise
Reduce complexity for integration

Renaming get_details to parse_uri is more concise on its functionality. Parse URI should get as much detail as possible from a single string input as well as any assumptions that the provider can make about the string without resolving the uri.

Reduce documentation confusion
Reduce complexity for integration

Adding init_connection as a required function will allow netman to prompt the provider for valid connection details, as well as give the provider an opportunity to resolve connection conflicts (required passwords for authentication, for example). Additionally, the object that is returned from this will now have a unique name associated with it, allowing this function to wrap the previous get_unique_name function that was used for file locking

Allow provider developer the ability to properly handle caching connections
Reduce complexity for integration

Consuming a returned boolean from a provider's init allows the provider a means of communication to inform netman it is not able to run currently. Netman will accept this and not use the listed provider for matching URIs

Allow provider developer the ability to "decline" being used (due to missing dependencies for example)

Allowing an optional close_connection function will allow the provider to better handle connection caching/management

Allow provider developer the ability to better manage caching connections.

These changes will be made and tested as the final API modification before 1.0 and thus will be considered stable.

Removal of Netrw disable prevents explore shim from functioning

The re-enabling of netrw has caused it to overtake the explorer shim when we are using it. This can be seen by setting the following lines at the top of init.lua

vim.g.loaded_netrwPlugin = 1
vim.g.loaded_netrw = 1

And then attempting to open a directory of SSH.
In doing so, the explorer shim properly invokes the Netman explorer buffer and all is well. However, re-removing them (as we did in the above commit) will cause Netrw to be invoked instead and the Netman explorer never triggers. Its worth investigating how other file explorers handle this (they may not).

Potential solutions include

Redisabling Netrw (not preferred)
Removing the Netrw autocommands for listed protocols that we utilize. They can be found here
API shenanigans based on protocols being registered from providers (:cry:)

To be clear, the main issue here is not at a provider level but instead at an explorer level as we are not able to prevent Netrw from launching when remote file systems are invoked

Create Socket/Channel communication layer for external services to utilize netman and providers

Considering certain explorers that currently exist (such as ChadTree), it is likely in our best interest to expose a socket/channel communication layer so these explorers that are written in other languages (and executed outside of neovim) are still able to communicate with netman to utilize the providers it providers for managing/manipulating remote resources.

Consider plugin autodetection/loading/hot reloading

As we move through provider development, the ability to hot reload changes to a plugin, or the ability to auto load new plugins into the session would be very helpful. Consider the work effort behind this.

Create Shims for "Popular" File Explorer Plugins

Once the shim manager is in place (See branch explorer), we will need to get some shims in place so that netman can self integrate with more popular explorers. Of these, the following shims likely need to be created

Remove useless "self" calls

API (and the providers associated) have alot of : access calls as opposed to . access calls. This is simply due to my early misunderstanding about the difference between the 2 and they should be removed as API does not pass back an instance of itself (it is a singleton pattern). This must be done before we complete work towards 1.1

Create "NotSupported" stubbing mechanic for unsupported (but required) functions

As providers expand in coverage, it is possible that certain required functions (EG: write, read, etc) are not actually supported by the provider.

An example, envision a tar.gz provider. This provider would be able to browse a gzipped tar, and read files in it, but it would not be able to write changes to it.

In the current setup, this is not something that is supported in a standard way. Standardize it.

How to fill out Issue

Please read Debugging before submitting an issue!
When filling out an issue, upload the output from the :Nmlogs command, run from within your current trouble session.

NOTE: When gathering your logs, please ensure you are running Netman in debug mode as described in Usage as that will provide the most helpful logs

Failure to provide the above listed required items may result in your issue being unceremoniously closed with a link to this issue as the reason.

Additionally, feel free to tag the issue with whatever labels you feel are most relevant to your issue.

Consider Provider Write Multiplexing

Research how difficult it is to implement and expose (safely) multiplex writing of buffers to providers.
An example

Nmwrite 1 sftp://myuser@myhost1/somefile.txt rsync://myuser@myhost2/somefile.txt docker://mycontainer///some_location/somefile.txt

Things to consider

How do we visually show the user which file/files your buffer is interfacing with
What do we do when/if there is a content mismatch between provider files?

I am not sold on how useful this would be but implementing it under the hood would not be difficult. The difficult lies on how to safely convey what is being done and how to allow users to control it.

Create Netman File Explorer UI

Netman currently does not have a UI in place for traversing directories, and instead relies purely on the user knowing the path to the file they wish to browse.