Patrick M. Walsh

Ever been curious about what, exactly, powers autocomplete? Me too!

“Any sufficiently advanced technology is indistinguishable from magic.” - Arthur C. Clarke

I’m fortunate to have learned how to develop software in the age of modern IDEs with support for just about any language I could possibly want to learn. I thought it was VSCode magic that I could just download a plugin and be off to the races with autocomplete, renaming, symbol-finding, etc. To a degree, it turns out, I was right - it was Microsoft that developed the Language Server Protocol to enable VSCode to be the “one true IDE” for any language under the sun. But the protocol isn’t just for VSCode, and I’ve used LSPs in a few different editors - it still feels like magic every time, though. I wanted to understand the magic, and maybe write a little bit of my own as a personal challenge.

Getting Started

I figured my first task is to receive a request - I had no idea how a language client (the IDE, in my case NeoVim) talks to a language server (the thing I’m writing), so it was time to do some research.

LSP is a protocol, which means it doesn’t care how a client and a server talk to each other - what’s important are the interfaces exposed by the client. According to the spec, there are a few options:

For my editor of choice, the default is stdio, so step one of this process is to try and receive input on stdio.

There’s a lot of ways to skin this cat, but I thought I would start with Go’s existing LSP implementation: gopls. (We’re doing this in Go, by the way) There’s a lot of complication in that repo to make things DRY, extensible, and performant, but the core of what I’m looking for can be found here:

...
ss = lsprpc.NewStreamServer(cache.New(nil), isDaemon, s.app.options)
...
stream := jsonrpc2.NewHeaderStream(fakenet.NewConn("stdio", os.Stdin, os.Stdout))
...
conn := jsonrpc2.NewConn(stream)
...
err := ss.ServeStream(ctx, conn)
...

OK! This looks server-y - but I have questions. Namely:

  1. What is lsprpc.NewStreamServer().ServeStream()?
  2. What is fakenet?
  3. What is in this jsonrpc2 package?

Let’s go through each of these to see how the professionals are doing it. This will go a little out of order so we start at the bottom and work our way up to ss.ServeStream(), which I think is what we’re after.

What is fakenet?

There’s just one file in the fakenet package: conn.go. It contains an implementation of net.Conn - something that I thought was a more concrete struct, but it looks like it’s actually an interface. This package will fulfill that interface for any io.ReadCloser and any io.WriteCloser pair that we pass into fakenet.NewConn() - essentially letting us use a bunch of net package tools on something that isn’t necessarily a network. Like stdin and stdout! We can see in the run() method that we do the following:

  1. Set up a bytearray buffer
  2. Set up an infinite loop
  3. Pipe the input channel to the buffer

Obviously there’s more, but right now we’re just trying to read stdin from the client.

What is jsonrpc2?

The spec says that LSP communication happens via json-rpc 2.0 protocol, and then specs out what that looks like. This package implements that protocol - for our purposes, we want to notice that the Read() method happens in two parts:

  1. Read the Content-Length header. Every JSON-RPC call begins with this header, which tells us the size of the body of the call
  2. Read the remainder of the JSON-RPC call. Reading the initial size of the body allows us to make a buffer of the appropriate size and run io.ReadFull() on a bufio.Reader, which is build from a net.Conn

We’ll also want to notice the Go() method - we’ll come back to that shortly.

What is lsprpc.NewStreamServer().ServeStream()?

Let’s take a look here! The most interesting lines to me are 98-108 - it looks like they:

  1. Call Go() on the connection - which, looking above, is a jsonrpc2.NewConn(). See, that didn’t take long - we’re already back. We’re only interested in the following parts of the Go() method: a. It calls Read() on the stream, which we looked at above b. After some error checking and protocol faffing about, we get around to calling a handler(), which is a callback passed into the Go() method
  2. Creates the handler() callback method to manage the result after we figure out what the call is.

Bringing it all together

OK! After that brief tour, let’s strip out the inessential parts so we can build something that will receive an LSP request. We need:

There are a couple of these that are easy - i.e., an infinite loop in Go is for {} - only 6 total characters!

The hard part is going to be managing the buffer. You’ll notice that the code above splits this into two categories:

  1. Read the Content-Length header
  2. Read the remainder of the body

We’ll dive into that in a moment. Before we do, though, a quick note: for this first part, instead of writing to Stdout, we’ll be writing to a temp file. We’re starting with the bare bones - maybe eventually we’ll get into parsing the JSON and responding to Stdout, but this baby step is a big enough bite to chew on for now.

Step one: arrange the editor accordingly

In NeoVim, it’s easy to connect to a custom server - simply add this to your configuration:

vim.lsp.start({
  name = 'test-server',
  cmd = {'go-server'},
  root_dir = vim.fs.dirname(vim.fs.find({'.git'}, { upward = true })[1])
})

Not to be all “RTFM”, but :help vim.lsp.start does a much better job than I will of describing what this method and each of its options does.

Step two: the most basic main function

OK, let’s start with nothing - fire up NeoVim with the above addition in your config, and you should get an error in :messages: Spawning language server with cmd: 'go-server' failed. The language server is either not installed, missing from PATH, or not executable. Perfect! Our editor just tried to run the command we provided in our vim.lsp.start() options, and couldn’t find it. Let’s give it something to run.

# Get your project set up
mkdir -p go-server && cd go-server && go mod init example/go-server && nvim ./main.go

We’ll start with the most basic main loop:

package main

import "os"

func main() {
  f, _ := os.OpenFile("/tmp/go-lsp.log", os.O_RDWR | os.O_CREATE | os.O_APPEND, 0666)
  f.WriteString("I opened this file\n")
}

Now that we have a file, let’s make sure that it’s accessible by NeoVim - which means it needs to be on our PATH. I happen to have ~/.local/bin/ as part of my PATH, so I’ll link it there. That way, we can keep doing builds and not need to worry about re-linking or re-adding our working folder.

go build -o go-server . && ln -s /path/to/this/go-server /Users/username/.local/bin/

Now what I’ve done is I can open a couple of terminal windows/panes/however you manage multiple terminals: run tail -f /tmp/go-lsp.log to watch the output there, and close and open vim in another.

You should see “I opened this file” pop up in your tail -f window! Don’t forget to periodically clear out this log - some of the LSP requests and responses are pretty large, and the file will grow over time, even in /tmp/.

OK! We’ve accomplished the following:

We’re all set to start putting together a basic main loop:

package main

import (
	"bufio"
	"errors"
	"fmt"
	"log"
	"os"
	"strconv"
	"strings"
)

func main() {
	logger, err := newLogger()
	if err != nil {
		log.Println("Error creating logger")
	}

	reader := bufio.NewReader(os.Stdin)
	for {
		n, err := readHeader(reader)
		if err != nil {
			logger.Println("ERROR:", err.Error())
		} else if n != 0 {
      logger.Println("Content length:", n)
    }
	}
}

func newLogger() (*log.Logger, error) {
	f, err := os.OpenFile("/tmp/go-lsp-log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		log.Println("Error opening log file")
		return nil, err
	}
	l := log.New(f, "go-lsp", log.LstdFlags)

	return l, nil
}

func readHeader(r *bufio.Reader) (int64, error) {
	line, err := r.ReadString('\n')
	if err != nil {
		return 0, errors.New("Error reading from bufio.Reader")
	}
	before, after, found := strings.Cut(line, ": ")
	if !found {
		return 0, errors.New("Unable to find colon delimiter in header - invalid header")
	}
	if strings.TrimSpace(before) != "Content-Length" {
		return 0, fmt.Errorf("Invalid header name. Expected 'Content-Length', got %s", before)
	}

	n, err := strconv.ParseInt(strings.TrimSpace(after), 10, 64)
	if err != nil {
		return 0, fmt.Errorf("Error parsing content length: %w", err)
	}

	return n, nil
}

Build, close NeoVim, and reopen - you should see something like this in your tail -f:

go-lsp2023/08/16 14:08:25 Content length: 3245
go-lsp2023/08/16 14:08:25 ERROR: Unable to find colon delimiter in header - invalid header
go-lsp2023/08/16 14:08:25 Content length: 0

Excellent! Get got our content length. We’ll take care of that error in a minute - basically we successfully get the content and then run our infinite loop again on the remainder of the request and we throw an error (as we should).

With our content length, we can make our buffer and fill it with the rest of the bufio.Reader contents. One note is that we need to discard two bytes of data before reading our Reader because there’s an additional \r\n separating the header and the contents. Another note is that I’ve seen this done with io.ReadAll - however, my attempts didn’t work since io.ReadAll will continue reading until EOF, which would be indicated here by a \n byte at the end. Neovim doesn’t provide this final byte, and I guess VSCode does, so for us the reader just waits for additional input, preventing us from moving along.

package main

import (
	"bufio"
	"errors"
	"fmt"
	"io"
	"log"
	"os"
	"strconv"
	"strings"
)

func main() {
	logger, err := newLogger()
	if err != nil {
		log.Println("Error creating logger")
	}

	reader := bufio.NewReader(os.Stdin)
	for {
		n, err := readHeader(reader)
		if err != nil {
			logger.Println("ERROR(readHeader):", err.Error())
		} else if n != 0 {
			logger.Println("In else")
			logger.Println("Content length:", n)
			contents := make([]byte, n)
			_, err := reader.Discard(2)
			if err != nil {
				logger.Println("ERROR(reader.Discard):", err.Error())
			} else {
				_, err := io.ReadFull(reader, contents)
				if err != nil {
					logger.Println("ERROR(io.ReadFull):", err.Error())
				} else {
					logger.Println(string(contents))
				}
			}
		}
	}
}

func newLogger() (*log.Logger, error) {
	f, err := os.OpenFile("/tmp/go-lsp-log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		log.Println("Error opening log file")
		return nil, err
	}
	l := log.New(f, "go-lsp", log.LstdFlags)

	return l, nil
}

func readHeader(r *bufio.Reader) (int64, error) {
	line, err := r.ReadString('\n')
	if err != nil {
		return 0, errors.New("Error reading from bufio.Reader")
	}
	before, after, found := strings.Cut(line, ": ")
	if !found {
		return 0, errors.New("Unable to find colon delimiter in header - invalid header")
	}
	if strings.TrimSpace(before) != "Content-Length" {
		return 0, fmt.Errorf("Invalid header name. Expected 'Content-Length', got %s", before)
	}

	n, err := strconv.ParseInt(strings.TrimSpace(after), 10, 64)
	if err != nil {
		return 0, fmt.Errorf("Error parsing content length: %w", err)
	}

	return n, nil
}

Give that a go - it might not be the best Go code, but it’s straightforward and easy to read, and I see a lot of JSON in my tail -f!

Wrapping up

I find this output pretty cool - a peek under the hood of what is now considered fundamental developer tooling. Of course, right now it’s not very inspectable. It’s a huge JSON object printed into a file. I could clean it up and maybe just write the JSON objects to the file so I would jq it to poke around some more - but a better course would be marshalling and unmarshalling these requests into objects that I can use.

In the next part, I’d like to dive more into this unmarshalling process, and maybe even respond to the client to make something happen.

Until then!

#Go #LSP