clojure-lsp Development#
Codebase architecture#
The codebase is dense but well structured, the diagram below will help understand the layers making easier to understand where you need to make your changes.
visualization above's code
---
config:
theme: dark
---
flowchart TB
subgraph Clients["Clients"]
VSCode["VS Code<br>(Calva)"]
Emacs["Emacs<br>(lsp-mode / eglot)"]
Vim["Vim/Neovim"]
IntelliJ["IntelliJ<br>(clojure-lsp-intellij)"]
REPL["Repl"]
OtherEditors["Other editors"]
end
subgraph subGraph1["Cli (cli/)"]
CLIMain["Process<br>(main.clj)"]
LSPServer["LSP Server<br>(server.clj)"]
end
subgraph subGraph2["Core Services"]
Handlers["Features handlers<br>(handlers.clj)"]
Startup["Startup & Init<br>(startup.clj)"]
Config["Configuration<br>(config.clj)"]
DB["Database<br>(db.clj)"]
end
subgraph subGraph3["Analysis Engine"]
KondoIntegration["clj-kondo Integration<br>(kondo.clj)"]
Parser["Parser<br>(parser.clj)"]
Queries["Analysis Queries<br>(queries.clj)"]
DepGraph["Dependency Graph<br>(dep_graph.clj)"]
end
subgraph subGraph4["Feature Implementations"]
Completion["Completion<br>(completion.clj)"]
Diagnostics["Diagnostics<br>(diagnostics.clj)"]
CodeActions["Code Actions<br>(code_actions.clj)"]
Rename["Rename<br>(rename.clj)"]
Hover["Hover<br>(hover.clj)"]
References["Find References"]
Format["Formatting<br>(format.clj)"]
Refactoring["Refactoring Features"]
end
subgraph subGraph5["Support Systems"]
FileManagement["File Management<br>(file_management.clj)"]
Classpath["Classpath Resolution<br>(classpath.clj)"]
SourcePaths["Source Paths<br>(source_paths.clj)"]
Logger["Logging<br>(logger.clj)"]
end
subgraph subGraph6["Core Library (lib/)"]
API["Public API<br>(api.clj)"]
InternalAPI["Internal API<br>(internal_api.clj)"]
subGraph2
subGraph3
subGraph4
subGraph5
end
subgraph subGraph7["External Tools"]
CljKondo["clj-kondo<br>(Static Analysis)"]
Cljfmt["cljfmt<br>(Code Formatting)"]
CljDepend["clj-depend<br>(Dependency Analysis)"]
ClojureDocs["ClojureDocs<br>(Documentation)"]
end
subgraph subGraph8["File System"]
ProjectFiles["Project Source Files<br>(.clj, .cljs, .cljc)"]
ConfigFiles["Config Files<br>(deps.edn, project.clj)"]
Dependencies["Dependencies<br>(JARs, Git repos)"]
CacheFiles["Cache Files<br>(.clj-kondo, .lsp)"]
end
VSCode --> CLIMain
Emacs --> CLIMain
Vim --> CLIMain
OtherEditors --> CLIMain
CLIMain --> API & LSPServer
LSPServer --> Handlers
REPL --> API
API --> InternalAPI
Handlers --> InternalAPI & Completion & Diagnostics & CodeActions & Rename & Hover & References & Format & Refactoring
InternalAPI --> Startup & Config & DB
Startup --> KondoIntegration & Classpath & SourcePaths
KondoIntegration --> CljKondo & Parser & CacheFiles
Format --> Cljfmt
DepGraph --> CljDepend
Hover --> ClojureDocs
Parser --> Queries
Queries --> DB
Completion --> Queries
Diagnostics --> Queries
CodeActions --> Queries
Rename --> Queries
References --> Queries
FileManagement --> ProjectFiles & ConfigFiles
Classpath --> Dependencies
CLIMain:::entryPoint
LSPServer:::entryPoint
Handlers:::coreService
Startup:::coreService
Config:::coreService
DB:::coreService
KondoIntegration:::coreService
Parser:::coreService
Completion:::feature
Diagnostics:::feature
CodeActions:::feature
Rename:::feature
Hover:::feature
References:::feature
Format:::feature
Refactoring:::feature
CljKondo:::external
Cljfmt:::external
CljDepend:::external
ClojureDocs:::external
ProjectFiles:::storage
ConfigFiles:::storage
Dependencies:::storage
CacheFiles:::storage
classDef entryPoint fill:#e1f5fe
classDef coreService fill:#f3e5f5
classDef feature fill:#e8f5e8
classDef external fill:#fff3e0
classDef storage fill:#fce4ec
Coding#
There are several ways of finding and fixing a bug or implementing a new feature:
- The Clojure Way
- Create a test for your bug/feature, then implement the code following the test (TDD).
- Build
clojure-lsp[.bat]usingbb debug-clieach time you have made changes, and test it manually in your client. This is the slowest option.
Whichever development path you choose: For final testing, it is good to rebuild the binary with bb debug-cli.
There are two custom LSP methods clojure/serverInfo/log and clojure/cursorInfo/log. They can assist in debugging.
The Clojure Way#
With a clojure-lsp + nREPL powered Clojure editor you can modify your editor session's clojure-lsp server using the Clojure REPL.
Here's demo video: https://www.youtube.com/watch?v=4UvT0yqBDw8
These are the steps:
bb debug-cli- to build aclojure-lsp[.bat]executable that includes cider-nrepl in the jar. This executable will be saved at the root of the project.- Configure your editor to use this
clojure-lsp[.bat]executable - Have your editor restart its clojure-lsp server
- Issue the clojure-lsp
serverInfocommand - Find the
portentry in the output - Connect your editors nREPL client to this port
- Hack away!
Seeing is believing. An easy way to convince yourself that you can actually change clojure-lsp mid-flight is to:
- Modify the
server-infofunction inlib/src/clojure_lsp/feature/development_info.clj- Say, you add a
:foo :barentry to the map returned
- Say, you add a
- Evaluate the new
server-infofunction definition - Issue the clojure-lsp
serverInfocommand - Find
:foo :barin the output
You have just modified the LSP server powering your editor while it was running! This is the Clojure way. No recompiling and restarting and reloading. That is some other, non-Clojure, way.
The details in how to perform these steps can vary a bit between the various Clojure editors/plugins.
Visual Studio Code with Calva#
-
This project comes with Calva configuration to use the
clojure-lsp[.bat]executable built in step 1 above. You can skip step 2, unless are running on MS-Widnows, in which case you should update the setting in.vscode/settings.jsonto add the.batextension, i.e."calva.clojureLspPath": "./clojure-lsp.bat". -
To restart the clojure-lsp server, use the VS Code command Developer: Reload Window
- The Hack away! step needs to start with you issuing the command Calva: Load Current File and Dependencies.
Emacs with CIDER#
- To configure Emacs to use the nREPL-enabled executable, run
(setq lsp-clojure-custom-server-command '("~/path/to/clojure-lsp/clojure-lsp")), adjusting the path as necessary. If you add this to your Emacs config, you can skip this step in the future. - To restart the clojure-lsp server, execute the Emacs command
lsp-workspace-restart. - To find the server info, execute
lsp-clojure-server-info. - To connect the nREPL client, run
cider-connect-clj, with "localhost" and the port.
If you re-connect regulary, you may want to add this Emacs shortcut:
(defun lsp-clojure-nrepl-connect ()
"Connect to the running nrepl debug server of clojure-lsp."
(interactive)
(let ((info (lsp-clojure-server-info-raw)))
(save-match-data
(when-let (port (and (string-match "\"port\":\\([0-9]+\\)" info)
(match-string 1 info)))
(cider-connect-clj `(:host "localhost"
:port ,port))))))
Vim with coc.nvim and Fireplace#
- Change
coc-settings.json(:CocConfig)clojure-lsp: {command: "~/path/to/clojure-lsp/clojure-lsp"}, adjusting the past as necessary. - To restart the clojure-lsp server use
:CocRestart - To find the server info,
:echo CocRequest('clojure-lsp', 'clojure/serverInfo/raw')['port'] - To find the server log,
:echo CocRequest('clojure-lsp', 'clojure/serverInfo/raw')['log-path'] - To connect the nREPL client, run
:Connect <port>
If you re-connect regulary, you may want to add something like this to your vimrc:
" Copies the log-path to your clipboard
nnoremap <silent> crsl :call setreg('*', CocRequest('clojure-lsp', 'clojure/serverInfo/raw')['log-path'])<CR>
" Connects to nrepl
nnoremap <silent> crsp :execute 'Connect' CocRequest('clojure-lsp', 'clojure/serverInfo/raw')['port']<CR>
Neovim with Conjure#
- Change the lsp config
cmdto "~/path/to/clojure-lsp/clojure-lsp", adjusting the path as necessary. - To restart the LSP:
- To stop the clojure-lsp server, use
:lua vim.lsp.stop_client(vim.lsp.get_active_clients()) - To start the clojure-lsp server, use
:edit - To find the server info or the log file, use
:lua clients = vim.lsp.get_active_clients() for k, client_data in ipairs(clients) do id = client_data.id end client = vim.lsp.get_client_by_id(id) result = client.request_sync("clojure/serverInfo/raw", {}, 5000, 15) print('port = ' .. result.result.port) print('log-path = ' .. result.result['log-path']) - To connect the nREPL client, run
:ConjureConnect <port>
Debugging & Profiling#
The nREPL includes tools for debugging and profiling clojure-lsp. See cli/dev/clojure_lsp/debug.clj.
If you're interested in using the profiling tools in that file, you'll need to be familiar with criterium and clj-async-profiler.
Note that the performance of clojure-lsp is highly dependent on the size of its db. If you load a repl with -A:build, you'll have access to the debugging tools, but the db will be nearly empty. Follow the [steps][#the-clojure-way] above to connect to an nREPL which has a populated db.
Automated Performance Testing#
clojure-lsp has unit tests to ensure that performance doesn't degrade with changes. They can be run stand alone using bb performance-test. Additionally, they are run by the continuous integration steps.
The implementation files for the performance tests are in the cli/integration-test/performance folder. These tests run various clojure-lsp tasks against the clojure-lsp codebase itself multiple times and report p50, p90, and p99 timings. The tests fail if these timings exceed their respective preset values.
Debugging with Flowstorm#
Flowstorm is a "tracing debugger". It records code flows and allows you to step through the flows later, after the code has run. To do this, it instruments your Clojure code while compiling them, adding its own (invisible) hooks.
The UI displays the code, stack, and local variables as you trace the code's flow. Click backwards to retrace the flow. You can also define a variable in the user space. There are many other features, see the Flowstorm page for more.
By creating a Flowstorm visualizer, you can directly see variables such as a rewrite-clj zloc without having to invoke z/string inside the REPL.
When building with bb debug-cli clojure-lsp is created using ahead-of-time compilation of class files. The resulting executable starts up quickly, but because the Clojure code has already been compiled Flowstorm can't instrument the Clojure at runtime. To get around this, you'll have to run the clojure-lsp code in the slower, but more flexible, way - from the command line.
To use Flowstorm with clojure-lsp, you need to 1. configure clojure-lsp according to The Clojure Way 2. create a new executable script containing the clojure-lsp command line; save it in the project as bin/clojure-lsp-dev 3. change your editor to use this new script
clojure-lsp configuration to use Flowstorm#
Let's follow the convention and name the script clojure-lsp-dev. The script will invoke clojure-lsp along with the options to include Flowstorm instrumenting and the Flowstorm GUI.
#!/usr/bin/env bash
clj -Sdeps '{:aliases
{:lsp
{:extra-paths ["cli/dev"
"lib/src"
"cli/src"]
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"
"-Dflowstorm.theme=dark"
"-Dclojure.storm.instrumentOnlyPrefixes=clojure-lsp"]
:classpath-overrides {org.clojure/clojure nil}
:extra-deps {com.github.flow-storm/clojure {:mvn/version "1.13.0-alpha5"}
com.github.flow-storm/flow-storm-dbg {:mvn/version "4.7.0"}}
:replace-deps
{clojure-lsp/clojure-lsp {:local/root "./cli"}
cider/cider-nrepl {:mvn/version "0.59.0"}}}}}' \
-M:lsp -m clojure-lsp.main "$@"
Explanation of the script
The script runs the clojure command line, telling `clj` that the deps.edn data will follow (`-Sdeps`) on the command line. Next there is a large quoted argument with the deps.edn data. Following that is a command telling `clj` to use the lsp alias (`-Mlsp`) we supplied with the deps.edn argument. And then the -m indicates the namespace for the main. The large deps.edn data argument is worth looking at chunk by chunk. {:extra-paths ["cli/dev"
"lib/src"
"cli/src"]
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"
"-Dflowstorm.theme=dark"
"-Dclojure.storm.instrumentOnlyPrefixes=clojure-lsp"]
:classpath-overrides {org.clojure/clojure nil}
:extra-deps {com.github.flow-storm/clojure {:mvn/version "1.13.0-alpha5"}
com.github.flow-storm/flow-storm-dbg {:mvn/version "4.7.0"}}
:replace-deps
{clojure-lsp/clojure-lsp {:local/root "./cli"}
cider/cider-nrepl {:mvn/version "0.59.0"}}}}}' \
Finally, change your editor to use the new executable. For Calva, you could change .vscode/settings.json to include the new script
{
"calva.clojureLspPath": "./bin/clojure-lsp-dev"
}
lsp-clojure-custom-server-command as described above.
Using Flowstorm with clojure-lsp#
- Launch Flowstorm's GUI by evaluating
:dbgin the REPL (see The Clojure Way for configuring the REPL) - Click the
Start/Stop recordingbutton in the upper left - In the editor, exercise part of clojure-lsp in your editor; for instance, extract a function
- Click the
Start/Stop recordingbutton again to pause recording - In the
Quick jumpwindow, type in the name of function you have targeted -extract-functionin this case. Choose clojure-lsp.refactor.transform/extract-function in the menu - Click through the function via the
Step over to next expression in the current framebutton
Flowstorm has several built-in ways to display variables (map, seq, indexed) but often it helps to display a variable in a custom way. For example, a zloc can be implemented as a vector, but it is usually easier to understand its meaning by seeing the string value.
A visualizer for rewrite-clj can make it easy to see the string values for zloc or node files values. See the Flowstorm visualizer documentation for detailed information. An example follows
(ns clojure-lsp.flowstorm-debug
(:require
[flow-storm.debugger.ui.data-windows.visualizers :as viz]
[flow-storm.runtime.values :as fs-values]
[rewrite-clj.zip :as z])
(:import
[javafx.scene.control Label]))
(comment
(fs-values/register-data-aspect-extractor
{:id :zloc-extractor
:pred (fn [val _] (try (some? (z/node val)) (catch Exception _ false)))
:extractor (fn [zloc _] {:rewrite-clj/zloc zloc})})
(viz/register-visualizer
{:id :zloc-viz
:pred (fn [val] (some? (:rewrite-clj/zloc val)))
:on-create (fn [val]
{:fx/node (Label. (z/string (:rewrite-clj/zloc val)))})})
(viz/add-default-visualizer
(fn [val] (try (some? (z/node (:val val))) (catch Exception _ false))) :zloc-viz)
:dbg
; rtc
)
cli/dev/clojure_lsp/flowstorm_debug.clj and evaluate the three methods. You can use :dbg to launch the UI. When the Data Window shows a zloc choose :zloc-viz in the dropdown and the display will show a string. (See the image above)
When you need to exit Flowstorm, simpily close it using the operating system's window manager.
Flowstorm's instrumentation does slow down clojure-lsp. If you are not using it, change your client from using clojure-lsp-dev back to clojure-lsp.
Testing#
Run bb tasks for a list of available dev tasks.
The codebase consists of the lib/ library and the cli/ program which depends on it.
bb test-lib,bb test-cli, andbb test(for both of the above).
cli/ produces a standalone .jar file that can be converted to a self contained clojure-lsp[.bat] script or a clojure-lsp[.exe] binary executable file.
bb prod-jar->cli\standalone jar,bb prod-cli-> executable script with embeddedcli/jar,bb debug-cli-> same asprod-clibut includes support forcider-nreplandclj-async-profile, andbb native-cli-> binary executable produced withgraalvm.
There are also cli/ integration tests. They require ./clojure[.bat|.exe] to have already been built with one of the above cli tasks (__NOTE:__always ensure that you rebuild the cli/ executable every time you make changes to lib/ or cli/ source code so tha the integration tests use an up to date executable).
bb integration-test
The same development version can be used to lint all of the source code.
bb lint
Writing tests#
A test should be able to run on all JDK versions in scope starting with 1.8 and across GNU/Linux, macos and MS-Windows operating systems.
The test author should be aware of the following important differences between *nix and windows:
- Line endings
- On *nix: the single Line Feed (LF) char, i.e.
\n. - On windows: the Carriage Return (CR) followed by the LF char (CRLF), i.e.
\r\n.
- On *nix: the single Line Feed (LF) char, i.e.
- Paths
- On *nix: Use
/as the path separator, absolute paths start with/. - On windows: use
\as the path separator, absolute paths start either with a drive letter followed by:\, i.e.[A-Za-Z]:\(e.g.c:\tempandD:\src) or with a double\\indicating a network path (e.g.\\computer39\temp).
- On *nix: Use
Below are a few hints to assist with writing test that work accross the different platforms.
- Line Endings
- When comparing strings, Use
clojure-lsp.test-helper.internal/string=with\nin your expected result.- e.g. use
(is (h/strings= "one\n" result)instead of(is (= "one\r\n" result))or(is (= "one\n" result)).
- e.g. use
- Use
h/str-includes?with\nin the string to search for instead ofclojure.string/includes?.- e.g.
(is (h/str-includes? (slurp "path") "something\n"))instead of(is (str/includes? (slurp "path") "something\n"))or(is (str/includes? (slurp "path") "something\r\n")).
- e.g.
- When comparing strings, Use
- Paths
- Always use
babashka.fs/canonicalizewhen converting a relative path to an absolute path. Avoiding using any of java File/Path absolute or canonical equivalent fns. This ensures that the drive letter on windows is always in capitals (e.g.D:\instead ofd:\). This is also the convention used throughout the codebase and it works as well with both existing and non-existing files. - Use
clojure-lsp.test-helper.internal/file-path,clojure-lsp.test-helper.internal/file->uriwith *nix paths. They are converted to the format expected by the OS.- e.g.
(load-code (h/file-path "/aaa.clj") "(ns aaa)")instead of(load-code "/aaa.clj" "(ns aaa)")or(load-code "c:\\aaa.clj" "(ns aaa)")
- e.g.
- Always use
Code Coverage with Clofidence#
It is useful to run the unit tests with a code coverage tool to verify that the unit tests cover the new and modified code reasonably.
Clofidence is a code coverage tool based on Flowstorm. It generates HTML output with the code color coded to indicate where coverage is good, missing, or low.
To use Clofidence with clojure-lsp, you must
1. modify deps.edn to include the Clofidence configuration
2. change scripts/make.clj so it uses the newly added Clofidence configuration
3. run the unit tests by the usual method, bb test
4. open the generated index.html (firefox lib/clofidence-output/index.html)
This is the page you'll see with index.html. It lists all namespaces in the project with a stacked bar chart showing the code coverage percentage.
When you click on a namespace, you'll see the details of the code coverage.
The following diff (against commit id 11f7ff7) changes the clojure-lsp configuration to support Clofidence. You may wish to update the flow-storm version.
diff --git a/lib/deps.edn b/lib/deps.edn
index 569ef77c..0e9e5096 100644
--- a/lib/deps.edn
+++ b/lib/deps.edn
@@ -34,6 +34,15 @@
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]
:extra-paths ["test"]
:main-opts ["-m" "kaocha.runner"]}
+ :clofidence {:classpath-overrides {org.clojure/clojure nil}
+ :extra-deps {com.github.flow-storm/clojure {:mvn/version "1.12.5"}
+ com.github.flow-storm/clofidence {:mvn/version "LATEST"}}
+ :exec-fn clofidence.main/run
+ :exec-args {:report-name "Hansel"
+ :test-fn kaocha.runner/-main*
+ :test-fn-args []}
+ :jvm-opts ["-Dclojure.storm.instrumentOnlyPrefixes=hansel"
+ "-Dclojure.storm.instrumentSkipRegex=.*test.*"]}
:cognitest {:extra-paths ["test"]
:extra-deps {clojure-lsp/test-helper {:local/root "../test-helper"}
io.github.cognitect-labs/test-runner
diff --git a/scripts/make.clj b/scripts/make.clj
index fd803036..0bf74d45 100644
--- a/scripts/make.clj
+++ b/scripts/make.clj
@@ -56,7 +56,7 @@
(defn ^:private unit-test [dir]
(println :running-unit-tests... dir)
- (clj! dir ["-M:test"])
+ (clj! dir ["-X:test:clofidence"])
(println))
(defn ^:private mv-here [file]
To remove the coverage output from Clofidence, remove the lib/clofidence-output folder (rm -r lib/clofidence-output).
To remove the Clofidence configuration after applying the above patch, simply revert the scripts/make.clj and lib/deps.edn files.