- Python 72.5%
- Svelte 12.7%
- Swift 6.4%
- Rust 3.7%
- TypeScript 3.6%
- Other 1.1%
f338f05ddcnetworking: membership hears which nodes report and which the cluster lists; a node counts once listed Two more inputs from exo, through NetworkingHandle.heard_from / listed (the same channel as forget: a call never waits and never takes the lock): - heard_from(node): exo on the node reported. This arms the wedge check (Suspect(Wedged) after WEDGE_TIMEOUT with the transport up), which nothing fed before. - listed(node): the cluster's state lists the node (it reported, or a snapshot names it). One this node has never seen is Suspect(Disconnected) from now, so it is Gone after the grace unless its transport shows it: a node that went away before this one started (a new coordinator can inherit such nodes) is removed like any other, instead of staying listed for good. A peer now counts in the quorum read once it is Alive and listed (reachable counts the same peers). A node that quits within seconds of starting, before its first report, is never in the cluster's state, so nothing removes it; it used to count until FORGET_AFTER, so a pair whose other node restarted quickly a few times read Minority for 5 min. Once Gone, a peer the cluster never listed leaves the view (there is nothing to remove). Tests: the listed input (Suspect then Gone for a node never seen; counts once its transport shows it; idempotent; timed from the end of Resuming); a node that leaves before it is listed never counts and leaves the view; Alive and listed in either order; a pair over three quick restarts stays a majority; a node waking from a 2-minute sleep, its reports armed and its peers' connections closing on waking, has no peer Gone or wedged within a grace of the end of Resuming. membership: tell the networking layer which nodes report and which the cluster lists Each node's Membership already passes NodeTimedOut on to the networking layer as it applies the cluster's events. It now also passes a node's report (NodeGatheredInfo) as heard_from (exo on it runs) and listed (the cluster lists it), and every node a state snapshot names as listed. Reports reach the coordinator from each node, and the other nodes through the coordinator. MembershipReader (Reachability plus the view with the quorum read) is what the master and the API will read; FakeMembership implements it for their tests and, like the real one, reaches only the nodes it is told of. Elections are unchanged: a node becoming wedged, or reporting again after it, starts no round (elections never saw a wedged node, and keep their own silence timeout for a wedged master), so the mapping skips it, as it skips Suspect becoming Gone. A change now carries the reason the node wasn't Alive before, for that. master: remove a node only when its membership says it is Gone and the quorum read allows it The master removed a node (NodeTimedOut, and InstanceDeleted for its instances) when the node's last report (last_seen) was 15 s old (NODE_SILENCE_TIMEOUT), with its own allowance for its own stalls (MASTER_STALL, then RECONNECT_ALLOWANCE). That clock-and-allowance pair is gone. It now removes the nodes its state lists that its own membership view has as Gone (Suspect for the 30 s grace period, or listed and never seen for as long), as far as its quorum read allows: - Majority: it removes them, and deletes their instances, as before. - Half (two nodes with one away, or an even split): their instances can't serve, so they are deleted, but the nodes stay listed as Gone. Once they have been Gone for FORGET_AFTER (the split hold, 5 min) they stop counting, the read is Majority, and they are removed. Removing them at once would make this side a majority of one before the split hold. - Minority: nothing, and nothing is sent: no NodeTimedOut, no InstanceDeleted. A Mac cut off from most of its cluster is most likely the one cut off; the others still run what it can't see. Once they have been Gone for FORGET_AFTER, the membership stops counting them (they stay in its view, as Gone), its read is Majority, and they are removed. A new model isn't placed on a node the master can't reach now (Suspect, or Gone and kept listed in an even split). Suspect nodes are never removed, Wedged ones included (until they are Gone), and the master never removes itself. The master's own stalls are the membership's too: it reads a clock that counts sleep, so a master whose Mac slept is Resuming and doesn't time the peers it lost toward Gone. MASTER_STALL read only the monotonic clock, which macOS stops during sleep, so a master waking from a real sleep removed every follower at once, and on real Macs, after a 50-120 s sleep, itself and its own instance too. last_seen stays in the state as a display field; nothing removes on it. _remove_silent_nodes_and_broken_instances keeps its shape and the synchronous indexing of #2584: each removal is in the state as soon as it is made, so a node or an instance is removed once. api: end a request at once when a device serving it can't be reached Every API now checks, on its own node's membership view, that it reaches the devices a request needs: - Before sending: a request whose model's instances all have a device this node can't reach (Suspect or Gone) ends at once with a 503, "A device serving this model can't be reached right now." (code device_unreachable). Readiness is judged on the instances it reaches, so a ready instance it can't reach doesn't make the model ready. - When the master places a request on an instance with such a device (it chose on its own view), the request ends at once. - While it runs: when a device leaves Alive (Membership.watch), the requests on its instances end at once with the same message, keeping what they got. The master keeps the same check as a backstop, on its own view: it doesn't send a request to an instance with a device it can't reach, and a request whose model has only such instances ends at once. Until now such requests waited: for output that couldn't arrive, until the device was removed (15 s of silence), and on a master cut off from most of the cluster, for as long as the cut lasted. master: an even split keeps a gone node's instances until the split hold, too On real Macs, a 2-node cluster lost the model of a follower asleep for 35 s: it was Gone at Suspect + 30 s, and its instance was ended then, but it was back about 5 s later (discovery holds off for 12 s after a stall, then it reconnects). With these constants a sleep is ridden out only up to about 27 s from the freeze. So with exactly half of the cluster reachable (two nodes with one away, or any even split) the master now keeps a Gone node's instances as well as the node, until the node stops counting after FORGET_AFTER (the split hold, SPLIT_HOLD, 5 min) and the read is Majority; then it removes the node and ends its instances, as a majority does at Gone. A lid closed for a minute or two then comes back to its models, as long as their ranks' connections held (if the ring broke, the runner fails and restarts, as before). Requests to the node fail fast meanwhile: every API checks reachability itself. A majority (three nodes or more) still removes a Gone node and ends its instances at Gone. No new timer: SPLIT_HOLD from the membership's table. membership: a removal still waiting is called off when the cluster lists the node again A removal of a node still Alive here now waits until the node leaves Alive (an announced quit). If the node reports after its removal, the cluster's state lists it again, but the waiting removal still took it out of the view once it was lost. The state then listed a node that this node no longer knew, so it never showed as Gone, and the master, which removes only what its view shows Gone, never removed it or ended its instances. listed() now calls off a waiting removal: the cluster lists the node again, so the removal no longer holds. Lost later, the node is Suspect and then Gone like any other listed node, and is removed. A quitting node stops reporting before it announces its departure, so an announced quit isn't affected. Test: a removal of a node still Alive here, followed by a report, leaves it in the view, counted, and Gone after the grace (it left the view at once). membership: a node the cluster's state lists counts in the quorum read, seen here or not A node that joins one side of a split adopts a state that lists the other side's nodes. It had never seen them, so they didn't count: on a 2-of-5 side it read Majority 2/2. As that side's master it would then remove them once they were Gone, 30 s later, instead of after the split hold. A listed node now counts from when it is listed (until it leaves, or has been Gone for FORGET_AFTER), whether this node has seen it or not. The joiner reads Minority 2/5, as the other node on its side does, and the other side's nodes stop counting, and are removed, only once they have been Gone for FORGET_AFTER. A node that is never listed (one that quits before its first report) still never counts. Tests: a node that joins the smaller side reads Minority 2/5 until the other side's nodes have been Gone for FORGET_AFTER (it read Majority 2/2); a listed node counts at once (it counted once Alive). membership: ask a node that reports, or is listed, while it is lost here A wedged node (its transport up, no reports) goes Gone, and a majority removes it: it leaves the view. When exo on it recovers and reports again, its token and connection never changed, so nothing showed it: listed() put it back as a node never seen, Suspect, Gone 30 s later, and it was removed again, every 30-35 s, until it restarted. Likewise a node Gone(Disconnected) whose reports keep arriving (through the coordinator) was never asked, and only a new token or connection could bring it back. Now, when exo lists a node, or hears from it, while its transport is lost here (Suspect or Gone, Disconnected), or when exo lists a node this one doesn't have, the node is asked whether it is still there, and an answer makes it Alive, from Gone too (an answer already undid Suspect). A node that doesn't answer is unchanged. At most one probe per node is in flight. A node listed but not in the view also has its wedge timeout armed: one that answers but never reports is Wedged after WEDGE_TIMEOUT rather than Alive for good. Tests: a wedged node removed while its transport stayed up is Alive once it reports and answers, and stays Alive (it was Gone and removed again every grace); a Gone node that keeps reporting is asked and back once it answers (it stayed Gone); a listed node that answers but never reports is wedged; an answer brings back a Gone node (it didn't). master: a launch onto a node it can't reach is refused, like a placement The master places a model (PlaceInstance) only on the nodes it can reach now: one Suspect, or Gone and kept listed in an even split, is left out. A launch with its nodes given (CreateInstance) wasn't checked, and that is what the API sends, for /place_instance too since #2581: onto such a node its model couldn't load, and the instance would be ended once the node was removed. It is now refused, with nothing sent, as a launch that doesn't fit is (#2584). The API sends a launch again until it shows (#2581), by when the node may be back, and calls it off after CLUSTER_COMMAND_TIMEOUT. The next change has the API place and launch only on the nodes its own node reaches, so this is the backstop for views that differ. Test: of two launches, one onto a node the master can't reach and one onto a node it reaches, only the second is launched (both were). The test of a launch onto a node the master dropped now also has the node reachable again when it is back. api: place, preview and launch only on nodes this node can reach /place_instance makes its placement in the API (#2581) and sends it as a CreateInstance, and /instance/previews places on the same topology. Neither left out the nodes this node can't reach now (Suspect, or Gone and kept listed in an even split), which stay listed for about 40 s, and for 5 minutes in a 2-node split or on a minority side. A launch could then be placed on a sleeping or cut-off node, whenever that node had more memory or the model's files, and the master, which doesn't launch onto a node it can't reach, left the user waiting 46 s for a 503 that blamed memory or a lost coordinator. The API now places and previews on the topology without the nodes its own membership can't reach, with the helper the master uses (reachable_topology, moved to exo.routing.membership). A launch naming such a node (POST /instance) is answered at once: a 503, "A device this launch needs can't be reached right now.", code device_unreachable, and nothing is sent. The master's refusal stays, as the backstop for views that differ. Tests: with the larger node unreachable, /place_instance launches on the other (it chose the unreachable one), and previews never offer it (they did); a launch naming it gets the 503 at once and sends nothing (it waited 46 s for the 503 about memory). api: tell the user to try again when a device can't be reached A device is unreachable from when it stops answering (Suspect), well before it is known to be gone, and it is often back a moment later: a brief network drop, or a Mac waking. The answers to a request and to a launch that need such a device now say so: - "A device serving this model can't be reached right now. Try again in a moment." - "A device this launch needs can't be reached right now. Try again in a moment." api: a model that would fit only with a device it can't reach is told so /place_instance places a model only on the nodes this node can reach now. When it fit only with one it can't reach (the larger Mac of a pair asleep, say), the answer was why it doesn't fit, a 400 about memory, which sent the user looking for memory to free. When placement fails on the reachable nodes but would succeed with all of them, the answer is now a 503, "A device that could hold this model can't be reached right now. Try again in a moment.", code device_unreachable. A model that doesn't fit even with every node is still told why, as before. Tests: the smaller Mac can't hold the model on its own and the larger one is unreachable: the 503 and its message, nothing sent (it was the 400 about memory); a model too big for both is still the 400. testkit: a short freeze or cut no longer costs a model A node is now removed only once its membership says it is Gone (Suspect for the 30 s grace) and the quorum read allows it, and a request that needs a device this node can't reach ends at once. The harness's expectations follow: - test_a_12s_pause_keeps_the_model and test_a_20s_cut_keeps_the_model pass: their strict expected-failure markers go. - test_a_cut_node_crowns_itself_and_rejoins_after_the_heal expects the model kept, and the chat through the master to end saying a device can't be reached. - A killed follower, and the old id of a restarted one, are removed at Gone, about 32 s after the kill: the bound for the killed follower is 45 s, as it is for the restarted one. The 40 s pause's description says when the node is removed now. New scenarios (test_membership_scenarios.py), each with bounds from the membership's timers: - a follower holding half a model cut off for 40 s keeps its view, requests through either side fail at once, and the master removes it no sooner than the lease plus the grace; - 2 nodes cut for 20 s: both fail fast, neither removes the other or the shared model, and it serves after the heal; - a crashed node the master reaches only through a third is removed within the re-probe interval, the probe and the grace; - a pair keeps its shared model through a 40 s SIGSTOP of the follower; - a chat streamed through a node that is then cut off ends when the far side is Suspect; - a pair whose follower restarts quickly three times, quitting before it reports, never reads a minority; - a master cut off from its followers is replaced, and a launch sent through a follower at the cut is answered 200. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .githooks | ||
| .github | ||
| .idea | ||
| .typings | ||
| .vscode | ||
| .zed | ||
| app/EXO | ||
| bench | ||
| dashboard | ||
| docs | ||
| nix | ||
| packaging | ||
| python | ||
| resources | ||
| rust | ||
| scripts | ||
| src/exo | ||
| tests | ||
| tmp | ||
| tools | ||
| .clauderules | ||
| .cursorrules | ||
| .envrc | ||
| .gitignore | ||
| .python-version | ||
| .swift-format | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| flake.lock | ||
| flake.nix | ||
| justfile | ||
| LICENSE | ||
| package-lock.json | ||
| PLATFORMS.md | ||
| pyproject.toml | ||
| README.md | ||
| RULES.md | ||
| TODO.md | ||
| uv.lock | ||
exo connects all your devices into an AI cluster. Not only does exo enable running models larger than would fit on a single device, but with day-0 support for RDMA over Thunderbolt, makes models run faster as you add more devices.
Features
- Automatic Device Discovery: Devices running exo automatically discover each other - no manual configuration.
- RDMA over Thunderbolt: exo ships with day-0 support for RDMA over Thunderbolt 5, enabling 99% reduction in latency between devices.
- Topology-Aware Auto Parallel: exo figures out the best way to split your model across all available devices based on a realtime view of your device topology. It takes into account device resources and network latency/bandwidth between each link.
- Tensor Parallelism: exo supports sharding models, for up to 1.8x speedup on 2 devices and 3.2x speedup on 4 devices.
- MLX Support: exo uses MLX as an inference backend and MLX distributed for distributed communication.
- Multiple API Compatibility: Compatible with OpenAI Chat Completions API, Claude Messages API, OpenAI Responses API, and Ollama API - use your existing tools and clients.
- Custom Model Support: Load custom models from HuggingFace hub to expand the range of available models.
Dashboard
exo includes a built-in dashboard for managing your cluster and chatting with models.
4 × 512GB M3 Ultra Mac Studio running DeepSeek v3.1 (8-bit) and Kimi-K2-Thinking (4-bit)
Benchmarks
Qwen3-235B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA
Source: Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5
DeepSeek v3.1 671B (8-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA
Source: Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5
Kimi K2 Thinking (native 4-bit) on 4 × M3 Ultra Mac Studio with Tensor Parallel RDMA
Source: Jeff Geerling: 15 TB VRAM on Mac Studio – RDMA over Thunderbolt 5
Quick Start
Devices running exo find each other automatically, without any manual configuration, when they are on the same network and run the same version of exo (see Adding another Mac). Each device provides an API and a dashboard for interacting with your cluster (runs at http://localhost:52415).
You can run exo with the macOS app, the easiest way on a Mac, or from source on macOS or Linux.
macOS App
exo ships a macOS app that runs in the background, in your Mac's menu bar.
The app needs a Mac with Apple silicon and macOS Sequoia 15.1 or later. RDMA over Thunderbolt needs macOS Tahoe 26.2 or later.
Download the latest version: EXO-latest.dmg. Open it, drag EXO to your Applications folder, and open EXO from there. The first time, macOS asks whether you want to open an app downloaded from the internet: click Open.
You can also install it with Homebrew:
brew install --cask exo
What EXO asks for, and why
-
Local Network access. macOS asks whether EXO may find devices on your local network. EXO needs this to find your other devices running exo and share the work with them. If you said no, turn EXO on in System Settings → Privacy & Security → Local Network, then quit and reopen EXO.
-
Thunderbolt network setup (optional, needs your administrator password). EXO offers to set up networking for Thunderbolt in its menu when Macs are connected with Thunderbolt cables; you can also do it in Settings › Cluster. Macs that share a Wi-Fi or Ethernet network don't need it. If you agree, EXO installs a small system service (
io.exo.networksetup) that:- creates a network location called
exoand switches this Mac to it; - gives each Thunderbolt port its own network service, which gets an address automatically;
- turns off Thunderbolt Bridge, which can flood the network with packets when three or more Macs are cabled in a loop.
The service runs when your Mac starts and about every 30 minutes, to keep these settings in place. The
exolocation starts with default settings, so anything you changed in your usual location (such as a fixed IP address, DNS servers or proxies) doesn't apply while it's in use. To undo the setup, uninstall EXO. To undo it but keep EXO, run the uninstall script described there: it restores your network settings and keeps your downloaded models, and EXO asks again before it sets up the network. - creates a network location called
-
Opening at login. EXO's welcome window has Open EXO at login ticked, so your Mac is ready to work with the others after a restart. Untick it there, or change it later in EXO's Settings › General or in System Settings → General → Login Items & Extensions.
-
Notifications. EXO doesn't ask at launch. When an update is ready to install, it leaves a quiet notification in Notification Center; you can turn these off in System Settings → Notifications. You can always check for updates in EXO's menu or Settings.
Adding another Mac
- Install EXO on the other Mac and open it.
- Put both Macs on the same local network (Wi-Fi or Ethernet), or connect them with a Thunderbolt cable.
- Make sure both Macs run the same version of exo. Devices running different versions don't join each other.
The new Mac shows up in the dashboard within a few seconds.
To see which version a Mac runs:
- in the app: choose About EXO from the menu bar icon, or open Settings › General;
- from source: run
uv run exo --version.
The app and a source checkout number their versions differently. The app uses its release number (such as 1.0.71), while a source checkout uses the version in pyproject.toml (such as 0.3.70). Because the default cluster name is the version, an app and a source checkout don't join each other by default. To mix them, give every device the same cluster name.
If a Mac doesn't show up:
- check that both Macs run the same version of exo;
- check that EXO is turned on in System Settings → Privacy & Security → Local Network on both Macs;
- guest Wi-Fi and some office networks stop devices from seeing each other: try Ethernet or a Thunderbolt cable instead;
- when exo sees another device that can't join (a different version or cluster name), it notes this in its log,
~/.exo/exo_log/exo.log.
Updating
EXO checks for updates by itself and tells you when one is ready. You can also use Check for Updates in EXO's menu or Settings. A Homebrew installation updates itself the same way.
Update every Mac in your cluster. Devices running different versions of exo don't join each other, so a Mac you haven't updated yet drops out of the cluster until you update it too; the dashboard then shows "DEVICE ON ANOTHER EXO VERSION". Mixing versions in one cluster isn't supported.
From source, pull the latest code, rebuild the dashboard and run uv sync again, on every device.
Cluster name
Devices join each other only when they use the same cluster name (in the CLI, the "namespace") and run the same build of exo (see below). By default the cluster name is exo, in the app and from source, the same for every version of exo, so a device can say that another one runs a different version instead of not seeing it.
To run separate clusters on one network, for example a test cluster next to your main one, give each cluster its own name:
- in the app: open Settings › Cluster from the menu bar icon and fill in Cluster name (leave it on Automatic for the default);
- from source: run
uv run exo --namespace my-cluster, or setEXO_ZENOH_NAMESPACE=my-cluster.
Devices also need the same wire version, a number exo bumps whenever what devices send each other changes, and the same build: the same app release, or for source builds the same exo package version and the same MLX and mlx-lm versions. Source builds of one package version are assumed to run the same exo code, so developers can run one checkout on several machines. A custom name doesn't override either, so devices with the same custom name but different versions of exo don't connect; before, they connected and then could stop making progress. A device that sees one under its cluster name logs "a device at … runs another version of exo" (or "another build") and lists it at GET /v1/devices-on-other-versions. The cluster name, wire version and build in use are logged at startup ("Discovery namespace: …"). A Mac and a Linux machine don't join each other, even on the same commit, because they run different builds of MLX. A Mac running exo from source and the Mac app at the same release don't join either, because the app names its own build.
Uninstalling the macOS app
The easiest way is in the app: click the EXO icon in the menu bar, open Settings › General, and click Uninstall EXO…. Enter your administrator password when asked. EXO then:
- removes its network setup service and the
exonetwork location, switches your Mac back to the Automatic location and turns Thunderbolt Bridge back on; - removes its login item;
- deletes its data folder,
~/.exo, but keeps downloaded models in~/.exo/modelsunless you untick Keep downloaded models; - moves EXO.app to the Trash.
If you've already deleted EXO.app, or removed it with Homebrew, run the uninstall script. You don't need a copy of this repository:
curl -fsSLO https://raw.githubusercontent.com/exo-explore/exo/main/app/EXO/uninstall-exo.sh
sudo bash uninstall-exo.sh # keeps downloaded models
sudo bash uninstall-exo.sh --delete-models # deletes them too
The script removes the network setup service, switches your Mac back to the Automatic location, turns Thunderbolt Bridge back on (recreating it if the network setup removed it), and deletes ~/.exo except downloaded models. It can't remove the login item: if EXO is still listed in System Settings → General → Login Items & Extensions, remove it there.
What each way removes:
| Way | Network settings (if you agreed to the Thunderbolt setup) | ~/.exo |
Downloaded models (~/.exo/models) |
|---|---|---|---|
| Uninstall EXO in Settings | Service removed, back to the Automatic location, Thunderbolt Bridge back on | Deleted | Kept, unless you untick Keep downloaded models |
uninstall-exo.sh |
Service removed, back to the Automatic location, Thunderbolt Bridge back on | Deleted | Kept, unless you pass --delete-models |
brew uninstall --cask exo |
Service removed, but your Mac stays on the exo location with Thunderbolt Bridge off |
Kept | Kept |
brew uninstall --zap --cask exo |
Same as brew uninstall. Also deletes the app's preferences and caches in ~/Library |
Kept (deleted only if empty) | Kept |
| Moving EXO.app to the Trash | The service removes itself within about 30 minutes, but your Mac stays on the exo location with Thunderbolt Bridge off |
Kept | Kept |
After brew uninstall or moving EXO.app to the Trash, run the uninstall script to restore your network settings and remove ~/.exo.
Run from Source (macOS)
If you have Nix installed, you can skip most of the steps below and run exo directly:
nix run .#exo
Note: To accept the Cachix binary cache (and avoid the Xcode Metal ToolChain), add these lines to /etc/nix/nix.conf, replacing your-username with your macOS username (whoami prints it):
trusted-users = root your-username
experimental-features = nix-command flakes
Then restart the Nix daemon: sudo launchctl kickstart -k system/org.nixos.nix-daemon
Prerequisites:
-
Xcode (provides the Metal ToolChain required for MLX compilation)
-
brew (for simple package management on macOS)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" -
uv (for Python dependency management)
-
node (for building the dashboard)
brew install uv node -
rust (stable, to build the Rust bindings)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -
macmon (optional, for GPU usage, power and temperature monitoring on Apple Silicon; without it exo falls back to psutil and monitors memory only)
Install the pinned fork revision used by this repo instead of Homebrew
macmon. Homebrewmacmon 0.6.1still crashes on Apple M5.cargo install --git https://github.com/vladkens/macmon \ --rev a1cd06b6cc0d5e61db24fd8832e74cd992097a7d \ macmon \ --force
Clone the repo, build the dashboard, install the dependencies, and run exo:
# Clone exo
git clone https://github.com/exo-explore/exo
# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..
# Install Python dependencies, including the MLX backend
uv sync --extra mlx
# Run exo
uv run exo
This starts the exo dashboard and API at http://localhost:52415/
Please view the section on RDMA to enable this feature on MacOS >=26.2!
Run from Source (Linux)
Prerequisites:
- uv (for Python dependency management)
- node (for building the dashboard) - version 18 or higher
- rust (stable, to build the Rust bindings)
Installation methods:
Option 1: Using system package manager (Ubuntu/Debian example):
# Install Node.js and npm
sudo apt update
sudo apt install nodejs npm
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Rust (using rustup)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Option 2: Using Homebrew on Linux (if preferred):
# Install Homebrew on Linux
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install dependencies
brew install uv node
# Install Rust (using rustup)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Note: The macmon package is macOS-only and not required for Linux.
Clone the repo, build the dashboard, install the dependencies, and run exo:
# Clone exo
git clone https://github.com/exo-explore/exo
# Build dashboard
cd exo/dashboard && npm install && npm run build && cd ..
# Install Python dependencies with the MLX backend for your hardware
# (NVIDIA: --extra mlx-cuda13 or --extra mlx-cuda12)
uv sync --extra mlx-cpu
# Run exo
uv run exo
This starts the exo dashboard and API at http://localhost:52415/
Important note for Linux users: Currently, exo runs on CPU on Linux. GPU support for Linux platforms is under development. If you'd like to see support for your specific Linux hardware, please search for existing feature requests or create a new one.
Configuration Options:
-
--no-worker: Run exo without the worker component. Useful for coordinator-only nodes that handle networking and orchestration but don't execute inference tasks. This is helpful for machines without sufficient GPU resources but with good network connectivity.uv run exo --no-worker -
--legacy-daemon: Run exo as a legacy SysV-style background daemon using double-fork daemonization. This is intended for legacy init scripts; systemd and launchd should run exo in the foreground without this flag.uv run exo --legacy-daemon
File Locations (Linux):
exo follows the XDG Base Directory Specification on Linux:
- Configuration files:
~/.config/exo/(or$XDG_CONFIG_HOME/exo/) - Data files:
~/.local/share/exo/(or$XDG_DATA_HOME/exo/) - Cache files:
~/.cache/exo/(or$XDG_CACHE_HOME/exo/) - Log files:
~/.cache/exo/exo_log/(with automatic log rotation) - Custom model cards:
~/.local/share/exo/custom_model_cards/
You can override these locations by setting the corresponding XDG environment variables.
Enabling RDMA on macOS
RDMA is a new capability added to macOS 26.2. It works on any Mac with Thunderbolt 5 (M4 Pro Mac Mini, M4 Max Mac Studio, M4 Max MacBook Pro, M3 Ultra Mac Studio).
Please refer to the caveats for immediate troubleshooting.
To enable RDMA on macOS, follow these steps:
- Shut down your Mac.
- Hold down the power button for 10 seconds until the boot menu appears.
- Select "Options" to enter Recovery mode.
- When the Recovery UI appears, open the Terminal from the Utilities menu.
- In the Terminal, type:
and press Enter.rdma_ctl enable - Reboot your Mac.
After that, RDMA will be enabled in macOS and exo will take care of the rest.
Important Caveats
- Devices that wish to be part of an RDMA cluster must be connected to all other devices in the cluster.
- The cables must support TB5.
- On a Mac Studio, you cannot use the Thunderbolt 5 port next to the Ethernet port.
- If running from source, please use the script found at
tmp/set_rdma_network_config.sh, which will disable Thunderbolt Bridge and set dhcp on each RDMA port. - RDMA ports may be unable to discover each other on different versions of MacOS. Please ensure that OS versions match exactly (even beta version numbers) on all devices.
Environment Variables
exo supports several environment variables for configuration:
| Variable | Description | Default |
|---|---|---|
EXO_DEFAULT_MODELS_DIR |
Default directory for model downloads and caches. Always first in the writable dirs list. | ~/.local/share/exo/models (Linux) or ~/.exo/models (macOS) |
EXO_MODELS_DIRS |
Colon-separated additional writable directories for model downloads. Checked in order after the default; first with enough free space is used. | None |
EXO_MODELS_READ_ONLY_DIRS |
Colon-separated read-only directories to search for pre-downloaded models (e.g., NFS mounts, shared storage). Models here cannot be deleted. | None |
EXO_OFFLINE |
Run without internet connection (uses only local models) | false |
EXO_ENABLE_IMAGE_MODELS |
Enable image model support | false |
EXO_ZENOH_NAMESPACE |
Custom namespace for cluster isolation (same as --namespace) |
exo version |
EXO_FAST_SYNCH |
Control MLX_METAL_FAST_SYNCH behavior (for JACCL backend) | Auto |
EXO_TRACING_ENABLED |
Enable distributed tracing for performance analysis | false |
Example usage:
# Use pre-downloaded models from NFS mount (read-only)
EXO_MODELS_READ_ONLY_DIRS=/mnt/nfs/models:/opt/ai-models uv run exo
# Download models to an external SSD (falls back to default dir if full)
EXO_MODELS_DIRS=/Volumes/ExternalSSD/exo-models uv run exo
# Run in offline mode
EXO_OFFLINE=true uv run exo
# Enable image models
EXO_ENABLE_IMAGE_MODELS=true uv run exo
# Use custom namespace for cluster isolation
EXO_ZENOH_NAMESPACE=my-dev-cluster uv run exo
Using the API
exo provides multiple API-compatible interfaces for maximum compatibility with existing tools:
- OpenAI Chat Completions API - Compatible with OpenAI clients
- Claude Messages API - Compatible with Anthropic's Claude format
- OpenAI Responses API - Compatible with OpenAI's Responses format
- Ollama API - Compatible with Ollama and tools like OpenWebUI
If you prefer to interact with exo via the API, here is an example creating an instance of a small model (mlx-community/Llama-3.2-1B-Instruct-4bit), sending a chat completions request and deleting the instance.
1. Preview instance placements
The /instance/previews endpoint will preview all valid placements for your model.
curl "http://localhost:52415/instance/previews?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"
Sample response:
{
"previews": [
{
"model_id": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"sharding": "Pipeline",
"instance_meta": "MlxRing",
"instance": {...},
"memory_delta_by_node": {"local": 729808896},
"error": null
}
// ...possibly more placements...
]
}
This will return all valid placements for this model. Pick a placement that you like.
To pick the first one, pipe into jq:
curl "http://localhost:52415/instance/previews?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit" | jq -c '.previews[] | select(.error == null) | .instance' | head -n1
2. Create a model instance
Send a POST to /instance with your desired placement in the instance field (the full payload must match types as in CreateInstanceParams), which you can copy from step 1:
curl -X POST http://localhost:52415/instance \
-H 'Content-Type: application/json' \
-d '{
"instance": {...}
}'
Sample response:
{
"message": "Command received.",
"command_id": "e9d1a8ab-...."
}
This command is asynchronous. Before sending inference requests, wait until the API sees the new instance for this model:
curl -N "http://localhost:52415/instance/await?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"
The endpoint returns an SSE stream. A successful wait emits a message with
"type": "ready" and the matching instance; a timeout emits "type": "timeout".
By default it waits indefinitely. Set timeout_seconds to a positive value to
bound the wait.
3. Send a chat completion
Now, make a POST to /v1/chat/completions (the same format as OpenAI's API):
curl -N -X POST http://localhost:52415/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"messages": [
{"role": "user", "content": "What is Llama 3.2 1B?"}
],
"stream": true
}'
4. Delete the instance
When you're done, delete the instance by its ID (find it via /state or /instance endpoints):
curl -X DELETE http://localhost:52415/instance/YOUR_INSTANCE_ID
Claude Messages API Compatibility
Use the Claude Messages API format with the /v1/messages endpoint:
curl -N -X POST http://localhost:52415/v1/messages \
-H 'Content-Type: application/json' \
-d '{
"model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"messages": [
{"role": "user", "content": "Hello"}
],
"max_tokens": 1024,
"stream": true
}'
OpenAI Responses API Compatibility
Use the OpenAI Responses API format with the /v1/responses endpoint:
curl -N -X POST http://localhost:52415/v1/responses \
-H 'Content-Type: application/json' \
-d '{
"model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"input": "Hello",
"stream": true
}'
Ollama API Compatibility
exo supports Ollama API endpoints for compatibility with tools like OpenWebUI:
# Ollama chat
curl -X POST http://localhost:52415/ollama/api/chat \
-H 'Content-Type: application/json' \
-d '{
"model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"messages": [
{"role": "user", "content": "Hello"}
],
"stream": false
}'
# List models (Ollama format)
curl http://localhost:52415/ollama/api/tags
Custom Model Loading from HuggingFace
You can add custom models from the HuggingFace hub:
curl -X POST http://localhost:52415/models/add \
-H 'Content-Type: application/json' \
-d '{
"model_id": "mlx-community/my-custom-model"
}'
Security Note:
Custom models requiring trust_remote_code in their configuration must be explicitly enabled (default is false) for security. Only enable this if you trust the model's remote code execution. Models are fetched from HuggingFace and stored locally as custom model cards.
Other useful API endpoints:*
- List all models:
curl http://localhost:52415/models - List downloaded models only:
curl http://localhost:52415/models?status=downloaded - Search HuggingFace:
curl "http://localhost:52415/models/search?query=llama&limit=10" - Inspect instance IDs and deployment state:
curl http://localhost:52415/state
For further details, see:
- API documentation in docs/api.md.
- API endpoints in src/exo/api/main.py, and their request and response types in src/exo/api/types/.
Benchmarking
The exo-bench tool measures model prefill and token generation speed across different placement configurations. This helps you optimize model performance and validate improvements.
Prerequisites:
- Nodes should be running with
uv run exobefore benchmarking - The tool uses the
/bench/chat/completionsendpoint
Basic usage:
uv run bench/exo_bench.py \
--model Llama-3.2-1B-Instruct-4bit \
--pp 128,256,512 \
--tg 128,256
Key parameters:
--model: Model to benchmark (short ID or HuggingFace ID)--pp: Prompt size hints (comma-separated integers)--tg: Generation lengths (comma-separated integers)--max-nodes: Limit placements to N nodes (default: 4)--instance-meta: Filter byring,jaccl, orboth(default: both)--sharding: Filter bypipeline,tensor, orboth(default: both)--repeat: Number of repetitions per configuration (default: 1)--warmup: Warmup runs per placement (default: 0)--json-out: Output file for results (default: bench/results.json)
Example with filters:
uv run bench/exo_bench.py \
--model Llama-3.2-1B-Instruct-4bit \
--pp 128,512 \
--tg 128 \
--max-nodes 2 \
--sharding tensor \
--repeat 3 \
--json-out my-results.json
The tool outputs performance metrics including prompt tokens per second (prompt_tps), generation tokens per second (generation_tps), and peak memory usage for each configuration.
Hardware Accelerator Support
On macOS, exo uses the GPU. On Linux, exo currently runs on CPU. We are working on extending hardware accelerator support. If you'd like support for a new hardware platform, please search for an existing feature request and add a thumbs up so we know what hardware is important to the community.
Contributing
See CONTRIBUTING.md for guidelines on how to contribute to exo.