Interwoven data streams connecting distributed servers

Open source / Version 1.3.0

WeaveFTPd

Distributed FTP infrastructure built for fast transfers, race integrity, and hands-on site operations.

Gosingle-binary daemon
TLS + mTLScontrol and data security
Linux + Windowsmainline + native beta branch

One control plane, many disks

Keep FTP state central. Move payloads directly.

WeaveFTPd separates authentication, policy, VFS, SITE commands, plugins, and race state from physical storage. In the default passthrough mode, encrypted data travels directly between the client and the selected slave.

Architecture

Built around the transfer path

Choose direct passthrough for maximum throughput or proxy mode when storage nodes cannot be reached by clients.

FTP clientControl + TLS data

FTP control

MasterAuth, VFS, policy

TLS payload

Slave storageDirect disk I/O

Best throughput when slave passive ports are reachable. Secure FXP is supported with SSCN and PROT P.

Capabilities

The pieces a real site needs

Transfer performance

Direct slave data paths, pooled buffers, Linux sendfile support, TCP tuning, and TLS session resumption.

Race integrity

SFV/CRC verification, X-DUPE handling, race stats, multi-disc naming, incomplete markers, and resilient retries.

Security by default

TLS data channels, CA-backed master/slave mTLS, fail-closed slave authentication, ACLs, and IP masks.

Operational plugins

Autonuke, pretime, release guard, requests, slowkick, spacekeeper, metadata, and dated directories.

IRC sitebot

Race announces, channel routing, FiSH encryption, TOP, requests, quota, free space, rules, and remote control.

Virtual filesystem

Central VFS state across multiple roots, mounted archives, remerge controls, metadata snapshots, and routing policy.

Deploy

Start with the layout that fits

The guided Linux installer is the shortest path. Docker keeps master, sitebot, and slaves separated. Native Windows support lives on the separate windows beta branch and uses PowerShell plus pure-Go SQLite.

Slave connections fail closed.Configure a per-slave mTLS identity or a matching IP/CIDR mask before connecting remote storage.

git clone https://github.com/WeaveFTPd/WeaveFTPd.git
cd WeaveFTPd
./setup.sh install
./weaveftpd

git clone https://github.com/WeaveFTPd/WeaveFTPd.git
cd WeaveFTPd/docker/deploy
./init.sh master
docker build --target daemon -t weaveftpd:latest ../..
docker build --target sitebot -t weaveftpd-sitebot:latest ../..
docker compose --env-file master.env \
  -f master-sitebot.compose.yml up -d master sitebot

git clone https://github.com/WeaveFTPd/WeaveFTPd.git
cd WeaveFTPd
git switch windows
Set-ExecutionPolicy -Scope Process Bypass
.\setup.ps1 -Mode install -Config both
.\run-master.ps1

Operator guide / Version 1.3.0

From first boot to multi-slave

The essential installation, security, networking, and maintenance guidance lives here. Use the wiki for the deeper plugin and command reference.

Browse the wiki

01 / Quick start

Install the Linux mainline

The guided installer creates missing configs, plugin files, certificates, and binaries while preserving existing live configuration.

git clone https://github.com/WeaveFTPd/WeaveFTPd.git
cd WeaveFTPd
./setup.sh install
./weaveftpd --config ./etc/config.yml

Before exposing the server: replace the example weaveftpd / weaveftpd login, set the public and passive addresses, and keep require_tls_control plus require_tls_data enabled.

Master config

Start from etc/config-example.yml. The master owns FTP control, authentication, ACLs, VFS state, SITE commands, routing, plugins, and race state.

Slave config

Start from etc/config-slave-example.yml. A pure slave needs the daemon, its physical roots, client-facing TLS, and its connection to the master.

02 / Master and slave

Choose the data path before opening ports

FTP control always terminates on the master. The configured transfer mode decides whether payloads go directly to a slave or are relayed by the master.

PassthroughRecommended. Clients and FXP peers must reach each slave's advertised address and passive range.

ProxyOnly the master must be public, but every byte crosses the master and consumes relay bandwidth.

RoutingSection/path policy selects eligible slaves, then load is balanced by active transfers, weight, and free space.

Typical ports: the FTP control listener is configured with listen_port, slaves connect to master.control_port, and passthrough data uses each slave's pasv_port_min through pasv_port_max.

03 / Security and NAT

Slave authentication fails closed

Every slave needs either its own mTLS identity or a source mask registered for that exact slave name. A global allowlist alone is not authentication.

Recommended: mTLS

Issue one certificate per slave. Its certificate Common Name must exactly equal slave.name, and the master certificate SAN must match slave.master_host.

./setup.sh certs "My Site"
./setup.sh slavecert SLAVE1

Alternative: source mask

Leave the master/slave client-certificate settings empty and register the connecting public IP or CIDR before starting the slave.

SITE SLAVE SLAVE1 ADDMASK 203.0.113.20/32

Behind NAT: set slave.bind_ip to the public address, slave.local_bind_ip to the machine's LAN address, and forward the slave passive range to that LAN address.

Verification matters: a source-mask setup keeps the slave control link encrypted, but does not verify the master certificate. Use mTLS for both identities. Active/SSCN PROT P FXP data connections also use a legacy compatibility client that encrypts traffic without certificate-authenticating the data peer.

Read the complete authentication guide

04 / Sitebot and FIFO

Keep the event transport local to the master

The daemon writes JSON-line events and the sitebot turns them into routed IRC output. Both processes must point to the exact same event_fifo path.

Native LinuxThe event path is a POSIX FIFO. Restart the daemon and sitebot after changing that path.

Separate containersMaster and sitebot may run separately when they share the same host-mounted /app/etc directory.

Remote slavesThey do not need the FIFO or a sitebot. Sitebot belongs on the master machine.

IRC TLS verification defaults to certificate checking. Use strict for a public CA or custom plus tls_ca_cert for a private CA. insecure keeps TLS encryption but accepts any server certificate, so use it only as an explicit legacy fallback.

Open the Sitebot guide

05 / Docker and Windows

Use the deployment path that matches the host

Docker on Linux

The deploy pack builds local images; no public container registry is required. It uses host networking for PASV/CPSV addresses, passive ranges, and FXP peer checks.

Run master and sitebot on the master host. Add the optional local-slave profile there, or initialize slave.compose.yml on another machine.

Docker deployment guide

Native Windows beta

Switch to the separate windows branch. Its PowerShell setup can download Go when missing, generates certificates without OpenSSL, uses pure-Go SQLite on Windows, and creates console runner scripts.

Windows uses an append-only event file instead of a POSIX FIFO. Native services are intentionally outside the current setup scope.

View the Windows branch

06 / Config and updates

Keep runtime state outside Git changes

Important files

etc/config.yml: Active daemon configuration
etc/permissions.yml: ACL and SITE command policy
etc/users/: User records and transfer totals
sitebot/etc/config.yml: IRC, routes, encryption, plugins
plugins/*/config.yml: Site-local daemon plugin settings

Normal update

./setup.sh update
./setup.sh install
./setup.sh build

update uses git pull --ff-only. Commit, stash, or back up local tracked edits first. The installer adds missing config pieces without replacing existing active configs.

Reload or restart: use SITE REHASH for supported policy/config changes. Binary updates, listener ports, TLS paths, mode, control port, and FIFO changes require process restarts. Configure recurring scans under slaves[].remerge.jobs[]; removed legacy background/slave remerge keys are ignored.

Update and maintenance guide

07 / Commands and plugins

The operational surface

WeaveFTPd includes the core site workflows directly, while daemon and sitebot plugins add policy, metadata, automation, announces, and IRC commands.

ZipscriptSFV/CRC checks, race stats, incomplete markers, ZIP checks, MP3/FLAC playlists, and media probing.

Daemon pluginsAutonuke, dateddirs, IMDb, PRE, pretime, releaseguard, requests, slowkick, spacekeeper, speedtest, and TVMaze.

SITE commandsAccounts, groups, ACL-governed admin, traffic, nukes, PRE, requests, search, rescan, remerge, and slave security.

Sitebot pluginsAnnounces, metadata, news, free space, requests, quota, TOP, rules, self-IP, control, and staff commands.

SITE command referenceSyntax, permissions, limits, traffic, remerge, and slave security. Daemon pluginsConfiguration and dedicated pages for each built-in module. Sitebot pluginsIRC commands, routing, themes, encryption, and operator controls. Version 1.3.0 changelogSlave authentication, race integrity, reliability, audio, and integration changes.

Community project

Build it, race it, report what breaks.

Bug reports, deployment feedback, feature requests, and focused pull requests are welcome.

Open an issue Pull requests