ts3java/ts3-client/README.md

# TS3J Client

A desktop TeamSpeak 3 client built on top of the [`ts3j`](../ts3j) reverse-engineered
TS3 protocol library. It looks and behaves like the official TS3 client and focuses
first on the features that matter most: **real-time voice with Opus encoding and
voice-activation detection (VAD)**.

The project is split so the frontend can be replaced (e.g. a future web UI via
TeaVM/CheerpJ) without touching the library:

| Module | Artifact | Responsibility |
|--------|----------|----------------|
| `core` | `ts3-client-core` | Frontend-agnostic library: protocol integration, server model, connection orchestration, the voice pipelines and their DSP. No UI, no platform audio API, no native code. |
| `desktop` | `ts3-client-desktop` | Desktop audio backend: PipeWire (or Java Sound) capture/playback + native Opus via the FFM API (project Panama), plus the global hotkey hooks. |
| `swing` | `ts3-client-swing` | Swing desktop UI + entry point. Depends on `core` and `desktop`. |

The core exposes `AudioBackend` / `VoiceInput` / `VoiceOutput`; the frontend injects
a concrete backend (`DesktopAudioBackend`) into `TeamspeakConnection`. A different
frontend supplies its own UI and audio backend while reusing `core` unchanged, as the
Android app in [`../android`](../android) does.

## Features

### Voice (the priority)
- **Native Opus codec** via a direct Panama (Foreign Function & Memory API) binding —
  no JNA, no other third-party FFI. The system `libopus` is used when installed,
  otherwise a bundled copy is extracted from the JAR. Only the Windows x86-64 build
  is bundled by default (build with `-Dnatives.all` to package every platform);
  elsewhere install libopus from your package manager. Encoding at 48 kHz, 20 ms frames.
- **Voice Activation Detection (VAD)** with the same three modes as the TS3 client:
  - **Volume Gate** — RMS/dBFS threshold with a live input meter and hangover.
  - **Automatic** — a Java port of WebRTC's RNN speech detector (`rnn_vad`), the one the
    TS3 client uses, with the same weights.
  - **Hybrid** — transmit only when loud enough **and** detected as speech.
  - Optional **VAD over Push-To-Talk** (keep detecting voice while PTT is available).
- **Capture pre-processing** — a Java port of the WebRTC audio-processing stages the TS3
  client enables, in its order: high-pass filter → **noise suppression** (WebRTC NS, TS3's
  four levels) → **typing attenuation** (the transient suppressor, told about key presses)
  → **automatic gain control** (AGC2, WebRTC's successor to the AGC1 TS3 uses). Checked
  against WebRTC built from source.
  - **Boost quiet speech** (0–20 dB) raises AGC2's cap on the amplified noise floor, which
    otherwise holds back a distant voice that is barely above the background noise.
  - Echo cancellation (WebRTC AEC3) is intentionally omitted — it needs the loudspeaker
    reference signal and matters mainly for open speakers, not the typical headset.
- **Push-to-Talk** — bind any key or mouse button as a global hotkey; transmits only while held.
- **Continuous** transmission mode.
- **Adaptive jitter buffer** — a port of the libspeex jitter buffer the TS3 client uses,
  set up and driven as it does: the delay follows the measured network jitter, and what it
  has learned carries over between talk bursts.
- **Playback mixing** — each speaker gets their own Opus decoder and audio line (a stream
  of their own in the PipeWire mixer), so one slow decode never blocks the others.
- **Whisper receive** — targeted voice is decoded and played like normal voice.
- Per-client **mute**, master **deafen**, adjustable mic gain / playback volume.
- **On-the-fly Opus tuning** — bitrate, complexity, VBR, FEC and voice/music codec
  can all be changed live from the options dialog and take effect on the running
  encoder immediately (no reconnect); a voice↔music switch transparently rebuilds
  the encoder because Opus fixes its application mode at creation.
- Configurable capture/playback **devices**.

### Server interaction
- Connect to any TS3 server (an identity is generated on first use if none exists).
- **Channel/client tree** styled like TS3, updated live from protocol events
  (joins, leaves, moves, channel create/edit/delete, nickname/mute/away changes).
- **Talk indicators** — speakers turn green live as they talk.
- **Info panel** — selecting a channel shows its topic and **description** (fetched
  on demand); selecting a client shows its **server groups** and **channel group**
  (names resolved from the server's group lists), talk power, platform and version.
- **Server-group badge** next to each client's nickname in the tree.
- **Spacer channels** — TS3 `[spacer]`/`[*spacer]`/`[c/l/r spacer]` names render as
  non-interactive separators (fill, centred, aligned).
- Double-click a channel to **join**; right-click a client to **poke**, open a
  **private chat**, or **locally mute** them.
- **Chat** to the current channel or the whole server; receive channel/server/private
  messages and **pokes**.
- **Avatars** — a selected client's avatar is shown in the info panel, fetched from the
  server's file repository (`/avatar_<id>` in channel 0, verified against the client's
  `client_flag_avatar` MD5) and cached on disk; **animated GIFs play**. Self → Set
  avatar… uploads your own picture (PNG/JPEG/GIF/BMP, within the server's
  `i_client_max_avatar_filesize`) or removes it.
- **Server bookmarks** — quick-connect menu with add/edit/remove management, each
  optionally pinned to a specific identity.
- **Identity management** (Tools → Identities) — keep several identities, mark one
  as the default, pick one per server or per bookmark, rename, raise an identity's
  security level, and import/export TeamSpeak-compatible `.ini` identity files.
- **Import from TeamSpeak 3 / myTeamSpeak** — the official client keeps its bookmarks
  and identities (the data a myTeamSpeak account synchronises) as protobuf
  `Item_Data` blobs in the `ProtobufItems` table of its `settings.db`. On first run,
  when this client has no identity of its own yet, it adopts all of them — identities,
  the default identity and its nickname, and the bookmarks with the identities they
  connect with; "Import from TeamSpeak 3…" in Manage Bookmarks and in Identities
  re-imports at any time (additive: known identities and bookmarks are left alone).
  The schema was recovered from the descriptors embedded in the client binary.
- **Contacts** (Tools → Contacts) — TeamSpeak's friend/blocked list: mark a client as
  **Friend**, **Blocked** or **Neutral** from its context menu (or drag it into the
  window), give it a custom and a phonetic nickname, choose how the name is shown
  (custom name and nickname / only custom / only nickname), and per contact **mute
  automatically**, **ignore** server/channel chat, private chat, pokes and away
  messages, and **allow or deny whispers**. Friends are coloured green and blocked
  clients red in the tree (optional), per-type defaults for new contacts are
  configurable ("Set Defaults"), and the global whisper policy lives in Options →
  Contacts. The list is stored in `~/.ts3jclient/contacts.txt` in the *exact* record
  format the official client keeps in the `Contacts` table of its `settings.db`; on
  first run it is seeded from an installed TeamSpeak 3 client's database (read with a
  small built-in SQLite reader — no driver needed), and can be re-imported any time.
- **Per-client volume** ("Change Volume…") — a dB gain on top of the master volume,
  remembered in the client's contact entry.
- **Ban List** (Tools → Ban List) — the server's bans in a searchable table (Add /
  Remove / Edit, own-ban filter and highlighting, choosable columns), with the
  official Add dialog: IP and nickname patterns read as IPv4/IPv6 wildcards, fixed
  strings or regular expressions, unique id, myTeamSpeak id, reason presets and a
  duration capped by `i_client_ban_max_bantime`.
- **Self status** — Away (with message) and Channel Commander toggles.
- **Status bar** shows the server name, user count and live ping.
- Change your nickname, mute/deafen from the toolbar.

### Hotkeys
- **Global hotkeys**, as in TeamSpeak 3: single keys or combinations (any key can act
  as the modifier) and mouse buttons including *Mouse 4* / *Mouse 5*, working
  system-wide rather than only while the window has focus. Configure them in
  **Options → Hotkeys**.
- Per binding you choose whether it **triggers on key down or on key up**, and whether
  it applies **on the active server** only or to every connected one (checkbox off).
  Push-to-talk style actions ignore the trigger setting and simply last while held.
- The action list is TeamSpeak's own, reverse-engineered from its hotkey dialog:
  three categories (*Server*, *Self*, *Misc*) and a **"Show advanced actions"**
  checkbox that reveals the rest, exactly as the original hides all but the everyday
  actions. Actions this client does not implement are listed but greyed out.
- Push-to-talk is one of those hotkeys; the button in **Options → Voice Activation**
  edits that binding.
- On Linux, capture uses **XInput2 raw events**, as TeamSpeak's own client does: no
  privileges needed and the keystroke is not swallowed. X11's RECORD extension is kept
  as a fallback for servers without XInput2.
- On **Windows**, capture uses **raw input** (`RIDEV_INPUTSINK`) through a message-only
  window: it observes rather than intercepts, so unlike a low-level keyboard hook it
  cannot swallow a keystroke or stall the system input queue, and it reports the side
  buttons and both edges of every key. Bindings are stored as scan codes, so they follow
  the physical key rather than the layout.
- On Wayland the hooks run inside Xwayland, so they see every X11 application, and keys
  aimed at native Wayland windows only where the compositor forwards them — under KWin
  that is *System Settings → Window Management → Legacy X11 App Support*, which is why
  TeamSpeak's hotkeys work there. Where it forwards nothing, the client reads
  `/dev/input/event*` instead, for which your user has to be in the `input` group.
  The Hotkeys tab says which backend is in use, or why none is.

### Notification sounds
- **Sound packs** in TeamSpeak's own format: a folder of waves plus a `settings.ini`
  mapping actions (`CONNECTION_CONNECTED`, `CLIENT_MOVED_TO_CURRENT_CHANNEL_STAYS`, …)
  to `play("file.wav")` entries, with `${clientType}`/`${channelname}`/… placeholders
  resolved per event. Packs are picked up from `~/.ts3jclient/sound`, an installed
  TeamSpeak 3 client (`$TS3_CLIENT_DIR` or the usual install paths) and a folder of
  your choosing, so the official packs work unchanged.
- **Per-action configuration** (Options → Notifications): switch any action's sound
  off, and mark actions as **important** (shown in bold, as in TS3) — important
  actions are the only ones still played while the speakers are muted.
- Sounds are mixed onto one playback line, so overlapping events never fight over
  the device; pack volume is separate from the voice volume.

### Icon packs
- **Icon packs** in TeamSpeak's own format: a `.zip` (or unpacked folder) of SVG/PNG
  artwork plus a `settings.ini` mapping icon keys (`CHANNEL_GREEN`, `PLAYER_ON`,
  `CONNECT`, …) to files. Packs are picked up from `~/.ts3jclient/gfx`, an installed
  TeamSpeak 3 client (`$TS3_CLIENT_DIR` or the usual install paths) and a folder of
  your choosing, so the official packs (`default_colored_2014.zip`,
  `default_mono_2014.zip`, the legacy `default.zip`) work unchanged.
- **Material**, a pack in the same format built from Google's Material icons, ships
  with the client (`core/src/main/resources/gfx/material`, generated by
  `tools/material-icon-pack.py`). Its monochrome icons paint with `currentColor`, which
  follows the theme's text colour; it covers every icon TeamSpeak's own packs do. It is
  the Android app's default, and the Android app draws TeamSpeak packs too — import one (e.g. `default_colored_2014.zip`) in its settings.
- The pack draws the tree (server, channel, client state), the toolbar, the menus,
  the context menus, the file browser and TeamSpeak's default group icons. Vector
  art is rasterised at the size it is drawn at; a pack's `FALLBACK` option decides
  whether icons it lacks come from `default.zip`, and anything still missing falls
  back to the icons the client draws itself.
- **Pick one in Options → Design**, which also shows a viewer of everything the pack
  contains; switching redraws the window right away.

### Display scaling (Linux)
- Java2D on X11 (and so under XWayland) only scales by whole numbers, so 125 % or
  150 % comes out at 100 %. On Linux the client pins Java2D to 1 and hands the full
  factor to FlatLaf instead, scaling its own sizes and icons to match.
- The factor follows the desktop: XSETTINGS `Xft/DPI` (GNOME, Plasma), else KWin's
  XWayland scale, else `GDK_SCALE`. For crisp text on Wayland, let X11 apps scale
  themselves (Plasma: *Display Configuration → Legacy Applications → Apply scaling
  themselves*; GNOME: the `xwayland-native-scaling` mutter feature), otherwise the
  compositor upscales the window.
- **Options → Design → Interface scale** overrides it (after a restart);
  `-Dflatlaf.uiScale=` or `-Dsun.java2d.uiScale=` on the command line bypasses all this.

## Requirements
- Java 25+ (the LTS; developed on Temurin 26)
- The native Opus library on the system:
  - Debian/Ubuntu: `sudo apt install libopus0`
  - Arch: `sudo pacman -S opus`
  - macOS: `brew install opus`

## Building
The build compiles `ts3j` from `../ts3j` alongside the client, so make sure the submodule is
checked out (`git submodule update --init`):

```bash
mvn -DskipTests package
```

This produces a runnable fat-jar at `swing/target/ts3-client.jar`.

## Running
```bash
java -jar swing/target/ts3-client.jar
# or, during development:
mvn -pl swing exec:java
```

Then use **Connections → Connect…**, enter a server address, port (default 9987),
nickname and identity, and connect. Open **Tools → Options** to pick audio devices and tune
voice activation while watching the live meter.

## Architecture
```
core/   com.ts3client
├── config                   the profile (~/.ts3jclient): Settings, Bookmarks, IdentityStore,
│                            AwayMessages, …; AppDirs finds it, ProfileFiles writes it
│                            privately and all at once
├── session
│   ├── ServerSession        one server tab's state and actions, whatever the frontend
│   └── Sessions             what the servers share: the microphone, away state, nickname
├── net
│   ├── TeamspeakConnection  ties socket + audio + model; runs requests on its own pool
│   ├── ConnectionEventHandler  server events → model changes, sounds and log lines
│   ├── ServerModel          channel/client state; ChannelNode/ClientEntry view models
│   ├── ChannelAdmin/BanAdmin/AvatarAdmin  channel editing, the ban list, our own avatar
│   ├── IconRepository       server icons (avatar/ holds the client avatar cache)
│   ├── filetransfer         channel file browsing, up- and downloads
│   └── ConnectionListener   frontend callbacks
├── audio                    the voice pipelines every platform shares
│   ├── AudioBackend/AudioIo what a platform supplies: devices, lines, Opus
│   ├── CaptureVoiceInput    capture → processing → VAD/PTT gate → Opus encode
│   ├── StreamingVoiceOutput per-speaker decode, each through a VoiceStream
│   ├── JitterBuffer         port of the libspeex jitter buffer TS3 uses
│   ├── PerSpeakerPlayout    a line per speaker (desktop); MixedPlayout mixes onto one (Android)
│   ├── processing           port of TS3's WebRTC capture chain: HPF, NS, transients, AGC2
│   ├── vad                  port of WebRTC's rnn_vad speech detector
│   └── opus                 codec interfaces
├── chatlog                  TS3-compatible chat logs, shared with the TS3 client by default
├── contacts                 friends and blocked clients, in TS3's own record format
├── text                     BBCode, TeamSpeak links, chat search
├── gfx
│   ├── IconPack             an icon pack zip/folder/classpath folder + its settings.ini mapping
│   ├── IconPacks            discovery of installed packs, the shipped Material one, lookup order
│   └── IconKeys             the icon keys the channel tree asks for
├── hotkey
│   ├── HotkeyAction         catalogue of TeamSpeak's hotkey actions (RE'd from its binary)
│   ├── Hotkey/HotkeyCombo   one binding: keys, trigger edge, server scope, argument
│   ├── Hotkeys              persisted bindings (~/.ts3jclient/hotkeys.properties)
│   ├── GlobalInputHook      platform hook interface: system-wide key/button events
│   └── HotkeyEngine         held-key tracking, combination matching, recording
├── sound
│   ├── SoundEvent           catalogue of actions (TeamSpeak's own event ids)
│   ├── SoundPack            a pack folder + its settings.ini mapping
│   ├── SoundPacks           discovery of installed packs
│   ├── NotificationSettings per-action enabled / important flags
│   ├── SoundNotifier        decides what is heard (pack + config + mute state)
│   ├── SoundPlayer          renders a sound
│   └── PackSoundPlayer      decodes, resamples and mixes sounds onto one line
└── teamspeak                the official client's own data files
    ├── TeamSpeakSettingsDb  locates settings.db; reads Contacts and the ProtobufItems sync store
    ├── SqliteReader         minimal read-only SQLite 3 file reader (no driver)
    ├── SyncItem             a synced bookmark / identity / folder (Item_Data)
    ├── SyncItemDecoder      protobuf wire decoding of Item_Data, via ProtobufReader
    └── TeamSpeakImporter    merges synced items into IdentityStore / Bookmarks

desktop/  com.ts3client.audio.desktop + com.ts3client.hotkey.desktop
├── DesktopAudioBackend      PipeWire where it runs, Java Sound elsewhere, and native Opus
├── Opus/NativeOpusCodec     Panama (FFM) binding to native libopus
├── AudioDevices             device enumeration + line opening (48 kHz/16-bit)
├── JavaSoundCapture/JavaSoundPlayback  Java Sound lines
├── pipewire                 PipeWire streams through FFM
└── hotkey.desktop
    ├── XInput2InputHook     global key/button capture via XInput2 raw events
    ├── XRecordInputHook     the same via X11's RECORD extension, as a fallback
    ├── EvdevInputHook       /dev/input fallback for Wayland sessions
    ├── WindowsInputHook     the same on Windows: raw input into a message-only window
    ├── RawInput             RAWINPUT layout and decoding, split out to be testable
    ├── X11KeyNamer/WindowsKeyNamer  layout-aware key labels per platform
    └── DesktopInputHooks    picks the backend that suits the session

swing/  com.ts3client
├── Main                     entry point (look & feel, settings, backend injection)
└── ui
    ├── MainFrame            window: menu, toolbar, tree | chat, status bar
    ├── ServerTreePanel      TS3-style channel/client tree (group badges, spacers)
    ├── Spacers              TS3 spacer-channel name parsing/rendering
    ├── InfoPanel            channel description / client group + details view
    ├── ChatPanel            chat log + input
    ├── SettingsDialog       the Options dialog: TS3-style page sidebar; one panel per page (ApplicationPanel, …, ChatLogsPanel)
    ├── HotkeysPanel         hotkey list (Options → Hotkeys)
    ├── HotkeyDialog         add/edit one hotkey: action, combination, trigger, scope
    ├── HotkeyService        bindings + engine + input hook, for the dialogs
    ├── HotkeyActions        carries a fired hotkey out on the client
    ├── NotificationsPanel   sound pack + per-action sound/important configuration
    ├── IconPackPanel        icon pack chooser + icon viewer (Options → Design)
    ├── DisplayScale         interface scale on Linux (fractional, via FlatLaf)
    ├── ConnectDialog        connect form
    ├── BookmarksDialog      manage saved servers
    ├── IdentitiesDialog     manage identities (new/import/export/improve)
    ├── IdentityChooser      identity drop-down shared by connect/bookmark forms
    ├── LevelMeter           dBFS meter with threshold marker
    ├── IconTheme            active icon pack: rasterising (JSVG) and caching
    ├── Icons                icon lookup by TeamSpeak key, with drawn fallbacks
    └── Theme                palette + fonts
```

## Known limitations / next steps
- Whisper is received/played but not yet **sendable** from the UI.
- Hotkey actions TeamSpeak offers but this client does not perform yet (listed but
  greyed out in the dialog): capture/playback/hotkey profiles and sound packs,
  whisper and push-to-whisper, recording, plugins, server groups, talk power, 3D
  sound, hardware ("local") microphone mute and the channel-traversal variants
  beyond "Switch to Channel".
- Global hotkeys cover Linux and Windows; **macOS has no backend**, so hotkeys there are
  inert (a `CGEventTap`, which needs Accessibility permission, is the way in). The
  Windows backend is written but has not been run on Windows — only its decoding is
  covered by tests.
- When no backend can start, push-to-talk does not work at all, not even with the window
  focused; a focus-scoped fallback hook would restore the pre-hotkey behaviour.