Building Desert Storm and Titan Halo
How DanBot Hosting went from SSH sessions and grep to one dashboard that watches every node.
Where this started
If you run a free or low cost game hosting platform, abuse is not a matter of if, but simply, a matter of when. This is especially true in the age of AI development. Somebody spins up a "Minecraft server" that is actually a Discord selfbot farm. Somebody uploads a requirements.txt full of packet manipulation libraries, XMRG mining software, etc. Somebody deletes their server from the panel but leaves 40 GB of volume behind, and multiply that by a few hundred and a node runs out of disk on a Sunday morning. Or in our case, back in late March.
For a long time our answer to all of this was a person with an SSH session, a few saved grep commands, and good instincts. That works right up until it does not. It does not scale across nodes, it does not leave a record of who did what, and it means the answer to "is this account abusive?" depends on which staff member is awake. It wasn't great, especially with very few people who had access to the systems.
So we built two things.
Titan Halo runs on each Pterodactyl node. It scans volumes, watches Docker resource and network usage, finds orphaned volumes and backups, and exposes all of it over an authenticated REST API.
Desert Storm is the internal dashboard. It pulls Pterodactyl, Paymenter, Discord, mail, VirtFusion, our own application database, and every Titan Halo node into one Next.js app with real role based access control and an audit log.
This post is about how they fit together, and about a handful of bugs I found along the way that were genuinely nasty. A couple of them could have destroyed customer data. One of them silently handed more access to staff than I intended. Those are the interesting parts, so those get the most space.
One thing up front: I am deliberately vague about the detection side. There is a section on how the rules engine is built, because the architecture is the interesting part and it generalises, but the signals it looks for, how they are weighted, and where the thresholds sit are all withheld. People who abuse hosting platforms read blogs too, and a detection write up detailed enough to be reproducible is also detailed enough to be evaded. Everything else here is described exactly as it is.
Two Systems, One Job
The split is simple and it has held up well.
+--------------------------------+
| Desert Storm |
| Next.js app, RBAC, audit |
| Pterodactyl / Paymenter / |
| Discord / mail / VirtFusion |
+---------------+----------------+
|
server side fan out, bearer tokens
never touch the browser
|
+---------------------------+---------------------------+
| | |
+-----v------+ +------v-----+ +------v-----+
| Titan Halo | | Titan Halo | | Titan Halo |
| node 1 | | node 2 | ... | node N |
+-----+------+ +------+-----+ +------+-----+
| | |
+--- volumes, docker stats, panel API, cron, logs -------+Titan Halo knows about one node and nothing else. It has no idea Desert Storm exists. It is a bag of bash scripts, a rules file, and an Express server that runs those scripts and parses their output into JSON.
Desert Storm knows about every node, holds every credential, and is the only thing a human ever logs into. That human being me.
Keeping the node agent dumb was a deliberate choice. Bash is what a node operator can debug at 3am with nothing but a terminal. Every script writes to logs/<category>/<script>/logs.txt and can be run by hand:
./titan-halo.sh detect-abuse # scan volumes
./titan-halo.sh docker-network # network I/O per container
./titan-halo.sh find-orphan-volumes # dry run, nothing is deletedThe API is a wrapper over exactly those commands. When something looks wrong in the dashboard, you can reproduce it on the node in one line. That property has paid for itself many times.
Titan Halo: The agent (non-AI) that lives on the Node
All configuration lives in a single config.json, read by both the Node server and the bash scripts. There is no second source of truth and no environment variable sprawl.
{
"apiToken": "...",
"apiPort": 5005,
"panelUrl": "https://panel.example.com",
"panelToken": "ptla_...",
"nodeId": "1",
"volumesDir": "/var/lib/pterodactyl/volumes",
"backupsDir": "/var/lib/pterodactyl/backups",
"maxServerCountDropPct": 20,
"abuseDetection": { "rulesFile": "config/abuse-rules.json" },
"tls": { "certPath": "", "keyPath": "" }
}The abuseDetection block has a handful of other tuning keys, covered loosely in the next section.
The bash side reads it with jq and exposes plain variables plus a couple of helpers:
PTERO_URL=$(jq -r '.panelUrl' "$CONFIG_JSON")
PTERO_API_KEY=$(jq -r '.panelToken' "$CONFIG_JSON")
PTERO_NODE_ID=$(jq -r '.nodeId' "$CONFIG_JSON")
VOLUMES_DIR=$(jq -r '.volumesDir' "$CONFIG_JSON")
# Usage: LOG_FILE=$(init_log "monitoring" "detect-abuse")
init_log() {
local log_dir="$LOG_BASE_DIR/$1/$2"
mkdir -p "$log_dir"
echo "$log_dir/logs.txt"
}TLS is optional and per node, because node config files are placed by hand and there is no way to flip an entire fleet at once. But if you ask for TLS, you get TLS or you get nothing:
// A node that was meant to serve HTTPS should never quietly come back up on
// plain HTTP, so an unreadable certificate is a startup failure, not a warning.
for (const [field, file] of [
["certPath", tls.certPath],
["keyPath", tls.keyPath],
]) {
if (!fs.existsSync(file)) {
console.error(`ERROR: tls.${field} not found: ${file}`);
process.exit(1);
}
}Silent downgrade to plaintext would leak the bearer token on every request. Refusing to boot is loud, and loud is recoverable.
Rules as Data, Not Code
The abuse scanner is rules driven. Detection logic lives in a versioned data file and the script is a generic engine over it. That separation is the point. Adding, tuning or retiring a signal is a data change that gets reviewed like any other change, rather than a patch to scanning code, and because the rule set carries a version number, every scan result records which version produced it. A report from three months ago can still be read correctly today.
This is the part of the system I am going to describe in shape only. Structure below, contents withheld, and that includes the tuning constants as well as the patterns:
abuse-rules.json
version rule set version, stamped onto every scan
<tuning> thresholds and scan bounds, values not published
<layers> one list of signals per detection layer
<suppressions> narrow, reviewed exceptions
<categories> labels for grouping findings in the dashboardDetection runs in layers. A volume is examined from several independent angles, and no single one of them is the detector. Each angle is individually useful and individually weak, which is exactly why there is more than one. Anything defeating a single layer in isolation is not defeating the scanner.
Findings are scored, not counted. Every hit produces a finding carrying a severity, severities are weighted, and a volume is reported only when its total clears a threshold. This is the most important design decision in the whole scanner. Any individual signal, taken alone, has a false positive rate that would drown a support team on a fleet this size. A combination of signals in one volume is a far stronger claim than any one of them, and it is a much harder thing to arrange accidentally. The weights and the threshold are configurable per deployment and are not published here.
Suppression is narrow, never global. Plenty of legitimate software depends on libraries that also turn up in abusive stacks. The tempting fix is to delete the offending rule, which trades one false positive for permanent blindness across the fleet. Instead a signal can be suppressed for a specific, known good case while staying fully active everywhere else. Suppressions live in the same versioned rule data as everything else, so they are reviewed rather than quietly accumulating.
Some benign shapes are filtered structurally. A category of legitimate software is named directly after the thing it defends against, and naive matching flags the defenders alongside the attackers. There is a small, deliberate set of filters for that class of confusion.
Cheap checks gate expensive ones. The obvious implementation walks every volume and runs every rule against every file, which on a node with well over a thousand volumes is a disk melting proposition, and running hourly it would be a permanent tax on customer I/O. The real ordering does the cheap, highly selective work first, in a single pass across the whole volumes directory rather than once per volume, and only escalates to the genuinely expensive analysis on volumes that have already produced a reason to look harder. Every expensive stage is bounded by configured limits. The overwhelming majority of volumes are dismissed after reading very little.
Nodes can also carry extra rules without forking the base file. A supplemental file is merged over the base at scan time, list fields concatenated and object fields overlaid, so a node with a specific local problem can be given a signal for it while continuing to track the shared rule set.
What comes out the other end is a scored, categorised alert per flagged volume, with its individual findings attached. That output is a plain text log, which is what makes the parsing seam described further down both testable and boring.
Major oversite: Almost deleting Customer Data
This is the one that still makes me uncomfortable, as I've been used to this (for those that remember my 2022 bot db destruction).
Titan Halo decides what is an "orphan" by diffing what exists on disk against what the panel says exists for this node. A volume directory with no matching server is an orphan. An orphan can be deleted.
The original implementation fetched the server list in one request:
# The original. Looks fine. Is not fine.
curl -s -H "Authorization: Bearer $PTERO_API_KEY" \
"$PTERO_URL/api/application/nodes/$PTERO_NODE_ID?include=servers"Pterodactyl paginates every list response. It paginates relationship includes too. So on any node with more servers than the page size, that request returned the first page and quietly stopped. Every server past the first page had no matching entry in the list.
Which is to say: every server past the first page looked like an orphan. To a script whose job is deleting orphaned volumes.
We caught this before it ran destructively in anger, but the gap between "we caught it" and "we did not" was thinner than I would like. The fix is not just pagination. It is pagination plus an explicit refusal to proceed on anything ambiguous:
fetch_server_uuids() {
page=1
total_pages=1
while [[ "$page" -le "$total_pages" ]]; do
curl -s -H "Authorization: Bearer $PTERO_API_KEY" \
-H "Accept: application/vnd.pterodactyl.v1+json" \
-w "%{http_code}" -o "$tmp_response" \
"$PTERO_URL/api/application/servers?per_page=$PTERO_PER_PAGE&page=$page" \
> "$tmp_status"
http_status=$(cat "$tmp_status")
if [[ "$http_status" != "200" ]]; then
echo "[ERROR] Pterodactyl API request failed on page $page (HTTP $http_status)"
exit 1
fi
# A truncated or non-JSON body must not be read as "no servers here".
if ! jq -e '.data' "$tmp_response" >/dev/null 2>&1; then
echo "[FATAL] Malformed API response on page $page. Aborting to prevent data loss."
exit 1
fi
# Re-read the page count every iteration: servers can be created while we
# page, and stopping at a stale total_pages would drop the tail of the list.
total_pages=$(jq -r '.meta.pagination.total_pages // 1' "$tmp_response")
mapfile -t -O "${#collected[@]}" collected < <(
jq -r --arg nid "$PTERO_NODE_ID" \
'.data[] | select((.attributes.node|tostring) == $nid) | .attributes.uuid' \
"$tmp_response"
)
page_count=$(jq -r '.data | length' "$tmp_response")
if [[ "$page_count" -eq 0 && "$page" -lt "$total_pages" ]]; then
echo "[FATAL] Empty page $page of $total_pages. Aborting to prevent data loss."
exit 1
fi
page=$((page + 1))
done
SERVER_UUIDS=("${collected[@]}")
if [[ "${#SERVER_UUIDS[@]}" -eq 0 ]]; then
echo "[FATAL] Zero servers returned from API. Aborting to prevent data loss."
exit 1
fi
_guard_server_count_drop "${#SERVER_UUIDS[@]}"
}Note the re-read of total_pages inside the loop. Servers get created while a scan is running. Caching the page count from the first response means the tail of the list gets dropped, and dropped servers are, again, "orphans".
The zero server guard catches a total API failure. It does not catch a partial one, which is the dangerous case: a panel mid migration returns 900 of 1200 servers, and 300 real volumes get marked for deletion. So there is a second guard that remembers the last successful count:
_guard_server_count_drop() {
local current="$1"
local state_file="$STATE_DIR/server-count-$PTERO_NODE_ID"
if [[ -f "$state_file" ]]; then
previous=$(cat "$state_file" 2>/dev/null || echo 0)
if [[ "$previous" =~ ^[0-9]+$ ]] && [[ "$previous" -gt 0 ]]; then
threshold=$(( previous * (100 - MAX_COUNT_DROP_PCT) / 100 ))
if [[ "$current" -lt "$threshold" ]]; then
if [[ "${TITAN_HALO_ALLOW_COUNT_DROP:-0}" == "1" ]]; then
echo "[WARN] Server count dropped $previous to $current; continuing on override."
else
echo "[FATAL] Server count dropped $previous to $current, below the ${MAX_COUNT_DROP_PCT}% guard."
echo "[FATAL] Aborting. If this drop is real, re-run with TITAN_HALO_ALLOW_COUNT_DROP=1."
exit 1
fi
fi
fi
fi
mkdir -p "$STATE_DIR"
echo "$current" > "$state_file"
}It is a heuristic, and it will occasionally block a legitimate mass cleanup. That is the correct trade. The override exists, it is one environment variable, and it re-baselines on the next successful run.
The general lesson, which I now apply to every integration we write: an API client that computes a difference must treat an incomplete response as an error, never as an empty set. A partial read that looks like a valid read is how automation deletes things.
No Terminal? No confirmation
The second serious bug lived in the destructive scripts themselves.
delete-orphan-volumes.sh was written to be interactive. It found orphans, printed them, and prompted for confirmation. The prompt was skipped when there was no TTY on stdin, on the reasonable sounding logic that you cannot prompt a cron job.
Then we added a REST API. And a job engine. Both of which invoke scripts with no TTY.
So the code path was: no terminal, therefore no prompt, therefore proceed. Every automated call deleted immediately, with no confirmation, no preview, and no diff. "I cannot ask you" had been implemented as "you said yes".
The fix required an automated caller to prove it had seen a dry run, not merely that it wanted to run:
# DESTRUCTIVE. Reports only unless --apply is passed.
#
# This script used to treat "no terminal on stdin" as consent, so every call
# from the API and the job engine deleted without a confirmation. A
# non-interactive --apply now additionally requires --expect-count N and aborts
# unless exactly N orphans are found, so an automated caller has to have seen a
# dry run and be acting on the number it reported.
# The only way to delete without a human at the keyboard.
if [[ "$APPLY" == true && ! -t 0 && -z "$EXPECT_COUNT" ]]; then
echo "[FATAL] --apply has no terminal to confirm on and no --expect-count."
echo "[FATAL] Run find-orphan-volumes.sh first and pass the count it reported."
exit 2
fiThe script also emits a machine readable count so the dry run and the apply run speak the same language:
echo "[ORPHANS] ${#ORPHANS[@]}"The API enforces the same contract, so nothing destructive is reachable by accident through any path, including the job engine:
function destructiveArgs(scriptKey, body) {
const cfg = DESTRUCTIVE_SCRIPTS[scriptKey];
if (!cfg) return { args: [] };
const { confirm, expectedCount } = body || {};
if (confirm !== true) {
return {
status: 400,
error:
`${scriptKey} deletes data and requires { "confirm": true, "expectedCount": N }. ` +
`Run GET /${cfg.dryRun} first and pass the number of orphans it reported.`,
};
}
if (!Number.isInteger(expectedCount) || expectedCount < 0) {
return {
status: 400,
error:
`"expectedCount" must be a non-negative integer, the number of orphans ` +
`GET /${cfg.dryRun} reported. The run aborts if the count no longer matches.`,
};
}
return { args: ["--apply", "--expect-count", String(expectedCount)] };
}The nice property of expectedCount is that it is not just a confirmation flag. It is a compare and swap on the state of the world. If anything changed between the dry run and the apply, whether a server was deleted, a server was created, or the panel is now returning different data, the count no longer matches and the run aborts. A stale confirmation cannot be replayed against a world that has moved on.
Gigachad Jobs vs Short HTTPS Runs
A full abuse scan on a busy node takes minutes. For example, Dono-01 at it's peak can handle nearly 3000 servers. That's 3000 docker containers, folders, and many more subfolders. Every HTTP client in the path, browsers, reverse proxies and fetch defaults alike, disagrees with that. Our first version just used long timeouts, and the result was requests dying halfway through a scan with no way to see progress or recover the output.
So script execution got decoupled from result reading. POST /jobs returns an id immediately. The caller polls or streams.
// --------------- Job Engine ---------------
// Scripts on a node with thousands of servers run for minutes, which is far
// longer than any sane HTTP timeout. Jobs decouple *running* a script from
// *reading* its result: POST /jobs returns immediately with an id, and the
// caller polls GET /jobs/:id for status, progress and output as it is produced.Two details in there took real debugging.
Cancelling has to kill the whole process tree. A bash script spawns find, grep, curl, and jq. Killing the bash wrapper leaves the children running, so a "cancelled" scan carries on hammering the disk while the UI happily shows it as stopped. The job is started in its own process group and the whole group is signalled:
// Bash scripts spawn children of their own (find, grep, curl). Running the job
// in its own process group means cancelling kills the whole tree, not just the
// bash wrapper, otherwise a cancelled scan keeps hammering the disk.
function killJob(job, signal) {
if (!job.child) return;
try {
process.kill(-job.child.pid, signal);
} catch {
try {
job.child.kill(signal);
} catch {
/* already gone */
}
}
}
// A script that ignores SIGTERM (or whose children do) would otherwise keep
// scanning after being "cancelled", so escalate if it has not exited.
function cancelJob(job) {
const child = job.child;
killJob(job, "SIGTERM");
if (!child) return;
setTimeout(() => {
if (job.child === child) killJob(job, "SIGKILL");
}, JOB_CANCEL_GRACE_MS).unref();
}The negative PID in process.kill(-pid, signal) is the whole trick, and the .unref() on the escalation timer stops a pending SIGKILL from holding the event loop open.
Concurrency has to be refused, not queued. Two simultaneous scans of the same volumes thrash the disk and race on the same log file, so a second request gets a 409 along with the job that is already running:
const running = findRunningJob(scriptKey);
if (running) {
return { error: `${scriptKey} is already running`, status: 409, job: running };
}Returning the running job in the error body means a client that lost track of its job id can attach to the existing one instead of failing.
Progress reporting is a tiny text protocol. Scripts write TH_PROGRESS lines to stderr, the engine parses them out of the stream, and they update a progress field instead of becoming output:
# In detect-abuse.sh, every 50 volumes:
echo "TH_PROGRESS current=$CURRENT total=$TOTAL stage=\"scanning volumes\"" >&2function parseProgressLine(line) {
const match = line.match(/^\s*TH_PROGRESS\s+(.+)$/);
if (!match) return null;
const progress = {};
const fields = /(\w+)=(?:"([^"]*)"|(\S+))/g;
let field;
while ((field = fields.exec(match[1])) !== null) {
const key = field[1];
const value = field[2] !== undefined ? field[2] : field[3];
if (key === "current" || key === "total") {
const num = parseInt(value, 10);
if (!Number.isNaN(num)) progress[key] = num;
} else {
progress[key] = value;
}
}
return Object.keys(progress).length > 0 ? progress : null;
}Any script can opt into a progress bar by printing one line. No shared library, no IPC, no coupling. Adding progress to a new script is a one line change.
Streaming without breaking my browser
My browser sucks. Most do, they don't have the correct features needed, but I digress.
Polling worked, but watching a scan through a 2 second poll is a miserable experience. So jobs gained a WebSocket stream, added as an optimisation rather than a replacement, because nodes update independently and an older node has to keep working:
// --------------- Job Streaming ---------------
// Subscribers receive the job's current state on connect, then deltas as the
// script produces them. Polling GET /jobs/:id remains fully supported, this is
// an optimisation, not a replacement, so an older client keeps working.The first version sent one frame per output line. A chatty scan produces thousands of lines in a burst, which is thousands of WebSocket frames, and the browser tab is the thing that suffers. Output is now coalesced on a short timer:
const JOB_STREAM_FLUSH_MS = 100;
function flushJob(job) {
if (job.flushTimer) {
clearTimeout(job.flushTimer);
job.flushTimer = null;
}
if (job.pendingLines.length > 0) {
broadcast(job, {
type: "log",
lines: job.pendingLines,
nextIndex: job.firstLineIndex + job.lines.length,
});
job.pendingLines = [];
}
if (job.progressDirty) {
broadcast(job, { type: "progress", progress: job.progress });
job.progressDirty = false;
}
}
function scheduleFlush(job) {
if (job.flushTimer || job.subscribers.size === 0) return;
job.flushTimer = setTimeout(() => {
job.flushTimer = null;
flushJob(job);
}, JOB_STREAM_FLUSH_MS);
}scheduleFlush returning early when there are no subscribers matters more than it looks. Without it, a cron triggered job with nobody watching still schedules a timer for every line it produces.
The other correctness detail is what a client gets on connect. A subscriber that only receives deltas has no idea what happened before it arrived, so it opens with a full snapshot and only then receives deltas:
function attachSubscriber(socket, job) {
job.subscribers.add(socket);
// Full current state up front, so a client that connects late, or reconnects
// after a drop, is immediately consistent.
socket.send(JSON.stringify({ type: "snapshot", job: serializeJob(job) }));
if (job.status !== "running") {
socket.send(JSON.stringify({
type: "status",
job: serializeJob(job, { includeLines: false }),
}));
socket.close(1000, "Job finished");
job.subscribers.delete(socket);
return;
}
// ...
}Reconnect after a dropped connection is therefore the same code path as first connect. There is no separate resume logic to get wrong.
Dashboard made out of glass?
Desert Storm is a Next.js App Router application. It currently covers Pterodactyl users, servers and nodes, Paymenter donations, Discord moderation and message building, a ticket system, mail search and sending, IP management, VirtFusion, a database explorer, and a page per Titan Halo capability with a per node drill down.
Access control is a flat permission list with roles composed from it, seeded into the database:
export const PERMISSIONS = [
'dashboard.view',
'pterodactyl.users.view',
'pterodactyl.users.manage',
'pterodactyl.servers.view',
'pterodactyl.servers.manage',
'pterodactyl.servers.destructive',
'discord.moderation.manage',
'ip_management.view',
'ip_management.manage',
'titan_halo.view',
'titan_halo.manage',
'system.db_explorer.view',
'members.manage',
'roles.manage',
'audit.view',
// ...
] as const;
export type PermissionKey = (typeof PERMISSIONS)[number];Note pterodactyl.servers.manage and pterodactyl.servers.destructive being separate. Suspending a server and deleting one are different actions with different consequences, and support staff need the first without the second.
The important rule, learned the hard way in an earlier iteration: permissions are enforced in the API route, not in the component that renders the button. Hiding a button is a user experience decision. Every route calls requirePermission and returns before doing anything else.
Multi node views fan out server side, so the browser makes one request no matter how many nodes exist, and one dead node does not take down the page:
const results = await Promise.allSettled(
nodes.map(async (node) => {
const result = await titanHaloFetch(node.nodeId, '/version');
if (!result.success) {
return { nodeId: node.nodeId, version: null, online: false };
}
const data = result.data as { version?: string } | undefined;
return { nodeId: node.nodeId, version: data?.version ?? null, online: true };
})
);
// Seed every configured node as offline first, then overwrite whatever answered.
// A node that never responds still appears in the response, as offline, rather
// than silently vanishing from the fleet view.
const versions: Record<number, NodeVersionInfo> = {};
for (const node of nodes) {
versions[node.nodeId] = { version: null, online: false };
}
for (const result of results) {
if (result.status === 'fulfilled') {
const { nodeId, version, online } = result.value;
versions[nodeId] = { version, online };
}
}Promise.allSettled rather than Promise.all, and pre-seeding the result map, means an unreachable node shows as offline instead of disappearing. A node that vanishes from a fleet dashboard is worse than a node reported as down, because nobody investigates something they cannot see.
Accidentially exposing data?
Desert Storm talks to nodes through a catch all proxy route: /api/titan-halo/[nodeId]/[...path]. It attaches the node's bearer token and forwards.
The original version forwarded the path and query string verbatim to any user holding titan_halo.view. Read that again with the token in mind. What it actually exposed was not "the endpoints the dashboard uses". It was every route the node has now, plus every route it ever gains in future, including anything added for local or operator use, to everyone holding the most basic view permission.
There was a second, subtler problem. Titan Halo serves cached script results on GET /monitoring/detect-abuse, and re-runs the script when you add ?refresh=true. Permissions were being derived from the HTTP method, so that request was treated as a read. It is not a read. It starts a multi minute, disk heavy scan on a production node. It is a mutation wearing a GET.
The fix was an explicit allowlist module that decides both questions:
/**
* Allowlist for the Titan Halo node proxy.
*
* Two things are decided here:
* - whether a method + path is forwardable at all, and which query params may
* travel with it (anything else is dropped rather than passed along);
* - which permission it needs, which is not always implied by the method.
* `GET /<script>?refresh=true` runs the script on the node, so it is a
* mutation wearing a GET, and is gated on `titan_halo.manage` like one.
*/
type Rule = {
pattern: RegExp;
permission: PermissionKey;
/** Query params allowed through. Any param not listed is dropped. */
allowedQuery?: readonly string[];
/**
* Params that turn the call into a mutation. Present with a truthy value,
* the rule's permission is raised to `titan_halo.manage`.
*/
escalatingQuery?: readonly string[];
};
const GET_RULES: Rule[] = [
{ pattern: /^\/version$/, permission: 'titan_halo.view' },
{ pattern: /^\/health$/, permission: 'titan_halo.view' },
{
pattern: new RegExp(`^/jobs/${UUID}$`),
permission: 'titan_halo.view',
allowedQuery: ['since'],
},
// Cached scan results. `refresh=true` re-runs the script on the node.
{
pattern: new RegExp(`^/(${NODE_SCRIPTS.join('|')})$`),
permission: 'titan_halo.view',
allowedQuery: ['refresh'],
escalatingQuery: ['refresh'],
},
];The forwarded URL is rebuilt from allowed parameters rather than passing the incoming query string through, so an unlisted parameter cannot ride along on an otherwise valid request.
Because the patterns are built by joining script key constants into alternations, a future script key containing a regex metacharacter would silently change what those patterns match. That is checked at import time, so the process fails to start rather than quietly widening the allowlist:
const SCRIPT_KEY = /^[a-z0-9-]+\/[a-z0-9-]+$/;
for (const key of [...NODE_SCRIPTS, ...NODE_ACTIONS]) {
if (!SCRIPT_KEY.test(key)) {
throw new Error(`Invalid Titan Halo script key: ${key}`);
}
}The route then gates on view first and escalates only after, so an unauthorised caller cannot use error responses to map which paths the allowlist accepts:
async function authorize(request, params, method) {
const session = await requirePermission('titan_halo.view');
if (session instanceof NextResponse) return session;
const decision = resolveProxyRoute(method, path, request.nextUrl.searchParams);
if (!decision.allowed) {
return NextResponse.json({ error: decision.error }, { status: 400 });
}
if (decision.permission !== 'titan_halo.view') {
const escalated = await requirePermission(decision.permission);
if (escalated instanceof NextResponse) return escalated;
}
return { nodeId, path: decision.path, session };
}Every mutating call through the proxy is also written to the audit log, with the action derived from the path rather than the route, because the proxy is generic and "updated a node" should not read the same as "started a scan":
function nodeAction(path: string) {
if (path === '/update') return 'titan_halo.node_updated' as const;
if (path === '/restart') return 'titan_halo.node_restarted' as const;
if (path === '/jobs') return 'titan_halo.job_started' as const;
return 'titan_halo.action_run' as const;
}The generalisable lesson: a proxy that attaches credentials is an authorisation boundary, and its default must be deny. Any pass through proxy is really a promise that every current and future endpoint behind it is safe for whoever can reach the proxy. That is not a promise anyone can keep.
For live job output, the browser never opens a WebSocket to a node at all. Desert Storm opens the wss connection server side, where the certificate is validated and the token lives, and republishes frames to the browser as Server Sent Events:
/**
* Server-Sent Events bridge for a Titan Halo job stream.
*
* The browser never talks to a node directly. This opens the wss connection
* server-side, where the node's certificate is validated and the node's bearer
* token lives, and republishes each frame to the browser as SSE.
*
* Progress is one-directional, so SSE is a better fit than a second WebSocket:
* it needs no extra server, and EventSource authenticates with the existing
* session cookie.
*/
export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';Two things in that bridge are easy to forget and annoying to diagnose. Aborts have to propagate, or every page reload during a long scan leaks a socket to the node:
request.signal.addEventListener('abort', () => {
try { socket.close(); } catch { /* already gone */ }
finish();
});And proxy buffering will happily defeat the entire feature, delivering your "live" stream in one lump at the end:
headers: {
'Content-Type': 'text/event-stream; charset=utf-8',
'Cache-Control': 'no-cache, no-transform',
Connection: 'keep-alive',
'X-Accel-Buffering': 'no',
}Sessions hammering the browser.
This one was not a security bug or a data loss bug. It was a slow, invisible tax that only showed up under real usage.
getCurrentSession() runs on every guarded route. Hydrating a session means a user lookup plus a fan out for roles, permissions, and identity providers. A single dashboard page paints from several endpoints in parallel, so one page load was paying that cost five or six times over, for data that had not changed between requests.
Worse, last_login_at was refreshed on every authenticated request. That is a row UPDATE on auth_users for every API call the dashboard makes. At that frequency the column is not information, it is index churn.
Both were fixed by being specific about what actually changes and when:
/**
* Short-lived cache of hydrated session users.
*
* The cache is deliberately small in scope:
*
* - It is per process. A deployment running several instances caches
* independently, which is fine for a window this short.
* - Only *active* users are cached. A suspended or invited user hydrates to
* null, and that is re-checked against the database every request rather
* than remembered.
* - Anything that changes what a user may do invalidates it explicitly, so
* the TTL is a backstop rather than the mechanism. Role and permission
* edits take effect immediately, not after it expires.
*/
const SESSION_CACHE_TTL_MS = 15_000;Those three bullets are the whole design. Caching a negative result is how a suspended account keeps working for another fifteen seconds, so suspended and invited users are never cached. And explicit invalidation is what makes the cache safe to have at all:
/** Drop one user's cached session: status, roles or sessions changed. */
export function invalidateSessionUser(userId: number): void {
cache.delete(userId);
}
/**
* Drop every cached session.
*
* For changes to a role *definition*: editing or deleting a role changes what
* every holder of it may do, and finding those holders would cost more than
* simply rehydrating.
*/
export function invalidateAllSessionUsers(): void {
cache.clear();
}Revoking a role has to take effect now, not in fifteen seconds. The TTL is only there so that a missed invalidation self corrects quickly instead of persisting until restart.
The write amplification got the same treatment, with the write moved off the response path entirely:
/**
* How stale `last_login_at` may get before it is written again.
*
* The column is really "last seen". It was refreshed on every authenticated
* request, which meant a row UPDATE on auth_users for every API call the
* dashboard made. At that rate the write is index churn rather than
* information; a few minutes of resolution says the same thing.
*/
const LOGIN_TOUCH_INTERVAL_MS = 5 * 60 * 1000;Same information, roughly two orders of magnitude fewer writes, and nobody in support noticed anything except that pages got faster.
Migrating to better-auth
When I found better-auth, I knew this was a lifesaver. It did all the heavylifting and it's one of the best products I've used within the Next.js ecosystem.
Desert Storm started with hand rolled authentication and moved to Better Auth: invite only onboarding, database backed individually revocable sessions, proper password reset. Public signup and social login are off entirely, since this is an internal tool.
The interesting part of any auth migration is the passwords. You cannot rehash what you cannot read, and asking every staff member to reset on the same afternoon is not a plan. So verification accepts both formats, and legacy hashes are verified in constant time:
async function verifyLegacyPassword(password: string, hash: string): Promise<boolean> {
const [salt, expectedHex, ...extra] = hash.split(':');
if (!salt || !expectedHex || extra.length > 0 || !/^[a-f0-9]+$/i.test(expectedHex)) {
return false;
}
const expected = Buffer.from(expectedHex, 'hex');
if (expected.length !== 64) return false;
const actual = (await scrypt(password, salt, expected.length)) as Buffer;
return crypto.timingSafeEqual(expected, actual);
}
export async function verifyPassword(data: { hash: string; password: string }): Promise<boolean> {
try {
if (await verifyBetterAuthPassword(data)) return true;
} catch {
// Legacy hashes are not in Better Auth's encoded format.
}
return verifyLegacyPassword(data.password, data.hash);
}The strict validation before the comparison is doing real work. timingSafeEqual throws on a length mismatch, so a malformed stored hash would otherwise turn a failed login into a 500. And it is crypto.timingSafeEqual rather than === because comparing secrets with early exit string equality leaks information through timing, even in a case as unglamorous as a legacy fallback path.
The migration keeps replaced tables as timestamped legacy_auth_* tables rather than dropping them, and the DB Explorer redacts password hashes, invite token hashes, and anything that looks like a secret or token column. An internal tool that shows you your own password hashes is an internal tool with a much larger blast radius than it needs.
Automated fleet updates
Updating Titan Halo by hand across every node does not scale, so nodes expose POST /update, and Desert Storm has an update manager page that shows every node's version and can update them.
Remote self update is one of those features where the failure mode is severe and specific: a node that boots into a crash loop is a node with no remote recovery path. You are now driving to a console, or opening a ticket with a data centre, over a version bump.
The ordering in that endpoint is entirely load bearing:
// Ordering is load-bearing. The restart is the point of no return, so the new
// code's dependencies must be installed AND verified before it happens. Any
// failure rolls the working tree back to the commit we started from, so a
// failed update leaves a node that still works rather than one that boots into
// a crash loop with no remote recovery path.Each decision in it exists because the alternative fails badly:
// Resolve the tracked upstream explicitly. A detached HEAD fails loudly here
// rather than silently pulling from whatever happens to be checked out.
const upstream = await runCommand(
"git rev-parse --abbrev-ref --symbolic-full-name @{u}",
{ timeout: GIT_TIMEOUT_MS }
);
// reset --hard rather than pull: a merge conflict would write conflict markers
// into server.js and the node would never boot again. Local edits to tracked
// files are discarded by design; config.json and logs/ are gitignored and
// survive untouched.
const reset = await runCommand(`git reset --hard ${upstream.stdout}`, {
timeout: GIT_TIMEOUT_MS,
});git pull on an unattended node is a genuinely bad idea. A conflict writes <<<<<<< markers into a JavaScript file, and the next restart is a syntax error on a machine you can only reach through the process that just died.
Install and rollback are asymmetric on purpose:
// npm ci installs the exact locked tree and removes anything extraneous, so
// added, changed and removed dependencies all apply. It also refuses to run if
// package.json and package-lock.json disagree, turning a silent half-install
// into a loud pre-flight failure.
const install = await runCommand("npm ci --omit=dev", { timeout: NPM_TIMEOUT_MS });
const rollback = async () => {
const back = await runCommand(`git reset --hard ${prevSha}`, { timeout: GIT_TIMEOUT_MS });
// Deliberately npm install, not npm ci: ci deletes node_modules before
// installing, so if it were the failure being recovered from, using it again
// could leave the node with no dependencies at all. install is
// non-destructive and rebuilds from the restored lockfile.
const reinstall = await runCommand("npm install --omit=dev", { timeout: NPM_TIMEOUT_MS });
return back.returncode === 0 && reinstall.returncode === 0;
};npm ci on the way in, npm install on the way out. ci wipes node_modules before installing, so using it in the recovery path means a failing recovery can leave the node with no dependencies at all, which is strictly worse than the state you were recovering from. Recovery paths must be non-destructive even when they fail.
And npm exiting zero is not proof the new code can load, so there is an explicit verification step before the point of no return:
// npm exiting 0 is not proof the new code can actually load.
const missing = verifyDependencies();
if (missing.length > 0) {
const rolledBack = await rollback();
return res.status(500).json({
success: false,
error: `Dependencies missing after install: ${missing.join(", ")}`,
rolledBackTo: prevSha,
rolledBack,
});
}Finally, the restart happens after the response has actually flushed, rather than after a guessed delay:
// Restart once the response has actually flushed, rather than guessing.
res.on("finish", restartSelf);A setTimeout based restart is a race: sometimes the caller gets its confirmation, sometimes the connection dies mid write and the operator has no idea whether the update applied. Cron sync failure is reported but not fatal, because a node running old cron entries still works, and a failure there silently stopping all six hourly jobs is exactly the kind of thing that goes unnoticed for weeks.
The smalls things that adds up.
Sometimes, the smallest things in life are the best (wish it was for all things).
The log endpoint takes a category and a script name straight from the URL. A .. segment survives Express's decoding, so path.join can walk out of the logs directory entirely:
function readLog(category, scriptName) {
const logPath = path.join(LOGS_ROOT, category, scriptName, "logs.txt");
// category and scriptName reach this function straight from the URL, and a
// `..` segment survives Express's decoding, so confirm the join stayed inside
// the logs directory before reading. Callers passing a known script's own
// category/name never trip this; it is here so no future caller can.
const relative = path.relative(LOGS_ROOT, logPath);
if (relative.startsWith("..") || path.isAbsolute(relative)) {
return { error: "Invalid log path", status: 400 };
}
// ...
}Every caller today passes a known constant, so this was not exploitable. It is there for the caller that gets added next year by someone who has not read this function. Resolve, then compare against the root, is the only path check that actually works. Substring checks on the raw input do not.
Another one: ioredis 6 negotiates RESP3 by default, which changes reply shapes, which changes how every read in the cache layer parses. One line pins it:
// ioredis 6 negotiates RESP3 (HELLO 3) by default and falls back to RESP2 on
// servers that do not know the command. 'legacy' keeps RESP2-compatible reply
// shapes so every read below parses the same way under either protocol.
replyMapping: 'legacy',That class of bug, a transport level default changing data shapes underneath application code that never asked, is very hard to find from the symptom alone.
Testing, Testing, and, Testing
Titan Halo's parsers are the seam between free form bash output and structured JSON that a dashboard renders. That makes them the highest value thing to test, and the easiest, because a log is just a string.
// Illustrative only. Severities, scores and the threshold below are made up,
// so this fixture does not describe how the real rule set is tuned.
const SAMPLE_LOG = `Abuse Detection Scan
Rules: 2
Generated: Sat Aug 29 20:00:00 EDT 2026
Threshold: 999
================================
[ALERT] severity=high category=example_category score=1234
Directory: /var/lib/pterodactyl/volumes/abc-123
Category: Example Category
Score: 1234
Findings:
- [high] Example Label (npm_dependency): package.json -> example-package
--------------------------------
[SUMMARY] volumes_scanned=1565 alerts=1 threshold=999
Scan complete.
`;
test("parseDetectAbuse extracts scan metadata", () => {
const data = parseDetectAbuse(SAMPLE_LOG);
assert.equal(data.rulesVersion, 2);
assert.equal(data.threshold, 999);
assert.equal(data.volumesScanned, 1565);
assert.equal(data.alertCount, 1);
});No fixtures on disk, no mocked node, no Docker. A string in, an object out. Desert Storm carries the same idea forward: the proxy allowlist, the permission model, the Pterodactyl error mapper, the server list filter, and the session cache all have tests, because they are all pure functions over data. The allowlist tests in particular assert the refusals, not just the approvals, which is the half people usually skip.
What Comes Next?
A few themes run through all of this, and they are not really about Pterodactyl.
Any code that computes a difference and then deletes must treat incomplete input as a fatal error. The orphan scanner is the clearest example, but the pattern shows up anywhere you reconcile two sources of truth.
Absence of a confirmation channel is not confirmation. "No TTY, so proceed" is the same bug as "no signature required for this amount".
A credential attaching proxy is an authorisation boundary. Allowlist it, rebuild the query string, and never let the HTTP method alone decide the permission.
Caches need explicit invalidation, and the TTL is only a backstop. Anything else means a revoked permission stays live for as long as your timeout.
As for what is actually next: quite a lot.
The next major piece of work for Titan Halo is documented internally as Final Frontier, an LLM driven server risk profiling layer that runs alongside the existing rules engine. Rules are precise and fast, and they catch known bad things. They do not catch the thing nobody has written a rule for yet. The plan is a local model doing behavioural profiling over the signals Titan Halo already collects, producing a risk score that sits next to the deterministic score rather than replacing it. Rules stay authoritative; the model gets to raise a hand.
Beyond that there is more on the roadmap for both projects: deeper cross node correlation, richer historical trend data for resource and network usage, more of the manual staff workflow moved into audited dashboard actions, and continued expansion of the detection layers.
Desert Storm and Titan Halo have already changed how DanBot Hosting operates. Abuse that used to be found by someone happening to look is now found on a schedule, scored, and attributed. Cleanup that used to be a risky manual rm -rf is now a dry run, a count, and a confirmation. And the answer to "who did that, and when?" is a query instead of a group chat.
There is a lot more to come from both.