Introduction

The Blackwall is a multi-host modular NixOS network. It utilises a single NixOS configuration to manage multiple physical interconnected hosts to distribute processing power, storage capabilities as well as enforce logical segmentation.

Multi-host

The original idea was a single device running everything ¹; having now looked into raid arrays and ZFS configs, I’ve decided it’d probably be better to utilise multiple types of devices to their own strengths

NixOS

I’ve also since become far more profficient with NixOS and believe it is the ‘end-game’ of home server OS options. By managing multiple hosts on a single modular nix configuration, it becomes infinitely easier to treat all hosts as if they’re part of a single interconnected system and brings the logical segmentation much closer to hypervisors like Proxmox, without virtualisation.

Theme

The aesthetic design of the Blackwall network is similar to its namesake. In Cyberpunk 2077, shortly after completing one of your first quests and receiving a new ‘ping’ hack, you can also find a lore book describing how a hacker named Rache Bartemoss released viruses upon the internet causing AI to go rampant and render the internet generally unusable.

In the present-day game, the Blackwall is a ‘firewall’ that protects the new state Intranets from the AIs in the old web.

Today we face a similar problem where due to AI, the general Internet has become unusable due to mass-produced low-effort content ². This server project serves as my efforts to create a closed-off space for my personal info and my data as well as work on my projects in peace.

Myself

I am a British/Irish computer scientist with a BSc (Honours) in Cyber Security from MMU, UK. I have experience working in:

• Web development
• Service Desk
• National CSIRT Analyst

My interests range from motorsports to bikes; I own a Honda Grom and a Mazda MX-5 NB ’03.

1. The original plan was to build everything in a Beelink ME Pro N150. This device is now Cynosure. ↩

2. Dead Internet Theory. ↩

Hosts

This page outlines the hosts within the Blackwall network. Each host serves a different purpose which allows for the most value to be extracted from a device.

Original idea (Cynosure - v1.0)

My original plan was to house all my services and data within a single Beelink ME Pro N150. Utilising the 2x 3.5“ HDD trays, I could mirror my drives allowing for easy-ish expansion and maintance of data. It also housed 3x NVMe slots which would work well for services.

However, since purchasing that device and adding a few services already, I’ve been exposed to new technologies and older overlooked concepts that were worth reconsidering.

New idea

A multi-host modular NixOS config can cordinate multiple completely separate machines, allowing for the best use of their hardware. This means a GPU-heavy device could host LLM software while an enterprise Proliant-style device could hold most the network’s storage capacities and user services. Other future requirements might include a Home Assistant hub and a VM host for security experiments.

Names have already been reserved for these conceptual hosts. Currently, the Beelink’s hostname is blackwall as it follows the afformentioned edition of the config. When there are finally multiple devices on the network, I’ll rename the Beelink to cynosure as planned.

Izanagi - v2.0

With the arrival of a second host, we can begin to split software between control and services. This release will be immense in terms of cost, size, time and patience.

Cynosure

The Cynosure host will take use FOSS software such as Prometheus, Uptime Kuma and Grafana for monitoring the overall status and health of systems and services.

I’ve read about Colmena which supposedly lets you declare the state of multiple machines from a single config instead of having to remote into each node to rebuild. I’m not sure how it works yet but Cynosure is likely the device that would control the deployments of other nodes.

Izanagi

This will be the mass storage and service node and will likely be the largest/heaviest node in the system. Actual specific makes/models shouldn’t be selected until I have the funds to puchase all together.

In order to process so much information at once and store it all safely, I’ll first likely need an rack-style server with enough space for all types of nodes I might have such as a desktop chassis or 19“ server. 18U should be sufficient.

This server will host critical irreplacable data such as passwords and personal information. The minimum launchable node should run some form of RAIDZ (mirror + parity + ZFS pooling). The goal is to reliably maintain important data in an economy where storage drives are increasingly expensive which means double parity may be required.

Tip

Each hostname comes from the world of Cyberpunk 2077. Mikoshi is a data fortress owned by Arasaka and Cynosure was Miltech’s direct response. Izanagi is an access point to Mikoshi while the Black Sapphire is skyscraper within Dogtown.

Architecture

This page details the architecture rules for the Blackwall server and all hosts involved.

Rules

1. Every host will have a single NVMe drive reserved for the OS.
2. Every password, API key or secret is to be stored via sops-nix.
3. Endpoints are kept to the minimum and user access is only via a tailnet ¹.
4. All software is managed and configured decleratively via the NixOS Blackwall config.
5. Changing which host runs what should be decided by which modules it inherits.
6. Deny by default.
7. Every host will broadcast logs and metrics to the Cynosure host.

1. A tailnet is a type of mesh VPN created by Tailscale or Headscale (the self-hosted Tailscale alternative). ↩

Topology

Reserved page for mermaid.js diagram explaining logical structure of the Network.

NixOS

Each node in the network is running NixOS to allow for easier control, admin and deployment over systems and services.

├── flake.lock
├── flake.nix
├── hosts
│   └── blackwall
│       ├── default.nix
│       └── hardware.nix
├── modules
│   ├── core
│   │   ├── secrets.nix
│   │   └── storage.nix
│   ├── development
│   │   ├── code-server.nix
│   │   ├── forgejo.nix
│   │   └── mdbook.nix
│   ├── gaming
│   │   ├── minecraft.nix
│   │   └── playit.nix
│   ├── networking
│   │   ├── caddy-tailscale.nix
│   │   └── tailscale.nix
│   └── service
│       ├── arr.nix
│       ├── books.nix
│       ├── glance.nix
│       ├── homepage.nix
│       ├── navidrome.nix
│       ├── vaultwarden.nix
│       └── vikunja.nix
├── README.md
└── secrets
    └── secrets.yaml

Security

Security auditing will probably be done with claude-code/abliterated Q3.8 with security/nix skills against the NixOS configuration.

Besides that, most security is handled by Tailscale which is a problem but headscale lacks features. Perhaps a sharing extension can be added to tailscale to allow a specific user to join?

Biggest thing is that I lack a static IP.

Networking

See Tailscale section in Security page

Maintenance

This page details any routine actions that must be performed to maintain the server.

Jellyfin - no downgrade mechanism so backup before any updates

Monitoring

SSH

There are 2 SSH keys involved in the Blackwall network.

Installation

mizutani@canto ~> ssh-keygen -t ed25519 -C "mizutani@canto" -f ~/.ssh/id_ed25519
The key fingerprint is:
SHA256:yXsXbb+1lqqgQYxS4NkBAsaDWgwlLj9vidiwnFBjmPs mizutani@canto
The key's randomart image is:
+--[ED25519 256]--+
|B*.o..           |
|+*= + .          |
|=o=o o           |
|o= .. o. .   .   |
|+ o. . oS   . o  |
|o*.+...  .   o . |
|ooE +  .... .   +|
|   .    o...   o+|
|       .   ...oo |
+----[SHA256]-----+

Storage Structure

Appdata vs Content

Content is data such as films or shows used by Jellyfin. Appdata is data such as watch history and user accounts. Neither of these data types are recoverable by a NixOS rebuild and thus must be protected in a way that OS data does not need to be.

Stacks

The Blackwall network groups data by the stack. This means that anything Arr* suite related (Jellyfin, Sonarr, Jellyseerr) is grouped together. Should the stack be moved from one host to another, data such as movies *and* user accounts would be migrated together.

Directory Structure

/data/
├── appdata/                    # ZFS dataset: frequent snapshots
│   ├── arr/                    # Media management stack
│   │   ├── jellyfin/
│   │   ├── radarr/
│   │   ├── sonarr/
│   │   ├── prowlarr/
│   │   └── transmission/
│   ├── git/                    # Development stack
│   │   ├── forgejo/
│   │   └── code-server/
│   └── games/
│       └── minecraft/
│
└── media/                      # ZFS dataset: infrequent snapshots
    ├── library/
    │   ├── movies/
    │   ├── tv/
    │   ├── music/
    │   └── books/
    └── downloads/
        ├── complete/
        ├── incomplete/
        └── torrents/

Classification Rules

Drives

RAID1

Services

Documents

There are many documents in life so document management software is required. This must either be easily interfaced with or connected to the chosen email client for usability. Documents also require signing so will need document signing software. Should likely purchase digital scanner and begin keeping receipts etc for later categorising. Adjacent to this is finance where ‘ActualBudget’ can supposedly connect to Monzo and receive transaction data.

Furthermore, will need to develop infrastructure for the kobo reader caduceus project. This includes some way to obtain comics, manga and books and send them to a physical kobo reader. Similar to this is ‘hammer.ink’ which works for literature writing.

Documentation

The Blackwall is documented via mdBook. This was briefly moved to Docusaurus before being switched back for ease of development.

Tools

I might need a virtual browser at some point so neko could be worth installing. I also wish to eventually expose services from beyond the tailnet so security tools like BunkerWeb could help harden any exposed software. SonarQube could also be helpful for improving code quality. ‘IT Tools’ and SnapOtter could also be useful as well as Uptime Kuma if not already mentioned.

Arr* Suite

Here are some Jellyfin-related software to add to the /arr stack:

• Jellystat - statistics
• Jellyfin rewind - music stats
• Lidarr - music
• Bazarr - subtitles
• Watcharr - watchlist

Nextsuite

For Nextcloud-adjacent apps, there’s:

• Talk
• News
• Office
• Tables

Non-server

This is not for server-based services accessed via a browser but instead for actual software to be run on the client device (elysia, canto etc) It should be noted there is a commitment to adwaita gnome.

Gnome Extensions:

• Dash to dock
• Blur my shell
• Vitals
• App menu maybe
• GSConnect
• OpenWeather
• Clipboard Indicator
• Just Perfection
• Removable Drive menu

Shell:

Chosen shell is fish and chosen terminal is kitty

• cbonsai
• Aliases
• arttime
• broot

Software Register

Cynosure

Izanagi

Future LLM Host

Archive

Education

• iFixit
• WikiBooks
• Wikipedia
• Wikiversity
• Wikivoyage

Technical

• Project Euler
• ArchWiki
• Nix Docs
• PhET Science and Math
• Cryptography Q&A
• Retrocomputing Q&A
• 3D Printing Q&A
• Network Engineering Q&A
• Computer Science Q&A
• Information Security Q&A
• Robotics Q&A
• Server Fault Q&A
• Ebook Q&A

Language

• Japanese Language Q&A
• Chinese Language Q&A
• Writing Q&A
• Anime & Manga Q&A

Nature

• Gardening and Landscaping Q&A
• Biology Q&A
• Earth Science Q&A
• Astronomy Q&A

TED Videos

• TED Worklife
• TED Competition
• TED Life
• TED Careers
• TED Ideas
• TED Wildlife
• TED Countdown
• TED Tech
• TED Work
• TED NASA
• TED AI
• TED Capitalism
• TED Leadership
• TED Conservation
• TED Finance
• TED Farming
• TED Environment
• TED Sustainability
• TED Business
• TED Relationships
• TED Energy
• TED Homelessness
• TED Family
• TED Resources
• TED Fossil Fuels
• TED Community
• TED Democracy
• TED Money
• TED Society
• TED Polution
• TED Fashion
• TED Investing
• TED Youth
• TED Data
• TED Travel
• TED Parenting
• TED Innovation

PGL

(Project Gutenberg Library)

• Science
• Technology
• Agriculture
• Medicine
• English Language
• General Works
• Social Sciences
• Fine Arts
• Geography and Anthropology
• Greek Language and Literature

Survival

• True Prepper
• Medline Plus
• Urban Prepper

Youtube

• Misfits
• Fitz (all channels)
• Arch
• Ahoy
• Barely Sociable
• Jschlatt
• BigT
• SMP Live
• Vsauce & D!NG
• Dani
• Disrupt
• Donut
• Speeeed
• evanf1997
• FranzJ
• FUNKe
• Half as Interesting
• In59Seconds
• iNoToRiOuS
• Internet Historian
• Isaacwhy
• Kento Bento
• Kurzgesagt
• LEMMiNO
• Los Ratones
• Lunch Club
• Michael Reeves
• My Name is Byf
• Nexpo
• Nightride
• NileRed/Blue
• Noodle
• OFFCANNY
• RaccoonEggs
• SovietWomble
• Speedless
• The Group Chat
• The Homie Talk Podcast
• THE SQUIDD
• theweeklyslap
• Vaati Vidya

Hardware

Rack

Hosts

Networking

AI

The Blackwall also plays host to a locally-hosted LLM stack.

The following software is used or involved:

Hardware

aflj

Hosting

Protocols

I think the most important skill is a skill that ‘routes’ requests to the right skill via some menu of sorts as well as adding any undiscovered skills. Could make expansion automatic and easy.

Notes

• List all services from blackwall.cam software list
• List any maintenance routines
• Security hardening
• Decide font scheme for all systems

Jellyfin worklist

• Re-configure with declarable accounts, settings etc
• Add more indexers
• Double check encoding is optimal
• Fonts
• F1/Sports streams? mybunny.tv?

Glance

• RSS Feed
• DNS widget
• Hardware stat widget
• Todo widget