Skip to main content

Constraints

Resource Contention and the Memory Bottleneck on Pear

While the archival mission of Project Codex is a primary objective, it does not exist in a vacuum. The host Pear is a multi-role server, functioning as the central nervous system for a variety of critical services including high-density photo storage, a collaborative Minecraft environment, and a media ecosystem powered by Jellyfin. Additionally, Pear hosts several localized websites and the very BookStack instance documenting this project. While these services—categorized as "Life Management" and "Entertainment"—place relatively light demands on the CPU and the 32TB archival array, they collectively create a severe strain on the system's most finite and precious resource: Random Access Memory (RAM).

The Anatomy of Memory Pressure in a Proxmox Environment

In a Proxmox-based virtualization environment, RAM is the "hard ceiling" that dictates how many concurrent services can operate reliably. Unlike CPU cycles, which can be "over-provisioned" and shared through time-slicing, RAM is a physical allocation that cannot be easily stretched without resorting to "Swap," which would catastrophically degrade performance—especially when dealing with the high-latency I/O of mechanical IronWolf drives.

The resource conflict on Pear is driven by two main factors. First, the ZFS file system used for the 32TB archival array is inherently memory-hungry, as it uses an Adaptive Replacement Cache (ARC) to store frequently accessed data in RAM to speed up performance. Second, Java-based applications like Minecraft and database-heavy services like Jellyfin require dedicated "RAM chunks" that remain locked and unavailable to the archival ingestion scripts. This creates a state of "Memory Contention," where the ambitious data extraction goals of Project Codex must be carefully balanced against the stability of the server's daily-use applications.

Strategic RAM Management: Tuning the ZFS ARC

To ensure that the "Silo Strategy" does not crash the Minecraft server or cause Jellyfin to stutter, we must implement aggressive memory management on Pear. The most effective lever at our disposal is the manual limitation of the ZFS ARC size. By default, ZFS on Proxmox may attempt to claim up to 50% of the host's total RAM for its cache.

For an archival project where we are primarily performing sequential writes and long-term storage rather than frequent "hot-data" reads, we can safely cap the ARC to a much smaller value (e.g., 4GB or 8GB). This "Starvation Strategy" forces ZFS to be more efficient with its memory usage, freeing up critical megabytes for the Mandarin and Quince VMs to perform their ingestion and processing tasks. While this might slightly slow down the "browsing" speed of the photo gallery, it ensures that the system as a whole remains stable under the heavy load of a multi-terabyte Kiwix ingestion.

Lean Virtualization: Containers vs. Virtual Machines

Another method for navigating the memory bottleneck is the strategic use of LXC (Linux Containers) instead of full Virtual Machines wherever possible. While Project Codex utilizes dedicated VMs like Mandarin for storage management, many of the "Life Management" services like the documentation site or small web servers should reside in LXCs.

Unlike a full VM, which requires a fixed, "frozen" allocation of RAM that is reserved even when the VM is idle, LXCs share the host's memory dynamically. If the Minecraft server is empty or the media server is not currently transcoding, that RAM remains available to the archival pipeline. This "Fluid Allocation" is essential for a resource-constrained host like Pear; it allows us to maintain a diverse ecosystem of services while still reserving enough peak memory to handle the high-intensity decompression of OpenAlex datasets or the extraction of large ZIM files.

Architectural Segregation: Decoupling Storage and Intelligence

A fundamental principle of the Project Codex implementation is the strict architectural separation between the Storage Layer and the AI Processing Layer. In a resource-constrained environment like the host Pear, attempting to run high-intensity data ingestion and large-scale AI inference within a single monolithic virtual machine would lead to catastrophic memory contention and system instability. By segregating these functions, we ensure that the "Vault" (the 32TB ZFS array) remains a stable, low-overhead environment for data preservation, while the "Engine" (the AI processing unit) can be isolated, tuned, and even power-cycled independently without risking the integrity of the archival filesystem.

The Intelligence Engine: Leveraging the NVIDIA 5060 Ti

The centerpiece of our AI Processing Layer is the NVIDIA 5060 Ti, a hardware asset of strategic importance. Acquired in early 2026 just before NVIDIA transitioned its production lines away from this specific mid-range architecture, this card represents the "sweet spot" for a high-efficiency home lab. With 16 GB of VRAM, the 5060 Ti provides the critical "Headroom" required to run modern open-weights models—such as Llama 3 (8B) or Mistral—entirely within GPU memory.

In the context of the "Silo Strategy," VRAM is a more significant bottleneck than raw compute speed. Having 16 GB allows us to load high-context models that can "read" through entire folders of our extracted Markdown files at once. Because we secured this card before the market shifted toward even more expensive, enterprise-only "AI Data Center" silicon, it serves as a protected legacy asset that allows the Sea Of Fate network to perform local, private inference without relying on the "AI-washed" cloud APIs of the major tech giants.

Implementing GPU Passthrough on Pear

To unlock the full potential of the 5060 Ti within the Proxmox environment, we utilize PCIe Passthrough. This technique allows the host Pear to "detach" the GPU from the physical hardware and hand it over directly to a specialized Virtual Machine (the "Intelligence Engine").

  • The Benefit of Isolation: By giving the AI VM direct access to the 5060 Ti, we bypass the performance overhead of virtualization. The VM treats the card as if it were plugged into a bare-metal motherboard, allowing for near-native speeds during LLM inference and data cleaning tasks.

  • Memory Protection: Because the 5060 Ti has its own dedicated 16 GB of VRAM, the AI's "thinking" process does not steal from the system's already-strained RAM pool. Only the small amount of system RAM required to run the VM's OS is consumed, protecting the memory needed for the Minecraft server and the ZFS ARC.

  • The "Cold Start" Strategy: Since the AI VM is segregated from the storage system, we can keep it powered down during the "Passive Ingestion" phases. When it is time to perform a "Search and Clean" operation on a newly downloaded ZIM extraction, we spin up the Intelligence Engine, perform the task, and shut it back down, preserving both electricity and system resources.

The Storage Layer: A Lean and Permanent Vault

While the AI VM is a high-performance, volatile environment, the Storage Layer is designed for absolute stability and lean resource usage. By keeping the 32TB IronWolf array managed by a separate, lightweight VM (or even at the Proxmox host level), we ensure that the data is always available to the "Life Management" services like Jellyfin or the photo storage system.

This separation prevents a "Logic Crash" in the AI engine from ever affecting the physical disks. If a experimental Python script or an unstable LLM causes the Intelligence VM to hang, the ZFS pool remains untouched and online. This "Firewalling of the Data" is the final structural defense of Project Codex; it ensures that the intelligence we are building can fail, iterate, and evolve without ever endangering the 2025 baseline we have worked so hard to preserve.

The OpenAlex Java Crisis: I/O Bottlenecks and Host Instability

As the project moves into the ingestion of massive scholarly datasets like OpenAlex, we encounter a specific technical conflict between Java-based data processing and the physical limitations of mechanical hard drives. OpenAlex snapshots are distributed as thousands of compressed JSONL files. To make this data usable, we must run an extraction and indexing pipeline—typically powered by a Java Virtual Machine (JVM)—that performs an immense volume of small, random "write" operations.

In a home lab environment, this presents three critical points of failure:

  1. HDD Write Saturation: Mechanical IronWolf drives excel at sequential storage but are fundamentally crushed by the "random I/O" patterns of a Java-based ingestion engine. Attempting to extract 1.6 TB of JSON directly to the ZFS pool causes "I/O Wait" to spike, effectively locking the host Pear and causing services like the Minecraft server to time out.

  2. The ARC vs. RAM Latency Trap: On a Proxmox host, the ZFS Adaptive Replacement Cache (ARC) is designed to shrink when the OS needs more memory. However, in practice, there is a significant "eviction delay." If a Java process suddenly demands 16GB of heap space, the ARC may not release that memory fast enough. This results in an Out-Of-Memory (OOM) event where the Proxmox kernel, sensing a total lack of free RAM, may kill the most resource-heavy process—often the storage VM itself.

  3. The Write-Amplification Spiral: Java's garbage collection and temp-file creation, when combined with ZFS's "Copy-on-Write" (CoW) nature, creates a "Write-Amplification" effect. Every small change in a JSON file causes a new block to be written, further choking the mechanical drives and wearing out the metadata headers of the pool.

The Infrastructure Split: Offloading to Kiwi

To prevent the total system collapse of Pear, the strategy requires Physical Isolation. We have designated a second Proxmox host, Kiwi, specifically to handle the "volatile" and "high-I/O" stages of the OpenAlex ingestion. By moving the Java processing engine to Kiwi, we decouple the "Thinking" from the "Storing."

  • The Staging Method: The raw OpenAlex snapshots are downloaded to Kiwi’s local SSDs. The Java engine then performs the heavy decompression and initial indexing on Kiwi, utilizing its own dedicated RAM pool. This ensures that the memory contention between the ZFS ARC and the JVM remains isolated to a host that isn't running our "Life Management" services.

  • Networked Serialization: Once the data is extracted and cleaned on Kiwi, it is streamed over the network to Pear as a high-speed sequential write. This transforms the "random I/O" nightmare into a "sequential I/O" task that the IronWolf drives can actually handle without stalling the entire system.

  • System Stability: By separating these hosts, we eliminate the risk of a "Cascade Failure." If the OpenAlex ingestion on Kiwi runs out of memory or saturates its disks, the primary vault and the documentation on Pear remain online and responsive.

Managing the Memory Pressure

The "ARC vs. Host" conflict is a known issue in Proxmox 8.x and 9.x. Even with Kiwi handling the processing, the storage host Pear must be tuned to prevent the ARC from becoming a liability.

  • The ARC Hard-Cap: We manually limit the ARC on Pear (e.g., to 8GB) to ensure there is always a "Safety Buffer" of free RAM that the Proxmox OS can claim instantly. This bypasses the dangerous "freeing delay" that occurs when the ARC is allowed to grow to 50% or more of system memory.

  • Proactive Swap Management: We ensure that Pear has a small, high-speed swap partition on the 4TB SSD rpool. This acts as a "pressure release valve," catching small memory spikes without the system needing to resort to killing active VMs.

By acknowledging these hardware-level constraints—the slow seek times of HDDs, the memory greed of Java, and the latency of ZFS ARC eviction—we build a system that is not only vast but stable. We are building a silo that can work hard in the background without making the rest of the network unusable.

Segregation of Duties: Storage, Intelligence, and Processing

To maintain system-wide uptime, we have moved away from a monolithic server design in favor of a decentralized, role-specific architecture. The primary vault on Pear is dedicated to long-term storage and lightweight services, while the heavy lifting of data transformation is geographically and logically isolated.

  • The Intelligence Engine: We have isolated our AI applications to a dedicated Virtual Machine on Pear, utilizing a 16GB NVIDIA 5060 Ti. By using PCIe passthrough, we ensure that the high VRAM requirements of LLM inference remain self-contained. This prevents the "intelligence" layer from competing with the ZFS filesystem for system RAM, while also allowing us to power down the GPU-heavy VM when it is not actively "cleaning" or "querying" the archive.

  • The Ingestion Specialist (Kiwi): Recognizing that massive datasets like OpenAlex rely on memory-intensive Java Virtual Machines (JVM) and high-volume "random" disk writes, we have offloaded these tasks to the host Kiwi. This prevents the "I/O Wait" death-spiral that occurs when mechanical HDDs are saturated by small, rapid writes, ensuring that the primary IronWolf array on Pear only ever sees steady, manageable sequential data streams.

Containerization vs. Virtualization: The Security-Stability Matrix

Our deployment strategy utilizes a mix of Docker containers and Virtual Machines, chosen specifically to balance efficiency with "Fault Containment." While Docker serves as our primary vehicle for running lightweight applications, we maintain strict boundaries to prevent "containment leaks" where a single runaway process could bring down the entire Proxmox host.

  • Docker for Efficiency: We utilize Docker for the majority of our services because it offers a significantly lower memory footprint than LXC or full VMs. By running our web tools and archival managers in Docker, we maximize our available RAM for the ZFS ARC and the Minecraft server.

  • VMs for Hard Isolation: For mission-critical tasks—specifically the Mandarin storage manager and the Intelligence Engine—we utilize full Virtual Machines. This provides a "Hard Boundary" at the kernel level. If a complex Java extraction script on Kiwi or an experimental AI model on Pear suffers a catastrophic memory leak or a kernel panic, the failure is contained within that specific VM. The Proxmox host remains unaffected, ensuring that a failure in the "Project Codex" pipeline never interrupts the "Life Management" services or requires a physical reboot of the server hardware.

The ARC-Host Sync: Preventing the Memory "Death Delay"

A final core tenet of our resource management is the active mitigation of the ZFS ARC latency. Because the Proxmox host can experience a delay between needing memory and the ZFS ARC releasing it, we have implemented a strict memory ceiling. By manually capping the ARC and utilizing high-speed SSD swap on our 4TB rpool, we create a "Pressure Release Valve." This ensures that Pear always has enough immediate, free RAM to handle sudden spikes from the Minecraft server or the Jellyfin transcoder without waiting for the slow eviction process of the archival cache. This careful tuning transforms Pear from a fragile, overextended server into a resilient, multi-host archival fortress.

Summary: Strategic Resource Stewardship and Multi-Host Sovereignty

The reality of the 2026 home lab is that we must operate with a philosophy of "Resource Intelligence." We acknowledge that Pear is a shared environment, and our archival mission must coexist with the memory, compute, and I/O requirements of the existing "Life Management" stack. By implementing a tiered hardware strategy—capping the ZFS ARC to prevent memory exhaustion, offloading high-intensity Java processing to the secondary host Kiwi, and isolating AI inference to a dedicated GPU-passthrough VM—we can successfully build our 32TB silo without compromising the daily stability of our Minecraft, Jellyfin, or web services. This disciplined approach ensures that the "vault" we are constructing remains a robust, background asset rather than a burden that collapses the very infrastructure it is meant to protect.

Back to top