The Problem Nobody Talks About
I spent three decades keeping UNIX systems running in production. Banks, telcos, healthcare - places where downtime meant actual consequences. When I started working with ML systems five years ago, I expected them to be different. They’re not.
The same infrastructure problems that plagued distributed systems in 1997 are plaguing ML systems in 2026. We’ve just renamed them and added GPUs.
Models fail to load. Predictions time out. Memory leaks crash inference servers. Log files grow until they fill the disk. Race conditions corrupt model state. The exact problems we solved in UNIX systems engineering, now dressed up in Python and TensorFlow.
The difference is that most ML teams don’t realize they’re building distributed systems. They think they’re doing “AI engineering” when they’re actually doing systems engineering with models attached.
What UNIX Got Right
UNIX survived because it had principles, not just implementations. These principles emerged from decades of production experience, refined by failure. They’re not sexy. They’re not new. But they work.
Principle 1: Everything Fails
UNIX assumes failure. Processes crash. Disks fill. Networks partition. This isn’t pessimism - it’s realism.
ML systems often assume success. The model loads. The prediction completes. The GPU is available. When these assumptions break, the system has no plan.
UNIX approach: Process supervision (init, systemd). If a daemon crashes, restart it. If it crashes repeatedly, stop trying and alert someone.
ML equivalent: Model serving should assume models fail to load, predictions timeout, and GPUs disappear. Every operation needs a failure path.
def serve_prediction(model_id, input_data, timeout_ms=100):
"""Production serving with UNIX-style failure handling"""
try:
# Try to load model with timeout
model = load_model_with_timeout(model_id, timeout_ms=5000)
# Predict with resource limits
result = predict_with_limits(
model,
input_data,
memory_mb=512,
timeout_ms=timeout_ms
)
return {"status": "success", "prediction": result}
except ModelLoadTimeout:
# Model didn't load in time - use fallback
log.error("model_load_timeout", model_id=model_id)
return {"status": "fallback", "reason": "model_load_timeout"}
except PredictionTimeout:
# Prediction took too long - fail fast
log.error("prediction_timeout", model_id=model_id)
return {"status": "timeout", "latency_ms": timeout_ms}
except MemoryError:
# Out of memory - clear and retry once
gc.collect()
log.warning("oom_retry", model_id=model_id)
# Return cached result or fallback
return get_cached_or_fallback(model_id, input_data)This isn’t paranoia. It’s the same thinking that made UNIX reliable: assume failure, handle it explicitly, fail gracefully.
Principle 2: Observability Through Logs
UNIX philosophy: everything important should emit structured logs. Not pretty dashboards. Not ML-specific “observability platforms.” Just logs.
Logs survive system crashes. Logs can be grep’d. Logs work when your monitoring infrastructure is down. Logs are boring, which is why they work.
(For a deeper dive on what to log in ML systems specifically, see The Observability Gap in ML Systems.)
ML systems often log the wrong things. Training loss curves. Model accuracy. Hyperparameters. These matter for research. They don’t help at 3AM when predictions are failing.
What to log:
# Model loading
log.info("model_load_start",
model_id=model_id,
version=version,
size_mb=model_size)
# Model loaded successfully
log.info("model_load_success",
model_id=model_id,
load_time_ms=elapsed,
memory_mb=memory_used)
# Prediction executed
log.info("prediction",
model_id=model_id,
input_hash=hash(input_data),
latency_ms=latency,
result_hash=hash(result))
# Resource exhaustion approaching
log.warning("memory_pressure",
used_mb=memory_used,
available_mb=memory_available,
threshold_pct=80)Notice what’s not there: model internals, feature importance, gradient norms. Those belong in training logs. Production logs answer operational questions: Did it work? How long did it take? What resources did it consume?
Principle 3: Processes Are Cheap, State Is Expensive
UNIX makes process creation cheap deliberately. Fork, exec, run, exit. No complex lifecycle management. No shared state between processes unless explicitly designed.
ML systems often invert this. They create one long-lived process (a “model server”) that holds expensive state (loaded models) and serves many requests. When that process fails, all state is lost.
UNIX pattern applied to ML:
Instead of one process serving all models, run one process per model. When a model crashes, only that model fails. Other models keep serving.
# Traditional approach - one server, all models
model_server --models=modelA,modelB,modelC
# UNIX approach - separate processes
model_server --model=modelA &
model_server --model=modelB &
model_server --model=modelC &Yes, this uses more memory. Yes, it’s “less efficient.” But it’s more reliable, which matters more in production. Memory is cheap. Debugging cascading failures at 3AM is expensive.
Principle 4: Text Streams Beat Custom Formats
UNIX pipes work because everything speaks text. ps | grep | awk | sort - different tools, same interface.
ML systems love custom formats. Pickle files. TensorFlow SavedModels. ONNX. Each format needs specific loading code. Each format can break in unique ways.
Better approach: Use standard formats for everything outside the model itself.
- Inputs: JSON or Protocol Buffers (parseable, validatable)
- Outputs: JSON (compatible with every tool)
- Configs: YAML or TOML (human-readable, version-controllable)
- Logs: Structured text (grep-able, parseable)
When the model serving process crashes, you can still parse its logs, validate its configs, and inspect its inputs. You don’t need model-specific tools to understand what happened.
Principle 5: Small Tools, Composed
UNIX provides cat, grep, sort, uniq - each does one thing. Composition creates power.
ML systems often build monoliths. One codebase for training, serving, monitoring, and retraining. When any part fails, the whole system is suspect.
Composition in ML:
# Training - writes model to disk
train_model --config=config.yaml --output=model.pt
# Serving - loads model from disk
serve_model --model=model.pt --port=8000
# Monitoring - reads serving logs
monitor_serving --logs=/var/log/serving --alert-threshold=0.95
# Retraining trigger - watches monitoring
retrain_trigger --monitor=localhost:8001 --threshold=0.90Each process does one thing. They communicate through files and logs. When serving breaks, training keeps working. When monitoring breaks, serving keeps working.
Where ML Systems Are Actually Different
UNIX principles apply broadly, but ML systems do have unique characteristics.
Statistical failure modes: A model can be “working” (no crashes) but producing garbage predictions. Traditional systems don’t have this problem - a crashed process is obviously broken. A model serving confidently wrong predictions requires different monitoring.
Resource elasticity: Models can consume wildly different resources for different inputs. A 10-word sentence and a 10,000-word document hit the same endpoint but need different resources. UNIX services tend to have more predictable resource usage.
Versioning semantics: Deploying a new model version isn’t like deploying new code. The interface stays the same but the behavior changes fundamentally. This requires different rollout strategies than traditional deployments.
These differences are real. But they don’t invalidate UNIX principles - they extend them.
Applying This Tomorrow
If you’re running ML systems in production and they’re unreliable, start with UNIX basics:
Week 1: Add structured logging to model serving. Log every load, every prediction, every failure. Make logs grep-able.
Week 2: Add process supervision. If your model server crashes, restart it automatically. If it crashes repeatedly, page someone.
Week 3: Add resource limits. Cap memory usage, timeout predictions, fail fast when resources are exhausted.
Week 4: Isolate failures. Run risky models in separate processes. Don’t let one bad model take down the whole service.
None of this is ML-specific. It’s just systems engineering. The same principles that kept Solaris servers running in 1997 will keep your ML systems running in 2026.
The Unsexy Truth
Production ML systems fail for boring reasons. Disk full. Memory leak. Network timeout. Configuration typo. Process crash.
These are solved problems. UNIX solved them decades ago. We just need to remember the solutions.
AI systems are systems first, AI second. Treat them like systems and they’ll be more reliable than if you treat them like magic.
The principles that kept UNIX running for 30 years still work. They’re just not as exciting as talking about transformers and embeddings. But at 3AM when production is down, you’ll care a lot more about process supervision than you will about attention mechanisms.
Start with the boring infrastructure. Make it reliable. Then add the AI on top. Not the other way around.