ZebraPipe: A New LocalLLM Orchestrator

ZebraPipe is designed for localLLM batch inferencing. It is open source!  Build or  modify  this with your LLM however you want! We used ChatGPT 5.6-sol working with it all day to build this powerful utility appliance.  It contains a complete supported The Tom's Fork of llama-server! This is bigly - because it offers you lots of compression options like turboquant which can greatly reduce your kv-cache sizes on larger contexts on small GPU's.

Please Note: Currently Uploading Should Be There In About 1 Hour

Supported Cuda Architectures

• All of them: 50, 60, 61, 70, 75, 80, 86, 89, 90, 100, 101, 103, 120a, 121a (Basically everything from a 1080 to a H200!)
• Port 8080 - Main llama-server inference window
• Port 8081 - Engine manager and settings
• Port 8082 - Batch queue handler!

Docker Pull and Run!

• Literally pull and run this with a single command!

docker pull cnmcdee/zebrapipe:latest
docker run -d \
  --name zebrapipe \
  --gpus all \
  --restart unless-stopped \
  -p 8080:8080 \
  -p 8081:8081 \
  -p 8082:8082 \
  -e MODEL_PATHS=/models \
  -v /opt/llm/models:/models:ro \
  -v /opt/llm/config:/config \
  -v /opt/llm/results:/results \
  cnmcdee/zebrapipe:latest

192.168.1.<your ip>:8080  # A standard llama-server 
192.168.1.<your ip>:8081  # A LLM Settings and Reload Manager
192.168.1.<your ip>:8082  # A prompt Queue Manager

Copy all Your Models

• Simply copy all your gguf model files to a created directory: /opt/llm/models/ Easy!
• Goto 192.168.1.<your ip>:8081 hit RESCAN MODELS

• Wait about 10 seconds - it takes a bit because it is interrogating the models, they will show up in the Discovered models
• Pick a model and load it

Port 8080 (llama-server Window)

• Will have your standard llama-server window. Note we are using the powerful Turboquant enabled The Tom's Version Fork!

Port 8081 (LLM Engine Manager)

• Is your Engine Manager, it will scan the mounted directory for all your models:

Of importance you can add all your custom options:

While monitoring the llama-server outputs at the bottom:

Port 8082 (Job Queue  Manager)

• You can enter numerous jobs and have them batched over night
• You can also have another LLM write up a json of tasks, and simply drag-n-drop them!

As they work you can watch the jobs

Bulk Export Results

• You can bulk export the results in one shot in the format you choose

Restarting is a Breeze

docker restart zebrapipe

Full Source Code

• Simply unpack this in a directory and run the build-stage.sh it will incrementally build this in pieces. It takes a Ryzen 9 3900 about an hour to build it so:

Build it in Pieces using BuildKit

• This builds using docker moby/buildkit So if you are modifying it you do not have to rebuild the entire system but can build it in pieces!

./build-all-stages.sh
# Or to build one stage:
./build-stage.sh <1 - 10>

Conclusion

This can become a powerful supporting tool for local batch inferencing jobs that can run all night.