Firstly expectations. What do we define as a 'workhorse' and an 'assistant,' well:

• Assistant LLM - Models that run inside a 8-12GB Graphics card and can do powerful lookups and contexts typical under or up to 64K.  These run great inside a 3060ti or a 3080. They use MoE (Mixture of Experts) a non-dense type model.
• Workhorse LLM - Models that run inside a 12-24 GB Graphics card and handle the large contexts (typically up to 256K), and or are capable of working all night without overflow by breaking up the job with Hermes or LLMQP.  They run great inside a 3080 / 4080 / 3090 and higher graphics card.
• Production LLM - Models that run on multiple 3090ti's, DGX Sparx, and Commercial equipment.  That term is debatable as many would only consider push-button get-application like Claude or Fable to qualify.

Advantages

• There are massive advantages to going to a LocalLLM. Firstly your proprietary data will not end up farmed into someone elses next token model
• Sensitive data is retained such as doctors, lawyers and client-privelege scenarios.
• Inventions and NDA type materials do not need to risk being harvested by the cloud LLM.
• This interview shows how concerned corporate is about going with a private LLM.

Let's start at the most basic actually useful LLM we can build < $500.

Starting at the Very Bottom - A Epyc 3151 Build (15,000 Specmark for $61). Then Add a Used GPU of FBM.

• 4 cores / 8 threads / 2.9 Ghz and a PCIe 3.0 16 GB/s for the cost of a Raspberry Pi! What's not to love!
• Add up to 256 GB DDR4 RAM.

Next we want to add a low-cost GPU. But the key here is to find the first models that have TensorCores, and also has 12 GB VRAM.  This is key, you can absolutely get by on the 8GB VRAM - for an assistant, but if you want to get to a 'workhorse' you want to try to find a 12 GB or higher VRAM.  16 GB VRAM like the 4080 has will be preferable, however it looks like their current price at the time of this writing has pushed up their prices significantly into the $1200 range.

You can get capable 8GB models and we proved it.  Naturally however we find that once you start sending large code bases the desire for the larger context > 128K becomes immediate.

Game Changer! Crash-Out! Good Production on a Ryzen 5 2600 (6-core/12thread AMD) w 3060ti/8GB VRAM

Crash-Out! Good Production on a Ryzen 5 2600 w 3060ti/8GB VRAM. We showed you can actually get very powerful productive capability on a 3060ti!

Hot Configthinkmeltprotonmail.com

If you get to the 12 GB model or higher - it then allows you to run large contexts. If you can find a 16GB (I know they are getting more expensive by the day) you can run the 128-256K Contexts. Turboquant is really key here.

Here is a table of possible configurations for lower-end compute. The key here is that they all have the Tensor Cores, and that is going to make a dramatic performance boost.

Keys are in the Tensor Cores, Problem is in the PciE Speed.

• 8 -9 B sized models will run inside a 8 GB model. We showed they are very capable with these kinds of setups. Sparse models lowered the active parameter counts and they can run really well on mixed layer loading. This is where some models load to the CPU and some load to the GPU.

9B Powerhouse? We look at Qwythos-9B-Claude-Mythos-5-1M-GGUF. 70-95T/s on a 4080. A LLM Boosters Dream Build.

We take a look at Qwythos and definitely were impressed!

Hot Configthinkmeltprotonmail.com

We also showed if you can find a 16 GB VRAM or higher and you have a TensorCore enabled card - you can get incredibly good 'Workhorse' level results. For instance:

Qwopus-3.6-35B-A3B-Coder Review. A Powerful LocalLLM Tuned for Coders. It is one of the best 35B sized models we have ever reviewed. Seriously.

Qwopus-3.6-35B-A3B-Coder is a localLLM dream for software developers. Incredibly accurate and powerful prompt processing!

Hot Configthinkmeltprotonmail.com

Why Was the 3090 Dual with NVLink so Popular over say Dual 4080's?

• Anywhere the model was split (and was a 'dense' model) it required massive bandwidth between the layers. If the model had to communicate over a PCIe 3.0 bus even at 8 GB/s bandwidth (16 GB/s) it would kill the model speed.  The NVLink was incredibly popular because it enabled up to 112 GB/s.  The only other option is to try to find a model that fit completely inside the GPU.
• The other option naturally was anything with a unified memory which implied that the RAM of the device shared with the GPU, and brought incredible speeds.

People Started Trying to Run LLM's on Everything

Even a old mining rig the BC-250 because of it's very fast unified memory speeds of 448 GB/s plus the fact that it sells for less than a single 3060ti made them very good candidates for the 8B platform.

Loktar 🇺🇸 (@loktar00) on X

This is actually CRAZY!!! Using llama.cpp RPC I have 2 BC-250’s setup so far, they’re able to run Qwen 27b at Q4, and 35b at Q4 as well. This is without extra CUs unlocked: Qwen 27b with MTP - 14.5 tk/sQwen 35b with MTP - 47 tk/s For $300 I’m getting these speeds! This is

X (formerly Twitter)Loktar 🇺🇸

Exploring the Older Nvidia Tesla / Quadro Etc.

• People even recompiled various LLM's so that they could run somewhat quickly on very older hardware, like the K80, and reported 11 Token/s (showing again even industrial older cards did not hold up as well as any cards that had Tensorcore.)

Chicken Little (@chkn_little) on X

@LottoLabs he wrote me today that he’s running Qwen3.6-35B-A3B-UD-IQ4_XS on the K80 using llama.cpp at 10 tps can a add the k80? I think it’s worth an entry

X (formerly Twitter)Chicken Little

The DGX Spark (Nvidia's LocalLLM Solution)

• At the time of this writing they haul heavy in the price tag - and the market demand confirms why. At $4679 on amazon.com their compute capability comes in at a whopping 1 PFLOP (1000 Trillion Floating Point Operations/s) + 20 core main compute + 128GB LPDDR5X. Because of their unified bandwidth they can send back and forth to the memory and the GPU without requiring going out to a PCIe lane, and this is fast coming in at 273 GB/s.  Please note B and b matters - if you are talking 10 Gbit - that sounds fast, but divide it by 8, then it comes in at 1.25 GBytes/s.  This is why nobody is running inference over the internet - well at least not yet..
• They could be interconnected at very high speeds allowing very large distributed models to split between them. The speed of the interconnect of the ConnectX-7 coming in at 200 Gb/s (Giga-bit/s)

The MAC Mini Pro Weighed in

• On the Macintosh side - due again the very fast interconnection capabilities between machines allowed for stacked units to run very large models quickly and capably.
• Demand was so high there were literally none for sale on Amazon with only second-hand sellers moving them on ebay.
• We do not really cover them simply in that they are not even available. 3-4 of them will cost as much as commerical H100 units etc.

Bus BandWidth Was Always What  Really Mattered

We can see that until PCIe 6.0 we won't even begin to match the speeds of the NVLink, consider the following chart.

The bandwidth always significantly mattered. Tools existed exactly for diagnostic testing, and If you want to get into the meaty details of measuring GPU-to-GPU bandwidth, PCI-e-to-GPU, GPU-to-Itself you can compile and install the mbw tool:  This is where you find out what stuff really can do - not the brochure sale..

Linux/LLM Speed Bandwidth Testing mbw / nvbandwidth

The #1 limiting factor if your LLM will run quickly or not is memory bandwidth. Memory Bandwidth of your RAM.Memory Bandwidth of your GPU.How quickly data can get between the GPU and the CPU over your very slow PCIe bus.Memory Bandwidth of your RAM. In Linux we

Hot Configthinkmeltprotonmail.com

This is where many people then sought out 'server' level motherboards in that they had the proper PCIe 4.0 x 7 such as the TRX40.  By getting up to 4.0 they could theoretically get better performance or run 4-8 card configurations. Used Epyc Motherboards were a good fit.

Re-tuned Moe + MTP + TurboQuant + MCP Was the Game Changing Solution

People started looking at all kinds of ways to speed things up, make them smaller, make them faster, and or make them more accurate.  What were all these different additives? Well when they all were added up they made a dramatic and powerful set of tools that reduced cache size, increase speed, and enabled tooling.

MoE - Mixture of Experts.  This allowed sparse models to have much less active parameters.  By doing this it greatly reduced the CPU/ GPU load, and allowed for the offloading and powerful fitting llama.cpp configurations such as:

--override-tensor "\.ffn_.*_exps\.weight=CPU" \

TurboQuant - kv_cache expanded quadratically, long contexts would destroy the VRAM and kept the hardware requirements very expensive. A full github and guide to install it.

MCP - Model Context Protocol allowed for powerful tool calling. The right tool calling allowed for LLM's to correct their work. Over and over until they get it right. Here are ten+ powerful MCP's that are all opensource!

MTP - Multiple Token Prediction came next. By running multiple parallel heads of certain layers it could see significant speedups in some instances up to 200%!

Re-Tuned Models - By taking strong base models and carefully applying token layers people were able to get significant boosts in performance.  

The Results - You are the winner

It created models that were  so powerful, capable and run on minimal hardware that the industry has not even fully realized how capable a localLLM is. Consider a 35B that can one-shot an entire Asteroids game, accurately, and cleanly and run on a 4080, it is right here. Again it only works because all of the above features are working at the same time.

Qwopus-3.6-35B-A3B-Coder Review. A Powerful LocalLLM Tuned for Coders. It is one of the best 35B sized models we have ever reviewed. Seriously.

Qwopus-3.6-35B-A3B-Coder is a localLLM dream for software developers. Incredibly accurate and powerful prompt processing!

Hot Configthinkmeltprotonmail.com

Or we show again how you can run capable models inside a 3060ti inside a 8GB VRAM.

Game Changer! Crash-Out! Good Production on a Ryzen 5 2600 (6-core/12thread AMD) w 3060ti/8GB VRAM

Crash-Out! Good Production on a Ryzen 5 2600 w 3060ti/8GB VRAM. We showed you can actually get very powerful productive capability on a 3060ti!

Hot Configthinkmeltprotonmail.com

Conclusion - Everyone Wants a LocalLLM. Configurations Matter.

You are not going to replace a $20 Billion dollar server farm with racks of H200 inference engines with a $1200 4080 Nivdia RTX you scored off FaceBook.  That's just not realistic. But configurations really matter. The right model, the right tuning, the right MCP agents they are capable, very quick, reliable and you never have to worry about your research being harvested by a corporate LLM.   You can take odd paths with the B-250 but find you outgrow a limited memory device, and find yourself going big for a DGX Spark. Corporate interests using tokenMaxxing budgets and getting billion dollar builds realized themselves even -  they jumped their own shark, and were getting massive Token bills. Suddenly Elon was limiting employees to $200/week, Meta was scaling things back, and it was becoming quickly public  everybody didn't want the prompt harvesting companies taking their proprietary data and using it to train the next 6T model.

• Commit the time. It can take a few days, but just learn a little bit at a time. If you would like a world-class LLM that encorporates all the bells and whistles start with this StudentLLM.
• This site is utterly CHOCK full of opensource guides for MCP tools, for various llama.cpp configurations, model reviews etc. It is a win for you!

Fine-tuned models are the LLM sleeper hit. The latest rage is taking opensource models then applying a retune layer to boost their ratings - like a lot! Huggingface.com is literally  forking hourly model drops. Yes absolutely there are a lot of incredulous and possibly dubious claims being made. Set that aside and just have a look.  Agent-A1 caught our attention in that it was claiming to be out-benching GPT 5.5 a commercial billion dollar OpenAI model, on a significant number of research based scientific benchmarks. Very interesting in that you could freely download and pull it. Cost is $0.

• GPT 5.5 has not disclosed the size of it's model. For all we know it could just be another 35B.  But that is highly doubted.
• Agent-A1 outperformed it in several benchmarks, lost in a few and that caught our attention, check this out.

If you would like to research the background paper this model is built upon:

https://arxiv.org/pdf/2606.30616

Here was the full post and chart:

David Hendrickson (@TeksEdge) on X

🎯 A better Qwen3.5-35B post-trained model could be the next Local AI agent winner! A must test. A new week and a new Qwen3.5 post-trained model is here. This time it’s Agents-A1 . It’s a 35B MoE model built for long-horizon agentic tasks. It stands out for stronger performance https://t.co/kwEufW…

X (formerly Twitter)David Hendrickson

What immediately caught our attention was this small underclass 35B - beating GPT 5.5 (high) in 7/10 benchmarks?! Really!?

Now the post did not explicitly link to which model as huggingface has multiple references to an  'Agents-A1' but we presume this is the InternScience/Agent-A1.

InternScience/Agents-A1 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

It has many rebuilds and gguf forks, people are constantly rehashing the models that can seem a little overwhelming. Anyways, in the end pick one..

We chose to go with the Tribbler fork as it looked  to already have added MTP support and a clean Q4 (4-bit Quantization). We also like it's 'working' descriptor. We like stuff that works, naturally.  Some other models we tried just didn't launch or seemed to be missing layers on their upload.

wget https://huggingface.co/Tribbler/Agents-A1-Q4_K_M-MTP-GGUF-working/resolve/main/agents-a1-q4_k_m-MTP.gguf?download=true

Note:

• We really usually focus on coding assistant LLMs. Ones  that will run on very modest hardware using the 3060/3080/4080 GPUs as a benchmark. The goal is to fully run local and see what you can get without requiring a $5000 compute node.
• This is a science / research based assistant LLM.  

We gave it the following configuration, while we waited the approximately 10 minutes for the  model to pull from huggingface.

/usr/bin/llama-server -m /home/c/models/agents-a1-q4_k_m-MTP.gguf?download=true \
--spec-type ngram-mod \
--spec-draft-n-max 3 \
-c 16384 \
--host 0.0.0.0 \
--no-mmap \
--n-gpu-layers -1
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
--flash-attn on \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
-b 1024 \
-ub 512 \
--temp 0.8 \
--top-p 0.7 \
--min-p 0.05 \
--repeat-penalty 1.2 \
--jinja \
--spec-draft-p-min 0.75 \
--host 0.0.0.0 \
--port 8080 \

Load-up looked good, it took up about 14.5 GB of our 16GB VRAM nvidia-smi shows:

Please Note - You CAN run this on a 3060ti / 3080 etc.

If your GPU is modest like a 3060ti you can still run this, simply beef up your CPU RAM, and then change your --n-gpu-layers 10 or 20 or something, while you are at it watch your nvidia loadout with nvidia-smi and see how it fits.  It will run slower, and that is the trade off where a bunch of layers might have to be done inside the much slower CPU.

Continuing..

We gave it a 16K context to start. Coders love large context models that can refactor whole git repositories, but from a research perspective one can probably get by with less for short questions.

We gave it one mcp tool out of the gate allowing it to do it's own Internet lookup research and had Grok 4 write up a challenging prompt.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

Grok gave us this

You are Agent-A1, an elite scientific reasoning agent with deep expertise across physics, chemistry, biology, computational science, and systems engineering. Your capabilities include rigorous multi-step reasoning, hypothesis generation, quantitative analysis, experimental design, error propagation, ethical evaluation, and critical assessment of scientific literature. You operate at the level of a multidisciplinary research team lead with access to state-of-the-art knowledge up to 2026.

**Core Task:**
Design a complete, feasible research program to achieve **room-temperature, ambient-pressure superconductivity** in a carbon-based or hybrid organic-inorganic material within the next 5–7 years. This must go beyond current hydride-based high-pressure approaches (e.g., LaH10 or carbonaceous sulfur hydride records).

**Requirements (address each explicitly and in sequence):**

1. **Theoretical Foundation (Step-by-Step Reasoning):**  
   Propose a detailed physical mechanism. Integrate concepts from BCS theory extensions, strong electron-phonon coupling, flat-band engineering, moiré superlattices, quantum geometry, excitonic pairing, or polariton-mediated superconductivity. Explain why your approach could enable Tc > 300 K at 1 atm. Include quantitative estimates (e.g., expected λ, μ*, density of states, coherence length) using simplified formulas or order-of-magnitude calculations. Discuss potential phase diagrams and competing orders (CDW, magnetism, Mott insulation).

2. **Material Design:**  
   Specify a candidate material family (e.g., twisted bilayer graphene derivatives, metal-organic frameworks with specific dopants, hydrogen-rich organic crystals under strain, or hybrid perovskite-like structures). Provide atomic-level structural details, synthesis precursors, and predicted electronic band structure features. Justify stability under ambient conditions using thermodynamic and kinetic arguments.

3. **Experimental Roadmap:**  
   Outline a phased 5–7 year research plan with milestones, required facilities (e.g., specific synchrotron, cryo-EM, or ultrafast laser setups), sample characterization methods (resistivity, magnetic susceptibility, specific heat, ARPES, Raman, neutron scattering), and control experiments. Include statistical power analysis for key measurements and potential failure modes with mitigation strategies.

4. **Computational Validation Pipeline:**  
   Detail a multi-scale modeling workflow (DFT → DFPT → Eliashberg equations → molecular dynamics → machine-learned potentials). Specify software/tools (e.g., Quantum ESPRESSO, VASP, LAMMPS, or custom ML frameworks) and estimated computational resources needed. Address known limitations like van der Waals corrections, strong correlation, or finite-size effects.

5. **Scalability, Safety, and Broader Impact:**  
   Analyze manufacturability at scale, cost projections, environmental impact, and dual-use risks. Propose open-science mechanisms while protecting IP where necessary. Discuss how success would transform energy, transportation, computing, and medicine.

6. **Critical Evaluation and Alternatives:**  
   Rigorously critique your own proposal (weaknesses, why it might fail based on historical precedents like LK-99). Provide at least two strong alternative approaches with comparative advantages/disadvantages. Assign Bayesian confidence scores (0–100%) to key claims with justification.

**Response Format:**
- Use numbered sections matching the requirements above.
- Include LaTeX for all equations (e.g., \( \lambda = \dots \), Eliashberg equations).
- Cite specific papers or arXiv preprints (real or plausibly recent) with DOIs/arXiv IDs where relevant.
- Maintain extreme intellectual honesty: flag uncertainties, unknown unknowns, and required breakthroughs.
- End with a concise executive summary and a prioritized list of immediate next experiments.

Begin your response now. Think deeply and systematically before outputting.

It immediately chugged out with a very strong 77 T/s tokens (recall we are using MTP heads=3) very impressive. It was theorizing some 'pairing interactions.'

It completed the task in a quick 1m 26s, and averaged 75.26 T/s so:

Tool Calling

Next we asked it to do some basic tool calling, part 1 was to see if it could clean out the Process Manager:

Resistive Research Internet Based Prompting

• We then gave it it's own open resistive based prompt.  Effectively the LLM is asked to conjure up it's own theory - but make multiple attempts to discredit it. If it passes write it up.  While you are at it - use the internet to find something that hasn't been solved yet:

Go on the internet find some quantum entanglement research that has never been solved. Make a powerful theory that could solve it, and then make three attempts to disprove it. When you are done explain why your theory is the solution. If it fails reattempt it until you have something that you know will work.

The Context Trade Off - Fast Small Questions or Big Slow Questions?

• At this point that single question overflowed the LLM, which kicked out and dropped immediately on the first query. Recall we gave it a very small (Jun 2026 standards) context of 16384, we modified our run configuration, dropped the MTP heads down to 1, and rerun it: Our new configuration was now setup for long-slow research where we just want the answer, the deep research, we don't mind it it needs to work for an hour while we go do something else.

/usr/bin/llama-server -m /home/c/models/agents-a1-q4_k_m-MTP.gguf?download=true \
--spec-type ngram-mod \
--spec-draft-n-max 1 \
-c 256000 \
--host 0.0.0.0 \
--no-mmap \
--n-gpu-layers -1
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
--flash-attn on \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
-b 1024 \
-ub 512 \
--temp 0.8 \
--top-p 0.7 \
--min-p 0.05 \
--repeat-penalty 1.2 \
--jinja \
--spec-draft-p-min 0.75 \
--host 0.0.0.0 \

We can see that MTP reduction was still chugging pretty fast - 56.33 T/s to start (that drops as context go up).

You Can Run This Too - Its All Opensource!

If you want to run this full setup yourself - just go work through the Student LLM, then add in the MCP agents. We do all of this on a Ryzen 9 128GB w/ a 4080.  It's a sub $2000 build if you just buy the parts.

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

Watching it work it chugged up cleanly approaching 30K tokens, and had decided to make a html presentation of it's work after many calls to the internet and doing some html page pulls.

At this point we left our 'house-scientist LLM' to work away while we did some house chores, eventually it came up with this:

Next we wanted to see if it could make some visual matplotlib graphs.  We have a  specialized open MCP tool that does graphing namely:

MCP Power Tool mcp_matplot - Add High Quality 2D/3D Plotting to A Small Context LocalLLM.

We show you how you can get *very* powerful plotting capability to your local LLM!

Hot Configthinkmeltprotonmail.com

Understand that for localLLMs or 'houseLLMs'  people on modest budgets typically are going to work on a contexts < 256K unless they start accumulating multiple 3090ti GPU's and custom builds.  That's just not realistic to the majority of us so the above MCP tool is specifically built to not return a large JSON object that will overflow the small context. This allows you to ask your localLLM to make these amazing charts without blowing out your Token Bank:

At this point, it did a really good job of looking at the tool, figuring out which plot it wanted to create that would interface with it's new QCIT theory:

Through the whole process we noted our 16GB VRAM 4080 had not failed:

It produced numerous plots and graphs.  Do note - the llm cannot see what it's producing, however you can give it feedback to corret it's work. We printed the plots to a PDF.

Conclusion - It's Powerful, Local, and Capable. It will Empower Junior and Basic Researchers If They Take a Bit of Time to Learn How to Set It Up and Use it.

Absolutely I would surmise that this LLM is very capable of becoming a research assistant to a junior researcher, and it can work effectively as a plotting assistant on a very humble budget. If you offload a lot of the layers it would even run respectibly on a 3060ti, or a 3080 no problem. It might only give you 20 T/s, go drink a coffee let it chug.

The LLM produced a number of 'graphs' which really was it's attempts at making a slide-deck visual for review.

If you could go back to 1995, and show this LLM to peple then using 386's and on dial-up modems  it would borderline science fiction fantasy..  Today we don't even barely consider the capabilities that we can use and obtain for free.

We literally finish one review, and another model drops.  The LLM ecosystem moves so fast right now that one can barely finish looking at one model before the next one becomes available. You the localLLM user are the winner.

Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We are using the StudentLLM configuration, and a 4080 GPU (16GB VRAM) on a Ryzen 9.  That's very modest hardware as in this instance much of the model needed off-loading to the CPU.  Irrespectively - you are the benefactor. In fact we even showed how potent 8B and 9B sized models can be (A, B)

Our configuration

/usr/bin/llama-server -m Qwopus3.6-35B-A3B-Coder-MTP-Q6_K.gguf?download=true \
--spec-type ngram-mod \
--spec-draft-n-max 3 \
-c 32768 \
--host 0.0.0.0 \
--no-mmap \
--n-gpu-layers -1
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
--flash-attn on \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
-b 1024 \
-ub 512 \
--temp 0.8 \
--top-p 0.7 \
--min-p 0.05 \
--repeat-penalty 1.2 \
--jinja \
--spec-draft-p-min 0.75 \
--host 0.0.0.0 \
--port 8080 \

Inspecting our nvidia-smi showed a clean loadout with 14.8GB used out of 16GB VRAM.

Out the gate it gave very good tool calling, scripted up some basic pythonic code and we naturally started the 'Benchie Benchmark' of asking it to write it's own Asteroid game.

• We get the LLM to research it's own specs so the prompt might be different each time it looks up a random site on the internet.

We use our own Process Manager which has internal to it a duckduckgo html pulling tool. This gives effective, fast internet capabilities, and it's opensource!

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

Within 1 min 47 seconds it had produced some quality asteroids, created a code drop point using the process manager MCP.  If you need these  powerful tool calls they are all available here:

• What was very nice is 48.32 Tokens/s generation.  Considering we are using a overflow model that was a whopping 29 GB, squished a bunch of layers to the GPU, and leaving a bunch on the CPU this is working very well.

At this point we were quite shocked. It nailed Asteroids cleanly. It was playable and accurately developed in a single prompt!  We typically accept that if a LLM can create Asteroids in 5-6 prompts it's still a very viable capable LLM, considering that we have seen SOTA models that can muck stuff up on a consistent basis. It always depends what the input feed to the built model was..

Of real interest is that we are running --spec-draft-n-max 3 and getting very good 48 Tokens/s

Long Context Testing

We then adjusted our configuration, backed off  --spec-draft-n-max 0 increased context -c 128000 and restarted it.

Our new configuration:

/usr/bin/llama-server -m Qwopus3.6-35B-A3B-Coder-MTP-Q6_K.gguf?download=true \
--spec-type ngram-mod \
--spec-draft-n-max 0 \
-c 128000 \
--host 0.0.0.0 \
--no-mmap \
--n-gpu-layers -1
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
--flash-attn on \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
-b 1024 \
-ub 512 \
--temp 0.8 \
--top-p 0.7 \
--min-p 0.05 \
--repeat-penalty 1.2 \
--jinja \
--spec-draft-p-min 0.75 \
--host 0.0.0.0 \
--port 8080 \

We gave it a very challenging prompt that most LLM's will normally not be able to complete on their own - that is self-prompt expansion and application development.

The load on the Nvidia 4080 sat as follows:

It showed no issues doing iterative research and development of it's task.

At 28,000 tokens it was still chugging at 40 Tokens/s not bad!

At about roughly approaching the 5 minute mark it was finished:

At this point we could see that it had generated two html detailed web pages, plus some python specifications, but did not actually generate the game itself.

That was on Us!

We realized at this point it had done exactly what we asked it to do - it upgraded the specification but did not actually write the game - we didn't ask it!

It was also interesting in that it ran concurrently with either the GPU floored at 90%+ or the GPU semi-idle at 30% while the cores on the Ryzen hauled.

It should be noted it built a beautiful structure using the Process Manager

Near perfect. Wow Now Go Write the Actual Game Please..

• This clearly is a prompt crusher of an LLM. It did exactly what we asked it to do.  There was a few flaws in the game, but we could tell that it would nail the corrections out of the gate.

This is a Top Tier LLM in the 35B Class

• There is no question that this model in its weight class is more than a daily driver, it is clearly up there with Ornith 1.0 and I leave it to you which one you want (run both!)  We would suspect that Ornith might work at a slightly higher level, but this is one of the most accurate prompt followers we have ever seen.  It is very rare for prompts to be fulfilled this accurately. But this is a utterly powerful LLM and we expect that it will become a flagship model at huggingface.com

What happens when everybody becomes an 'LLM Booster?'  What is that exactly you ask? It is when people take world class SOTA models and use them consistently to 'pull up' smaller models and boost their capabilities.  We strongly believe that 'LLM Boosting' will see the public win over all attempts to regulate it.

This is exactly what is happening with this model, as described:

Qwythos-9B is a full-parameter reasoning model post-trained on over 500 million tokens of high-quality Claude Mythos / Claude Fable traces with chain-of-thought generated in-house by Empero AI's internal rethink tool. It dominates the base Qwen3.5-9B under matched evaluation (+34 pts MMLU, +30 pts gsm8k-strict, +19 pts gsm8k-flex), supports native function calling per the Qwen3.5 spec, and ships with a 1,048,576-token (1M) context window via YaRN rope-scaling enabled by default.

It can be easily pulled in the quantization of your choice:

empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

wget https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF/resolve/main/Qwythos-9B-Claude-Mythos-5-1M-MTP-Q8_0.gguf?download=true

What is really nice about a 9B model is people on limited budgets can pull these models easily and test it for themselves.  3060ti video cards from 10 years ago do ship in 12GB VRAM flavors frequently and you can have your own localLLM running powerfully and reliably.  You would need clearly a smaller quantization naturally and that might introduce artifacts but in the end, everybody wins for once with low-cost open models like these!

We do not run a pile of tests on a LLM, instead we simply say Go write some python code and pay attention to Token generation speed and watch what it decides to do.  Why? Because if it is quick, and capable of tool calling most LLM's will correct their work anyways. It is more valuable one could say-  to have a learning correcting LLM, with fluid tool calling even if it's smaller and maybe less adept.  You can have a $12 Billion dollar Grok 4 make piles of mistakes along with an 8B.  We have personally spent hours prompting SOTA models that cannot get it - only to have a Qwen 3.6 Moe nail it out of the gate. The key is allowing them to read their own mistakes and learn from them. Anyways..

It was given two tools, which are opensource right here:

70 Tokens/s.  We consider anything above 50 a production level. It handled tool calling right out of the gate.  We have seen 35B's get tripped up on this.

Our configuration run was as follows:

/usr/bin/llama-server -m Qwythos-9B-Claude-Mythos-5-1M-MTP-Q8_0.gguf?download=true \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
-c 65536 \
--no-mmap \
--flash-attn off \
--n-cpu-moe 20 \
--n-gpu-layers -1 \
--cache-type-k turbo2 \
--cache-type-v turbo2 \
-b 1024 \
-ub 512 \
--temp 0.7 \
--top-p 0.9 \
--min-p 0.05 \
--repeat-penalty 1.1 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \
--spec-draft-p-min 0.50

Under the current configuration we are only using 10.2 GB / 16GB VRAM. That means bigger contexts are well available! We turned the context up, typically about 130,000 sized context is enough for a LLM to attempt a one-shot Asteroids. -c 256000 What is really nice about this model is it comes 1M Context right out of the hop. That's impressive. Sometimes people do not need a $20 Billion dollar model they just need a modest model that can work.

With a 256,000 context our GPU load went up to 12GB / 16GB of VRAM. Still lots of room left.

One-Shotting Asteroids

• We treat the one-shot of Asteroids similar to the now defactor 'benchie' that 3D printers use as a benchmark test.

We gave it two tools for this job, namely:

It ran tool calling no issues at all, very fluid. We LOVE these 8B < 12B models simply in that they run fast, work hard, do good tool calling and make great coding assistants.

This has to be a typo, it claims it has finished Asteroids in 37 Seconds. Seriously?!

• How the process manager tool works it is it like a 'digital notepad' that can look up stuff on the internet.  It is a powerful coding assistant that the LLM's can save work between themselves, recall their previous work in a new context and give code -drops.

2.7 seconds to do a code drop. That's fast.

Examining the code show it has a bit of hallucinatory errors. We are using Turboquant compression along with a lot of additives to the configuration so let's see if it can solve it's own problems.

Comically it suggested adjusting the following run configuration, I think --temp 0.2 will not work but lets try it.

/usr/bin/llama-server -m Qwythos-9B-Claude-Mythos-5-1M-MTP-Q8_0.gguf?download=true \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
-c 256000 \
--no-mmap \
--flash-attn off \
--n-cpu-moe 20 \
--n-gpu-layers -1 \
--cache-type-k turbo2 \
--cache-type-v turbo2 \
-b 1024 \
-ub 512 \
--temp 0.2 \
--top-p 0.7 \
--min-p 0.05 \
--repeat-penalty 1.2 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \
--spec-draft-p-min 0.50

At this point we have no idea how having --temp 0.2 works so well but it generated a complete game that puts out a blank screen.

Please Note - We do not use the expectation of a LLM 'one-shoting' entire systems.  We have seen multi-billion dollar LLM's make piles of mistakes so we see these as assistive processes that we work with. When you start thinking like this and start really exploring the capabilities of these models you will find out how utterly powerful they are.

We re-ran this with --temp 0.8 and were very pleased with it's basic results.

Conclusion

• We love 8 and 9B models simply in that they run fast. It had very good tool calling, and was capable of correcting it's errors. Would I use this as a large project context manager on large code bases. Most likely not.  However for a 9B they give exceptionally good support, they run very fast, and it gave a good effort. Absolutely I would have this writing small sub-sections of code or looking up support boilerplate, or as a support model that can run locally checking code and other tasks.  
• This is a really good model in it's weight class.  Given the right MCP tool like these ones it becomes a small powerhouse. It had no issues using the Javascript end-point to compile and test its work.

Ornith 1.0 as of it's debut only days ago  is currently the 'opensource upset' and will it dethrone the Qwen 3.6 dominance?  Originally released without MTP it only took 72 hours for  five MTP capable mixture models soon followed:

We noticed the neko-legends/s-batman build was getting reports of very high token speeds.  In this case we already have very fast capable configurations ready. Do note that this is a 4-bit quantization our other models were 6-bit (Q6) - that can be a big factor. We preferred typically running Q6 as it gave near 8-bit performance without any real performance degradations, so lets see how this runs:

Our config:

• We just tried this configuration as a 'recycle' from our last Ornith 1.0 model

Ornith MTP FrakenModel 1.0 (Try 01) w/MTP. Slower than it’s Original?

We explore if MTP is hitting Ornith 1.0. We were not able to get significant breakthroughs - again configurations matter!

Hot Configthinkmeltprotonmail.com

/usr/bin/llama-server -m ornith-1.0-35b-aeon-ultimate-uncensored-nvfp4-gguf-mtp.gguf?download=true \
--spec-type draft-eagle3,ngram-mod \
--spec-draft-n-max 4 \
-c 131072 \
--no-mmap \
--flash-attn off \
--n-cpu-moe 30 \
--n-gpu-layers -1 \
--cache-type-k turbo4 \
--cache-type-v q8_0 \
-b 1024 \
-ub 512 \
--temp 0.7 \
--top-p 0.9 \
--min-p 0.05 \
--repeat-penalty 1.1 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \
--spec-draft-p-min 0.75

It booted up clean and quickly on our 4080, some snapshots:

Under that configuration about 10.5 GB of the VRAM was used up, which is promising leaving 5GB for large contexts

12.2 Tokens/s. Sadly.

We turned down one parameter and run it again: --spec-draft-n-max 2, next we also let it work it's own problem.

We keep a very healthy large number of completely opensource MCP tools, one in particular is very good as it allows your LLM to research on the internet by itself:

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

• It continued researching on our behalf, at 12 Tokens/s.

Ongoing

• We are still researching the speeds. It should be understood that the performance of a 5090 is vastly more in terms of VRAM and Cuda Cores over a 4080, and to get a margin drop in this size is very realistic.
• The real perf gains we suspect is using both MoE and MTP.   That is what we need. We are most likely running a non-Moe.  However what probably was the mistake was because we presumed these models have Moe because the base models do too!

Update, within hours there are scratch 4, now 5  variants of Ornith, This is a test of the first one:

Ornith 1.0 (the original)  released only days ago,  has caused  a real upset. Claiming significantly higher benchmarks as a 'fine-tuned' model derivative from it's predecessors of Qwen 3.5 and Gemma 4, it hit the Opensource ranks with 50/50 splits. Some were positive, others neutral, others didn't see benefits in their benchmarks.  Qwen 3.6 had dominated the localspace for some now, and an upset was in the works. Some using the new Ornith 1.0 did see significantly better tool calling, others suggested it was a half-mung model. Overall if it's benchmarks were true it was a very significant leap forward irrespective if one felt it was 'Benchmaxxed'. You can decide for yourself, we have an entire setup guide as well.

GitHub - deepreinforce-ai/Ornith-1

Contribute to deepreinforce-ai/Ornith-1 development by creating an account on GitHub.

GitHubdeepreinforce-ai

It only took literally hours for someone to modify it, add MTP (Multi-Token Protocol) support and to give it a 'franken-model' naming:

skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We immediately set out to pull it and see if we could break the 100T/s on our 4080 rig. Our run configuration after some mangling is sorta a bit of a franken-config in itself because we had to force off-load some of the tensors with our configuration cheat code:

--override-tensor "\.ffn_.*_exps\.weight=CPU" \

/usr/bin/llama-server -m ornith-1.0-35b-Q6_K-MTP-final.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
-c 16384 \
-ngl 99 \
--flash-attn on \
--cache-type-k turbo2 \
--cache-type-v turbo2 \
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
-b 1024 \
-ub 512 \
--temp 0.7 \
--top-p 0.9 \
--min-p 0.05 \
--repeat-penalty 1.1 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \

It fired up cleanly, without much fuss:

We were not completely really sold on this setup. MTP typically should give some 'slop' but in exchange you can have super-fluid fast token generations. We measured about 37-40 Token/s typically  is the same as just running a straight Ornith 1.0 Moe?

We kept trying different configurations over and over, we also tried variants like this:

/usr/bin/llama-server -m ornith-1.0-35b-Q6_K-MTP-final.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-c 8192 \
-ngl 20 \
--flash-attn on \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
-b 1024 \
-ub 512 \
--temp 0.7 \
--top-p 0.9 \
--min-p 0.05 \
--repeat-penalty 1.1 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \
--spec-draft-p-min 0.75

Final Try - Stacking --spec-type

We finally tried this and it did get pretty fluid, but at no point did we see 200% gains, or 150% gains, and or the 125% gains.

/usr/bin/llama-server -m ornith-1.0-35b-Q6_K-MTP-final.gguf \
--spec-type draft-eagle3,ngram-mod \
--spec-draft-n-max 4 \
-c 131072 \
--no-mmap \
--flash-attn on \
--n-cpu-moe 30 \
--n-gpu-layers -1 \
--cache-type-k turbo4 \
--cache-type-v q8_0 \
-b 1024 \
-ub 512 \
--temp 0.7 \
--top-p 0.9 \
--min-p 0.05 \
--repeat-penalty 1.1 \
--jinja \
--host 0.0.0.0 \
--port 8080 \
-np 1 \
--spec-draft-p-min 0.75

Conclusion

• We have tried offloading the model, but it has not showed us any significant improvements in speed over just running a standard Ornith 1.0 however there are still two more MTP models to go.
• We fully recognize that the huggingface site is not using llama.cpp but an alternate.  We prefer it simply in that we can run very large contexts.
• One needs to understand that MTP can be a 'trade-off' you may be forced to run a smaller context - but you should see very fast token generation.  We have not been able to thus far get there (we tried a bunch of varying context sizes).
• Configurations really matter.  In one setup we were getting 17T/s adjusting it slightly and we were back cooking at 37 T/s.  It really is worth your time to learn your LLM and the best configuration for your GPU!  After 50 T/s if you can get your LLM there it becomes quite 'fluid' in that is is responsively flooding you with answers to keep you in flow.

Why?

• Imagine setting up a 37 Token/s workhorse that can run all night have big contexts, and  then switching out your config and getting 100 Token/s fluid fast tool calling for fast local questions. It is the 'best-of-both-worlds' type setup. A simple way to explain it is like it's similar to a truck pulling hard in 2nd gear pulling a trailer load, and then moving quickly to take you for snacks!  

The idea is simple. Most files now may fit into a single context.  What if you take a leading model and have it review line-by-line entire code bases for you? What would be the result?

• Yes - I know claude and others are probably doing this  but what about doing it from a houseLLM with a localGPU using the best model of the day Ornith 1.0

What is amazing is this can be extended for security testing, and or pentesting simply by modifying the prompt!

This is complex? Just learn a bit at a time, and work through the powerful StudentLLM. Don't underestimate it - when you finish it - it is actually one of the worlds most advanced localLLM configurations, and you can run 70B, 35B Moe, MTP, Turboquant, whatever you want!

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

Yes you can use an excellent Hermes agent, in this case we boiler-plated some code to really learn the process.

Hermes Agent — The Agent That Grows With You

Hermes Agent — a standalone terminal app and a native application for macOS, Windows, and Linux. Install it and start a conversation with the agent that grows with you.

Hermes Agent

Here is some boilerplate code, and we were wondering if it has become possible to have these kinds of iterative loops working on our behalf.

• The git repo is pulled, in our instance we want to learn from this excellent fork of llama.cpp

GitHub - TheTom/turboquant_plus

Contribute to TheTom/turboquant_plus development by creating an account on GitHub.

GitHubTheTom

• A list of files are queried and they are prompt serially marched one by one through the reviewing Ornith.
• It produces an output fork that is automatically tested and compiled and this sub-loop sends back the errors if any until it passes.
• If a dramatic improvement is found the new repo -> base repo and the process is started.

Some code

import asyncio
import subprocess
import json
from pathlib import Path
from typing import List, Dict, Optional
import git  # pip install GitPython
import aiohttp
import re
import time


class GitAlgorithmOptimizerAgent:
    def __init__(self, server_url: str = "http://192.168.1.3:8080", work_dir: str = "/tmp/git_scan"):
        self.server_url = server_url.rstrip("/")
        self.work_dir = Path(work_dir)
        self.work_dir.mkdir(parents=True, exist_ok=True)
        self.max_build_attempts = 5
        self.build_timeout = 300  # seconds

    async def _query_llm(self, prompt: str, max_tokens: int = 2000) -> str:
        payload = {
            "prompt": prompt,
            "n_predict": max_tokens,
            "temperature": 0.7,
            "top_p": 0.9,
        }
        async with aiohttp.ClientSession() as session:
            async with session.post(f"{self.server_url}/completion", json=payload) as resp:
                if resp.status != 200:
                    text = await resp.text()
                    raise Exception(f"LLM error: {resp.status} - {text}")
                data = await resp.json()
                return data.get("content", "")

    async def clone_repo(self, repo_url: str, branch: str = "main") -> Path:
        repo_name = repo_url.split("/")[-1].replace(".git", "")
        local_path = self.work_dir / repo_name
        if local_path.exists():
            print(f"Using existing clone at {local_path}")
            return local_path

        print(f"Cloning {repo_url}...")
        subprocess.run(["git", "clone", "--depth", "1", "-b", branch, repo_url, str(local_path)], check=True)
        return local_path

    def scan_source_files(self, repo_path: Path, max_size: int = 12000) -> List[Dict]:
        """Scan source files across multiple languages: Python, C, C++, CUDA."""
        source_files = []
        extensions = {
            '.py': 'Python',
            '.c': 'C',
            '.cpp': 'C++',
            '.h': 'C/C++ Header',
            '.hpp': 'C++ Header',
            '.cu': 'CUDA',
            '.cuh': 'CUDA Header'
        }

        for ext, lang in extensions.items():
            for file_path in repo_path.rglob(f"*{ext}"):
                try:
                    with open(file_path, "r", encoding="utf-8") as f:
                        content = f.read()
                    relative_path = str(file_path.relative_to(repo_path))
                    source_files.append({
                        "path": relative_path,
                        "language": lang,
                        "content": content[:max_size],
                        "size": len(content)
                    })
                except Exception as e:
                    print(f"Error reading {file_path}: {e}")

        print(f"Scanned {len(source_files)} source files across Python, C, C++, and CUDA.")
        return source_files

    async def analyze_for_optimizations(self, file_info: Dict) -> Dict:
        lang = file_info.get("language", "Unknown")
        prompt = f"""You are an expert algorithm optimizer for {lang} code.
Analyze the following code for performance bottlenecks and suggest faster alternatives.

File: {file_info['path']}
Language: {lang}
Code:
{file_info['content']}

Focus on time complexity, data structures, memory usage, compiler optimizations, and language-specific best practices.
Respond with valid JSON only:
{{"suggestions": [{{"original_snippet": "...", "improved_snippet": "...", "reason": "..."}}], "overall_assessment": "brief summary"}}"""

        try:
            response = await self._query_llm(prompt, max_tokens=3000)
            json_match = re.search(r'\{.*\}', response, re.DOTALL)
            if json_match:
                return json.loads(json_match.group(0))
            return {"error": "No JSON found", "raw": response[:500]}
        except Exception as e:
            return {"error": str(e)}

    def save_results_to_local_git_fork(self, repo_path: Path, results: List[Dict],
                                       branch_name: str = "algorithm-optimizations"):
        """Commit analysis results and any code changes to a new branch."""
        try:
            repo = git.Repo(repo_path)

            if branch_name in [b.name for b in repo.branches]:
                repo.git.checkout(branch_name)
            else:
                repo.git.checkout("-b", branch_name)

            report_path = repo_path / "optimization_report.json"
            with open(report_path, "w", encoding="utf-8") as f:
                json.dump(results, f, indent=2)

            repo.index.add([str(report_path)])
            repo.index.commit(f"Add algorithmic optimization analysis report (branch: {branch_name})")

            print(f"Results committed to local branch '{branch_name}' in {repo_path}")

        except Exception as e:
            print(f"Git commit error: {e}")

    def _run_build(self, repo_path: Path) -> tuple[bool, str]:
        """Attempt CMake build (llama.cpp fork)."""
        build_dir = repo_path / "build"
        build_dir.mkdir(exist_ok=True)

        try:
            cmake_cmd = [
                "cmake", "-B", str(build_dir), "-S", str(repo_path),
                "-DGGML_CUDA=ON",
                "-DCMAKE_BUILD_TYPE=Release"
            ]
            print("Running CMake configure...")
            config_result = subprocess.run(cmake_cmd, cwd=repo_path, capture_output=True, text=True,
                                           timeout=self.build_timeout)
            if config_result.returncode != 0:
                return False, f"CMake configure failed:\n{config_result.stderr}\n{config_result.stdout}"

            print("Running build...")
            build_cmd = ["cmake", "--build", str(build_dir), "-j", "8", "--config", "Release"]
            build_result = subprocess.run(build_cmd, cwd=repo_path, capture_output=True, text=True,
                                          timeout=self.build_timeout * 2)

            if build_result.returncode == 0:
                return True, build_result.stdout
            else:
                return False, f"Build failed:\n{build_result.stderr}\n{build_result.stdout}"
        except subprocess.TimeoutExpired:
            return False, "Build timed out."
        except Exception as e:
            return False, f"Build error: {str(e)}"

    async def _fix_build_errors(self, repo_path: Path, build_error: str, max_fix_attempts: int = 3) -> bool:
        """Iteratively fix build errors using LLM."""
        for attempt in range(1, max_fix_attempts + 1):
            print(f"Build fix attempt {attempt}/{max_fix_attempts}...")

            # Gather context from C/C++/CUDA files
            key_files = []
            for ext in ['.cpp', '.c', '.cu', '.h', '.hpp', '.cuh']:
                for file_path in list(repo_path.rglob(f"*{ext}"))[:8]:
                    try:
                        with open(file_path, "r", encoding="utf-8") as f:
                            content = f.read()[:6000]
                        key_files.append({
                            "path": str(file_path.relative_to(repo_path)),
                            "content": content
                        })
                    except:
                        pass

            prompt = f"""You are an expert C++/CUDA/CMake build fixer for a llama.cpp-based project.
The project failed to build with the following error:

{build_error[:8000]}

Key source file snippets:
{json.dumps(key_files, indent=2)}

Provide precise fixes. Respond with valid JSON only:
{{
  "fixes": [
    {{"file_path": "relative/path/to/file.cpp", "original_snippet": "...", "improved_snippet": "...", "explanation": "..."}},
    ...
  ],
  "cmake_changes": "any suggestions for CMakeLists.txt or flags",
  "overall_plan": "brief summary"
}}"""

            try:
                response = await self._query_llm(prompt, max_tokens=3500)
                json_match = re.search(r'\{.*\}', response, re.DOTALL)
                if not json_match:
                    continue
                fix_data = json.loads(json_match.group(0))

                applied = False
                for fix in fix_data.get("fixes", []):
                    file_path = repo_path / fix["file_path"]
                    if file_path.exists():
                        try:
                            with open(file_path, "r", encoding="utf-8") as f:
                                content = f.read()
                            updated = content.replace(fix["original_snippet"], fix["improved_snippet"])
                            with open(file_path, "w", encoding="utf-8") as f:
                                f.write(updated)
                            print(f"Applied fix to {fix['file_path']}")
                            applied = True
                        except Exception as e:
                            print(f"Failed to apply fix: {e}")

                success, output = self._run_build(repo_path)
                if success:
                    print("Build succeeded after fixes!")
                    return True
                else:
                    build_error = output
                    time.sleep(2)
            except Exception as e:
                print(f"Fix attempt error: {e}")

        print("Max fix attempts reached.")
        return False

    async def run_analysis(self, repo_url: str, max_files: int = 30, max_cycles: int = 3) -> None:
        """Main iterative optimization cycle supporting multiple languages."""
        repo_path = await self.clone_repo(repo_url)

        for cycle in range(1, max_cycles + 1):
            print(f"\n=== Starting Optimization Cycle {cycle}/{max_cycles} ===")

            files = self.scan_source_files(repo_path)
            results = []
            for file in files[:max_files]:
                print(f"Analyzing {file['path']} ({file['language']})...")
                analysis = await self.analyze_for_optimizations(file)
                results.append({"file": file["path"], "language": file["language"], "analysis": analysis})

            branch_name = f"algorithm-optimizations-cycle-{cycle}"
            self.save_results_to_local_git_fork(repo_path, results, branch_name)

            print("Attempting build...")
            success, build_output = self._run_build(repo_path)
            if not success:
                print("Initial build failed. Entering fix sub-cycle.")
                build_fixed = await self._fix_build_errors(repo_path, build_output)
                if not build_fixed:
                    print("Failed to fix build. Stopping this cycle.")
                    break
            else:
                print("Build succeeded.")

            print(f"Cycle {cycle} completed.")

        print("\nAll optimization cycles finished.")


# Usage
async def main():
    agent = GitAlgorithmOptimizerAgent(server_url="http://192.168.1.3:8080")
    await agent.run_analysis("https://github.com/TheTom/turboquant_plus", max_cycles=30)
    print("Full iterative process complete.")


if __name__ == "__main__":
    asyncio.run(main())

Our specs:

• 4080 16GB VRAM / Ryzen 9 3900 w/128 GB Ram, Our run configuration is Ornith 1.0 with this setup:

Ornith 1.0 Breakout - the HouseLLM BenchMaxx? Builds a CNN (Convolutional Neural Network) on the first Prompt!

Ornith 1.0 a MIT licensed supermodel gives crushing results, and very good performance!

Hot Configthinkmeltprotonmail.com

• Outside that we simply let it work.
• The Moe + 4080 was kept very busy:

It looked to take about a minute to review each file:

Ingest was solid we came in at 640 Tokens/s and our generation was a solid 36T/s consistent across tasks

As you can see here we are actually wasting  resources.. We are only using about 10 GB of our card no matter what file we look at - there may be very large files that come so we will not touch it for now

Finished - Not at all.

• We still need to add some feed-back compilation loops and speed testing of the builds. That needs all the setups and configurations plus the cmake. However the seed is started.  
• From this point it can be built and added upon.  If we make significant improvements we will post it here!

Update - Adding a Compilationn and Repair Sub-Fork

• We are part of an iterative cycle. We see things that need fixing, we are really an 'agent' that prompts the LLM to do much boilerplating work.
• Millions and millions of researchers iterating through things accelerating their research and development.
• Results go logarithmic everywhere there is a large distributive bell-curve.

Multiple Path Iteration

• This modification of the code will create a different git every time it runs.  
• Overwriting sequential iterations may have the algorithms passing by a 'sweet-spot'

import asyncio
import subprocess
import json
from pathlib import Path
from typing import List, Dict
import git  # pip install GitPython
import aiohttp
import re
import time
import shutil


class GitAlgorithmOptimizerAgent:
    def __init__(self, server_url: str = "http://192.168.1.3:8080", work_dir: str = "/tmp/git_scan"):
        self.server_url = server_url.rstrip("/")
        self.work_dir = Path(work_dir)
        self.work_dir.mkdir(parents=True, exist_ok=True)
        self.max_build_attempts = 5
        self.build_timeout = 300  # seconds

    async def _query_llm(self, prompt: str, max_tokens: int = 2000) -> str:
        payload = {
            "prompt": prompt,
            "n_predict": max_tokens,
            "temperature": 0.7,
            "top_p": 0.9,
        }
        async with aiohttp.ClientSession() as session:
            async with session.post(f"{self.server_url}/completion", json=payload) as resp:
                if resp.status != 200:
                    text = await resp.text()
                    raise Exception(f"LLM error: {resp.status} - {text}")
                data = await resp.json()
                return data.get("content", "")

    async def create_independent_fork(self, base_repo_path: Path, cycle: int) -> Path:
        """Create a fresh independent fork (copy) for this iteration to enable multi-dimensional exploration."""
        fork_name = f"turboquant_plus_fork_cycle_{cycle}_{int(time.time())}"
        fork_path = self.work_dir / fork_name
        
        print(f"Creating independent fork for cycle {cycle}: {fork_path}")
        if fork_path.exists():
            shutil.rmtree(fork_path)
        
        shutil.copytree(base_repo_path, fork_path, dirs_exist_ok=True)
        return fork_path

    def scan_source_files(self, repo_path: Path, max_size: int = 12000) -> List[Dict]:
        """Scan source files across Python, C, C++, CUDA."""
        source_files = []
        extensions = {
            '.py': 'Python',
            '.c': 'C',
            '.cpp': 'C++',
            '.h': 'C/C++ Header',
            '.hpp': 'C++ Header',
            '.cu': 'CUDA',
            '.cuh': 'CUDA Header'
        }
        
        for ext, lang in extensions.items():
            for file_path in repo_path.rglob(f"*{ext}"):
                try:
                    with open(file_path, "r", encoding="utf-8") as f:
                        content = f.read()
                    relative_path = str(file_path.relative_to(repo_path))
                    source_files.append({
                        "path": relative_path,
                        "language": lang,
                        "content": content[:max_size],
                        "size": len(content)
                    })
                except Exception as e:
                    print(f"Error reading {file_path}: {e}")
        
        print(f"Scanned {len(source_files)} source files.")
        return source_files

    async def analyze_for_optimizations(self, file_info: Dict) -> Dict:
        lang = file_info.get("language", "Unknown")
        prompt = f"""You are an expert algorithm optimizer for {lang} code.
Analyze the following code for performance bottlenecks and suggest faster alternatives.

File: {file_info['path']}
Language: {lang}
Code:
{file_info['content']}

Focus on time complexity, data structures, memory usage, compiler optimizations, and language-specific best practices.
Respond with valid JSON only:
{{"suggestions": [{{"original_snippet": "...", "improved_snippet": "...", "reason": "..."}}], "overall_assessment": "brief summary"}}"""

        try:
            response = await self._query_llm(prompt, max_tokens=3000)
            json_match = re.search(r'\{.*\}', response, re.DOTALL)
            if json_match:
                return json.loads(json_match.group(0))
            return {"error": "No JSON found", "raw": response[:500]}
        except Exception as e:
            return {"error": str(e)}

    def save_results_to_local_git_fork(self, repo_path: Path, results: List[Dict], branch_name: str):
        """Commit results to the independent fork branch."""
        try:
            repo = git.Repo(repo_path)
            if branch_name in [b.name for b in repo.branches]:
                repo.git.checkout(branch_name)
            else:
                repo.git.checkout("-b", branch_name)

            report_path = repo_path / "optimization_report.json"
            with open(report_path, "w", encoding="utf-8") as f:
                json.dump(results, f, indent=2)

            repo.index.add([str(report_path)])
            repo.index.commit(f"Add algorithmic optimization analysis report (cycle branch: {branch_name})")

            print(f"Results committed to branch '{branch_name}' in independent fork {repo_path.name}")

        except Exception as e:
            print(f"Git commit error: {e}")

    def _run_build(self, repo_path: Path) -> tuple[bool, str]:
        """CMake build for the independent fork."""
        build_dir = repo_path / "build"
        build_dir.mkdir(exist_ok=True)

        try:
            cmake_cmd = [
                "cmake", "-B", str(build_dir), "-S", str(repo_path),
                "-DGGML_CUDA=ON",
                "-DCMAKE_BUILD_TYPE=Release"
            ]
            print(f"Configuring build in {repo_path.name}...")
            config_result = subprocess.run(cmake_cmd, cwd=repo_path, capture_output=True, text=True, timeout=self.build_timeout)
            if config_result.returncode != 0:
                return False, f"CMake configure failed:\n{config_result.stderr}\n{config_result.stdout}"

            print(f"Building {repo_path.name}...")
            build_cmd = ["cmake", "--build", str(build_dir), "-j", "8", "--config", "Release"]
            build_result = subprocess.run(build_cmd, cwd=repo_path, capture_output=True, text=True, timeout=self.build_timeout * 2)
            
            if build_result.returncode == 0:
                return True, build_result.stdout
            else:
                return False, f"Build failed:\n{build_result.stderr}\n{build_result.stdout}"
        except subprocess.TimeoutExpired:
            return False, "Build timed out."
        except Exception as e:
            return False, f"Build error: {str(e)}"

    async def _fix_build_errors(self, repo_path: Path, build_error: str, max_fix_attempts: int = 3) -> bool:
        """LLM-driven iterative build fixing on this independent fork."""
        for attempt in range(1, max_fix_attempts + 1):
            print(f"Build fix attempt {attempt}/{max_fix_attempts} on {repo_path.name}...")

            key_files = []
            for ext in ['.cpp', '.c', '.cu', '.h', '.hpp', '.cuh']:
                for file_path in list(repo_path.rglob(f"*{ext}"))[:8]:
                    try:
                        with open(file_path, "r", encoding="utf-8") as f:
                            content = f.read()[:6000]
                        key_files.append({"path": str(file_path.relative_to(repo_path)), "content": content})
                    except:
                        pass

            prompt = f"""You are an expert C++/CUDA/CMake build fixer.
Project: {repo_path.name}
Build error:

{build_error[:8000]}

Key files:
{json.dumps(key_files, indent=2)}

Respond with valid JSON only containing fixes."""
            # (Full prompt structure as in previous version - abbreviated here for brevity)

            try:
                response = await self._query_llm(prompt, max_tokens=3500)
                json_match = re.search(r'\{.*\}', response, re.DOTALL)
                if not json_match:
                    continue
                fix_data = json.loads(json_match.group(0))

                for fix in fix_data.get("fixes", []):
                    file_path = repo_path / fix.get("file_path", "")
                    if file_path.exists():
                        try:
                            with open(file_path, "r", encoding="utf-8") as f:
                                content = f.read()
                            updated = content.replace(fix["original_snippet"], fix["improved_snippet"])
                            with open(file_path, "w", encoding="utf-8") as f:
                                f.write(updated)
                            print(f"Applied fix to {fix.get('file_path')}")
                        except Exception as e:
                            print(f"Patch error: {e}")

                success, output = self._run_build(repo_path)
                if success:
                    print(f"✅ Build succeeded on independent fork {repo_path.name}!")
                    return True
                build_error = output
            except Exception as e:
                print(f"Fix error: {e}")

        print(f"❌ Max fixes reached for {repo_path.name}")
        return False

    async def run_analysis(self, repo_url: str, max_files: int = 30, max_cycles: int = 3) -> None:
        """Multi-dimensional independent fork iteration."""
        base_repo_path = await self.clone_repo(repo_url)
        
        for cycle in range(1, max_cycles + 1):
            print(f"\n=== Starting Independent Fork Cycle {cycle}/{max_cycles} ===")
            
            # Create fresh independent fork for this path
            fork_path = await self.create_independent_fork(base_repo_path, cycle)
            
            files = self.scan_source_files(fork_path)
            results = []
            for file in files[:max_files]:
                print(f"Analyzing {file['path']} ({file['language']}) in fork cycle {cycle}...")
                analysis = await self.analyze_for_optimizations(file)
                results.append({
                    "file": file["path"], 
                    "language": file["language"], 
                    "analysis": analysis
                })

            branch_name = f"optimizations-cycle-{cycle}"
            self.save_results_to_local_git_fork(fork_path, results, branch_name)

            # Build + Fix sub-cycle on this independent fork
            print(f"Building independent fork for cycle {cycle}...")
            success, build_output = self._run_build(fork_path)
            if not success:
                print("Initial build failed - entering fix sub-cycle.")
                build_fixed = await self._fix_build_errors(fork_path, build_output)
                if not build_fixed:
                    print(f"Failed to produce working fork for cycle {cycle}. Continuing to next path.")
                    continue
            
            print(f"✅ Working independent fork created for cycle {cycle}: {fork_path}")

        print("\nAll multi-dimensional optimization cycles completed.")


# Usage
async def main():
    agent = GitAlgorithmOptimizerAgent(server_url="http://192.168.1.3:8080")
    await agent.run_analysis("https://github.com/TheTom/turboquant_plus", max_cycles=3)
    print("Multi-path iterative process complete.")


if __name__ == "__main__":
    asyncio.run(main())

Tom’s llama.cpp fork is, at the time of this writing, one of the world’s most advanced inference engines. Built on the base llama.cpp repository, it is one of the few engines that simultaneously supports TurboQuant, MTP, and MoE. We want to highlight the dramatic contributions of ‘TheTom’ and the enormous effort invested in this project. We have simplified the explanation below so that readers new to the topic can follow along.

GitHub - TheTom/llama-cpp-turboquant: LLM inference in C/C++

LLM inference in C/C++. Contribute to TheTom/llama-cpp-turboquant development by creating an account on GitHub.

GitHubTheTom

Let's review.

• Large language models (LLMs) initially relied on 16-bit (FP16) representations for their key-value (KV) caches. These were later reduced to 8-bit formats, and further quantization techniques enabled even greater memory savings. However, aggressive quantization often led to issues such as context loss, increased hallucinations, or nonsensical outputs.
• An additional challenge was the quadratic (not logarithmic) growth in KV cache memory requirements as context length increased during inference. This made long-context processing extremely demanding, often necessitating expensive high-end servers equipped with large amounts of high-bandwidth memory (HBM).
• The introduction of TurboQuant (from Google Research) represented a major advancement. This vector quantization technique dramatically reduces KV cache memory usage—by a factor of at least 6×—while preserving model accuracy and mitigating the previous scaling issues. For the theoretical paper, see: https://arxiv.org/abs/2504.19874.
• The problem was nobody had integrated TurboQuant into a localLLM inference engine - that is until Tom did it on his fork!

What does that mean?

• When these new vector quantization algorithms were introduced it enabled a giant boost in context lengths (how long a LLM can read and regurgitate.)
• So what once required a $250,000 server could now run on a house GPU like a 3060ti or a 4080, 3090.
• Now where you once could only have small 'paragraphic' conversations with your LLM by implementing Turboquant you could fling entire code bases at your local GPU and it can handle it!

It just Gets Better, MoE, MTP, and TurboQuant, with MCP Dramatically Empowered it Even More.

• Turboquant allowed your LLM to have really long analysis chains within it's working contexts.
• MoE (Mixture-Of-Experts) allowed the active number of parameters to be dramatically reduced, by doing this non-dense models run really quick
• MTP (Multiple-Token-Prediction) allowed parallel token prediction and this enabled token speeds to double or triple! Please note you specifically need a MTP enabled model - so check.
• MCP (Model Context Protocol) agents gave agentic workflows. From this it enabled your LLM to check  its work!

However it cannot be stressed enough - none of this would be possible without finding a solution to the context length problem (how long can your model talk before token-prediction) eats up the VRAM of your GPU.

Because of this what was impossible became very quickly possible. For instance highly performative LLM's can now run smoothly and reliable on small graphic cards which we proved:

Game Changer! Crash-Out! Good Production on a Ryzen 5 2600 (6-core/12thread AMD) w 3060ti/8GB VRAM

Crash-Out! Good Production on a Ryzen 5 2600 w 3060ti/8GB VRAM. We showed you can actually get very powerful productive capability on a 3060ti!

Hot Configthinkmeltprotonmail.com

The research and work this requires is incredibly complex, and you can follow the rapid developments as these world-class LLM's break through. You can follow and study this work and the incredible work that goes into these:

turbo KV: correct Lloyd-Max centroids (4.125 bpw), PDL + fused-MMA decode for turbo4/3/2, perplexity 32K fix by TheTom · Pull Request #197 · TheTom/llama-cpp-turboquant

SummaryBrings turbo4 KV cache to parity-or-better with the strongest external turbo4 implementation (spiritbuun&#39;s CUDA fork) across Mean KLD, decode, and prefill at equal bits (4.125 bpw). All...

GitHubTheTom

Again a giant thank-you to Tom. This kind of opensource software will shape the world!

Whoa.  It just gets better by the day. Check out these incredible performance statistics for the new Ornith 1.0 which claims a fine-tuned combo-prodigy of Gemma 4 and Qwen 3.5.

Note: We initially labelled our headline 'CRUSHES' in light of its Ornith's Terminal bench jump over a  Qwen 3.6 benchmark of 41 to 64.2 by Ornith.  However to avoid 'hype-claims' we toned this down. However that's a 56% improvement in the same sized model. That's reaaally significant. Crushing? I will let you decide!

Pull it from Hugging Face..

wget https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF/resolve/main/ornith-1.0-35b-Q6_K.gguf?download=true

Qwen 3.6 was everybody's 'daily driver' but the benchmarks were just too good to ignore - check this out!

We run a custom llama.cpp -  that has  both MTP (Multiple-Token-Prediction) and Turboquant.  Here is the full guide of getting this specialized llama.cpp running! It is one of the worlds most advanced Opensource models at this time.

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

We just tossed a random basic run configuration at this, thusly:

• Note we typically run a Q6 model that gives near perfect performance, and then tinker with our offloading our key is often found by specialized offloading, namely this configuration. Q6 does not fit on our 4080, but Moe (Mixture-of-Experts) reduces the active parameters at any one time greatly enabling these models to run on limited hardware.
• This is NOT a MTP model at this time and we were quickly corrected on it.

--override-tensor "\.ffn_.*_exps\.weight=CPU" \

We gave it a incredibly hard task out of the gate, with tool calling and it kicked in right way with a swift 36 Token/s

While it was working away we sent out configuration off to grok and it suggested a more powerful configuration, and corrected us on the MTP is not quite there yet for it.

Update - we have two configurations. The first one is a more modest but stable run configuration:

/usr/bin/llama-server --jinja \
-m /home/c/models/ornith-1.0-35b-Q6_K.gguf \
--host 192.168.1.3 \
--n-gpu-layers -1 \
--n-cpu-moe 30 \
--chat-template-kwargs '{"preserve_thinking":true}' \
-c 131072 \
--flash-attn 0 \
--context-shift \
--repeat-penalty 1.10 \
--cache-type-k turbo4 \
--cache-type-v q8_0 \
--no-mmap \

This second configuration seemed to induce some hallucinations at the 100K prompt mark. The key - learn to work a bit with your LLM like a car. Find out where it's strengths and weaknesses lie.

#!/bin/bash
/usr/bin/llama-server --jinja \
  -m /home/c/models/ornith-1.0-35b-Q6_K.gguf \
  --host 192.168.1.3 \
  -ngl 99 \
  --n-cpu-moe 0 \                  # Adjust if MoE confirmed
  --chat-template-kwargs '{"preserve_thinking":true}' \
  -c 131072 \                      # Start here; scale up after testing
  -b 4096 \                        # Batch size for prompt eval
  -ub 1024 \                       # Ubatch for memory/speed balance
  --flash-attn 1 \
  --context-shift \
  --repeat-penalty 1.12 \
  --cache-type-k q8_0 \            # Higher precision K, aggressive V
  --cache-type-v turbo3 \
  --no-mmap \
  --mlock \
  --threads $(nproc) \             # Or 16–24 for your CPU
  --parallel 1                     # Increase if multi-user

• Out of the gate it successfully utilized the advanced python tool, and then started work on a convolutional neural network using numpy - seriously!

Agentic Server Primer: Llama.cpp MCP Lesson 3: Adding Python Tooling Capability To your HouseLLM.

Agentic Server Primer: Llama.cpp MCP Lesson 3: Python

Hot Configthinkmeltprotonmail.com

Prompt 2 - One-shoting Asteroids.

• Asteroids is becoming the new 'benchie' from 3D printers. Anyways we just threw a wild prompt at it as:

• This is wild - our prompt asked it to develop a better prompt and then go do it! It handled it very well,

To be continued .. We may update and see how this LLM does in the morning we will let it chug all night!

At 40,000 Tokens it Hardly Touched the 4080 GPU RAM.

• We were only 10900K on our Turboquant fill, nice!!!
• It was still busy coding and testing using the MCP agents.

We use very powerful opensource MCP agents, please, never leave your LLM in a handicap state - get them today!

PowerChest home Agent Agents MCP Tools to Put your HouseLLM into Turbo. MCP TOOLS 1-9++

Downloads Page for all your MCP tooling needs!

Hot Configthinkmeltprotonmail.com

How can you get amazing plotting capabilities out of a small LLM? Easy. Don't ask for the information back. What does that mean? We simply mute the json return, redirecting it to nice png graphs in a subdirectory, just not back to your LLM. It never sees the massive dataset that will kill most LLM contexts. By doing it this  way we let the MCP agent simply generate a viewing point /history and have robust calling capability!

This MCP is now easily pulled and run as a standalone docker container!

docker pull cnmcdee/mcp_matplot:latest
docker run -d \
  --name mcp_matplot \
  -p 0.0.0.0:5016:5016 \
  -v "$(pwd)/plots:/app/plots" \
  --restart unless-stopped \
  cnmcdee/mcp_matplot:latest

Let's get started!

• We are going to build a highly robust docker that has a lot of parts like scipy, numpy, matplotlib
• It must also be a very forgiving MCP Agent.  Just like humans - different LLM's may decide to make the tool calls in different ways.  To accomodate this we tried to make each tool end-point absolutely as accomodating as possible.

Secondly we need straight-pipe return html end-points so we defined /history as a viewing endpoint for seeing whatever your LLM's are drawing. You coverse with your LLM in your typical llama.cpp box and view the output at another box. For instance,

The code template is a specialized type of mcp namely:

@mcp.custom_route("/html", methods=["GET"]) , anyways..

Adding it to your llama.cpp is pretty easy. MCP Add Server.. click.

Then you will see in your list as:

• Finally we want it to just pass straight-shot matplotlib or similar code directly to the MCP agent, have it run the code and generate the output.  This is the only really good way to get it to  make large-detail custom graphing without taking out your small contexts.
• Yep we added some small cleaning tools that will allow you to clean out the production directory..
• It should be understood that the matplotlib library functions can accept varying sized arrays. So in many cases just passing the code will be the best way, it runs it / plots it. If you try passing raw stuff - you will run out of context very easily.

How do you use it? That's easy - just ask the LLM to use it! Remember all the large graphs and numbers go to the directory they are never returned to the LLM to overflow small contexts. This is by design..

Test the execute_matplot_lib code with some beautiful 3d plots that show landscapes

In the end this is the code, a little messy but if you work through it you can add to it as you see fit, naturally cutting and pasting portions of it into any LLM that will happily write the rest for you!

#region import directives
from fastmcp import FastMCP
from starlette.middleware import Middleware
from starlette.middleware.cors import CORSMiddleware
from starlette.responses import HTMLResponse
import uvicorn
import json
import matplotlib
import matplotlib.pyplot as plot
import io
import base64
import os
from datetime import datetime
import matplotlib.pyplot as plot
import io
import base64
import json
import os
from pathlib import Path
from datetime import datetime
import base64
from pathlib import Path
import json
import base64
import io
import traceback
from typing import List, Optional, Union, Dict, Any
import matplotlib.pyplot as plt
import subprocess, sys # Ensure it's imported at module level
#endregion
#region Settings and Dependencies
PLOTS_DIR = Path("plots")   # Make sure it's a Path object
PLOTS_DIR.mkdir(parents=True, exist_ok=True)
active_port = 5016
# Use non-interactive backend
matplotlib.use('Agg')
mcp = FastMCP(name="Matplotlib MCP Tool")
# Global list to track plots in the order they were created
plot_history = []
# Ensure plots directory exists
os.makedirs(PLOTS_DIR, exist_ok=True)
def ensure_dependencies():
    """Install a broad scientific Python stack to support 95%+ of LLM scientist calls."""
    required_packages = [
        "numpy", "pandas", "matplotlib", "scipy",
        "seaborn", "scikit-learn", "statsmodels",
        "plotly", "kaleido",  # plotly static image export
        "sympy",  # symbolic math / equations
        "pillow",  # image handling
    ]

    missing = []
    for pkg in required_packages:
        try:
            __import__(pkg.replace("-", "_").split(".")[0])  # handle names like scikit-learn
        except ImportError:
            missing.append(pkg)

    if missing:
        print(f"Installing scientific dependencies: {missing}")
        try:
            # Use --no-input to avoid interactive prompts
            subprocess.check_call([
                                      sys.executable, "-m", "pip", "install", "--quiet", "--no-input", "--upgrade"
                                  ] + missing)
            print("✅ Scientific dependencies installed successfully.")
        except subprocess.CalledProcessError as e:
            print(f"⚠️ Partial installation failure: {e}. Some advanced features may be limited.")
        except Exception as e:
            print(f"⚠️ Failed to auto-install packages: {e}")
    else:
        print("✅ All core scientific packages already available.")
    return True
# Run once when module loads
ensure_dependencies()
def save_plot_to_file(fig, plot_type="plot"):
    """Save plot to plots directory and return filename."""
    global plot_history

    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    counter = len(plot_history) + 1
    filename = f"plots/plot_{counter:03d}_{plot_type}_{timestamp}.png"

    fig.savefig(filename, bbox_inches='tight', dpi=200)
    plot_history.append({
        "filename": filename,
        "plot_type": plot_type,
        "timestamp": timestamp,
        "number": counter
    })
    return filename
#endregion
#region plot_2dplot
@mcp.tool()
def plot_2dplot(
    x: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    y: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Plot",
    xlabel: str = "X",
    ylabel: str = "Y",
    color: str = "blue"
) -> str:
    """
    Create a line plot. Very tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [x, y, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If we still have a dict in 'x', use it
        if isinstance(x, dict):
            params = x

        # Extract all possible values
        x = params.get('x', x)
        y = params.get('y', y)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)

        # Extra safety: try parsing string values that look like lists
        if isinstance(x, str) and x.startswith('['):
            try:
                x = json.loads(x)
            except Exception:
                pass
        if isinstance(y, str) and y.startswith('['):
            try:
                y = json.loads(y)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(x, 'tolist'):
            x = x.tolist()
        if hasattr(y, 'tolist'):
            y = y.tolist()

        # Final validation
        if x is None or y is None:
            return json.dumps({
                "success": False,
                "message": "Both 'x' and 'y' are required.",
                "error": "Missing x or y data",
                "received_params": {"x": str(type(x)), "y": str(type(y))}
            }, indent=2)

        if not isinstance(x, (list, tuple)) or not isinstance(y, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "x and y must be lists of numbers.",
                "error": f"Invalid types: x={type(x)}, y={type(y)}"
            }, indent=2)

        if len(x) != len(y):
            return json.dumps({
                "success": False,
                "message": "x and y must have the same length.",
                "error": f"Length mismatch: x={len(x)}, y={len(y)}"
            }, indent=2)

        # === Create the plot ===
        fig = plot.figure(figsize=(10, 6))
        plot.plot(x, y, color=color)
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True)

        filename = save_plot_to_file(fig, "line")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Line plot created and saved successfully.",
            "plot_type": "line",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create line plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dscatter
@mcp.tool()
def plot_2dscatter(
    x: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    y: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Scatter Plot",
    xlabel: str = "X",
    ylabel: str = "Y",
    color: str = "blue",
    size: float = 50,
    alpha: float = 0.7
) -> str:
    """
    Create a scatter plot using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [x, y, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If first argument is a dict, use it
        if isinstance(x, dict):
            params = x

        # Extract all possible values
        x = params.get('x', x)
        y = params.get('y', y)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        size = params.get('size', size)
        alpha = params.get('alpha', alpha)

        # Extra safety: try parsing string values that look like lists
        if isinstance(x, str) and x.startswith('['):
            try:
                x = json.loads(x)
            except Exception:
                pass
        if isinstance(y, str) and y.startswith('['):
            try:
                y = json.loads(y)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(x, 'tolist'):
            x = x.tolist()
        if hasattr(y, 'tolist'):
            y = y.tolist()

        # Final validation
        if x is None or y is None:
            return json.dumps({
                "success": False,
                "message": "Both 'x' and 'y' are required.",
                "error": "Missing x or y data",
                "received_params": {"x": str(type(x)), "y": str(type(y))}
            }, indent=2)

        if not isinstance(x, (list, tuple)) or not isinstance(y, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "x and y must be lists of numbers.",
                "error": f"Invalid types: x={type(x)}, y={type(y)}"
            }, indent=2)

        if len(x) != len(y):
            return json.dumps({
                "success": False,
                "message": "x and y must have the same length.",
                "error": f"Length mismatch: x={len(x)}, y={len(y)}"
            }, indent=2)

        # === Create the scatter plot ===
        fig = plot.figure(figsize=(10, 6))
        plot.scatter(x, y, color=color, s=size, alpha=alpha)
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True)

        filename = save_plot_to_file(fig, "scatter")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Scatter plot created and saved successfully.",
            "plot_type": "scatter",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create scatter plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dbar
@mcp.tool()
def plot_2dbar(
    x: Union[List[float], List[int], List[str], Dict[str, Any], str, None] = None,
    y: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Bar Plot",
    xlabel: str = "Categories",
    ylabel: str = "Values",
    color: str = "blue",
    alpha: float = 0.8
) -> str:
    """
    Create a bar plot using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [x, y, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If first argument is a dict, use it
        if isinstance(x, dict):
            params = x

        # Extract all possible values
        x = params.get('x', x)
        y = params.get('y', y)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        alpha = params.get('alpha', alpha)

        # Extra safety: try parsing string values that look like lists
        if isinstance(x, str) and (x.startswith('[') or x.startswith('{')):
            try:
                x = json.loads(x)
            except Exception:
                pass
        if isinstance(y, str) and (y.startswith('[') or y.startswith('{')):
            try:
                y = json.loads(y)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(x, 'tolist'):
            x = x.tolist()
        if hasattr(y, 'tolist'):
            y = y.tolist()

        # Final validation
        if x is None or y is None:
            return json.dumps({
                "success": False,
                "message": "Both 'x' and 'y' are required.",
                "error": "Missing x or y data",
                "received_params": {"x": str(type(x)), "y": str(type(y))}
            }, indent=2)

        if not isinstance(x, (list, tuple)) or not isinstance(y, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "x and y must be lists.",
                "error": f"Invalid types: x={type(x)}, y={type(y)}"
            }, indent=2)

        if len(x) != len(y):
            return json.dumps({
                "success": False,
                "message": "x and y must have the same length.",
                "error": f"Length mismatch: x={len(x)}, y={len(y)}"
            }, indent=2)

        # === Create the bar plot ===
        fig = plot.figure(figsize=(10, 6))
        plot.bar(x, y, color=color, alpha=alpha)
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True, axis='y')

        # Rotate x labels if they are strings (categories)
        if any(isinstance(label, str) for label in x):
            plot.xticks(rotation=45, ha='right')

        filename = save_plot_to_file(fig, "bar")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Bar plot created and saved successfully.",
            "plot_type": "bar",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create bar plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dbarh
@mcp.tool()
def plot_2dbarh(
        x: Union[List[float], List[int], List[str], Dict[str, Any], str, None] = None,
        y: Union[List[float], List[int], Dict[str, Any], str, None] = None,
        title: str = "Horizontal Bar Plot",
        xlabel: str = "Values",
        ylabel: str = "Categories",
        color: str = "blue",
        alpha: float = 0.8
) -> str:
    """
    Create a horizontal bar plot using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [x, y, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If first argument is a dict, use it
        if isinstance(x, dict):
            params = x

        # Extract all possible values
        x = params.get('x', x)
        y = params.get('y', y)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        alpha = params.get('alpha', alpha)

        # Extra safety: try parsing string values that look like lists
        if isinstance(x, str) and (x.startswith('[') or x.startswith('{')):
            try:
                x = json.loads(x)
            except Exception:
                pass
        if isinstance(y, str) and (y.startswith('[') or y.startswith('{')):
            try:
                y = json.loads(y)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(x, 'tolist'):
            x = x.tolist()
        if hasattr(y, 'tolist'):
            y = y.tolist()

        # Final validation
        if x is None or y is None:
            return json.dumps({
                "success": False,
                "message": "Both 'x' and 'y' are required.",
                "error": "Missing x or y data",
                "received_params": {"x": str(type(x)), "y": str(type(y))}
            }, indent=2)

        if not isinstance(x, (list, tuple)) or not isinstance(y, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "x and y must be lists.",
                "error": f"Invalid types: x={type(x)}, y={type(y)}"
            }, indent=2)

        if len(x) != len(y):
            return json.dumps({
                "success": False,
                "message": "x and y must have the same length.",
                "error": f"Length mismatch: x={len(x)}, y={len(y)}"
            }, indent=2)

        # === Create the horizontal bar plot ===
        fig = plot.figure(figsize=(10, 6))

        # For horizontal bars, we typically swap x and y roles
        # y = categories (on vertical axis), x = values (on horizontal axis)
        plot.barh(y, x, color=color, alpha=alpha)  # Note: barh(y, width=x)

        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True, axis='x')

        # Rotate y labels if they are long strings
        if any(isinstance(label, str) for label in y):
            plot.yticks(rotation=0)  # Usually no need to rotate for horizontal

        filename = save_plot_to_file(fig, "barh")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Horizontal bar plot created and saved successfully.",
            "plot_type": "barh",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create horizontal bar plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dhist
@mcp.tool()
def plot_2dhist(
    data: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Histogram",
    xlabel: str = "Value",
    ylabel: str = "Frequency",
    color: str = "blue",
    bins: int = 10,
    alpha: float = 0.7
) -> str:
    """
    Create a histogram using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        candidates = [data, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        if isinstance(data, dict):
            params = data

        # Extract values
        data = params.get('data', data)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        bins = params.get('bins', bins)
        alpha = params.get('alpha', alpha)

        # Parse stringified lists
        if isinstance(data, str) and (data.startswith('[') or data.startswith('{')):
            try:
                data = json.loads(data)
            except Exception:
                pass

        # Convert numpy/pandas
        if hasattr(data, 'tolist'):
            data = data.tolist()

        # Validation
        if data is None:
            return json.dumps({
                "success": False,
                "message": "'data' is required for histogram.",
                "error": "Missing data",
                "received_params": {"data": str(type(data))}
            }, indent=2)

        if not isinstance(data, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "data must be a list of numbers.",
                "error": f"Invalid type: data={type(data)}"
            }, indent=2)

        # === Create histogram ===
        fig = plot.figure(figsize=(10, 6))
        plot.hist(data, bins=bins, color=color, alpha=alpha)
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True, axis='y')

        filename = save_plot_to_file(fig, "hist")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Histogram created and saved successfully.",
            "plot_type": "hist",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create histogram.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dpie
@mcp.tool()
def plot_2dpie(
    values: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    labels: Union[List[str], Dict[str, Any], str, None] = None,
    title: str = "Pie Chart",
    colors: Union[List[str], None] = None
) -> str:
    """
    Create a pie chart using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        params: Dict[str, Any] = {}

        candidates = [values, labels, title, colors]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        if isinstance(values, dict):
            params = values

        values = params.get('values', values)
        labels = params.get('labels', labels)
        title = params.get('title', title)
        colors = params.get('colors', colors)

        if isinstance(values, str) and values.startswith('['):
            try:
                values = json.loads(values)
            except Exception:
                pass
        if isinstance(labels, str) and labels.startswith('['):
            try:
                labels = json.loads(labels)
            except Exception:
                pass

        if hasattr(values, 'tolist'):
            values = values.tolist()

        if values is None:
            return json.dumps({
                "success": False,
                "message": "'values' is required for pie chart.",
                "error": "Missing values"
            }, indent=2)

        if not isinstance(values, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "values must be a list of numbers.",
                "error": f"Invalid type: values={type(values)}"
            }, indent=2)

        # === Create pie chart ===
        fig = plot.figure(figsize=(10, 6))
        plot.pie(values, labels=labels, colors=colors, autopct='%1.1f%%', startangle=90)
        plot.title(title)
        plot.axis('equal')

        filename = save_plot_to_file(fig, "pie")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Pie chart created and saved successfully.",
            "plot_type": "pie",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create pie chart.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#region plot_2dboxplot
@mcp.tool()
def plot_2dboxplot(
    data: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Box Plot",
    xlabel: str = "Category",
    ylabel: str = "Value",
    color: str = "lightblue"
) -> str:
    """
    Create a box plot using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        params: Dict[str, Any] = {}

        candidates = [data, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        if isinstance(data, dict):
            params = data

        data = params.get('data', data)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)

        if isinstance(data, str) and data.startswith('['):
            try:
                data = json.loads(data)
            except Exception:
                pass

        if hasattr(data, 'tolist'):
            data = data.tolist()

        if data is None:
            return json.dumps({
                "success": False,
                "message": "'data' is required for boxplot.",
                "error": "Missing data"
            }, indent=2)

        if not isinstance(data, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "data must be a list.",
                "error": f"Invalid type: data={type(data)}"
            }, indent=2)

        # === Create boxplot ===
        fig = plot.figure(figsize=(10, 6))
        plot.boxplot(data, patch_artist=True, boxprops=dict(facecolor=color))
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True, axis='y')

        filename = save_plot_to_file(fig, "boxplot")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Box plot created and saved successfully.",
            "plot_type": "boxplot",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create box plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)

#endregion
#region plot_2dfill
@mcp.tool()
def plot_2dfill(
    x: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    y: Union[List[float], List[int], Dict[str, Any], str, None] = None,
    title: str = "Area Plot",
    xlabel: str = "X",
    ylabel: str = "Y",
    color: str = "blue",
    alpha: float = 0.5
) -> str:
    """
    Create a filled area plot using matplotlib plot.fill().
    Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [x, y, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If first argument is a dict, use it
        if isinstance(x, dict):
            params = x

        # Extract all possible values
        x = params.get('x', x)
        y = params.get('y', y)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        alpha = params.get('alpha', alpha)

        # Extra safety: try parsing string values that look like lists
        if isinstance(x, str) and (x.startswith('[') or x.startswith('{')):
            try:
                x = json.loads(x)
            except Exception:
                pass
        if isinstance(y, str) and (y.startswith('[') or y.startswith('{')):
            try:
                y = json.loads(y)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(x, 'tolist'):
            x = x.tolist()
        if hasattr(y, 'tolist'):
            y = y.tolist()

        # Final validation
        if x is None or y is None:
            return json.dumps({
                "success": False,
                "message": "Both 'x' and 'y' are required.",
                "error": "Missing x or y data",
                "received_params": {"x": str(type(x)), "y": str(type(y))}
            }, indent=2)

        if not isinstance(x, (list, tuple)) or not isinstance(y, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "x and y must be lists of numbers.",
                "error": f"Invalid types: x={type(x)}, y={type(y)}"
            }, indent=2)

        if len(x) != len(y):
            return json.dumps({
                "success": False,
                "message": "x and y must have the same length.",
                "error": f"Length mismatch: x={len(x)}, y={len(y)}"
            }, indent=2)

        # === Create the filled area plot ===
        fig = plot.figure(figsize=(10, 6))
        plot.fill(x, y, color=color, alpha=alpha)
        plot.plot(x, y, color=color)  # Add line on top of fill
        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True)

        filename = save_plot_to_file(fig, "fill")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Area (fill) plot created and saved successfully.",
            "plot_type": "fill",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create area plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)

#endregion
#region plot_2dviolinplot
@mcp.tool()
def plot_2dviolinplot(
        data: Union[List[float], List[int], List[List[float]], Dict[str, Any], str, None] = None,
        title: str = "Violin Plot",
        xlabel: str = "Category",
        ylabel: str = "Value",
        color: str = "lightblue",
        showmeans: bool = True,
        showmedians: bool = True
) -> str:
    """
    Create a violin plot using matplotlib. Highly tolerant of different LLM calling styles.
    """
    try:
        # === Aggressive parameter extraction for LLM flexibility ===
        params: Dict[str, Any] = {}

        # Try to detect if any argument is a dict or JSON string
        candidates = [data, title, xlabel, ylabel, color]

        for candidate in candidates:
            if isinstance(candidate, dict):
                params = candidate
                break
            elif isinstance(candidate, str) and candidate.strip().startswith("{"):
                try:
                    params = json.loads(candidate)
                    break
                except Exception:
                    continue

        # If first argument is a dict, use it
        if isinstance(data, dict):
            params = data

        # Extract all possible values
        data = params.get('data', data)
        title = params.get('title', title)
        xlabel = params.get('xlabel', xlabel)
        ylabel = params.get('ylabel', ylabel)
        color = params.get('color', color)
        showmeans = params.get('showmeans', showmeans)
        showmedians = params.get('showmedians', showmedians)

        # Extra safety: try parsing string values that look like lists
        if isinstance(data, str) and (data.startswith('[') or data.startswith('{')):
            try:
                data = json.loads(data)
            except Exception:
                pass

        # Convert numpy/pandas objects
        if hasattr(data, 'tolist'):
            data = data.tolist()

        # Validation
        if data is None:
            return json.dumps({
                "success": False,
                "message": "'data' is required for violin plot.",
                "error": "Missing data",
                "received_params": {"data": str(type(data))}
            }, indent=2)

        # Ensure data is a list (can be list of lists for multiple violins)
        if not isinstance(data, (list, tuple)):
            return json.dumps({
                "success": False,
                "message": "data must be a list of numbers or list of lists.",
                "error": f"Invalid type: data={type(data)}"
            }, indent=2)

        # === Create the violin plot ===
        fig = plot.figure(figsize=(10, 6))

        # Violin plot can accept list of arrays
        violin_parts = plot.violinplot(data, showmeans=showmeans, showmedians=showmedians)

        # Color the violins
        for pc in violin_parts['bodies']:
            pc.set_facecolor(color)
            pc.set_edgecolor('black')
            pc.set_alpha(0.7)

        plot.title(title)
        plot.xlabel(xlabel)
        plot.ylabel(ylabel)
        plot.grid(True, axis='y')

        filename = save_plot_to_file(fig, "violin")

        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight')
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')
        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": "Violin plot created and saved successfully.",
            "plot_type": "violin",
            "image": img_base64,
            "filename": filename,
            "plot_number": len(plot_history) if 'plot_history' in globals() else 0,
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create violin plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)

#endregion

#region execute_matplotlib_code

@mcp.tool()
def execute_matplotlib_code(
    code: str,
    filename: str = None,
    title: str = "Matplotlib Plot",
    dpi: int = 150
) -> str:
    """
    Robust matplotlib code executor using save_plot_to_file().
    """
    try:
        if not code or not isinstance(code, str):
            raise ValueError("Code parameter is required.")

        # Shared result container
        exec_result = {"saved_files": [], "final_filename": None}

        local_env: Dict[str, Any] = {
            "plt": plt,
            "os": os,
            "save_plot_to_file": save_plot_to_file,
            "exec_result": exec_result,
            "__builtins__": __builtins__,
            "PLOTS_DIR": PLOTS_DIR,   # pass the Path object
        }

        enforced_code = f"""
import matplotlib.pyplot as plt
import os
from pathlib import Path

# User's code
{code}

saved_filenames = []

if plt.get_fignums():
    for i, fig_num in enumerate(plt.get_fignums()):
        fig = plt.figure(fig_num)
        suffix = "plot" if i == 0 else f"plot_{{i}}"
        fname = save_plot_to_file(fig, suffix)
        saved_filenames.append(fname)
        print(f"Saved plot: {{fname}}")
else:
    fig = plt.figure(figsize=(10, 6))
    plt.title('{title}')
    plt.grid(True)
    fname = save_plot_to_file(fig, "default")
    saved_filenames.append(fname)
    print(f"Saved default plot: {{fname}}")

exec_result["saved_files"] = saved_filenames
exec_result["final_filename"] = saved_filenames[-1] if saved_filenames else None
print("FINAL_FILENAME:" + (exec_result["final_filename"] or "unknown"))
"""

        exec_globals = {"__builtins__": __builtins__}
        exec(enforced_code, exec_globals, local_env)

        final_filename = exec_result.get("final_filename") or "unknown_plot.png"

        return json.dumps({
            "success": True,
            "message": "Matplotlib code executed successfully.",
            "filename": final_filename,
            "saved_files": exec_result.get("saved_files", []),
            "full_path": str(PLOTS_DIR / final_filename),
            "image": None,
            "image_url": f"/plots/{final_filename}",
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to execute matplotlib code.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)
#endregion
#endregion


#region plot_3dplot
@mcp.tool()
def plot_3d(
    x: Union[List, Dict, str, None] = None,
    y: Union[List, Dict, str, None] = None,
    z: Union[List, Dict, str, None] = None,
    title: str = "3D Plot",
    xlabel: str = "X",
    ylabel: str = "Y",
    zlabel: str = "Z",
    color: str = "blue",
    plot_type: str = "scatter"  # "scatter", "line", "surface"
) -> str:
    """
    Robust 3D/2D plot tool. Handles parameters passed as:
    - Individual arguments
    - Single dictionary (common in tool calling)
    - JSON-stringified values
    - Mixed formats
    """
    try:
        # === 1. Aggressive parameter extraction ===
        params: Dict[str, Any] = {}

        # Check each argument for a dict or JSON dict string
        candidates = [x, y, z, title, xlabel, ylabel, zlabel, color, plot_type]
        for cand in candidates:
            if isinstance(cand, dict):
                params = cand
                break
            if isinstance(cand, str) and cand.strip().startswith(('{', '[')):
                try:
                    parsed = json.loads(cand)
                    if isinstance(parsed, dict):
                        params = parsed
                        break
                except Exception:
                    continue

        # If x itself is a dict, treat it as the full params (very common pattern)
        if isinstance(x, dict):
            params = x

        # === 2. Extract and override with params ===
        def extract(key, default):
            return params.get(key, default)

        x = extract('x', x)
        y = extract('y', y)
        z = extract('z', z)
        title = extract('title', title)
        xlabel = extract('xlabel', xlabel)
        ylabel = extract('ylabel', ylabel)
        zlabel = extract('zlabel', zlabel)
        color = extract('color', color)
        plot_type = extract('plot_type', plot_type).lower().strip()

        # === 3. Robust data normalization ===
        def normalize_data(data: Any) -> List:
            if data is None:
                return None
            if isinstance(data, str):
                data = data.strip()
                if data.startswith(('[', '{')):
                    try:
                        data = json.loads(data)
                    except Exception:
                        pass
            # Convert numpy/pandas
            if hasattr(data, 'tolist'):
                data = data.tolist()
            if isinstance(data, (list, tuple)):
                # Handle list of dicts or deeply nested cases if needed
                return [float(v) if isinstance(v, (int, float, str)) and str(v).replace('.', '', 1).isdigit() else v
                        for v in data]
            return data

        x = normalize_data(x)
        y = normalize_data(y)
        z = normalize_data(z)

        # === 4. Auto-detect dimensionality ===
        is_3d = z is not None and len([v for v in (x, y, z) if v is not None]) >= 3

        # === 5. Create plot ===
        fig = plot.figure(figsize=(12, 9))
        ax = fig.add_subplot(111, projection='3d' if is_3d else None)

        if is_3d:
            if plot_type == "line":
                ax.plot(x, y, z, color=color)
            else:  # default scatter
                ax.scatter(x, y, z, color=color, s=50)
            ax.set_zlabel(zlabel)
        else:
            # 2D fallback
            if plot_type == "line":
                ax.plot(x, y, color=color)
            else:
                ax.scatter(x, y, color=color)

        ax.set_title(title)
        ax.set_xlabel(xlabel)
        ax.set_ylabel(ylabel)
        ax.grid(True)

        # Save to file (your existing helper)
        filename = save_plot_to_file(fig, "3d" if is_3d else "2d")

        # Base64 for immediate return
        buf = io.BytesIO()
        fig.savefig(buf, format='png', bbox_inches='tight', dpi=150)
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode('utf-8')

        plot.close(fig)

        return json.dumps({
            "success": True,
            "message": f"{'3D' if is_3d else '2D'} {plot_type} plot created successfully.",
            "plot_type": "3d_" + plot_type if is_3d else plot_type,
            "is_3d": is_3d,
            "image": img_base64,
            "filename": filename,
            "plot_number": len(globals().get('plot_history', [])),
            "error": None
        }, indent=2)

    except Exception as e:
        return json.dumps({
            "success": False,
            "message": "Failed to create 3D/2D plot.",
            "error": str(e),
            "traceback": traceback.format_exc()
        }, indent=2)

#endregion


@mcp.tool()
def remove_active_plots():
    """
    Remove all entries from plot_history and delete the corresponding image files.
    """
    try:
        deleted_files = 0
        history_entries = 0

        if 'plot_history' in globals() and plot_history:
            history_entries = len(plot_history)

            for plot in plot_history[:]:  # copy to avoid modification during iteration
                filename = plot.get("filename")
                if filename:
                    file_path = PLOTS_DIR / filename
                    if file_path.exists():
                        file_path.unlink()
                        deleted_files += 1

            plot_history.clear()

            result = {
                "success": True,
                "message": f"Removed {history_entries} plot history entries and deleted {deleted_files} image files.",
                "history_cleared": history_entries,
                "files_deleted": deleted_files
            }
        else:
            result = {
                "success": True,
                "message": "plot_history was empty or not available. No action taken.",
                "history_cleared": 0,
                "files_deleted": 0
            }

        return json.dumps(result, indent=2)

    except Exception as e:
        error_result = {
            "success": False,
            "message": f"Error while removing active plots: {str(e)}",
            "history_cleared": 0,
            "files_deleted": 0
        }
        return json.dumps(error_result, indent=2)
@mcp.tool()
def clean_plots_directory():
    """
    Completely clean the plots directory by deleting all files inside it.
    """
    try:
        if not PLOTS_DIR.exists():
            result = {
                "success": False,
                "message": f"Plots directory {PLOTS_DIR} does not exist.",
                "files_deleted": 0
            }
            return json.dumps(result, indent=2)

        files_before = [f for f in PLOTS_DIR.glob("*.*") if f.is_file()]
        file_count = len(files_before)

        if file_count == 0:
            result = {
                "success": True,
                "message": "Plots directory is already empty.",
                "files_deleted": 0
            }
            return json.dumps(result, indent=2)

        # Delete all files
        for file_path in files_before:
            file_path.unlink()

        # Clear history if it exists
        if 'plot_history' in globals():
            plot_history.clear()

        result = {
            "success": True,
            "message": f"Successfully cleaned plots directory. Removed {file_count} files.",
            "files_deleted": file_count
        }
        return json.dumps(result, indent=2)

    except Exception as e:
        error_result = {
            "success": False,
            "message": f"Error while cleaning plots directory: {str(e)}",
            "files_deleted": 0
        }
        return json.dumps(error_result, indent=2)

@mcp.tool()
def get_plot_history():
    """Return history of all generated plots."""
    try:
        return json.dumps({
            "success": True,
            "total_plots": len(plot_history),
            "history": plot_history,
            "error": None
        })
    except Exception as e:
        return json.dumps({
            "success": False,
            "error": str(e)
        })
# ====================== Direct HTML for Browser ======================
@mcp.custom_route("/html", methods=["GET"])
async def serve_html(request):
    """Serves a clean HTML page directly to web browsers."""
    html_content = """<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>MCP Matplotlib Server</title>
    <style>
        body { font-family: system-ui, Arial, sans-serif; margin: 0; padding: 40px; background: #f0f2f5; }
        .card { max-width: 900px; margin: 40px auto; background: white; padding: 40px; border-radius: 12px; box-shadow: 0 8px 25px rgba(0,0,0,0.1); }
    </style>
</head>
<body>
    <div class="card">
        <h1>✅ MCP Matplotlib Server Ready</h1>
        <p>Plots are automatically saved to the <strong>plots/</strong> folder in creation order.</p>
        <p>Use <strong>get_plot_history</strong> tool to see all generated plots.</p>
    </div>
</body>
</html>"""
    return HTMLResponse(content=html_content)
@mcp.custom_route("/plots", methods=["GET"])
async def serve_plots_gallery(request):
    """Display all saved plots in a clean HTML gallery in the order they were created."""
    try:
        plots_html = ""
        for plot in plot_history:
            rel_path = plot["filename"]
            plots_html += f"""
            <div class="plot-card">
                <h3>Plot #{plot['number']} — {plot['plot_type'].title()}</h3>
                <p><small>{plot['timestamp']}</small></p>
                <img src="{rel_path}" alt="{plot['plot_type']}" style="max-width:100%; border:1px solid #ddd; border-radius:8px;">
                <hr>
            </div>
            """

        if not plot_history:
            plots_html = "<p><em>No plots generated yet.</em></p>"

        html_content = f"""<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>MCP Plot Gallery</title>
    <style>
        body {{ font-family: system-ui, Arial, sans-serif; margin: 0; padding: 20px; background: #f0f2f5; }}
        .container {{ max-width: 1200px; margin: 0 auto; }}
        .plot-card {{ background: white; padding: 20px; margin-bottom: 30px; border-radius: 12px; box-shadow: 0 4px 15px rgba(0,0,0,0.1); }}
        h1 {{ text-align: center; color: #1a365d; }}
        img {{ display: block; margin: 15px auto; }}
    </style>
</head>
<body>
    <div class="container">
        <h1>📊 MCP Generated Plots Gallery</h1>
        <p style="text-align:center;">Total plots: <strong>{len(plot_history)}</strong> (in creation order)</p>
        {plots_html}
    </div>
</body>
</html>"""

        return HTMLResponse(content=html_content)

    except Exception as e:
        error_html = f"""<!DOCTYPE html>
<html><body><h1>Error loading gallery</h1><p>{str(e)}</p></body></html>"""
        return HTMLResponse(content=error_html)
@mcp.custom_route("/history", methods=["GET"])
async def serve_plots_history(request):
    """Scan the plots directory and generate self-contained HTML with embedded plots."""
    try:
        if not PLOTS_DIR.exists():
            raise FileNotFoundError(f"Plots directory not found: {PLOTS_DIR}")

        image_files = sorted(
            list(PLOTS_DIR.glob("*.png")) + list(PLOTS_DIR.glob("*.jpg")) + list(PLOTS_DIR.glob("*.jpeg")),
            key=lambda x: x.stat().st_ctime,
            reverse=True
        )

        plots_html = ""
        for idx, file_path in enumerate(image_files, 1):
            filename = file_path.name
            timestamp = datetime.fromtimestamp(file_path.stat().st_ctime).strftime("%Y%m%d_%H%M%S")

            # Detect plot type
            plot_type = "Plot"
            name_lower = filename.lower()
            if "line" in name_lower:
                plot_type = "Line"
            elif "bar" in name_lower:
                plot_type = "Bar"
            elif "pie" in name_lower:
                plot_type = "Pie"
            elif "scatter" in name_lower:
                plot_type = "Scatter"

            # Embed image as base64
            with open(file_path, "rb") as f:
                img_data = base64.b64encode(f.read()).decode("utf-8")
            img_src = f"data:image/png;base64,{img_data}"  # change to jpeg if needed

            plots_html += f"""
            <div class="plot-card">
                <h3>Plot #{idx} — {plot_type}</h3>
                <p><small>{timestamp}</small><br><small>{filename}</small></p>
                <img src="{img_src}" alt="{plot_type}" style="max-width:100%; border:1px solid #ddd; border-radius:8px;">
                <hr>
            </div>
            """

        if not image_files:
            plots_html = "<p><em>No image files found in the directory.</em></p>"

        html_content = f"""<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>MCP Plots History</title>
    <style>
        body {{ font-family: system-ui, Arial, sans-serif; margin: 0; padding: 20px; background: #f0f2f5; }}
        .container {{ max-width: 1200px; margin: 0 auto; }}
        .plot-card {{ background: white; padding: 20px; margin-bottom: 30px; border-radius: 12px; box-shadow: 0 4px 15px rgba(0,0,0,0.1); }}
        h1 {{ text-align: center; color: #1a365d; }}
        img {{ display: block; margin: 15px auto; }}
    </style>
</head>
<body>
    <div class="container">
        <h1>📊 MCP Plots Directory History</h1>
        <p style="text-align:center;">Total plots: <strong>{len(image_files)}</strong> (newest first)</p>
        {plots_html}
    </div>
</body>
</html>"""

        return HTMLResponse(content=html_content)

    except Exception as e:
        error_html = f"""<!DOCTYPE html>
<html><body><h1>Error loading history</h1><p>{str(e)}</p></body></html>"""
        return HTMLResponse(content=error_html)
@mcp.custom_route("/test", methods=["GET"])
async def serve_plots_gallery(request):
    """Display all saved plots in a clean HTML gallery with embedded plots."""
    try:
        plots_html = ""
        for plot in plot_history:
            filename = plot["filename"]
            file_path = Path(PLOTS_DIR) / filename  # PLOTS_DIR must be defined

            if file_path.exists():
                # Read and embed as base64
                with open(file_path, "rb") as f:
                    img_data = base64.b64encode(f.read()).decode("utf-8")
                img_src = f"data:image/png;base64,{img_data}"
            else:
                img_src = ""  # fallback

            plots_html += f"""
            <div class="plot-card">
                <h3>Plot #{plot['number']} — {plot['plot_type'].title()}</h3>
                <p><small>{plot['timestamp']}</small></p>
                <img src="{img_src}" alt="{plot['plot_type']}" style="max-width:100%; border:1px solid #ddd; border-radius:8px;">
                <hr>
            </div>
            """

        if not plot_history:
            plots_html = "<p><em>No plots generated yet.</em></p>"

        html_content = f"""<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>MCP Plot Gallery</title>
    <style>
        body {{ font-family: system-ui, Arial, sans-serif; margin: 0; padding: 20px; background: #f0f2f5; }}
        .container {{ max-width: 1200px; margin: 0 auto; }}
        .plot-card {{ background: white; padding: 20px; margin-bottom: 30px; border-radius: 12px; box-shadow: 0 4px 15px rgba(0,0,0,0.1); }}
        h1 {{ text-align: center; color: #1a365d; }}
        img {{ display: block; margin: 15px auto; }}
    </style>
</head>
<body>
    <div class="container">
        <h1>📊 MCP Generated Plots Gallery</h1>
        <p style="text-align:center;">Total plots: <strong>{len(plot_history)}</strong> (in creation order)</p>
        {plots_html}
    </div>
</body>
</html>"""

        return HTMLResponse(content=html_content)

    except Exception as e:
        error_html = f"""<!DOCTYPE html>
<html><body><h1>Error loading gallery</h1><p>{str(e)}</p></body></html>"""
        return HTMLResponse(content=error_html)

active_port = 5016
# ── Server Startup ─────────────────────────────────────────────────────
if __name__ == "__main__":
    middleware = [
        Middleware(
            CORSMiddleware,
            allow_origins=["*"],
            allow_credentials=True,
            allow_methods=["*"],
            allow_headers=["*"],
        )
    ]

    app = mcp.http_app(
        path="/mcp",
        middleware=middleware,
        stateless_http=True,      # ← This fixes the session ID error
        # transport="http"        # Optional: force plain HTTP transport
    )

    print("🚀 MCP Matplotlib Server started!")
    print("→ plots saved to: ./plots/")
    print(f"→ MCP Endpoint:    http://localhost:{active_port}/mcp")
    print(f"→ Gallery:         http://localhost:{active_port}/mcp/plots")
    print(f"→ Info Page:       http://localhost:{active_port}/mcp/html")

    uvicorn.run(
        app,
        host="0.0.0.0",
        port=active_port,
        log_level="info"
    )

Once that is done we will look at dockerizing it.. so requirements.txt will become:

fastapi
uvicorn[standard]
starlette
fastmcp
matplotlib
numpy
pandas
scipy
seaborn
scikit-learn
statsmodels
plotly
kaleido
sympy
pillow
python-multipart

And a nice Dockerfile to build an image would then naturally become:

# =============================================================================
# MCP Matplotlib Server - Docker Image
# =============================================================================
FROM python:3.11-slim

WORKDIR /app

# System dependencies for matplotlib + plotly
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libglib2.0-0 libsm6 libxext6 libxrender-dev libfontconfig1 \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Ensure plots directory exists
RUN mkdir -p /app/plots

# Copy application
COPY app.py .

EXPOSE 5016

ENV PYTHONUNBUFFERED=1 \
    MPLBACKEND=Agg

# Health check
HEALTHCHECK CMD curl --fail http://localhost:5016/mcp/html || exit 1

# Explicitly run app.py
CMD ["python", "app.py"]

You would be able to stand it up with:

docker build . -t mcp_matplot --no-cache

This is going to make a pretty large image! Ah! My bloat! No it's all good,

To run it typically - just use the bash snippet at the top of this!

It worked! It seriously worked! We took one of the world's most advanced localLLM configurations and proved that it can run a production capable serious level  assistant.  Let's define what that actually means.

• A production assistant is a powerful coding and research assistant. No it won't generate 1000 T/s, but on a $200 house GPU to run a 65,000 Token Context W/35t per second generation is insanely powerful. To have it cleanly write and start asteroids with 5-6 prompts is very impressive. After about 30 Tokens/s and fast response times it looks fluid, it moves as fast as your attention span. It's very good!
• The cost of this is free. It will be able to go on the internet - look stuff up, report back to you. You will also become a competent life-long maintainer of LLM's. Companies will love you!
• Save your money.
• We are using one of the latest gemma models namely gemma-4-12B-it-qat-UD-Q4_K_XL.gguf

wget https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF?show_file_info=gemma-4-12B-it-qat-UD-Q4_K_XL.gguf

• We ran this on a Ryzen 5 2600 w/16GB RAM on a 3060ti with only 8GB. This entire machine dumps on facebook for < $400.
• We were able to launch with a 65535 Token context by using the correct Moe and the right configuration.
• We added in a lot of MCP power-ups into this to pull it all off.  How that works is it starts a job, saves it to the process-manager, the next prompt picks it up and continues working on it.
• We used the specialized 'The Tom' fork of llama.cpp with turbo3 for both --cache-type-k turbo3 and --cache-type-k turbo3.

The Build

Have patience! This can take some effort, but in essence you can do this too!

Step 1.  Install Your StudentLLM

Get all your drivers / cmake / nvcc / and your custom fork installed. Just follow the StudentLLM to completion and come back here for the model setup!

• Please understand it does the very advanced turboquant fork of llama.cpp. You really need this because you need your turboquant compression on your caches. Got it!?

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

CheckList!

• You have nvidia-drivers, nvidia-smi, nvcc, and The Tom turboquant fork, of llama.cpp done? It is all in the guide above. Just work through it!

If you have all that awesome!

Create a  ~/models folder where you have downloaded your model and put in it this script /gemma_c_3060.sh which you will need to make executable with chmod +x gemma_c_3060.sh naturally..

Pulling your model, repeating for clarity:

wget https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF?show_file_info=gemma-4-12B-it-qat-UD-Q4_K_XL.gguf

/usr/bin/llama-server \
--jinja \
-m /home/c/models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--ctx-size 65535 \
--n-gpu-layers -1 \
--cache-type-k turbo3 \
--cache-type-v turbo3 \
--no-mmap \
--flash-attn on \
--override-tensor "\.ffn_.*_exps\.weight=CPU" \
-fa on \
--parallel 1 \
--batch-size 512 \
--ubatch-size 512 \
--threads 12 \
--temp 0.7 \
--top-p 0.9 \
--top-k 40 \
--repeat-penalty 1.15 \
--min-p 0.1 \
--host 0.0.0.0 --port 8080

If it is running it will look like this on your port 8080 http://192.168.1.<your ip>:8080

If you open up another terminal and type watch nvidia-smi it will show you as it goes.

There are a LOT of parts that come together to pull this off, lets review some parts of it.

• We want to thank The Tom and his wonderful TurboQuant fork that enabled kv_cache compression.  Without it none of this would of been possible! Thanks Tom!
• --cache-type-k turbo3 and --cache-type-v turbo3

GitHub - TheTom/turboquant_plus

Contribute to TheTom/turboquant_plus development by creating an account on GitHub.

GitHubTheTom

• We used specialized off-loading moe (mixture of experts) that enabled the lighter models to offload to the CPU, and the heavier layers to stay on the GPU --override-tensor "\.ffn_.*_exps\.weight=CPU" \
• We added a repeat penalty to make sure that halucinations of the model were kept to a minimum.

Step 2. Adding MCP Powerups!

Next you will need some MCP agentic agents. They allow the gemma model to test it's own code. Just work through each one, after you get your docker installed.

• Process Manager (very important - it will allow you to continue a context across a series of prompts and will give you code-drops!

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

It can be easily installed and run with docker w/

docker pull docker.io/cnmcdee/mcp-process-manager:latest
docker run -d --name mcp-process-manager --restart unless-stopped -e "FLASH_ENV=production" -p 0.0.0.0:5008:5008 cnmcdee/mcp-process-manager:latest

• Javascript Node MCP (This very powerful agent will allow your LLM to call and test it's javascript / html code!

Agentic Server Primer: Llama.cpp MCP Lesson 5: Adding javascript via a Python api plugin.

We go through a full working example of creating your own MCP tools.

Hot Configthinkmeltprotonmail.com

It can be easily installed and run with docker w/

docker pull docker.io/cnmcdee/mcp-javascript:latest
docker run -d --name mcp-javascript --restart unless-stopped -e "FLASH_ENV=production" -p 0.0.0.0:5003:5003 cnmcdee/mcp-javascript:latest

• Python super-venv. This powerful MCP agent will allow your LLM to build whatever environment it needs on the fly.  We tested it and this configuration worked very well, installing obscure libraries.

Agentic Server Primer: Llama.cpp MCP Lesson 3: Adding Python Tooling Capability To your HouseLLM.

Agentic Server Primer: Llama.cpp MCP Lesson 3: Python

Hot Configthinkmeltprotonmail.com

It can be easliy installed and run with docker w/

docker pull cnmcdee/mcp-python:latest
docker run -d --name mcp-python --restart unless-stopped -e "FLASH_ENV=production" -p 0.0.0.0:5015:5015 cnmcdee/mcp-python:latest

Adding these was simply a matter of clicking on MCP servers and inputting their IP with always the following /mcp

Exceptionally  Good Tool Calling

• We were highly impressed how fluid and functional the tool calling is with this model. It was able to read and give a summation of cnn.com in about 26 seconds, generating a 38T/s. That's impressive!   Recall about 2 years ago this required about $70,000 in equipment and many models didn't even have the capability to look stuff up.  
  

Asking it to write Asteroids (The New Benchie)

• Asteroids is becoming the 3d-printer benchie  A benchie-boat is a 'frame of test reference' for a 3d-printer just as asking your LLM to see if it can write a rudimentary asteroids game.
• 2 years ago this required $70,000 in equipment and a H200. Today we just proved you can seriously work on these types of projects using a $400 left over computer.

Why it works!

• The real key in this is the process manager mcp above that can save work-points. The LLM can start a task, and upon further prompting or even a whole new context.
• A llm is significantly enhanced by the  ability to correctly tool-call and check it's output. This showed very strong tool calling.

Adding LLMQP and getting it coding all night!

• Finally if you need a prompt-babysitter this is it. You can setup 20 prompts - goto bed and have it work on your stuff all night.

docker pull docker.io/cnmcdee/llmqueue:latest
docker run -d --name mcp-llmqueue --restart unless-stopped -p 0.0.0.0:5012:5012 cnmcdee/llmqueue:latest

LLMQP Drops! A New Queue Dispatcher. Let your LLM CODE ALL NIGHT.

LLM Queue Dispatcher. A Powerful Harness Drop will queue your localLLM all night and keep it working!

Hot Configthinkmeltprotonmail.com

We saw this post and it piqued our interest. A production-assistant level LLM that claimed it finally worked really well on a 8GB VRAM cards aka 3060ti?  This would make mass-adoption reachable. Now a $200 GPU could be working for you, if it can do decent tool-calling  and support kv-cache (Turboquant) it could be the next game-changer in the LLM revolutions!

Alok (@analogalok) on X

gemma-4-12B-agentic-fable5-composer2.5 V2 is out. the agentic upgrade to the model trained on Fable 5’s reasoning. Running it now with TurboQuant llama.cpp on a single RTX 4060( 8 GB VRAM) at 30 tokens/second with full 25000 context and reasoning: # The benchmarks v2 is built https://t.co/XP15TN2…

X (formerly Twitter)Alok

• Benchmarked 360% higher than standard Gemma4 (claimed)
• Tuned for agentic / coding tasks.
• Runs on a 3060ti..

Pre-Install Supports

If you need a full walk-through guide to installing one of the world's most advanced setups in llama.cpp this guide will carefully walk you through the whole process.

• Learn how to build and manager your own LLM and all the drivers and GPU supports.

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

Some direct model links:

3060ti Configurations - 64K Maximum Context Q3_K_M (3-bit Quantization)

• We tested this on a 4080, but screenshot it to see how it actually fit..
• It rolled into a repetitive loop so we added --repeat-penalty 1.12

/usr/bin/llama-server \
  --jinja \
  -m /home/c/models/gemma4-v2-Q3_K_M.gguf \
  --ctx-size 65536 \
  --n-gpu-layers 99 \
  --cache-type-k turbo3 \
  --cache-type-v turbo3 \
  --no-mmap \
  -fa on \
  --parallel 1 \
  --batch-size 512 \
  --ubatch-size 512 \
  --threads 12 \
  --repeat-penalty 1.12 \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --host 0.0.0.0 --port 8080

Results - About 7196 MB on load / BLASTING at 71 Tokens/s out (on a 4080)

Note - tool calling did not work, or was really problematic. We really tried a bunch of stuff. Summary great for local assistantant lookups. It's very good.

Standard Configuration - Gemma4-v2-Q8_0 without MTP (40 Token's/s) - Not Bad!

• We recycled the following configuration, tuned it up a bit, and since this is a smaller model it left lots of room
• It worked very well with turbo3 \ turbo4 quantization and integrated cleanly with mtp (Multiple Token Prediction)
• It presented some breaks and errors. So it's not quite there.

/usr/bin/llama-server --jinja \
-m /home/c/models/gemma4-v2-Q8_0.gguf \
--ctx-size 16384 \
--n-gpu-layers 99 \
--no-mmap -fa on \
--jinja \
--temp 1.0 --top-p 0.95 --top-k 64 \
--host 0.0.0.0 --port 8080 \
--cache-type-k turbo3 \
--cache-type-v turbo4
--context-shift \
--spec-type draft-mtp \
--spect-draft-n-max 4

• It ran at about 40 T/s, but was the tool calling fixed??

We are using powerful agentic tools that are completely opensource - check them out!

Conclusion.

This is a very good local llm, but we tried extensively to get any tooling calls to work. It either seemed to hallucinate a lot - or did not make the tooling calls at all. With no tooling enabled at all it seemed to work quite well.  However with no tools given to it - it would produce very good quality assistant suggestions very quickly - at about 70 tokens/s which is fluid and capable for most people's needs!

We are still looking!

Things are moving incredibly fast in the LLM world - as people are now able to push out tools in hours that used to take years (and most of them in the past never really got done.)  

We constantly want to promote sites that track this information - so we really want to showcase producthunt.com - a site showcasing stuff dropping daily!

There are people creating some really cool products, sites, and programs - check this out!

We have had Grok 4 spit up a javascript RSS feed parser. Use this to dissect any RSS web page.

Save your Context and Come Back

This process manager is very powerful in that your LLM can now save it's work and spread it across many contexts.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

Get your LLM coding all Night! LLMQP

This LLM will enable you to queue multiple prompts which will execute one after another.

LLMQP Drops! A New Queue Dispatcher. Let your LLM CODE ALL NIGHT.

LLM Queue Dispatcher. A Powerful Harness Drop will queue your localLLM all night and keep it working!

Hot Configthinkmeltprotonmail.com

We had so many issues troubleshooting and getting our Nvidia drivers to work with our particular kernel of Linux (ParrotOS latest) that we wrote this guide.  Pretty much every LLM needs the full suite of drivers/nvcc etc,  so this  guide might help you - with your current Linux kernel. You can ask a SOTA level model that will get you most of the way, but  in the end it is these recipes that work. This is what worked for us.

• Because we are getting into cuda nvcc compiler development this guide will cover the basics of setting up a cuda-toolkit on top of custom compiling the very latest drivers.

In our case we are running a very recent version of Linux, and all the standard nvidia drivers were consistently breaking.  We were seeing constant errors at /var/log/nvdia-installer.log Our errors:

┌─[✗]─[c@parrot]─[~]
└──╼ $uname -a
Linux parrot 7.0.9+parrot7-amd64 #1 SMP PREEMPT_DYNAMIC Parrot 7.0.9-1parrot1 (2026-05-28) x86_64 GNU/Linux

Firstly we purged out the drivers, and installed some supports:

sudo apt purge nvidia*
sudo apt update
sudo apt install dwarves libelf-dev build-essential linux-headers-$(uname -r)

Next we will work from this github:

• The purpose of this is to install the kernel level drivers that sit under the nvidia main drivers.

GitHub - NVIDIA/open-gpu-kernel-modules: NVIDIA Linux open GPU kernel module source

NVIDIA Linux open GPU kernel module source. Contribute to NVIDIA/open-gpu-kernel-modules development by creating an account on GitHub.

GitHubNVIDIA

So:

Part 1. Getting to nvidia-smi

git clone https://github.com/NVIDIA/open-gpu-kernel-modules
cd open-gpu-kernel-modules/

Next:

make modules -j$(nproc)
sudo make modules_install -j$(nproc)

At this point you may reach errors thus:

sudo apt install mokutil && mokutil --sb-state 
sudo depmod -a
sudo modprobe nvidia

Installing the main Nvidia 6.10 drivers w/ --no-kernel-modules Option

• At this point we will need to get the latest main drivers and specifically install them as in:

wget https://us.download.nvidia.com/XFree86/Linux-x86_64/610.43.02/NVIDIA-Linux-x86_64-610.43.02.run
chmod +x NVIDIA-Linux-x86_64-610.43.02.run
sudo ./NVIDIA-Linux-x86_64-610.43.02.run --no-kernel-modules

You will get a warning where you are installing the latest Nvidia drivers - without installing the kernel drivers because you did that manually above:

• So..

This actually worked, wow!

We pulled this off while in GUI X-11 mode, as in:

After sudo reboot we actually have our basic nvidia-smi:

• Almost there!

Part II: Getting to nvcc / cuda-toolkit

• If your nvida-smi driver is installed we can now focus on the nvidia-cuda-toolkit,

Try 1: script

# Download and install the CUDA repository keyring
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

# Refresh package lists
sudo apt update

# Install the CUDA Toolkit (includes nvcc and development components)
sudo apt install cuda-toolkit

Or Try 2: Direct download from

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-ubuntu2604.pin
sudo mv cuda-ubuntu2604.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.3.0/local_installers/cuda-repo-ubuntu2604-13-3-local_13.3.0-610.43.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2604-13-3-local_13.3.0-610.43.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2604-13-3-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-3

At this point, we are looking at large downloads as it pulls in the supports.

A successful run will give this at the end:

Processing triggers for desktop-file-utils (0.28-1) ...
Processing triggers for mailcap (3.74) ...
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
--------------------------------------------------
[!] Scanning application launchers
Removing duplicate or broken launchers...
[!] Launchers have been successfully updated!
--------------------------------------------------

A Working CLion Example

• This is a working CMakeLists.txt / main.cpp / json_support.h example for your reference for Clion (Jetbrains product)
• It also does a minimal cuda call.

cmake_minimum_required(VERSION 3.18)

set(CMAKE_CUDA_COMPILER "/usr/local/cuda-13.3/bin/nvcc")

project(MyImGuiCuda LANGUAGES CXX CUDA)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CUDA_STANDARD 17)
set(CMAKE_CUDA_ARCHITECTURES 75)

set(CUDAToolkit_ROOT "/usr/local/cuda-13.3")
list(APPEND CMAKE_PREFIX_PATH "/usr/local/cuda-13.3")
find_package(CUDAToolkit REQUIRED)

find_package(OpenGL REQUIRED)
find_package(glfw3 REQUIRED)

include(FetchContent)

FetchContent_Declare(imgui GIT_REPOSITORY https://github.com/ocornut/imgui.git GIT_TAG v1.91.8-docking)
FetchContent_Declare(implot GIT_REPOSITORY https://github.com/epezent/implot.git GIT_TAG v0.16)
FetchContent_Declare(cpr GIT_REPOSITORY https://github.com/libcpr/cpr.git GIT_TAG 1.11.1)
FetchContent_Declare(nlohmann_json GIT_REPOSITORY https://github.com/nlohmann/json.git GIT_TAG v3.11.3)

FetchContent_MakeAvailable(imgui implot cpr nlohmann_json)

set(IMGUI_SOURCES
        ${imgui_SOURCE_DIR}/imgui.cpp
        ${imgui_SOURCE_DIR}/imgui_draw.cpp
        ${imgui_SOURCE_DIR}/imgui_tables.cpp
        ${imgui_SOURCE_DIR}/imgui_widgets.cpp
        ${imgui_SOURCE_DIR}/backends/imgui_impl_glfw.cpp
        ${imgui_SOURCE_DIR}/backends/imgui_impl_opengl3.cpp
)

set(IMPLOT_SOURCES
        ${implot_SOURCE_DIR}/implot.cpp
        ${implot_SOURCE_DIR}/implot_items.cpp
)

add_executable(my_imgui_cuda
        main.cpp
        json_support.cpp
        json_support.h
        ${IMGUI_SOURCES}
        ${IMPLOT_SOURCES}
)

target_include_directories(my_imgui_cuda PRIVATE
        ${imgui_SOURCE_DIR}
        ${imgui_SOURCE_DIR}/backends
        ${implot_SOURCE_DIR}
        ${CUDAToolkit_INCLUDE_DIRS}
)

target_link_libraries(my_imgui_cuda PRIVATE
        glfw
        OpenGL::GL
        CUDA::cudart
        cpr::cpr
        nlohmann_json::nlohmann_json
        ${CMAKE_DL_LIBS}
)

set_target_properties(my_imgui_cuda PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

And inside our main.cpp we have some polygon.io boilerplate:

#include <GLFW/glfw3.h>
#include <cuda_runtime.h>
#include <iostream>

#include "imgui.h"
#include "imgui_impl_glfw.h"
#include "imgui_impl_opengl3.h"
#include "implot.h"

#include <iostream>
#include <string>
#include "json_support.h"


int main()
{

    // region test_api call
    // Create the client with the base URL
    JsonSupport api("https://api.massive.com", 15000);   // 15 second timeout

    // Your API key (replace with your real key)
    std::string api_key = "";

    // Build the query
    std::string endpoint = "/v3/reference/options/contracts?"
                           "apiKey=" + api_key +
                           "&underlying_ticker=AAPL"
                           "&contract_type=call"
                           "&limit=20"
                           "&order=asc";

    // Make the request
    nlohmann::json response = api.get(endpoint);

    // Check for errors
    if (response.contains("error") || response.contains("status_code")) {
        std::cerr << "API Error: " << response.dump(4) << std::endl;
        return 1;
    }

    // Print results
    if (response.contains("results")) {
        std::cout << "Found " << response["results"].size() << " contracts\n\n";

        for (const auto& contract : response["results"]) {
            std::cout << contract["ticker"].get<std::string>()
                      << " | Strike: " << contract["strike_price"]
                      << " | Exp: " << contract["expiration_date"].get<std::string>()
                      << " | Type: " << contract["contract_type"].get<std::string>()
                      << std::endl;
        }
    } else {
        std::cout << response.dump(4) << std::endl;
    }


    // endregion


    // region GLFW window
    glfwInit();
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
    glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);
    GLFWwindow* window = glfwCreateWindow(1280, 720, "ImGui + CUDA Example", nullptr, nullptr);
    glfwMakeContextCurrent(window);
    glfwSwapInterval(1);
    // endregion
    // region ImGui setup
    IMGUI_CHECKVERSION();
    ImGui::CreateContext();
    ImPlot::CreateContext();
    ImGuiIO& io = ImGui::GetIO(); (void)io;
    ImGui::StyleColorsDark();
    ImGui_ImplGlfw_InitForOpenGL(window, true);
    ImGui_ImplOpenGL3_Init("#version 330");
    // endregion
    // region Simple CUDA check
    int deviceCount = 0;
    cudaGetDeviceCount(&deviceCount);
    std::cout << "CUDA devices: " << deviceCount << std::endl;

    float values[100] = {0};  // Sample data for plotting

    while (!glfwWindowShouldClose(window)) {
        glfwPollEvents();

        ImGui_ImplOpenGL3_NewFrame();
        ImGui_ImplGlfw_NewFrame();
        ImGui::NewFrame();

        ImGui::Begin("Hello ImGui + CUDA");
        ImGui::Text("CUDA devices detected: %d", deviceCount);
        ImGui::End();

        ImGui::Begin("Simple Plot");
        if (ImPlot::BeginPlot("Sample Plot")) {
            ImPlot::PlotLine("Values", values, 100);
            ImPlot::EndPlot();
        }
        ImGui::End();

        // Render
        ImGui::Render();
        int display_w, display_h;
        glfwGetFramebufferSize(window, &display_w, &display_h);
        glViewport(0, 0, display_w, display_h);
        glClearColor(0.1f, 0.1f, 0.1f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT);
        ImGui_ImplOpenGL3_RenderDrawData(ImGui::GetDrawData());
        glfwSwapBuffers(window);
    }
    // endregion
    // region  Cleanup
    ImGui_ImplOpenGL3_Shutdown();
    ImGui_ImplGlfw_Shutdown();
    ImPlot::DestroyContext();
    ImGui::DestroyContext();
    glfwDestroyWindow(window);
    glfwTerminate();
    // endregion

    return 0;
}

And inside our json_support.h

// region JsonSupport Class
#pragma once

#include <cpr/cpr.h>
#include <nlohmann/json.hpp>
#include <string>
#include <optional>

class JsonSupport {
public:
    explicit JsonSupport(std::string base_url = "", int timeout_ms = 10000)
        : base_url_(std::move(base_url)), timeout_ms_(timeout_ms) {}

    // Set authentication token once (used for all subsequent requests)
    void set_bearer_token(const std::string& token) {
        bearer_token_ = token;
    }

    void set_timeout(int timeout_ms) {
        timeout_ms_ = timeout_ms;
    }

    // GET request
    nlohmann::json get(const std::string& endpoint) {


        std::string url = build_url(endpoint);

        cpr::Header headers = build_headers();

        auto response = cpr::Get(
            cpr::Url{url},
            headers,
            cpr::Timeout{timeout_ms_}
        );

        return handle_response(response);
    }

    // POST request with optional JSON body
    nlohmann::json post(const std::string& endpoint,
                        const nlohmann::json& payload = nlohmann::json{}) {
        std::string url = build_url(endpoint);
        cpr::Header headers = build_headers();
        headers["Content-Type"] = "application/json";

        auto response = cpr::Post(
            cpr::Url{url},
            headers,
            cpr::Body{payload.dump()},
            cpr::Timeout{timeout_ms_}
        );

        return handle_response(response);
    }

private:
    std::string base_url_;
    int timeout_ms_;
    std::string bearer_token_;

    std::string build_url(const std::string& endpoint) const {
        if (base_url_.empty()) {
            return endpoint;
        }
        if (endpoint.empty()) {
            return base_url_;
        }
        // Simple URL joining
        if (base_url_.back() == '/' && endpoint.front() == '/') {
            return base_url_ + endpoint.substr(1);
        } else if (base_url_.back() != '/' && endpoint.front() != '/') {
            return base_url_ + "/" + endpoint;
        }
        return base_url_ + endpoint;
    }

    cpr::Header build_headers() const {
        cpr::Header headers;
        if (!bearer_token_.empty()) {
            headers["Authorization"] = "Bearer " + bearer_token_;
        }
        return headers;
    }

    nlohmann::json handle_response(const cpr::Response& response) const {
        if (response.status_code >= 200 && response.status_code < 300) {
            if (response.text.empty()) {
                return nlohmann::json::object();
            }
            try {
                return nlohmann::json::parse(response.text);
            } catch (...) {
                return nlohmann::json{{"raw_response", response.text}};
            }
        } else {
            nlohmann::json error;
            error["status_code"] = response.status_code;
            error["error"] = response.error.message;
            if (!response.text.empty()) {
                error["body"] = response.text;
            }
            return error;
        }
    }
};
// endregion

This is interesting because it will produce a super-fast gui interface, rudimentary but cool!

Save your Context and Come Back

This process manager is very powerful in that your LLM can now save it's work and spread it across many contexts.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

Get your LLM coding all Night! LLMQP

This LLM will enable you to queue multiple prompts which will execute one after another.

LLMQP Drops! A New Queue Dispatcher. Let your LLM CODE ALL NIGHT.

LLM Queue Dispatcher. A Powerful Harness Drop will queue your localLLM all night and keep it working!

Hot Configthinkmeltprotonmail.com

The significance of this cannot be stressed enough - imagine not only getting MTP (multiple-token-prediction) on your LLM which almost  doubles the speed , but then getting TurboQuant KV compression - allowing you to run VERY large contexts on minimal hardware!

Again we want to hugely thank TheTom for this specialized fork.

GitHub - TheTom/llama-cpp-turboquant: LLM inference in C/C++

LLM inference in C/C++. Contribute to TheTom/llama-cpp-turboquant development by creating an account on GitHub.

GitHubTheTom

Background Supports.

• The supports are always the same you need the latest cmake, the latest nvcc, the latest nvidia drivers, simply go over to the StudentLLM and work through it, the only difference is come back here for a different configuration and a MTP enabled model.
• Because it references TheTom Turboquant fork of Llama.cpp it will automatically enable MTP - which he added to his fork last week! So this is the model that will both have MTP and Turboquant as of 2026-Jun-21.
• We had no idea this had occurred until Tom personally messaged me about it!

StudentLLM - Qwen2.5-coder-7b-instruct-q6-k / Qwen3.5 Agentic on a Ryzen 5-2600/ 3060ti. Production LLM or not? YES!

We Look a StudentLLM setup to get as much productivity out of limited hardware as we can.

Hot Configthinkmeltprotonmail.com

If you have worked through the above guide you will have just taught yourself how to compile from source one of the most advanced LLM's in the world, and it's supports.

A Hot Config

• Please note you MUST use a MTP enabled MOE type model, we have gotten stunning results from Carnice-Qwen3.6.. so do support this guys work. Buy him a coffee!

wget https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/blob/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf

/usr/bin/llama-server --jinja \
-m /home/c/models/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf  \
--host 192.168.1.3 \
--n-gpu-layers -1 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--n-cpu-moe 30 \
--chat-template-kwargs '{"preserve_thinking":true}' \
-c 252144 \
--flash-attn 1 \
--context-shift \
--repeat-penalty 1.12 \
--cache-type-k turbo3 \
--cache-type-v turbo4 \

The results were shocking. We had it go over a 38K large multiple level asteroid game. We were used to our localLLM taking  a good half hour to produce. No. Nuts! It was done in a few minutes.

Some INSANE ingest/production Numbers. Check this out.

• This is not 'high-end' gear - this is just a 4080 on a Ryzen 9 12-core. 128 GB RAM / 16 GB  VRAM.
• Please note you will need to adjust your --n-cpu-moe 30 to something larger or smaller, if you are doing small contexts load your GPU like 90%. I use this configuration to utilize about 12GB of my 16GB GPU.  It allows for very large contexts because of kv_cache compression and still gives whopping fast speeds, like FAST.
• If I was doing super-large contexts I can adjust that value UP, and load most of the model to the CPU. It is a speed/size trade off.

Next we ran bench-loop on the whole setup...

benchloop run --endpoint http://192.168.1.3:8080 --provider openai_compat --model Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf

We modified our run configuration so it used 15.7GB of the 16 GB card. The idea is the benchmark will probably run inside the last 1 GB, and it was CRUSHING through it. Our benchmark config... --n-cpu-moe 24

/usr/bin/llama-server --jinja \
-m /home/c/models/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf  \
--host 192.168.1.3 \
--n-gpu-layers -1 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--n-cpu-moe 24 \
--chat-template-kwargs '{"preserve_thinking":true}' \
-c 252144 \
--flash-attn 1 \
--context-shift \
--repeat-penalty 1.12 \
--cache-type-k turbo3 \
--cache-type-v turbo4 \

It was just sailing through the benchmark, and  we were excited to see what results were about to pour in. We had never seen ingest over really about 150, and never saw much above 27.  We effectively saw it doubled.

It gave VERY good results. For some reason dataextract either gets a 15/15 or a 1 with these models, to tweak. It should be noted that potentially we should disable our MCP agents before benchmarking as it adds unnecessary overhead.

We noticed that we gave it almost no-cache and it was failing at one point with an error of:

W slot update_slots: id  0 | task 32013 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cp

We backed off our model and followed recommendations by adjusting the --n-cpu-moe 26 and adding --swa-full Our next run configuration is as follows, which loaded the GPU to 14.5GB out of 16GB.

/usr/bin/llama-server --jinja \
-m /home/c/models/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf  \
--host 192.168.1.3 \
--n-gpu-layers -1 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--n-cpu-moe 29 \
--chat-template-kwargs '{"preserve_thinking":true}' \
-c 252144 \
--swa-full \
--flash-attn 1 \
--repeat-penalty 1.12 \
--cache-type-k turbo3 \
--cache-type-v turbo4 \

It should be noted that our Tokens/s dropped from 58 to about 45 ish on various things, and we are currently studying this for better results. We stopped at this point because irrespective of the bench the results are a game changer!

Power Tools.  MCP Agents.

Once you get this highly powerful LLM working you can increase it's ability MASSIVELY by adding MCP agents. Here are complete opensource walk throughs of a pile of MCP agents to get you started. If you are trying to  learn - start on #1 the Calculator and when you get the hang of it work up!

PowerChest home Agent Agents MCP Tools to Put your HouseLLM into Turbo. MCP TOOLS 1-9++

Downloads Page for all your MCP tooling needs!

Hot Configthinkmeltprotonmail.com

Process-Manger MCP Agent is Exceptionally Powerful

One tool stood out in particular the Process Manager MCP tool.  Using it you can have your localLLM pick up over multiple prompts, break a job up into multiple tasks, give you 'code-drops' of entire code bases.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant w/ Code Drop.

Agentic Server Primer: Llama.cpp MCP Lesson 8: Process Manager Web Enabled Research Assistant

Hot Configthinkmeltprotonmail.com

LLMQP - Get your houseLLM coding all night

Once you have the hang of a bunch of coding agents, you can use this MCP agent to get your LLM working through dozens of prompts - saving it's work to the Process Manager.

LLMQP Drops! A New Queue Dispatcher. Let your LLM CODE ALL NIGHT.

LLM Queue Dispatcher. A Powerful Harness Drop will queue your localLLM all night and keep it working!

Hot Configthinkmeltprotonmail.com

Conclusion

Even though it seemed that bench-loop didn't really give this configuration a very high rating, we were extremely sold on how well it performed from our own anecdotal observations.  It not just a work horse now, it is a endurance workhorse.  We were very used to having our houseLLM take up to 1.5 hours to produce anything significant and it was like the dishes - you set it and come back and it produced something useful.  Now if you were around pre-LLM days this would take weeks to do manually, and you could spend days looking for code examples. So this is fast compared to hand-coding.  And it's free. You do not have to pay for it, and it's real benefit comes in when you learn how to leverage your localLLM to code for you all night!  Yes it will not replace a SOTA commerical model, but it's yours, and ownership is it's own ability, it also shows you know how to build, promote, and maintain your own LLM that empowers you that you are competent in using them!