Uncategorized

The AMD Vulkan Chronicles

Leave a comment

This is a classic “Developer in the Trenches” story. It’s got a ghost in the machine, a GPU that won’t cooperate, and a happy ending with a dark-horse model.

Below is a summary of our troubleshooting session and a blog post draft you can share with the LLM community.

Executive Summary

  • The Goal: Run an LLM with Tool-Calling capabilities on a Windows machine with an AMD Radeon RX 5500 XT (8GB VRAM) using Ollama and the Vulkan backend.
  • The Conflict: * Gemma 3 1B suffered from “Split-Brain” syndrome, offloading 576MB to the CPU despite having 7.3GB of free VRAM.
    • Llama 3.2 1B ignored user-defined context limits and tried to allocate a massive 128k context window (8.3GB), causing an ErrorOutOfDeviceMemory crash.
    • Environmental Ghost: A hidden system-level variable (OLLAMA_CONTEXT_LENGTH: 131072) was overriding session settings.
  • The Solution: Switched to Qwen 2.5 Coder 1.5B. It proved to be the most stable for AMD/Vulkan, respected memory limits, and handles tool-calling/Greek characters natively.

Blog Post: The AMD Vulkan Chronicles

Title: Fighting the 128k Ghost: My Journey to Stable Local LLMs on AMD

If you’ve ever tried to run local LLMs on Windows with an AMD card, you know the “Vulkan Dance.” Last night, I went ten rounds with Ollama trying to get a simple 1B model to run fully on my Radeon RX 5500 XT. Here is the post-mortem of that battle.

1. The Ghost in the Registry

I started by trying to run Gemma 3 1B. My GPU has 8GB of VRAM, and the model is tiny. It should have been a breeze. Instead, I saw the dreaded “Split-Brain”: Ollama was shoving 500MB+ onto my CPU.

The Culprit: My logs revealed a hidden system variable: OLLAMA_CONTEXT_LENGTH: 131072. No matter what I typed in PowerShell, the system was forcing a 128k context window, which ate my VRAM for breakfast.

Lesson: Always check your ollama serve logs for the env="map[...]" section. If there’s a variable there you didn’t set, find it in your Windows Environment Variables and kill it.

2. The Llama 3.2 “Memory Leak”

Next, I tried Llama 3.2 1B. This resulted in a total panic: ggml_vulkan: Device memory allocation failed. Even though I set my context to 32k, the Llama runner ignored me and tried to reserve 8.3GB of VRAM for a 128k window. On an 8GB card, that’s an instant crash.

3. The “Split-Brain” Mystery

Even after freeing up 7.3GB of VRAM, some models (like Gemma 3 QAT) still insisted on putting a fraction of the weights on the CPU. This is often due to VRAM fragmentation. Windows and AMD drivers are conservative; if they can’t find one perfectly continuous block of memory, they bail to the CPU, tanking your tokens-per-second.

4. The Winner: Qwen 2.5 Coder 1.5B

After failing with the “big names,” I pivoted to Qwen 2.5 Coder 1.5B.

  • Stability: It loaded 100% onto the GPU instantly.
  • Speed: I went from a 3.7-second “handshake” lag to near-instant responses.
  • Utility: Since I’m building a tool to manage my Greek supermarket lists (including baby supplies for my 3-person family), the “Coder” variant’s strictness with JSON and tool-calling was exactly what I needed.

My Final Setup Script:

If you’re on an AMD card, don’t just run ollama run. Use a clean PowerShell session:

PowerShell

$env:OLLAMA_NUM_CTX="16384"  # Start safe
$env:OLLAMA_VULKAN="1"
ollama serve

Final Takeaway: Don’t get married to a specific model. If the drivers hate the architecture, pivot to Qwen. It’s the hidden gem of the small-model world for AMD users.

Uncategorized

The Challenge of Sharing Environment Variables Across Multi-Stage Docker Builds

Leave a comment

Multi-stage builds in Docker are powerful, enabling streamlined images and optimized build processes by allowing each FROM statement to start with a fresh layer. However, a known limitation is the inability to natively share environment variables between stages. Environment variables set in one stage aren’t directly accessible in subsequent stages due to Docker’s isolation design for layers.

To transfer environment variables, developers use workarounds like writing variables to a file and copying it to subsequent stages. For example:

dockerfile
# First Stage
FROM base-image AS stage1
RUN echo "MY_VAR=$MY_VAR" > /env_vars

# Second Stage
FROM final-image
COPY --from=stage1 /env_vars /env_vars
RUN export $(cat /env_vars)

This approach reads from a file to replicate environment variables, providing a way to reuse configuration across stages despite Docker’s isolation boundaries.

ChatGPT, Software Engineering, Uncategorized

Should You Return to Your Old Job? Pros and Cons

Leave a comment

With the current state of the market in software engineering, I can understand there are people out there who cannot find a decent and stable job. I might be exaggerating, but I do believe that the “peak” of software engineering was reached in ’21. It’s all downhill from there on.

Returning to a former job can be appealing for those seeking stability, familiarity, or a role they truly enjoyed, but it’s a decision that requires careful consideration. Especially in software engineering, and in the software industry, where job hopping is considered the norm. I will try to analyze a breakdown of the pros and cons.

Pros

  1. Familiarity: You already know the team, culture, and work processes, allowing for a smoother transition.
  2. Shortened Learning Curve: Since you’re familiar with the role, you can often pick up where you left off, leading to quicker productivity.
  3. Potential Respect or Promotion: Returning employees are sometimes valued for their previous loyalty, especially if they left on good terms, which could open doors to promotions or new responsibilities.

Cons

  1. Stagnation: Returning could limit new learning and growth, as it may feel like going back instead of moving forward in your career.
  2. Possible Tensions: If you left for reasons related to culture or management, those issues might still be there.
  3. Missed New Opportunities: Going back might prevent you from exploring fresh environments or companies that could offer new perspectives, skills, or compensation.

Mr. ChatGPT, could you please…?

The above, just as you guessed, came from a single, one-liner prompt:

Write the pros and cons of returning to your old job.

While I can see the reasoning behind it, and most of the pros do make sense, my belief is that the scale tips towards the cons. As usual, I’ll try to elaborate my train of thought.

Being primarily a millennial, when it comes to the job market, I was born on the wrong side of the history. Right when I managed to make a break and by shedding tears and spilling blood, graduated from my masters, I ended up right in the big financial crisis. Businesses closing down, political unrest, and not to mention that my current job aspirations were describing a profession that, at the time, half of the people could not grasp it even.

Therefore, just as my other peers born on the cusp of the millennia, we believe that having a job is actually a blessing than a burden. So, millennial compromise usually outnumbers gen z’s pickiness, or general “high maintenance” of all the younger people joining the industry. Hence, if usually millennials are changing jobs, they do so because there are strong reasons behind them.

Familiarity

The kind of reasons, once accounted and acted upon, do not easily change. Hence, the pro argument of Familiarity is a strong con. What would be the reason of returning to the old job, if, there was a familiar strong reason for leaving? If that reason has changed, yes, it makes sense, but this leads to other…kind of arguments (essentially a different company, or position), so let’s not explore that.

Shortened Learning Curve

This argument, while seems to be a big positive one, I will try to reverse it.

Indeed, since an employee left, was familiar at the least, or was doing a very good job at best, in his/her previous post. That leads to shorter time to production and increased productivity. However, we do not fully know the reasons for leaving (whether that was compensation based, or something else), but for most of the people in the industry, a short learning curve is something they generally avoid. Not learning something new (especially with all the new tools that come up every day), is considered to be job-killing. Not to mention the challenging part. It is extremely challenging to be able to comprehend, learn and adapt in order to fix something broken. Or create something new.

Potential Respect or Promotion

I cannot think a worse argument than this. The main reason behind it is quite simple. If there was enough respect, there would have been a promotion, and the employee would not have reached the exit door. GPT, is also mentioning “loyalty” and “leaving on good terms”. Both of those things are, obviously discarded, right from when someone opens the exit door…

Concluding

Returning to a former position for me does not really make any sense. Being an advocate of change and generally positive feedback, when someone leaves his post, there surely is a larger reason that has not change for him to return. If that reason has changed, the most of the times we are talking about a different company, with different managers and different colleagues. Not the same company anymore…

To quote the big economist: “This time is different”. No — this time its not different than the previous time, at least not in the software industry.