Search for a command to run...
Software Engineer | Open Source Enthusiast | Mentor | Learner I love documenting stuff that I come across and find interesting. Hoping that you will love reading it and get to know something new :)
No comments yet. Be the first to comment.
In the world of large language models (LLMs) like OpenAI’s GPT, Anthropic’s Claude, or Meta’s LLaMA, one of the most powerful techniques for guiding model behaviour is prompt engineering.Few-Shot Prompting establishes context by referencing LLMs. The...
AI development often presents an illusion of standardization. When engineering structured workflows, JSON Schema is universally relied upon to force Large Language Models into returning predictable, t
Large language models (LLMs) like Llama, Qwen, and DeepSeek are transforming how software interacts with data. However, moving these models from a local experimental script to a highly available, mult
Artificial Intelligence (AI) models, especially large language models (LLMs) like GPT, Claude, or Gemini, have transformed how we interact with computers. Yet, terms like tokens, token limits, quotas, tokenizers, and parameters can seem confusing at ...
JavaScript development involves many concepts and tools that help developers organize, share, and reuse code effectively. For newcomers, terms such as library, framework, UMD, and CDN can seem technical, and understanding their roles is essential for...
Large language models (LLMs) such as GPT, LLaMA, and Falcon have billions of parameters. While these models are powerful, their size makes them challenging to run and adapt for specific use cases. Three key concepts help solve this problem: quantizat...
Artificial Intelligence powers everything from chatbots to medical imaging, but there's one concept at the heart of most modern AI applications: inference.
While training large models like GPT or LLaMA gets all the attention, it's inference that actually brings AI to life in apps, assistants, and automations. Whether you’re a developer, researcher, or just curious about how tools like ChatGPT or Ollama work under the hood, understanding inference models is essential.
Let’s break it down.
An inference model is a trained AI or machine learning model that’s being used to make predictions or generate outputs on new data.
You're not teaching the model anymore — you're using what it already knows to do something useful:
Example:
Training: Feed a model millions of images of cats and dogs.
Inference: Show it a new image → it tells you “cat”.
In the world of large language models (LLMs), inference means:
Prompt → Model → Text Response
🧾 Input is tokenized (e.g., “Hello” → [15496])
⚙️ The model processes the tokens through multiple layers
📤 An output is generated (text, number, label, etc.)
For example, asking an LLM:
“Explain quantum computing in simple terms”
results in a full paragraph written by the model — this is inference.
You can run inference in two main ways:
Use libraries like PyTorch, Transformers, or TensorFlow:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct")
inputs = tokenizer("What is inference?", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))
Full control
Customisation
Good for research
Heavy setup
Requires managing memory, GPUs, dependencies
Platforms like Ollama, Groq, LM Studio, and Hugging Face Inference API provide:
One-command setup
Efficient model execution
REST APIs or CLI tools
# Start LLaMA 3 locally
ollama run llama3
Fast and optimised
Low setup effort
Can run locally (Ollama) or in cloud (Groq)
Limited customisation
Depends on platform support
Tools like Ollama, LM Studio, or WebUI work completely offline
Great for privacy, edge computing, and offline environments
| Tool/Platform | Internet Required? | Notes |
| Ollama | ❌ No | Local inference engine |
| GroqCloud | ✅ Yes | Ultra-fast cloud inference |
| Hugging Face API | ✅ Yes | Hosted models |
| PyTorch (offline) | ❌ No | After model is downloaded |
Here are common tools for running inference in different ways:
| Tool | Description | Type |
| Ollama | Local LLM runner with REST API | Local |
| GroqCloud | Fast cloud inference for LLMs | Cloud |
| Hugging Face | Model hub + APIs | Cloud/Local |
| Text Generation WebUI | GUI for running models locally | Local |
| ONNX Runtime | Optimized inference for any backend | Local/Cloud |
| vLLM | Fast, scalable server for LLMs | Local/Server |
Inference is the final step in the machine learning pipeline — and arguably the most important. It’s where trained models turn theory into action, powering everything from virtual assistants to real-time translation and intelligent automation.
Reach out in the comments below or on LinkedIn to let me know what you think of it.
For more updates, do follow me here :)