How I Came to Understand Pipeline, Queue, and Worker
This is where it started
I took over a system from a coworker. He called it a “queue system,” but as he kept explaining, it somehow became a pipeline.
The actual architecture was pretty simple: one Flask service receiving API requests, plus 3 workers, each handling a stage of data processing in sequence.
So is a queue just a pipeline? And what’s the relationship between queue and worker? That’s when I started digging…
Getting these terms straight first
Is a pipeline just a queue? No. They’re actually terms at different levels.
A queue is more like a to-do list: when a task comes in, you add it to the list and process them in order.
By contrast, there’s another way to handle tasks: the ones that come in later get processed first.
A pipeline, on the other hand, is the processing flow itself — data has to go through a series of steps in order before it’s considered done.
flowchart TB
root["Is a pipeline<br> a queue?"]
root --> Q["queue<br>FIFO — first in, first out<br>Line up; earlier tasks first"]
root --> S["stack<br>LIFO — last in, first out<br>Also called a stack-based queue"]
root --> P["pipeline<br>An ordered processing flow<br>Doesn't conflict with queue"]
style root fill:#dce9f5,stroke:#5d8aa8 Between receive and run: the broker
In a queue system, the queue stores tasks, workers run them, and a broker in the middle handles message passing.
Take Celery, a common Python worker framework: a typical setup uses Redis as the “broker.” I used to see this word a lot when I worked in international trade — it meant a middleman or a customs broker. In software, people usually call it a message intermediary.
The broker holds the queue I mentioned earlier — the data structure that stores the to-do list. When a task is created, there’s a place to put it, and workers pick tasks up from there to run.
The whole setup looks something like this:
flowchart TB
A["Main app<br>(e.g. Flask API)"] -->|"call task.delay()"| C1["Celery client/app<br>Turns a function call into<br>a task message, sent to the broker"]
subgraph B["broker (message intermediary)"]
Q["queue (FIFO container)<br>Holds pending tasks"]
end
C1 -->|send task message| Q
Q -->|fetch task| C2["Celery worker<br>Pulls tasks from the broker<br>and runs them"]
C2 --> T["Run the task"]
classDef celery fill:#f9e79f,stroke:#b7950b
class C1,C2 celery Redis isn’t really a formal broker
While researching, I came across this line: “Strictly speaking, Redis doesn’t meet the definition of a broker.” So… why does everyone still use it?
Later, while looking into LLM resumable streaming, I noticed the author of this article also used Redis — specifically the Redis Pub/Sub pattern. Digging into that happened to answer my earlier question:
When Redis first came out, it was just a list data structure sitting in memory — really convenient to use, and it happened to fit what a queue needs: a place to temporarily store tasks. That’s why it caught on. But in a more rigorous production environment where you can’t afford to drop tasks, Redis isn’t as good a fit.
A broker that strictly meets the definition needs “task management” capabilities. Later on, Redis Stream did evolve to serve as a message broker.
flowchart TB
DEF["Strict definition of a broker"]
DEF --> A1["ACK"]
DEF --> A2["Error retries"]
DEF --> A3["delivery guarantee<br>at-least-once /<br>exactly-once"]
A1 -.-> R
A2 -.->|"Judge Redis by the above"| R
A3 -.-> R
R["Redis:<br>Not a “formal” broker.<br>Historical evolution:"]
R --> L["Redis List<br>Original design; convenient, often used as a queue.<br>No ACK."]
R --> ST["Redis Stream (after List)<br>Supports ACK<br>Designed for message brokering."]
R --> PS["Redis Pub/Sub<br>No ACK<br>Fire-and-forget realtime broadcast."]
style DEF fill:#dce9f5,stroke:#5d8aa8
style ST fill:#d5e8d4,stroke:#82b366
style L fill:#f5f5f5,stroke:#999999
style PS fill:#f5f5f5,stroke:#999999 So worker and queue always get used together… or do they?
Back to the beginning: the queue system that kicked off this whole investigation does look like the classic combo — queue plus worker. So whenever you have tasks to process, should you just apply this architecture?
Not necessarily. The two can be considered completely separately. Different situations call for different tools, and they don’t have to be bound together.
I drew a diagram of when to use a queue, when you should add a worker, and what factors decide that — it helped me build a mental model —
Simply put: the volume of tasks decides whether you need a queue; how heavy the tasks are decides whether you need a worker.
③ Queue only
Sign-up rush: API replies “received”; sign-ups go into a queue to be written later.
④ queue + worker
User uploads an image → resize / convert
① Handle in the main process
Manual admin export of a small dataset (tens of rows)
② Background worker
Nightly batch reports (fixed time, predictable volume)
The chart is wide — scroll sideways to explore; pinch to zoom for a clearer view.