Learn · Q-learning
Q-learning in a grid world
An agent starts in the corner knowing nothing. Each move it takes earns a small penalty; reaching the goal pays out, falling in a pit costs dearly. Through trial and error it fills in a table of action values and slowly discovers a route. Train it, then watch it walk the policy it learned.
Cyan cells are high-value (close to the goal), red cells are low-value (near a pit). Gold arrows show the best action the agent currently believes in. Dark blocks are walls.
How strongly each new experience overwrites the old estimate.
How much the agent values future reward versus reward right now.
Chance of taking a random action instead of the greedy one.
Agent, environment and reward
Reinforcement learning frames a problem as an agent acting in an environment. Here the environment is the grid: each cell is a state, and from any cell the agent can take one of four actions — up, down, left or right. Bumping a wall or the edge just leaves it where it was.
Every step returns a reward: a small −0.02 for a plain move (so short paths beat long ones), +1 for reaching the goal, and −1 for landing in a pit. The goal and pits end the episode; so does hitting the step cap. The agent never sees the rules — it only sees rewards, and has to work the rest out.
The Q-learning update
The agent keeps a table Q(s, a) — one number for every state–action pair, all starting at zero — estimating the long-run reward of taking action a in state s. After each move from s to s′ it nudges that estimate toward what it just observed:
Q(s, a) ← Q(s, a) + α · [ r + γ · maxa′ Q(s′, a′) − Q(s, a) ]
The bracket is the temporal-difference error: the reward r just received, plus the discounted best value available from the next state, minus what we predicted. At a terminal cell there is no next state, so that future term is zero. Repeat this over thousands of steps and the table converges to the optimal action values.
Exploration vs exploitation (ε)
If the agent always took the action it currently thinks is best, it would lock onto the first vaguely-good route and never find anything better. So it acts ε-greedy: with probability ε it picks a random action to explore, and otherwise it exploits the greedy action. High ε learns the map broadly but wanders; low ε commits fast but can miss the goal. Real systems usually start high and decay ε over time.
What α and γ do
α (learning rate) controls how much each experience moves the estimate. Near 0 the agent barely updates and learns painfully slowly; near 1 it overwrites its estimate with the latest (noisy) sample and the values jitter. γ (discount) sets how far ahead it plans: at γ = 0 it's greedy for immediate reward only; near 1 the +1 at the goal propagates back across the whole grid, which is what carves the smooth cyan gradient leading home.
Caveat: this is a toy. The world is tiny, fully observed and deterministic, and the table has just 36 × 4 entries — so plain tabular Q-learning is enough. Real problems have enormous or continuous state spaces where the table is replaced by a neural network (deep Q-networks), rewards are sparse and noisy, and exploration is a research problem in its own right. The update rule above, though, is exactly the one those systems build on. More on the Learn shelf.