What is Q-learning and how does this reinforcement learning work?

Q-learning is a a machine learning technique that enables a model to learn through repetition and improve over time with a learning approach typical of mammals. It is a form of reinforcement learning that usually uses Python as its preferred programming language.

In a way, Q-learning imitates the way in which young children and animals carry out certain actions as good or bad. Thus, when an action is classified as good, the individual is rewarded, whilst when it is classified as bad, they are penalised.

How does Q-learning work?

We’re going to explain how Q-learning works in simple terms, using an example to make it clearer. Essentially, Q-learning is a method that helps an agent (such as a robot, a programme, etc.) learn to make the best decisions in a given environment in order to achieve the highest possible reward.

Imagine a game where A robot must move around a grid to find a treasure. The grid has different positions (states), and the robot can move up, down, left or right (actions).

Let’s define a couple of key terms in reinforcement learning in order to explain how Q-learning works:

• States (S): These are the different positions on the grid. Each position represents a state.

• Shares (A): These are the robot’s possible movements: up, down, left, right.

• Reward (R): This is what the robot gains or loses after making a move. For example, +10 points for finding the treasure and -1 point for every move that does not lead to the treasure. The value of the rewards will depend on how we configure the Q-learning algorithm.

• Q-value (Q): It is a figure that tells us how effective it is for the robot to perform a certain action from a certain position.

What is a Q-table?

Before we explain how this machine learning technique works, we need to explain what a Q-table is. A Q-table, or Q-table, is a table or matrix that uses the Q-learning algorithm to store information on the best actions that an agent (such as a robot or a programme) should take in different states to maximise its long-term reward.

The Q-table has the following features:

1. Rows: Each row of the table represents a possible state in which the agent may find itself.

2. Columns: Each column in the table represents an action that the agent can take from that state.

3. Cells (Q-values, Q(s, a)): Each cell in the table contains a Q-value, which is a number indicating the quality of taking a specific action in a specific state. The higher the Q-value, the better that action is considered to be in that state (Bellman equation).

Q-learning process

Now, let’s explain how our agent goes about improving its responses until it achieves an optimal result:

Initialisation

We start with a Q-table full of zeros. This means that the agent has no information about which action is best in any state. This table has one row for each state (position on the grid) and one column for each action (possible moves).

Let’s visualise the Q-table for this same example. The robot must find the treasure on a 2×2 grid. Therefore, on the grid we represent four states (positions in which the robot can be found) which we can express in coordinates. The robot, for its part, has a range of four actions it can perform (move up, down, right or left). This is what the Q-table would look like:

Example of a Q-table                                                                                                                                                                                                                   

Shares

It is now up to the agent to decide what action to take. If the agent has not received any reward or penalty from the Q-learning algorithm, it will act randomly. If it has been trained previously, it will make decisions based on its past experiences.

Update

The robot receives rewards or penalties – that is, positive or negative feedback – depending on whether its position is getting closer to or further away from the treasure. Every time the robot moves, the Q-table must update the calculation of rewards for each combination of actions performed by the robot, depending on its current state.

Iteration and termination

This process is iterative; in other words, it is repeated until the Q-learning model can take no further actions and reaches a termination state, ideally having achieved a satisfactory outcome. Each time the algorithm terminates, this is called an episode.

Real-world applications of Q-learning

The theory is all well and good, but what’s the point of it all? Q-learning, like any other machine learning technique, can be put to practical use in many disciplines. Here are a few examples:

• Robotics: Robots can use Q-learning to learn how to navigate unfamiliar environments, avoid obstacles and achieve specific objectives.

• Video games: It is used to create TASs (tool-assisted speedruns) that can play and improve performance in video games, ranging from simple games such as chess to complex ones such as StarCraft.

• Autonomous driving: Cars learn to drive safely by obeying traffic signs and reacting appropriately to different situations on the road.

• Product recommendations: It improves recommendations on platforms such as Netflix, YouTube and Amazon by learning from users’ preferences and adapting to their tastes at all times.

• Industry and manufacturing: Anticipate machinery failures and plan maintenance to minimise downtime.