The tokenizer, Byte-Pair Encoding on this occasion, interprets every token within the enter textual content right into a corresponding token ID. Then, GPT-2 makes use of these token IDs as enter and tries to foretell the subsequent most definitely token. Lastly, the mannequin generates logits, that are transformed into possibilities utilizing a softmax operate.

For instance, the mannequin assigns a likelihood of 17% to the token for “of” being the subsequent token after “I’ve a dream”. This output primarily represents a ranked checklist of potential subsequent tokens within the sequence. Extra formally, we denote this likelihood as *P(of | I’ve a dream) = 17%*.

Autoregressive fashions like GPT predict the subsequent token in a sequence primarily based on the previous tokens. Think about a sequence of tokens *w = (w*₁*, w*₂*, …, w*ₜ*)*. The joint likelihood of this sequence *P(w)* may be damaged down as:

For every token *wᵢ* within the sequence, *P(wᵢ | w₁, w₂, …, wᵢ₋₁)* represents the conditional likelihood of *wᵢ* given all of the previous tokens (*w₁, w₂, …, wᵢ₋₁*). GPT-2 calculates this conditional likelihood for every of the 50,257 tokens in its vocabulary.

This results in the query: how will we use these possibilities to generate textual content? That is the place decoding methods, equivalent to grasping search and beam search, come into play.

Grasping search is a decoding methodology that takes probably the most possible token at every step as the subsequent token within the sequence. To place it merely, it solely retains the most definitely token at every stage, discarding all different potential choices. Utilizing our instance:

**Step 1**: Enter: “I’ve a dream” → Most certainly token: “ of”**Step 2**: Enter: “I’ve a dream of” → Most certainly token: “ being”**Step 3**: Enter: “I’ve a dream of being” → Most certainly token: “ a”**Step 4**: Enter: “I’ve a dream of being a” → Most certainly token: “ physician”**Step 5**: Enter: “I’ve a dream of being a health care provider” → Most certainly token: “.”

Whereas this strategy would possibly sound intuitive, it’s essential to notice that the grasping search is short-sighted: it solely considers probably the most possible token at every step with out contemplating the general impact on the sequence. This property makes it quick and environment friendly because it doesn’t must hold observe of a number of sequences, but it surely additionally implies that it may miss out on higher sequences which may have appeared with barely much less possible subsequent tokens.

Subsequent, let’s illustrate the grasping search implementation utilizing graphviz and networkx. We choose the ID with the best rating, compute its log likelihood (we take the log to simplify calculations), and add it to the tree. We’ll repeat this course of for 5 tokens.

`import matplotlib.pyplot as plt`

import networkx as nx

import numpy as np

import timedef get_log_prob(logits, token_id):

# Compute the softmax of the logits

possibilities = torch.nn.practical.softmax(logits, dim=-1)

log_probabilities = torch.log(possibilities)

# Get the log likelihood of the token

token_log_probability = log_probabilities[token_id].merchandise()

return token_log_probability

def greedy_search(input_ids, node, size=5):

if size == 0:

return input_ids

outputs = mannequin(input_ids)

predictions = outputs.logits

# Get the anticipated subsequent sub-word (right here we use top-k search)

logits = predictions[0, -1, :]

token_id = torch.argmax(logits).unsqueeze(0)

# Compute the rating of the anticipated token

token_score = get_log_prob(logits, token_id)

# Add the anticipated token to the checklist of enter ids

new_input_ids = torch.cat([input_ids, token_id.unsqueeze(0)], dim=-1)

# Add node and edge to graph

next_token = tokenizer.decode(token_id, skip_special_tokens=True)

current_node = checklist(graph.successors(node))[0]

graph.nodes[current_node]['tokenscore'] = np.exp(token_score) * 100

graph.nodes[current_node]['token'] = next_token + f"_{size}"

# Recursive name

input_ids = greedy_search(new_input_ids, current_node, length-1)

return input_ids

# Parameters

size = 5

beams = 1

# Create a balanced tree with peak 'size'

graph = nx.balanced_tree(1, size, create_using=nx.DiGraph())

# Add 'tokenscore', 'cumscore', and 'token' attributes to every node

for node in graph.nodes:

graph.nodes[node]['tokenscore'] = 100

graph.nodes[node]['token'] = textual content

# Begin producing textual content

output_ids = greedy_search(input_ids, 0, size=size)

output = tokenizer.decode(output_ids.squeeze().tolist(), skip_special_tokens=True)

print(f"Generated textual content: {output}")

`Generated textual content: I've a dream of being a health care provider.`

Our grasping search generates the identical textual content because the one from the transformers library: “I’ve a dream of being a health care provider.” Let’s visualize the tree we created.

`import matplotlib.pyplot as plt`

import networkx as nx

import matplotlib.colours as mcolors

from matplotlib.colours import LinearSegmentedColormapdef plot_graph(graph, size, beams, rating):

fig, ax = plt.subplots(figsize=(3+1.2*beams**size, max(5, 2+size)), dpi=300, facecolor='white')

# Create positions for every node

pos = nx.nx_agraph.graphviz_layout(graph, prog="dot")

# Normalize the colours alongside the vary of token scores

if rating == 'token':

scores = [data['tokenscore'] for _, information in graph.nodes(information=True) if information['token'] just isn't None]

elif rating == 'sequence':

scores = [data['sequencescore'] for _, information in graph.nodes(information=True) if information['token'] just isn't None]

vmin = min(scores)

vmax = max(scores)

norm = mcolors.Normalize(vmin=vmin, vmax=vmax)

cmap = LinearSegmentedColormap.from_list('rg', ["r", "y", "g"], N=256)

# Draw the nodes

nx.draw_networkx_nodes(graph, pos, node_size=2000, node_shape='o', alpha=1, linewidths=4,

node_color=scores, cmap=cmap)

# Draw the sides

nx.draw_networkx_edges(graph, pos)

# Draw the labels

if rating == 'token':

labels = {node: information['token'].cut up('_')[0] + f"n{information['tokenscore']:.2f}%" for node, information in graph.nodes(information=True) if information['token'] just isn't None}

elif rating == 'sequence':

labels = {node: information['token'].cut up('_')[0] + f"n{information['sequencescore']:.2f}" for node, information in graph.nodes(information=True) if information['token'] just isn't None}

nx.draw_networkx_labels(graph, pos, labels=labels, font_size=10)

plt.field(False)

# Add a colorbar

sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)

sm.set_array([])

if rating == 'token':

fig.colorbar(sm, ax=ax, orientation='vertical', pad=0, label='Token likelihood (%)')

elif rating == 'sequence':

fig.colorbar(sm, ax=ax, orientation='vertical', pad=0, label='Sequence rating')

plt.present()

# Plot graph

plot_graph(graph, size, 1.5, 'token')