“含义机器”可视化大规模的语言模型将单词分解为令牌并处理它们

使用大规模语言模型（例如Chatgpt，Claude和Grok）的生成AI适当地对用户单词响应，就像人类一样。但是，有很大的区别

The website 'Meaning Machine' provides a visually easy-to-understand view of how large-scale language models process language, which is different from how humans process language. Generative AI using large-scale language models such as ChatGPT, Claude, and Grok respond appropriately to user words just like a human. However, there is a big difference between how large-scale language models process language and how humans process language.

网站的“含义机器”提供了视觉上易于理解的视图，以了解大规模的语言模型处理语言，这与人类处理语言的方式不同。使用大规模语言模型（例如Chatgpt，Claude和Grok）的生成AI适当地对用户单词响应，就像人类一样。但是，大规模语言模型处理语言与人类如何处理语言之间存在很大的区别。

The website 'Meaning Machine' provides a visually easy-to-understand view of how large-scale language models process language.

网站的“意义机器”提供了视觉上易于理解的视图，以了解大规模的语言模型处理语言。

Meaning Machine · Streamlit

意思是机器·简化

Below that, the input sentence is split into words, and each word is shown with a numeric ID when it is represented as a 'token' by the large-scale language model.

在此之下，输入句子被分为单词，当大规模语言模型将其表示为“令牌”时，每个单词都会以数字ID表示。

Joshua Hathcock, the developer of Meaning Machine, explains that large-scale language models do not process entire sentences together, but rather split words and character sets into numerical IDs called tokens and process them abstractly. For example, in the case of the GPT model on which ChatGPT is based, common words such as 'The,' 'young,' 'student,' 'didn't,' and 'submit' are often represented by a single token, but rare words are split into multiple tokens made up of combinations of subwords. The large-scale language model then identifies the grammatical role of each token and infers the subject, verb, object, etc. of the sentence. In the example sentence, the subject is 'student,' the verb is 'submit,' and the object is 'report.'

意义机器的开发人员Joshua Hathcock解释说，大规模的语言模型不会将整个句子一起处理在一起，而是将单词和字符设置为称为令牌的数字ID并抽象地处理它们。例如，在GPT模型的情况下，ChatGpt所基于的，例如“'''Young，''学生，“没有”和“提交”等常见词，通常由一个令牌表示，但是稀有的单词分为多个令牌，由子词组合组成。然后，大规模的语言模型识别每个令牌的语法作用，并渗透句子的主题，动词，对象等。在示例句子中，主题是“学生”，动词为“提交”，对象为“报告”。

The large-scale language model tags each token with its part of speech (POS), maps dependencies in a sentence, and structures and represents the sentence.

大规模的语言模型标记每个令牌及其语音的一部分（POS），句子中的地图依赖项以及结构并代表句子。

The meaning of the dependency strings is explained in the table at the bottom of the Meaning Machine page.

依赖关系字符串的含义在含义机器页面底部的表中说明。

Each token is then converted into a list (vector) of hundreds of numbers that capture its meaning and context. The figure below shows each token in the example sentence visualized in two dimensions through dimensionality reduction.

然后将每个令牌转换为捕获其含义和上下文的数百个数字的列表（向量）。下图显示了示例句子中通过降低维度可视化的每个令牌。

Below that is a tree showing the dependencies of each token, which shows which tokens depend on which other tokens, and what the whole picture means.

下面是一棵树，显示了每个令牌的依赖性，该树显示了哪些令牌取决于其他令牌以及整个图片的含义。

You can navigate through the dependencies by dragging the bar at the bottom of the diagram left and right.

您可以通过在左右图底部拖动栏来浏览依赖项。

In Meaning Machine, you can enter any sentence you like into the input form at the top of the page to see how the large-scale language model converts each word into a token and how it captures the dependencies of the entire sentence.

在“含义机器”中，您可以将自己喜欢的任何句子输入页面顶部的输入形式中，以查看大规模语言模型如何将每个单词转换为令牌，以及它如何捕获整个句子的依赖项。

'These technical steps reveal something deeper: language models don't understand language the way humans do,' Hathcock said. 'They simulate language convincingly, but in a fundamentally different way. When you or I say 'dog,' we might recall the feel of fur, the sound of a bark, and even an emotional response. But when a large-scale language model sees the word 'dog,' it sees a vector of numbers formed by the frequency with which 'dog' appears near words like 'bark,' 'tail,' 'vet,' and so on. This is not wrong; it has statistical meaning. But this has no substance, no basis, no knowledge.' In other words, large-scale language models and humans process language fundamentally differently, and no matter how human-like a response may be, there are no beliefs or goals.

哈斯科克说：“这些技术步骤揭示了更深入的问题：语言模型不像人类那样理解语言。” ``他们令人信服地模拟语言，但根本不同。当您或我说“狗”时，我们可能会回想起皮毛的感觉，树皮的声音，甚至是情感反应。但是，当一个大规模的语言模型看到“狗”一词时，它看到了一个数字向量，该数字是由“狗”出现在'bark'''''''''tail'''''''''''等频率的频率中。这不是错的；它具有统计含义。但这没有实质，没有基础，没有知识。”换句话说，大规模的语言模型和人类从根本上处理语言，无论反应可能有多像人类，都没有信念或目标。

Despite this, large-scale language models are already widely used in society, creating people's resumes, filtering content, and sometimes even determining what is valuable. Since AI is already becoming a social infrastructure, Hathcock argued that it is important to know the difference in performance and understanding of large-scale language models.

尽管如此，大规模的语言模型已经在社会中广泛使用，创建了人们的简历，过滤内容，有时甚至确定什么是有价值的。由于AI已经成为社会基础设施，因此Hathcock认为，了解大型语言模型的性能和理解的差异很重要。