G2PT：图生成预训练 Transformer

来自塔夫茨大学、东北大学和康奈尔大学的研究人员开发了图生成预训练变压器（G2PT），这是一种自回归模型，旨在通过下一个令牌预测来学习图结构。

Graph generation is a critical task in diverse fields like molecular design and social network analysis, owing to its capacity to model intricate relationships and structured data. Despite recent advances, many graph generative models heavily rely on adjacency matrix representations. While effective, these methods can be computationally demanding and often lack flexibility, making it challenging to efficiently capture the complex dependencies between nodes and edges, especially for large and sparse graphs. Current approaches, including diffusion-based and auto-regressive models, encounter difficulties in terms of scalability and accuracy, highlighting the need for more refined solutions.

图生成是分子设计和社交网络分析等不同领域的一项关键任务，因为它能够建模复杂的关系和结构化数据。尽管最近取得了进展，但许多图生成模型严重依赖邻接矩阵表示。虽然有效，但这些方法可能对计算要求很高，并且通常缺乏灵活性，使得有效捕获节点和边之间的复杂依赖关系具有挑战性，特别是对于大型稀疏图。当前的方法，包括基于扩散和自回归模型，在可扩展性和准确性方面遇到了困难，这凸显了对更精细解决方案的需求。

In a recent study, a team of researchers from Tufts University, Northeastern University, and Cornell University introduces the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model designed to learn graph structures through next-token prediction. Unlike traditional methods, G2PT employs a sequence-based representation of graphs, encoding nodes and edges as sequences of tokens. This approach streamlines the modeling process, making it more efficient and scalable. By leveraging a transformer decoder for token prediction, G2PT generates graphs that maintain structural integrity and flexibility. Moreover, G2PT can be readily adapted to downstream tasks, such as goal-oriented graph generation and graph property prediction, serving as a versatile tool for various applications.

在最近的一项研究中，来自塔夫茨大学、东北大学和康奈尔大学的研究人员团队介绍了图生成预训练 Transformer (G2PT)，这是一种自回归模型，旨在通过下一个标记预测来学习图结构。与传统方法不同，G2PT 采用基于序列的图表示，将节点和边编码为标记序列。这种方法简化了建模过程，使其更加高效和可扩展。通过利用 Transformer 解码器进行令牌预测，G2PT 生成保持结构完整性和灵活性的图表。此外，G2PT可以很容易地适应下游任务，例如面向目标的图生成和图属性预测，作为各种应用的多功能工具。

Technical Insights and Benefits

技术见解和优势

G2PT introduces a novel sequence-based representation that decomposes graphs into node and edge definitions. Node definitions specify indices and types, whereas edge definitions outline connections and labels. This approach fundamentally differs from adjacency matrix representations, which focus on all possible edges, by considering only the existing edges, thereby reducing sparsity and computational complexity. The transformer decoder effectively models these sequences through next-token prediction, offering several advantages:

G2PT 引入了一种新颖的基于序列的表示，将图分解为节点和边定义。节点定义指定索引和类型，而边定义概述连接和标签。这种方法从根本上不同于邻接矩阵表示，邻接矩阵表示通过仅考虑现有边缘来关注所有可能的边缘，从而减少稀疏性和计算复杂性。 Transformer 解码器通过下一个令牌预测有效地对这些序列进行建模，具有以下几个优点：

The researchers also explored fine-tuning methods for tasks like goal-oriented generation and graph property prediction, broadening the model’s applicability.

研究人员还探索了面向目标的生成和图属性预测等任务的微调方法，扩大了模型的适用性。

Experimental Results and Insights

实验结果和见解

G2PT has been evaluated on various datasets and tasks, demonstrating strong performance. In general graph generation, it matched or exceeded the state-of-the-art performance across seven datasets. In molecular graph generation, G2PT achieved high validity and uniqueness scores, reflecting its ability to accurately capture structural details. For instance, on the MOSES dataset, G2PTbase attained a validity score of 96.4% and a uniqueness score of 100%.

G2PT 已在各种数据集和任务上进行了评估，展示了强大的性能。在一般图形生成方面，它在七个数据集上达到或超过了最先进的性能。在分子图生成方面，G2PT 取得了很高的有效性和独特性分数，反映了其准确捕捉结构细节的能力。例如，在 MOSES 数据集上，G2PTbase 的有效性得分为 96.4%，唯一性得分为 100%。

In a goal-oriented generation, G2PT aligned generated graphs with desired properties using fine-tuning techniques like rejection sampling and reinforcement learning. These methods enabled the model to adapt its outputs effectively. Similarly, in predictive tasks, G2PT’s embeddings delivered competitive results across molecular property benchmarks, reinforcing its suitability for both generative and predictive tasks.

在面向目标的一代中，G2PT 使用拒绝采样和强化学习等微调技术将生成的图与所需的属性对齐。这些方法使模型能够有效地调整其输出。同样，在预测任务中，G2PT 的嵌入在分子特性基准上提供了有竞争力的结果，增强了其对生成任务和预测任务的适用性。

Conclusion

结论

The Graph Generative Pre-trained Transformer (G2PT) represents a thoughtful step forward in graph generation. By employing a sequence-based representation and transformer-based modeling, G2PT addresses many limitations of traditional approaches. Its combination of efficiency, scalability, and adaptability makes it a valuable resource for researchers and practitioners. While G2PT shows sensitivity to graph orderings, further exploration of universal and expressive edge-ordering mechanisms could enhance its robustness. G2PT exemplifies how innovative representations and modeling approaches can advance the field of graph generation.

图生成预训练 Transformer (G2PT) 代表了图生成领域向前迈出的深思熟虑的一步。通过采用基于序列的表示和基于变压器的建模，G2PT 解决了传统方法的许多局限性。它集效率、可扩展性和适应性于一体，使其成为研究人员和从业者的宝贵资源。虽然 G2PT 对图排序表现出敏感性，但对通用和富有表现力的边排序机制的进一步探索可以增强其鲁棒性。 G2PT 举例说明了创新的表示和建模方法如何推动图形生成领域的发展。

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 60k+ ML SubReddit.

查看论文。这项研究的所有功劳都归功于该项目的研究人员。另外，不要忘记在 Twitter 上关注我们并加入我们的 Telegram 频道和 LinkedIn 群组。不要忘记加入我们 60k+ ML SubReddit。

🚨 FREE UPCOMING AI WEBINAR (JAN 15, 2025): Boost LLM Accuracy with Synthetic Data and Evaluation Intelligence–Join this webinar to gain actionable insights into boosting LLM model performance and accuracy while safeguarding data privacy.

🚨 即将举行的免费人工智能网络研讨会（2025 年 1 月 15 日）：利用合成数据和评估智能提高 LLM 准确性 - 加入此网络研讨会，获得可操作的见解，以提高 LLM 模型的性能和准确性，同时保护数据隐私。