多层变压器编码器，用于学习IVF过程中关键元素的学习表示

在爱德华兹（Edwards）中，我们使用多层变压器编码器来学习IVF过程中关键元素的表示。

In Edwards, we used a multi-layer Transformer Encoder to learn the representations of the key elements in IVF process. These key elements included demographic data (more details shown in Table 1), treatment plans, hormone profiles, and follicular measurements (more details shown in Table 2), categorized and mapped into a lookup dictionary. We applied a self-supervised training method, Masked LM15, as the pre-training strategy. During the pre-training process, these elements were projected into a high-dimension trainable vectorized embedding space through the aforementioned lookup dictionary, the characteristics of each element, and the context of IVF process were thus captured and represented by the vectored embedding space and the parameters of the Transformer Encoder. The downstream tasks (e.g., predicted treatment plans, final outcomes of IVF cycles, etc) are addressed by fine-tuning the pre-trained model. In addition, we developed Edwards-Pro by integrating the knowledge-based decision support system proposed by our previous study7 into Edwards, in order to improve the accessibility of this approach, as well as to improve the predictions of treatment plans.

在爱德华兹（Edwards）中，我们使用多层变压器编码器来学习IVF过程中关键元素的表示。这些关键要素包括人口统计数据（表1所示的更多详细信息），治疗计划，激素概况和卵泡测量（更多详细信息，如表2所示），并将其分类并映射到查找字典中。我们应用了一种自我监督的训练方法，即蒙面的LM15作为训练前策略。在训练过程中，通过上述查找词典，每个元素的特征，将这些元素投射到可训练的高维度训练的矢量化嵌入空间中，因此，IVF过程的上下文被矢量嵌入空间和变压器编码的参数捕获并表示。下游任务（例如，预测的治疗计划，IVF周期的最终结果等）是通过微调预训练的模型来解决的。此外，我们通过将我们以前的研究7提出的基于知识的决策支持系统整合到Edwards中，以提高这种方法的可访问性，并改善治疗计划的预测，从而开发了Edwards-Pro。

We used historical clinical data collected over almost ten years from New Hope Fertility Center (NHFC) to train and verify our approach. The clinical data including the aforementioned key elements were collected from patients’ monitoring in every visit. The dataset for training the deep learning model contained 30,552 IVF cycles with 239,047 monitoring visits from January 2013 to December 2021. Another dataset of 1,804 cycles containing 8,364 visits from January 2022 to July 2022 was used as the validation dataset. More details about the data preprocessing, model architecture, and training strategies are addressed in Section 4 and Figure 1.

我们使用了近十年来从新希望生育中心（NHFC）收集的历史临床数据来训练和验证我们的方法。每次访问中都从患者的监测中收集了包括上述关键要素在内的临床数据。用于训练深度学习模型的数据集包含30,552个IVF周期，从2013年1月到2021年12月，有239,047次监视访问。另一个包含2022年1月至2022年7月8364次访问的1,804个周期的数据集用作验证数据集。有关数据预处理，模型架构和培训策略的更多详细信息在第4节和图1中介绍。

Our approach provides predictions for two distinct phases in IVF COS cycles. Phase I focuses on key elements during monitoring visits, such as treatment plans, hormone profiles, and follicular measurements. Predictions for these elements in visit #n are based on all data from the previous #n-1 visits. Phase II targets the final outcomes of IVF cycles, such as MII rate, 2PN rate, and blastulation rate (more details shown in Table 3), predicted using data from the entire IVF cycle (Table 4).

我们的方法为IVF COS循环中的两个不同阶段提供了预测。第一阶段的重点是监测访问期间的关键要素，例如治疗计划，激素概况和卵泡测量。访问＃N中这些元素的预测基于上一个＃N-1访问中的所有数据。 II阶段的目标是使用整个IVF循环的数据预测IVF周期的最终结果，例如MII率，2PN速率和泄漏速率（表3所示的更多详细信息）。

Both phases were framed as classification tasks for two reasons: 1. Classification tasks align naturally with our approach, where key elements of the IVF process are categorized into data points for the training and validation datasets. 2. Clinically, REI specialists typically make decisions based on ranges of hormone profiles and follicular measurements rather than exact values. Additionally, the rates of MII, 2PN, and blastulation, defined as proportions of retrieved oocytes, are more accurate criteria for assessing IVF outcomes, as they correlate closely with patient factors such as age, ovarian reserve, and stimulation response.

这两个阶段均被构架为分类任务的原因有两个：1。分类任务与我们的方法自然对齐，其中IVF过程的关键要素分为培训和验证数据集的数据点。 2。从临床上讲，REI专家通常会根据激素概况和卵泡测量而不是精确的值做出决策。此外，定义为检索卵母细胞比例的MII，2PN和泄漏的速率是评估IVF结局的更准确标准，因为它们与年龄，卵巢储备和刺激反应等患者因素紧密相关。

We designed a targeted evaluation strategy for these two-phase predictions. For Phase I, which can be applied during any monitoring visit, we divided the 1,804 cycles in the validation dataset into 8,364 input sequences. In each sequence, data from visits beyond the predicted monitoring visit were excluded. For Phase II, we used the full dataset from each cycle, as final IVF outcomes depend on the entire ovarian stimulation process. To benchmark our deep learning model, we implemented traditional machine learning approaches referenced in prior studies6,8. Additionally, we developed a sequential learning baseline model-Sequence-to-Sequence (Seq2Seq)16, based on Long Short-Term Memory (LSTM) units17, to assess our model’s ability to capture temporal features effectively.

我们为这些两相预测设计了有针对性的评估策略。对于可以在任何监视访问期间应用的第一阶段，我们将验证数据集中的1,804个周期分为8,364个输入序列。在每个序列中，排除了预测监视访问以外的访问数据。对于第二阶段，我们使用了每个周期中的完整数据集，因为最终的IVF结果取决于整个卵巢刺激过程。为了基准我们的深度学习模型，我们实施了先前研究中引用的传统机器学习方法6,8。此外，我们基于长期记忆（LSTM）单位开发了一个顺序学习基线模型 - 序列对序列（SEQ2SEQ）16，以评估我们模型有效捕获时间特征的能力。

The main distinction between Edwards-Pro and Edwards lies in Edwards-Pro’s enhanced ability to predict treatment plans; both models performed identically for other prediction categories. In nearly all treatment plan categories (Table 5), sequential learning models, including Seq2Seq, Edwards, and Edwards-Pro-outperformed traditional machine learning approaches, achieving improvements of at least 10% in average precision (AP), 14% in the area under the receiver operating characteristic curve (AUROC), and 4% in top-2 accuracy. The exception was the Follitropin category, which had an imbalanced label set; while AdaBoost achieved the best AP (93.0%), this was due to predicting only the dominant class. For categories linked to clinical judgment, such as Day# (next visit date), Follitropin (COS dosage), and oral contraceptives, Edwards-Pro improved Edwards’s performance by 2.9% (AP), 5.8% (AUROC), and 11.6% (top-2 accuracy). In clinical assessment-related predictions (Table 6), sequential learning models excelled across all categories except FSH and follicular measurements, both of which had imbalanced datasets similar to Follitropin. Conversely, for E

Edwards-Pro和Edwards之间的主要区别在于Edwards-Pro预测治疗计划的增强能力。两种模型对于其他预测类别都相同。在几乎所有的治疗计划类别（表5）中，包括SEQ2SEQ，EDWARDS和Edwards-Pro脱颖而出的传统机器学习方法在内的顺序学习模型，在平均精度（AP）中取得了至少10％的改善，在接收器操作特征曲线（AUROC）下的区域中，提高了14％，在TOP-2准确度中提高了4％。例外是follitropin类别，其标签设置不平衡。尽管Adaboost获得了最佳AP（93.0％），但这是由于仅预测主要类别。对于与临床判断相关的类别，例如Day＃（下一个访问日期），Follitropin（COS剂量）和口服避孕药，Edwards-Pro将Edwards的表现提高了2.9％（AP），5.8％（AUROC）和11.6％（前2位准确性）。在与临床评估相关的预测中（表6），除了FSH和滤泡测量以外的所有类别中，顺序学习模型都具有卓越的方式，这两种模型的数据集类似于follitropin。相反，对于e