多層變壓器編碼器，用於學習IVF過程中關鍵元素的學習表示

在愛德華茲（Edwards）中，我們使用多層變壓器編碼器來學習IVF過程中關鍵元素的表示。

In Edwards, we used a multi-layer Transformer Encoder to learn the representations of the key elements in IVF process. These key elements included demographic data (more details shown in Table 1), treatment plans, hormone profiles, and follicular measurements (more details shown in Table 2), categorized and mapped into a lookup dictionary. We applied a self-supervised training method, Masked LM15, as the pre-training strategy. During the pre-training process, these elements were projected into a high-dimension trainable vectorized embedding space through the aforementioned lookup dictionary, the characteristics of each element, and the context of IVF process were thus captured and represented by the vectored embedding space and the parameters of the Transformer Encoder. The downstream tasks (e.g., predicted treatment plans, final outcomes of IVF cycles, etc) are addressed by fine-tuning the pre-trained model. In addition, we developed Edwards-Pro by integrating the knowledge-based decision support system proposed by our previous study7 into Edwards, in order to improve the accessibility of this approach, as well as to improve the predictions of treatment plans.

在愛德華茲（Edwards）中，我們使用多層變壓器編碼器來學習IVF過程中關鍵元素的表示。這些關鍵要素包括人口統計數據（表1所示的更多詳細信息），治療計劃，激素概況和卵泡測量（更多詳細信息，如表2所示），並將其分類並映射到查找字典中。我們應用了一種自我監督的訓練方法，即蒙面的LM15作為訓練前策略。在訓練過程中，通過上述查找詞典，每個元素的特徵，將這些元素投射到可訓練的高維度訓練的矢量化嵌入空間中，因此，IVF過程的上下文被矢量嵌入空間和變壓器編碼的參數捕獲並表示。下游任務（例如，預測的治療計劃，IVF週期的最終結果等）是通過微調預訓練的模型來解決的。此外，我們通過將我們以前的研究7提出的基於知識的決策支持系統整合到Edwards中，以提高這種方法的可訪問性，並改善治療計劃的預測，從而開發了Edwards-Pro。

We used historical clinical data collected over almost ten years from New Hope Fertility Center (NHFC) to train and verify our approach. The clinical data including the aforementioned key elements were collected from patients’ monitoring in every visit. The dataset for training the deep learning model contained 30,552 IVF cycles with 239,047 monitoring visits from January 2013 to December 2021. Another dataset of 1,804 cycles containing 8,364 visits from January 2022 to July 2022 was used as the validation dataset. More details about the data preprocessing, model architecture, and training strategies are addressed in Section 4 and Figure 1.

我們使用了近十年來從新希望生育中心（NHFC）收集的歷史臨床數據來訓練和驗證我們的方法。每次訪問中都從患者的監測中收集了包括上述關鍵要素在內的臨床數據。用於訓練深度學習模型的數據集包含30,552個IVF週期，從2013年1月到2021年12月，有239,047次監視訪問。另一個包含2022年1月至2022年7月8364次訪問的1,804個週期的數據集用作驗證數據集。有關數據預處理，模型架構和培訓策略的更多詳細信息在第4節和圖1中介紹。

Our approach provides predictions for two distinct phases in IVF COS cycles. Phase I focuses on key elements during monitoring visits, such as treatment plans, hormone profiles, and follicular measurements. Predictions for these elements in visit #n are based on all data from the previous #n-1 visits. Phase II targets the final outcomes of IVF cycles, such as MII rate, 2PN rate, and blastulation rate (more details shown in Table 3), predicted using data from the entire IVF cycle (Table 4).

我們的方法為IVF COS循環中的兩個不同階段提供了預測。第一階段的重點是監測訪問期間的關鍵要素，例如治療計劃，激素概況和卵泡測量。訪問＃N中這些元素的預測基於上一個＃N-1訪問中的所有數據。 II階段的目標是使用整個IVF循環的數據預測IVF週期的最終結果，例如MII率，2PN速率和洩漏速率（表3所示的更多詳細信息）。

Both phases were framed as classification tasks for two reasons: 1. Classification tasks align naturally with our approach, where key elements of the IVF process are categorized into data points for the training and validation datasets. 2. Clinically, REI specialists typically make decisions based on ranges of hormone profiles and follicular measurements rather than exact values. Additionally, the rates of MII, 2PN, and blastulation, defined as proportions of retrieved oocytes, are more accurate criteria for assessing IVF outcomes, as they correlate closely with patient factors such as age, ovarian reserve, and stimulation response.

這兩個階段均被構架為分類任務的原因有兩個：1。分類任務與我們的方法自然對齊，其中IVF過程的關鍵要素分為培訓和驗證數據集的數據點。 2。從臨床上講，REI專家通常會根據激素概況和卵泡測量而不是精確的值做出決策。此外，定義為檢索卵母細胞比例的MII，2PN和洩漏的速率是評估IVF結局的更準確標準，因為它們與年齡，卵巢儲備和刺激反應等患者因素緊密相關。

We designed a targeted evaluation strategy for these two-phase predictions. For Phase I, which can be applied during any monitoring visit, we divided the 1,804 cycles in the validation dataset into 8,364 input sequences. In each sequence, data from visits beyond the predicted monitoring visit were excluded. For Phase II, we used the full dataset from each cycle, as final IVF outcomes depend on the entire ovarian stimulation process. To benchmark our deep learning model, we implemented traditional machine learning approaches referenced in prior studies6,8. Additionally, we developed a sequential learning baseline model-Sequence-to-Sequence (Seq2Seq)16, based on Long Short-Term Memory (LSTM) units17, to assess our model’s ability to capture temporal features effectively.

我們為這些兩相預測設計了有針對性的評估策略。對於可以在任何監視訪問期間應用的第一階段，我們將驗證數據集中的1,804個週期分為8,364個輸入序列。在每個序列中，排除了預測監視訪問以外的訪問數據。對於第二階段，我們使用了每個週期中的完整數據集，因為最終的IVF結果取決於整個卵巢刺激過程。為了基准我們的深度學習模型，我們實施了先前研究中引用的傳統機器學習方法6,8。此外，我們基於長期記憶（LSTM）單位開發了一個順序學習基線模型 - 序列對序列（SEQ2SEQ）16，以評估我們模型有效捕獲時間特徵的能力。

The main distinction between Edwards-Pro and Edwards lies in Edwards-Pro’s enhanced ability to predict treatment plans; both models performed identically for other prediction categories. In nearly all treatment plan categories (Table 5), sequential learning models, including Seq2Seq, Edwards, and Edwards-Pro-outperformed traditional machine learning approaches, achieving improvements of at least 10% in average precision (AP), 14% in the area under the receiver operating characteristic curve (AUROC), and 4% in top-2 accuracy. The exception was the Follitropin category, which had an imbalanced label set; while AdaBoost achieved the best AP (93.0%), this was due to predicting only the dominant class. For categories linked to clinical judgment, such as Day# (next visit date), Follitropin (COS dosage), and oral contraceptives, Edwards-Pro improved Edwards’s performance by 2.9% (AP), 5.8% (AUROC), and 11.6% (top-2 accuracy). In clinical assessment-related predictions (Table 6), sequential learning models excelled across all categories except FSH and follicular measurements, both of which had imbalanced datasets similar to Follitropin. Conversely, for E

Edwards-Pro和Edwards之間的主要區別在於Edwards-Pro預測治療計劃的增強能力。兩種模型對於其他預測類別都相同。在幾乎所有的治療計劃類別（表5）中，包括SEQ2SEQ，EDWARDS和Edwards-Pro脫穎而出的傳統機器學習方法在內的順序學習模型，在平均精度（AP）中取得了至少10％的改善，在接收器操作特徵曲線（AUROC）下的區域中，提高了14％，在TOP-2準確度中提高了4％。例外是follitropin類別，其標籤設置不平衡。儘管Adaboost獲得了最佳AP（93.0％），但這是由於僅預測主要類別。對於與臨床判斷相關的類別，例如Day＃（下一個訪問日期），Follitropin（COS劑量）和口服避孕藥，Edwards-Pro將Edwards的表現提高了2.9％（AP），5.8％（AUROC）和11.6％（前2位準確性）。在與臨床評估相關的預測中（表6），除了FSH和濾泡測量以外的所有類別中，順序學習模型都具有卓越的方式，這兩種模型的數據集類似於follitropin。相反，對於e