Figure 2. Fig. 2 Overview of annotation workflow. The MeSH vocabulary (TTL/CSV) is semantically normalized into Minimal Word-Tagged (MWT) dictionaries, which are mapped to a controlled vocabulary and passed to the Whatizit Tool Docker alongside the RELISH documents (XML). Following entity normal ization, annotated RELISH documents are produced and used to generate a MeSH concept-document mapping index (TSV). The example illustrates how recognized MeSH terms are normalized to the same MeSH concept and identifier (Influenza, Human; D007251) and how the corresponding MeSH identifier is subsequently used in the normalized corpus解説: MeSHアノテーションワークフローの概要です。MeSH語彙が正規化され、WhatizitツールでRELISH文書にMeSH用語がアノテーションされます。これにより、「influenza」や「human flu」といった異なる表記が、最終的に同じMeSH識別子「MeSHD007251」に正規化される例が示されています。
本研究では、MeSHベースのアノテーションを文書埋め込みに統合する3つのハイブリッド手法が比較されています。これらの手法は「プレアノテーション」、「ポストアノテーション」、「ポストリダクション」と名付けられています。研究チームはこれらのハイブリッド手法を、TF-IDFのような伝統的な手法、Word2Vec、fastText、Doc2Vecといった標準的なニューラル埋め込み、さらにBioBERT、SciBERT、SPECTER、SapBERTといった公開されているTransformerベースモデルとベンチマーク比較しました。評価指標としては、コサイン類似度やWord Mover’s Distance（WMD）が用いられました。ベンチマーク実験は、専門家が文書ペア間の関連性を手作業でラベル付けした「RELISHコーパス」に基づいています。このコーパスは、2クラス（関連 vs. 非関連）および3クラス（関連、部分関連、非関連）の判断を提供します。
Figure 1. Fig. 1 Overview of the experimental workflow. Documents from the RELISH corpus (PubMed titles and abstracts) are processed through two parallel pipelines: standard text preprocessing and MeSH annotation using the MeSH vocabulary. The resulting representations are fed into three categories of document representation methods: standard text methods (TF-IDF, Word2Vec, fastText, Doc2Vec), transformer methods (BioBERT, SciBERT, SPECTER) and MeSH-augmented methods (pre-annotation, post-annotation, post-reduction) and document similarity is computed using cosine similarity or Word Mover’s Distance (WMD). Performance is evaluated using Precision@N and nDCG@N解説: 本研究の実験ワークフローの概要図です。RELISHコーパスの文書が前処理され、MeSH語彙を用いたアノテーションパイプラインと標準テキスト処理を経て、様々な文書表現手法（標準テキスト、Transformer、MeSH強化型）に入力されます。その後、類似度計算と評価が行われます。
Figure 3. Fig. 3 Pre-annotation hybrid pipeline. The annotated RELISH dataset (TSV) is split into training and test sets. During model training, preprocessed tokens from the annotated text are passed through an embedding model to generate word embeddings, including embeddings associated with normalized MeSH identifiers. Document-level embeddings are then computed by extracting and aggregating these embeddings via centroid calculation. The ex ample illustrates this process at a whole word-level for clarity, the exact tokenization (e.g., whole-word vs. subword) depends on the specific embedding model used解説: プレアノテーションハイブリッドパイプラインの概念図です。アノテーションされたRELISHデータセットのテキストからトークンが生成され、埋め込みモデルで単語埋め込みが作成されます。その後、これらの単語埋め込みを統合することで、文書レベルの埋め込みが計算されます。
Figure 4. Fig. 4 Post-annotation (b) and post-reduction (c) hybrid pipelines. In both strategies, the RELISH dataset is preprocessed and passed through an em bedding model trained on plain text. MeSH concept embeddings are derived from the MeSH concept-document index and used to extend the model vocabulary. In post-annotation, MeSH IDs are appended to their corresponding terms in the document to enrich the final document embeddings. In post-reduction, MeSH terms are replaced by their corresponding MeSH IDs before embedding, allowing concept-level identifiers to be directly embed ded into the document representation. The example illustrates this process at a whole word-level for clarity, the exact tokenization (e.g., whole-word vs. subword) depends on the specific embedding model used解説: ポストアノテーション（b）とポストリダクション（c）のハイブリッドパイプラインの概念図です。どちらの戦略も、MeSH概念埋め込みをモデルの語彙に拡張するために使用されますが、MeSHの情報を文書表現に統合する方法が異なっています。
Table 3. Table 3 Precision@5 under the 2-class relevance setting for each base embedding architecture and its MeSH-augmented variants. Bold indicates an improvement over the corresponding baseline; “–” denotes a strategy not applicable to that architecture Base architecture解説: 2クラス関連性設定におけるPrecision@5を、各ベース埋め込みアーキテクチャとそのMeSH強化型バリアントで比較した表です。Doc2VecとfastTextのハイブリッド手法でベースラインからの改善が太字で示されています。
Figure 6. Fig. 6 Effect of MeSH augmentation on retrieval performance measured by Precision@5 under the 2-class relevance setting. Values represent the differ ence in Precision@5 between each MeSH-augmented hybrid method and its corresponding baseline embedding model without MeSH enrichment. Bars are grouped by base architecture and colored by integration strategy; positive values (right) indicate improved retrieval, negative values (left) indicate a decrease解説: 2クラス関連性設定におけるPrecision@5に対するMeSH強化の効果を示しています。各バーは、MeSH強化型ハイブリッド手法が対応するベースライン埋め込みモデルと比較してPrecision@5でどれだけ変化したかを表しています。
Figure 5. Fig. 5 Distribution of MeSH concept document frequency in the RELISH corpus. For each annotated MeSH concept, the number of documents in which it occurs is counted. The distribution is strongly right-skewed: roughly 9,700 concepts occur in only a single article, whereas a small number of generic concepts appear in a large share of documents, with the most frequent concept occurring in 43,157 articles). The x-axis is truncated at 50 articles, which covers 83.8% of concepts. Only a small subset of the vocabulary is therefore broadly shared across the corpus, consistent with the long-tailed distribution of MeSH concepts typically observed in biomedical text解説: RELISHコーパスにおけるMeSH概念の文書頻度分布を示しています。ほとんどの概念は少数の記事にしか出現せず、ごく一部の概念が多くの記事に広く分布している「ロングテール」な傾向が見られます。
Table 2. Table 2 Overview over the occurrences of MeSH concepts across the RELISH Corpus Frequency MeSH concept count i nline-eq-IEq2 20,564 i nline-eq-IEq3 22,224 i nline-eq-IEq4 24.413 i nline-eq-IEq5 28,120 i nline-eq-IEq6 34,525解説: RELISHコーパス全体におけるMeSH概念の出現頻度をまとめたものです。各頻度範囲で該当するMeSH概念の数が示されており、少なくとも1回出現する概念の総数が34,525であることが分かります。
Figure 7. Fig. 7 Mean within-category Jaccard overlap of MeSH concepts for document pairs in the RELISH test set, stratified by relevance level. Relevant docu ment pairs generally exhibit greater concept overlap than partially relevant and non-relevant pairs, particularly within biomedical categories such as diseases, chemicals and drugs, organisms and health care, indicating that relevance information is concentrated within specific regions of the MeSH hierarchy解説: RELISHテストセットにおけるMeSH概念のカテゴリ内Jaccardオーバーラップを、関連性レベル（非関連、部分関連、関連）ごとに層別化して平均値で示しています。関連文書ペアは、特に疾患や化学物質などのカテゴリで、他のペアよりも概念オーバーラップが大きい傾向が見られます。
Figure 8. Fig. 8 Distribution of lexical variant richness for annotated MeSH concepts in the RELISH test collection. The majority of concepts occur under a single observed surface form, while only a small fraction exhibit substantial lexical diversity. This limits the potential gains obtainable through synonym normal ization alone解説: RELISHテストコレクションにおけるアノテーションされたMeSH概念の語彙変形の豊富さの分布です。大部分の概念は単一の表面形式で出現し、複数の語彙変形を持つ概念は少数に留まっていることが示されています。
は、MeSH概念あたりの語彙変形の豊富さの分布を示しています。
検索性能の比較
Table 4. Table 4 Comparison of retrieval performance under the 2-class relevance setting (relevant + partially relevant versus non-relevant). Approach Training type P@5 nDCG@5 whatizit-dictionary – 0.820 0.642 word2doc2vec P 0.793 0.617 word2doc2vec T 0.815 0.662 pre-annotation-word2doc2vec T 0.797 0.648 post-annotation-word2doc2vec T 0.812 0.657 postreduction-word2doc2vec T 0.810 0.655 wmd-word2vec P 0.777 0.636 wmd-word2vec T 0.813 0.666 pre-annotation-wmd-word2vec T 0.802 0.665 post-annotation-wmd-word2vec T 0.802 0.666 postreduction-wmd-word2vec T 0.792 0.657 fasttext P 0.771 0.611 fasttext T 0.800 0.656 pre-annotation-fasttext T 0.793 0.653 post-annotation-fasttext T 0.808 0.656 postreduction-fasttext T 0.807 0.664 doc2vec T 0.823 0.648 pre-annotation-doc2vec T 0.839 0.660 BioBERT-base P 0.782 0.623 BioBERT-base F 0.907 0.777 BioBERT-large P 0.802 0.628 BioBERT-large F 0.890 0.754 SciBERT P 0.793 0.624 SciBERT F 0.875 0.730 SPECTER P 0.864 0.732 SPECTER F 0.879 0.766 SapBERT P 0.877 0.746 SapBERT F 0.887 0.775解説: 2クラス関連性設定における様々な検索手法のPrecision@5およびnDCG@5の比較です。標準手法、MeSH強化型ハイブリッド手法、Transformerベース手法が含まれ、ファインチューニングされたBERTベースモデルが優れた性能を示しています。
Figure 9. Fig. 9 Absolute improvement in Precision@5 (inline-eq-IEq7P@5) between trained and pre-trained hybrid embedding models for the 2-class relevance setting. Bars are sorted in descending order of improvement. FastText-based hybrid methods (post-fasttext, postred-fasttext) show the largest gains, while pre-word 2doc2vec shows the smallest improvement, suggesting that annotation timing has a limited and architecture-dependent impact on retrieval precision
Figure 10
Figure 10. Fig. 10 Absolute improvement in Precision@5 (inline-eq-IEq8P@5) between trained and pre-trained hybrid embedding models for the 3-class relevance setting. Overall gains are larger compared to the 2-class setting, reflecting more improvement when distinguishing between relevant, partially relevant and non-relevant documents. FastText-based methods again lead, while wmd-based methods show the most consistent performance across annotation strategies
Figure 11
Figure 11. Fig. 11 Vocabulary characteristics versus retrieval performance under the 2-class relevance setting. (a) Vocabulary size versus Precision@5 shows that larger vocabularies do not consistently lead to higher retrieval precision. wmd-word2vec achieves among the highest precision despite having one of the largest vocabularies, while word2doc2vec achieves the highest precision with a comparatively smaller vocabulary. (b) Out-of-vocabulary (OOV) count versus Precision@5 shows that higher OOV counts do not necessarily reduce performance, word2doc2vec and wmd-word2vec maintain the highest pre cision despite exhibiting the largest OOV counts, while fastText-based methods show near-zero OOV counts due to subword-based representation but do not outperform word-level methods. Marker shapes distinguish method types: circles indicate wmd-word2vec-based methods, diamonds indicate word2doc2vec-based methods and squares indicate fastText-based methods
Leave a Reply