Highway Transformer

Highway Transformer ：自門增強型自我關注網絡

此倉庫是使用Self-Dependency Unit的 Transformer-XL 的示範程式碼。這項工作與 Gating-enhanced Transformer 變體密切相關，例如 Google 的 Switch Transformer。

柴業坤等。等人。 ， Highway Transformer ：自門增強型自我關注網路 (ACL 2020)

要求

• PyTorch >= 1.1.0
• TensorboardX >= 1.8
• 張量板 >= 1.14
• 4 個 GPU ，每個8GB內存，用於運行 12 層 Transformer-XL

資料下載

bash getdata.sh

運行 6 層 Transformer-XL

 cd pytorch/xl_L6_scripts && bash < script-name > .sh train --work_dir " PATH_TO_WORK_DIR " 

可視化您的結果

 cd XL-L6-results && tensorboard --logdir=.

結果

• 不同模型設定的線圖，其中最上面的線（紅色）是基準模型（即原始 Transformer-XL）。
• 在添加自依賴單元（請參閱底部兩條曲線）後，很明顯， Highway Transformer加快了訓練和評估過程中的收斂過程。

培訓業務流程	訓練損失

評估bpc	評估損失

引文

對於學術背景下的歸屬，請將本作品引用為：

Highway Transformer: Self-Gating Enhanced Self-Attentive Networks", author = "Chai, Yekun and Jin, Shuo and Hou, Xinwen", booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics", month = jul, year = "2020", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/2020.acl-main.616", pages = "6887--6900" }">

 @inproceedings{chai-etal-2020-highway,
    title = " Highway Transformer : Self-Gating Enhanced Self-Attentive Networks",
    author = "Chai, Yekun  and
      Jin, Shuo  and
      Hou, Xinwen",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.616",
    pages = "6887--6900"
}