tab transformer pytorch

탭 변환기

Pytorch에서 표 형식 데이터에 대한 주의 네트워크인 Tab Transformer를 구현합니다. 이 간단한 아키텍처는 GBDT 성능의 아주 넓은 범위에 포함되었습니다.

업데이트: Amazon AI는 실제 표 형식 데이터 세트(배송비 예측)에 대한 Attention을 통해 GBDT를 능가했다고 주장합니다.

설치하다

$ pip install tab-transformer-pytorch

용법

 import torch
import torch . nn as nn
from tab_transformer_pytorch import TabTransformer

cont_mean_std = torch . randn ( 10 , 2 )

model = TabTransformer (
    categories = ( 10 , 5 , 6 , 5 , 8 ),      # tuple containing the number of unique values within each category
    num_continuous = 10 ,                # number of continuous values
    dim = 32 ,                           # dimension, paper set at 32
    dim_out = 1 ,                        # binary prediction, but could be anything
    depth = 6 ,                          # depth, paper recommended 6
    heads = 8 ,                          # heads, paper recommends 8
    attn_dropout = 0.1 ,                 # post-attention dropout
    ff_dropout = 0.1 ,                   # feed forward dropout
    mlp_hidden_mults = ( 4 , 2 ),          # relative multiples of each hidden dimension of the last mlp to logits
    mlp_act = nn . ReLU (),                # activation for final mlp, defaults to relu, but could be anything else (selu etc)
    continuous_mean_std = cont_mean_std # (optional) - normalize the continuous values before layer norm
)

x_categ = torch . randint ( 0 , 5 , ( 1 , 5 ))     # category values, from 0 - max number of categories, in the order as passed into the constructor above
x_cont = torch . randn ( 1 , 10 )               # assume continuous values are already normalized individually

pred = model ( x_categ , x_cont ) # (1, 1) 

FT 트랜스포머

Yandex의 이 문서는 이 reddit 게시물에서 제공하는 위 다이어그램에 표시된 것처럼 연속 숫자 값을 삽입하는 더 간단한 방식을 사용하여 Tab Transformer를 개선합니다.

Tab Transformer와의 편리한 비교를 위해 이 저장소에 포함됨

 import torch
from tab_transformer_pytorch import FTTransformer

model = FTTransformer (
    categories = ( 10 , 5 , 6 , 5 , 8 ),      # tuple containing the number of unique values within each category
    num_continuous = 10 ,                # number of continuous values
    dim = 32 ,                           # dimension, paper set at 32
    dim_out = 1 ,                        # binary prediction, but could be anything
    depth = 6 ,                          # depth, paper recommended 6
    heads = 8 ,                          # heads, paper recommends 8
    attn_dropout = 0.1 ,                 # post-attention dropout
    ff_dropout = 0.1                    # feed forward dropout
)

x_categ = torch . randint ( 0 , 5 , ( 1 , 5 ))     # category values, from 0 - max number of categories, in the order as passed into the constructor above
x_numer = torch . randn ( 1 , 10 )              # numerical value

pred = model ( x_categ , x_numer ) # (1, 1) 

감독되지 않은 훈련

논문에 설명된 비지도 학습 유형을 받으려면 먼저 카테고리 토큰을 적절한 고유 ID로 변환한 다음 model.transformer 에서 Electra를 사용할 수 있습니다.

토도

• https://arxiv.org/abs/2203.05556을 고려하십시오.

인용

 @misc { huang2020tabtransformer ,
    title   = { TabTransformer: Tabular Data Modeling Using Contextual Embeddings } ,
    author  = { Xin Huang and Ashish Khetan and Milan Cvitkovic and Zohar Karnin } ,
    year    = { 2020 } ,
    eprint  = { 2012.06678 } ,
    archivePrefix = { arXiv } ,
    primaryClass = { cs.LG }
}

 @article { Gorishniy2021RevisitingDL ,
    title   = { Revisiting Deep Learning Models for Tabular Data } ,
    author  = { Yu. V. Gorishniy and Ivan Rubachev and Valentin Khrulkov and Artem Babenko } ,
    journal = { ArXiv } ,
    year    = { 2021 } ,
    volume  = { abs/2106.11959 }
}

 @article { Zhu2024HyperConnections ,
    title   = { Hyper-Connections } ,
    author  = { Defa Zhu and Hongzhi Huang and Zihao Huang and Yutao Zeng and Yunyao Mao and Banggu Wu and Qiyang Min and Xun Zhou } ,
    journal = { ArXiv } ,
    year    = { 2024 } ,
    volume  = { abs/2409.19606 } ,
    url     = { https://api.semanticscholar.org/CorpusID:272987528 }
}