openNMT 관련 파라미터

Transformer model 사용

- config (quickstart에 있는 ex와 example안에 transformer yml 참고)

 

• param_init_glorot & param_init 0: correct initialization of parameters;
• position_encoding: add sinusoidal position encoding to each embedding; (true)
• optim: "adam", decay_method: "noam", warmup_steps: 8000: use special learning rate;
• batch_type: "tokens", normalization: "tokens" : batch and normalize based on number of tokens and not sentences;
• accum_count 4: compute gradients based on four batches;
• label_smoothing 0.1: use label smoothing loss.

param_init: 0.0
param_init_glorot: 'true'
position_encoding: 'true'
optim: "adam"
decay_method: "noam"
batch_type: "tokens"
normalization: "tokens"
accum_count: 8
label_smoothing: 0.1

Tokenization parameter

- tokenizer : sentencepiece -  onmt.transforms.misc.SentencePieceTransform

- 먼저 sentencepiece로 만든 모델의 path

- Common options for the tokenization transforms 

• src_subword_model: path of source side (or both if shared) subword model; 
• tgt_subword_model: path of target side subword model;
• src_subword_nbest: number of candidates for subword regularization (sentencepiece), source side; 1
• tgt_subword_nbest: number of candidates for subword regularization (sentencepiece), target_side; 1
• src_subword_alpha: smoothing parameter for sentencepiece regularization / dropout probability for BPE, source side; 0.0
• tgt_subword_alpha: smoothing parameter for sentencepiece regularization / dropout probability for BPE, target side.
• src_subword_type: type of subword model for source side (from ["none", "sentencepiece", "bpe"]);
• tgt_subword_type: type of subword model for target side (from ["none", "sentencepiece", "bpe"]);
• src_onmttok_kwargs: additional kwargs for pyonmttok Tokenizer class, source side;
• tgt_onmttok_kwargs: additional kwargs for pyonmttok Tokenizer class, target side.

src_subword_model: examples/subword.spm.model
tgt_subword_model: examples/subword.spm.model
src_subword_nbest: 1
tgt_subword_nbest: 1
src_subword_alpha: 0.0
tgt_subword_alpha: 0.0
src_subword_type: sentencepiece
tgt_subword_type: sentencepiece
src_onmttok_kwargs: "{'mode': 'aggressive', 'spacer_annotate': True}"
tgt_onmttok_kwargs: "{'mode': 'aggressive', 'spacer_annotate': True}"