Q: Piyush
I don't have enough information to answer that.
The context provided discusses technical details about answer span prediction models and BERT architecture, but does not contain any information about "Piyush."
3 source chunks
tor S ∈ RH and an end vector E ∈ RH during
fine-tuning. The probability of word i being the
start of the answer span is computed as a dot prod-
uct betweenTi andS followed by a softmax over
all of the words in the paragraph: Pi = eS·Ti
∑
j eS·Tj .
The analogous formula is used for the end of the
answer span. The score of a candidate span from
positioni to positionj is defined asS·Ti +E·Tj,
and the maximum scoring span where j ≥ i is
used as a prediction. The training objective is the
BERT
E[CLS] E1 E[SEP]... EN E1’ ... EM’
C
T1
T[SEP]...
TN
T1’ ...
TM’
[CLS] Tok
1
[SEP]... Tok
N
Tok
1 ... Tok
M
Question Paragraph
BERT
E[CLS] E1 E2 EN
C
T1
T2
TN
Single Sentence
...
...
BERT
Tok 1 Tok 2 Tok N...[CLS]
E[CLS] E1 E2 EN
C
T1
T2
TN
Single Sentence
B-PERO O
...
...E[CLS] E1 E[SEP]
Class
Label
... EN E1’ ... EM’
C
T1
T[SEP]...
TN
T1’ ...
TM’
Start/End Span
Class
Label
BERT
Tok 1 Tok 2 Tok N...[CLS] Tok 1[CLS][CLS] Tok
1
[SEP]... Tok
N
Tok
level tasks from the GLUE benchmark (Wang
et al., 2018a). Left-to-right language model-
July 13, 2026, 3:25 p.m.
Q: What is the masked language model objective?
Based on the context provided, the masked language model (MLM) objective works as follows:
1. **Random masking**: The model randomly masks some of the tokens from the input (specifically 15% of all WordPiece tokens in each sequence).
2. **Prediction task**: The objective is to predict the original vocabulary id of the masked tokens.
3. **Implementation**: The final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary, similar to a standard language model.
4. **Key difference from similar approaches**: Unlike denoising auto-encoders, the model only predicts the masked words rather than reconstructing the entire input.
The MLM objective was inspired by the Cloze task and helps BERT alleviate the unidirectionality constraint by allowing the model to use bidirectional context when predicting masked tokens.
3 source chunks
replacement only occurs for 1.5% of all tokens
(i.e., 10% of 15%), this does not seem to harm
the model’s language understanding capability. In
Section C.2, we evaluate the impact this proce-
dure.
Compared to standard langauge model training,
the masked LM only make predictions on 15% of
tokens in each batch, which suggests that more
pre-training steps may be required for the model
BERT alleviates the previously mentioned unidi-
rectionality constraint by using a “masked lan-
guage model” (MLM) pre-training objective, in-
spired by the Cloze task (Taylor, 1953). The
masked language model randomly masks some of
the tokens from the input, and the objective is to
predict the original vocabulary id of the masked
arXiv:1810.04805v2 [cs.CL] 24 May 2019
LM” (MLM), although it is often referred to as a
Cloze task in the literature (Taylor, 1953). In this
case, the final hidden vectors corresponding to the
mask tokens are fed into an output softmax over
the vocabulary, as in a standard LM. In all of our
experiments, we mask 15% of all WordPiece to-
kens in each sequence at random. In contrast to
denoising auto-encoders (Vincent et al., 2008), we
only predict the masked words rather than recon-
structing the entire input.
July 12, 2026, 4:43 p.m.
Q: What two pretraining tasks does BERT use?
I don't have enough information to answer that. The context mentions that during pre-training, the BERT model is trained on unlabeled data over "different pre-training tasks," but it does not specify what those two pretraining tasks are.
3 source chunks
and the bidirectionality they enable.
A.5 Illustrations of Fine-tuning on Different
Tasks
The illustration of fine-tuning BERT on different
tasks can be seen in Figure 4. Our task-specific
models are formed by incorporating BERT with
one additional output layer, so a minimal num-
ber of parameters need to be learned from scratch.
Among the tasks, (a) and (b) are sequence-level
tasks while (c) and (d) are token-level tasks. In
the figure, E represents the input embedding, Ti
strated the importance of transfer learning from
large pre-trained models, where an effective recipe
is to fine-tune models pre-trained with Ima-
geNet (Deng et al., 2009; Yosinski et al., 2014).
3 BERT
We introduce BERT and its detailed implementa-
tion in this section. There are two steps in our
framework: pre-training and fine-tuning. Dur-
ing pre-training, the model is trained on unlabeled
data over different pre-training tasks. For fine-
tuning, the BERT model is first initialized with
tuning, the BERT model is first initialized with
the pre-trained parameters, and all of the param-
eters are fine-tuned using labeled data from the
downstream tasks. Each downstream task has sep-
arate fine-tuned models, even though they are ini-
tialized with the same pre-trained parameters. The
question-answering example in Figure 1 will serve
as a running example for this section.
A distinctive feature of BERT is its unified ar-
chitecture across different tasks. There is mini-
July 12, 2026, 4:43 p.m.