ABSTRACT: Implementing technology in a modern-day classroom is an ongoing challenge. 
In this paper, we created a system for an automatic assessment of student answers using Latent Semantic Analysis (lsa) – a method with an underlying assumption that words with similar meanings will appear in the same contexts. 
The system will be used within digital lexical flashcards for L2 vocabulary acquisition in a clil classroom. 
Results presented in this paper indicate that while lsa does well in creating semantic spaces for longer texts, it fell somewhat short of detecting topics in answers and word definitions. 
The answers were classified using knn, for both binary and multinomial classification. 
The results of knn classification are as follows: precision P = 0.73, recall R = 1.00, F1 = 0.85 for binary classification, and P = 0.50, R = 0.47, F1 = 0.46 score for the multino-mial classifier. 
The results are to be taken with a grain of salt, due to a small test and training dataset. 
KEYWORDS: lsa, clil, l2 vocabulary acquisition, cosine similarity, knn 
Employing technology to improve language learning outcomes is the problem scientists have wrestled with since the 1960s. 
In this paper, we will present a model for an automatic assessment of student answers using Latent Semantic Analysis (lsa). 
In further development, the model will be implemented within digital lexical flashcards (henceforth referred to as: flashcards) for learning vocabulary in English as a Second Language (esl) classes. 
Previous research (Landauer, Foltz, and Darrell 1998; Lemaire and Dessus 2003; Lifchitz, Jhean-Larose, and Denhière 2009) shows that many cognitive abilities in humans, including vocabulary acquisition, are well-represented by lsa. Furthermore, assessments provided by lsa largely correlated with those done by evaluators (Landauer et al. 1997; Graesser et al. 2000; Lemaire and Dessus 2003; Landauer, Laham, and Foltz 2003; Picca, Jaccard, and Eberlé 2015). 
Flashcards have so far proven to be a good tool for textscl2 vocabulary acquisition, combining interval (Ashcroft, Cvitkovic, and Praver 2018) and conscious learning (Nation 2006; Hung 2015) - two approaches that enhance learning outcomes, especially at the lower levels of language knowledge (Ashcroft, Cvitkovic, and Praver 2018). 
Taking everything previously said into account, we believe that developing this model will help us tackle several methodological problems, and contribute to digitalisation of l2 classroom at the Faculty of Mining and Geology, University of Belgrade. 
In accordance with the paper’s interdisciplinary approach, our aims are twofold - pedagogical and methodological. 
The former is to examine current general and geological vocabulary knowledge of the Faculty’s students and associates, as well as to improve present teaching methods by helping develop digital learning materials. 
The latter aim is to look into lsa’s application in assessing student answers in the geological domain, and with a form of a definition, and to see to which extent the model’s assessment correlates with the evaluators’. 
Conforming to the aforementioned aims, our hypotheses are as follows - (1) creation of the system will help digitalise learning materials; (2) lsa will be successful in assessing student answers. 
Knowing a word includes knowing its form, meaning and use (Nation 2006). 
The time that a student spends learning, along with the student’s involvement in the learning process, affects vocabulary learning outcomes. 
Spaced (or distributed) learning, i.e. learning in many small sessions increasing the breaks between each session, showed the best results when it comes to vocabulary learning (Nation 2006). 
In reference to the level of student involvement in the learning process, some authors presume that word form is learned implicitly as a result of frequent exposure to a word, while word meaning that is abstract and belongs to the domain of semantics, is acquired by explicit learning. 
Explicit learning uses an array of tools, such as dictionaries, word lists, and flashcards (Nation 2006; Hung 2015; Ma 2009). 
Several researches display that flashcards give the best results when it comes to l2 vocabulary acquisition outcomes, especially at the lower levels of language knowledge (Spiri 2008; Nakata 2008; Hung 2015; Averianova 2015; Yüksel, Mercanoğlu, and Yılmaz 2022). 
Flashcards provide simultaneous explicit and interval learning of vocabulary, together with learning word form, meaning and use in context (Ma 2009). 
Nowadays, students can also learn using digital flashcards, which have several advantages compared to their paper predecessors - user receives feedback on their spelling and grammar errors at once, the system can use pictures and sound and can be accessed from different devices. 
Moreover, students can now learn whenever and wherever making digital flashcards a great tool for spaced learning. 
Additionally, they can be used for both learning and tests, as well as gamification of the learning process. 
Today, the most commonly exploited systems are Anki and Quizlet (Ashcroft, Cvitkovic, and Praver 2018). 
Learning geological terminology in esl classes is explored in Beko, Obradović, and Stanković (2015). 
The paper highlights several issues, including difficulties students have in finding a suitable learning method, low level of language knowledge at the begging of studies, and lack of translation of geological terminology to Serbian, which makes translational tasks even more difficult (Beko, Obradović, and Stanković 2015). 
Given that our model will be monolingual, we will not address the last-mentioned issue. 
Faculty of Mining and Geology currently uses language tools such as a thesaurus of geological terminology in Serbian and English, comprised of roughly 2800 words (Beko, Obradović, and Stanković 2015), and a digital mining terminology platform RudOnto. 
Additionally, a system of flashcards rgf Flashcards was developed, using Anki. The flashcards were integrated into the Faculty’s Moodle platform.  
Given that our flashcards will be used in a clil classroom, that integrates learning content from a certain domain with language learning (Beko 2013; Ðerić 2019; Baten, Van Hiel, and De Cuypere 2020), whereby c1 entry language knowledge is expected, the flashcards can be used to facilitate vocabulary acquisition process for students with lower levels of English-language knowledge, consequently making following learning materials and classes easier. 
Latent Semantic Analysis lsa is a theory and method for extraction and representation of word meaning in context, whereby statistical calculations are applied to a large text corpus (Landauer et al. 1997). 
Thus far, research has shown that lsa can largely represent human cognitive abilities, such as vocabulary acquisition, word categorisation, semantic priming, discourse comprehension, and essay assessment (Landauer, Foltz, and Darrell 1998). 
When creating a lsa model, we start by computing a document-term matrix, thereby forming a semantic space comprised of all terms and all documents in our corpus (Deerwester et al. 1990; Landauer, Foltz, and Dar-rell 1998; Lemaire and Dessus 2003). 
The next step is applying a weighting function to each matrix cell, which assigns small weights to high-frequency terms, and high weights to terms that appear in some, but not all document (Deerwester et al. 1990; Martin and Berry 2013). 
After this, singular value decomposition (svd) is performed (Equation 1). 
This is an instrumental part of lsa, because it makes it possible to present word meaning relative to the context it appears in (Lemaire and Dessus 2003). 
If m and n are integers and M is an arbitrary matrix m × n, then decomposition of a matrix Mmn is the following: M=U×S×VT	(1) 
where: 
– M: is an orthogonal m × n matrix (document-term matrix), where m is a number of documents and n the number of terms; 
– U: is an orthogonal document-topic m × r matrix, where m is a number of documents and r the number of topic; 
– S: is a diagonal r×r matrix, in which all values but those on the diagonal are equal to 0. The diagonal values in S represent how much each topic explains variance in the data; 
– V : is an orthogonal n × r matrix (term-topic), where n is the number of documents and r is the number of topics. 
Multiplying the three svd matrices, we can approximately reconstruct the original matrix M. 
Reduced matrix’s Mk dimensionality ought to be optimal, and accurately represent relationships between elements in the original matrix M (Landauer, Foltz, and Darrell 1998). 
For checking the validity of the number of dimensions, an external validation criterion is used (Landauer, Foltz, and Darrell 1998). 
In this paper, we will use a cosine similarity measure between students’ and correct answers (Rahutomo, Kitasuka, and Aritsugi 2012). 
Lsa has hitherto been used for answer assessment, providing feedback, answering student questions, as well as assessing student essay accuracy and coherency, in several smart games. 
In the essay assessment task, it displayed a high degree of correlation with evaluator assessments (Landauer et al. 1997; Graesser et al. 2000; Lemaire and Dessus 2003; Landauer, Laham, and Foltz 2003; Dikli 2006; Lafourcade and Zampa 2009; Picca, Jaccard, and Eberlé 2015). 
Our data is constituted of three parts - 
(1) unit texts written for Prof. Dr. Lidija Beko’s English-language textbook in preparation, 12 units with 3 texts each; 
(2) vocabulary for each unit, split into three categories -	general vocabulary (663 words), geological vocabulary (280 words), and minerals (18 words); 
(3) student answers collected by tests via the Faculty’s Moodle platform, for subjects English 1–4. 
There were three groups of tests, for three groups of participants. 
All groups had the same questions with different examples and were formed using an example from Jhean-Larose et al. (2010). Some questions (e.g. question six) were adjusted to the research aims. 
The test description is to be found in Table 1. 
Table 1: Test used for answer collection 
As mentioned, tests were implemented in Moodle platform enhanced with hp5 extinction and shared with the participants. 
The test was completed by 14 participants, and 451 answers were collected. 
For anonymity purposes, we created a unique numerical id for each participant. 
The most answers were collected for the first group of the test, and the fewest for the third. 
Answer assessment by an evaluator was done in two steps. 
First, each answer was assessed on a scale from 1 to 5 (Table 2). 
Table 2: Evaluator grading scale 
Secondly, answers marked with 1 were labeled as incorrect (i), while the rest were labeled as correct (c). 
The assessment criterion was answer similarity with the golden standard -- word definition from the textbook, as well as the evaluator’s language competence. 
Since our model does not take into account grammar and spelling, neither did the evaluator when assessing student answers. 
Due to an inconsistent output, question 5 was not included in the analysis, while questions 3, 6, 7, and 8 were estimated fit for analysis using lsa. 
This selection left us with 238 answers. 
After removing missing values, 72 answers remained. 
Text preparation was conducted in accordance with methods found in the literature (Deerwester et al. 1990; Dikli 2006; Lifchitz, Jhean-Larose, and Denhière 2009), which we adapted to our goals and our data. 
The first step in text preparation was text lemmatisation using SpaCy library. After obtaining lemmatised text surrogates for each part of our data, we removed punctuation and special characters using regular expressions and changed text to lowercase. 
In addition, we removed Latin abbreviations and plurals from the vocabulary (e.g. data sing. datum, hypothesis pl. hypotheses). 
An example of text before and after preparation is displayed in Table 3. 
The examples are extracted from different texts. 
Table 3: Processed text 
We noticed inconsistencies when it comes to the lemmatisation of some verbs. 
For example, verb split was not lemmatised in an act of splitting into a category. 
However, going through other examples with similar structures (verb-adverb-verb or noun-verb-adverb), e.g. an action of taking something; the process of breaking something down we did not detect the same mistake. 
Furthermore, the verb unstratified was lemmatised to unstratifie when standing alone, while it remained in its original form in the answer incoherent loose unstratified. 
Latin words, such as antennae or Pinaceae, were not lemmatised at all. 
For developing our lsa model, Scikit-Learn Python library was used. 
First, we constructed a tf-idf matrix (Term Frequency-Inverse Document Frequency), with documents in matrix rows, terms in matrix columns, and relative term frequencies in each of the documents in matrix cells (Jurafsky and Martin 2023). 
When constructing a tf-idf matrix, it is necessary we decide how many terms are to describe each document. 
Trying out options between 700 and 5000 terms, we decided that the number of terms in tf-idf matrix for unit texts be 1000, that minimal term frequency would be 3, and that a term can appear in no more than 80% of the documents to be found in the matrix. 
In this step, we also removed stop words. Stop words list was comprised of the English language stop words from nltk library and stop words specific for our corpus (km, kmh, mm, meter, one, two, three, etc., yet, well. . . ). 
Initially, we applied the same parameters to the rest of our data, i.e. vocabulary and participant answers, but this gave poor results. Thus, we lowered the number of dimensions to 700 and minimal frequency to 1, and increased maximum frequency to 100% of the documents, while the stop word list contained only definite and indefinite article -- a/an, the. 
Svd parameters are the same for all parts of the data. 
In order to determine an appropriate number of topics, we extracted 15 terms with the highest weights for each topic and examined if there are any overlaps between topics, or if the model has missed something. 
Finally, we opted for 10 topics. 
Then, we assigned a name to each topic based on the first 100 terms with the highest weights. 
Some topics, such as Topic0, Topic1, Topic4, con-tained more general terms that are woven through most texts. 
On the other hand, topics Topic3, Topic5, and Topic7 contain terms from different geology branches, like tectonic plates, volcanology, and erosion (Table 4). 
Table 4: Text topics and terms with highest weights for each topic 
After obtaining topic vectors for all parts of the data, we measured cosine similarities between all texts so as to get the most similar ones. 
Next, we calculated a final score for each answer as a mean of cosine similarity of answer a and: a) vector of the text in which the word defined in the answer is used; b) vector of the correct answer (golden standard); c) vector of the most similar answer b to the target answer a. 
The higher the similarity score of document a and document b, the higher the connection between the documents (Rahutomo, Kitasuka, and Aritsugi 2012). 
Finally, answers were classified into two and more categories according to accuracy and based on the obtained answer score. For classification purposes, both binominal (Correct / Incorrect) and multinomial (criteria displayed in Table 2), knn (K-Nearest Neighbour) algorithm was employed (Li, Yu, and Lu 2003; Peterson 2009). 
Prior to proceeding to answer classification, let us take a look at how the data is distributed among topics. This will provide us with insight into our model’s validity. 
We observed the most uneven topic distribution in texts, while it was somewhat more uniform, although still irregular, in vocabulary and answers. 
We believe that the reason behind lower standard deviation (std) (Urdan 2005) in vocabulary and answers is a more coherent text form, compared to unit texts. 
In unit texts, maximal topic values vary between 0.5617 in Volcanology, to 0.3638 in Dating, while minimal values fluctuate from 0.3878 for Earth-Formation, all the way to -0.001 for topic Dating, and std in topics is high. 
Maximal values in definitions are, to a degree, more evenly distributed. 
Topic Weathering (0.7067) has the highest maximum value, while the lowers is that of Landslides (0.3547). 
Almost all minimal topic values are negative, apart from the topic EarthFormation, with a minimal value of 0.0193. 
While topics are assigned well to some geological terms, e.g. debris has high values in EarhFormation, Weathering and Landslides, the model failed to recognise latent topics in others, which is shown for example in a low value of topic Fossils in definitions of terms fossil,10 fossilised, fossilisation. This means that, according to our model, topic of Fossils does not describe the afore  mentioned terms in a high degree. 
The distribution of general vocabulary among topics was harder to evaluate since topics pertain to geology. 
The phrasal verb wear away has high values for topics EarthFormation, Erosion and Weathering, which aligns with its meaning. 
However, words such as adverbs therefore, yet, anyway etc. were probably the most difficult ones to distribute since they appear in most topics. 
In participant answers, we find topic TectonicPlates has the highest max-imal value (0.7042), while the lowest one is that of Landslides, with just 0.3612. 
Minimal values are for the most part negative, and have values between -0.5557 for Minerals and 0.0000 for EarthFormation. 
Answers to the same question mainly have similar topic distribution. Most answers to the question global warming have the highest values for the topic EarthFormatoin, and the lowest for Erosion and Landslides. 
All topic values for short, incomplete answers are equal to 0. 
We will begin the result presentation by displaying dominant topics in texts, definitions, and participants’ answers. 
In the following section, we will show the way in which we calculated answer accuracy, and analyse the results. 
Finally, we will evaluate the results of bi- and multinomial knn classification algorithms. 
Three dominant topics were extracted for each sample in each part of the data. 
In unit texts, EarthFormation is the most frequent as the first dominant topic, appearing in 21 documents, then in 11 as the second dominant topic, and in 2 documents as the third dominant topic (Figure 1). 
Dating is the only topic not found as the first dominant, but it comes second by frequency as the second dominant one, along with topic Minerals. 
All topics appear as the second dominant topic, Landslides coming up just once in unit text Mass Wasting and Types of Landslides, while Erosion is the only topic not appearing in the place of a third dominant topic (Figure 1). 
Figure 1: Dominant topic frequency in unit texts. 
Analysing values of dominant topics in unit texts, we can see that the first dominant topic values are the least scattered, ranging from approx. 0.3 to 0.6. 
The second and third dominant topic values are more scattered and somewhat lower, stretching from a bit below 0.1 to about 0.4 for the second, and from negative values up to 0.4 for the third dominant topic (Figure 2). 
For the most part, the model did well in detecting dominant topics in unit texts. Nonetheless, in the text Mineral Evolution, Minerals is not to be found among the dominant topics, but it appears in the following text of the same unit, Physical Properties, that discusses the origin and physical properties of minerals. 
Figure 2: Dominant topic values in unit texts. 
Based on the results, we can conclude that there are large differences in values of the first and second dominant topics of a document. 
In spite of that, the second and third dominant topics often more closely specify what a text is about. Therefore, all three topics should be taken into consideration when analysing the results. 
EarthFormation is also the most frequent first dominant topic in vocabulary, and most frequent overall, the same as in texts. 
After that, comes TectonicPlace, in close frequency with Fossils, Volcanology and Landslides, while Minerals is the least frequent first dominant topic (Figure 3). 
Figure 3: Dominant topics in vocabulary. 
In a definition of the word glaciation, topics Weathering, EarthFormation and Volcanology are found as dominant. 
On the other hand, earthflow is primarily placed in topics Fossils, EarthFormation, and Tectonic-Plates, even though, intuitively, we would perhaps list Erosion, Landslides and Weathering. 
In Figure 4 can see that values pertaining to the first dominant topic are more scattered than in unit texts, fluctuating between over 0.7 and be-low 0.1. 
Second dominant topic values are slightly lower than those of the first dominant topic, but equally irregularly distributed, while values of the third dominant topic in vocabulary are predominantly low – below 0.4, and relatively homogeneous. 
Figure 4: Dominant topics values in vocabulary. 
Lastly, EarthFlow scores first in frequency in all three places in the participants’ answers, emerging 31 times as the first, 25 times as second, and 18 times as the third dominant topic. 
Landslides is second most frequent as the first dominant topic, having the largest values in 10 answers, while Erosion is second as the second dominant topic with 12 answers. 
All topics are of similar frequency as the third dominant topic, apart from Landslides, which appears only 2 times (Figure 5). 
Figure 5: Dominant topics in participant answers. 
Generally, the same topics are assigned to long answers to the same question. 
For example, all answers defining global warming have EarthFormation, Volcanology and either Minerals or Dating as dominant topics. 
In the participants’ definitions of a notion of sedimentary rock, EarthFormation and Landslides take turns as the first two dominant topics, while Fossils, Dating and Erosion rotate at the third place. 
Zero values are explained by the unsorted answers (Figure 6). 
Figure 6: Dominant topic values in participant answers. 
After going through topic distribution, values, and dominant topics in all parts of the data, we compared extracted dominant topics of the same notions in answers, vocabulary, and relevant unit texts (Table 5). 
Table 5: Dominant topics for the term transpire 
Table 6: Dominant topics for the term global warming 
For example, extracted dominant topics for answer, definition, and unit text for the term transpire do not align, and their values differ in all parts of the data (Table 5). 
On the contrary, we detected a high degree of similarity between topics and their values for the term global warming (Table 6). 
We believe that the reason for this is that we have several long answers for the term global warming, all containing terms with big weights in topics, which made answer distribution easier for the model. 
To measure the similarity of unit texts in the textbook, we applied cosine similarity measure to topic vectors of all texts, retrieved after applying svd to a previously constructed tf-idf matrix. 
Based on the cosine similarity results, we can see how well our lsa model recognised latent topics in unit texts. 
Analysing the results, the supposition is that latent topics in texts are well-detected and that the most similar texts indeed convey similar topics, 
so a text about Wagner’s hypothesis that explains an assumption of the existence of Pangaea, has the highest similarity with a text about tectonic plates. Furthermore, a text about volcanology is closely matched to a text about igneous rocks (Table 7). 
Table 7: Examples of the most similar texts 
In this section of the paper, we will present results of cosine similarity measured between a vector of answer a and: (1) vector of a unit text containing the target word; (2) vector of the correct answer; (3) vector of the most similar answer b. 
In order to see if there are any differences between scores of correct and incorrect answers, we will take a look at the distribution of answer score values among two and more grading categories obtained by the evaluator’s answer assessment. 
When comparing answers and texts, we measured the cosine similarity of an answer and each of the three texts of a unit in which the defined notion is explained and then computed a mean of the three values so as to obtain the similarity between an answer and the entire unit. 
The results of this computation should be taken with a grain of salt because text vectors assuredly differ from unit vectors. 
We noticed greater similarities between units and long answers, especially the ones referring to geological terms, such as global warming, and sedimentary rock. 
Contrastingly, we did not obtain steady results for an answer to the definition of hypothesis, where we have a similarity of 0 for one answer, and a rather high similarity for another answer (Table 8). 
Table 8: Cosine similarity results - Examples for answer-text similarity; qid – question id, pid – participant id. 
As displayed in Figure 7, measured cosine similarity values between an-swers and units are somewhat higher in correct than in incorrect answers. 
In the five-category distribution, we have the greatest variance in categories 3 and 5. 
The lowest values are detected in category 1. 
Outliers are answers to questions chemical weathering and convergent (Figure 7). 
Figure 7: Answer-text similarity value distribution. 
The completeness and correctness of an answer are best represented by the cosine similarity of an answer and the correct answer. 
Each of the answers was compared to only one correct answer, so not only are we comparing similarities of two short texts, but also gain a direct insight into the lsa’s comprehension of short text and definitions. 
For example, a participant with id 4 defined the word convergent as a mathematical term, which is not incorrect, but it is not the meaning we were looking for, so the answer’s similarity to the correct answer is rather low (Table 9). 
Furthermore, one of the definitions of straightforward is an incomplete answer that has low similarity to the correct answer as well. 
On the other hand, some correct answers have negative cosine similarity values to the correct answer, such as petrologist20 (Table 9). 
Answer length might explain this kind of result. 
Table 9: Cosine similarity results - Examples for similarity of an answer and the correct answer; qid – question id, pid – participant id. 
Additionally, high weights of functional words (be, of, in, to, or, by, etc.) might have contributed to the results. 
In further research, we could solve this by removing functional words with high weights from answers and definitions, and see if the results improve. 
As we can see, the range of correct answers’ values is greater than that of incorrect. 
The reason behind low values of correct answers is probably short answers evaluated as correct (Figure 8). 
In the five-category evaluation, completely correct answers (grade 5) have the greatest value range. 
Low values are mostly answers where a participant used synonymous words in their definition with respect to the correct answer (Figure 8). 
Figure 8: Answer-definition similarity value distribution. 
The third measured value is a cosine similarity of all answers, obtained in the same manner as the similarity of all texts in the textbook. 
Unlike texts, the model fell short in detecting most similar answers. 
Similarity values spread from about 0.7 to 0.9. 
If the answer vector is 0, the first answer in the data was computed as the most similar one, with a similarity of 0 (Table 10). 
Answers that do not share the same terms are evaluated as most similar but also answers to the same question that do share many terms, as well as answers to different questions that share the same terms, such as answers to questions hydrological cycle and seabed, both containing terms earth, ocean, surface (Table 10). 
Table 10: Most similar answers. 
Since we only took the highest similarity score for each of the answers, the value distribution of correct and incorrect answers is nearly equal, while in the five categories, answers with grade 2 have the most scattered values (Figure 9). 
Figure 9: Answer similarity value distribution. 
Finally, we calculated a final score for each answer, as a mean of the three aforementioned values. 
The lowest value in the final score is 0, which is the score of previously explained short answers, while long answers show little variance between the three values. 
In two-category distribution, the final score has lower values in correct than in incorrect values, and values of correct answers have a greater range. 
Answer similarity in all probability contributed to high values of incorrect answers (Figure 10). 
In five categories, we can see that values of answers graded 2 are most scattered, while the densest ones are those of answers with grade 1, and higher grades have relatively similar final scores (Figure 10). 
Figure 10: Final score value distribution. 
Considering the displayed results, we can conclude that lsa did well in detecting topics in unit texts. 
Yet, when it comes to vocabulary and answers, results are not as great, especially concerning detecting latent topics in very short answers, and in answers pertaining to words that do not fall under the domain of geology. 
Also, there is not much difference between cosine similarities of correct and incorrect answers, which leads up to questioning our answer assessment methodology. 
The final step was answer classification - binary and multinomial. 
For classification purposes knn algorithm was employed (Li, Yu, and Lu 2003; Peterson 2009; Chen 2018), labels corresponded with the evaluator assessment, while the classification criteria were the final answer score. 
Recall, precision and F1 score were used to evaluate the knn models (Géron 2022). 
In binary classification, answers were classified as correct or incorrect. 
The data is comprised of 60 correct and 12 incorrect answers. 
Due to this discrepancy, the model classified all answers as correct (Table 11a)). 
Calculated model precision was 73%, recall 100%, and F1 = 0.85. 
Given that our data set is rather small, containing only 72 observations, consequently so is the test set with mere 15 observations, the presented results do not reflect actual model validity. 
Since the data is randomly split into a training and test set, it can just so happen that all the observations in the test set have the same label. 
The same goes for multinomial classification - category frequency is uneven, where incorrect and completely correct answers (grades 1 and 5) are least frequent, thus, we expect that this will affect the results of the classi-fication model. 
Grades 3, 4 and 5 are equally represented in the data, with 17, 21 and 22 answers, respectively. 
As displayed in the confusion matrix, answers graded with 1 were the hardest for the model to classify. 
Seven out of six answers with grade 5 ended up in the test set, and our model classified 3 of them correctly (Table 11b). 
Table 11: confusion matrices 
Estimated model precision was 50%, recall 47%, and F1 = 0.46, which is significantly worse compared to the results of binary classification. 
We believe that this is a result of a small data set, uneven label representation in the training and test set, along with similar value distribution of final answer scores in different grade categories. 
In this paper, we discussed the application of latent semantic analysis for the assessment of short answers. 
In accordance with the set pedagogical goals of this paper, we extrapolated that the utilisation of flashcards for l2 vocabulary acquisition gives favourable results, particularly at the lower levels of language knowledge. 
As students of the Faculty of Mining and Geology come from different educational backgrounds and usually enter their studies with a low level of English, we strongly believe that using a system of flash-cards that accompany the subject textbook would greatly help students to make progress faster and get to a level of vocabulary knowledge suitable for following clil lectures. 
Reflecting on the methodological aims of the paper, we determined that developing this model helped us recognise the advantages and disadvantages of our approach. 
One of the greatest advantages of the model is good topic modeling of longer texts and vocabulary and answers pertaining to geology. 
We deem that the biggest downside is its inability to detect topics of very short answers. Supposedly, we could overcome this by expanding the data set. 
In further research, we will aim to expand our data set. 
Our second goal is to add a system for spelling and grammar assessment. 
In order to improve the results obtained using lsa, we will lower the number of dimensions and try and see if the model improves. Additionally, creating separate semantic spaces for words that are not geological notions, i.e. general vocabulary, might be a good idea. 
When comparing answers and unit texts, we believe that we would get more meaningful results if we extract just a fragment of the text where a certain geological notion is explained or a word belonging to the general vocabulary used. 
Lastly, instead of computing the similarity of all answers, we would proceed to calculate the similarity of answers to the same question. 
The presented model development laid a foundation for the development of a system for automatic answer assessment in digital flashcards. 
Comparing the goals and aims of clil methodology and the outcomes of using flashcards in teaching, we concluded that this technology would greatly complement the textbook in preparation, whose author is Prof. Dr Lidija Beko. 
Our claim is supported by the Faculty’s students’ positive attitude towards using digital flashcards in an l2 classroom expressed in previous research. 
In further research, we will aim to accomplish the project’s main goal --the development of a digital flashcard system that will be implemented in the classroom. 
As this paper is a result of my Master’s thesis written for an interdisciplinary master’s program Social Sciences and Computing at the University of Belgrade, I would hereby like to thank my mentor, Prof. Dr. Ranka Stanović, for her patience, guidance, knowledge, and infinite support. 
Also, I would like to express my gratitude to Prof. Dr. Lidija Beko, for inviting me to use her textbook in preparation as the material for my thesis. The textbook will be used for courses English 1–4 at the Faculty of Mining and Geology, University of Belgrade. 
Finally, I want to thank Prof. Dr. Jelena Jovanović, who was a member of the committee for my thesis, because her questions contributed to improvements applied in this version of the paper. 
