Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability

S Roy - arXiv preprint arXiv:2106.02016, 2021

… A 4-gram language model trained on common crawl dataset is used for decoding 6 7[28] … 4https://github.com/pytorch/fairseq/tree/master/examples/ wav2vec 5https://github.com/facebookresearch/wav2letter/tree/master/recipes …


Semantically Meaningful Sentence Embeddings

Text embedding is an approach used in Natural Language Processing (NLP) to represent words, phrases, sentences, and documents. It is the process of obtaining numeric representations of text to feed into machine learning models as vectors (arrays …


SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation

S Ren, L Zhou, S Liu, F Wei, M Zhou, S Ma

… Table 1: The datasets used in our experiments. Lang: language; Mono: monolingual; Para: parallel; #Sent: number of sentences in the monolingual corpus; NC: NewsCrawl; CC: CommonCrawl; BE: BigEst Estonian corpus; Wiki: Wiki dumps …


Semi-automatic Construction of Sight Words Dictionary for Filipino Text Readability

JM Imperial - Knowledge Management and Acquisition for Intelligent …

… The Filipino word embedding model has been pre-trained from the articles of Wikipedia and the publicly-available webscraped data of the Common Crawl project using the Continuous Bag-of-Words (CBOW) model [2]. Word …


Semi-supervised log-based anomaly detection via probabilistic label estimation

L Yang, J Chen, Z Wang, W Wang, J Jiang, X Dong… - 2021 IEEE/ACM 43rd …, 2021

… Then, PLELog adopts the pre-trained word vectors based on Common Crawl Corpus using the FastText algorithm [46] (which can effectively capture the intrinsic relationship among words in natural language), to extract the …


Sentence encoding for Dialogue Act classification

N Duran, S Battle, J Smith - Natural Language Engineering, 2021

In this study, we investigate the process of generating single-sentence representations for the purpose of Dialogue Act (DA) classification, including several aspects of text pre-processing and input representation which are often overlooked …


Sentence simplification with ruGPT3

… One of the latest works [12] uses an unsupervised approach to automatically create training corpora for simplification in multiple languages from raw Common Crawl web data and train simplification systems in any language with a controllable generation mechanism …


Sentiment Analysis of Code-Mixed Roman Urdu-English Social Media Text using Deep Learning Approaches

A Younas, R Nasim, S Ali, G Wang, F Qi - 2020 IEEE 23rd International Conference on …

… Similarly, authors have developed XLM-RoBERTa (XLM-R) - a multilingual language model trained on 2.5 terabyte of filtered CommonCrawl data using BERT-large architecture [9]. It provides strong advantage over mBERT model …


Sentiment analysis of Korean movie reviews using XLM-R

NR Shin, TH Kim, DY Yun, SJ Moon, C Hwang - International Journal of Advanced …, 2021

… Among them, the Facebook AI team unveiled the XLM-R (XLM-RoBERTa), an upgraded XLM model. XLM-R solved the data limitation and the curse of multilinguality by training XLM with 2TB or more refined CC (CommonCrawl), not Wikipedia data …


Sentiment Analysis of Latin Poetry: First Experiments on the Odes of Horace

R Sprugnoli, F Mambrini, M Passarotti, G Moretti - Eighth Italian Conference on …, 2021

In this paper we present a set of annotated data and the results of a number of unsupervised experiments for the analysis of sentiment in Latin poetry. More specifically, we describe a small gold standard made of eight poems by Horace, in …


Sentiment Analysis of Sinhala News Comments

S Ranathunga, IU Liyanage - Transactions on Asian and Low-Resource Language …, 2021

… 2009], and the Sinhala Common Crawl dataset1 are examples of un-annotated Sinhala corpora. Common crawl is the largest among these, with about 110 million uncleaned tokens … Both SinMin and Common Crawl contain a lot of noise …


K Ahmad, F Alam, J Qadir, B Qolomany, I Khan, T Khan… - arXiv preprint arXiv …, 2021

Page 1. 1 Sentiment Analysis of Users' Reviews on COVID-19 Contact Tracing Apps with a Benchmark Dataset Kashif Ahmad∗, Firoj Alam†, Junaid Qadir‡, Basheer Qolomany§, Imran Khan¶, Talhat Khan¶, Muhammad Suleman …


Sentiment Analysis Using XLM-R Transformer and Zero-shot Transfer Learning on Resource-poor Indian Language

A Kumar, VHC Albuquerque - Transactions on Asian and Low-Resource Language …, 2021

… It is a scaled cross-lingual sentence encoder trained on 2.5 Tb of data across 100 languages data filtered from CommonCrawl texts. Compared to the original version, the biggest update of XLM-Roberta is a significant …


Sentiment Polarity Classification of Corporate Review Data with a Bidirectional Long-Short Term Memory (biLSTM) Neural Network Architecture

… Therefore, it was decided to proceed with fastText embeddings. Accordingly, pre-trained vectors for Dutch language were added to the biLSTM model as one of the layers. Moreover, these vectors were trained on Common Crawl and Wikipedia using fastText …


Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning

S Lee, HB Lee, J Lee, SJ Hwang - arXiv preprint arXiv:2110.02600, 2021

Page 1. Preprint SEQUENTIAL REPTILE: INTER-TASK GRADIENT ALIGNMENT FOR MULTILINGUAL LEARNING Seanie Lee1∗, Hae Beom Lee1∗, Juho Lee1,2, Sung Ju Hwang1,2 KAIST1, AITRICS2, South Korea …


SG-MLP: Switch Gated Multi-Layer Perceptron Model for Natural Language Understanding

G Son, S Kim, SJ Joo, W Cho, JE Nah - … of the Korea Information Processing Society …, 2021

… SG-MLP Small 은 약 16GB 에 해당하는 Book Corpus 와 Wiki 데이터로 학습 되었으며 이외의 모델은 Common Crawl’s Web Crawl Corpus 를 한 차례 정제한 Allen·C4·En[8] 데이터를 학습에 활용하였으며 이는 약 305GB 정도의 분량이다. 보유한 하드웨어 …


Shall I Work with Them? A Knowledge Graph-Based Approach for Predicting Future Research Collaborations

N Kanakaris, N Giarelis, I Siachos, N Karacapilidis - Entropy, 2021

We consider the prediction of future research collaborations as a link prediction problem applied on a scientific knowledge graph. To the best of our knowledge, this is the first work on the prediction of future research collaborations that …


Shallow Discourse Parsing for German

Page 1. Shallow Discourse Parsing for German Peter Bourgonje Dissertation eingereicht bei der Humanwissenschaftlichen Fakultät der Universität Potsdam 2021 Page 2. Datum der Einreichung: 22-07-2020 Wissenschaftlicher Betreuer und Gutachter: Prof …


H Bao, Y Nagano, K Nozawa - arXiv preprint arXiv:2110.02501, 2021

Page 1. Sharp Learning Bounds for Contrastive Unsupervised Representation Learning *Han Bao, Yoshihiro Nagano, Kento Nozawa The University of Tokyo & RIKEN AIP {tsutsumi@ms., nagano@, nozawa@ms.}ku-tokyo.ac.jp October 7, 2021 …


Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification

C Park, S Eo, H Moon, HS Lim - Proceedings of the 2021 Conference of the North …, 2021

… corpus. In mBART (Liu et al., 2020), the CC25 corpus was composed of a total of 25 languages extracted from CommonCrawl (CC) (Lample and Conneau, 2019; Wenzek et al., 2019) and used for unified vocabulary extraction …


Show, tell and summarise: learning to generate and summarise radiology findings from medical images

S Singh, S Karimi, K Ho-Shon, L Hamey - Neural Computing and Applications, 2021

Radiology plays a vital role in health care by viewing the human body for diagnosis, monitoring, and treatment of medical problems. In radiology practice,.


SJ_AJ@ DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

SM Jayanthi, A Gupta - arXiv preprint arXiv:2102.01051, 2021

… mon Crawl data6. The common crawl data consists of textual data both in native scripts as well as in romanized script for some of the languages. Among the three languages in this task, only Tamil has Romanized data in the Common Crawl …


Sketch-Based Creativity Support Tools Using Deep Learning

F Huang, E Schoop, D Ha, J Nichols, J Canny - Artificial Intelligence for Human …, 2021

Abstract Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines’ ability in understanding and generating …


SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training

A Bapna, Y Chung, N Wu, A Gulati, Y Jia, JH Clark… - arXiv preprint arXiv …, 2021

Unsupervised pre-training is now the predominant approach for both text and speech understanding. Self-attention models pre-trained on large amounts of unannotated data have been hugely successful when fine-tuned on downstream …


J Piskorski, B Babych, Z Kancheva, O Kanishcheva… - The 8th Workshop onBalto …, 2021

Page 1. Slav-NER: the 3rd Cross-lingual Challenge on Recognition, Normalization, Classification, and Linking of Named Entities across Slavic languages Jakub Piskorski,1 Bogdan Babych,2 Zara Kancheva,3 Olga Kanishcheva …


SlovakBERT: Slovak Masked Language Model

M Pikuliak, Š Grivalský, M Konôpka, M Blšták… - arXiv preprint arXiv …, 2021

… is a suite of RoBERTa-style LMs. The models support 100 languages, including Slovak. Training data are based on CommonCrawl Web-crawled corpus. Slovak part has 23.2 GB (3.5B tokens). The XLM-R models differ in their …


Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-Resource Languages

… We select these languages because they are the languages supported by the British Broadcasting Corporation (BBC) News, which was our main source of data.4 We also obtain additional data from the Common Crawl …


Smart integration of sensors, computer vision and knowledge representation for intelligent monitoring and verbal human-computer interaction

T Mavropoulos, S Symeonidis, A Tsanousa… - Journal of Intelligent …, 2021

The details presented in this article revolve around a sophisticated monitoring framework equipped with knowledge representation and computer vision capabi.


SOAR: A Synthesis Approach for Data Science API Refactoring

A Ni, D Ramos, A Yang, I Lynce, V Manquinho… - arXiv preprint arXiv …, 2021

… is greatly reduced. However, GloVe is trained with Common Crawl [30] which contains raw webpages, which is a mismatch from our domain of textual data (ie, data science and programming). API matching. Given the representation …


Social network analysis using deep learning: applications and schemes

AM Abbas - Social Network Analysis and Mining, 2021

Abstract Online social networks (OSNs) are part of daily life of human beings. Millions of users are connected through online social networks. Due to very large number of users and huge amount of data, social network analysis is a challenging …


Social Networks Analysis to Retrieve Critical Comments on Online Platforms

S Bhandari, R Raju - arXiv preprint arXiv:2102.10495, 2021

… 2019 IEEE International Conference on Healthcare Informatics (ICHI), pp. 1–3, 2019. [13] S. Nagel, “Cc-news.” https://commoncrawl.org/2016/10/news-datasetavailable/, 2016. [14] M. Heidari and JH Jones, “Using bert to extract …


Social Norm Bias: Residual Harms of Fairness-Aware Algorithms

M Cheng, M De-Arteaga, L Mackey, AT Kalai - arXiv preprint arXiv:2108.11056, 2021

… We use the dataset1 and task described by De-Arteaga et al. [25]. The dataset, containing 397,340 biographies spanning twenty-eight occupations, is obtained by using regular expressions to filter the Common Crawl for online biographies …


SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification

S Rosenthal, P Atanasova, G Karadzhov, M Zampieri… - Proceedings of the Findings …, 2021

… relations between words. First is an embedding layer initialized with a concatenation of the GloVe 300-dimensional (Pennington et al., 2014) and FastText's Common Crawl 300-dimensional embeddings (Grave et al., 2018). It is …


Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language Models

CP Carrino, J Armengol-Estapé, OG Bonet… - arXiv preprint arXiv …, 2021

… The CommonCrawl1 is very a large repository of crawled websites. However, it needs preprocessing to extract the text relevant to the user. The OSCAR corpus [10] was built applying language identification to CommonCrawl …


Sparse Distillation: Speeding Up Text Classification by Using Bigger Models

Q Ye, M Khabsa, M Lewis, S Wang, X Ren, A Jaech - arXiv preprint arXiv:2110.08536, 2021

Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time. However, the improved inference speed may be still unsatisfactory for certain time-sensitive applications. In …


Spreadsheet Data Transformation for Ontology Engineering in Petrochemical Equipment Inspection Tasks

NO Dorodnykh, AY Yurin - … Conference on Intelligent Information Technologies for …, 2021

… world. The exploration of the Web crawl (“Google Crawl”, “Common Crawl”, “ClueWeb”) discovered hundreds of millions of tables containing relational data [3]. Presumably, billions of valuable facts can be extracted from them …


SQYQP@ Vaxxstance: Stance Detection for the Antivaxxers Movement

… embeddings, Multilingual cased BERT embeddings are employed, which are trained on cased text in the top 104 languages with the largest Wikipedias, as well as XML-RoBERTa embeddings, which are trained on 2.5 TB of newly …


Stacked Embeddings and Multiple Fine-Tuned XLM-RoBERTa Models for Enhanced Hostility Identification

XLM Fine-Tuned - Combating Online Hostile Posts in Regional …

… XLM-RoBERTa. XLM-RoBERTa [10] is a large multilingual model trained on the CommonCrawl Dataset. There are two versions: base and large; both have around 250k words in the vocabulary, and the base has 250M parameters, while large has 560M …


StarryThoughts: Facilitating Diverse Opinion Exploration on Social Issues

H Kim, H Kim, KJ Jo, J Kim - Proceedings of the ACM on Human-Computer …, 2021

… After translation, we computed each opinion's embedded vector with the algorithm using pre-trained GloVe word embeddings built upon 42B tokens from Common Crawl [49]. 4.3 Implementation details The front-end of StarryThoughts is implemented with React …


Step-unrolled Denoising Autoencoders for Text Generation

N Savinov, J Chung, M Binkowski, E Elsen, A Oord - arXiv preprint arXiv:2112.06749, 2021

… results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. … • We demonstrate good qualitative results for unconditional generation and inpainting on Colossal Clean …


Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?

R Choenni, E Shutova, R van Rooij - arXiv preprint arXiv:2109.10052, 2021

09/21/21 - In this paper, we investigate what types of stereotypical information are captured by pretrained language models. We present the f...


Stock Volume Prediction Based on Polarity of Tweets, News, and Historical Data Using Deep Learning

N Jawahar, J Chelladurai, I Sakthivel, B Bajracharya - 2020 2nd International …, 2020

… token in order to predict the entity of that token. The CNN core model is pre-trained with GloVe vectors on Common Crawl, with 86.43% precision and 86.37% recall for NER. A python script is written that uses Psycopg to extract …


Storytelling Exhibitions: Identity, Truth and Wonder


S Farhadkhani, R Guerraoui, LN Hoang - arXiv preprint arXiv:2106.02398, 2021

Page 1. arXiv:2106.02398v1 [cs.LG] 4 Jun 2021 Strategyproof Learning: Building Trustworthy User-Generated Datasets Sadegh Farhadkhani IC School, EPFL Lausanne, Switzerland sadegh.farhadkhani@epfl.ch Rachid Guerraoui …


Streaming cascade-based speech translation leveraged by a direct segmentation model

J Iranzo-Sánchez, J Jorge, P Baquero-Arnal… - Neural Networks, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Streamlining the Identification of Emerging Tasks in the O* NET System Using Natural Language Processing (NLP): Technical Summary

… In our case, we used embeddings from the Global Vectors for Word Representation (GloVe; Pennington, Socher, & Manning, 2014) algorithm trained on a “common crawl” of the internet consisting of 42 billion tokens (ie, wordlike pieces of text) …


Structured Text Generation for Spanish Freestyle Battles using Neural Networks

P Dal Bianco, I Mindlin, L Lanzarini, F Ronchetti… - 2021 XLVII Latin American …, 2021

… It was trained on a version of the Common Crawl database [6], which consists of around a trillion words. GPT-3 achieves very good results for text generation, via training first on large databases such as the one mentioned and then performing fine-tuning …


Study of automatic text summarization approaches in different languages

Y Kumar, K Kaur, S Kaur - Artificial Intelligence Review, 2021

Nowadays we see huge amount of information is available on both, online and offline sources. For single topic we see hundreds of articles are available, co.


Studying the Relationship between Social Position and the Meaning of Work using Siamese BERT Networks and Generalized K-Means Clustering

… 1https://www.glassdoor.com 2Specifically, I used the 300-dimensional embeddings with a 1.9 million-token vocabulary that was pretrained on Common Crawl data. 2 Page 3. Review Occupation Flexible working hours …


Style Change Detection on Real-World Data using LSTM-powered Attribution Algorithm

… lab [18]. They were trained on Common Crawl and Wikipedia data, using continuous bag of words with position-weights, in dimension 300, with character n-grams of length five, a window of size five and ten negatives. Unlike …


SU-NLP at CheckThat! 2021: Check-Worthiness of Turkish Tweets

… Loodos_ALBERT5: pre-trained on the same dataset as Loodos_BERT. • mBERT6: multilingual BERT model pre-trained on Wikipedia in 104 languages. • XLM-RoBERTa 7: this cross-lingual model was pretrained on the CommonCrawl corpus in 100 languages …


A Martinez, K Sudoh, Y Matsumoto - Journal of Natural Language Processing, 2021

Page 1. IA0248_04martinez (2021-03-05 13:42) General Paper Sub-Subword N-Gram Features for Subword-Level Neural Machine Translation Ander Martinez † , Katsuhito Sudoh † and Yuji Matsumoto †,†† Neural machine …


Subword Pooling Makes a Difference

J Ács, Á Kádár, A Kornai - arXiv preprint arXiv:2102.10864, 2021

… We pick these two since they are architecturally similar (both have 12 layers and the same hidden size) making our comparison easier. mBERT was trained on Wikipedia while XLM-RoBERTa was trained on CommonCrawl (Wenzek et al., 2020) …


Subwords-Only Alternatives to fastText for Morphologically Rich Languages

T Ghukasyan, Y Yeshilbashyan, K Avetisyan - Programming and Computer Software, 2021

… Furthermore, as a result of embedding whole words, fastText models typically weigh a few gigabytes (Facebook's Common Crawl vectors' .bin and .vec files weigh 7.3 GB and 4.5 GB respectively [8]). This especially becomes …


Surprise Language Challenge: Developing a Neural Machine Translation System between Pashto and English in Two Months

A Birch, B Haddow, AVM Barone, J Helcl, J Waldendorf…

… Finally, we also use the Pashto-English corpus that was submitted by the Bytedance team to the WMT 2020 cleaning shared task (Koehn et al., 2020). For pretraining the German–English model we use exisiting WMT data …


Surveillance of COVID-19 Pandemic using Social Media: A Reddit Study in North Carolina

C Whitfield, Y Liu, M Anwar - arXiv preprint arXiv:2106.04515, 2021

… The custom NER models trained for this study were based on spaCy's multi-task convolutional neural network (CNN) which was trained using the OntoNotes 6 corpus, and contain GloVe vectors [31] that were trained on Common Crawl …


Survey of Generative Methods for Social Media Analysis

S Matwin, A Milios, P Prałat, A Soares, F Théberge - arXiv preprint arXiv:2112.07041, 2021

This survey draws a broad-stroke, panoramic picture of the State of the Art (SoTA) of the research in generative methods for the analysis of social media data. It fills a void, as the existing survey articles are either much narrower in their scope or are dated …


SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German

P Dogan-Schönberger, J Mäder, T Hofmann - arXiv preprint arXiv:2103.11401, 2021

… Page 2. 2. Related Work For various high-resource languages, there are parallel text corpora [2, 3, 4] and monolingual text corpora such as [5], Google News, Common Crawl that are used for the tasks of language modeling and machine translation …


Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

W Fedus, B Zoph, N Shazeer - arXiv preprint arXiv:2101.03961, 2021

Page 1. SWITCH TRANSFORMERS: SCALING TO TRILLION PARAMETER MODELS WITH SIMPLE AND EFFICIENT SPARSITY William Fedus∗ Google Brain liamfedus@google.com Barret Zoph∗ Google Brain barretzoph@google.com …


Page 1. SYMBOLIC AND NEURAL APPROACHES TO NATURAL LANGUAGE INFERENCE Hai Hu Submitted to the faculty of the University Graduate School in partial fulfillment of the requirements for the degree …


Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

P West, C Bhagavatula, J Hessel, JD Hwang, L Jiang… - arXiv preprint arXiv …, 2021

The common practice for training commonsense models has gone from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from-machine-to-corpus-to-machine …


Syntax-Enhanced Pre-trained Model

Z Xu, D Guo, D Tang, Q Su, L Shou, M Gong, W Zhong… - arXiv preprint arXiv …, 2020

… pre-training and fine-tuning stages. We construct a pre-training dataset by applying an offthe-shell dependency parser (Qi et al., 2020) to one billion sentences from common crawl news. With these data, we introduce a syntax …


Systemic racism: individuals and interactions, institutions and society

MR Banaji, ST Fiske, DS Massey - Cognitive Research: Principles and Implications, 2021

ST Fiske, DS Massey - Cognitive Research, 2021

Systemic racism is a scientifically tractable phenomenon, urgent for cognitive scientists to address. This tutorial reviews the built-in systems that undermine life opportunities and outcomes by racial category, with a focus on challenges to Black …


Tab2KG: Semantic Table Interpretation with Lightweight Semantic Profiles

… Subsequent steps such as property mapping are based on the results of this lookup step. However, as shown by Ritze et al. [14], only about 3% of the tables contained in the 3.5 billion HTML pages of the Common …


TABBIE: Pretrained Representations of Tabular Data

H Iida, D Thai, V Manjunatha, M Iyyer - arXiv preprint arXiv:2105.02584, 2021

… TaBERT's pretraining data was not publicly released at the time of our work, but their dataset consists of 26.6M ta- bles from Wikipedia and the Common Crawl. We thus form a pretraining dataset of equivalent size by combining …


Tabular Data Concept Type Detection Using Star-Transformers

Y Zhou, S Singh, C Christodoulopoulos - 2021

… We directly fine-tune TURL's concept type detection model on our dataset using its re- leased code. 4.2 Training Details We use 300 dimensional Glove [12] word embeddings pre-trained on 42 billion uncased tokens from Common …


Tackling Cyber-Aggression: Identification and Fine-Grained Categorization of Aggressive Texts on Social Media using Weighted Ensemble of Transformers

O Sharif, MM Hoque - Neurocomputing, 2021

… This model is trained with monolingual Bengali CommonCrawl corpus and utilises the BERT base model’s architecture. XLM-R is a cross-lingual model which outdoes m-BERT in various benchmarks. This model is built over the of 100 …


Tackling Italian University Assessment Tests with Transformer-Based Language Models

D Puccinelli, S Demartini, PL Ferrari - 2021

Cloze tests are a great tool to asses reading proficiency as well as analytical thinking, and are therefore employed in admission and assessment tests at various levels of the education system in multiple countries. In Italy, cloze tests are administered to …


Targeted Feedback Generation for Constructed-Response Questions

J Zhao, K Larson, W Xu, N Gattani, C Thille - 2020

… TextRank, BERT basic, and BERT fine-tuned models achieve better FPR scores. For non-transformer-based approaches, we ap- ply multitask CNN (trained on OntoNotes) and GloVe (1.2m 300d vectors trained on Common Crawl) as our embedding methods …


Task Aware Multi-Task Learning for Speech to Text Tasks

S Indurthi, MA Zaidi, NK Lakumarapu, B Lee, H Han… - ICASSP 2021-2021 IEEE …, 2021

… We use the following datasets for ST English-German, ASR English, MT English-German tasks: MT En-De: We use the Open Subtitles [9] and WMT 19 corpora. WMT 19 consists of Common Crawl, Europarl v9, and News …


Taxonomy Enrichment with Text and Graph Vector Representations

I Nikishina, M Tikhomirov, V Logacheva, Y Nazarov…


TBCOV: Two Billion Multilingual COVID-19 Tweets with Sentiment, Entity, Geo, and Gender Labels

M Imran, U Qazi, F Ofli - arXiv preprint arXiv:2110.03664, 2021

… Although the original XLM-R model is trained on one hundred languages using more than two terabytes of filtered CommonCrawl data4, its Twitter variant XLM-T achieves better performance on a large multilingual benchmark for sentiment analysis17 …


Technology-Skill Complementarity and Labor Displacement: Evidence from Linking Two Centuries of Patents with Occupations

L Kogan, D Papanikolaou, LDW Schmidt, B Seegmiller

Abstract We construct new technology indicators using textual analysis of patent documents and occupation task descriptions that span of two centuries (1850–2010). At the industry level, improvements in technology are associated with higher labor …


Temporal Effects on Pre-trained Models for Language Processing Tasks

O Agarwal, A Nenkova - arXiv preprint arXiv:2111.12790, 2021

Keeping the performance of language technologies optimal as time passes is of great practical interest. Here we survey prior work concerned with the effect of time on system performance, establishing more nuanced terminology for discussing the …


Term-community-based topic detection with variable resolution

A Hamm, S Odrowski - arXiv preprint arXiv:2103.13550, 2021

… The pre-trained fastText models are shallow neural nets with output vectors of dimension 300 that were trained on huge text collections from Wikipedia and Common Crawl, using the so called CBOW task of predicting a word by its surrounding words …


Term-Community-Based Topic Detection with Variable Resolution. Information 2021, 12, 221

Page 1. information Article Term-Community-Based Topic Detection with Variable Resolution Andreas Hamm * and Simon Odrowski Citation: Hamm, A.; Odrowski, S. Term-Community-Based Topic Detection with …


B Kennedy, A Ashokkumar, RL Boyd, M Dehghani - 2021

Page 1. Running Head: TEXT ANALYSIS METHODS FOR PSYCHOLOGY Text Analysis for Psychology: Methods, Principles, and Practices Brendan Kennedy1, Ashwini Ashokkumar2, Ryan Boyd3–5, Morteza …


Text Classification for Predicting Multi-level Product Categories

H Jahanshahi, O Ozyegen, M Cevik, B Bulut, D Yigit… - arXiv preprint arXiv …, 2021

… next-sentence pretraining objectives, and training with larger batch sizes. For our experiments, we use the multi-lingual version of this architecture pretrained on 2.5TB of CommonCrawl data in 100 languages using a …


Text Normalization for Low-Resource Languages of Africa

A Zupon, E Crew, S Ritchie - arXiv preprint arXiv:2103.15845, 2021

… The size of available corpora also varies greatly from 10K sentences (Swahili Newscrawl) to 1M sentences (Afrikaans Web and Mixed). The OSCAR9 corpus is a multilingual corpus of data obtained by language classification and filtering of the Common Crawl corpus …


Text Similarity Techniques for Matching Employee Objectives

A Elnaggar, M Ghanem, F Matthes, AD Mckinnon…

… https://huggingface.co./albert-xxlarge-v2 11. https://huggingface.co./distilbertbase-uncased 12. https://huggingface.co./roberta-base 13. https:// commoncrawl.org/2016/10/news-dataset-available/ 14. https://github.com …


Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents

U Bhattacharya, N Rewkowski, A Banerjee, P Guhan… - arXiv preprint arXiv …, 2021

… W = [w1 ... ws ... wTsen ], with Tsen being the maximum sentence length, using word embeddings ws ∈ R300. We obtain the word em- beddings using the GloVe model pre-trained on the Common Crawl corpus [52]. We opt …


Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents** This work has been supported in part by ARO Grants …

U Bhattacharya, N Rewkowski, A Banerjee, P Guhan… - 2021 IEEE Virtual Reality …, 2021

… W = [w1 ... ws ... wTsen], with Tsen being the maximum sentence length, using word embeddings ws ∈ R300. We obtain the word embeddings using the GloVe model pre-trained on the Common Crawl corpus [52]. We opt …


TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling

P Rileya, N Constantb, M Guob, G Kumarc, D Uthusb…

… For this purpose, we train a single general-purpose TextSETTR model, with the same configuration as our model from Section 3, except fine-tuned for 200k steps on English Common Crawl data (the same “C4” data that T5 pretrained on) instead of Amazon reviews …


Textual Analysis of News for Stock Market Prediction

AV Bogdanov, M Bogan, A Stankus - … Conference on Computational Science and Its …, 2021

… So, for Word2Vec there are pre-trained word-vectors on Google news with a dimension of 300. There is a dictionary for GloVe trained on various textual data, including on news with Common Crawl with a dimension of 300 …


TGIF: Tree-Graph Integrated-Format Parser for Enhanced UD with Two-Stage Generic-to Individual-Language Finetuning

T Shi, L Lee - arXiv preprint arXiv:2107.06907, 2021

… Representation The first step is to extract contextual representations. For this purpose, we use the pre-trained XLM-R model (Conneau et al., 2020), which is trained on multilingual CommonCrawl data and supports all 17 languages in the shared task …


The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task Overview and Evaluation Results (WebNLG

T Ferreira, C Gardent, N Ilinykh, C van der Lee, S Mille…

… 3.3 Mono-task, Bilingual Approaches cuni-ufal. The mBART model (Liu et al., 2020) is pre-trained for multilingual denoising on the large-scale multilingual CC25 corpus extracted from Common Crawl, which contains …


The Advantages of Human Evaluation of Sociomedical Question Answering Systems

V Firsanova - International Journal of Open Information Technologies, 2021

… XLM-RoBERTa is a model for one hundred languages trained on CommonCrawl data. According to the developers, this model improved the performance on low-resource languages of other crossand multilingual models developed earlier. This model is …


The Croatian psycholinguistic database: Estimates for 6000 nouns, verbs, adjectives and adverbs

A Peti-Stantić, M Anđel, V Gnjidić, G Keresteš… - Behavior Research Methods, 2021

Psycholinguistic databases containing ratings of concreteness, imageability, age of acquisition, and subjective frequency are used in psycholinguistic and.


The Danish Gigaword Corpus

L Derczynski, ITU Copenhagen, MR Ciosici, R Baglini…

… Massive, monolithic, automatically collected datasets of web content, such as Common Crawl, support the training of large language models but suffer from quality issues (Radford et al., 2019) and bias (Ferrer et al., 2021) …


The Data Set Knowledge Graph: Creating a Linked Open Data Source for Data Sets

ABSTRACT Several scholarly knowledge graphs have been proposed to model and analyze the academic landscape. However, although the number of data sets has increased remarkably in recent years, these knowledge graphs do not primarily …


The Demand for Executive Skills

J Fuller, S Hansen, T Ramdas, R Sadun - 2021

… tions. 'Off-the-shelf' models—typically estimated on large corpora that are representative of written language such as Wikipedia and Common Crawl for English—do exist, but se- mantic relatedness in generic English may …


The Effect of Domain and Diacritics in Yorubá–English Neural Machine Translation

DI Adelani, D Ruiter, JO Alabi, D Adebonojo, A Ayeni… - Proceedings of Machine …, 2021

… Fine-tuning mT5 We examine a transfer learning approach by fine-tuning a massively multilingual model mT5 (Xue et al., 2021). mT5 had been pre-trained on 6.3T tokens originating from Common Crawl in 101 languages (including Yor`ubá) …


The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding

A Prasad, MA Rehan, S Pathak, P Jyothi - arXiv preprint arXiv:2107.09931, 2021

… XLM-R uses a similar training objective as mBERT but is trained on orders of magnitude more data from the CommonCrawl corpus spanning 100 languages and yields competitive results on low-resource languages(Conneau et al., 2020) …


The FinSim-2 2021 Shared Task: Learning Semantic Similarities for the Financial Domain

Y Mansar, J Kang, IE Maarouf - Companion Proceedings of the Web Conference 2021, 2021

… where english words of similar meaning are grouped into sets called synsets, Wikidata is a knowledge base maintained by the Wikimedia Foundation and contains entities like "MSCI World" and finally WebIsALOD is a hypernymy …


The Impact of Main Content Extraction on Near-Duplicate Detection

M Fröbe, M Hagen, J Bevendorff, M Völske, B Stein… - arXiv preprint arXiv …, 2021

… Widely available web crawls—most notably the ClueWebs† and the Common Crawl‡—contain the (near-)duplicate documents that the crawler encountered during the crawling process. While the inclusion of near-duplicates enables many …


The Impact of Positional Encodings on Multilingual Compression

V Ravishankar, A Søgaard - arXiv preprint arXiv:2109.05388, 2021

… The second corpus uses the English Wikipedia as the training split, and Common Crawl as validation. We present corpus statistics in Table 3. For each corpus, we learn and apply a BPE vocabulary of size 2048. Train Validation Bible 30602 9080 Wikipedia 50000 20000 …


The information retrieval anthology 2021: inaugural status report and challenges ahead

M Potthast, B Stein, M Hagen - ACM SIGIR Forum, 2021

… Originally developed as a reproducible baseline search engine for the ClueWeb09, the ClueWeb12, and the Common Crawl, ChatNoir's distributed Elasticsearch backend also enables the inclusion of other kinds of indexes …


The Information Retrieval Anthology

M Potthast, S Günther, J Bevendorff, JP Bittner… - 2021

… To allow users to easily search the IR Anthology, we provide a dedicated search index accessible via our search engine ChatNoir [6], a proven web search engine indexing around 5 billion web pages from the ClueWeb crawls and Common Crawl versions …


The IWSLT 2021 BUT Speech Translation Systems

HK Vydana, M Karafi'at, L Burget - arXiv preprint arXiv:2107.06155, 2021

… Corpora #Sentences Audio Source text Target Text MT -Train-set ParaCrawl v3 31M - OpenSubtitles 2018 12M - Rapid 2019 1.5M - Europarl v9 1.81M - News Commentary 365K - Common Crawl 2.4M - Wikititles …


The Language of Situational Empathy

KE ZHOU, LM AIELLO, S SCEPANOVIC, D QUERCIA… - 2021

… Specifically, we averaged the 300-dimensional embedding vectors for each lexicon word using GloVe embeddings [55] trained on the Common Crawl corpus8. The above two approaches used a machine learning algorithm on top of our lexicon for the final prediction …


The NiuTrans End-to-End Speech Translation System\\for IWSLT 2021 Offline Task

C Xu, X Liu, X Liu, L Wang, C Huang, T Xiao, J Zhu - arXiv preprint arXiv:2107.02444, 2021

… Task Corpora Size Time ASR LibriSpeech 281241 960h Common Voice 1000000 1387h Total 1281241 2347h MT CommonCrawl 2014304 - Europarl 1802849 - ParaCrawl 31528317 - Wiki 5714363 - OpenSubtitles 14449099 - Total 55508932 - ST …


The NiuTrans Machine Translation Systems for WMT21

S Zhou, T Zhou, B Wei, Y Luo, Y Mu, Z Zhou, C Wang… - arXiv preprint arXiv …, 2021

… For the ParaCrawl v7.1, we only selected 8.5 million data according to the score of sentences provided by the dataset. We chose all of News Crawl and News Commentary and 12 million data sampled from Common Crawl for the Japanese monolingual data …


The Pile: An 800GB Dataset of Diverse Text for Language Modeling

L Gao, S Biderman, S Black, L Golding, T Hoppe… - arXiv preprint arXiv …, 2020

… The growing need for data in language modeling has caused most existing large-scale language models to turn to the Common Crawl for most or all of their data (Brown et al., 2020; Raffel et al., 2019). While training on the …


The Potential of Automated Text Analytics in Social Knowledge Building

R Németh, J Koltai - Pathways Between Social Science and Computational …, 2021

… Finally, big data are often samples themselves, resulting from an unknown or hard-to-formalize sampling procedure. See, eg, Common Crawl (http://commoncrawl.org), an open repository of textual web page data …


The Power of Anchor Text in the Neural Retrieval Era

M Fröbe, S Günther, M Probst, M Potthast, M Hagen

… and Eiron and McCurley on MS MARCO, we extract anchor texts from Common Crawl snapshots. We randomly select one snapshot from each year from 2016 to 2021 (each containing 1.7–3.4 billion documents) and extract the anchor texts of …


The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing

M Tomalin, B Byrne, S Concannon, D Saunders… - Ethics and Information …, 2021

This article probes the practical ethical implications of AI system design by reconsidering the important topic of bias in the datasets used to train auton.


The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

V Nikoulina, M Tezekbayev, N Kozhakhmet… - arXiv preprint arXiv …, 2021

… English-to-German translation. The authors used the CommonCrawl-840B GloVe model (Pennington et al., 2014) for English word vectors, which was completely fixed during pretraining, and we follow their setup. This entails that …


The SPECTRANS System Description for the WMT21 Terminology Task

NBDCB Faye, ZYKHM Mojca, PJBYG Wisniewski…

… The latter did better for sacreBLEU and Window Overlap Accuracy (n=3) but probably having seen the terminological resources in the training data gave an edge for Exact-Match Accuracy to our model trained with Common Crawl and the terminological …


D Iter, D Grangier - arXiv preprint arXiv:2109.10274, 2021

Page 1. The Trade-offs of Domain Adaptation for Neural Language Models David Grangier Google, Mountain View, CA grangier@google.com Dan Iter∗ Stanford, Palo Alto, CA daniter@stanford.edu Abstract This work connects …


The University of Edinburgh's Bengali-Hindi Submissions to the WMT21 News Translation Task

Abstract We describe the University of Edinburgh’s Bengali↔ Hindi constrained systems submitted to the WMT21 News Translation task. We submitted ensembles of Transformer models built with large-scale back-translation and finetuned on subsets …


The University of Edinburgh's English-German and English-Hausa Submissions to the WMT21 News Translation Task

… Page 2. • Crawled parallel: ParaCrawl, WikiMatrix, CommonCrawl, and WikiTitles … For back-translation, we use 9.5 million monolingual Hausa sentences from Common Crawl, Extended Common Crawl, and News Crawl provided by the task organizers …


The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021

D Liu, M Du, X Li, Y Hu, L Dai - arXiv preprint arXiv:2107.00279, 2021

… sentences from it. Corpus Sentences EN→DE MuST-C(v2) 229.7k Europarl 1828.5k Rapid-2019 1531.3k WIT3-TED 209.5k Commoncrawl 2399.1k WikiMatrix 6227.2k Wikititles 1382.6k Paracrawl 82638.2k EN→JA WIT3-TED 225.0 …


The USYD-JD Speech Translation System for IWSLT 2021

L Ding, D Wu, D Tao - arXiv preprint arXiv:2107.11572, 2021

… Corpus #Sent. commoncrawl English 4,366,344 commoncrawl Swahili 38,928 + upsampling (14×) 544,992 … Mono. Corpus for Tagged BT #Sent. Totally collected corpus commoncrawl English 30,513,498 Cleaned corpus …


The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English

… We use two input word representations: the 300-d GloVe vectors trained on Common Crawl (Pennington et al., 2014), which is the dominant representation in NER, and the 1024-d contextual ELMo (Peters et al …


The Volctrans Neural Speech Translation System for IWSLT 2021

C Zhao, Z Liu, J Tong, T Wang, M Wang, R Ye, Q Dong… - arXiv preprint arXiv …, 2021

… Data Combination and Sampling Strategy We train transformer models with different combina5http://www.statmt.org/wmt20/ translation-task.html, including Common Crawl, tions of data sets because increasing the model's diversity can benefit the model ensemble …


The Web Is Your Oyster--Knowledge-Intensive NLP against a Very Large Web Corpus

A Piktus, F Petroni, V Karpukhin, D Okhonko… - arXiv preprint arXiv …, 2021

… CCNet processes Common Crawl by performing deduplication, language identification and quality filtering (articles are split into three quality tiers: head, … We pick the CCNet snapshot corresponding to the August 2019 Common Crawl …


TheEyeCorpus: Experiments in Reducing NLP Bias and Identifiability for Large LMs

… 2020), a state of the art large language model, that used terabytes of the mostly unfiltered Common Crawl. Making corpuses is usually as simple as downloading and extracting the common crawl database. This isn’t the most ideal approach …


Thor at Touché 2021: Argument Retrieval for Comparative Questions

E Shirshakova, A Wattar - Working Notes of CLEF, 2021

… doi:10.1145/2348283.2348429. [4] J. Bevendorff, B. Stein, M. Hagen, M. Potthast, Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl, in: L. Azzopardi, A. Hanbury, G. Pasi, B. Piwowarski (Eds.), Advances in Information Retrieval …


Page 1. Abstract Three Essays on the Role of Unstructured Data in Marketing Research Ishita Sunity Kumar Chakraborty 2021 This thesis studies the use of firm and user-generated unstructured data (eg, text and videos) for …


Time and Cost Prediction Models for Language Classification Over a Large Corpus on Spark

JB Rodrigues, GC Vasconcelos, PRM Maciel - 2020 IEEE Symposium Series on …, 2020

… All nodes were managed by a Debian GNU/Linux 9 OS, Apache Spark 2.3, and Apache Hadoop 2.9. 2 The Common Crawl Foundation - http://commoncrawl.org/ 1) Analysis of Variance: Fig. 1 displays the results over the log-transformed data …


TMEKU System for the WAT2021 Multimodal Translation Task

Y Zhao, M Komachi, T Kajiwara, C Chu - red

… Subsequently, we utilize fastText (Bojanowski et al., 2017) to learn subword embeddings. We use a pre-trained model4 containing two million word vectors trained with subword information on Common Crawl (600B tokens) …


Token-Level Multilingual Epidemic Dataset for Event Extraction

S Mutuvi, E Boros, A Doucet, G Lejeune, A Jatowt… - International Conference on …, 2021

… 8. XLM-RoBERTa-base was trained on 2.5TB of CommonCrawl data in 100 languages. References. 1. Aiello, AE, Renson, A., Zivich, PN: Social media-and internet-based disease surveillance for public health. Annu. Rev. Public …


Touché Task 2: Comparative Argument Retrieval. A Document-based Search Engine for Answering Comparative Questions

D Helmrich, D Streitmatter, F Fuchs, M Heykeroth - Working Notes of CLEF, 2021

… eg faster). The results are sentences from the Common Crawl2 retrieved via ElasticSearch [7] and then reduced to comparative sentences (either by certain keywords or a machine learning approach) and ranked. The system …


Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias throughout the Machine Learning Lifecyle

YT Cao, H Daumé - Computational Linguistics, 2021

Page 1. Toward Gender-Inclusive Coreference Resolution An Analysis of Gender and Bias throughout the Machine Learning Lifecyle∗ Yang Trista Cao∗∗ University of Maryland Hal Daumé III† Microsoft Research University of Maryland …


Toward the Understanding of Deep Text Matching Models for Information Retrieval

L Chen, Y Lan, L Pang, J Guo, X Cheng - arXiv preprint arXiv:2108.07081, 2021

… For the input word embeddings, we initialize the embedding layer with the 300-dimensional Glove[29] word vectors pre-trained in the 840B Common Crawl corpus4. For the out-of … 3https://github.com/NTMC-Community …


Towards a Machine Learning Based Generalizable Framework for Detecting COVID-19 Misinformation on Social Media

Page 1. University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln …


TOWARDS A ROMANIAN END-TO-END AUTOMATIC SPEECH RECOGNITION BASED ON DEEPSPEECH2

… [17]) layers that have 768 units. The output of the LSTM layers is 7 https://commoncrawl org/ 8 We used langdetect to detect Romanian sentences: https://github.com/Mimino666/ langdetect. Page 4. Andrei-Marius AVRAM, Vasile PĂIȘ, Dan TUFIȘ 398 …


Towards better subtitles: A multilingual approach for punctuation restoration of speech transcripts

NM Guerreiro, R Rei, F Batista - Expert Systems with Applications, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Towards Change Detection in Privacy Policies with Natural Language Processing

A Adhikari, R Dewri - 2021 18th International Conference on Privacy …, 2021

Privacy policies notify users about the privacy practices of websites, mobile apps, and other products and services. However, users rarely read them and struggle to understand their contents. Due to the complicated nature of these documents, it gets …


Towards Continual Knowledge Learning of Language Models

J Jang, S Ye, S Yang, J Shin, J Han, G Kim, SJ Choi… - arXiv preprint arXiv …, 2021

… 3CC-RECENTNEWS consists of 221,779 articles (∼168M tokens), which is estimated to be about 750 times smaller than C4, a cleansed version of the April 2019 Common Crawl dataset (https://commoncrawl.org/) that was …


X Garcia, N Constant, AP Parikh, O Firat - arXiv preprint arXiv:2103.06799, 2021

Page 1. Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution Xavier Garcia and Noah Constant and Ankur P. Parikh and Orhan Firat Google Research Mountain View California …


Towards Controllable Neural Generation of Arguments

Page 1. Page 2. TOWARDS CONTROLLABLE NEURAL GENERATION OF ARGUMENTS A Dissertation Presented to The Khoury College of Computer Sciences of Northeastern University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy …


Towards Generalization in Dialog through Inductive Biases

Abstract Generalization is imperative in dialog research. Data-driven models have been shown to be capable of performing specific tasks in constrained contexts, given ample data. However, the complexity of human communication necessitates …


Towards Learning a Joint Representation from Transformer in Multimodal Emotion Recognition

JJ Deng, CHC Leung - International Conference on Brain Informatics, 2021

… modality of speech audio. As for text representation, we adopt fine-tuned Text-To-Text Transfer Transformer (T5) [18] model trained by a common crawl (C4) dataset to extract text embeddings. Operations of transfer learning …


Page 1. Towards Learning Language Agnostic Features for NLP in Low-resource Languages Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science and …


Towards Learning Terminological Concept Systems from Multilingual Natural Language Text

L Wachowiak, C Lang, B Heinisch, D Gromann - 3rd Conference on Language, Data …, 2021

… [6]. We further added to the data by manually annotating around 200 acronym-term pairs from ACTER with relations other than synonymy and another 271 sample sentences from the Common Crawl News Corpus5 … 5 …


Y Liang, C Zhou, F Meng, J Xu, Y Chen, J Su, J Zhou - arXiv preprint arXiv …, 2021

Page 1. Towards Making the Most of Dialogue Characteristics for Neural Chat Translation Yunlong Liang1∗, Chulun Zhou2∗, Fandong Meng3, Jinan Xu1†, Yufeng Chen1, Jinsong Su2 and Jie Zhou3 1Beijing Key Lab of Traffic …


Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

G Chen, S Ma, Y Chen, D Zhang, J Pan, W Wang… - arXiv preprint arXiv …, 2021

… 2018, SPM) learned on the full Common Crawl data that includes 250k subword tokens. We do not apply additional preprocessing, such as truecasing or normalizing punctuation/characters. Following XLMR, we add the [BOS] and [EOS] tokens at the …


Towards Model Understanding

While deep learning models have become increasingly accurate over the last decade, concerns about their (lack of) interpretability have taken a center stage. In response, a growing sub-field on interpretability and analysis of these models has …


Towards More Effective and Economic Sparsely-Activated Model

H Jiang, K Zhan, J Qu, Y Wu, Z Fei, X Zhang, L Chen… - arXiv preprint arXiv …, 2021

L Chen, Z Dou, X Qiu, Z Guo, R Lai, J Wu, E Hu…

… RealNews is a large scale dataset about news articles from Common Crawl OpenWebText dataset contains a large amount of text data from social media platform Reddit. They scraped all outbound links from Reddit as data sources and …


Towards Multi-modal Entity Resolution for Product Matching

… Finally a procedure to query the internet archive2 for missing images is used. Due to its slow speed this method is only applied selectively to achieve higher coverage of images for the experiments. 1http://commoncrawl.org/ 2https://archive.org/ Page 3 …


E Tokarchuk, J Rosendahl, W Wang, P Petrushkov… - arXiv preprint arXiv …, 2021

Page 1. Towards Reinforcement Learning for Pivot-based Neural Machine Translation with Non-autoregressive Transformer Evgeniia Tokarchuk 1 2 Jan Rosendahl 2 Weiyue Wang 2 Pavel Petrushkov 1 Tomer Lancewicki 1 Shahram Khadivi 1 Hermann Ney 2 …


Towards Selecting Informative Content for Cyber Threat Intelligence

P Panagiotou, C Iliou, K Apostolou, T Tsikrika… - 2021 IEEE International …, 2021

… NER model. For their implementation, we used the Delft tool3, and the Glove Common Crawl embeddings4. The Bi- directional LSTM-CNNs-CRF architecture of [29] performs the best according to all metrics considered. We …


Towards Target-dependent Sentiment Classification in News Articles

… category-dataset Page 10. 10 F. Hamborg et al. 23. Nagel, S.: Common Crawl: News Crawl (2016), https://web. archive.org/web/20191118111519/https://commoncrawl org/2016/10/ news-dataset-available/ 24. Nakov, P., Ritter …


Towards the smart use of embedding and instance features for property matching

D Ayala, I Hernández, D Ruiz, E Rahm - 2021 IEEE 37th International Conference on …, 2021

… To compute embeddings, we use the pre-trained GloVe approach [25]1, specifically for the uncased Common Crawl corpus that includes 300-dimensional vectors for 1.9 million words. Unknown words are mapped to a vector filled with zeroes …


Towards Zero-Shot Cross-Lingual Named Entity Disambiguation

A Barrena Madinabeitia, A Soroa Echave… - 2021

A Barrena, A Soroa, E Agirre - Expert Systems with Applications, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


Traceability recovery between bug reports and test cases-a Mozilla Firefox case study

G Gadelha, F Ramalho, T Massoni - Automated Software Engineering, 2021

Automatic recovery of traceability between software artifacts may promote early detection of issues and better calculate change impact. Information Retriev.


Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction

… Data collection approaches using the CommonCrawl1 have flourished as they allow for faster download and processing by skipping (or more precisely outsourcing) the crawling phase … 1https://commoncrawl.org Page 2. 123 …


Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

O Press, NA Smith, M Lewis - arXiv preprint arXiv:2108.12409, 2021

Page 1. TRAIN SHORT, TEST LONG: ATTENTION WITH LINEAR BIASES ENABLES INPUT LENGTH EXTRAPOLATION Ofir Press1,2 Noah A. Smith1,3 Mike Lewis2 1Paul G. Allen School of Computer Science & Engineering …


Training Domain Specific Multilingually Aligned Word Embeddings

… FastText The pre-trained Fasttext embeddings follow the training procedure proposed in [5]. They use a mixture of Wikipedia and CommonCrawl as training data and learn 300 dimensional embeddings with a CBOW model defined …


J Shen, J Liu, T Liu, C Yu, J Han - arXiv preprint arXiv:2106.00139, 2021

Page 1. Training ELECTRA Augmented with Multi-word Selection Jiaming Shen∇⋆, Jialu Liu O , Tianqi Liu O , Cong Yu O , Jiawei Han∇ ∇University of Illinois Urbana-Champaign, IL, USA, 3 Google Research, NY, USA ∇{js2 …


Transfer Learning for Automated Responses to the BDI Questionnaire

C Spartalis, G Drosatos, A Arampatzis - Working Notes of CLEF, 2021

… batches, and over more data. More specifically, its pretraining corpus also includes CC-News (portion of the CommonCrawl News dataset [44]), OpenWebText [45], and Stories [46]). Furthermore, there are some modifications …


X Huang, J Xu, M Sun, Y Liu - arXiv preprint arXiv:2105.14809, 2021

Page 1. Transfer Learning for Sequence Generation: from Single-source to Multi-source Xuancheng Huang1, Jingfang Xu4, Maosong Sun1,3, and Yang Liu1,2,3∗ 1Dept. of Comp. Sci. & Tech., BNRist Center, Institute for AI, Tsinghua …


Transfer Learning Model for Disrupting Misinformation During a COVID-19 Pandemic

R Raju, S Bhandari, SA Mohamud, EN Ceesay - 2021 IEEE 11th Annual Computing …, 2021

… 19–27, IEEE Computer Society, 2015. [30] S. Nagel, “Cc-news.” https://commoncrawl.org/ 2016/10/news-datasetavailable/, 2016. [31] A. Gokaslan and V. Cohen, “Openwebtext corpus.” https://skylion007.github.io/OpenWebTextCorpus/, 2016 …


Transfer Learning Techniques for Sequence Labeling in Network File System Specifications

… English Wikipedia and BookCorpus (Zhu et al., 2015), and RoBERTa (Liu et al., 2019) is pre-trained on Common-Crawl News (Nagel, 2016), OpenWebText (Radford et al., 2019), and Stories (Trinh and Le, 2019) …


Transformer-Based Automatic Punctuation Prediction and Word Casing Reconstruction of the ASR Output

J Švec, J Lehečka, L Šmídl, P Ircing - International Conference on Text, Speech, and …, 2021

… For experiments with English data, we used the Google's T5-base English model (version from October 21th, 2019) 3 trained from Common Crawl data 4 . We replicated the same pre-processing procedure to obtain the …


Transformer-based identification of stochastic information cascades in social networks using text and image similarity

P Kasnesis, R Heartfield, X Liang, L Toumanidis… - Applied Soft Computing, 2021

… An advantage of RoBERTa against BERT for text-based information cascade identification is its pretrained architecture which benefits from a more diverse range of datasets (larger corpus). For example, its training corpus includes the CommonCrawl News dataset.


Page 1. Page 2. E¨OTV¨OS LORÁND UNIVERSITY FACULTY OF SCIENCE TRANSFORMERS AND THEIR APPLICATIONS IN NATURAL LANGUAGE PROCESSING MASTER'S THESIS Nurzhan Sarzhan MSc mathematics …


Transforming Term Extraction: Transformer-Based Approaches to Multilingual Term Extraction Across Domains

C Lang, L Wachowiak, B Heinisch, D Gromann

… XLM-R (Conneau et al., 2020) is a multilingual variant of BERT, which was pretrained in 100 languages using 2.5 terabytes of Common Crawl data. Moreover, it makes use of the improved training routine introduced by RoBERTa (Liu et al., 2019) …


Translate and Classify: Improving Sequence Level Classification for English-Hindi Code-Mixed Data

D Gautam, K Gupta, M Shrivastava - Proceedings of the Fifth Workshop on …, 2021

… auto-encoder. It has been pre-trained using the BART (Lewis et al., 2020) objective on large-scale monolingual corpora of 25 languages extracted from Common Crawl1 (Wenzek et al., 2020; Conneau et al., 2020). Both English …


TrollsWithOpinion: A Dataset for Predicting Domain-specific Opinion Manipulation in Troll Memes

S Suryawanshi, BR Chakravarthi, M Arcan, S Little… - arXiv preprint arXiv …, 2021

… 5.2 Deep Learning baselines Bag of words (BOW) : This unimodal baseline used text representation in the form of 300-dimensional pre-trained GloVe Pennington et al. (2014) (common crawl) word embeddings, which were fed to a classification layer …


TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect

A Messaoudi, A Cheikhrouhou, H Haddad, N Ferchichi… - arXiv preprint arXiv …, 2021

… They also concluded that a model trained on a CommonCrawl-based corpus performed consistently better than the one trained on the French Wikipedia. They suggested that a 4 GB heterogeneous dataset in terms of genre and style is large enough as a …


Turkic Interlingua: A Case Study of Machine Translation in Low-resource Languages

… production. Table 4 shows results from the CCAligned dataset, which is a parallel dataset compiled from 68 Common Crawl snapshots … shown in Table 6. Data for OSCAR is also a compilation of Common Crawl snapshots and follows a similar filtering process …


Twin Question Pair Classification

A Sharma, SS Jha, S Arora, S Garg, S Tayal - Smart and Sustainable Intelligent …, 2021

… 4 Embedding Embedding maps words in a text document to number space (these number-space representations are called word vectors). We have to use Fast text embedding [6]. The Fast text model for English is pre-trained …


Twitter Dataset and Evaluation of Transformers for Turkish Sentiment Analysis

A Köksal, A Özgür - 2021 29th Signal Processing and Communications …, 2021

… Kelime Büyüklü˘gü 128,000 119,547 250,002 Kullanılan Diller Türkçe Türkçe + 103 dil Türkçe + 99 dil E˘gitim Derlemi Oscar + Wikipedia + Opus Wikipedia CommonCrawl vektör uzunlu˘gu, dikkat ve gizli katmanların sayısı bu üç mo- delde de aynıdır …


Page 1. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2726–2738 August 1–6, 2021. ©2021 Association for Computational Linguistics 2726 Two Parents, One Child: Dual Transfer for Low-Resource Neural Machine Translation …


Two-stage Visual Cues Enhancement Network for Referring Image Segmentation

Y Jiao, Z Jie, W Luo, J Chen, YG Jiang, X Wei, L Ma - arXiv preprint arXiv:2110.04435, 2021

… As for language processing, GloVe [27] word embeddings pretrained on Common Crawl 840B tokens are firstly used for constructing word embedding look-up table. Then a LSTM takes GloVe word embeddings as input for capturing context information …


Type-and Token-based Word Embeddings in the Digital Humanities

A Ehrmanntraut, T Hagen, L Konle, F Jannidis - Proceedings http://ceur-ws. org ISSN, 2021

Abstract In the general perception of the NLP community, the new dynamic, context-sensitive, token-based embeddings from language models like BERT have replaced the older static, type-based embeddings like word2vec or fastText, due to their better …


UGTO: Uncommon Words and Proper Nouns

X Zhong, E Cambria - Time Expression and Named Entity Recognition, 2021

… they are (1) word2vec, which is trained on the Google News dataset [13], (2) GloVe, which is trained on the Wikipedia 2014 and Gigaword 5 corpora [20], and (3) FastText, which is trained on the Wikipedia 2017, UMBC corpus …


N Alvarez-Gonzalez, A Kaltenbrunner, V Gómez - arXiv preprint arXiv:2109.01900, 2021

Page 1. Uncovering the Limits of Text-based Emotion Detection Nurudin Alvarez-Gonzalez ∗, Andreas Kaltenbrunner, Vicenç Gómez nuralgon@gmail.com, andreas.kaltenbrunner@ upf.edu, vicen.gomez@upf.edu Universitat Pompeu Fabra. Barcelona, Spain. Abstract …


Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model

… trained word em- beddings in the gensim library ( ˇRehurek and Sojka, 2010), with the results given in Table 2. The FastText embeddings generally outperform the other embeddings on this task, with the 2M word model trained …


Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

H Liu, V Lai, C Tan - arXiv preprint arXiv:2101.05303, 2021

… Race. There are 7,214 defendants in this dataset. • BIOS [9]. This dataset contains hundreds of thousands of online biographies from the Common Crawl corpus. The task is to predict a person's profession given a biography …


Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus

J Abadji, PJ Ortiz Suárez, L Romary, B Sagot - … of the Workshop on Challenges in the …, 2021

… The most recent Common Crawl dump contains 64,000 shards. Each shard is composed of numerous records, and each record holds textual content along with metadata. While CommonCrawl shards hold document-level …


Unified Open-Domain Question Answering with Structured and Unstructured Knowledge

B Oguz, X Chen, V Karpukhin, S Peshterliev… - arXiv preprint arXiv …, 2020

… KBQA is ultimately limited in its coverage of facts and the types of questions it can answer. On the other hand, large collections of text such as Wikipedia or CommonCrawl promise to be a richer source of knowledge for truly open do- main question answering systems …


Universal Sentence Representation Learning with Conditional Masked Language Model

Z Yang, Y Yang, D Cer, J Law, E Darve - arXiv preprint arXiv:2012.14388, 2020

… For training English sentence encoders with CMLM, we use three Common Crawl dumps … Similarly, we also train QuickThought, a competitive unsupervised sentence representations learning model, on the same …


University of Copenhagen Participation in TREC Health Misinformation Track 2020

LC Lima, DB Wright, I Augenstein, M Maistro - arXiv preprint arXiv:2103.02462, 2021

… The corpus of the Total Recall and Ad Hoc Tasks consits of news articles in the CommonCrawl News crawl, sampled from January, 1st 2020 … Since there are no credibility assessments for the Common Crawl News sample …


University of Regensburg@ PAN: Profiling Hate Speech Spreaders on Twitter

KO Akomeah, U Kruschwitz, B Ludwig - CLEF, 2021

… This preprocessor is a companion to the BERT models for preprocsssing plain text inputs into the input format expected by BERT. The model uses a vocabulary for multilingual models extracted from Wikipedia, CommonCrawl, and translation pairs from the Web …


Unraveling Social Perceptions & Behaviors towards Migrants on Twitter

A Khatua, W Nejdl - arXiv preprint arXiv:2112.06642, 2021

We draw insights from the social psychology literature to identify two facets of Twitter deliberations about migrants, ie, perceptions about migrants and behaviors towards mi-grants. Our theoretical anchoring helped us in identifying two prevailing …


Unsupervised Clustering of Structured and Unstructured Text Collections

Page 1. Lars B u n g u m Doctoral theses at NTNU, 202 1 :179 ISBN 978-82-326-5390-4 (printed ver.) ISBN 978-82-326-6966-0 (electronic ver.) ISSN 1503-8181 (printed ver.) ISSN 2703-8084 (electronic ver.) Do cto ral …


Unsupervised Cross-Domain and Cross-Lingual Methods for Text Classification, Slot-Filling, and Question-Answering

Page 1. UNSUPERVISED CROSS-DOMAIN AND CROSS-LINGUAL METHODS FOR TEXT CLASSIFICATION, SLOT-FILLING, AND QUESTION-ANSWERING by Jitin Krishnan A Dissertation Submitted to the Graduate Faculty …


Unsupervised Neural Machine Translation with Generative Language Models Only

JM Han, I Babuschkin, H Edwards, A Neelakantan… - arXiv preprint arXiv …, 2021

We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: few-shot amplification, distillation, and backtranslation. We first use the …


Uppsala NLP at SemEval-2021 Task 2: Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguation

H You, X Zhu, S Stymne - arXiv preprint arXiv:2104.03767, 2021

… 3 Multilingual Language Models 3.1 XLMR XLMR (XLM-RoBERTa) is a scaled cross-lingual sentence encoder (Conneau et al., 2020), which is trained on 2.5T of data obtained from Common Crawl that covers more than 100 languages …


UPV at TREC Health Misinformation Track 2021 Ranking with SBERT and Quality Estimators

IB Schlicht, AFM de Paula, P Rosso - arXiv preprint arXiv:2112.06080, 2021

Health misinformation on search engines is a significant problem that could negatively affect individuals or public health. To mitigate the problem, TREC organizes a health misinformation track. This paper presents our submissions to this …


UQuAD1. 0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension

S Kazi, S Khoja - arXiv preprint arXiv:2111.01543, 2021

… The Third model used is XLM-Roberta, a multi-lingual pre-trained transformer model from common Crawl trained on 100 languages, including Urdu. XLM-Roberta outperformed previous multi-lingual models such as mBERT and XLM on a variety of …


User Identification in Online Social Networks using Graph Transformer Networks

KNP Kumar, ML Gavrilova - 2021 18th International Conference on Privacy …, 2021

… We used the model with 2 million word vectors trained with subword information on Common Crawl (600B tokens). (v) Skip-thought Vectors (StV) [30]: A Gated Recurrent Unit (GRU) encoder trained to predict adjacent sentences in a books …


Using Embedding-Based Similarities to Improve Lexical Resources

NV Loukachevitch, MM Tikhomirov, EA Parkhomenko - Lobachevskii Journal of …, 2021

In this paper we discuss the usefulness of applying semi-automatic checking procedures to existing thesauri for natural language processing—large man.


Using mixed methods to study the historical use of web beacons in web tracking

J Nielsen - International Journal of Digital Humanities, 2021

… pixel. The approach is inspired by the study of web beacons by Ruohonen and Leppänen (2018) and Schelter and Kunegis' study of third-party embeddings in the Common Crawl 2012 corpus (Schelter & Kunegis, 2016, 2018) …


USING SEMANTIC RELATIONSHIPS TO ENHANCE NEURAL WORD EMBEDDINGS

Y Yin, X Sun, H Lv, P Wang, H Ma, D Yang

Neural language models have significantly improved current natural language understanding tasks. However, distributional semantics, derived from neural language models is less competitive in computing semantic relatedness or similarity …


Page 1. Using Visual Features to Generate Language Context Undergraduate Honors Research Thesis Presented in Partial Fulfillment of the Requirements for Undergraduate Honors Research Thesis in the College of Engineering at The Ohio State University By …


Page 1. Using Word Embeddings to Determine Concepts of Values In Insurance Claim Spreadsheets Vemund Justnes Thesis submitted for the degree of Master in Informatics: Programming and System Architecture 60 credits Institute for Informatics …


Using word embeddings to probe sentiment associations of politically loaded terms in news and opinion articles from news media outlets

D Rozado, M al-Gharbi - Journal of Computational Social Science, 2021

… word embeddings also absorb many of the biases that are prevalent within a cultural system, as a result of embedding models being trained on culturally relevant language artifacts that are presumed to contain cultural biases, such …


Utility-Preserving Anonymization of Textual Documents

Page 1. UTILITY-PRESERVING ANONYMIZATION OF TEXTUAL DOCUMENTS Fadi Abdulfattah Mohammed Hassan ADVERTIMENT. L'accés als continguts d'aquesta tesi doctoral i la seva utilització ha de respectar els drets de la persona autora …


Utility-Preserving Privacy Protection of Textual Documents via Word Embeddings

F Hassan, D Sanchez, J Domingo-Ferrer - IEEE Transactions on Knowledge and Data …, 2021

Page 1. 1041-4347 (c) 2021 IEEE. Personal use is permitted, but republication/ redistribution requires IEEE permission. See http://www.ieee.org/ publications_standards/publications/rights/index.html for more information. This …


UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP

… Manning, 2019). We use XLM-R masked LM (Conneau et al., 2020) as our vicinity model θmlm, which is trained on massive multilingual corpora (2.5 TB of Common-Crawl data in 100 languages). The Page 4. 1981 Algorithm 1 …


UzBERT: pretraining a BERT model for Uzbek

B Mansurov, A Mansurov - arXiv preprint arXiv:2108.09814, 2021

… Page 3. 3 Uzbek Wikipedia and the Common Crawl corpus. According … 2006], where full words are replaced by their abbreviations to save space in print. The filtered Common Crawl corpus5 used by XLM-R [Conneau et al. 2019 …


Page 1. Validating Word Embedding as a Tool for the Psychological Sciences by Victor Swift A thesis submitted in conformity with the requirements for the degree of Doctorate of Philosophy Department of Psychology University of …


Variational Autoencoder with Interactive Attention for Affective Text Generation

R Chen, J Wang, X Zhang - CCF International Conference on Natural Language …, 2021

… The word embeddings used in this experiment was pre-trained on Common Crawl 840B 1 by GloVe [10]. For baselines, the polarity label \(\mathbf {c}\) was treated as a scalar (ie, 0 or 1) and directly concatenated into the latent variables. For …


VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

F Luo, W Wang, J Liu, Y Liu, B Bi, S Huang, F Huang…

… mBART (Liu et al., 2020b) Encoder-decoder 680M 12 12 25 250k CommonCrawl VECO Flexible 662M 24* 50 250k CommonCrawl + Translation … For monolingual training datasets, we reconstruct CommonCrawl Corpus used in XLM-R (Conneau et al., 2019) …


Vera: Prediction Techniques for Reducing Harmful Misinformation in Consumer Health Search

R Pradeep, X Ma, R Nogueira, J Lin - Proceedings of the 44th Annual International …, 2021

… The document collection comprises news articles from the CommonCrawl news crawl spanning the first four months of 2020 (January 1st, 2020 to April 30th, 2020), covering the onset of the COVID-19 pandemic. This corpus …


VERB: Visualizing and Interpreting Bias Mitigation Techniques for Word Representations

A Rathore, S Dev, JM Phillips, V Srikumar, Y Zheng… - arXiv preprint arXiv …, 2021

… It also provides a larger GLoVe embedding (300-dimensional with the 100K most frequent words) from the Common Crawl corpus (https://commoncrawl.org/). 5 USE CASES We demonstrate the efficacy of VERB on several examples …


VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?

H Nguyen, L Phan, J Anibal, A Peltekian, H Tran - arXiv preprint arXiv:2110.04257, 2021

… Common Crawl is a publicly web archive that provides text scraped from websites. These datasets (C4, CC25, and Common Crawl) may include Vietnamese Wikipedia languages website that help the model excel in constrained domain corpus (Wikipedia) …


Visual and Affective Multimodal Models of Word Meaning in Language and Mind

S De Deyne, DJ Navarro, G Collell, A Perfors - Cognitive Science, 2021

Abstract One of the main limitations of natural language‐based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how wel...


Visual Grounding Strategies for Text-Only Natural Language Processing

D Sileo - arXiv preprint arXiv:2103.13942, 2021

… A linear projection maps the region features to the input space of the cross-modal encoder. We also provide K rank 6We use the CommonCrawl version https:// fasttext.cc/docs/en/english-vectors.html. and discard stop-words with NLTK …


Visualizing large-scale high-dimensional data via hierarchical embedding of KNN graphs

H Zhu, M Zhu, Y Feng, D Cai, Y Hu, S Wu, X Wu… - Visual Informatics, 2021

JavaScript is disabled on your browser. Please enable JavaScript to use all the features on this page. Skip to main content Skip to article …


VITALITY: Promoting Serendipitous Discovery of Academic Literature with Transformers & Visual Analytics

A Narechania, A Karduni, R Wesslen, E Wall - arXiv preprint arXiv:2108.03366, 2021

… Unlike many pre-trained language models that use a general corpus like Wikipedia or the Common Crawl [14, 43, 49], SPECTER was pre-trained on academic literature (sciBERT [?]) and fine-tuned with citations which …


VOH. CoLAB at TREC 2020 Health Misinformation Track⋆

… 2.1 Collection The collection contains 65 million news articles from CommonCrawl News4 corresponding to the period from January to April of 2020 … Recently, vinegar has been promoted as a disinfectant (...) 4 …


Voted In, Standing Out: Public Response to Immigrants' Political Accession

… newspapers, covering the general elections from 2010–2019.8 This data is from Common Crawl, which is an open repository of web crawl data. We assume that an article refers to a candidate's ethnic group when three conditions are met: 1) the publication date is …


We Need to Talk About Data: The Importance of Data Readiness in Natural Language Processing

F Olsson, M Sahlgren - arXiv preprint arXiv:2110.05464, 2021

In this paper, we identify the state of data as being an important reason for failure in applied Natural Language Processing (NLP) projects. We argue that there is a gap between academic research in NLP and its application to problems outside …


Web Archive Analytics

M Völske, J Bevendorff, J Kiesel, B Stein, M Fröbe… - INFORMATIK 2020, 2021

… in Figure 5, beginning with the bottom-most data acquisition layerȷ Primary sources for data ingestion include web crawls and web archives, such as the aforementioned Internet Archive, the Common Crawl,13 the older … 13 …


Web Content Authentication: A Machine Learning Approach to Identify Fake and Authentic Web Pages on Internet

J Ashok, P Badoni - … Technology for Competitive Strategies (ICTCS 2020) …

… SpringerLink (Springer, 21 June 2017). www. link. springer. com/chapter/10.1007/978- 3-319-69784-0_15 42. Link to GitHub where code is hosted. https://github. com/Jkrish1011/Web-Content-Authentic ator 43. https://commoncrawl. org/


Web Table Classification based on Visual Features

B Bühler, H Paulheim - arXiv preprint arXiv:2103.05110, 2021

… Lehmberg et al. [17] conduct web table classification in order to create a large open corpus of relational HTML tables, ie, the WebDataCommons HTML Ta- bles Dataset3 based on the 2012 version of the Common Crawl web corpus …


WeChat Neural Machine Translation Systems for WMT21

X Zeng, Y Liu, E Li, Q Ran, F Meng, P Li, J Xu, J Zhou - arXiv preprint arXiv …, 2021

… 4.2 Dataset For each language pair, the bilingual training data is the combination of all parallel data released by WMT21. For monolingual data, we filter out clean data from News Crawl, Common Crawl and Ex- tended Common Crawl …


What Can Unsupervised Machine Translation Contribute to High-Resource Language Pairs?

K Marchisio, M Freitag, D Grangier - arXiv preprint arXiv:2106.15818, 2021

… As bilingual training data, we use News Commentary v13, Europarl v7, Common Crawl and EU Press Release. We deduplicate and filter out pairs with > 250 to- kens in either language or length ratio over 1.5. This results in 5.2 million parallel sentence pairs …


What Does Your Bio Say? Inferring Twitter Users' Depression Status From Multimodal Profile Information Using Deep Learning

S Ghosh, A Ekbal, P Bhattacharyya - IEEE Transactions on Computational Social …, 2021

… Word embedding has been trained on the Common Crawl corpus (840 billion tokens). Padding arrays of zeros are done to make input sequences the same length. In our training set, we create an embedding matrix (E) that contains the embedding …


What is this cluster about? Explaining textual clusters by extracting relevant keywords

A Penta, A Pal - Knowledge-Based Systems, 2021


What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study

M Abdalla, K Vishnubhotla, SM Mohammad - arXiv preprint arXiv:2110.04845, 2021

The degree of semantic relatedness (or, closeness in meaning) of two units of language has long been considered fundamental to understanding meaning. Automatically determining relatedness has many applications such as question …


What Transformers Might Know About the Physical World: T5 and the Origins of Knowledge

H Shi, P Wolff - Proceedings of the Annual Meeting of the Cognitive …, 2021

… stack of decoders. All versions of T5 were trained on a cleaned version of the common crawl called the Colossal Cleaned Common Crawl (C4). The training corpus is over two times larger than Wikipedia. The largest version …


What Vision-Language ModelsSee'when they See Scenes

M Cafagna, K van Deemter, A Gatt - arXiv preprint arXiv:2109.07301, 2021

Page 1. What Vision-Language Models 'See' when they See Scenes Michele Cafagna1 Kees van Deemter2 Albert Gatt1,2 1University of Malta, Institute of Linguistics and Language Technology 2Universiteit Utrecht, Information …


What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus

A Sasha Luccioni, JD Viviano - arXiv e-prints, 2021

AS Luccioni, JD Viviano - arXiv preprint arXiv:2105.02732, 2021

Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this …


When FastText Pays Attention

V Novotný, M Štefánik, EF Ayetiran, P Sojka - 2021

… (Hyper)-Parameter Optimization 3. Experiments & 4. Results I 3.3. (Hyper-)Parameter Optimization Dataset Number of tokens 2017 English Wikipedia 2,423,655,228 English Common Crawl 823,575,128,431 Table: Our datasets and their sizes in tokens …


When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

V Novotný, M Štefánik, EF Ayetiran, P Sojka - arXiv preprint arXiv:2104.09691, 2021

… For qualitative evaluation and performance estimation, we use the deduplicated English Common Crawl4 as the training corpus. See the statistics in Table 1. Dataset Number of tokens 2017 English Wikipedia 2,423,655,228 English Common Crawl 823,575,128,431 …


When few-shot learning meets large-scale knowledge-enhanced pre-training: Alibaba at FewCLUE

Z Xu, C Wang, P Li, Y Li, M Wang, B Hou, M Qiu… - … International Conference on …, 2021

… In the pre-training steps, we use CLUECorpus2020 [28], which contains 100 GB Chinese raw texts retrieved from Common Crawl. We also employ 100 million high-quality Chinese knowledge SPOs from our in-house KG as the structured knowledge source …


When in Doubt, Summon the Titans: Efficient Inference with Large Models

AS Rawat, M Zaheer, AK Menon, A Ahmed, S Kumar - arXiv preprint arXiv …, 2021

Scaling neural networks to "large" sizes, with billions of parameters, has been shown to yield impressive results on many challenging problems. However, the inference cost incurred by such large models often prevents their application in most …


When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer

A Deshpande, P Talukdar, K Narasimhan - arXiv preprint arXiv:2110.14782, 2021

… (same) which uses different splits of Wikipedia data (hence, same domain), and (2) non-parallel (diff) which uses Wikipedia data for the original and common crawl data (web text) for the derived language (hence, different domain). We use the …


Which Apple Keeps Which Doctor Away Colorful Word Representations with Visual Oracles

Z Zhang, H Yu, H Zhao, M Utiyama - IEEE/ACM Transactions on Audio, Speech, and …, 2021

Recent pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally provide more effective contextualized word …


Which is Better for Deep Learning: Python or MATLAB? Answering Comparative Questions in Natural Language

V Chekalina, A Bondarenko, C Biemann, M Beloucif…

… with respect to an aspect specified by the user. First, using the Elasticsearch BM25, CAM retrieves sentences containing the two compared objects and the comparison aspect from the Common Crawl-based corpus featuring …


Page 1. Which model is helpful in solving privacy, memorization, and bias problems? Soyoung Yoon * 1 Hyunji Lee * 1 Abstract Recently, language models of huge size are achieving stunning performance on natural language understanding tasks …


Who Blames or Endorses Whom? Entity-to-Entity Directed Sentiment Extraction in News Text

K Park, Z Pan, J Joo - arXiv preprint arXiv:2106.01033, 2021

… For each of the target media outlets, we collected news articles shared throughout 2020 until September from the Common Crawl corpus, which has been collecting web data since 20088. The total number of retrieved …


Who is Accountable for Data Bias?

… The OpenAI researchers use the data from Common Crawl, which provides a corpus for collaborative research, analysis, and education. It is an open data set containing petabytes of data collected since 2008 from raw web …


Why Do Masked Neural Language Models Still Need Semantic Knowledge in Question Answering?

… Lastly, RoBERTa is pre-trained with much larger data. Especially, the data is about 160GB, which is composed of CommonCrawl News dataset [40], Open WebText corpus [41], and STORIES corpus [42] in addition to the pre-training data of BERT …


Wide and Deep Learning in Crowdfunding Success Prediction

The thesis studies to which extent deep learning models can predict whether crowdfunding campaigns will be successfully funded by drawing upon a dataset of 246,891 projects and 40 structured and text features from the crowdfunding platform …


with Graph Neural Networks

… example “pen” and “pencil”, or “salt” and “pepper”. We load word embeddings from a FastText 155 model [18], pre-trained on the Common Crawl dataset of over 600 billion tokens. Names of objects 156 are converted to word …


WITS: Wikipedia for Italian Text Summarization

Abstractive text summarization has recently improved its performance due to the use of sequence to sequence models. However, while these models are extremely data-hungry, datasets in languages other than English are few. In this work, we introduce WITS (Wikipedia …


Page 1. IN DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS , STOCKHOLM SWEDEN 2020 Word embeddings for the classification of main conditions with samples of patient text describing symptoms …


Word embeddings reveal social group attitudes and stereotypes in large language corpora

… Surprisingly, less than half of the 45 languages studied by DeFranza and colleagues revealed a male-good/female-bad2 association across data from Wikipedia and the Common Crawl. This … For example, in the Common Crawl, 67% of languages with …


Word Vector Representations using Shallow Neural Networks

… We obtain better performance in both languages on the downstream task with far smaller training data, compared to recently released, common crawl versions; and character n-grams appear useful for Swedish, a morphologically rich language. Keywords …


Word-level Korean-English Quality Estimation

S Eo, C Park, J Seo, H Moon, H Lim - Annual Conference on Human and Language …, 2021

Abstract 기계번역 품질 예측 (Quality Estimation, QE) 은 정답 문장에 대한 참조없이 소스 문장과 기계번역 결과를 통해 기계번역 결과에 대한 품질을 수준별 주석으로 나타내주는 태스크이며, 다양한 활용도가 있다는 점에서 꾸준히 연구가 수행되고 있다 …


Words as a window: Using word embeddings to explore the learned representations of Convolutional Neural Networks

D Dharmaretnam, C Foster, A Fyshe - Neural Networks, 2021

… here. Elmo is trained on a random sample of Wikipedia and the common crawl, and we use an average of the three 512-dimensional output layers … 2016). The Lexvec model has 300 dimensions and is trained on the common crawl …


WTR: A Test Collection for Web Table Retrieval

Z Chen, S Zhang, BD Davison - arXiv e-prints, 2021

… PA, USA ABSTRACT We describe the development, characteristics and availability of a test collection for the task of Web table retrieval, which uses a largescale Web Table Corpora extracted from the Common Crawl. Since …


XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Z Chi, S Huang, L Dong, S Ma, S Singhal, P Bajaj… - arXiv preprint arXiv …, 2021

… different languages. 4 Experiments 4.1 Setup Data We use the CC-100 (Conneau et al., 2020) dataset for the replaced token detection task. CC- 100 contains texts in 100 languages collected from the CommonCrawl dump. We use …


XLM-K: Improving Cross-Lingual Language Model Pre-Training with Multilingual Knowledge

X Jiang, Y Liang, W Chen, N Duan - arXiv preprint arXiv:2109.12573, 2021

… 4.1 Implementation Details Data and Model Structure For the multilingual masked language modeling task, we use Common Crawl dataset (Wenzek et al. 2020) … 2020) was trained on Common Crawl dataset (Wenzek et al. 2020), which covers our training data Wikipedia …


XLM-T: A Multilingual Language Model Toolkit for Twitter

F Barbieri, LE Anke, J Camacho-Collados - arXiv preprint arXiv:2104.12250, 2021

… 2.1). For the monolingual experiments we also include a FastText (FT) baseline (Joulin et al., 2017), which relies on monolingual FT embeddings trained on Common Crawl and Wikipedia (Grave et al., 2018) as initialization for each language lookup table …


XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders

S Ma, J Yang, H Huang, Z Chi, L Dong, D Zhang… - arXiv preprint arXiv …, 2020

… cross-lingual representations. In this work, we adopt XLM-R BASE (Conneau et al., 2020) as the pretrained encoder. It was trained in 100 languages, using more than two terabytes of filtered CommonCrawl data. XLM-R is …


XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

A Babu, C Wang, A Tjandra, K Lakhotia, Q Xu, N Goyal… - arXiv preprint arXiv …, 2021

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 …


XRR: Explainable Risk Ranking for Financial Reports

TW Lin, RY Sun, HL Chang, CJ Wang, MF Tsai

… permutation test with p < 0.05. Table 1: Performance comparison 3. GloVe [20] representations are 300-dimensional word vectors6 trained on 840 billion tokens of Common Crawl data. In the following experiments, we denote …


XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

S Ruder, N Constant, J Botha, A Siddhant, O Firat, J Fu… - arXiv preprint arXiv …, 2021

… Rebalanced mBERT (RemBERT; Chung et al., 2021) is a more efficient, scaled-up reparameterization of mBERT. These models have been pre-trained on unlabeled data in around 100 languages from Wikipedia …


XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

S Mukherjee, AH Awadallah, J Gao - arXiv preprint arXiv:2106.04563, 2021

… To address these issues, we ex- plore techniques to automatically generate largescale task-specific transfer sets leveraging a very large bank of web sentences from Common Crawl in Section 4.1. 3 Distillation Framework Overview …


You too Brutus! Trapping Hateful Users in Social Media: Challenges, Solutions & Insights

M Das, P Saha, R Dutt, P Goyal, A Mukherjee… - arXiv preprint arXiv …, 2021

… We experiment with the following text-classifiers. fastText: We leverage the pre-trained fastText embedding by Grave et al. [14] which has been trained on Common Crawl and Wikipedia. Each user's document has been represented as a 100 dimensional vector …


Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

S Wu, X Zhao, T Yu, R Zhang, C Shen, H Liu, F Li… - arXiv preprint arXiv …, 2021

… c) After this step, the data size of Common Crawl is decreased from 866,304GB to 12,200GB. • Traditional to Simplified Chinese: Converts traditional Chinese characters to simplified Chinese characters. • Sensitive Words Filtering: Removes …


Zero-Resource Multi-Dialectal Arabic Natural Language Understanding

… XLM-R is a cross-lingual model, and we choose it since it is reported to perform better than mBERT, the multilingual model from Google [5]. XLM-R also uses Common Crawl for training, which is more likely to have …


Zeroshot Crosslingual Transfer of a Gloss Language Model for Semantic Change Detection

… finetuned. data with MLM (Masked Language Modeling) objective. But in contrast to BERT and RoBERTa, it is pretrained not on monolingual data but on 2.5 TB of texts from CommonCrawl in 100 languages. 3 System overview …


ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

KK Maurya, MS Desarkar, Y Kano, K Deepshikha - arXiv preprint arXiv:2106.01597, 2021

… languages. It is a transformer-based sequence- to-sequence pre-trained model. The model is trained on monolingual data in many languages from Wikipedia Common Crawl corpus with BART language model objective. Particularly …