Distributed Representation

Features of fasttext

• Improved objective function
  • Consideration of negative samples
  • This should not affect training time.
• Change in optimization method
  • Use of stochastic optimization
  • If it affects the learning time, it should be this one
• Implementation in C language
  • This is the most effective, isn’t it?

If we implement it in ## pytorch, it won’t be much different from word2vec. It would depend on the amount of data to be trained.

After the distributed representation has been obtained, the After the distributed representation is obtained, machine learning can be used to classify it.

Models that can be used include

• Decision Tree
• SVM Support Vector Machine
• NN Neural Networks

and others.

SVM is included in NN in a broad sense.

In this section, we will use the decision tree method.

Bagging

• Image of majority voting with multiple decision trees
• Simple theory
  • Decision trees are highly explainable and are a classic machine learning model.
  • Computational load is light compared to deep learning
    • Depends on the size of the model
• Not much explainability
  • Do we want to analyze each of the multiple decision trees?

``py from sklearn.ensemble import BaggingClassifier from sklearn.tree import DecisionTreeClassifier

. BERT is a model that can be powerfully applied to natural language processing tasks.

However, it does not do a good job of capturing sentence-wise features.

Some claim that sentence features appear in [ CLS\ ], but This paper](https://arxiv.org/abs/1908.10084) claims that it does not contain that much useful information for the task.

Sentence BERT is a model that extends BERT to be able to obtain features per sentence.

The following are the steps to create Sentence BERT in Japanese.

• BART is a model for document summarization
• Derived from the same transformer as BERT
• Unlike BERT, it has an encoder-decoder structure
  • This is because it is intended for sentence generation

This page shows the steps to run a tutorial on BART.

Procedure

1. install transformers

Run ``sh pip install transformers

Run summary

2. Run the summary
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig

model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')

ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')

# Generate Summary
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])
```

On 2021/01/18, the output was MyMy friends.

Interesting.

## Where I got stuck.
Error when the version of pytorch is different from the one specified in transformers.

pip install -U torch

.

A vector of documents can be obtained using Universal Sentence Encoder.

Features

Supports multiple languages.

Japanese is supported.

Can handle Japanese sentences as vectors.

Usage

Clustering, similarity calculation, feature extraction.

Usage

Execute the following command as preparation.

pip install tensorflow tensorflow_hub tensorflow_text numpy   

Trained models are available.

See the python description below for details on how to use it.

import tensorflow_hub as hub  
import tensorflow_text
import numpy as np  
# for avoiding error  
import ssl  
ssl._create_default_https_context = ssl._create_unverified_context  

def cos_sim(v1, v2):  
   return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))  
  
embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")  
  
texts = ["I saw a comedy show yesterday." , "There was a comedy show on TV last night." , "I went to the park yesterday." , "I saw a comedy show last night.", "Yesterday, I went to the park."]  
vectors = embed(texts)  
```.

See the following link for more details

[Try Universal Sentence Encoder in Japanese](https://qiita.com/kenta1984/items/9613da23766a2578a27a)


### Postscript
```py
import tensorflow_text

Without this line, you will get an error like ``Sentencepiece not found! error. This line is not explicitly used in the sample source, but is required for the actual execution. This line is not explicitly used in the sample source, but is required in the actual runtime.

huggingface has released a Japanese model for BERT.

The Japanese model is included in transformers.

However, I stumbled over a few things before I could get it to actually work in a Mac environment, so I’ll leave a note.

Preliminaries: Installing mecab

The morphological analysis engine, mecab, is required to use BERT’s Japanese model.

The tokenizer will probably ask for mecab.

This time, we will use homebrew to install Mecab and ipadic.

Summary of what I’ve done with Fasttext to the document classification problem.

• Facebook research has published a document classification library using Fasttext.
• Fasttext is easy to install in a python environment.
• Run time is fast.

Preliminaries

I decided to tackle the task of document classification, and initially thought.

NeuralClassifier: An Open-source Neural Hierarchical Multi-label Text Classification Toolkit

NeuralClassifier: An Open-source Neural Hierarchical Multi-label Text Classification Toolkit. However, it was not very accurate.