--Get Tweets with Twitter API --Add words to Mecab --Use Word2Vec
I used the original ruby script.
# gem install twitter
require "twitter"
client = Twitter::REST::Client.new do |config|
config.consumer_key = ""
config.consumer_secret = ""
config.access_token = ""
config.access_token_secret = ""
end
@result = client.search("Science University").take(10000)
File.open("tus.csv", 'w') do |file|
@result.each do |tweet|
file.write(tweet.text.gsub(/(\s)/,""))
file.write("\n")
end
end
You will have a csv file. There are many articles about getting tokens, so I will omit it.
cd /usr/local/lib/mecab/dic
mkdir userdic
cd userdic
touch tus.csv
echo 'Science University,,,1,noun,General,*,*,*,*,Rikadai,Rikadai,Rikadai' >> tus.csv
#compile
/usr/local/Cellar/mecab/0.996/libexec/mecab/mecab-dict-index \
-d /usr/local/lib/mecab/dic/ipadic \
-u tus.dic \
-f utf-8 \
-t utf-8 tus.csv
#Press Enter to compile
reading tus.csv ... 1
emitting double-array: 100% |###########################################|
done!
#pass through path
vi /usr/local/etc/mecabrc
#Specify where the generated dic is
userdic = /usr/local/lib/mecab/dic/userdic/tus.dic
word2vec
# coding: UTF-8
import pandas as pd
import numpy as np
import MeCab
tweets = pd.read_csv('/Users/Hiroto/git/scripts/tus.csv').tweet
#Create a word-separated file
wakati = ""
for tweet in tweets:
mt = MeCab.Tagger("-Owakati")
wakati = wakati + mt.parse(tweet)
f = open('tus_wakati.txt', 'w')
f.write(wakati)
f.close()
# word2vec
from gensim.models import word2vec
data = word2vec.Text8Corpus('tus_wakati.txt')
model = word2vec.Word2Vec(data, size=100)
Similarity of the subject
out=model.most_similar(positive=[u'Science University'],topn= 100)
for x in out:
print(x[0],x[1])
| word | Degree of similarity |
|---|---|
| Ne | 0.9801737666130066 |
| U | 0.9679325222969055 |
| world | 0.9637500643730164 |
| inequality | 0.9604602456092834 |
| Yeah | 0.9603763818740845 |
| So | 0.9602923393249512 |
| is | 0.9574853181838989 |
| That kind of | 0.9568058252334595 |
| Lol | 0.9534944295883179 |
| darkness | 0.9462004899978638 |
| ! | 0.9435620307922363 |
| ? | 0.9433774948120117 |
| Raw | 0.942541241645813 |
| From | 0.9420970678329468 |
| Good | 0.9348764419555664 |
| Yo | 0.9348678588867188 |
| 。 | 0.9291704893112183 |
| Feeling | 0.929074764251709 |
| Me | 0.9288586378097534 |
| together | 0.9273968935012817 |
| 0.9265207052230835 | |
| Is | 0.9249017238616943 |
| Secret meeting | 0.9227114915847778 |
| Teru | 0.9216452836990356 |
| To go | 0.9207674264907837 |
| God | 0.9192628264427185 |
| Good luck | 0.918117880821228 |
| Ah ~ | 0.9180813431739807 |
| Disagreeable | 0.9164369106292725 |
| reason | 0.9164099097251892 |
| Waka | 0.9158462882041931 |
| Understood | 0.915264368057251 |
| ) | 0.913904070854187 |
| Is | 0.9111155867576599 |
| Delicious | 0.9105844497680664 |
| Nana | 0.9098367691040039 |
| Man | 0.909660816192627 |
| Shit | 0.9095121622085571 |
| so | 0.907973051071167 |
| If | 0.906628429889679 |
| meaning | 0.9065468311309814 |
| Sophia | 0.905195415019989 |
| Or | 0.9034873247146606 |
| Guy | 0.9014643430709839 |
| Go | 0.8999437689781189 |
| What | 0.8993074893951416 |
| Drink | 0.8984052538871765 |
| march | 0.8983776569366455 |
| Say | 0.8976813554763794 |
| Ta | 0.8964160680770874 |
| Often | 0.896243691444397 |
| eat | 0.8960259556770325 |
| want to see | 0.8957585096359253 |
| Child | 0.8946411609649658 |
| nice to meet you | 0.8943185806274414 |
| Want | 0.8941484689712524 |
| Stunning | 0.893967866897583 |
| zebra | 0.8935203552246094 |
| Too | 0.8934850692749023 |
| you | 0.8934849500656128 |
| illumination | 0.8927890062332153 |
| go | 0.8927274942398071 |
| Ichi | 0.8926646709442139 |
| Is | 0.8919773697853088 |
| arithmetic | 0.8915943503379822 |
| ( | 0.8915064930915833 |
| why | 0.8907312154769897 |
| Humanities | 0.8906354904174805 |
| Hmm | 0.8897289037704468 |
| - | 0.8896894454956055 |
| Yeah | 0.8896220922470093 |
| Department | 0.8895649313926697 |
| K | 0.8881763219833374 |
| Thoughts | 0.8881138563156128 |
| I don't know | 0.8880779147148132 |
| school | 0.8879990577697754 |
| But | 0.8878818154335022 |
| Incident | 0.8878498077392578 |
| Please | 0.8875197172164917 |
| Know | 0.8871732354164124 |
| Iwa | 0.8870071172714233 |
| Personality | 0.8869134187698364 |
| Hey | 0.8867558240890503 |
| Soukei | 0.8866025805473328 |
| I'd love to | 0.8860080242156982 |
| I wonder | 0.8857483267784119 |
| But | 0.8853344321250916 |
| Stop | 0.8850265145301819 |
| age | 0.8849031925201416 |
| k | 0.884624719619751 |
| which one | 0.8840593695640564 |
| Or | 0.8840340971946716 |
| Live | 0.883965253829956 |
| Discount | 0.8836942911148071 |
| By all means | 0.8836302757263184 |
| Crying | 0.8831743597984314 |
| yumalaonvae | 0.883036196231842 |
| o | 0.8830046653747559 |
| Note | 0.8829131126403809 |
| why | 0.8827589154243469 |
** Inequality **, ** Darkness ** is like science What are "secret meetings" and "zebras" ...
--It's not working well because you haven't removed the trash from your tweets (maybe) --The number of tweets acquired is small (1696 tweets this time) --Even if you use the ruby code for 10,000, you can only get 1696 tweets. ――I wanted you to come out with "proffesional" or "unit"
Recommended Posts