From here: http://mccormickml.com/2016/04/12/googles-pretrained-word2vec-model-in-python/
12 Apr 2016
In this post I’m going to describe how to get Google’s pre-trained Word2Vec model up and running in Python to play with.
As an interface to word2vec, I decided to go with a Python package called gensim. gensim appears to be a popular NLP package, and has some nice documentation and tutorials, including for word2vec.
You can download Google’s pre-trained model here. It’s 1.5GB! It includes word vectors for a vocabulary of 3 million words and phrases that they trained on roughly 100 billion words from a Google News dataset. The vector length is 300 features.
Loading this model using gensim is a piece of cake; you just need to pass in the path to the model file (update the path in the code below to wherever you’ve placed the file).
However, if you’re running 32-bit Python (like I was) you’re going to get a memory error!
This is because gensim allocates a big matrix to hold all of the word vectors, and if you do the math…
…that’s a big matrix!
Assuming you’ve got a 64-bit machine and a decent amount of RAM (I’ve got 16GB; maybe you could get away with 8GB?), your best bet is to switch to 64-bit Python. I had a little trouble with this–see my notes down at the end of the post.
Inspecting the Model
I have a small Python project on GitHub called inspect_word2vec that loads Google’s model, and inspects a few different properties of it.
If you’d like to browse the 3M word list in Google’s pre-trained model, you can just look at the text files in the vocabulary folder of that project. I split the word list across 50 files, and each text file contains 100,000 entries from the model. I split it up like this so your editor wouldn’t completely choke (hopefully) when you try to open them. The words are stored in their original order–I haven’t sorted the list alphabetically. I don’t know what determined the original order.
Here are some the questions I had about the vocabulary, which I answered in this project:
Does it include stop words?
Answer: Some stop words like “a”, “and”, “of” are excluded, but others like “the”, “also”, “should” are included.
Does it include misspellings of words?
Answer: Yes. For instance, it includes both “mispelled” and “misspelled”–the latter is the correct one.
Does it include commonly paired words?
Answer: Yes. For instance, it includes “Soviet_Union” and “New_York”.
Does it include numbers?
Answer: Not directly; e.g., you won’t find “100”. But it does include entries like “###MHz_DDR2_SDRAM” where I’m assuming the ‘#’ are intended to match any digit.
Here’s a selection of 30 “terms” from the vocabulary. Pretty weird stuff in there!
Al_Qods
Surendra_Pal
Leaflet
guitar_harmonica
Yeoval
Suhardi
VoATM
Streaming_Coverage
Vawda
Lisa_Vanderpump
Nevern
Saleema
Saleemi
rbracken@centredaily.com
yellow_wagtails
P_&C;
CHICOPEE_Mass._WWLP
Gardiners_Rd
Nevers
Stocks_Advance_Paced
IIT_alumnus
Popery
Kapumpa
fashionably_rumpled
WDTV_Live
ARTICLES_##V_##W
Yerga
Weegs
Paris_IPN_Euronext
##bFM_Audio_Simon
网友评论