On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words.
All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.
The only code released by the 'Paragraph Vector' paper authors was a small patch, from Mikolov, that added paragraph-vectors to the original `word2vec.c` implementation in a very simple way: treating the 1st token of each line as a special paragraph-vector, still string-named (and allocated in the same lookup dictionary). Only by convention (a special prefix on those paragraph-vector tokens) could collisions with similarly-named word-vectors avoided.
That's a nice minimal way to demo/test the idea, but limited and fragile in other ways. The initial gensim implementation did something similar, then I changed it to use a separate doc-vectors space, to better support a lot of options (including the PV-DM mode with a concatenative input layer – which has never been confirmed to perform as well as the original paper implied).
Embeddings are shallow, and what comes after them is usually less than 5 layers of LSTM, not a deeeeeep neural net (maybe, deep only horizontally, on the axis of words).
All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.