Python, LDA : How to get the id of keywords instead of the keywords themselves with Gensim?

Discussion in 'Python' started by Amy21, Jan 20, 2017.

  1. Amy21


    Jan 20, 2017
    Likes Received:

    I am applying the LDA method using Gensim to extract keywords from documents. I can extract topics, and then assign these topics and key words associated to the documents.

    I would like to have the ids of these terms (or key words) instead of the terms themselves. I know that corpus extract a list of couples [(term_id, term_frequency) ...] of document i but I can't see how could I use this in my code to extract only the ids and assign it to my results.

    My code is as follows :

    Code (Text):
    ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=num_topics, id2word = dictionary, passes=passes, minimum_probability=0)

    # Assinging the topics to the document in corpus
    lda_corpus = ldamodel[corpus]

    # Find the threshold, let's set the threshold to be 1/#clusters,
    # To prove that the threshold is sane, we average the sum of all probabilities:
    scores = list(chain(*[[score for topic_id,score in topic] \
                         for topic in [doc for doc in lda_corpus]]))

    threshold = sum(scores)/len(scores)

    for t in range(len(topic_tuple)):

        key_words.append([topic_tuple[t][j][0] for j in range(num_words)])
        df_key_words = pd.DataFrame({'key_words' : key_words})

        documents_corpus.append([j for i,j in zip(lda_corpus,doc_set) if i[t][1] > threshold])
        df_documents_corpus = pd.DataFrame({'documents_corpus' : documents_corpus})

        documents_corpus_id.append([i for d,i in zip(lda_corpus, doc_set_id) if d[t][1] > threshold])
        df_documents_corpus_id = pd.DataFrame({'documents_corpus_id' : documents_corpus_id})

    result.append(pd.concat([df_key_words, df_documents_corpus, df_documents_corpus_id ], axis=1))
    Thank you in advance and ask me if more information are needed.
    Amy21, Jan 20, 2017
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.