Rectangle 27 0

With the help of Ben's superior document reading skills, I believe this is possible using the posterior() function.

Nice work! test.topics[[2]] is the matrix with topics as cols, new documents as rows and cell values as posterior probabilities.

r - Predicting LDA topics for new data - Stack Overflow

r lda topic-modeling
Rectangle 27 0

I had recently a similar project, usually, at least some of this steps are done:

  • Stop-words removal: you can easily do this via the removeWords(your corpus, stopwords("english")) from the tm package. Further you can construct your own stop word list and remove it via the same function.
  • usually you also numbers and punctuation's (see tm package) are removed.
  • also very common is stemming (see Wikipedia for an explanation) and removing sparse terms, this helps to reduce the dimension of your term document matrix with only little loss of information (both in tm and RWeka package).
  • some people also like to work only with nouns/proper nouns or noun phrases. See here for an overview and some word lists and part of speech dictionaries you can be found on Kevin's Word List Page.
  • regarding splitting in paragraphs: this should be possible with the NgramTokenizer from Rweka package see tm package FAQ.
  • A nice article about pre-processing in general can be found here or more scientific here.
  • One more example of R + topic models can be found Ponweiser 2012

I learned that text mining is a bit different. Things which improved results in one case do not work in an other case. It is a lot of testing which parameters and which pre-processing steps improve your results...So have fun!

That's terrific advice and I'll put it to use. Much appreciated! Appears I don't have the reputation to upvote you guys yet, but oh well, thank you!

How to go about data preparation for topic modeling in R (topicmodels,...

r lda topic-modeling
Rectangle 27 0

There's no code in your question, so it's not really suitable for this site. That said, here are some comments that might be useful. If you supply code you'll get more specific and useful answers.

  • Yes. Breaking the text into chunks is common and advisable. Exact sizes are a matter of taste. It is often done within R, I've done it before making the corpus. You might also subset only nouns, like @holzben suggests. Here some code for cutting a corpus into chunks: corpus_chunk <- function(x, corpus, n) { # convert corpus to list of character vectors message("converting corpus to list of vectors...") listofwords <- vector("list", length(corpus)) for(i in 1:length(corpus)) { listofwords[[i]] <- corpus[[i]] } message("done") # divide each vector into chunks of n words # from f <- function(x) { y <- unlist(strsplit(x, " ")) ly <- length(y) split(y, gl(ly%/%n+1, n, ly)) } message("splitting documents into chunks...") listofnwords1 <- sapply(listofwords, f) listofnwords2 <- unlist(listofnwords1, recursive = FALSE) message("done") # append IDs to list items so we can get bibliographic data for each chunk lengths <- sapply(1:length(listofwords), function(i) length(listofnwords1[[i]])) names(listofnwords2) <- unlist(lapply(1:length(lengths), function(i) rep(x$bibliodata$x[i], lengths[i]))) names(listofnwords2) <- paste0(names(listofnwords2), "_", unlist(lapply(lengths, function(x) seq(1:x)))) return(listofnwords2) }

Yes, you might make a start with some code and then come back with a more specific question. That's how you'll get the most out of this site.

For a basic introduction to text mining and topic modelling, see Matthew Jockers' book Text Analysis with R for Students of Literature

If you're already a little familiar with MALLET, then try rmallet for topic modelling. There's lots of code snippets on the web that use this, here's one of mine.

Thank you for your suggestions, Ben. Apologies for going off-topic with this. I should have realized that. Won't happen again.

No worries, your first two questions would be perfect for this forum if you just add a bit of code to show what you've already tried. Why not ask them separately when you've got some code ready (and you can code something that others here can reproduce)?

How to go about data preparation for topic modeling in R (topicmodels,...

r lda topic-modeling