### FudanNLP Verb-Object and Location Adverbial Rule Source: https://github.com/fudannlp/fnlp/blob/master/models/dict_dep.txt This rule defines a verb as a core word and illustrates complex dependency patterns involving locations. It shows how a location ('地名') can act as a determiner ('定语') for '附近' (nearby), which then forms a '的字结构' (de-structure) that further modifies another location acting as an object ('宾语') of the initial verb. ```FudanNLP Rule 0 * 动词 -1 核心词 1 * 地名 2 定语 2 附近 名词 3 的字结构 3 的 结构助词 4 定语 4 * 地名 0 宾语 ``` -------------------------------- ### FudanNLP Core Noun Phrase Rule Source: https://github.com/fudannlp/fnlp/blob/master/models/dict_dep.txt This rule identifies core words that are nouns, locations, entities, or organizations, and shows how the '的' particle can modify them, indicating a possessive or descriptive relationship (语态 - mood/aspect). ```FudanNLP Rule 0 * 名词|地名|实体名|机构名 -1 核心词 1 的 * 0 语态 ``` -------------------------------- ### Basic Usage of gensim LdaModel for Training and Inference Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Illustrates fundamental operations with the gensim LdaModel, including initial model training, inferring topic distributions for new documents, and updating the model with additional corpora. It also shows how to configure asymmetric alpha priors. ```Python >>> lda = LdaModel(corpus, num_topics=10) >>> doc_lda = lda[doc_bow] >>> lda.update(other_corpus) ``` ```Python >>> lda = LdaModel(corpus, num_topics=100) # train model >>> print(lda[doc_bow]) # get topic probability distribution for a document >>> lda.update(corpus2) # update the LDA model with additional documents >>> print(lda[doc_bow]) >>> lda = LdaModel(corpus, num_topics=50, alpha='auto', eval_every=5) # train asymmetric alpha from data ``` -------------------------------- ### APIDOC: LdaModel.show_topic method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC show_topic(topicid, topn=10) ``` -------------------------------- ### APIDOC: LdaModel.sync_state method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC sync_state() ``` -------------------------------- ### APIDOC: LdaState.get_Elogbeta method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC get_Elogbeta() ``` -------------------------------- ### gensim.models.ldamodel.LdaModel.do_mstep Method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Executes the M-step (maximization) of the online LDA algorithm, updating the topics using linear interpolation between the existing topics and collected sufficient statistics from another source. ```APIDOC do_mstep(rho, other) M step: use linear interpolation between the existing topics and collected sufficient statistics in other to update the topics. Parameters: rho: Learning rate parameter. other: Sufficient statistics collected from another source (e.g., from do_estep). ``` -------------------------------- ### APIDOC: LdaModel.print_topic method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC print_topic(topicid, topn=10) ``` -------------------------------- ### gensim.models.ldamodel.LdaModel Class Constructor Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Documents the constructor for the LdaModel class, used to estimate Latent Dirichlet Allocation model parameters. It details various parameters such as num_topics, id2word, alpha, eta, and distributed options, along with their purpose and default values. ```APIDOC class gensim.models.ldamodel.LdaModel(corpus=None, num_topics=100, id2word=None, distributed=False, chunksize=2000, passes=1, update_every=1, alpha='symmetric', eta=None, decay=0.5, eval_every=10, iterations=50, gamma_threshold=0.001) Bases: gensim.interfaces.TransformationABC The constructor estimates Latent Dirichlet Allocation model parameters based on a training corpus. If not given, the model is left untrained (presumably because you want to call update() manually). Parameters: corpus: An iterable training corpus. If not provided, the model is initialized but not trained. num_topics: The number of requested latent topics to be extracted from the training corpus. id2word: A mapping from word ids (integers) to words (strings). Used for vocabulary size, debugging, and topic printing. distributed: Boolean to force distributed computing. See gensim web tutorial for cluster setup. chunksize: Number of documents to be processed at a time during training. passes: Number of passes through the corpus during training. update_every: How often to update the model (e.g., after every 'update_every' chunks). alpha: Hyperparameter affecting sparsity of document-topic distributions. Defaults to symmetric 1.0/num_topics prior. Can be 'asymmetric' or 'auto'. eta: Hyperparameter affecting sparsity of topic-word distributions. Defaults to symmetric 1.0/num_topics prior. Can be a scalar or a matrix. decay: Weight of the previous model when updating (for online training). eval_every: Frequency of perplexity estimation logging (e.g., every 'eval_every' model updates). Set to None to disable. iterations: Number of iterations for inference per document. gamma_threshold: Threshold for gamma convergence. ``` -------------------------------- ### APIDOC: LdaState.blend2 method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Alternative, more simple blend. ```APIDOC blend2(rhot, other, targetsize=None) ``` -------------------------------- ### APIDOC: LdaModel.log_perplexity method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC log_perplexity(chunk, total_docs=None) ``` -------------------------------- ### APIDOC: LdaState.get_lambda method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt No description ```APIDOC get_lambda() ``` -------------------------------- ### APIDOC: LdaModel.show_topics method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt For num_topics number of topics, return num_words most significant words (10 words per topic, by default). The topics are returned as a list – a list of strings if formatted is True, or a list of (probability, word) 2-tuples if False. If log is True, also output this result to log. Unlike LSA, there is no natural ordering between the topics in LDA. The returned num_topics <= self.num_topics subset of all topics is therefore arbitrary and may change between two LDA training runs. ```APIDOC show_topics(num_topics=10, num_words=10, log=False, formatted=True) ``` -------------------------------- ### APIDOC: LdaState.reset method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Prepare the state for a new EM iteration (reset sufficient stats). ```APIDOC reset() ``` -------------------------------- ### gensim.models.ldamodel.LdaModel.do_estep Method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Performs the E-step (inference) on a chunk of documents, accumulating the collected sufficient statistics in a given state object or the model's internal state. This is a core part of the online LDA algorithm. ```APIDOC do_estep(chunk, state=None) Performs inference on a chunk of documents, and accumulates the collected sufficient statistics in state (or self.state if None). Parameters: chunk: A chunk of documents to process. state: An optional state object to accumulate statistics into. If None, uses the model's internal state. ``` -------------------------------- ### APIDOC: LdaModel.print_topics method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt For num_topics number of topics, return num_words most significant words (10 words per topic, by default). The topics are returned as a list – a list of strings if formatted is True, or a list of (probability, word) 2-tuples if False. If log is True, also output this result to log. Unlike LSA, there is no natural ordering between the topics in LDA. The returned num_topics <= self.num_topics subset of all topics is therefore arbitrary and may change between two LDA training runs. ```APIDOC print_topics(num_topics=10, num_words=10) ``` -------------------------------- ### APIDOC: LdaModel.save method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Save the model to file. Large internal arrays may be stored into separate files, with fname as prefix. ```APIDOC save(fname, *args, **kwargs) ``` -------------------------------- ### APIDOC: LdaModel.load method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Load a previously saved object from file (also see save). Large arrays are mmap’ed back as read-only (shared memory). ```APIDOC classmethod load(fname, *args, **kwargs) ``` -------------------------------- ### APIDOC: LdaState.load method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Load a previously saved object from file (also see save). If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. Default: don’t use mmap, load large arrays as normal objects. ```APIDOC classmethod load(fname, mmap=None) ``` -------------------------------- ### gensim.models.ldamodel.LdaModel.inference Method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Estimates the variational parameters (gamma) for each document in a given chunk, representing their topic weights. This function is read-only and can optionally collect sufficient statistics needed to update the model's topic-word distributions. ```APIDOC inference(chunk, collect_sstats=False) Given a chunk of sparse document vectors, estimate gamma (parameters controlling the topic weights) for each document in the chunk. This function does not modify the model (=is read-only aka const). The whole input chunk of document is assumed to fit in RAM; chunking of a large corpus must be done earlier in the pipeline. Parameters: chunk: A chunk of sparse document vectors. collect_sstats: If True, also collect sufficient statistics needed to update the model’s topic-word distributions. Returns: A 2-tuple (gamma, sstats). gamma is of shape len(chunk) x self.num_topics. sstats is None if collect_sstats is False. ``` -------------------------------- ### APIDOC: LdaState.save method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Save the object to file (also see load). If separately is None, automatically detect large numpy/scipy.sparse arrays in the object being stored, and store them into separate files. This avoids pickle memory errors and allows mmap’ing large arrays back on load efficiently. You can also set separately manually, in which case it must be a list of attribute names to be stored in separate files. The automatic check is not performed in this case. ignore is a set of attribute names to not serialize (file handles, caches etc). On subsequent load() these attributes will be set to None. ```APIDOC save(fname, separately=None, sep_limit=10485760, ignore=frozenset([])) ``` -------------------------------- ### APIDOC: gensim.models.ldamodel.LdaState class Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Encapsulate information for distributed computation of LdaModel objects. Objects of this class are sent over the network, so try to keep them lean to reduce traffic. ```APIDOC class gensim.models.ldamodel.LdaState(eta, shape) Bases: gensim.utils.SaveLoad ``` -------------------------------- ### APIDOC: LdaState.blend method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Given LdaState other, merge it with the current state. Stretch both to targetsize documents before merging, so that they are of comparable magnitude. Merging is done by average weighting: in the extremes, rhot=0.0 means other is completely ignored; rhot=1.0 means self is completely ignored. This procedure corresponds to the stochastic gradient update from Hoffman et al., algorithm 2 (eq. 14). ```APIDOC blend(rhot, other, targetsize=None) ``` -------------------------------- ### gensim.models.ldamodel.LdaModel.bound Method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Estimates the variational bound of documents from a given corpus, providing a measure of model fit. It can optionally use pre-inferred variational parameters (gamma). ```APIDOC bound(corpus, gamma=None, subsample_ratio=1.0) Estimates the variational bound of documents from corpus: E_q[log p(corpus)] - E_q[log q(corpus)] Parameters: corpus: The corpus of documents for which to estimate the bound. gamma: Optional variational parameters on topic weights for each corpus document (2D matrix). If not supplied, will be inferred from the model. subsample_ratio: Ratio of documents to subsample for bound estimation. ``` -------------------------------- ### APIDOC: LdaModel.update_alpha method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Update parameters for the Dirichlet prior on the per-document topic weights alpha given the last gammat. Uses Newton’s method, described in Huang: Maximum Likelihood Estimation of Dirichlet Distribution Parameters. (http://www.stanford.edu/~jhuang11/research/dirichlet/dirichlet.pdf) ```APIDOC update_alpha(gammat, rho) ``` -------------------------------- ### APIDOC: LdaState.merge method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Merge the result of an E step from one node with that of another node (summing up sufficient statistics). The merging is trivial and after merging all cluster nodes, we have the exact same result as if the computation was run on a single node (no approximation). ```APIDOC merge(other) ``` -------------------------------- ### APIDOC: LdaModel.update method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Train the model with new documents, by EM-iterating over corpus until the topics converge (or until the maximum number of allowed iterations is reached). corpus must be an iterable (repeatable stream of documents), In distributed mode, the E step is distributed over a cluster of machines. This update also supports updating an already trained model (self) with new documents from corpus; the two models are then merged in proportion to the number of old vs. new documents. This feature is still experimental for non-stationary input streams. For stationary input (no topic drift in new documents), on the other hand, this equals the online update of Hoffman et al. and is guaranteed to converge for any decay in (0.5, 1.0>. ```APIDOC update(corpus, chunksize=None, decay=None, passes=None, update_every=None, eval_every=None, iterations=None, gamma_threshold=None) ``` -------------------------------- ### gensim.models.ldamodel.LdaModel.clear Method Source: https://github.com/fudannlp/fnlp/blob/master/example-data/data-lda.txt Clears the internal state of the LDA model, primarily used in the distributed algorithm to free up memory. ```APIDOC clear() Clears model state (free up some memory). Used in the distributed algorithm. ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.