Solvable null model for the distribution of word frequencies

J F Fontanari1, L I Perlovsky

  • 1Instituto de Física de São Carlos, Universidade de São Paulo, Caixa Postal 369, 13560-970 São Carlos, São Paulo, Brazil.

Related Concept Videos

Construction of Frequency Distribution01:15

Construction of Frequency Distribution

A frequency distribution table can be constructed using the steps given below.
First, make a table with two columns—one with the title of the data that needs to be organized, and the other column for frequency. [Draw a third column for tally marks if needed]. Then, take a look at the items given in the data set and decide if an ungrouped frequency distribution table or a grouped frequency distribution table would be more suitable. If there are large sets of different values, then it is best to...
Expected Frequencies in Goodness-of-Fit Tests01:19

Expected Frequencies in Goodness-of-Fit Tests

A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).
Poisson Probability Distribution01:09

Poisson Probability Distribution

A Poisson probability distribution is a discrete probability distribution. It gives the probability of a number of events occurring in a fixed interval of time or space if these events happen at a known average rate and independently of the time since the last event. For example, a book editor might be interested in the number of words spelled incorrectly in a particular book. It might be that, on average, there are five words spelled incorrectly in 100 pages. The interval is 100 pages.
The...
Mean From a Frequency Distribution01:11

Mean From a Frequency Distribution

Sometimes, data gathered from an experiment on a large sample or population are organized into concise tables. In such cases, the frequency of the quantitative data set is plotted in the form of a table. Or else, the data values are grouped into the quantity’s intervals, which form classes, and their respective frequencies are known. That is, the data values are distributed over different categories or classes. This is known as frequency distribution.
When such a data set is encountered, the...
Relative Frequency Distribution00:55

Relative Frequency Distribution

A relative frequency distribution is the proportion or fraction of times a value occurs in a data set. To find the relative frequencies, one can divide each frequency by the total number of data points in the sample. It is very similar to a regular frequency distribution, except that instead of reporting how many data values fall in a class, a relative frequency distribution reports the fraction of data values that fall in a class. These fractions or proportions are called relative frequencies...
Determination of Expected Frequency01:08

Determination of Expected Frequency

Suppose one wants to test independence between the two variables of a contingency table. The values in the table constitute the observed frequencies of the dataset. But how does one determine the expected frequency of the dataset? One of the important assumptions is that the two variables are independent, which means the variables do not influence each other. For independent variables, the statistical probability of any event involving both variables is calculated by multiplying the individual...