statistics

Statistics::OLS is a Perl module to perform ordinary least squares and associated statistics.

SYNOPSIS

use Statistics::OLS;

my $ls = Statistics::OLS->new();

$ls->setData (@xydataset) or die( $ls->error() );
$ls->setData (@xdataset, @ydataset);

$ls->regress();

my ($intercept, $slope) = $ls->coefficients();
my $R_squared = $ls->rsq();
my ($tstat_intercept, $tstat_slope) = $ls->tstats();
my $sigma = $ls->sigma();
my $durbin_watson = $ls->dw();

my $sample_size = $ls->size();
my ($avX, $avY) = $ls->av();
my ($varX, $varY, $covXY) = $ls->var();
my ($xmin, $xmax, $ymin, $ymax) = $ls->minMax();

# returned arrays are x-y or y-only data
# depending on initial call to setData()
my @predictedYs = $ls->predicted();
my @residuals = $ls->residuals();

I wrote Statistics::OLS to perform Ordinary Least Squares (linear curve fitting) on two dimensional data: y = a + bx. The other simple statistical module I found on CPAN (Statistics::Descriptive) is designed for univariate analysis. It accomodates OLS, but somewhat inflexibly and without rich bivariate statistics. Nevertheless, it might make sense to fold OLS into that module or a supermodule someday.

Statistics::OLS computes the estimated slope and intercept of the regression line, their T-statistics, R squared, standard error of the regression and the Durbin-Watson statistic. It can also return the residuals.

It is pretty simple to do two dimensional least squares, but much harder to do multiple regression, so OLS is unlikely ever to work with multiple independent variables.

This is a beta code and has not been extensively tested. It has worked on a few published datasets. Feedback is welcome, particularly if you notice an error or try it with known results that are not reproduced correctly.

MaxEntropy is a Perl5 module for Maximum Entropy Modeling and Feature Induction.

SYNOPSIS

use Statistics::MaxEntropy;

# debugging messages; default 0
$Statistics::MaxEntropy::debug = 0;

# maximum number of iterations for IIS; default 100
$Statistics::MaxEntropy::NEWTON_max_it = 100;

# minimal distance between new and old x for Newtons method;
# default 0.001
$Statistics::MaxEntropy::NEWTON_min = 0.001;

# maximum number of iterations for Newtons method; default 100
$Statistics::MaxEntropy::KL_max_it = 100;

# minimal distance between new and old x; default 0.001
$Statistics::MaxEntropy::KL_min = 0.001;

# the size of Monte Carlo samples; default 1000
$Statistics::MaxEntropy::SAMPLE_size = 1000;

# creation of a new event space from an events file
$events = Statistics::MaxEntropy::new($file);

# Generalised Iterative Scaling, "corpus" means no sampling
$events->scale("corpus", "gis");

# Improved Iterative Scaling, "mc" means Monte Carlo sampling
$events->scale("mc", "iis");

# Feature Induction algorithm, also see Statistics::Candidates POD
$candidates = Statistics::Candidates->new($candidates_file);
$events->fi("iis", $candidates, $nr_to_add, "mc");

# writing new events, candidates, and parameters files
$events->write($some_other_file);
$events->write_parameters($file);
$events->write_parameters_with_names($file);

# dump/undump the event space to/from a file
$events->dump($file);
$events->undump($file);

This module is an implementation of the Generalised and Improved Iterative Scaling (GIS, IIS) algorithms and the Feature Induction (FI) algorithm as defined in (Darroch and Ratcliff 1972) and (Della Pietra et al. 1997). The purpose of the scaling algorithms is to find the maximum entropy distribution given a set of events and (optionally) an initial distribution.

Also a set of candidate features may be specified; then the FI algorithm may be applied to find and add the candidate feature(s) that give the largest `gain in terms of Kullback Leibler divergence when it is added to the current set of features.

Events are specified in terms of a set of feature functions (properties) f_1...f_k that map each event to {0,1}: an event is a string of bits. In addition of each event its frequency is given. We assume the event space to have a probability distribution that can be described by
The module requires the Bit::SparseVector module by Steffen Beyer and the Data::Dumper module by Gurusamy Sarathy. Both can be obtained from CPAN just like this module.

Statistics::LineFit module least squares line fit, weighted or unweighted.

SYNOPSIS

use Statistics::LineFit;
$lineFit = Statistics::LineFit->new();
$lineFit->setData (@xValues, @yValues) or die "Invalid data";
($intercept, $slope) = $lineFit->coefficients();
defined $intercept or die "Cant fit line if x values are all equal";
$rSquared = $lineFit->rSquared();
$meanSquaredError = $lineFit->meanSqError();
$durbinWatson = $lineFit->durbinWatson();
$sigma = $lineFit->sigma();
($tStatIntercept, $tStatSlope) = $lineFit->tStatistics();
@predictedYs = $lineFit->predictedYs();
@residuals = $lineFit->residuals();
(varianceIntercept, $varianceSlope) = $lineFit->varianceOfEstimates();

The Statistics::LineFit module does weighted or unweighted least-squares line fitting to two-dimensional data (y = a + b * x). (This is also called linear regression.) In addition to the slope and y-intercept, the module can return the square of the correlation coefficient (R squared), the Durbin-Watson statistic, the mean squared error, sigma, the t statistics, the variance of the estimates of the slope and y-intercept, the predicted y values and the residuals of the y values. (See the METHODS section for a description of these statistics.)
The module accepts input data in separate x and y arrays or a single 2-D array (an array of arrayrefs). The optional weights are input in a separate array. The module can optionally verify that the input data and weights are valid numbers. If weights are input, the line fit minimizes the weighted sum of the squared errors and the following statistics are weighted: the correlation coefficient, the Durbin-Watson statistic, the mean squared error, sigma and the t statistics.
The module is state-oriented and caches its results. Once you call the setData() method, you can call the other methods in any order or call a method several times without invoking redundant calculations. After calling setData(), you can modify the input data or weights without affecting the modules results.
The decision to use or not use weighting could be made using your a priori knowledge of the data or using supplemental data. If the data is sparse or contains non-random noise, weighting can degrade the solution. Weighting is a good option if some points are suspect or less relevant (e.g., older terms in a time series, points that are known to have more noise).

Statistics::Hartigan is a Perl extension for the stopping rule proposed by Hartigan J. Hartigan, J. (1975). Clustering Algorithms. John Wiley and Sons, New York, NY, US.

SYNOPSIS

use Statistics::Hartigan;
&hartigan(InputFile, "agglo", 6, 10);

Input file is expected in the "dense" format -
Sample Input file:

6 5
1 1 0 0 1
1 0 0 0 0
1 1 0 0 1
1 1 0 0 1
1 0 0 0 1
1 1 0 0 1

Hartigan J. uses the Within Cluster/Group Sum of Squares (WGSS) to estimate the number of clusters a given data naturally falls into. The is goal is to minimize WG.

Statistics::GaussHelmert is a general weighted least squares estimation module.

SYNOPSIS

use Statistics::GaussHelmert;

# create an empty model
my $estimation = new Statistics::GaussHelmert;

# setup the model given observations $y, covariance matrices
# $Sigma_yy, an initial guess $b0 for the unknown parameters.
$estimation->observations($y);
$estimation->covariance_observations($Sigma_yy);
$estimation->initial_guess($b0);

# specify the implicit model function and its Jacobians by using
# closures.
$estimation->observation_equations(sub { ... });
$estimation->Jacobian_unknowns(sub { ... });
$estimation->Jacobian_observations(sub { ... });

# Maybe we want to impose some constraints on the unknown
# parameters, this is not mandatory
$estimation->constraints(sub { ... });
$estimation->Jacobian_constraints(sub { ... });

# start estimation
$estimation->start(verbose => 1);

# print result
print $estimation->estimated_unknown(),
$estimation->covariance_unknown();

This module is a flexible tool for estimating model parameters given a set of observations. The module provides function for a linear estimation model, the underlying model is called Gauss-Helmert model.

Statistics::GaussHelmert is different to modules such as Statistics::OLS in the sense that it may fit arbitrary functions to data of any dimensions. You have to specify an implicit minimization function (in contrast to explicit functions as in traditional regression methods) and its derivatives with respects to the unknown and the observations. You may also specify constraint function on the unknowns (with its derivative). Furthermore you already need an approximate solution. For some problems it is easy finding approximate solutions by directly solving for the unknown parameters with some well chosen observations.

Statistics::LTU is an implementation of Linear Threshold Units.

SYNOPSIS

use Statistics::LTU;

my $acr_ltu = new Statistics::LTU::ACR(3, 1); # 3 attributes, scaled

$ltu->train([1,3,2], $LTU_PLUS);
$ltu->train([-1,3,0], $LTU_MINUS);
...
print "LTU looks like this:n";
$ltu->print;

print "[1,5,2] is in class ";
if ($ltu->test([1,5,2]) > $LTU_THRESHOLD) { print "PLUS" }
else { print "MINUS" };

$ltu->save("ACR.saved") or die "Save failed!";
$ltu2 = restore Statistics::LTU("ACR.saved");

EXPORTS

For readability, LTU.pm exports three scalar constants: $LTU_PLUS (+1), $LTU_MINUS (-1) and $LTU_THRESHOLD (0).

Statistics::LTU defines methods for creating, destroying, training and testing Linear Threshold Units. A linear threshold unit is a 1-layer neural network, also called a perceptron. LTUs are used to learn classifications from examples.
An LTU learns to distinguish between two classes based on the data given to it. After training on a number of examples, the LTU can then be used to classify new (unseen) examples. Technically, LTUs learn to distinguish two classes by fitting a hyperplane between examples; if the examples have n features, the hyperplane will have n dimensions. In general, the LTUs weights will converge to a define the separating hyperplane.

The LTU.pm file defines an uninstantiable base class, LTU, and four other instantiable classes built on top of LTU. The four individual classes differs in the training rules used:

ACR - Absolute Correction Rule
TACR - Thermal Absolute Correction Rule (thermal annealing)
LMS - Least Mean Squares rule
RLS - Recursive Least Squares rule

Each of these training rules behaves somewhat differently. Exact details of how these work are beyond the scope of this document; see the additional documentation file (ltu.doc) for discussion.

Statistics::RankOrder contains algorithms for determining overall rankings from a panel of judges.

SYNOPSIS

use Statistics::RankOrder;

my $r = Statistics::RankOrder->new();

$r->add_judge( [qw( A B C )] );
$r->add_judge( [qw( A C B )] );
$r->add_judge( [qw( B A C )] );

my %ranks = $r->mean_rank;
my %ranks = $r->trimmed_mean_rank(1);
my %ranks = $r->median_rank;
my %ranks = $r->best_majority_rank;

This module offers algorithms for combining the rank-ordering of candidates by a panel of judges. For the purpose of this module, the term "candidates" means candidates in an election, brands in a taste-test, competitors in a sporting event, and so on. "Judges" means those rank-ordering the candidates, whether these are event judges, voters, etc. Unlike "voting" algorithms (e.g. majority-rule or single-transferable-vote), these algorithms require judges to rank-order all candidates. (Ties may be permissible for some algorithms).

Algorithms included are:

Lowest-Mean
Trimmed-Lowest-Mean
Median-Rank
Best-of-Majority

In this alpha version, there is minimal error checking. Future versions will have more robust error checking and may have additional ranking methods such as pair-ranking methods.

Statistics::GammaDistribution Perl module represents a gamma distribution.

SYNOPSIS

use Statistics::GammaDistribution;
my $g = Statistics::GammaDistribution->new();
$g->set_order(8.5);
print $g->rand(1.0);

my @alpha = (0.5,4.5,20.5,6.5,1.5,0.5);
my @theta = $g->dirichlet_dist(@alpha);

METHODS

$gamma = Statistics::GammaDistribution->new();

No parameters necessary.

$variate = $gamma->rand( SCALE );

This function returns a random variate from the gamma distribution. The distribution function is,

p(x) dx = {1 over Gamma(a) b^a} x^{a-1} e^{-x/b} dx
for x > 0.

Where a is the order and b is the scale. Unless supplied as a parameter, SCALE is assumed to be 1.0 if not supplied.
$gamma->get/set_order( ORDER );

Gets/sets the order of the distribution. Order must be greater than zero.

@theta = $gamma->dirichlet_dist( ALPHA );

Takes a K-sized array of real numbers (all greater than zero), and returns a K-sized array containing random variates from a Dirichlet distribution. The distribution function is

p(theta_1, ..., theta_K) dtheta_1 ... dtheta_K =
(1/Z) prod_{i=1}^K theta_i^{alpha_i - 1} delta(1 -sum_{i=1}^K theta_i) dtheta_1 ... dtheta_K

for theta_i >= 0 and alpha_i >= 0. The normalization factor Z is
Z = {prod_{i=1}^K Gamma(alpha_i)} / {Gamma( sum_{i=1}^K alpha_i)}

The random variates are generated by sampling K values from gamma distributions with parameters order=alpha_i, scale=1, and renormalizing. See A.M. Law, W.D. Kelton, Simulation Modeling and Analysis (1991).

RRDStats is a Coyote Linux and BrazilFW add-on package for network traffic monitoring, link quality control, and QOS classes monitoring.

RRD Statistics project is based on RRDtool for storing data to round robin databases, and a slightly modified RRDcgi for visualizing data through a Web interface.

Here are some key features of "RRD Statistics":

· Realtime graphical statistics for bandwidth usage and link quality
· Graphical statistics of QOS priority classes usage
· Historical data stored for one week

Configuration:

All default configuration is stored in /etc/rrd.config. This version supports web based configuration and there is no need to manual configuration for basic package functionality. Just install the packages and browse to your web administration interface (by default its http://192.168.0.1:8180). There should be new link at left menu labeled "RRDStats configuration"

There are some basic options you should set up to fit your configuration. First get sure, the RRDstats package is enabled (its the first option at configuration screen). After that should you set up your line speed (just some basic approximation is good enough). The last this you should set up is your internet gateway IP address. This IP address is used to measure your internet link latency and packet loss.

Ignore other configuration options for now, save your configuration and reboot router. After your system boots up, you can browse RRD statistics.

After system startup, package is initialiazed with /etc/rc.d/pkgs/rc.rrdstats. This file start another copy of tiny webserver which listens by default on port 8080. It reads its homepage files from /var/rrd/www/ directory. After webserver startup there are also started some data gathering threads.

They read transfered data from network interfaces, QOS classes and measure link latency. These values are then stored in RRD databases. RRD databases are by default stored in /var/rrd/data/ directory

For further information how RRD databases work, please visit their homepage. Simply said RRD database has constant size, it does not grow over time and stores average data over period of time.

Last component of RRDStats package are .cgi and template files which display data from RRD databases using web interface. As said before, these files and templates are stored in /var/rrd/www/ and its subdirectories.

Statistics::ChisqIndep is a Perl module to perform chi-square test of independence (a.k.a. contingency tables).

Synopsis

#example for Statistics::ChisqIndep
use strict;
use Statistics::ChisqIndep;
use POSIX;
# input data in the form of the array of array references
my @obs = ([15, 68, 83], [23,47,65]);
my $chi = new Statistics::ChisqIndep;
$chi->load_data(@obs);
# print the summary data along with the contingency table
$chi->print_summary();
#print the contingency table only
$chi->print_contingency_table();
#the following output is the same as calling the function of print_summary
#all of the detailed info such as the expected values, degree of freedoms
#and totals are accessible as object globals
#check if the load_data() call is successful
if($chi->{valid}) {
print "Rows: ", $chi->{rows}, "n";
print "Columns: ", $chi->{cols}, "n";
print "Degree of Freedom: ", $chi->{df}, "n";
print "Total Count: ", $chi->{total}, "n";
print "Chi-square Statistic: ",
$chi->{chisq_statistic}, "n";
print "p-value: ", $chi->{p_value}, "n";
print "Warning:
some of the cell counts might be too low.n"
if ($chi->{warning});
#output the contingency table
my $rows = $chi->{rows}; # # rows
my $cols = $chi->{cols}; # # columns
my $obs = $chi->{obs}; # observed values
my $exp = $chi->{expected}; # expected values
my $rtotals = $chi->{rtotals}; # row totals
my $ctotals = $chi->{ctotals}; #column totals
my $total = $chi->{total}; # total counts
for (my $j = 0; $j < $cols; $j++) {
print "t",$j + 1;
}
print "trtotaln";
for (my $i = 0; $i < $rows; $i ++) {
print $i + 1, "t";
for(my $j = 0 ; $j < $cols; $j ++) {
#observed values can be accessed
#in the following way
print $obs->[$i]->[$j], "t";
}
#row totals can be accessed
# in the following way
print $rtotals->[$i], "n";
print "t";
for(my $j = 0 ; $j < $cols; $j ++) {
#expected values can be accessed
#in the following way
printf "(%.2f)t", $exp->[$i]->[$j];
}
print "n";
}
print "ctotalt";
for (my $j = 0; $j < $cols; $j++) {
#column totals can be accessed in the following way
print $ctotals->[$j], "t";
}
#output total counts
print $total, "n";
}

This is the module to perform the Pearsons Chi-squared test on contingency tables of 2 dimensions. The users input the observed values in the table form and the module will compute the expected values for each cell based on the independence hypothesis. The module will then compute the chi-square statistic and the corresponding p-value based on the observed and the expected values to test if the 2 dimensions are truly independent.