statistics tools

Statistics::GaussHelmert is a general weighted least squares estimation module.

SYNOPSIS

use Statistics::GaussHelmert;

# create an empty model
my $estimation = new Statistics::GaussHelmert;

# setup the model given observations $y, covariance matrices
# $Sigma_yy, an initial guess $b0 for the unknown parameters.
$estimation->observations($y);
$estimation->covariance_observations($Sigma_yy);
$estimation->initial_guess($b0);

# specify the implicit model function and its Jacobians by using
# closures.
$estimation->observation_equations(sub { ... });
$estimation->Jacobian_unknowns(sub { ... });
$estimation->Jacobian_observations(sub { ... });

# Maybe we want to impose some constraints on the unknown
# parameters, this is not mandatory
$estimation->constraints(sub { ... });
$estimation->Jacobian_constraints(sub { ... });

# start estimation
$estimation->start(verbose => 1);

# print result
print $estimation->estimated_unknown(),
$estimation->covariance_unknown();

This module is a flexible tool for estimating model parameters given a set of observations. The module provides function for a linear estimation model, the underlying model is called Gauss-Helmert model.

Statistics::GaussHelmert is different to modules such as Statistics::OLS in the sense that it may fit arbitrary functions to data of any dimensions. You have to specify an implicit minimization function (in contrast to explicit functions as in traditional regression methods) and its derivatives with respects to the unknown and the observations. You may also specify constraint function on the unknowns (with its derivative). Furthermore you already need an approximate solution. For some problems it is easy finding approximate solutions by directly solving for the unknown parameters with some well chosen observations.

The Text Statistics application is a simple tool that was designed to give users statistics about the input text. Just enter the text that you want and view how many words you have, the special characters number, space or line break number.

What do you know about your BLOB data?

BLOB Statistics is a tool for collecting, viewing and analyzing information about stored BLOB data. With BLOB Statistics, you can determine the number of images, office documents, sounds, videos (etc.) stored in your Database (or Warehouse), how much space is used, what is the minimum, average and maximum size of data, etc.

Here are some key features of "BLOB Statistics":

· Automatically determines database BLOB objects and creates statistics for them
· Recognizes most popular data formats -Generates statistics for the entire database, for each table, for data formats
· Produces reports with drill-down details
· Supports ODBC and desktop Databases
· BLOB Statistics supports the following reports:
· Database Summary Report - The Summary Report contains summary information about BLOBs for the entire database.
· Table Summary Report - The Table Summary Report contains summary information for each table.
· Format Summary Report - The Format Summary Report includes information on all recognized formats.
· Formats by Tables Report - This report allows you to determine which formats are used in each particular table.
· Tables by Formats Report - This report allows you to determine in which tables each particular format is used.
· Database: Recognized/Unrecognized Data - This report allows you to compare the properties of recognized and unrecognized data for the entire database.
· Tables: Recognized/Unrecognized Data - This report allows you to compare the properties of recognized and unrecognized data for each table.


Limitations:

· Adds DEMO strings to reports

Statistics::OLS is a Perl module to perform ordinary least squares and associated statistics.

SYNOPSIS

use Statistics::OLS;

my $ls = Statistics::OLS->new();

$ls->setData (@xydataset) or die( $ls->error() );
$ls->setData (@xdataset, @ydataset);

$ls->regress();

my ($intercept, $slope) = $ls->coefficients();
my $R_squared = $ls->rsq();
my ($tstat_intercept, $tstat_slope) = $ls->tstats();
my $sigma = $ls->sigma();
my $durbin_watson = $ls->dw();

my $sample_size = $ls->size();
my ($avX, $avY) = $ls->av();
my ($varX, $varY, $covXY) = $ls->var();
my ($xmin, $xmax, $ymin, $ymax) = $ls->minMax();

# returned arrays are x-y or y-only data
# depending on initial call to setData()
my @predictedYs = $ls->predicted();
my @residuals = $ls->residuals();

I wrote Statistics::OLS to perform Ordinary Least Squares (linear curve fitting) on two dimensional data: y = a + bx. The other simple statistical module I found on CPAN (Statistics::Descriptive) is designed for univariate analysis. It accomodates OLS, but somewhat inflexibly and without rich bivariate statistics. Nevertheless, it might make sense to fold OLS into that module or a supermodule someday.

Statistics::OLS computes the estimated slope and intercept of the regression line, their T-statistics, R squared, standard error of the regression and the Durbin-Watson statistic. It can also return the residuals.

It is pretty simple to do two dimensional least squares, but much harder to do multiple regression, so OLS is unlikely ever to work with multiple independent variables.

This is a beta code and has not been extensively tested. It has worked on a few published datasets. Feedback is welcome, particularly if you notice an error or try it with known results that are not reproduced correctly.

Web Statistics Montage is a free tool that compiles web statistics from multiple web sites and sends them to you in an easy-to-read, easy-to-compare format without having to log in anywhere! Just set up a simple configuration file, upload the files to your site and youre done.

Web Statistics Montage gathers statistics like search engine spider activity, the PageRank of each of your sites, the unique visitors to your site over the last 3 months, what keywords were used to get to your site since that last report was generated, and much, much more!

Web Statistics Montage requires that you have a web hosting account that uses the cPanel control panel with AWStats installed.

MaxEntropy is a Perl5 module for Maximum Entropy Modeling and Feature Induction.

SYNOPSIS

use Statistics::MaxEntropy;

# debugging messages; default 0
$Statistics::MaxEntropy::debug = 0;

# maximum number of iterations for IIS; default 100
$Statistics::MaxEntropy::NEWTON_max_it = 100;

# minimal distance between new and old x for Newtons method;
# default 0.001
$Statistics::MaxEntropy::NEWTON_min = 0.001;

# maximum number of iterations for Newtons method; default 100
$Statistics::MaxEntropy::KL_max_it = 100;

# minimal distance between new and old x; default 0.001
$Statistics::MaxEntropy::KL_min = 0.001;

# the size of Monte Carlo samples; default 1000
$Statistics::MaxEntropy::SAMPLE_size = 1000;

# creation of a new event space from an events file
$events = Statistics::MaxEntropy::new($file);

# Generalised Iterative Scaling, "corpus" means no sampling
$events->scale("corpus", "gis");

# Improved Iterative Scaling, "mc" means Monte Carlo sampling
$events->scale("mc", "iis");

# Feature Induction algorithm, also see Statistics::Candidates POD
$candidates = Statistics::Candidates->new($candidates_file);
$events->fi("iis", $candidates, $nr_to_add, "mc");

# writing new events, candidates, and parameters files
$events->write($some_other_file);
$events->write_parameters($file);
$events->write_parameters_with_names($file);

# dump/undump the event space to/from a file
$events->dump($file);
$events->undump($file);

This module is an implementation of the Generalised and Improved Iterative Scaling (GIS, IIS) algorithms and the Feature Induction (FI) algorithm as defined in (Darroch and Ratcliff 1972) and (Della Pietra et al. 1997). The purpose of the scaling algorithms is to find the maximum entropy distribution given a set of events and (optionally) an initial distribution.

Also a set of candidate features may be specified; then the FI algorithm may be applied to find and add the candidate feature(s) that give the largest `gain in terms of Kullback Leibler divergence when it is added to the current set of features.

Events are specified in terms of a set of feature functions (properties) f_1...f_k that map each event to {0,1}: an event is a string of bits. In addition of each event its frequency is given. We assume the event space to have a probability distribution that can be described by
The module requires the Bit::SparseVector module by Steffen Beyer and the Data::Dumper module by Gurusamy Sarathy. Both can be obtained from CPAN just like this module.

Smart Photo Statistics is a tool designed to gather several types of photographic statistics. The user can view statistics such as: camera model, focal length, flash used, sensitivity etc.

The data can be exported either as text or in a graphics format.

Statistics::LineFit module least squares line fit, weighted or unweighted.

SYNOPSIS

use Statistics::LineFit;
$lineFit = Statistics::LineFit->new();
$lineFit->setData (@xValues, @yValues) or die "Invalid data";
($intercept, $slope) = $lineFit->coefficients();
defined $intercept or die "Cant fit line if x values are all equal";
$rSquared = $lineFit->rSquared();
$meanSquaredError = $lineFit->meanSqError();
$durbinWatson = $lineFit->durbinWatson();
$sigma = $lineFit->sigma();
($tStatIntercept, $tStatSlope) = $lineFit->tStatistics();
@predictedYs = $lineFit->predictedYs();
@residuals = $lineFit->residuals();
(varianceIntercept, $varianceSlope) = $lineFit->varianceOfEstimates();

The Statistics::LineFit module does weighted or unweighted least-squares line fitting to two-dimensional data (y = a + b * x). (This is also called linear regression.) In addition to the slope and y-intercept, the module can return the square of the correlation coefficient (R squared), the Durbin-Watson statistic, the mean squared error, sigma, the t statistics, the variance of the estimates of the slope and y-intercept, the predicted y values and the residuals of the y values. (See the METHODS section for a description of these statistics.)
The module accepts input data in separate x and y arrays or a single 2-D array (an array of arrayrefs). The optional weights are input in a separate array. The module can optionally verify that the input data and weights are valid numbers. If weights are input, the line fit minimizes the weighted sum of the squared errors and the following statistics are weighted: the correlation coefficient, the Durbin-Watson statistic, the mean squared error, sigma and the t statistics.
The module is state-oriented and caches its results. Once you call the setData() method, you can call the other methods in any order or call a method several times without invoking redundant calculations. After calling setData(), you can modify the input data or weights without affecting the modules results.
The decision to use or not use weighting could be made using your a priori knowledge of the data or using supplemental data. If the data is sparse or contains non-random noise, weighting can degrade the solution. Weighting is a good option if some points are suspect or less relevant (e.g., older terms in a time series, points that are known to have more noise).

Fast Statistics description
Fast Statistics is a Statistical and Graphical Analysis program for MS Excel Fast Statistics is a Statistical and Graphical Analysis program for MS Excel, it can work as an Excel add-in and stand-alone software without excel installed, perform from basic data manipulations to the most advanced statistical analyses and produce sophisticated reports and charts. This is a ideal tool for Six Sigma, descriptive statistics, ANOVA, regression, dummy regression, intelligent data input, factor analysis, nonlinear regression, box plots, nonparametric tests, and correspondence analysis.

Fast Statistics includes more than 50 functions covering data and statistical-analysis requirements (preparing, describing, analyzing, and modeling data). Excel utilities have also been included to facilitate charting and data manipulation. Complementary modules are available.

Fast Statistic also gives professionals engaged in quality improvement the powerful , reliable, and easy to use tools they need to effectively analyze hard data while managing large and complex projects. It is the ideal package for Six Sigma and other quality improvement projects. From Statistical Process Control to Design of Experiments, it offers you the methods you need to implement every phase of your quality project.

Here are some key features of "Fast Statistics":

Easy to Use
· works as an Excel Add-ins
· Read and edit Excel 95/97/2000/XP files (do not need MS Excel installation)
· Intelligent data input system
· Comprehensive HTML Help
Data and File Management
· Export HTML Statistical and Graphical Analysis report
· Import and export: Excel, text, and other data formats
· Import Microsoft Excel workbook sheets individually
· support import currency formats
· Data manipulation
· Search and replace in Data window
· support Double-precision worksheets
Simulation and Distributions
· Random number generator
· Density, distribution, and inverse cumulative distribution functions
Statistics
· Basic statistics: Display descriptive statistic, Correlation, Covariance
· Hypothesis test: Sign test, Wilcoxon rank sum test, Z test, T test, Chi squared test, F test
· Analysis of Variance(ANOVA): One way ANOVA, Two way ANOVA, Interaction plot, Main effect plot
· Regression: Multiple linear regression, Dummy-variable regression, Non-linear regression
· Nonparametrics : Friedman, Kruskal-Wallis
· Goodness of Fit
· Time Series: Trend analysis, Moving average, Single exponential smoothing, Double exponential smoothing, Decomposition, Winters Method, Autocorrelation, Autocovariance, Partial autocorrelation
· DOE: Full factorial design, Fractional factorial design, Analyze factorial design, Factorial plot
Graphics
· Pictorial gallery and streamlined dialog boxes simplify graph creation
· Interactively edit attributes (axes, scale, etc.) and recreate custom graphs with new data
· Intuitive tool to place multiple graphs on one page
· Set your own preferences for graph attribute defaults
· support Rotating, Scrolling and Moving 3D data graphs
· Generate Animation chart
· Basic Graph: Line, HorizLine, Area, Point(Scatter), Bar, HorizBar, Pie, Shapes, Stairs Line, Arrows, Bubble
· Statistical Graph: Histogram, Box plot, Contour, Time series plot, 3D plot, Probability plot
· Quality Control Charts: X chart, R chart, S chart, P chart, NP chart, C chart, U chart, Pareto chart, EWMA
· Capability analysis