indexing

PDL::Indexing Perl module contains a tutorial on how to index piddles.

This manpage should serve as a first tutorial on the indexing and threading features of PDL.

This manpage is still in alpha development and not yet complete. "Meta" comments that point out deficiencies/omissions of this document will be surrounded by square brackets ([]), e.g. [ Hopefully I will be able to remove this paragraph at some time in the future ]. Furthermore, it is possible that there are errors in the code examples. Please report any errors to Christian Soeller (c.soeller@auckland.ac.nz).

Still to be done are (please bear with us and/or ask on the mailing list, see PDL::FAQ):

document perl level threading
threadids
update and correct description of slice
new functions in slice.pd (affine, lag, splitdim)
reworking of paragraph on explicit threading

Indexing and threading with PDL

A lot of the flexibility and power of PDL relies on the indexing and looping features of the perl extension. Indexing allows access to the data of a pdl object in a very flexible way. Threading provides efficient implicit looping functionality (since the loops are implemented as optimized C code).

Pdl objects (later often called "pdls") are perl objects that represent multidimensional arrays and operations on those. In contrast to simple perl @x style lists the array data is compactly stored in a single block of memory thus taking up a lot less memory and enabling use of fast C code to implement operations (e.g. addition, etc) on pdls.

pdls can have children

Central to many of the indexing capabilities of PDL are the relation of "parent" and "child" between pdls. Many of the indexing commands create a new pdl from an existing pdl. The new pdl is the "child" and the old one is the "parent". The data of the new pdl is defined by a transformation that specifies how to generate (compute) its data from the parents data. The relation between the child pdl and its parent are often bidirectional, meaning that changes in the childs data are propagated back to the parent. (Note: You see, we are aiming in our terminology already towards the new dataflow features. The kind of dataflow that is used by the indexing commands (about which you will learn in a minute) is always in operation, not only when you have explicitly switched on dataflow in your pdl by saying $a->doflow. For further information about data flow check the dataflow manpage.)

Another way to interpret the pdls created by our indexing commands is to view them as a kind of intelligent pointer that points back to some portion or all of its parents data. Therefore, it is not surprising that the parents data (or a portion of it) changes when manipulated through this "pointer". After these introductory remarks that hopefully prepared you for what is coming (rather than confuse you too much) we are going to dive right in and start with a description of the indexing commands and some typical examples how they might be used in PDL programs. We will further illustrate the pointer/dataflow analogies in the context of some of the examples later on.

There are two different implementations of this ``smart pointer relationship: the first one, which is a little slower but works for any transformation is simply to do the transformation forwards and backwards as necessary. The other is to consider the child piddle a ``virtual piddle, which only stores a pointer to the parent and access information so that routines which use the child piddle actually directly access the data in the parent. If the virtual piddle is given to a routine which cannot use it, PDL transparently physicalizes the virtual piddle before letting the routine use it.

Currently (1.94_01) all transformations which are ``affine, i.e. the indices of the data item in the parent piddle are determined by a linear transformation (+ constant) from the indices of the child piddle result in virtual piddles. All other indexing routines (e.g. ->index(...)) result in physical piddles. All routines compiled by PP can accept affine piddles (except those routines that pass pointers to external library functions).

Note that whether something is affine or not does not affect the semantics of what you do in any way: both

$a->index(...) .= 5;
$a->slice(...) .= 5;

change the data in $a. The affinity does, however, have a significant impact on memory usage and performance.

SPyDI Is a powerful engine to create distributed full text indexing systems and distributed search engines.

Simple Python Distributed Indexing library supports harvesting, crawling (pull mehtods), and push methods (via a Web interface or SPyRO Web services).

It supports boolean and vector Information retrieval models. It has few dependencies, and comes with its own HTTP server and HTML embedded pages language (called pyew and wey pages), and session manager.

It can use the SMTP of the Python library. It supports replacing the default modules with some better modules (Apache, exim, etc).

Whats New in This Release:

· Monarca updates to support SPyROs new HTTP protocol management.
· Some bugfixes in pyew pages.
· General code cleanup.

RunIt! enables you to add files, websites and programs to your RunIt! index, all according to keywords you selected, so it is easy for you to remember how to find that file, then with a press of hotkeys (also assigned by you), youre able to run it or perform a websearch in one of the following search engines: Google, Yahoo! or MSN. RunIt! also allows you to index your related files, simply by adding a common keyword to all of them. For example, you can add the keyword "games" to every game you add to the index, thus when you type "games" in the run program module, you will recieve a list of all games added to index. RunIt! also integrates itself to your system shell, so that you can add items to your index simply by right-clicking them and choosing "Add to RunIt!" from your right-click menu.

ruby index/search is a general indexing framework for ruby. With it, you can create collections of documents, then index and search them. Currently, both inverted indexing and LSA indexing are supported, with rudimentary result clustering in the works.

The indices may be marshalled out (with Marshal.dump()), then pulled back in with Marshal.load(), to allow for relatively quick search operations in both LSA and inverted indices.

Contentment::Index is a Perl module that provides categorization and indexing features.

SYNOPSIS

# Get a list of the available indexes
my @indexes = Contentment::Index->indexes;

for my $index (@indexes) {

# Get a list of avilable terms
my @terms = $index->terms;

for my $term (@terms) {

# Get a list of generators
my @generators = $index->generators;

}

}

One frequently controversial component of a CMS is the categorization system. Contentment attempts to avoid this problem by providing a framework for building categorization systems so that any controversy just leads to the replacement of whatever system someone doesnt like.

That is, rather than create some all encompassing categorization system or some dead-simple one that your grandma who thinks the boogey-man is going to jump out of the her CD-ROM drive can use, you can have either or both or neither depending upon your need. Thus, instead of providing a category or taxonomy system, Contentment provides the indexing system, which allows you to implement whatever category system you prefer.

Bio::Index::Swissprot is a Perl Interface for indexing (multiple) Swissprot .dat files (ie flat file swissprot format).

SYNOPSIS

# Complete code for making an index for several
# Swissprot files
use Bio::Index::Swissprot;
use strict;

my $Index_File_Name = shift;
my $inx = Bio::Index::Swissprot->new(-filename => $Index_File_Name,
-write_flag => WRITE);
$inx->make_index(@ARGV);

# Print out several sequences present in the index
# in gcg format
use Bio::Index::Swissprot;
use Bio::SeqIO;
use strict;

my $out = Bio::SeqIO->new( -format => gcg, -fh => *STDOUT );
my $Index_File_Name = shift;
my $inx = Bio::Index::Swissprot->new(-filename => $Index_File_Name);

foreach my $id (@ARGV) {
my $seq = $inx->fetch($id); # Returns Bio::Seq object
$out->write_seq($seq);
}

# alternatively
my ($id, $acc);
my $seq1 = $inx->get_Seq_by_id($id);
my $seq2 = $inx->get_Seq_by_acc($acc);

Inherits functions for managing dbm files from Bio::Index::Abstract.pm, and provides the basic funtionallity for indexing Swissprot files, and retrieving the sequence from them. Heavily snaffled from James Gilberts Fasta system. Note: for best results use strict.

Details on configuration and additional example code are available in the biodatabases.pod file.

Index & Shutdown tells Sherlock to Index your hard drives and then shut down.

You can run it at the end of each day when you are ready to shut down. Your drive(s) need to be indexed periodically for Sherlocks "Find by Content" to work properly. Before you can search the contents of a disks files, you must create an index of the disk. If you have modified files on the disk since it was last indexed and you want to search the modified content, you need to update the index. The only way to be sure that find by content will always work is to index your drives each day. You could set up a schedule in Sherlock to index the drives each day at a specified time but that slows down your system while you are trying to use it.
Index & Shutdown avoids that problem by indexing your drives during shutdown.

NOTE : Does work under OS X as a classic app.

Whats New in This Release:

· Fix - Works as classic app with OS X. Previous versions would incorrectly report the OS version and would not run.
· Note: May still not work with beta version of OS X but should work with final version.

FTP Index provides a FTP indexer and search engine.

FTP Index is a search engine for FTP servers. It scans servers for definable filetypes and stores the results in a MySQL database.

It utilizes the ftpls tool from the ftpcopy package for indexing the servers.

It scans multiple servers at the same time by running with multiple processes.

Whats New in This Release:

· fixed a nasty bug that caused the loss of the half all found files.

Creates indexed pdf documents from text files. Designed to aid creating an electronic distribution method for legacy system reports, since many mainframe type print spools are plain text.

Allows indexing, customizing page settings, font size, font face, and super-imposing text over an image in the case of using pre-printed forms. Supports unlimited levels of indexing bookmarks in documents and system/user configuration files.

Suitable for use in an intranet gateway for generating PDF documents in real-time.

Whats New in This Release:

· This fixes a bug for page breaking when the number of lines is a multiple of the lines per page, thanks to Carlo Benna