trunk/doc/Search.pod

=head1 BackupPC search and archival extension

BackupPC search and archival extension is used to:

=over 4

=item *

find files in C<pool> by substring of filename

=item *

see state of one or all of your shares in some particular point in time
(file creation date on clients or backup date is supported as filter)

=item *

archive selected backups (per host, share and increment) to archival
medium (CD, DVD, tape etc.) with rich descriptions, archival locations,
multiple copies and so on.

=back

=head2 Requirements

This extension is based on PostgreSQL RDBMS and HyperEstraier full-text
search engine (and it's perl bindings).

If you are using Debian, you are in luck. All required packages are
allready part of C<unstable> distribution and can be installed with:

  # apt-get install postgresql
  # apt-get install hyperestraier libestraier-dev libqdbm-dev

Now you can skip to installation of L<HyperEstraier perl bindings> below.

If you don't have pre-packaged binaries for your installation, you will
need to install additional packages by hand.

=head3 PostgreSQL

You can use packages provided by your distribution or follow installation
instructions on PostgreSQL site.

=head3 QDBM

First, you need qdbm on which HyperEstraier depends. Installation is simple.

  $ tar xvfz qdbm-1.8.31.tar.gz 
  $ cd qdbm-1.8.31
  $ ./configure --enable-zlib
  $ make
  $ sudo make install

=head3 HyperEstraier

Also quite simple.

  $ tar xvfz hyperestraier-0.5.4.tar.gz
  $ cd hyperestraier-0.5.4
  $ ./configure
  $ make
  $ sudo make install

Then you will have to install perl bindings for HyperEstraier.

=head3 HyperEstraier perl bindings

This might take a bit more work. You will need to have C<swig> and C<g++> installed
before you can install perl bindings.

  $ tar xvfz hyper_estraier_wrappers-0.0.10.tar.gz 
  $ cd swig_hest/perl/
  $ swig -c++ -perl5 -o HyperEstraier_wrap.cpp  ../HyperEstraier.i
  $ cp ../HyperEstraierWrapper.cpp ./
  $ perl Makefile.PL

You might need to remove C<estconfig --mtlibs> from C<Makefile.PL> if it
fails to run with following message:

 Unrecognized argument in LIBS ignored: 'configuration'
 Unrecognized argument in LIBS ignored: 'helper'
 Unrecognized argument in LIBS ignored: 'for'
 Unrecognized argument in LIBS ignored: 'Hyper'
 Unrecognized argument in LIBS ignored: 'Estraier'
 Writing Makefile for HyperEstraier

Working C<Makefile.PL> for my particular installation is:

 # File : Makefile.PL
 use ExtUtils::MakeMaker;
 WriteMakefile(
        NAME    => 'HyperEstraier',
        INC     => sprintf('%s', `pkg-config --cflags hyperestraier`),
        LIBS    => [sprintf('-lstdc++ %s %s' , `pkg-config --libs hyperestraier`)],
        OBJECT  => 'HyperEstraierWrapper.o HyperEstraier_wrap.o'
 );

After succesfull C<perl Makefile.pl>, you can compile and install it.

  $ make
  $ sudo make install

=head3 CPAN modules

You will also need a few additional cpan modules

=over 4

=item File::Pid
=item File::Which
=item File::Path
=item File::Slurp

=item Spreadsheet::WriteExcel

=item Term::Menus
=item XML::Writer
=item Algorithm::Diff
=item Archive::Tar::Stream

=back

Last module, C<Spreadsheet::WriteExcel> is needed only if you want to use
C<BackupPC_xls_report> to generate Excel reports from your backup data.

Easiest way to install them is using C<cpan> shell.

  $ sudo cpan File::Pid Spreadsheet::WriteExcel

=head1 Creation of initial database

Once you have all components installed, you should initially create data
about increments in RDBMS and full-text search engine index.

First, edit C<config.pl> and setup C<SearchDSN> to valid perl DSN (for example,
C<dbi:Pg:dbname=backuppc> and C<SearchUser> to database user which has priviledges
over that database. You might need to add additional directives in PostgreSQL's
C<pg_hba.conf> so that selected user can be connected without password (I know, it's not
perfect, but I trust local users on backuppc machine).

Then, create new database for backuppc.

  $ createdb backuppc

Then invoke C<BackupPC_updatedb> for the first time with argument to create database
schema:

  $ sudo -u backuppc /data/backuppc/bin/BackupPC_updatedb -c

You can also force full-text reindex by C<-i> flag. This will add all files which are in
database but for some reason are missing in full-text index.

Now, you can setup cron job which will daily check your backups and update database
and full-text index. This is as simple as invoking

  /data/backuppc/bin/BackupPC_updatedb

As C<backuppc> user and redirecting output to log file.

B<Documentation is still incomplete>.

Now that you finished installation, you can select new options from
menu on the left and example search and archival.

=head1 Reporting in Excel

Often, it is useful to be able to present report about your BackupPC hosts, number
of snapshots, total size and other useful info. While all those information can
be accessed using web browser, for analysis it's useful to have them in spreadsheet
form. With this data, you can monitor changes on your backup cycle, effects of changes
on your server or network to your BackupPC installation and so on.

You can create Excel spreadsheet (which works perfectly with Gnumeric also) using
following command:

  $ sudo -u backuppc /data/backuppc/bin/BackupPC_xls_report

=head1 Additional requirements

You will also need several other command-line utilities to make
C<BackupPC_burnArchiveCLI> run. Those include:

=over 4

=item mkisofs

=item cdrecord

Or equivavalent utility, on Debian you might want to use C<dvdrecord>.

=item eject

Please make a symlink from C</dev/cdrom> to you CD/DVD burner.

=item split

=back

=head1 Troubleshoting

=head2 md5sum problems on Debian

If you are running Debian stable and expiriencing error message like

 -: No such file or directory

your C<md5sum> comes from C<coreutils> (which doesn't know how to handle C<->
as input) as opposed to C<md5sum> which comes with C<textutils> (which does).

Fix is simple, correct md5sum is allready installed, you just have to do:

 sudo cp /usr/bin/md5sum.textutils /usr/bin/md5sum

=head1 Related projects

BackupPC allready has archival host which might suit your needs better (and
it's quite easier to install).

=over 4

=item PostgreSQL

L<http://www.postgresql.org/>

=item HyperEstraier

L<http://hyperestraier.sourceforge.net/>

=back

=head1 Authors

Ivan Klaric C<< <iklaric@gmail.com> >>

Dobrica Pavlinusic C<< <dpavlin@rot13.org> >>

=head1 Licence

This extension is released under GPL licence, same as BackupPC.

1	dpavlin	94	=head1 BackupPC search and archival extension
2
3			BackupPC search and archival extension is used to:
4
5			=over 4
6
7			=item *
8
9			find files in C<pool> by substring of filename
10
11			=item *
12
13			see state of one or all of your shares in some particular point in time
14			(file creation date on clients or backup date is supported as filter)
15
16			=item *
17
18			archive selected backups (per host, share and increment) to archival
19	dpavlin	105	medium (CD, DVD, tape etc.) with rich descriptions, archival locations,
20			multiple copies and so on.
21	dpavlin	94
22			=back
23
24			=head2 Requirements
25
26			This extension is based on PostgreSQL RDBMS and HyperEstraier full-text
27			search engine (and it's perl bindings).
28
29	dpavlin	96	If you are using Debian, you are in luck. All required packages are
30			allready part of C<unstable> distribution and can be installed with:
31	dpavlin	94
32	dpavlin	96	# apt-get install postgresql
33			# apt-get install hyperestraier libestraier-dev libqdbm-dev
34
35	dpavlin	105	Now you can skip to installation of L<HyperEstraier perl bindings> below.
36	dpavlin	96
37			If you don't have pre-packaged binaries for your installation, you will
38			need to install additional packages by hand.
39
40	dpavlin	105	=head3 PostgreSQL
41	dpavlin	94
42			You can use packages provided by your distribution or follow installation
43			instructions on PostgreSQL site.
44
45	dpavlin	105	=head3 QDBM
46	dpavlin	94
47			First, you need qdbm on which HyperEstraier depends. Installation is simple.
48
49			$ tar xvfz qdbm-1.8.31.tar.gz
50			$ cd qdbm-1.8.31
51			$ ./configure --enable-zlib
52			$ make
53			$ sudo make install
54
55	dpavlin	105	=head3 HyperEstraier
56	dpavlin	94
57			Also quite simple.
58
59			$ tar xvfz hyperestraier-0.5.4.tar.gz
60			$ cd hyperestraier-0.5.4
61			$ ./configure
62			$ make
63			$ sudo make install
64
65	dpavlin	105	Then you will have to install perl bindings for HyperEstraier.
66	dpavlin	94
67	dpavlin	105	=head3 HyperEstraier perl bindings
68
69			This might take a bit more work. You will need to have C<swig> and C<g++> installed
70			before you can install perl bindings.
71
72	dpavlin	94	$ tar xvfz hyper_estraier_wrappers-0.0.10.tar.gz
73			$ cd swig_hest/perl/
74			$ swig -c++ -perl5 -o HyperEstraier_wrap.cpp ../HyperEstraier.i
75			$ cp ../HyperEstraierWrapper.cpp ./
76			$ perl Makefile.PL
77
78			You might need to remove C<estconfig --mtlibs> from C<Makefile.PL> if it
79			fails to run with following message:
80
81			Unrecognized argument in LIBS ignored: 'configuration'
82			Unrecognized argument in LIBS ignored: 'helper'
83			Unrecognized argument in LIBS ignored: 'for'
84			Unrecognized argument in LIBS ignored: 'Hyper'
85			Unrecognized argument in LIBS ignored: 'Estraier'
86			Writing Makefile for HyperEstraier
87
88			Working C<Makefile.PL> for my particular installation is:
89
90			# File : Makefile.PL
91			use ExtUtils::MakeMaker;
92			WriteMakefile(
93			NAME => 'HyperEstraier',
94	dpavlin	96	INC => sprintf('%s', `pkg-config --cflags hyperestraier`),
95			LIBS => [sprintf('-lstdc++ %s %s' , `pkg-config --libs hyperestraier`)],
96	dpavlin	94	OBJECT => 'HyperEstraierWrapper.o HyperEstraier_wrap.o'
97			);
98
99			After succesfull C<perl Makefile.pl>, you can compile and install it.
100
101			$ make
102			$ sudo make install
103
104	dpavlin	105	=head3 CPAN modules
105
106			You will also need a few additional cpan modules
107
108			=over 4
109
110			=item File::Pid
111	dpavlin	235	=item File::Which
112			=item File::Path
113			=item File::Slurp
114	dpavlin	105
115	dpavlin	108	=item Spreadsheet::WriteExcel
116
117	dpavlin	177	=item Term::Menus
118	dpavlin	198	=item XML::Writer
119	dpavlin	217	=item Algorithm::Diff
120			=item Archive::Tar::Stream
121	dpavlin	177
122	dpavlin	94	=back
123
124	dpavlin	108	Last module, C<Spreadsheet::WriteExcel> is needed only if you want to use
125			C<BackupPC_xls_report> to generate Excel reports from your backup data.
126
127	dpavlin	105	Easiest way to install them is using C<cpan> shell.
128	dpavlin	94
129	dpavlin	108	$ sudo cpan File::Pid Spreadsheet::WriteExcel
130	dpavlin	105
131			=head1 Creation of initial database
132
133	dpavlin	94	Once you have all components installed, you should initially create data
134			about increments in RDBMS and full-text search engine index.
135
136	dpavlin	105	First, edit C<config.pl> and setup C<SearchDSN> to valid perl DSN (for example,
137			C<dbi:Pg:dbname=backuppc> and C<SearchUser> to database user which has priviledges
138			over that database. You might need to add additional directives in PostgreSQL's
139			C<pg_hba.conf> so that selected user can be connected without password (I know, it's not
140			perfect, but I trust local users on backuppc machine).
141	dpavlin	94
142	dpavlin	105	Then, create new database for backuppc.
143	dpavlin	94
144	dpavlin	105	$ createdb backuppc
145	dpavlin	94
146	dpavlin	105	Then invoke C<BackupPC_updatedb> for the first time with argument to create database
147			schema:
148
149			$ sudo -u backuppc /data/backuppc/bin/BackupPC_updatedb -c
150
151			You can also force full-text reindex by C<-i> flag. This will add all files which are in
152			database but for some reason are missing in full-text index.
153
154			Now, you can setup cron job which will daily check your backups and update database
155			and full-text index. This is as simple as invoking
156
157			/data/backuppc/bin/BackupPC_updatedb
158
159			As C<backuppc> user and redirecting output to log file.
160
161			B<Documentation is still incomplete>.
162
163	dpavlin	94	Now that you finished installation, you can select new options from
164			menu on the left and example search and archival.
165
166	dpavlin	108	=head1 Reporting in Excel
167
168			Often, it is useful to be able to present report about your BackupPC hosts, number
169			of snapshots, total size and other useful info. While all those information can
170			be accessed using web browser, for analysis it's useful to have them in spreadsheet
171			form. With this data, you can monitor changes on your backup cycle, effects of changes
172			on your server or network to your BackupPC installation and so on.
173
174			You can create Excel spreadsheet (which works perfectly with Gnumeric also) using
175			following command:
176
177			$ sudo -u backuppc /data/backuppc/bin/BackupPC_xls_report
178
179	dpavlin	177	=head1 Additional requirements
180
181			You will also need several other command-line utilities to make
182			C<BackupPC_burnArchiveCLI> run. Those include:
183
184			=over 4
185
186			=item mkisofs
187
188	dpavlin	199	=item cdrecord
189
190			Or equivavalent utility, on Debian you might want to use C<dvdrecord>.
191
192			=item eject
193
194	dpavlin	204	Please make a symlink from C</dev/cdrom> to you CD/DVD burner.
195
196	dpavlin	199	=item split
197
198	dpavlin	177	=back
199
200	dpavlin	255	=head1 Troubleshoting
201	dpavlin	177
202	dpavlin	255	=head2 md5sum problems on Debian
203
204			If you are running Debian stable and expiriencing error message like
205
206			-: No such file or directory
207
208			your C<md5sum> comes from C<coreutils> (which doesn't know how to handle C<->
209			as input) as opposed to C<md5sum> which comes with C<textutils> (which does).
210
211			Fix is simple, correct md5sum is allready installed, you just have to do:
212
213			sudo cp /usr/bin/md5sum.textutils /usr/bin/md5sum
214
215	dpavlin	94	=head1 Related projects
216
217			BackupPC allready has archival host which might suit your needs better (and
218			it's quite easier to install).
219
220			=over 4
221
222			=item PostgreSQL
223
224			L<http://www.postgresql.org/>
225
226			=item HyperEstraier
227
228			L<http://hyperestraier.sourceforge.net/>
229
230			=back
231
232			=head1 Authors
233
234			Ivan Klaric C<< <iklaric@gmail.com> >>
235
236			Dobrica Pavlinusic C<< <dpavlin@rot13.org> >>
237
238			=head1 Licence
239
240			This extension is released under GPL licence, same as BackupPC.
241