0.9.9e/doc/IIF.txt

IIF, MARC and Z39.50


IIF is the "Information Interchange Format", a record serialization format
specified in ISO standard 2709, also published as ANSI
>       http://www.niso.org/standards/resources/Z39-2.pdf       Z39.2.
IIF is mostly a plaintext format, in that almost any information is encoded
using ASCII characters (no binary numbers) and the only control characters
used are byte values 29 (record terminator RT), 30 (field terminator FT)
and 31 (as subfield delimiter).


>       http://www.loc.gov/marc/        MARC
("MAchine Readable Catalogue") is actually a family of largely incompatible
standards (
>       http://www.loc.gov/marc/marcdocz.html   USMARC
,
>       http://www.ifla.org/VI/3/p1996-1/sec-uni.htm    UNIMARC
, UKMARC, ...) that evolved from MARC I (1965).
While the main concern of the MARC standards is to specify actual data models
(assigning tags and subfield codes, which can be used perfectly well in
Malete, CDS/ISIS or other databases), they also specify a variant of IIF as
suggested common format for data exchange, which we here refer to as "MARC".
(This file syntax seems to be mostly the same for all MARC standards).


>       ftp://ftp.loc.gov/pub/z3950/official/part1.txt  Z39.50
is a network protocol to search and retrieve records.
It supports various query "languages", the most commonly used of which
is called Type-1 query. Type-1 is similar to the queries as supported
by Malete and CDS/ISIS, however, much more general and complex.
Terms can be searched for in any indexed field or with restriction
to one or more "attributes".

Attributes are basically the tags used in the index, which are almost always
different from those used in records. While it is common for records to use
any of the various MARCs or even completely different formats, the attributes
used in bibliographical systems are typically those specified by the Bib-1
attribute set (e.g. assigning 4 to title).


Z39.50 allows a client to select a record format from various conversions
supported by a server. When a MARC format is selected,
the data is actually transmitted serialized according to IIF.


*       IIF and MARC serialized records

IIF specifies a serialization for records. Like the Malete record data file,
an IIF file is simply a stream of such records; there is no additional
file header.

A record has
-       a 24 byte leader, containing 16 bytes structural data
        and 8 bytes application data (x, imported as "MARC leader").
        The format for MARC is LLLLLxxxxx22BBBBBxxx4500.
        The Ls and Bs are total record length (including leader and a terminating RT)
        and  start of data (field values, after an FT terminating the dictionary).
        The first '2' denotes that every field starts with two indicator bytes,
        the second is the subfield identifier length including the delimiter char.
-       a "dictionary" array with one entry per field containing 3 bytes tag,
        and n and m bytes for length and offset.
        n and m are digits at leader offset 20 and 21, MARC uses 4 and 5.
        In general IIF, leader byte 22 may specify a number of implementation
        defined entry bytes.
-       the actual field values, each terminated by the FT character.

As opposed to folklore, MARC does NOT use a '$' as subfield delimiter,
nor a '#' for unused indicators. Rather, the examples in the specs
use a '$' to REPRESENT the subfield delimiter control character 31 (^_),
and a '#' to REPRESENT a blank. The RT(29, ^]) is sometimes represented as '\'
and the FT(30, ^^) as '^' or '@'.


*       Malete IIF import and export

The malete tool provides two rather simplistic
>       CmdLine commands
iifimp and iifexp.

The command specific options are:
-       Ffile
        specify full filename for the IIF files.
        Default is the basename of the Malete database with extension .iif.
        On UNIX, a filename '-' selects stdin/out.
-       Nomarc (literally)
        do not assume the MARC structure 22/450 on import. Requires proper IIF data.
-       P[iic]
        on export, prepend indicators ii and, where needed, subfield c.
        A single -P uses two blanks as indicators and subfield '0'.
        Suggested to produce at least syntactically correct MARC.
-       Rid (literally)
        on import, use a numeric control number (1st field, if it has tag 1)
        as record id. Note that on export, the record id is always used as
        control number unless the record already has one,
        since this is specified as a must not only by MARC, but by IIF.

*       creating proper IIF from WinIsis

In Database-Export, set the subfield separator to \031 and
output line length to 0.

If the fields do not contain valid MARC data, use a reformatting FST like
$
001 0 MFN
044 0 |00^a|,v44
024 0 |00^a|,v24
026 0 |00|,v26
070 0 (|00^a|,v70/)
$
Make sure, that
-       the first output field is tag 1 containing some unique id
-       every field starts with two indicator characters
        (really should be blank, but that would be stripped during export)
-       the indicators are followed by a delimiter and subfield identifier
Still the output is not 100% correct, since WinIsis sets
number of indicators and identifier length to 0, where MARC specifies 2.
However, many other MARC processors, including zebraidx, ignore these settings.


*       making MARC data available via Z39.50

MARC records can be made easily available using indexdata's
>       http://www.indexdata.dk/zebra/  zebra.

If records in your IIF file use tags and subfields conforming to, say, USmarc, 
simply check out the test/usmarc example in the zebra distribution.
Put your data in the records subdir and run "zebraidx update records; zebrasrv".

If your data was exported from WinIsis, you may want to put a line
"encoding Cp850" in the .abs file.


You must use recordType: grs.marc.something, meaning that it's general
structured data in some marc file format.
The sample usmarc.abs uses the "marc usmarc.mar" statement,
and usmarc.mar (in the zebra/tab directory) contains "reference USmarc",
stating that the marc input actually IS in USmarc.
This need not be the truth, it just means that the records will be served
as is, if a client asks for USmarc.
However, only the tags listed in "elm" statements in the .abs files
will be indexed.


Note that zebra's indexing support is not as flexible as that of CDS/ISIS:
you can only select fields or subfields to be indexed in one of a couple
of modes (like word or phrase). To take full advantage of sophisticated
CDS/ISIS FSTs, include them in your export reformatting FST.
Use some otherwise unused field tags to hold the index terms and "elm"
statements to map them to bib-1 attributes.
Omit those fields from the display mapping.


To keep the data in its native format (say CDS), change the elm
statements to map the fields to index to the corresponding bib-1 attributes
for searching, e.g. "elm 024     Conference-name   !",
and, instead of using the "marc usmarc.mar" statement,
create one or more maptabs to map the full record to one or more
USmarc a/o other presentation formats as applicable.
Check out the gils-usmarc.map example in the zebra/tab directory.


Consult the
>       http://www.indexdata.dk/zebra/doc/      zebra documentation
for details.


*       links
-       ISO2709 "Information Interchange Format", a.k.a. ANSI/NISO
>       http://www.niso.org/standards/resources/Z39-2.pdf       Z39.2 
-       Machine Readable Catalogues
>       http://www.loc.gov/marc/specifications/specrecstruc.html        (US) MARC 21
        ,
>       http://lcweb.loc.gov/marc/      overview 
        , references at the
>       http://www.oasis-open.org/cover/marc.html       Cover Pages
-       Z39.50
>       ftp://ftp.loc.gov/pub/z3950/official/   official spec
        , overview at
>       http://www.oclcpica.org/content/45/pdf/z3950_handbook_paper.pdf OCLC|Pica
        , links at
>       http://www.indexdata.dk/technologies/z3950/     indexdata
        , makers of excellent free Z39.50 software. 
-       Uncle Aung's
>       http://uncleaung.com/zisis/     Zisis

---
        $Id: IIF.txt,v 1.5 2004/09/23 11:44:04 kripke Exp $
1	dpavlin	604	IIF, MARC and Z39.50
2
3
4			IIF is the "Information Interchange Format", a record serialization format
5			specified in ISO standard 2709, also published as ANSI
6			> http://www.niso.org/standards/resources/Z39-2.pdf Z39.2.
7			IIF is mostly a plaintext format, in that almost any information is encoded
8			using ASCII characters (no binary numbers) and the only control characters
9			used are byte values 29 (record terminator RT), 30 (field terminator FT)
10			and 31 (as subfield delimiter).
11
12
13			> http://www.loc.gov/marc/ MARC
14			("MAchine Readable Catalogue") is actually a family of largely incompatible
15			standards (
16			> http://www.loc.gov/marc/marcdocz.html USMARC
17			,
18			> http://www.ifla.org/VI/3/p1996-1/sec-uni.htm UNIMARC
19			, UKMARC, ...) that evolved from MARC I (1965).
20			While the main concern of the MARC standards is to specify actual data models
21			(assigning tags and subfield codes, which can be used perfectly well in
22			Malete, CDS/ISIS or other databases), they also specify a variant of IIF as
23			suggested common format for data exchange, which we here refer to as "MARC".
24			(This file syntax seems to be mostly the same for all MARC standards).
25
26
27			> ftp://ftp.loc.gov/pub/z3950/official/part1.txt Z39.50
28			is a network protocol to search and retrieve records.
29			It supports various query "languages", the most commonly used of which
30			is called Type-1 query. Type-1 is similar to the queries as supported
31			by Malete and CDS/ISIS, however, much more general and complex.
32			Terms can be searched for in any indexed field or with restriction
33			to one or more "attributes".
34
35			Attributes are basically the tags used in the index, which are almost always
36			different from those used in records. While it is common for records to use
37			any of the various MARCs or even completely different formats, the attributes
38			used in bibliographical systems are typically those specified by the Bib-1
39			attribute set (e.g. assigning 4 to title).
40
41
42			Z39.50 allows a client to select a record format from various conversions
43			supported by a server. When a MARC format is selected,
44			the data is actually transmitted serialized according to IIF.
45
46
47			* IIF and MARC serialized records
48
49			IIF specifies a serialization for records. Like the Malete record data file,
50			an IIF file is simply a stream of such records; there is no additional
51			file header.
52
53			A record has
54			- a 24 byte leader, containing 16 bytes structural data
55			and 8 bytes application data (x, imported as "MARC leader").
56			The format for MARC is LLLLLxxxxx22BBBBBxxx4500.
57			The Ls and Bs are total record length (including leader and a terminating RT)
58			and start of data (field values, after an FT terminating the dictionary).
59			The first '2' denotes that every field starts with two indicator bytes,
60			the second is the subfield identifier length including the delimiter char.
61			- a "dictionary" array with one entry per field containing 3 bytes tag,
62			and n and m bytes for length and offset.
63			n and m are digits at leader offset 20 and 21, MARC uses 4 and 5.
64			In general IIF, leader byte 22 may specify a number of implementation
65			defined entry bytes.
66			- the actual field values, each terminated by the FT character.
67
68			As opposed to folklore, MARC does NOT use a '$' as subfield delimiter,
69			nor a '#' for unused indicators. Rather, the examples in the specs
70			use a '$' to REPRESENT the subfield delimiter control character 31 (^_),
71			and a '#' to REPRESENT a blank. The RT(29, ^]) is sometimes represented as '\'
72			and the FT(30, ^^) as '^' or '@'.
73
74
75			* Malete IIF import and export
76
77			The malete tool provides two rather simplistic
78			> CmdLine commands
79			iifimp and iifexp.
80
81			The command specific options are:
82			- Ffile
83			specify full filename for the IIF files.
84			Default is the basename of the Malete database with extension .iif.
85			On UNIX, a filename '-' selects stdin/out.
86			- Nomarc (literally)
87			do not assume the MARC structure 22/450 on import. Requires proper IIF data.
88			- P[iic]
89			on export, prepend indicators ii and, where needed, subfield c.
90			A single -P uses two blanks as indicators and subfield '0'.
91			Suggested to produce at least syntactically correct MARC.
92			- Rid (literally)
93			on import, use a numeric control number (1st field, if it has tag 1)
94			as record id. Note that on export, the record id is always used as
95			control number unless the record already has one,
96			since this is specified as a must not only by MARC, but by IIF.
97
98			* creating proper IIF from WinIsis
99
100			In Database-Export, set the subfield separator to \031 and
101			output line length to 0.
102
103			If the fields do not contain valid MARC data, use a reformatting FST like
104			$
105			001 0 MFN
106			044 0 \|00^a\|,v44
107			024 0 \|00^a\|,v24
108			026 0 \|00\|,v26
109			070 0 (\|00^a\|,v70/)
110			$
111			Make sure, that
112			- the first output field is tag 1 containing some unique id
113			- every field starts with two indicator characters
114			(really should be blank, but that would be stripped during export)
115			- the indicators are followed by a delimiter and subfield identifier
116			Still the output is not 100% correct, since WinIsis sets
117			number of indicators and identifier length to 0, where MARC specifies 2.
118			However, many other MARC processors, including zebraidx, ignore these settings.
119
120
121			* making MARC data available via Z39.50
122
123			MARC records can be made easily available using indexdata's
124			> http://www.indexdata.dk/zebra/ zebra.
125
126			If records in your IIF file use tags and subfields conforming to, say, USmarc,
127			simply check out the test/usmarc example in the zebra distribution.
128			Put your data in the records subdir and run "zebraidx update records; zebrasrv".
129
130			If your data was exported from WinIsis, you may want to put a line
131			"encoding Cp850" in the .abs file.
132
133
134			You must use recordType: grs.marc.something, meaning that it's general
135			structured data in some marc file format.
136			The sample usmarc.abs uses the "marc usmarc.mar" statement,
137			and usmarc.mar (in the zebra/tab directory) contains "reference USmarc",
138			stating that the marc input actually IS in USmarc.
139			This need not be the truth, it just means that the records will be served
140			as is, if a client asks for USmarc.
141			However, only the tags listed in "elm" statements in the .abs files
142			will be indexed.
143
144
145			Note that zebra's indexing support is not as flexible as that of CDS/ISIS:
146			you can only select fields or subfields to be indexed in one of a couple
147			of modes (like word or phrase). To take full advantage of sophisticated
148			CDS/ISIS FSTs, include them in your export reformatting FST.
149			Use some otherwise unused field tags to hold the index terms and "elm"
150			statements to map them to bib-1 attributes.
151			Omit those fields from the display mapping.
152
153
154			To keep the data in its native format (say CDS), change the elm
155			statements to map the fields to index to the corresponding bib-1 attributes
156			for searching, e.g. "elm 024 Conference-name !",
157			and, instead of using the "marc usmarc.mar" statement,
158			create one or more maptabs to map the full record to one or more
159			USmarc a/o other presentation formats as applicable.
160			Check out the gils-usmarc.map example in the zebra/tab directory.
161
162
163			Consult the
164			> http://www.indexdata.dk/zebra/doc/ zebra documentation
165			for details.
166
167
168			* links
169			- ISO2709 "Information Interchange Format", a.k.a. ANSI/NISO
170			> http://www.niso.org/standards/resources/Z39-2.pdf Z39.2
171			- Machine Readable Catalogues
172			> http://www.loc.gov/marc/specifications/specrecstruc.html (US) MARC 21
173			,
174			> http://lcweb.loc.gov/marc/ overview
175			, references at the
176			> http://www.oasis-open.org/cover/marc.html Cover Pages
177			- Z39.50
178			> ftp://ftp.loc.gov/pub/z3950/official/ official spec
179			, overview at
180			> http://www.oclcpica.org/content/45/pdf/z3950_handbook_paper.pdf OCLC\|Pica
181			, links at
182			> http://www.indexdata.dk/technologies/z3950/ indexdata
183			, makers of excellent free Z39.50 software.
184			- Uncle Aung's
185			> http://uncleaung.com/zisis/ Zisis
186
187			---
188			$Id: IIF.txt,v 1.5 2004/09/23 11:44:04 kripke Exp $