README File for FRENCH GIGAWORD SECOND EDITION =============================================== INTRODUCTION ------------ French Gigaword Second Edition is a comprehensive archive of newswire text data that has been acquired over several years by the Linguistic Data Consortium (LDC) at the University of Pennsylvania. The two distinct international sources of French newswire in this edition, and the time spans of collection covered for each, are as follows: - Agence France-Presse (afp_fre) May 1994 - Dec 2008 - Associated Press French Service (apw_spa) Nov 1994 - Dec 2008 The seven-letter codes in parentheses include the three-character source name abbreviations and the three-character language code ("fre") separated by an underscore ("_") character. The three-letter language code conforms to LDC's new internal convention based on the ISO 639-3 standard. The seven-letter codes are used in both the directory names where the data files are found, and in the prefix that appears at the beginning of every data file name. It is also used (in all UPPER CASE) as the initial portion of the DOC "id" strings that uniquely identify each news story. DATA FORMAT AND SGML MARKUP --------------------------- Each data file name consists of the 7-letter prefix plus another underscore character, followed by a 6-digit date (representing the year and month during which the file contents were generated by the respective news source), followed by a ".gz" file extension, indicating that the file contents have been compressed using the GNU "gzip" compression utility (RFC 1952). So, each file contains all the usable data received by LDC for the given month from the given news source. All text data are presented in SGML form, using a very simple, minimal markup structure; all text consists of printable ASCII, white space, and printable code points in the "Latin1 Supplement" character table, as defined by the Unicode Standard (ISO 10646) for the "accented" characters used in French. The Supplement/accented characters are presented in UTF-8 encoding. The file "gigaword_f.dtd" in the "dtd" directory provides the formal "Document Type Declaration" for parsing the SGML content. The corpus has been fully validated by a standard SGML parser utility (nsgmls), using this DTD file. The markup structure, common to all data files, can be summarized as follows: The Headline Element is Optional -- not all DOCs have one The Dateline Element is Optional -- not all DOCs have one

Paragraph tags are only used if the 'type' attribute of the DOC happens to be "story" -- more on the 'type' attribute below...

Note that all data files use the UNIX-standard "\n" form of line termination, and text lines are generally wrapped to a width of 80 characters or less.

For every "opening" tag (DOC, HEADLINE, DATELINE, TEXT, P), there is a corresponding "closing" tag -- always. The attribute values in the DOC tag are always presented within double-quotes; the "id=" attribute of DOC consists of the 7-letter source/language abbreviation (in CAPS), an underscore, an 8-digit date string representing the date of the story (YYYYMMDD), a period, and a 4-digit sequence number starting at "0001" for each date (e.g. "AFP_FRE_19951231.0001"); in this way, every DOC in the corpus is uniquely identifiable by the id string. There are cases where we assigned a sequence number to a document, and later found that the document was empty or very noisy. In such cases, we have removed the document from the collection, but did not reassign sequence numbers to the rest of the collection for the same day. As a result there may be some gaps in sequence numbers. Every SGML tag is presented alone on one line, separate from other tags, and from the text content (so a simple process like the UNIX "grep -v '<'" will eliminate all tags, and retain all the text content). The structure shown above represents some notable differences relative to the markup strategy used in previous LDC publications of newswire data; these are intended to facilitate bulk processing of the present corpus. The major differences are: - Earlier corpora usually organized the data as one file per day, or limited the average file size to one megabyte (MB). Typical compressed file sizes in the current corpus range from about 0.1 MB to about 10 MB; this equates to a range of about 0.5 to 30 MB per file when the data are uncompressed. In general, these files are not intended for use with interactive text editors or word processing software (though many such programs are likely to work reasonably well with these files). Rather, it's expected that the files will be used as input to programs that are geared to dealing with data in such quantities, for filtering, conditioning, indexing, statistical summary, etc. - Earlier corpora tended to use different markup outlines (different tag sets) depending on the source of the data, because different sources came to us with different structural properties, and we had chosen to preserve these as much as possible (even though many elements of the delivered structure may have been meaningless for research use). The present corpus uses only the information structure that is common to all sources and serves a clear function: headline, dateline, and core news content (usually containing paragraphs). The "dateline" is a brief string typically found at the beginning of the first paragraph in each news story, giving the location the report is coming from, and sometimes the news service and/or date; since this content is not part of the initial sentence, we separate it from the first paragraph (this was not done prior to the Gigaword corpora). - Earlier corpora tended to include "custom" SGML entity references, which were intended to preserve things like special punctuation or typesetting instructions (e.g. "&QL;", "&UR;", "&MD;", etc). The present corpus uses only two SGML entity references: ``&'' and ``>'' (only the lower-case forms are used). The former represents the literal ampersand "&" character (e.g. as sometimes appears in abbreviated company names: AT&T). The latter is the "right-angle bracket" (>), (sometimes used as typesetting "decoration", and also occurring in the name "i>TELE"). All other specialized control characters have been filtered out, and unusual punctuation marks (such as the underscore "_", sometimes used in APW_FRE to represent an "em-dash" character) have been converted to simple equivalents (e.g. hyphens). - In earlier corpora, newswire data were presented as streams of undifferentiated "DOC" units; depending on the source and corpus, varying amounts of quality checking and filtering were done to eliminate noisy or unsuitable content (e.g. test messages). The preparation of text data for release in Gigaword still involves a number of source-specific checks and operations, owing to intrinsic differences among the sources. In fact even for a single source, we tend to experience significant transitions in the nature of incoming data over the span of the collection; for example, APW_FRE switched from transmission via dedicated serial modem to a database-driven, XML-formatted internet delivery protocol in June 2004. However, now that we have standardized on a uniform markup format for all sources (in fact, the same format used for all Gigaword releases regardless of language), we are better able to provide a more uniform degree of quality control compared to earlier corpora. For all of the documents in this corpus, we have applied a rudimentary (and _approximate_) categorization of DOC units into four distinct "types". The classification is indicated by the `` type="string" '' attribute that is included in each opening ``DOC'' tag. The four types are: * story : This is by far the most frequent type, and it represents the most typical newswire item: a coherent report on a particular topic or event, consisting of paragraphs and full sentences. As indicated above, the paragraph tag "

" is found only in DOCs of this type; in the other types described below, the text content is rendered with no additional tags or special characters -- just lines of tokens (strings of letters, numbers and punctuation) separated by white space, roughly equivalent to the original format as received. * multi : This type of DOC contains a series of unrelated "blurbs", each of which briefly describes a particular topic or event; this is typically applied to DOCs that contain "summaries of today's news", "news briefs in ... (some general area like finance or sports)", and so on. Each paragraph-like blurb by itself is coherent and composed of sentences, but it does not bear any necessary relation of topicality or continuity relative to it neighboring sections, and it may contain its own distinct dateline. * advis : (short for "advisory") These are DOCs which the news service addresses to news editors -- they are not intended for publication to the "end users" (the populations who read the news); as a result, DOCs of this type tend to contain obscure abbreviations and phrases, which are familiar to news editors, but may be meaningless to the general public. We also find a lot of formulaic, repetitive content in DOCs of this type (contact phone numbers, word counts, etc). * other : This represents DOCs that clearly do not fall into any of the above types -- in general, items of this type are intended for broad circulation (they are not advisories), they may be topically coherent (unlike "multi" type DOCS), and they typically do not contain paragraphs or sentences (they aren't really "stories"); these are things like lists of sports scores, stock prices, temperatures around the world, and so on. The general strategy for categorizing DOCs into these four classes was, for each source, to discover the most common and frequent clues in the text stream that correlated with the three "non-story" types, and to apply the appropriate label for the ``type=...'' attribute whenever the DOC displayed one of these specific clues. When none of the known clues was in evidence, the DOC was classified as a "story". This means that the most frequent classification error will tend to be the use of `` type="story" '' on DOCs that are actually some other type. But the number of such errors should be fairly small, compared to the number of "non-story" DOCs that are correctly tagged as such. Note that the markup was applied algorithmically, using logic that was based on less-than-complete knowledge of the data. For the most part, the HEADLINE, DATELINE and TEXT tags have their intended content; but due to the inherent variability (and the inevitable source errors) in the data, users may find occasional mishaps where the headline and/or dateline were not successfully identified (hence show up within TEXT), or where an initial sentence or paragraph has been mistakenly tagged as the headline or dateline. DATA QUANTITIES --------------- The "docs" directory contains two plain-text tables (data_stats_*) that describe the quantities of data by source and month (i.e. by file), broken down according to the four "type" categories. The overall totals for each source are summarized below. Note that the "Totl-MB" numbers show the amount of data you get when the files are uncompressed (i.e. approximately 15 gigabytes, total); the "Gzip-MB" column shows totals for compressed file sizes as stored on the DVD-ROM; the "K-wrds" numbers are simply the number of white space-separated tokens (of all types) after all SGML tags are eliminated. Source #Files Gzip-MB Totl-MB K-wrds #DOCs AFP_FRE 172 2408 4079 560000 2060803 APW_FRE 171 2280 1719 241324 872573 TOTAL 343 4688 5789 801324 2933376 The following tables present "Text-MB", "K-wrds" and "#DOCS" broken down by source and DOC type; "Text-MB" represents the total number of characters (including white space) after SGML tags are eliminated. Text-MB K-wrds #DOCs type="advis": AFP_FRE 88 11788 48712 APW_FRE 14 2303 9235 TOTAL 102 14091 57947 type="multi": AFP_FRE 58 8411 10269 APW_FRE 194 29828 52240 TOTAL 252 38239 62509 type="other": AFP_FRE 177 58514 8411 APW_FRE 82 193981 29828 TOTAL 260 252495 38239 type="story": AFP_FRE 1824 198440 27216 APW_FRE 728 87662 13006 TOTAL 2552 286102 40222 GENERAL AND SOURCE-SPECIFIC PROPERTIES OF THE DATA -------------------------------------------------- Most of the text data (all of AFP_FRE, most of APW_FRE) were received at LDC via dedicated, 24-hour/day electronic feeds (leased phone lines in the case of APW_FRE, a local satellite dish for AFP_FRE). These 24-hour transmission services were all susceptible to "line noise" (occasional corruption of text content), as well as service outages both at the data source and at our receiving computers. Usually, the various disruptions of a newswire data stream would leave tell-tale evidence in the form of byte values falling outside the range of printable ASCII characters, or recognizable patterns of anomalous ASCII strings. The portion of APW_FRE data beginning with 200406 was received as bulk electronic text archives via internet retrieval. As such, they were not susceptible to modem line-noise or related disruptions, though this does not guarantee that the source data are free of mishaps. Starting with 200406, and steadily since then, APW_FRE data have been received via internet-based subscription systems, whereby first-issue stories and editing updates are sent throughout the day to a dedicated client process running at the LDC; this process maintains a local database and story cache that holds the latest version of each distinct story for a limited number of days (in contrast to the older modem-based service, where updated versions and editing directives simply accumulated in an ever-growing archive of the continuous data stream). In the new setup, the harvesting of stories into a cumulative archive is simply a matter of taking a daily snapshot of the client-program's story cache, removing stories from the snapshot if they had been captured on a previous day, and adding the remainder to the archive. As a result, the data collected in this manner tends to include less duplication of story content (because repeated transmissions of a given story, with or without minor edits, are generally not retained in the final archive). All the data have undergone a consistent extent of quality control, to eliminate out-of-band content and other obvious forms of corruption. Naturally, since the source data are all generated manually on a daily basis, there will be a small percentage of human errors common to all sources: missing white space, incorrect or variant spellings, badly formed sentences, and so on, as are normally seen in newspapers. No attempt has been made to address this property of the data. As indicated above, a common feature of the modem-based archives is that stories may be repeated in the course of daily transmissions (or daily archiving). Sometimes a later transmission of a story comes with minor alterations (fixed spelling, one or more paragraphs added or removed); but just as often, the collection ends up with two or more DOCs that are fully identical. In general, though, this practice affects a relatively small minority of the overall content. We have not attempted to eliminate these duplications; however, we plan to make information about duplicate and similar articles available on our web site as supplemental information for this corpus. (See the "ADDITIONAL INFORMATION and UPDATES" section below.) Finally, some of the modem services typically show a practice of breaking long stories into chunks, and sending the chunks as separate DOC units, with each unit having the normal structural features of a full story. Normally, when this sort of splitting is done, cues are provided in the text of each chunk that allow editors to reconstruct the full report; but these cues tend to rely heavily on editorial skills -- it is taken for granted by each news service that the stories will be reassembled manually as needed -- so the process of combining the pieces into a full story is not amenable to an algorithmic solution, and no attempt has been made to do this. On the whole, however, this practice is quite rare in the French newswire modem services. The following sections explain further data properties that are particular to each source. AFP_FRE: Intermittent noise in the satellite/modem receiving system was particularly pervasive in the late summer of 2002 and again in the late summer of 2003; the resulting loss of usable stories is reflected in the relatively small sizes of the monthly files for 200208, 200209, 200308 and 200309. Symptoms of the reception problems were typically obvious enough to be caught by algorithms, but it's possible that a noticeable number of stories in these files may contain more subtle problems, in the form of occasional "word" tokens having garbled orthography or inappropriate punctuation characters within or around the word. Apart from the data transmission issues, the AFP_FRE content shows a high degree of internal consistency, in terms of day-to-day content and typographic conventions. During 2007, LDC's AFP feed switched to a new delivery method. Although the data content appears to be fairly consistent with previously collected content, LDC has not done detailed analysis to determine the level of consistency. There is a gap in AFP coverage from Aug 2006 through Dec 2006. APW_FRE: During the period collected via modem, this service provided up to six other languages besides French via the same modem connection, with DOCs in all languages interleaved at random. Of course, we have extracted just the French content for publication here, based on flags provided in the header of each story; however we did notice a handful of cases where the header information was incorrect. The modem portion as a whole may contain a small number of articles in English, German or Spanish. The service draws news from quasi-independent offices around the world, so there tends to be more variability here in terms of typographic conventions; there is also a noticeably higher percentage of non-story content, especially in the "other" category: tables of sports results, stocks, weather, etc. Starting in May 2004, APW switched to a dedicated internet delivery system, eliminating the problems of modem noise and also creating a much better environment for limiting or avoiding duplicate content in stories.APW is now delivered by an NNTP-based service, with the document data presented in an XML format. ADDITIONAL INFORMATION AND UPDATES ---------------------------------- Additional information, updates, and bug fixes may be available in the LDC catalog entry for this corpus (LDC2009T28) at: http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2009T28 ---------------------------------------------------------------------- Original README file written by David Graff, November 2006 and updated Oct. 2009 by Angelo Mendonça. Linguistic Data Consortium