CSC343 -- Introduction To Databases Weeks 7 And 9: XML .

CSC343 -- Introduction to DatabasesDefaultnamespaceExample item xmlns http://www.acmeinc.com/jp#suppliesxmlns:toy “http://www.acmeinc.com/jp#toys” name backpack /name reservedtoy feature keywordnamespace toy:item toy:name cyberpet /toy:name /toy:item /feature /item XML -- 27CSC343 -- Introduction to DatabasesMore NamespacesÆ Scopes of declarations are color-coded: item xmlns “http://www.foo.org/abc”xmlns:cde “http://www.bar.com/cde” name /name New default; feature overshadows old default cde:item cde:name /cde:name /cde:item /feature item xmlns “http://www.foobar.org/”xmlns:cde “http://www.foobar.org/cde” name /name cde:name /cde:name Redeclaration of /item cde;cde overshadows old /item declarationXML -- 2814

CSC343 -- Introduction to DatabasesNamespaces (cont’d)Æxmlns “http://foo.com/bar” doesn’t mean there isa document at this URL: using URLs is just aconvention; a namespace is just an identifier.ÆNamespaces aren’t part of XML 1.0, but all XMLprocessors understand this feature nowÆA number of prefixes have become “standard”and some XML processors might understandthem without any declaration. E.g.,9xs for http://www.w3.org/2001/XMLSchema9xsl for http://www.w3.org/1999/XSL/Transform9Etc.XML -- 29CSC343 -- Introduction to DatabasesDocument Type Definition (DTD)ÆA DTD is a grammar specification for an XMLdocument – you can think of it as a schema.ÆDTDs are optional – don’t need to bespecified; if specified, a DTD can be part ofthe document (at the top); or it can be givenas a URLÆA document that conforms (i.e., parses) w.r.t.its DTD is said to be valid.validÆXML processors are not required to checkvalidity, even if DTD is specified; but they arerequired to test well-formedness.XML -- 3015

CSC343 -- Introduction to DatabasesAttaching a DTD to a DocumentÆDTD specified as part of a document: ?xml version “1.0” ? !DOCTYPE Report [ DTD Report spec ] Name of the DTD Report /Report ÆDTD can also be specified as a standalone thing ?xml version “1.0” ? !DOCTYPE Report“http://foo.org/Report.dtd” Report /Report XML -- 31CSC343 -- Introduction to DatabasesExample DTD ConstructsElement’scontentsÆ !ELEMENT elt-name( contents )/EMPTY/ANY An attr for eltÆ !ATTLIST elt-name pe of attribute Optional/mandatoryÆCan define other things, like macros (calledentities in XML jargon)XML -- 3216

CSC343 -- Introduction to DatabasesDTD LanguageÆ !DOCTYPE root-element [ doctypedeclaration. ] — determines name of rootelement and contains document typedeclarationsÆ !ELEMENT element-name content-model —associates a content model to every elementÆContent models:9EMPTY: no content is allowed9ANY: any content is allowed9(#PCDATA element-name .)*: "mixedcontent", arbitrary sequence of character dataand listed elements;9Deterministic regular expression (cont'd).XML -- 33CSC343 -- Introduction to DatabasesDTD Language: Regular ExpressionsÆDeterministic regular expression overelement names: sequence of elementsmatching the expression choice: (. . .) sequence: (.,.,.) optional: .? zero or more: .* one or more: . XML -- 3417

CSC343 -- Introduction to DatabasesDTD Language: AttributesÆ !ATTLIST element-name attr-name attr-typeattr-default . — declares which attributesare allowed or required in which elementsÆAttribute types:9CDATA: any value is allowed (the default)9(value .): enumeration of allowed values9ID, IDREF, IDREFS: ID attribute values mustbe unique (contain "element identity"),IDREF attribute values must match some ID(reference to an element)9ENTITY, ENTITIES, NMTOKEN,NMTOKENS, NOTATION: just forget these.XML -- 35CSC343 -- Introduction to DatabasesDTD Language:Attribute DefaultsÆ#REQUIRED: the attribute must be explicitlyprovided.Æ#IMPLIED: attribute is optional, no defaultprovided.Æ"value": if not explicitly provided, this value isinserted by default.Æ#FIXED "value": as above, but only this valueis allowed.XML -- 3618

CSC343 -- Introduction to DatabasesDTD Example !DOCTYPE Report [ !ELEMENT Report (Students, Classes, Courses) !ELEMENT Students (Student*) Zero or more !ELEMENT Classes (Class*) !ELEMENT Courses (Course*) !ELEMENT Student (Name, Status, CrsTaken*) !ELEMENT Name (First,Last) Has text content !ELEMENT First (#PCDATA) Empty element, !ELEMENT CrsTaken EMPTY no content !ELEMENT Class (CrsCode,Semester,ClassRoster) !ELEMENT Course (CrsName) !ATTLIST Report Date CDATA #IMPLIED Same attribute in !ATTLIST Student StudId ID #REQUIRED different elements !ATTLIST Course CrsCode ID #REQUIRED !ATTLIST CrsTaken CrsCode IDREF #REQUIRED !ATTLIST ClassRoster Members IDREFS #IMPLIED ] XML-- 37CSC343 -- Introduction to Databases !ELEMENT collection (description,recipe*) !ELEMENT description ANY !ELEMENT recipe (title, ingredient*, preparation, comment?,nutrition) !ELEMENT title (#PCDATA) !ELEMENT ingredient (ingredient*,preparation)? !ATTLIST ingredient name CDATA #REQUIREDamount CDATA #IMPLIEDunit CDATA #IMPLIED !ELEMENT preparation (step*) !ELEMENT step (#PCDATA) !ELEMENT comment (#PCDATA) !ELEMENT nutrition EMPTY !ATTLIST nutrition protein CDATA #REQUIREDcarbohydrates CDATA #REQUIREDfat CDATA #REQUIREDcalories CDATA #REQUIREDalcohol CDATA #IMPLIED AnotherExampleXML -- 3819

CSC343 -- Introduction to DatabasesLimitations of DTDsÆDon’t understand namespaces.ÆVery limited assortment of data types (juststrings).ÆVery weak wrt consistency constraints(ID/IDREF/ IDREFS only).ÆCan’t express unordered contents conveniently.ÆAll element names are global: can’t have oneName type for people and another forcompanies, e.g., !ELEMENT Name (Last, First) !ELEMENT Name (#PCDATA) can’t be in the same DTDXML -- 39CSC343 -- Introduction to DatabasesXML SchemaÆProposed in order to rectify drawbacks of DTDs.ÆAdvantages:9Integrated with namespaces;9Many built-in types;9User-defined types;9Has local element names;9Powerful key and referential constraints.ÆDisadvantages: Unwieldy, much more complexthan DTDsXML -- 4020

CSC343 -- Introduction to DatabasesXML Query LanguagesÆXPath – core query language. Very limited, aglorified selection operator. Very useful, though:used in XML Schema, XSLT, XQuery, many otherXML standards.ÆXSLT – a functional document transformationlanguage. Very powerful, very complicated.ÆXQuery – W3C standard. Very powerful, fairlyintuitive, SQL-styleÆSQL/XML – attempt to marry SQL and XML, part ofSQL:2003XML -- 41CSC343 -- Introduction to DatabasesWhy Query XML?ÆNeed to extract parts of XML documents.ÆNeed to transform documents into differentforms.ÆNeed to relate – join – parts of the same ordifferent documents.XML -- 4221

CSC343 -- Introduction to DatabasesXPathÆAnalogous to path expressions in objectoriented languages (e.g., OQL).ÆExtends path expressions with query facility.ÆXPath views an XML document as a tree9Root of the tree is a new node, which doesn’tcorrespond to anything in the document9Internal nodes are elements;9Leaves are either Attributes, Text nodes, Comments; Or other things that we won't discuss (e.g.,processing instructions, )XML -- 43CSC343 -- Introduction to DatabasesXPath Document TreeRoot of XML treeRoot of XML documentXML -- 4422

CSC343 -- Introduction to Databases and Corresponding Document Æ A fragment of the report document used earlier: ?xml version “1.0” ? !-- Some comment -- Students Student StudId “111111111” Name First John /First Last Doe /Last /Name Status U2 /Status CrsTaken CrsCode “CS308” Semester “F1997” / CrsTaken CrsCode “MAT123” Semester “F1997” / /Student Student StudId “987654321” Name First Bart /First Last Simpson /Last /Name Status U4 /Status CrsTaken CrsCode “CS308” Semester “F1994” / /Student /Students !-- Some other comment -- XML -- 45CSC343 -- Introduction to DatabasesTerminologyÆParent/child nodes, as usual.ÆChild nodes (that are of interest to us) are oftypes text, element, attribute.ÆAncestor/descendant nodes – as usual intrees.XML -- 4623

CSC343 -- Introduction to DatabasesXPath BasicsÆAn XPath expression takes a document tree asinput and returns a multi-set of nodes of the tree.ÆExpressions that start with / are absolute pathexpressions9Expression / – returns root node of XPath tree;9 /Students/Student – returns all StudentStudentelements that are children of Students elements,which in turn must be children of the root;9 /Student – returns empty set (no such childrenat root).ÆThe basic idea here is similar to that of directorypaths.XML -- 47CSC343 -- Introduction to DatabasesMore XPath BasicsÆCurrent (or context node) – exists during theevaluation of XPath expressions (and in otherXML query languages)Æ . – denotes the current node; . – denotes theparent foo/bar – returns all bar-elementsthat arebarchildren of foo nodes, which in turn arechildren of the current node; ./foo/bar – same; ./abc/cde – all cde e-children of abc echildren of the parent of the current node.ÆExpressions that don’t start with / are relative(to the current node).XML -- 4824

CSC343 -- Introduction to DatabasesAttributes, Text, etc.Denotes anattributeÆ/Students/Student/@StudentId – returns allStudentId a-children of Student,Student which are echildren of Students,Students which are children of theroot.Æ/Students/Student/Name/Last/text( ) – returns allt-children of Last e-children of ÆXPath provides means to select other documentcomponents as well.XML -- 49CSC343 -- Introduction to DatabasesBasic Idea and SemanticsÆ An XPath expression is: locationStep1/locationStep2/ locationStep1/locationStep2/ Æ Location step:step Axis::nodeSelector[predicate]Æ Navigation axis:axis9 child, parent – have seen;9 ancestor, descendant, ancestor-or-self, descendantor-self – will see later;This is called full (rather9 some other -- will see later.than abbreviated) syntax.Æ Node selector:selector node name or wildcard; e.g.,9 ./child::Student (we used ./Student, which is anabbreviation)9 ./child::* – any e-child (abbreviation: ./*)Æ Predicate:Predicate a selection condition; e.g.,Students/Student[CourseTaken/@CrsCode “CSC343”]XML -- 5025

CSC343 -- Introduction to DatabasesComplete Set of AxesÆChild — the children of the context nodeÆDescendants — all descendants (children );ÆParent — the parent (empty if at the root)ÆAncestor — all ancestors from the parent to the rootÆFollowing-sibling — siblings to the rightÆPreceding-sibling — siblings to the leftÆFollowing — all following nodes in the document,excluding descendantsÆPreceding — all preceding nodes in the document,excluding ancestorsÆAttribute — the attributes of the context nodeÆNamespace — namespace declarations in contextnodeÆSelf — the context node itselfÆdescendant-or-self — the union of descendant andselfÆancestor-or-self — the union of ancestor and selfXML -- 51CSC343 -- Introduction to DatabasesAxisDirectionsXML -- 5226

CSC343 -- Introduction to DatabasesNode TestsÆTesting by node type:9text() — chardata node;9comment() — comment node;9processing-instruction() — processinginstruction node;9node() — any node (not including attributesand namespace declarations);ÆTesting by node name:9Name — nodes with that name9* — any nodeXML -- 53CSC343 -- Introduction to DatabasesEssential PredicatesÆ[attribute::name "flour"]: test equality of an attributeÆ[attribute::name! "flour"]: test inequality of anattributeÆ[attribute::amount '0.5' and attribute::unit 'cup']: testtwo things at once (also or)Æ[position() 2]: test position among siblingsÆ[attribute::amount '0.5']: a syntax errorÆ[attribute::amount<'0.5']: a useless test oflexicographical ]: whatyou meant to write instead!An entire location path may be used as a predicateÆ[attribute::amount]: the node has an amount attributeÆ[descendant::ingredient]: the node has a nestedingredientXML -- 5427

CSC343 -- Introduction to DatabasesXPath SemanticsThe meaning of the expression locStep1/locStep2/ isthe set of all document nodes obtained as follows:9Find all nodes reachable by locStep1 from thecurrent node;9For each node N in the result, find all nodesreachable from N by locStep2; take the union ofall these nodes;9For each node in the result, find all nodesreachable by locStep3,locStep3 etc.;9The value of the path expression on a documentis the set of all document nodes found afterprocessing the last location step in theXML -- 55expression.CSC343 -- Introduction to Databases More Generally ÆlocationStep1/locationStep2/ means:9Find all nodes specified by locationStep19For each such node N: Find all nodes specified by locationStep2using N as the current node Take union9For each node returned by locationStep2 do thesame using locationStep3, ÆlocationStep axis::node[predicate]9Find all nodes specified by axis::node9Select only those that satisfy predicateXML -- 5628

CSC343 -- Introduction to DatabasesMore Navigational PrimitivesÆSecond CrsTaken child of first Student child ofStudents:Students/Students/Students Student[1]/Student CrsTaken[2]CrsTakenÆAll last CourseTaken elements within eachStudent element:/Students/Student/CrsTaken[last( )]ÆAll href attributes in cite elements in the first 5sections of an article document:child::section[position() 6] / descendant::cite /attribute::hrefXML -- 57CSC343 -- Introduction to DatabasesWildcardsÆWildcards are useful for unknown document structures.ÆThe // wildcard descends down any number of levels(including 0):9 //CrsTaken – all CrsTaken nodes under the root;9 Students//@Name – all Name attribute nodes underthe elements Students, who are children under thecurrent node.ÆNote: ./Last and Last are same; but .//Last and //Lastare different.ÆThe * wildcard: * – any element:Student/*/text() @* – any attribute: Students//@*XML -- 5829

CSC343 -- Introduction to DatabasesSelection PredicatesÆRecall: Location step Axis::nodeSelector[predicate]ÆPredicate:9XPath expression const built-in function XPath expression (equality predicate);9XPath expression (returns false if result is empty);9 built-in predicate;9 a Boolean combination thereof;ÆAxis::nodeSelector[predicate] Axis::nodeSelectorbut contains only the nodes that satisfy predicate.ÆBuilt-in predicates include ones for string matching,set manipulation, etc. Built-in function include largeassortment of functions for string manipulation,aggregation, etc.XML -- 59CSC343 -- Introduction to DatabasesXPath Queries – ExamplesÆStudents who have taken CSC343://Student[CrsTaken/@CrsCode “CSC343”]ÆComplex example://Student[Status “U3” and starts-with(.//Last,“A”)and contains(concat(.//@CrsCode),“ESE”)and not(.//Last .//First) ]ÆAggregation: sum( ), count( )//Student[sum(.//@Grade) div count(.//@Grade) 3.5]XML -- 6030

CSC343 -- Introduction to DatabasesXPath Queries cont’dÆTesting whether a subnode exists:9//Student[CrsTaken/@Grade] – students whohave a grade (for some course)9//Student[Name/First or CrsTaken/@Semesteror Status/text() “U4”] – studentswho have either a first name or have taken acourse in some semester or have status U4ÆUnion operator, :9//CrsTaken[@Semester “F2001”] //Class[Semester “F1990”]union lets us define heterogeneous collections ofnodes.XML -- 61CSC343 -- Introduction to DatabasesXQuery – XML Query LanguageÆIntegrates XPath with earlier proposed querylanguages: XQL, XML-QLÆSQL-style, not functional-styleÆ2004: XQuery 1.0XML -- 6231

CSC343 -- Introduction to DatabasesAn ExampleXML -- 63CSC343 -- Introduction to DatabasesSomeQueriesXML -- 6432

CSC343 -- Introduction to Databasestranscript.xml Transcripts Transcript Student StudId “111111111” Name “John Doe” / CrsTaken CrsCode “CS308” Sem “F97” Gr “B” / CrsTaken CrsCode “MAT123” Sem “F97” Gr “B” / CrsTaken CrsCode “EE101” Sem “F1997” Gr “A” / CrsTaken CrsCode “CS305” Sem “F1995” Gr “A” / /Transcript Transcript Student StudId “987654321” Name “Bart Simpson” / CrsTaken CrsCode “CS305” Sem “F1995” Gr “C” / CrsTaken CrsCode “CS308” Sem “F1994” Gr “B” / /Transcript cont’d XML -- 65CSC343 -- Introduction to Databasestranscript.xml (cont’d) Transcript Student StudId “123454321” Name “Joe Blow” / CrsTaken CrsCode “CS315” Sem “S97” Gr “A” / CrsTaken CrsCode “CS305” Sem “S96” Gr “A” / CrsTaken CrsCode “MAT123” Sem “S96” Gr “C” / /Transcript Transcript Student StudId “023456789” Name “Homer Simpson”/ CrsTaken CrsCode “EE101” Sem “F1995” Gr “B” / CrsTaken CrsCode “CS305” Sem “S1996” Gr “A” / /Transcript /Transcripts XML -- 6633

CSC343 -- Introduction to DatabasesXQuery BasicsXQueryÆ General structure:expressionFORvariable declarationsWHERE conditionRETURN documentcommentÆ Example:(: students who took MAT123 :)See next slideFOR t criptWHERE t/CrsTaken/@CrsCode “MAT123”RETURN t/StudentÆ Result: Student StudId “111111111” Name “John Doe” / Student StudId “123454321” Name “Joe Blow” / XML -- 67CSC343 -- Introduction to DatabasesXQuery Basics (cont’d)Æ Previous query doesn’t produce a well-formed XMLdocument; the following does:Query inside StudentList XML{FOR t IN doc(“transcript.xml”)//TranscriptWHERE t/CrsTaken/@CrsCode “MAT123”RETURN t/Student} /StudentList Æ FOR binds t to Transcript elements one by one, filtersusing WHERE, then places Student-childrenas eStudentchildren of StudentList using RETURN.XML -- 6834

CSC343 -- Introduction to DatabasesDoc Restructuring with XQueryÆ Reconstruct lists of students taking each class usingthe Transcript records:FOR c IN N ClassRoster CrsCode { c/@CrsCode}Sem { c/@Sem} { FOR t IN doc(“transcript.xml”)//TranscriptWHERE t/CrsTaken/[@CrsCode c/@CrsCodeand @Semester c/@Sem]RETURN t/Student ORDER BY t/Student/@StudId} /ClassRoster ORDER BY c/@CrsCodeXML -- 69CSC343 -- Introduction to DatabasesDocument Restructuring (cont’d)Æ Output elements have the form: ClassRoster CrsCode “CS305” Sem “F1995” StudentStudId “111111111”Name “John Doe” / StudentStudId “987654321”Name “BartSimpson”/ /ClassRoster Æ Problem: the above element will be output twice –John Doe’sonce when c is bound to CrsTaken CrsCode “CS305” Sem “F1995” Grade “A” / Bart Simpson’sand once when it is bound to CrsTaken CrsCode “CS305” Sem “F1995” Grade “C” / Æ Note: grades are different – distinct( ) won’t eliminatetranscript records that refer to same class!XML -- 7035

CSC343 -- Introduction to DatabasesDocument Restructuring (cont’d)ÆSolution: instead ofFOR c Document onnext slideFOR c IN doc(“classes.xml”)//Classclasses.xmlwhere clas

CSC343 -- Introduction to Databases Weeks 7 and 9: XML Data and Query Processing Semistructured Data, HTML XML and DTDs XPath, XQuery XML -- 2 CSC343 -- Introduction to Databases Hypertext ÆMost human knowledge exists today in document format (books etc.) ÆNeed technologies that s