Autonomous Systems - A Rigorous Architectural Characterization
Autonomous Systems - A RigorousArchitectural Characterization2019 IEEE SERVICES CongressMilano, July 9, 2019Joseph SifakisVerimag Laboratory
Next-generation autonomous systems – The IoT VisionThe IoT allows objects to be sensed or controlled remotely across a networkinfrastructure, achieving more direct integration of the physical world intocomputer-based systems, and resulting in improved efficiency and predictability.2
Next-generation autonomous systems – The IoT VisionThe Internet of ThingsIndustrial IoT Human IoT People’s explicit orRules can bechanged, buthuman-drivenAutonomouschanges areexternal to normal behaviorAutonomous transport systemsIndustry 4.0Smart gridsInteractivearbitrary actionsdynamically triggercontrol sequences orrule changesIntelligent servicesSemantic web
Next-generation autonomous systems – Main CharacteristicsNext-generation autonomous systems emerge from the needs to further automateexisting complex organizations by progressive and incremental replacement of humanagents by autonomous agents.Such systems exhibit “broad intelligence” by using and producing knowledge in orderto Manage dynamically changing sets of potentially conflicting goals – thisreflects the trend of transitioning from “narrow” or “weak” AI to “strong” or“general” AI. Cope with uncertainty of complex and unpredictable environments Harmoniously, collaborate with human agents e.g. “symbiotic” autonomy.The dystopian AI mythWhen should we trust machines that can make mistakes and are notaccountable for their behavior?
Next-generation autonomous systems – Current limitations Criticality requirements for next-generation autonomous systems cannot beachieved under the current state of the art poor trustworthiness of infrastructures and systems e.g. impossibility toguarantee safety and security; impossibility to guarantee response times in communication thus timelinesswhich is essential for autonomous reactive systems; Integration of mixed-criticality systems is hard to achieve because criticalsystems and best-effort systems are developed following two completelydifferent and diverging design paradigms; New practices emerge Extensive use of learning-enabled components breaking with the traditionalcritical systems engineering practice – end-to-end AI-based solutions; In contrast with the current systems engineering practice (*), criticalsoftware is customized by updates – Tesla cars software may be updatedon a monthly basis.(*) An aircraft is certified as a product that cannot be modified including all itscomponents even HW – aircraft makers purchase and store an advance supply ofthe microprocessors that will run the software, sufficient to last for the estimated 50year production!
Next-generation autonomous systems – Facing the challengeSystems Engineering comes to a turning point moving from small size centralized nonevolvable automated systems to next-generation autonomous systems We need a general reference semantic model that could be a basis for evaluatingsystem autonomy - Not just a list of “self”-prefixed terms e.g. as Self-healing, Selfoptimized, Self-protected, Self-aware, Self-organized, etc. What are the technical solutions for enhancing a system’s autonomy?For each enhancement, what are the implied technical difficulties and risks? There is a strong and urgent need to lay out a common engineering foundation forthe development of next-generation autonomous systems.Essential issues to be addressed:1. integration of model-based and data-driven techniques in “hybrid” design flowsallowing to determine trade offs between trustworthiness and performance;2. means for faithful modeling and simulation of a system in its physicalenvironment (which includes humans);3. combine empirical and proof-based validation for assessing trustworthinessand performance – open the way for new standards.
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward7
The Concept of Autonomy – Basic DefinitionsAn autonomous system involves two different types of components, agents and objects,operating in a common environment so that their coordinated collective behavior meetssome global goals An agent is a reactive system (controller) interacting with components of itsenvironment so that specific goals are met; It can monitor objects and from theirenvironment and change their states and can coordinate its actions with otheragents. An object is a physical or virtual component whose behavior can be controlledby system agents i.e. it is integrated as such when the system is designed The environment consists of the elements of the physical and virtualinfrastructure of the system that are used for the coordination betweencomponents (agents and objects) e.g. geographic coordinates to determineconnectivity relationships, available communication infrastructure, devices forobservability/controllability of objectsNote that A component may be agent or object depending on its role in the system It is an interesting question indeed how are related system and agent goals
The Concept of Autonomy – Basic DefinitionsSYSTEM nt1Agent2Agent1InternalEnvnt1InternalEnvnt2SYSTEM Agents Objets System EnvironmentAgents Agent1 Agent2Objects Traffic light Pedestrian Human Driven carSystem Environment (External Envnt1 External Envnt2)x(Internal Envnt1 Internal Envnt2)
The Concept of Autonomy – Find the DifferencesThermostatAutomatic train shuttleSoccer-playing robotChess-playing robotRobocarEach system consists of agents acting as controllers on their environmentand pursuing individual goals so that the collective behavior meets thesystem global goals.
The Concept of Autonomy – Meeting GoalsGiven a set of goals and the model of an environment to be controlled, there aremethods for computing plans enforcing the satisfaction of the goals.Initc1c2GOALS- Never reach Bad- Eventually reach c3TargetPLANA (possibly infinite)tree with alternatingcontrollable anduncontrollable actionsENVIRONMENT MODELA (possibly infinite) state graphwith controllable (green) anduncontrollable (red) actions
The Concept of Autonomy – From Automation to AutonomyEnvironmentStimuliMeeting GoalsRoom Heating/coolingdeviceTemperatureExplicit controllerCars Passengers equipmentDynamicconfiguration ofcars State of equipmentExplicit controller on line adaptationStaticconfiguration ofpawnsOn-line planning stored knowledgeDyn. Changing goalsSoccer robot Regions in the field DynamicPlayers Ballconfiguration ofplayers/ballOn-line planning stored/generatedknowledgeDyn. changing goalsRobocarOn-line planning stored/generatedknowledgeDyn. changing goalsThermostatShuttleChess robotChess board pawnsSingle goalVehicles/obstacles DynamicRoad/communication configuration ofequipmentvehicles/obstacles State of equipmentMany fixed goals
The Concept of Autonomy – Architectural CharacterizationSituation entExternalSensorsExternalEnvironmentAdaptive DecisionGoal onsensory informationExternalActuators
The Concept of Autonomy – Architectural Characterization Autonomy is the capacity of an agent to achieve a set of coordinated goals by itsown means (without human intervention) adapting to environment variations. Itcombines five complementary functions: Perception e.g. interpretation of stimuli, removing ambiguity from complexinput data and determining relevant information; Reflection e.g. building/updating a faithful environment run-time model fromwhich strategies meeting the goals can be computed; Goal management e.g. choosing among possible goals the most appropriateones for a given configuration of the environment model; Planning to achieve a particular goal; Self-awareness/adaptation e.g. the ability to create new situational knowledgeand new goals through learning and reasoning These functions are implementation-agnostic Insights on Automation vs. Autonomy; Human-assisted vs. Machine Empowered autonomy
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward15
Trusting Autonomous Systems – Autonomy LevelSAE AYTONOMY LEVELSLevel 0No automationLevel 1Driver assistance required (“hands on”)The driver still needs to maintain full situational awareness and control of thevehicle e.g. cruise control.Level 2Partial automation options available(“hands off”)Autopilot manages both speed and steering under certain conditions, e.g.highway driving.Level 3Conditional Automation(“eyes off”)The car, rather than the driver, takes over actively monitoring the environmentwhen the system is engaged. However, human drivers must be prepared torespond to a "request to intervene”Level 4High automation (“mind off”)Self driving is supported only in limited areas (geofenced) or under specialcircumstances, like traffic jamsLevel 5Full automation (“steering wheel optional”)No human intervention is required e.g. a robotic taxi
Trusting Autonomous Systems – Autonomy LevelSelf-awarenessHuman AssistedAutonomy
Trusting Autonomous Systems –The Automation FrontierTask Criticality1TrustedHumanTrustedSystem0System Trustworthiness1How we decide whether a System can be trusted for performing a Task: SystemTrustworthiness: the system will behave as expected despite any kind ofmishaps e.g. resilience to errors, failures, attacks. Task Criticality: characterizes the severity of the impact of an error in the fulfilment ofthe task e.g. driving a car, operating on a patient, nuclear plant control.
Trusting Autonomous Systems – Automated vs.Non-automatedTask ainDrivingSystemNursing lTrustedSystemSystem Trustworthiness1Automated systems: simple decision process or small impact of failures.Non-automated systems: require good situation awareness and multiplegoal management.
Trusting Autonomous Systems – Symbiotic SystemsCarDrivingTask stem Trustworthiness 1 Autonomous systems extensively use knowledge; they cannot be effectivelyimplemented without massive use of AI-based techniques. Problem: choose the appropriate degree of autonomy (machine empowered vs.human-assisted operation e.g. SAE degrees of autonomy for vehicles ).
Trusting Autonomous Systems – The Role of InstitutionsSocial acceptance of Truth is a complex process where institutions play an important role100 years afterGalileo is WRONG!!Galileo is RIGHT!! Institutions shape public perceptions about what is TRUE, RIGHT, SAFE, etc In modern societies independent institutions guarantee trustworthiness of technicalinfrastructure and common services based on standards and regulations e.g. FDA.,FAA, NTHSA, in the US. Most critical systems standards require conclusive model-based evidence e.g.based on the laws of Physics a bridge will not collapse for a century. Suchstandards not applicable to AI-based systems – self-driving cars are “self-certified”!
Trusting Autonomous Systems – Shaping Factors1Task CriticalityTrustedHumanTrustedSystem01System TrustworthinessPerformance: for low criticality, trade quality of service for performance;Bias: human error is more acceptable than machine failure.
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward23
Hybrid Design Flows – The PrincipleDesign-timeData-based approach(Sufficient evidence)Deployment“Hybrid” approach(Guarantees Sufficient evidence )DeploymentModel-based dRun-time assuranceRun-time AssuranceExecution PlatformLearning-enabled AgentPERFORMANCEDIR mechanismsExecution Platform“Hybrid” Autonomous AgentRun-timeExecution PlatformAutomated AgentTRUSTWORTHINESS 24
Hybrid Design Flows – Model-based Trustworthiness Current approaches guarantee trustworthiness at design time by applying a more or less exhaustive risk analysis that identifies all kind of harmful events FataltechniquesStatesguaranteeing tolerance: any single harmful event leads to non-fatal states DIR (Detection, Isolation, Recovery) mechanisms leading from non-fatal states totrustworthy states These approaches cannot be directly applied to autonomous systems Lack of predictability and environment complexity make practically impossibleidentification at design time of all harmful events and corresponding DIR mechanisms Use of learning-enabled componentsNon-Trustworthy StatesTrustworthy StatesNominalBehaviorFatal StatesNon-Fatal StatesXNon-fatalState25
Hybrid Design Flows – Model-based TrustworthinessPre-crash failure typology covering 99.4% of light-vehicle crashes for 5,942,000 cases.Source: Pre-Crash Scenario Typology for Crash Avoidance Research, DOT HS 810 767, April 2017.FDIR approaches are not anymore applicable due to overwhelming complexity!
Hybrid Design Flows – Model-based GuaranteesMobileye’s Responsibility-Sensitive Safety: Compute lower bounds of the distancebetween two cars that guarantee safety. (“On a Formal Model of Safe and ScalableSelf-driving Cars” Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua,Mobileye, 2017)See also “The Safety Force Field” David Nistér, Hon-Leung Lee, JuliaNg, Yizhou Wang, Nvidia White Paper, March 2019
Hybrid Design Flows – Control for Safety and PerformanceThe general problem:1. An agent provides critical services and possibly some non-critical services.2. The agent uses a variable amount of free resources F (measured in space, time,memory, energy, etc.) such that Fmin F and 2F/ t2 amax Fmin is sufficient for the system to ensure the critical services Critical services should be absolutely ensured (safety) The rest of the available resources should be used in the best possiblemanner to ensure non critical services (performance). Safety cannot be dissociated from performance e.g. overtaking on a two lane road The problem needs to be solved for a humongous number of configurations: use learning-enabled techniques to recognize types of configurations for each identified type, apply a model-based protocol
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward29
Modeling and Simulation – Basic Modeling ConceptsCurrently, most simulation systems use ad-hoc techniques coupling an autonomousmonolithic agent to game SW. They lack features for Building scenarios that capture behavior corner cases and high risk situations Building environment models incrementally and compositionally Different levels of abstraction from fine grain simulation of cyber physicalcomponents to high level simulationWhat is the value of results reported by Waymo: 27 000 cars running 24/7, 10million miles simulated per day, 7 Billion miles of simulation.We need component-based modeling frameworks integrating:1. Libraries of component types for both agents and objects, as well as librariesof architecture patterns and protocols;2. Expressive component coordination primitives supporting parametricdescription and various types of dynamism such as componentcreation/deletion and mobility;3. Self-organization by supporting multi-mode coordination e.g. a componentcan live in many different “worlds” and migrate according to pursued goals.4. Knowledge management and application for situational awareness andgeneration of new goals accordingly.
Modeling and Simulation – State-aware SimulationDR-BIP (Dynamic Reconfigurable BIP)MOTIFMap A system is a set of (architecture) motifsAddress function: @C1C2C3C4Component instancesConfiguration rulesInteraction rules A motif is a coordination mode consisting of A set of components, instances of types of agentsor objects A map that is a graph (N,E) used to describerelations between components e.g. geographical,organizational, etc. An address function @ mapping components intonodes of the map Interaction rules: define interactions (atomicmultiparty synchronization) between components Configuration rules:- Mobility of components (change of @)- Creation/deletion of components- Dynamic change of the mapThe meaning of systems models is defined using operational semantics
Model-based Approach – State-aware SimulationInteraction rule:for all a,a’:vehicle, if [dist(@(a),@(a’)) l] then exchange(a.speed,a’.speed).Mobility rule :for all a:vehicle if @(a) n and @-1(n 1) empty then @(a): n 1.
Model-based Approach – Refined Agent ModelPerceptionsensory informationKnowledgeRepository Agent typesObject TypesMap enerationSensorsEnvironmentmodelAgent’s Environment geapplicationGoalmanagementPlanningEnvironment ModelcommandsActuators
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward34
In Search of a Foundation – Validation Machine learning techniques cannot be formally verified as they are not developedbased on formal goals e.g. specifying how a dog looks different from a cat instead, we are showing a whole bunch of pictures so they can learn just like ahuman learns the differences between a cat and a dog. Pushing model-based validation techniques to the limits Increasing confidence in ML-models which remain mostly “black boxes” Metamorphic testing: φ1, φ2 if y f(x) then φ2(y) f(φ1(x)) Determining reference models (oracles) i.e. interpretability, explainability,“causal modeling” Combining proof-based and empirical validation techniques
In Search of a Foundation – Model-based ValidationFormalization of goals for autonomous systems is extremely hard e.g. “behavioralcompetencies” for self-driving cars (California 19.20.21.22.23.24.25.Detectand Respondto Speed LimitChangesand SpeedAdvisories1.Detectand Respondto SpeedLimitChangesand Speed AdvisoriesPerform High-Speed Merge (e.g., Freeway)Perform Low-Speed MergeMove Out of the Travel Lane and Park (e.g., to the Shoulder for Minimal Risk)Detect and Respond to Encroaching Oncoming VehiclesDetectPassingand No Passingand PerformPassing6.DetectPassingand NoZonesPassingZonesand ManeuversPerform Passing ManeuversPerform Car Following (Including Stop and Go)Detect and Respond to Stopped VehiclesDetect and Respond to Lane ChangesDetect and Respond to Static Obstacles in the Path of the VehicleDetect Traffic Signals and Stop/Yield SignsRespond to Traffic Signals and Stop/Yield SignsNavigateIntersectionsand PerformandTurnsPerform Turns13.NavigateIntersectionsNavigate RoundaboutsNavigate a Parking Lot and Locate SpacesDetect and Respond to Access Restrictions (One-Way, No Turn, Ramps, etc.)Detect and Respond to Work Zones and People Directing Traffic in Unplanned or Planned ppropriateRight-of-WayDecisionsFollow Local and State Driving LawsFollow Police/First Responder Controlling Traffic (Overriding or Acting as Traffic Control Device)Follow Construction Zone Workers Controlling Traffic Patterns (Slow/Stop Sign Holders).Respond to Citizens Directing Traffic After a CrashDetect and Respond to Temporary Traffic Control DevicesDetect and Respond to Emergency VehiclesYield for Law Enforcement, EMT, Fire, and Other Emergency Vehicles at Intersections, Junctions, and Other TrafficControlled Situations26. Yield to Pedestrians and Bicyclists at Intersections and Crosswalks27. Provide Safe Distance From Vehicles, Pedestrians, Bicyclists on Side of the RoadDetect/RespondDetoursand/or OtherTemporaryChanges in Traffic Patterns28. 28.Detect/Respondto Detourstoand/orOther TemporaryChangesin Traffic Patterns36
Rigorous System Design – Model-based Validation Formal verification is applicable when goals that can be explicitly formalized as requirements Is tractable for moderate model complexity - only monolithic verificationtechniques of finite state systems can be automated; Is not enough! Autonomy is about controller synthesis under both safety andoptimization constraints; A more natural approach is to achieve correctness by design. The V-model, Systems Engineering Process recommended by Safety Standardssuch as ISO262621. assumes that all the system requirements are initiallyknown, can be clearly formulated and understood.2. assumes that system development is top-down from aset of requirements. Nonetheless, systems are neverdesigned from scratch; they are built by incrementallymodifying existing systems and component reuse.3. considers that global system requirements can bebroken down into requirements satisfied by systemcomponents.37
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Discussion Valuing knowledge The way forward38
Discussion – An Interesting AnalogyFast thinking vs. Slow thinking (D. Kahneman’s “Thinking Fast and Slow”)System 1: “Fast” Thinking Non-conscious – automatic – effortless; Without self-awareness or control; Handles all kind of empirical implicitknowledge e.g. walking, speaking,playing the pianoSystem 2: “Slow” Thinking Conscious – controlled– effortful; With self-awareness and control Is the source of any reasonedknowledge e.g. mathematical,scientific, technical.Neural Networks vs. Conventional ComputersNEURAL NETWORKALGORITHMstep1step2YES Generate empirical knowledge aftertraining (Data-based knowledge). Distinguish “cats from dogs” exactly askids do – Cannot be verified!NO Execute algorithms (Model-basedknowledge) . Deal with explicitly formalizedknowledge – Can be verified!
Discussion – The Knowledge Hierarchy tics,ComputingMathematics, ComputingScientific&TechnicalKnowledgeImplicit Empirical KnowledgeFacts and Syllogisms
Discussion – The Knowledge Hierarchy athematics, taknowledgeScientific&TechnicalKnowledgeMachine KnowledgeLearningML-basedData AnalyticsImplicit Empirical KnowledgeFacts and Syllogisms
Discussion – Scientific vs. ML-generated Knowledge1. EXPERIMENTF1,F2, FnF(Force)2. LEARNINGa1,a2, anF ma(model)m(mass)a(Acceleration)GalileoNEURAL NETWORKI1,I2, Inc1,c2 cn3. EXPLANATIONImage?{Cat,Dog}Image
Discussion – Scientific vs. Machine-generated KnowledgeLimitations of the scientific approach1. Phenomena are explainable provided we have the adequate mathematical model2. Cognitive complexity: there is a limit in the size of the relations that human mindcan deal with: relations of rank five (one predicate four arguments)3. We are “lucky”: basic physical laws are easy to understand !!BUT our lack of understanding of complex phenomena does not necessarily meanthat they are not subject to laws – Simply their complexity exceeds our cognitivecapabilitiesCan computers help overcome these limitations?For many domains of knowledge e.g. earth sciences, epidemiology, economics,phenomena are irreducibly complex and depend on a large number of parameters. The development of all encompassing theoretical models seems practicallyimpossible. -Theories are necessarily partial as they consider drastic abstractions. Computers allow the validation of empirical models e.g. combining theoretical andad hoc models. The combination of data analytics (actionable knowledge) and machine learning(expert knowledge) can help study complex phenomena and predict their behavior.
Autonomous Systems The concept of autonomy Should we trust autonomous systems? In Search of a Foundation “Hybrid” design flows Modeling and Simulation ValidationOVERVIEW Complexity Issues Autonomic Complexity Design Complexity Discussion The value of knowledge The way forward44
Discussion – Standards for Next-Gen Autonomous Systems Autonomy should be associated with functionality and not with specific techniques– while ML is essential it is not only way to build perceptors and adaptivecontrollers. Current trends render obsolete conventional critical systems engineering principlesand standards such as such as ISO26262 and DO178B , that require conclusiveevidence that the system can cope with any type of harmful event. they cannot handle machine learning components; they cannot handle design flows for autonomous systems – they give asystem credit for a human assistant ultimately being responsible for safety. they require guarantees at design time and stringent predictability that areimpossible to provide IoT autonomous systems. Consequently, there is no Independent safety certification for autonomoussystems! Automotive and medical products are self-certified by their manufacturersaccording to guidelines that determine how to provide sufficient evidencethat the developed system is reliable enough.
Discussion – Should be worried about dystopian AI futures?The role of AI systems will depend on choices we make about when we trustthem and when we do not. Making these choices wisely1. is a matter of social awareness and of sense of political responsibility: When machines use knowledge in critical decision processes makesure that it is truthful, unbiased, neutral, fair, etc. (precautionaryprinciple). Always question motives, objectives and biases of existing systems.2. requires new scientific foundations allowing the development of trustevaluation tools We need a “new kind of scientific approach” based on a « hybrid »model-based and data-based approach seeking tradeoffs betweentrustworthiness and performance. We should develop and apply rigorous regulations and standards forthe development and use of such systems (as for all artifacts fromtoasters to bridges and aircraft).No self-regulation, no self-certification !!Building trustworthy next-generation autonomous systems goes for farbeyond the current AI challenge.
Thank YouJoseph SifakisAutonomous Systems -- An Architectural Characterization,November 2018https://arxiv.org/abs/1811.10277
xt-generation autonomous systems - Main Characteristics . Next-generation autonomous systems emerge from the needs to further automate existing complex organizations by progressive and incremental replacement of human agents by autonomous agents. Such systems exhibit "broad intelligence" by using and producing knowledge in order to
Page 2 Autonomous Systems Working Group Charter n Autonomous systems are here today.How do we envision autonomous systems of the future? n Our purpose is to explore the 'what ifs' of future autonomous systems'. - 10 years from now what are the emerging applications / autonomous platform of interest? - What are common needs/requirements across different autonomous
Autonomous Differential Equations 1. A differential equation of the form y0 F(y) is autonomous. 2. That is, if the right side does not depend on x, the equation is autonomous. 3. Autonomous equations are separable, but ugly integrals and expressions that cannot be solved for y make qualitative analysis sensible. 4.
Florida Statutes - Autonomous Vehicles (2016) HB 7027, signed April 4. th. 2016 - updates: F.S. 316.85 -Autonomous Vehicles; Operation (1) "A person who possesses a valid driver license may operate an autonomous vehicle in autonomous mode on roads in this state if the vehicle is equipped with autonomous technology, as defined in s. 316. .
systems to autonomous systems. Firstly, the autonomy and autonomous systems in different fields are summarized. The article classifies and summarizes the architecture of typical automated systems and infer three suggestions for building an autonomous system architecture: extensibility, evolvability, and collaborability.
2 Research and development case study: Robotics and autonomous systems research Introduction This case study on robotics and autonomous systems research is one of a series that we have developed to support and complement our published report on research and development. Our examination of robotics and autonomous systems research
text, autonomous robots and mobile manipulators form the forefront of recent developments. In the Master's Program 'Autonomous Systems' (MAS) students will learn the practical skills and intellectual abilities necessary for the design and develop-ment of such autonomous systems. This Master's Program takes the form of a "Master by Research". So .
of autonomous military robots, or “autonomous weapon systems.” The treatment of such subjects in the ethics, robotics, and popular literature has generally assumed that autonomous systems either fit perfectly into existing legal regimes or threaten long-standing paradigms. This Article demonstrates that neither assumption is correct.
Annual Thanksgiving Service at St Mark’s Church St Mark’s Rise, Dalston E8 on Sunday 19 September 2004 at 4 pm . 2 . Order of Service Processional hymn — all stand All things bright and beautiful, All creatures great and small, All things wise and wonderful: The Lord God made them all. Each little flower that opens, Each little bird that sings, He made their glowing colors, He made their .
