Scenic: Language-Based Scene Generation
Scenic: Language-Based Scene GenerationDaniel FremontXiangyu YueTommaso DreossiShromona GhoshAlberto L. Sangiovanni-VincentelliSanjit A. SeshiaElectrical Engineering and Computer SciencesUniversity of California at BerkeleyTechnical Report No. TechRpts/2018/EECS-2018-8.htmlApril 18, 2018
Copyright 2018, by the author(s).All rights reserved.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission.
Scenic: Language-Based Scene GenerationDaniel J. FremontXiangyu YueTommaso DreossiLogic and the Methodology of ScienceUniversity of California, Berkeleydfremont@berkeley.eduElectrical Engineering and ComputerSciencesUniversity of California, Berkeleyxyyue@berkeley.eduElectrical Engineering and ComputerSciencesUniversity of California, Berkeleytommasodreossi@berkeley.eduShromona GhoshAlbertoSangiovanni-VincentelliElectrical Engineering and ComputerSciencesUniversity of California, Berkeleyshromona.ghosh@berkeley.eduElectrical Engineering and ComputerSciencesUniversity of California, Berkeleyalberto@berkeley.eduSanjit A. SeshiaElectrical Engineering and ComputerSciences & Logic and theMethodology of ScienceUniversity of California, Berkeleysseshia@berkeley.eduFigure 1: Three scenes generated from a single Scenic scenario representing bumper-to-bumper traffic.ABSTRACTKEYWORDSSynthetic data has proved increasingly useful in both training andtesting machine learning models such as neural networks. The major problem in synthetic data generation is producing meaningfuldata that is not simply random but reflects properties of real-worlddata or covers particular cases of interest. In this paper, we showhow a probabilistic programming language can be used to guide datasynthesis by encoding domain knowledge about what data is useful.Specifically, we focus on data sets arising from scenes, configurations of physical objects: for example, images of cars on a road. Wedesign a domain-specific language, Scenic, for describing scenariosthat are distributions over scenes. The syntax of Scenic makes iteasy to specify complex relationships between the positions andorientations of objects. As a probabilistic programming language,Scenic allows assigning distributions to features of the scene, aswell as declaratively imposing hard and soft constraints over thescene. A Scenic scenario thereby implicitly defines a distributionover scenes, and we formulate the problem of sampling from thisdistribution as scene improvisation. We implement an improviser forScenic scenarios and apply it in a case study generating syntheticdata sets for a convolutional neural network designed to detect carsin road images. Our experiments demonstrate the usefulness of ourapproach by using Scenic to analyze and improve the performanceof the network in various scenarios.scenario description language, synthetic data, deep learning, probabilistic programming, automatic test generation, fuzz testing1INTRODUCTIONMachine learning (ML) is increasingly used in safety-critical applications, thereby creating an acute need for techniques to gainhigher assurance in ML systems [1, 34, 35]. The traditional MLapproach to this problem is to test the learned model1 in its environment, gathering more data, and retraining if performance isinadequate. However, collecting real-world data can be slow andexpensive, as it must be preprocessed and correctly labeled beforeuse. Furthermore, it can be hard to observe and reproduce cornercases that are rare but nonetheless necessary to test against (forexample, a car accident). As a result, recent work has investigatedtraining and testing models with synthetically generated data, whichcan be produced in bulk with correct labels and giving the designerfull control over the distribution of the data [17, 19, 39].Generating meaningful synthetic data can be a nontrivial tasksince the input space of ML models is often large and unstructured.This is certainly true of the domain we consider in this paper: scenescomprising configurations of objects in space. Suppose we wanted adata set consisting of images of cars on a road. If we simply sampleduniformly at random from all possible configurations of, say, 12cars, we would get data that was at best unrealistic, with cars facingsideways or backward, and at worst physically impossible, withcars intersecting each other. Instead, we want scenes like thoseUC Berkeley EECS Technical Report, April 18, 20182018.1 Theterm “model” is commonly used in machine learning to refer to the learnedclassifier/predictor, e.g., a neural network.1
UC Berkeley EECS Technical Report, April 18, 2018Fremont et al.Theft Auto V (GTAV2 ), a high fidelity graphics videogame. Ourexperiments illustrate several ways Scenic can be used: generating specialized test sets to assess the accuracy of theML system under particular conditions (e.g. in rain); generating instances of hard cases for the system so that theycan be emphasized when retraining, improving accuracy inthe hard case without impacting the typical case; generalizing a known failure case in many directions, exploring the sensitivity of the system to different features anddeveloping a more general scenario for retraining.These experiments show that Scenic is a very useful tool for understanding and improving the performance of an ML system.In summary, the main contributions of this work are: Scenic, a domain-specific probabilistic programming language for describing scenarios that are distributions overconfigurations of physical objects; scene improvisation, an approach to generating a diverseset of concrete scenes from a Scenic scenario that drawsinspiration from control improvisation [11]; an implementation of an improviser for Scenic scenarios,with an interface to GTAV for producing realistic images; a case study showing how Scenic can be used in practice toanalyze and improve the accuracy of SqueezeDet, a practicaldeep neural network for autonomous driving [42].The paper is structured as follows: we begin by discussing relatedwork in Sec. 2. Section 3 gives examples highlighting the majorfeatures of Scenic and motivating various choices in its design.In Sec. 4 we describe Scenic’s syntax and semantics in detail, andin Sec. 5 we discuss the scene improvisation problem. Section 6describes the experimental setup and results of our car detectioncase study. Finally, we conclude in Sec. 7 with a summary anddirections for future work.in Fig. 1, where the cars are laid out in a consistent and realisticway. Furthermore, we may want scenes that are not only realisticbut represent particular scenarios of interest, e.g., parked cars, carspassing across the field of view, or bumper-to-bumper traffic as inFig. 1. In general, we need a way to guide data generation towardscenes that make sense for our application.In this paper, we take a programming languages approach, designing a domain-specific scenario description language, Scenic. Ascenario is a distribution over scenes. Scenic is thus a probabilistic programming language, allowing the user to programmaticallyspecify a scenario of interest. Furthermore, it allows the user toboth construct objects in a straightforward imperative style andimpose hard and soft constraints declaratively. Scenic also providesreadable, concise syntax for common geometric relationships thatwould otherwise require complex nonlinear expressions and constraints. In addition, Scenic provides a notion of classes allowingproperties of objects to be given default values depending on otherproperties: for example, we can define a Car so that by default itfaces in the direction of the road at its position. Finally, Scenic provides an easy way to generalize a concrete scene by automaticallyadding noise.The variety of constructs in Scenic makes it possible to writescenarios anywhere on a spectrum from concrete scenes (i.e. individual test cases) to extremely broad classes of abstract scenes (seeFig. 2). A scenario can be reached by moving along the spectrumfrom either end: the top-down approach is to progressively constrain a very general scenario, while the bottom-up approach is togeneralize from a concrete example (such as a known failure case),for example by adding random noise.Probably most usefully,one can write a scenarioGeneric scenarioin the middle which is far(“a car on the road”)more general than simplyadding noise to a singleStructured scenarioscene but has much more(“a badly parked car”)structure than a completely random scene: forExample noiseexample, the traffic sce(“a car near 1.2 m 4 m”)nario depicted in Fig. 1.Concrete exampleWe will illustrate all three(“a car at 1.2 m 4 m”)ways of developing a scenario in this paper.Figure 2: Spectrum of scenariosGenerating concretefrom general to specific.scenes from a Scenic scenario requires samplingfrom the probability distribution it implicitly defines. This problem, while closely relatedto the general probabilistic programming inference problem [16], istheoretically interesting in its own right. We call it scene improvisation, as it can be viewed as a variant of control improvisation [10, 11],a class of problems involving the random generation of sequencessubject to hard and soft constraints as well as distribution requirements.Finally, we demonstrate the usefulness of Scenic with a casestudy testing and improving the reliability of a convolutional neuralnetwork designed to perform object detection in autonomous cars.We implemented an improviser for Scenic scenarios and used itto generate scenes which were rendered into images by Grand2RELATED WORKData Generation and Testing for ML. There has been a largeamount of work on generating artificial data sets for specific applications, including text recognition [19], text localization [17], roboticobject grasping [39], and autonomous driving [8, 20]. Closely related is work on domain adaptation, which attempts to correct differences between synthetic and real-world input distributions. Domainadaptation has enabled synthetic data to successfully train modelsfor several other applications including 3D object detection [23, 36],pedestrian detection [40], and semantic image segmentation [33].Such work provides important context for our paper, showing thatmodels trained exclusively on synthetic data sets (possibly domainadapted) can achieve acceptable performance on real-world inputs.The major difference in our work is that we do not focus on anyspecific application but provide, through Scenic, a general way tospecialize data generation for any application whose data derivesfrom scenes.Some works have also explored the idea of using adversarialexamples (i.e. misclassified examples) to retrain and improve models [41, 43]. Some of these methods generate misclassifying examples by looking at the model gradient and by finding minimal inputperturbations that lead to a misclassification [14, 26, 29, 38]. Othertechniques assume the model to be gray/black-boxes and focus on2 GTAV:2https://www.rockstargames.com/
Scenic: Language-Based Scene GenerationUC Berkeley EECS Technical Report, April 18, 2018input modifications or high-level properties of the model [5, 18,21, 31]. Finally, Generative Adversarial Networks (GANs) [13], aparticular kind of neural network able to generate synthetic data,have been used to augment training sets [22, 24]. The differencewith Scenic is that GANs require an initial training set or pretrained model, while Scenic produces synthetic data involvingonly simulators.Model-Based Test Generation. Techniques to guide generationtowards useful outputs have been proposed in both automatedtesting and procedural generation [3]. A popular approach is toprovide example outputs, as in mutational fuzz testing [37] andexample-based scene synthesis [9]. While these methods are easyto use, they do not provide fine-grained control over the generateddata. Another approach is to give rules or a grammar specifyinghow the data can be generated, as in generative fuzz testing [37] andprocedural generation from shape grammars [27]. While grammarsprovide much greater control, they are hard to read and write anddo not easily allow enforcing global properties. Readability andease of use can be improved by instead providing a program in adomain-specific language with nondeterminism [7]. Conversely,directly writing constraints as in constrained-random verification[28] allows global properties but is even more difficult than writinga grammar. Scenic improves on these methods by simultaneouslyproviding fine-grained control, enforcement of global properties,specification of probability distributions, and the ease of use of animperative programming language.Figure 3: A scene generated from a simple scenario.Finally, the third line creates an additional Car. Notice that wehave not specified anything about where the cars are or how theyare oriented; despite this, the scenario will always yield reasonablescenes: an example is shown in Fig. 3. This is because Scenic enforces several default requirements: all objects must be contained inthe workspace, must not intersect each other, and must be visiblefrom the ego object. Furthermore, Scenic allows defining defaultvalues for all object properties. The definition of Car in carLibbegins as follows (slightly simplified):1 class Car:2position: Point on road3heading: roadDirection at self.positionHere road is a region, one of Scenic’s primitive types, defined incarLib to specify which points in the workspace are on a road.Similarly, roadDirection is a vector field specifying the prevailingtraffic direction at such points. The operator F at X simply getsthe direction of the field F at point X , so the default value for acar’s heading is the road direction at its position. The defaultposition, in turn, is a Point on road (we will explain this syntaxshortly), which means a uniformly random point on the road.So our scenario, despite being so brief, will yield images wherethe cars are positioned realistically. In fact, the rest of the definitionof Car in carLib specifies reasonable default distributions for carmodel and color, so these aspects of the cars will also be plausible.On the other hand, our scenario is still extremely general (“onecar, anywhere on the road”), and we might want to specify the cars’positions more precisely. To create a car that is 20–40 m ahead ofthe camera, for example, we could write:1 Car offset by (-10, 10) @ (20, 40)The interval notation (X , Y ) creates a uniform distribution onthe interval, and X @ Y creates a vector from xy coordinates (as inSmalltalk [12]). This vector is then used as an offset to position theCar, and since no reference point is explicitly given the ego object isused. So the car is placed randomly up to 10 m left or right of the egoobject and 20–40 m ahead (again, automatically conditioned on thefact that the car does not intersect buildings, etc.). This illustratesanother general principle in Scenic, that relative positions, headings,and distances are with respect to the ego object by default. This makesit possible to write compact code by assigning an object to ego andbuilding a local part of the scenario around it (possibly reassigningego to construct another part, eventually leaving it assigned to thedesired camera location).Probabilistic Programming Languages. The semantics (and tosome extent, the syntax) of Scenic are similar to other probabilistic programming languages such as Prob [16], Church [15], andBLOG [25]. In probabilistic programming the focus is usually oninference rather than generation, and in particular to our knowledgeprobabilistic programming languages have not previously beenused for test generation. However, the most popular inference techniques are based on sampling and so could be directly applied tothe scene improvisation problem, as we discuss in Sec. 5.3OVERVIEWWe use several Scenic scenarios from our autonomous car casestudy to motivate and illustrate the main features of the language.Appendix A shows images rendered from the scenarios describedin this section, as well as more complex scenarios such as that inFig. 1, with complete Scenic code.Basics: Objects, Regions, Vector Fields, Distributions, Defaults. Tostart, suppose we want scenes of one car viewed from another onthe road. We can simply write:1 import carLib2 ego Car3 CarFirst, we import a library containing everything specific to ourcase study: the definition of Car as a type of object, as well asinformation about locations and directions of roads (from now onwe suppress this line). Scenic itself contains only general geometricoperations and predicates that we will see below.The second line creates a Car and assigns it to the special variableego specifying the ego object which is the reference point for thescenario. In particular, rendered images from the scenario are fromthe perspective of the ego object.3
UC Berkeley EECS Technical Report, April 18, 2018Fremont et al.Using offset by as above overrides the default position of theCar, but leaves the default orientation (“face along the direction ofthe road”) unchanged. Suppose for greater realism we don’t want torequire the car to be exactly aligned with the road, but to be withinsay 5 of it. We could try:1 Car offset by (-10, 10) @ (20, 40), \2facing (-5, 5) degbut this is not quite what we want, since this sets the orientationof the Car in global coordinates (i.e. within 5 of North). Insteadwe can use Scenic’s general operator X relative to Y , whichcan interpret vectors and headings as being in a variety of localcoordinate systems:1 Car offset by (-10, 10) @ (20, 40), \2facing (-5, 5) deg relative to roadDirectionNote that roadDirection is a vector field and so defines a differentcoordinate system at each point in space: here Scenic automatically uses the position of the Car being defined. If we wanted theheading to be up to 5 off of the ego car’s orientation instead, wecould simply write (-5, 5) deg relative to ego.but these would proliferate since we would need to handle allpossible combinations of ways to specify different properties (e.g.do we want to require a specific model? Are we overriding the widthprovided by the model for this specific car?). Instead of having suchmonolithic constructors, Scenic factors the definition of objectsinto potentially-interacting but syntactically-independent parts:1 spot curb offset by -0.5 @ 02 Car left of spot, with model BUSHere left of X and with model M are specifiers which do nothave an order, but which together specify the properties of thecar. Scenic works out the dependencies between properties (here,position is provided by left of, which depends on width, whosedefault value depends on model) and evaluates them in the correctorder. To use the default model distribution we would simply leaveoff with model BUS; keeping it affects the position appropriatelywithout having to specify BUS more than once.Specifying Multiple Properties Together. Recall that we definedthe default position for a Car to be a Point on road: this is anexample of another specifier, on region, which specifies positionto be a uniformly random point in the given region. This specifierillustrates another feature of Scenic, namely that more complexspecifiers can specify multiple properties at a time. Consider thefollowing scenario, which creates a parked car given a region curbdefined in carLib:1 spot OrientedPoint on visible curb2 Car left of (spot offset by -0.25 @ 0)The function visible region returns the part of the region thatis visible from the ego object. The specifier on visible curbwill then set position to be a uniforml
synthesis by encoding domain knowledge about what data is useful. Specifically, we focus on data sets arising fromscenes, configura-tions of physical objects: for example, images of cars on a road. We design a domain-specific language,Scenic, for describing scenarios that are distributions over scenes. The syntax of Scenic makes it
William Shakespeare (1564–1616). The Oxford Shakespeare. 1914. The Tempest Table of Contents: Act I Scene 1 Act I Scene 2 Act II Scene 1 Act II Scene 2 Act III Scene 1 Act III Scene 2 Act III Scene 3 Act IV Scene 1 Act V Scene 1 Act I. Scene I. On a Ship at
Act I, Scene 1 Act I, Scene 2 Act I, Scene 3 Act II, Scene 1 Act II, Scene 2 Act II, Scene 3 Act III, Scene 1 20. Act I, Scene 1–Act III, Scene 1: Summary . Directions: Summarize what you what you have read so far in Divided Loyalties (Act I-Act III, Scene1). 21. Act III, Scenes 2 and 3:
introduction to the San Juan iSlandS Scenic Byway Scenic Byway Status The San Juan Islands Scenic Byway is designated as a state scenic and recreational highway through state legislation [RCW 47.39.020(63)], which includes the Washington State Ferries routes to and from the islands, as we
Act I Scene 1 4 Scene 2 5 Scene 3 8 Scene 4 15 Scene 5 18 Scene 6 21 Scene 7 23 Act II Scene 1 26 . For brave Macbeth--well he deserves that name-- Disdaining fortune, with his brandish'd steel, . and every one d
A Midsummer Night's Dream Reader Summary 1.1 2 Act 1, Scene 1 6 Summary 1.2 16 Act 1, Scene 2 20 Summary 2.1 (a) 30 Act 2, Scene 1 (a) 34 Summary 2.1 (b) 42 Act 2, Scene 1 (b) 46 Summary 2.2 50 Act 2, Scene 2 54 Summary 3.1 64 Act 3, Scene 1 66 Summary 3.2 80 Act 3, Scene 2 96 Summary 4.1 106 Act 4, Scene 1 108
Christmas Play Written by LINDSAY FINLEY Produced by. TABLE OF CONTENTS Foreword3 Characters4 From the Author 5 Introduction6 Scene 1 8 Scene 2 10 Scene 3 12 Scene 4 14 Scene 5 16 Scene 6 18 Scene 7 22 Song Sugg
b. Be aware of any persons or vehicles leaving the crime scene. c. Approach the scene cautiously. d. Assess the scene for officer safety. e. Remain alert and attentive. f. Tr eat location(s) as a crime scene until determined to be otherwise. Performance Given a crime scene scenario, the student is expected to demonstrate overall scene aware-
of general rough paths. However, in this paper, we will focus on the case where the driving signal is of bounded variation. Following [6] we interpret the whole collection of iterated integrals as a single algebraic object, known as the signature, living in the algebra of formal tensor series. This representation exposes the natural algebraic structure on the signatures of paths induced by the .
