SOFTWARE OpenAccess Asubstrate Formodular,extensible Data .

1y ago
22 Views
1 Downloads
3.18 MB
15 Pages
Last View : 1m ago
Last Download : 4m ago
Upload by : Rosemary Rios
Transcription

(2020) 5:1Matelsky et al. Big Data SOFTWAREBig Data AnalyticsOpen AccessA substrate for modular, extensibledata-visualizationJordan K. Matelsky, Joseph Downs, Hannah P. Cowley, Brock Wester and William uapl.eduThe Johns Hopkins UniversityApplied Physics Laboratory, Laurel,MD, USAAbstractBackground: As the scope of scientific questions increase and datasets grow larger,the visualization of relevant information correspondingly becomes more difficult andcomplex. Sharing visualizations amongst collaborators and with the public can beespecially onerous, as it is challenging to reconcile software dependencies, dataformats, and specific user needs in an easily accessible package.Results: We present substrate, a data-visualization framework designed tosimplify communication and code reuse across diverse research teams. Our platformprovides a simple, powerful, browser-based interface for scientists to rapidly buildeffective three-dimensional scenes and visualizations. We aim to reduce the limitationsof existing systems, which commonly prescribe a limited set of high-level components,that are rarely optimized for arbitrarily large data visualization or for custom data types.Conclusions: To further engage the broader scientific community and enableseamless integration with existing scientific workflows, we also present pytri, aPython library that bridges the use of substrate with the ubiquitous scientificcomputing platform, Jupyter. Our intention is to lower the activation energy required totransition between exploratory data analysis, data visualization, and publication-qualityinteractive scenes.Keywords: Visualization, Data science, Big data, Graph visualization, WebGL, Browser,Jupyter, Python, JavaScriptBackgroundUsing modern web-based visualization frameworks [1, 2] makes it easy to generatebeautiful, interactive, and informative visualizations of scientific data. These renderingssimplify the processes of exploring data and sharing insights with the community. In manydomains, this has become a key step in the research and discovery pipeline [3].One challenge with these technologies is the difficulty of adapting prior visualizationwork to a new use-case. These tools are often built to be single-purpose rather than interoperable. Therefore it can be difficult or even impossible to combine aspects of disparatevisualization scenes, even when the visualizations use the same technologies or frameworks. This challenge leads to software duplication instead of reuse, and complicates theportability of these products between research efforts. Often, the developers of modernvisualization systems have chosen to either enjoy wide adoption at the expense of domain The Author(s). 2020 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to theCreative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ) applies to the data made available in this article, unless otherwise stated.

Matelsky et al. Big Data Analytics(2020) 5:1specific tooling (e.g., plotly and matplotlib), or have focused on scientific subdomains atthe expense of extensibility (e.g., common GIS or biological rendering software such asneuroglancer [4] and FreeSurfer [5]). As a result, combining visuals from more than oneanalysis or modality often requires significant engineering effort [6].Scientists and software developers have produced several frameworks designed to remedy these challenges [3, 7, 8] and we leverage architectural ideas from some of theseframeworks in our solution, called substrate. We follow a compositional model similar in spirit to component-based engines such as React or Vue [4, 8–10], whereina visualization is comprised of many individual, independent components. Furthermore, we have developed pytri, a Python module that enables Python developers toaccess and interact with WebGL-based substrate from inside a Jupyter environmentwithout requiring prior JavaScript or WebGL knowledge. This ecosystem includes integrated Jupyter notebook capabilities found in systems such as Mayavi or Neuroglancer[4, 7] alongside generalized visualization capabilities found in systems such as Plotly orMatplotlib [11, 12].Unlike many existing Jupyter visualization packages, pytri visualizations are fully customizable, even by the end user. Users are not constrained by the limits of prepackagedvisualization data structures or plot types, and can combine prebuilt components alongside custom, purpose-built visualization components. Users directly interact with theirunderlying data as usual, while these tools bring visualization capabilities fully into thedata analytics platform.Our tools are designed to enable the visualization of large-scale data or customdatatypes, coregister multimodal data, and simplify the process of sharing or reproducinganalyses — all without disrupting the data science process.We have designed our approach to be usable with minimal technical knowledge: Auser can enter this ecosystem with only a basic knowledge of JavaScript or Python,though power-users and proficient software developers will find these libraries easy tomanipulate and extend to suit their needs.We first describe the software design of substrate, and then introduce pytri inorder to render substrate scenes from common Python data libraries such as numpy[13], networkx [14], or pandas [15]. Finally, we share example use-cases in which theinteroperability provided by substrate can reduce the engineering overhead of a newvisualization project. Code and tutorials are available online at the links in the Availabilityof Data and Material section.ImplementationTo fully separate the responsibilities of substrate, our scene engine, and pytri, ourPython integration library, our software architecture mandates that all rendering andDocument Object Model (DOM) manipulation is handled by substrate and all Pythonmanipulations and datatype translation take place in pytri.substratesubstrate is a JavaScript library that exposes a simple but powerful developer-facingapplication program interface (API). This abstraction enables otherwise disparate visualization projects to share resources and logic easily. The substrate JavaScript libraryis intended to simplify component-reusability of commonly used data-visualizationPage 2 of 15

Matelsky et al. Big Data Analytics(2020) 5:1structures such as scatter plots or 3D-embedded graph visualizations. Each atomic visualization component implements a common Layer API. Each Layer in a scene handlesits own data management and interactivity in isolation, while a parent Visualizerobject manages a list of these Layers in much the same way as a React application rerenders its components as needed when its internal state changes [9]. For example, aScatterLayer implements Layer by accepting an array of [ x, y, z]-tuples, and it willrender these data when initialized. In the same scene, a GraphLayer might render a 3Dgraph embedding, as shown in Fig. 1. These layers exist in the same 3D coordinate system,but are managed independently so changes to one does not require a re-render of theother components in the scene. Comprehensive Layer API documentation is availableonline.We formalize a universal Layer interface that accommodates common visualizationtasks. Layers must include:1.2.3.requestInit (function) is called before the visualization starts: Thisfunction generally includes instructions to provision objects in a 3D scene,or to request data from a remote source with a long round-trip;requestRender (function) runs on every frame. In static,non-animated Layers, this function may be empty or remainunimplemented in order to conserve compute power;children (array attribute) lists all objects in a scene associated with aparticular Layer. When a Layer is removed from the visualization, allobjects in this list are cleaned up and garbage-collected by substrateinternally.This simple interface enables many different visualization objects or groups of objectsto coexist in a scene without interfering with one another. Such namespacing conflicts areFig. 1 A GraphLayer and a ScatterLayer, embedded in the same 3D space. substrate overlaysthe two and controls the two independently. a One layer renders a graph object. b A second layer renders ascatter of points in 3D space. c The layers are composited by the substrate Visualizer into a singleimage. d The rendered scene, viewed from the substrate camera in finished formPage 3 of 15

Matelsky et al. Big Data Analytics(2020) 5:1a common pitfall when combining conventional, separately-developed assets into a singlescene using WebGL wrapper frameworks.In order to improve the accessibility of our codebase to new developers and data scientists, we use three.js as a convenience to wrap WebGL. Despite the prevalenceof three.js in our current codebase, substrate aims to be framework-agnostic.Authors of new Layers may choose to write WebGL directly, or use another wrapper or framework. substrate will support these Layers provided they subscribeto the substrate.Layer and substrate.Visualizer interfaces. In this way,substrate works in a similar extensible fashion to the Uber deck.gl project, thoughdeck.gl re-implements full scene rendering from the ground up, whereas substrateuses the three.js industry standard [2].The compositional property of Layers can be expressed in code using the syntaxshown in Fig. 2. Here we show a simple Visualizer containing two Layers; this shortsnippet can run a complete visualization without any extra configuration.One common use of separate Layers is to place objects of interest — such as a mesh— in one Layer, and place lighting, axes, or other environmental factors in another. Thisenables a researcher to share their core data, such as the 3D mesh, with other researchers,without including extraneous features such as light sources or grids and axes. (Thissimultaneously enables an artist to reuse their lighting layout across projects.) In Fig. 3,we illustrate a sample Layer implementation that can be ported to any substratevisualization. Our add-and-remove-layer demo provides an example in which aMeshLayer is added or removed, without affecting other objects in the scene.Layers written for one visualization or application may be repurposed or reused inanother scene without additional developer effort. This means that for most visualizationuse-cases, such as graph displays or scatter plots, no substrate knowledge is requiredat all; instead, prebuilt Layers are available for public use, including a ScatterLayer,GraphLayer, MeshLayer, and many others, the most common of which are listed inTable 1.In some cases, specific data requirements may mean that researchers cannot use theseexisting, prebuilt Layer implementations. If a developer decides to implement theirown Layer from scratch, it can be trivially integrated into other visualizations, as allsubstrate Layers subscribe to the same simple interface and are interchangeable. Forexample, the big-data neuroscience community has developed a custom layer that visualizes larger-than-RAM imagery by shuttling data in and out of memory as it enters andFig. 2 A sample Visualizer, with two Layers. One renders a 3D scatter-plot, and the other rendersnodes and edges of an undirected graphPage 4 of 15

Matelsky et al. Big Data Analytics(2020) 5:1Page 5 of 15Fig. 3 A sample implementation of a Layer that generates a point-cloud from the data provided in theconstructor. This exact implementation can be dropped into any substrate visualization withoutmodification. This code, and other Layer examples, are available onlineleaves the substrate camera viewport. Social graph research teams have developedrepresentations of graphs with enriched visual cues to signal node and edge attributes.When research use-cases require customization, engineers can easily implementLayers that suit their specific need. Prebuilt Layers are sufficient for many applications,and require no engineering knowledge or ability from the end-user.Table 1 Available layers built into the substrate packageLayer nameDescriptionUseful for.AxisLayerContains an RGB axis at the scene origin.orienting the user in spaceGraphLayerA graph renderer with support for many millionsof nodes or edges.complex networksImageLayerImport 2D images and render them as a plane.displaying static iamgeryLightingLayerThree-point lighting scheme, as commonly usedin film and 3D graphics.easy lighting setupsLineSegmentsLayerA list of line segments to be rendered individually.paths or wireframesMeshLayerArbitrary 3D meshes, imported in common fileformats such as OBJ.arbitrary 3D solidsScatterLayerPoint-clouds of many millions of points.colored scatter plotsVolumeLayerA visualization of dense 3D voxelwise data.rendering volumetric data

Matelsky et al. Big Data Analytics(2020) 5:1Our brownian-particle-motion example demonstrates how a developer caneasily implement a custom Layer, while still taking advantage of prebuilt code. Weenvision that community users may request to merge commonly used Layers into thesubstrate codebase to extend native functionality and cover a diverse set of use-cases.As groups work together to achieve research goals, these researchers may separatelydevelop Layers (e.g. a raw experimental Layer and an annotation Layer for theanalysis) which can be combined in the same scene when needed.pytriRequiring a scientist to exit their research environment in order to engage withvisualizations reduces the time-efficacy of research [16, 17]. In order to providea convenient, inline visualization solution for data scientists, we created pytri, aPython package that enables visualization of substrate Layers without leaving aJupyter notebook [18] or other IPython environment (Fig. 4). Jupyter is a standardresearch platform for many communities. By bringing composable, extensible visualization to this platform, data scientists can quickly visualize and explore data ina familiar environment without needing to understand the underlying substratecodebase.Our initial use-case demanded performant large-scale graph visualization. Thoughstandalone visualization software existed for this scale of graph, our team found it onerous to exit our data science platform to view the raw data. Furthermore, we encounteredissues with the performance of many popular 3D plotting libraries in Python, as theywere unable to handle the size of graphs we needed to visualize. We built substrateto gracefully handle graph data containing many millions of edges. We then added corresponding hooks in pytri to empower users to create large-scale graph visualizationsquickly without leaving the familiar Python environment.pytri combines substrate capability with Python datatypes by leveraging the Jupyter Notebook IPython.display module, to which pytri delegates responsibility for interaction with the Jupyter DOM through both JavaScript(IPython.display.JavaScript) and HTML (IPython.display.HTML) [18].These direct DOM manipulations, unlike the comparable Jupyter Widget architecture, enable pytri visualizations and data to persist even in a static HTML exportof the Jupyter notebook. This enables the distribution of reproducible visualizationswithout requiring the end-user to install or configure software packages. In otherwords, researchers can produce static HTML files with interactive data visualizationswhich can be shared by email, by online publication, or through sharing the originalsource code.ResultsThe following use-cases illustrate substrate’s flexibility, idiomatic brevity, and abilityto handle custom data. By lowering the overhead associated with task-switching betweenvisualization and analysis, substrate provides an opportunity for a team to moreintimately explore their data and iterate on analyses in realtime.One of the key advantages of substrate is its use as a general framework for visualization across many domains. Here we highlight a few diverse applications that benefitfrom this work.Page 6 of 15

Matelsky et al. Big Data Analytics(2020) 5:1Fig. 4 Here, pytri runs two simultaneous cells of different substrate visualizations inside a Jupyternotebook, using WebGL to enable real-time user interaction with the visualization. This high-detail mesh wasgenerated using manual annotations from a recent electron microscopy study [19]Use case: analyzing biological imaging dataBiomedical research is one of many domains that benefits from the recent explosion in bigdata [20]. As such, there is pressing need for research tools that can adapt to the demandsof large-scale datasets. Such a customizable and interactive framework for visualizationcan help researchers understand otherwise uninterpretable data. Volumetric imagery isone datatype that is particularly important both in biomedical research as well as in thePage 7 of 15

Matelsky et al. Big Data Analytics(2020) 5:1clinical setting. Many existing frameworks for data visualization lack a lightweight toolthat preserves the vital spatial relationships found in volumetric imagery.When creating visualizations using substrate and pytri, the researcher has thespeed and flexibility necessary to transform a typical visualization product from a staticcommunication tool to a dynamic exploration aid. As an exercise in using pytri fordata exploration and scientific communication, we present the following use-cases, showcasing similar opportunities across six orders of magnitude (e.g., magnetic resonanceimaging (MRI) to electron microscopy (EM)). We focus on an exciting area of emergingresearch called connectomics, which focuses on estimating brain connectivity maps atvarious resolutions.We first investigate slices of MRI data volumes using ImageLayer, as demonstratedin Fig. 5. Navigation through the dense 3D data can be done programmatically throughdirect calls to the pytri API, or through the Jupyter UI, so the researcher never has toleave their data analysis environment. Beginning with this volumetric data representation,we can overlay fiber tracts representing estimates of major axonal bundles in the brain,and the derived nodes and edges associated with a connectome [21]. More specifically, theconnectome is visualized using a GraphLayer overlaid on a MeshLayer of the surfaceof the patient’s brain as computed from structural MRI. When the analysis is ready to beshared, the researcher can easily package these visualizations for others by exporting theJupyter notebook to an interactive HTML page.With EM data, we show the flexibility of the tooling by displaying imagery slices, alongwith commonly used derived data representations such as meshes generated from manualor automated segmentation methods and skeleton traces. These tools are important torapidly explore and validate large reconstructions. Because our visualization tools existwithin an analytics environment, users can compute quantitative analyses in the sameenvironment, reducing impediments to discovery. substrate’s layer-based frameworkFig. 5 A view of MRI data using volumetric rendering. A graph of estimated brain region connectivity isrendered in red and yellow. Estimated fibers are rendered in cyan. The white brain volume is the directoutput of a Diffusion Tensor Imagery MRI scanPage 8 of 15

Matelsky et al. Big Data Analytics(2020) 5:1allows the researcher to overlay multiple data sources, to reorient the view to focus onone detailed section of the data at a time, and to fully leverage data visualization in theresearch and sharing process.Use case: astronomic

a convenient, inline visualization solution for data scientists, we created pytri,a Python package that enables visualization of substrate Layers without leaving a Jupyter notebook [18] or other IPython environment (Fig. 4). Jupyter is a standard research platform for many communities. By bringing composable, extensible visu-alization to this .

Related Documents:

Progress Software Corporation September 2016 5 OpenAccess SDK Version 8.1 Highlights This section highlights what’s new in OpenAccess SDK 8.1: OpenAccess SDK SQL Engine and IP SDK

2.1 XML (Extensible Markup Language) 13 2.2 XSD (XML Schema Definition) 18 2.3 MathML (Mathematical Markup Language) 23 2.4 SPS (StyleVision Power Stylesheet) 25 2.5 XSL (Extensible Style Language) 27 2.6 XSLT (Extensible Style Language Transformations) 31 2.7 XSL:FO (Extensible Style Language: Formatting Objects) 32 2.8 XPath (XML Path Language) 33 3 Estudi de l'estàndard XML DocBook 37 3.1 .

www.lenel.com OpenAccess Alliance Program Factory Certified Product (FCP) Interface Document OAAP Member Information Member Company Name SMI Global Ltd. Member Contact Name: Dick Clark Phone: 44 7815158203 Email: dick@smi-global.net OnGuard Information OnGuard Version/Build, Firmware OnGuard 7.1 OnGuard 7.2

Luqueetal.BMCMedicalInformaticsandDecisionMaking (2016) 16:9 DOI10.1186/s12911-016-0246-y RESEARCH ARTICLE OpenAccess Optimalsequenceoftestsforthe .

Extensible Markup Language (XML), Extensible 3D (X3D), Extensible Stylesheet Language (XSL), US Message Text Format (USMTF), XML-MTF, Land Command and Contol Information Exchange Model (LC2IEDM), Generic Hub (GH), Amphibious Raid 16. PRICE CODE 17. SECURITY Unclassified 18. SECURITY Unclassified 19. SECURITY Unclassified 20. LIMITATION OF ABSTRACT UL NSN 7540-01-280-5500 Standard Form 298 (Rev .

tres tipos principales de software: software de sistemas, software de aplicación y software de programación. 1.2 Tipos de software El software se clasifica en tres tipos: Software de sistema. Software de aplicación. Software de programación.

GGobi is a direct descendent of a data visualization system called XGobi that has been around since the early 1990's. GGobi's new features include multiple plotting windows, a color lookup table manager, and an XML (Extensible Markup Language) . an interactive and dynamic software system for data visualization. GGobi is the result of a .

O-Cloud Standard Hardware Management Unit Baseband Unit RAN Control Plane RAN Data Plane O-Cloud Standard Hardware . and security policies Extensible Platform Service-based architecture extensible with 3rd party . Capgemini Engineering 5G gNB Software Framework enables support for 3GPP / O-RAN Architecture & Interfaces. * CU, DU, and xApps .

Rational Software Architect unifies them and is built on top of the open and extensible Eclipse platform, which leverages several industry standards. Rational Software Architect is mainly used by software architects and senior developers within a development team for specifying and maintaining all aspects of an application's software architecture.

Resource Planning (CRP) methodology for algorithm designs. The resulting near-optimum algorithm operates within an extensible CRP shell and uses two pluggable scheduling and forwarding heuristics to produce near-optimal performance. In this paper we present the shell architecture, the algorithm, and the simulation

Extensible Access Control Markup Language (XACML) Version 3.0 Extensible Markup Language (XML) The PROV Data Model (PROV-DM) The Provenance Notation (PROV-N) The PROV Ontology (PROV-O) The PROV XML Schema (PROV-XML) 3 Definitions, Abbreviations and Symbols 3.1 Definitions For the purposes of this International Standard, the following definitions apply. The definitions defined in clause 3 of .

compliance with Annex II criteria set out in the Regulation 1025/20121 on European standardisation. XHTML stands for Extensible Hypertext Markup Language. The W3C technical specification is a reformulation of HTML4 as an application of XML. HTML (Hypertext Markup Language) is the Markup Language for formulating Web pages in the World Wide Web and XML (Extensible Markup Language) defines a set .

IEEE Std 1636.1 IEEE Standard for (SIMICA): Exchanging Test Results and Session Information via the eXtensible Markup Language (XML) IEEE Std 1636.2 IEEE Standard for (SIMICA): Exchanging Maintenance Action Information via the Extensible Markup Language (XML) IEEE Std 1671 IEEE Standard for Automatic Test Markup Language (ATML) for Exchanging Automatic Test Equipment and Test Information via .

Building an Effective & Extensible Data & Analytics Operating Model To keep pace with ever-present business and technology change and challenges, organizations need operating models built with a strong data and analytics foundation. Here’s how your organization can build one incorporating a range of

WorkCentre 5665/5675/5687 WorkCentre Pro 45 (continued on the next page) (continued on the next page) Xerox Extensible Interface Platform (E

an ideal solution for indoor/outdoor compact substations and is available in non-extensible, extensible and modular formats to suit various application requirements. All of the switching functions are insulated with SF6 gas and sealed in a stainless s

What is XML XML stands for eXtensible Markup Language. A markup language is used toprovide information about a document. Tags are added to the document to provide the extra information. HTML tags tell a browser how to display the document. XML

XML: a markup language used to describe information. DOM: a programming interface for accessing and updating documents. DTD and XML Schema: describes the structure and content of XML documents. XSLT: XSL stands for eXtensible Stylesheet Language, and is a style sheet language for

Perl::Critic-Static Code Analysis for Perl-Includes over 120 policies-Most based on Perl Best Practices-Grouped into levels and themes-Configurable and extensible for local policies and styles 31 An extensible framework for creating and applying coding standards to Perl source code Perltidy address the layout of code.

Le fabricant et l’utilisateur d’un additif alimentaire sont tenus: a. de transmettre à l’OSAV toute nouvelle information scientifique ou techni-que susceptible d’influer sur l’évaluation de la sécurité de cet additif; et b. d’informer l’OSAV, sur demande, des usages de l’additif concerné. Art. 11 Modification des annexes L’OSAV adapte régulièrement les annexes de la .