A Multi-core Parallel Mosaic Alorithm For Multi-view Uav Images
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, ChinaA MULTI-CORE PARALLEL MOSAIC ALORITHM FOR MULTI-VIEW UAV IMAGESXin Pan a, Xian Zhao a,*, Dongyang Gao a, Xianhu Li aaSchool of Geomatics and Urban Spatial Information, Beijing University of Civil Engineering and Architecture, 102616 Beijing,China – (panxin, gaodongyang, lixianhu)@stu.bucea.edu.cn, zhaoxian@bucea.edu.cnKEY WORDS: Unmanned Aerial Vehicle, Image Mosaic, Parallel ComputationABSTRACT:As the spread of the error and accumulation often lead to distortion or failure of image mosaic during the multi-viewUAV(Unmanned Aerial Vehicle) images stitching. In this paper, to solve the problem we propose a mosaic strategy to construct amosaic ring and multi-level grouping parallel acceleration as an auxiliary. First, the input images will be divided into several groups,each group in the ring way to stitch. Then, use SIFT for matching, RANSAC to remove the wrong matching points. And then,calculate the perspective transformation matrix. Finally weaken the error by using the adjustment equation. All these steps runbetween different groups at the same time. By using real UAV images, the experiment results show that this method can effectivelyreduce the influence of accumulative error, improve the precision of mosaic and reduce the mosaic time by 60%. The proposedmethod can be used as one of the effective ways to minimize the accumulative error.1INTRODUCTIONAccording to the low cost, high flexibility and less limited byweather and the other characteristics, UAV has been animportant way of acquiring data in measuring, GIS and remotesensing. So multi-view UAV image mosaic is also a researchhotspot. One of the most influential factors for mosaic is theexistence of accumulated errors. By its type it can be dividedinto two types. One is caused by symmetry error and the other isasymmetric error. To solve the problem, the researchers haveproposed the use of satellite images with geographicinformation as a reference image for image mosaic (Keall,2004); transformation of the benchmark map of the mosaicstrategy (Li, 2008). The above method can effectively improvethe quality of mosaic results under certain circumstances, butthe treatment of accumulative error is not very satisfied.Because there are many uncontrolled factors caused erroraccumulation, this kind of situation makes analysis andelimination the accumulative error of great difficulty, soassuming that the position error between adjacent images is themost important factor (Li, 2008).In the paper, we proposed a multi-core parallel ring mosaicalgorithm for multi-view UAV to illustrate problem above. Themosaic process is divided into several levels, which consists ofmultiple sets of adjacent images, and then two adjacent imagesin each group will be cut into four sub-images with overlapareas, between the two images using the feature-based imagemosaic algorithm, and then stitch the four images to ringsuccessively, then use the adjustment model of the imagetransformation model to eliminate the error, after these themodel will be re-brought into the mosaic process to completethe mosaic before the real stitch of the two images. Each levelof mosaic will be finished by the above-mentioned mosaicalgorithm completed in parallel. Through the experiment, themethod can effectively reduce the impact of accumulative error,and greatly reduce the operation time.2THE BASIC THEORIES OF IMAGE MOSAICThe feature-based image mosaic method described aboveincludes the following three processes. The SIFT feature isextracted from the two input images and then perform the imagematching; use random sampling consistency (RANSAC) forremoval of mismatch.; The perspective transformation matrix isused as the relationship between the images. Finally, use one ofimage fusion methods to smooth stitch edge and improve visualeffects.2.1Image Matching and Elimination of MismatchingSIFT is a well-known operator in the field of computer visionand can be used for pattern recognition and image matching.The operator can find the important local information of thefeature points accurately to determine the matching relation ofthe feature points, and finally reach the effect of prevent fromrotation, translation and scaling to find the matching featurepoints accurately (Lowe, 2004). The key idea of the algorithm isto detect in the scale space first and then locate the key pointaccurately and determine its main direction to make it invariant;finally establish the descriptor of the key points. However, thiscannot still guarantee that all the matching points are correct,but also need the elimination process.RANSAC is an abbreviation for random sampling consistency.It is a robust model parameter estimation algorithm. Since theinput matching point data contains the wrong pair of points,unlike the least squares method, the model parameterscalculated by the least squares method are the smallest sum ofthe distance from all points, but RANSAC can remove thewrong points, calculate the parameters that are appropriate forproper matching points. Even if in the case of the number ofwrong pairs is large, RANSAC is still valid (Fischler, 1987).The calculation process of RANSAC is as follows: Firstly,extract the points randomly to calculate the model. Aftercalculating the model parameters, the point set is divided intoinner and outer points. If the number of points is enough, themodel parameters are correct, so that recalculate the model thatsatisfied all the inner points. The process will be repeated a*Corresponding authorFund Project: National Nature Science Foundation of China (40771178)This contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.845
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, Chinacertain number of times to select the optimal model parameters,finally use precision to evaluate the quality of the model.2.2Image TransformationAn important step in image mosaic is to determine thetransformations between two images, such as affinetransformation and perspective transformation. Affinetransformation can describe the rotation, scaling, translationbetween images and is "parallelism", and in additionperspective transformation can also describe the vertical sweepand horizontal sweep and other movements, the transformationis not "parallelism ". Affine transformation can be considered asa special case of perspective transformation. Table 1 lists theparameters of perspective transformation and affinetransformation. The perspective transformation matrix has thefollowing form: h11 h12 H h21 h22 h 31 h32h13 h23 1 h11Affinetransformationh11h21h31h21h22h23h31h32g ( x, y) ag1 ( x, y) (1 a) g2 ( x, y)Where (x, y) means coordinate in image, g(x, y) means pixelvalue after fusion, g1(x, y), g2(x, y) means two images to befused, a is weight parameter, its value between (0,1) (Yang,2007). In theory for each image the weight of the middle part ishigher than the weight of the edge.h21h31h21h22h2300Image FusionIn image mosaic, the problem of which is closely related toimage transformation is the choice of image fusion. Based onthe two images to be stitched are of better registration, theinformation of two images are integrated into the same imagefor a comprehensive expression, can not only reflect thecharacteristics of the two images but also reduce the redundancyof the data. Image fusion algorithm directly affects the visualeffects of synthetic images. At present, scholars around theword put forward a lot of image fusion methods such as directMULTI-LEVEL PARALLEL CLOSURE MOSAICALGORITHM FOR MULTI-VIEW IMAGEIn the multi-vision UAV image mosaic, the most influencefactor is accumulative error, by type it can be divided intosymmetry errors and asymmetry errors. The relevant scholarshave proposed that the accumulative error can be halved bydivided between the adjacent images to eliminate the influenceof it, and is most useful to asymmetric error even the precisioncan be doubled (Tang, 2003). The paper proposes a multi-coreparallel ring mosaic algorithm designed to reduce theaccumulative error while using multi-core technology to processalgorithm parallel to reduce the operation time.3.1Table 1. Image transformation parameters2.3The basic idea of the linear weighted transition is similar to thedirect averaging method, but the pixel values of the overlappingregions are not simply averaged but weighted averaged. Formlike：3Where h11,h12,h21,h22 are scaling and rotation factors；h13,h23 arehorizontal and vertical translation factors; h31,h32 are affinetransformation factors (Zhao, 2006). As the degree of freedomof perspective matrix is eight, so at least four pairs of pointsneeded to calculate the parameters.Perspectivetransformationaverage method, linear weighted transition method andmulti-resolution method (Burt, 1983). In order to bettereliminate the lighting problem during shooting, reduce theghosting produced by geometric transformation and reduce thediscontinuity and mutation of the color at the mosaic edge, thepaper uses the linear weighted transition method.Multi-level Grouping and ParallelismSince accumulative error can be scattered to effectively weakenits influence on the final result, so that the multi-level groupingparallel technology is proposed. OpenMP interface is amultiprocessor programming model for memory sharing modeland is suitable for this case. For n pieces of input images, it canbe divided into [log2n] 1levels, where [x] means rounding.Here it is assuming that the input image is eight, three levels ofmosaic, which also means eight input images will be stitchedinto four images and then mosaic by the same way, in the endstitched into the output image. The idea of the algorithm is thateach level is divided into several groups, each group is parallel.Between levels are serial. Because the input image of the nextlevel is the output image of the previous level. The principle ofmulti-level and parallelism mosaic is shown in Figure 1.Figure 1. Principle of multi-level and parallelismThis contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.846
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, ChinaParallel computing is achieved by combining plementation is described in the form of pseudo-code://start parallel process#omp sections //Lv1 Mosaic start{#omp sectionImg12 Mosaic2img(Img2, Img1)#omp sectionImg34 Mosaic2img(Img4,Img3);#omp sectionImg56 Mosaic2img(Img6,Img5);#omp sectionImg78 Mosaic2img(Img8,Img7);} //End of Lv1 Mosaic# omp sections //Lv2 Mosaic start{# omp sectionImg14 Mosaic2img(Img34,Img12);# omp sectionImg58 Mosaic2img(Img78, Img56);}//End of Lv2MosaicMosaic2img(Img58,Img14);//Lv3 Mosaic(2) Mosaic the four sub-images together with the orderof the clockwise. Determining the perspectivetransformation matrix HD of Img2D and Img1D, theoutput is ImgD. By the same way HUD is the matrix ofImgD and Img1U, the result is ImgLUD. The relationbetween ImgLUD and Img2U is HU, the output is A.(3) In Img2D with the non-overlapping region fromImg2U, to find a point P with coordinate (x, y), thecoordinate of Img2 is (x’, y’). Because they are the samepoint in Img2 itself, and the process of cutting Img2 intosub-images is known, the following relationship exists: x' x H y ' y whereH HU HUD H D x x ' x H y y ' y (1)Where function Mosaic2img(Img2, Img1) executes the mosaicof Img1 and Img2, then return the result of it, each ompSections describes a level, different Sections are serial whichmeans when the execution of former Sections is done, the nextSections shall begin to work. In each Sections contains anumber of sections, each section corresponds to a thread callMosaic2img (A, B) to perform image mosaic. Different sectionsin Sections are parallel relationships. This will combine the ideaof multi-level grouping and parallel computing to speed up theoperation of the algorithm.(4) By using formula (1) to establish the adjustmentequation and linearize it, the following formula can beobtained:3.2Where Vx, Vy represents the correction number for x and y; x0, y0 Indicates the initial value of the closed error, h11, h12, h13, h21, h32 indicate therequired parameters.Mosaic Ring Composition MethodFor the traditional method that mismatching still exists andthere is no consideration of influence of errors, at the time theparameters of H matrix are not correct. Based on thecharacteristics of UAV images and research of relevant scholars,a ring mosaic method is proposed to reduce the influence ofaccumulative errors.Since the operations done by H on the entire image are the same,the closed error calculated by using the matching pairs is thesame for each point on the image. Since the points fromoutside the overlapping area is an external point, the H is correctif the above relation H is satisfied; otherwise, the closure errorcan be obtained and eliminated. Assuming that the two imagesto be stitched are Img1 and Img2, the algorithm steps are asfollows:(1) Clip Image1 and Imag2 in to four sub-images:Img1U,Img1D and Img2U,Img2D.The method of clippingis shown as Figure 2. x x x x x Vx x0 h h11 h h12 h h13 h h31 h h32 1112133132(2) y y y y V y h h h h y h y 0 h21 21 h22 22 h23 23 h31 31 h32 32For all of the matching points there are:V BX L(3)Where Vx1 1 Vy V n Vx Vy n h11 h 12 X h31 h32 x01 1 y0 L n x0 n y0 Figure 2. Method of clipping sub-imagesThis contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.847
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, China x1 y1 c c 0 0 B n n x y c c 0 0 Where yb c2 yb c 2 nn x a y b 0 0c2c2 y n 1 x n a y nb c c c2c2 adjacent images, and the adjustment method is added to ensurethe precision of stitching. On the other hand, because of theintroduction of multi-core parallel computation, operation timeis greatly shortened, although the algorithm cannot guaranteethe real-time processing, but compared to single-core computingthe running speed has been greatly improved. Figure 3 showsthe overall process of this algorithm. xa10 0 0cc2x1 y1 1 xa0c c c c210cxn0c(4) a h11 x n h12 y n h13 nn b h21 x h22 y h23 c h xn h y n 13132 The least squares solution is:X BT PB BT PL 1(5)Where P is the weight matrix, let forty pairs of pointsinvolved in the calculation is of unit weight, each time thenumber of points involved in the calculation of H is p1p2 40 p 1p 0 pn 1pn , so4 0 40 pn H newEXPERIMENT AND RESULT ANALYSISThe experimental images are several multi-view images takenby a UAV from China. The height of the shooting is slightlychanged. However, due to the cross wind, the attitude of theaircraft is not stable enough. In order to prove the effectivenessand advantages of the algorithm, the same input images are usedto compare the traditional mosaic method and the proposedalgorithm.By using formula (5), H will be updated from X andremembered as Hnew : h11 h11 h12 h12 h21 h21 h22 h22 h hh32 h3231 31Figure 3. Process of mosaic operationh13 h13 h23 h23 (6) 1 (5) using Hnew as perspective transformation matrix tofinish the mosaic.For all eight input images are shown in Figure 4. By using thealgorithm, Figure 4(a) and (b) are as examples to cut into foursub-images are shown in Figure 5, the traditional mosaic resultsshown in Figure 6(a)，the result of ring mosaic is shown inFigure 6(b). By comparison Figure 6(a) with Figure 6(b) , it canbe shown that due to the interference of the accumulative error,there are mosaic gap and edge of the dislocation on the left sideof Figure6(a), but by using this algorithm the problems are wellhandled. Compared with traditional method, the closeddifference of corresponding points is within one pixel by usingthis algorithm.Because of the mosaic method, on the one hand, theaccumulative error can be effectively distributed betweenThis contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.848
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, China(d) Input image4(c) Input image3(b) Input image2(a)Input image1(h) Input image8(g) Input image7(f) Input image6(e) Input image5Figure 4. Input images(d) Sub-image 4(c) Sub-image 3(b) Sub-image 2(a) Sub-image 1Figure 5. Four sub-images(a) Result of traditional mosaic(b) Result of ring mosaicFigure 6. Result comparisonBecause of the high demand for time of image mosaic, theconsumption of process time is also an important indicator. Asthe time of mosaic processing is affected by many factors, suchas scene complexity and performance of the hardware, so thetime of program is only indicate that multi-core parallelcomputation plays an absolute positive role in improving speed.The input images are all 434*619 with depth in 24 bit. Theequipment of experiments are as follows CPU: i5-2410M;memory: 2G; operation system: Win7 32bit. The timecomparison of the algorithm about whether using parallelacceleration is shown in Table 2. It can be seen that after theacceleration, the operation time has reduced by 40%.This contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.849
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W7, 2017ISPRS Geospatial Week 2017, 18–22 September 2017, Wuhan, ChinaProjectLV1 mosaicLV2 mosaicLV3 mosaicTotalTime（s）Without parallel acceleration27.371114.56978.4748750.415667With parallel acceleration13.25088.57238.067829.8909Table 2. Operation time comparison5CONCLUSIONIt can be seen from the above experiments that the multi-imagemulti-level parallel mosaic algorithm proposed in this paper caneffectively distribute the accumulative error between theadjacent images and avoid the accumulation of the accumulatederrors in the multi-view image mosaic. The adjustment modelweakens the error to ensure the accuracy of the final mosaicresults, while the parallel acceleration of the introduction canincrease the speed and reduce the time cost. However, due tothe huge amount of calculation, in daily use we should be basedon the need of accuracy of mosaic and time to make the choice.The proposed method also provides a reference for theassociated multi-view image mosaic.the ACM, 24(6), 726-740.Keall, P. 2004. 4-dimensional computed tomography imagingand treatment planning. Seminars in Radiation Oncology, 14(1),81-90.LI Congli, XUE Mogen, LENG Xiaoyan, LU Wei. 2008.Analysis and elimination on aerial recon sequential imagestitching accumulative error. Journal of Image and Graphics JImage Graph, 13(4), 814-819.Lowe, D. G. 2004. Lowe, d.g.: distinctive image features fromscale-invariant key-points. int. j. comput. vision 60(2), 91-110.International Journal of Computer Vision, 60(2).REFERENCESTang, J., Shi-Wen, G. U., & Cai, Z. X. 2003. Theoreticalanalysis on image stitching error. Mini-micro Systems.Burt, P. J. 1983. A multiresolution spline with application toimage mosaics. Acm Transactions on Graphics, 2(4), 217-236.Yang,ZL. 2008. Research on image registration and mosaicbased on feature point. Xidian University.Fischler, M. A., & Bolles, R. C. 1987. Random sampleconsensus: a paradigm for model fitting with applications toimage analysis and automated cartography. Communications ofZhao,Hui. 2006. Research on image registration algorithmbased on point feature. Shandong University.This contribution has been es-XLII-2-W7-845-2017 Authors 2017. CC BY 4.0 License.850
Parallel computing is achieved by combining OpenMP technology in multi-level mosaic. The algorithm implementation is described in the form of pseudo-code: //start parallel process . #omp sections //Lv1 Mosaic start {#omp section . Img12 Mosaic2img(Img2, Img1) #omp section . Img34 Mosaic2img(Img4,Img3); #omp section . Img56 Mosaic2img(Img6,Img5);
Mosaic Group tables. For the Mosaic table, list below the top three to four Mosaic Segments in your area. You want to identify the groups that make up the first 50% of your population. Mosaic Population Percentage Mosaic Segments & Groups Experian Marketing Services Mosaic USA is a household-based
1. micro mosaic 5 petal flower form base w/raised 6 section center dome - 1 ¾ x 11 ½" 2. micro mosaic 6 petal round dome shape - 1 1/8 x 10 ½ 3. milk gls embossed to resemble micro mosaic - 1 1/8 x 7 4. micro mosaic hexagon form w/floral center - 1 x 9 5. micro mosaic w/floral bouquet center surrounded by 8 floral petals - 1 ½ x 7 ¾ 6.
mosaic variants from a cohort of 12,000 samples submitted f or clinical exome sequencing (ES) at Baylor Genetics. Results: We found 120 mosaic variants involving 107 genes, including 80 mosaic SNVs in proband samples and 40 in parental/grandparental samples. Average mosaic alternate allele fraction (AAF) detected in autosomes and in X-linked
www.biotone.fr - www.shop.biotone.fr Smart Li-ion Power www.rexton.com - www.sivantos.com Nouveau Nouveau TM Tinnitus Smart Key Bluetooth Direct App. Smart Remote TM Smart Mic App. Smart Direct. Piles 13 13 Platine Mosaic P 80 8C 48-20 Mosaic M 80 8C 48-20 Platine Mosaic P 60 8C 32-16 Mosaic M 60 8C 32-16 Or Mosaic P 40 8C
Introduction to Mosaic 2009 Groups and Types September 2010 From September 2010, Audiences London's Snapshot London benchmarking package has analysed box office and other audience data using the Mosaic 2009 Groups and Types. The number . Mosaic is a product of the Experian company www.experian.co.uk .
MOSAIC Program Overview 4 In keeping with the goal of MOSAIC to achieve the form-factor of 1-sun panels while approaching the harvesting performance of CPV, opportunities for MOSAIC technology are expected to lie in the region between CPV and 1-sun projections, as depicted. Figure 3. PV system energy harv
Window Manager (GNOME, Unity, KDE etc) may over-ride MOSAIC settings. 1x3 MOSAIC –but three separate Desktops MOSAIC is running –i.e. Windows should open full screen 1x3 MOSAIC –Single Desktop Option "nvidiaXineramaInfo" "False" Option "RANDR" "Disable"
Accounting implications of the effects of coronavirus At a glance This In depth considers the impact of the new coronavirus (‘COVID-19’ or ‘the virus’) on the financial statements for periods ending after 31 December 2019 of entities whose business is affected by the virus. There are broad IFRS implications, including: non-financial assets; financial instruments and leases; revenue .
