Supplementary Material For Rel3D: A Minimally Contrastive .

Supplementary Material for Rel3D: A MinimallyContrastive Benchmark for Grounding SpatialRelations in 3DAnkit Goyal† , Kaiyu Yang† , Dawei Yang†‡ , Jia Deng†‡University of Michigan, Ann Arbor, MI†Princeton University, Princeton, NJ{agoyal, kaiyuy, daweiy, jiadeng}@princeton.edu1Predicate VocabularyThere are in total 27336 images in Rel3D. Figure. 1 plots the number of images per predicate.onunderoveron top ofto the side ofinto the left of (wrt you)to the right of (wrt you)faces awaybelowfaces towardsin front ofbehindaroundleaning againstpoints awayneartouchingbehind (wrt you)aligned topoints towardsinsidecoveringin front of (wrt you)to the right ofto the left offar frompassing throughto the side of (wrt you)outside0250 500 750 1000 1250 1500 1750Number ImagesFigure 1: Number of images per predicate in Rel3D34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

2Object VocabularyTable 1: Object categories in Rel3D along with their source.ShapeNetSem Only YCB Only Both in YCB and ShapeNetSem SimpleWordSpeakerCanCereal BoxBuildingComputerFruitsBottleMountainPicture dTreeCameraWallMedia ruckBirdTeapotAirplaneBowlForkSpoonCouchChild BedVaseToiletSinkSuitcaseBikeTable 2: Mapping between categories in ShapeNetSem and Rel3D. All categories in ShapeNetSemnot present in Rel3D either have few shapes or have spatial relations similar to some category inRel3D.2

ShapeNetSem ure Table,AccentTableCameraMedia WithMirrorChild ChairsIn sYesYesYes3 5 shapesRel3D Cat.PictureChairSpeakerSimilar Cat. in Rel3DPlantChairChairComputerChairTablePicture FrameCouchBottleMedia ictureBottleMedia StorageCupBedBookChairBookChairPlantComputerCereal meraMedia StorageTablePlantGunMedia StorageChild BedMedia StorageVaseMedia StorageChair

Lamp,WallLampTvStand,Media moire,Wardrobe,ChestOfDrawersPicture PlantMedia StoragePictureBlockWasherKnifeMedia ControllerMedia StorageChairPlantMedia StorageMedia StoragePicture FrameMedia tureKnifePlantRingCapTableChairMedia mal

real BoxPaintingWasherBusBed,DoubleBedCereal YesYesYesYesYesYesYesYes5Media StorageTruckMedia StorageBedMedia StorageTableSinkCellPhoneBedAnimalBlock, BowlChairClockTableYesClockStoneBottleYesForkMedia aseControllerAnimalController, al BoxPictureWasherBusBedCereal Box

nBoatFishTablePlate, ruckYesAnimalBedDeskBedYesChairPictureBowlBlock, BookDoorChairMedia StorageYesYesYesDeskTableYesMedia nimalMedia StorageCameraBikeBike

No7YesPictureYesYesYesYesYesCapBowlMedia StorageYesChairYesYesMedia StorageMedia dia StorageTruckBowlSinkTeapotAnimalControllerChild ctureGlassesTableYesYesYesYesChild BedAnimalYesPlantYesClockBedBlock, Controller

Toy,AirplaneNoAirplaneTable 3: Mapping between objects in YCB and Rel3D.Object IDRel3D Category Reason for not including in Rel3D001 chips canCan002 master chef canCan003 cracker boxCereal Box004 sugar boxCereal Box005 tomato soup canCan006 mustard bottleBottle007 tuna fish canCan008 pudding boxCereal Box009 gelatin boxCereal Box010 potted meat canCan011 bananaFruits012 strawberryFruits013 appleFruits014 lemonFruits015 peachBad reconstruction016 pearFruits017 orangeFruits018 plumBad reconstuction019 pitcher baseBad reconstuction021 bleach cleanserBottle022 windex bottleLargely distorted023 wine glassMissing Object024 bowlBad reconstuction025 mugCup026 spongeBlock027 skilletBad reconstuction028 skillet lidMissing object029 plateBad reconstuction030 forkBad reconstuction031 spoonBad reconstuction032 knifeBad reconstuction033 spatulaMissing object035 power drillOnly 1 shape036 wood blockBlock037 scissorsOnly 1 shape038 padlockBad reconstruction039 keyMissing object040 large markerOnly 1 shape041 small markerBad reconstuction042 adjustable wrenchBad reconstuction043 phillips screwdriverBad reconstuction044 flat screwdriverBad reconstuction048 hammerOnly 1 shape049 small clampMissing .obj file050 medium clampBad reconstuction051 large clampBad reconstuction052 extra large clampOnly 1 Shape053 mini soccer ballBad reconstuction054 softballBall055 baseballBall056 tennis ballBall8

057 racquetball058 golf ball059 chain061 foam brick062 dice063-a marbles063-b marbles063-c marbles063-d marbles063-e marbles063-f marbles065-a cups065-b cups065-c cups065-d cups065-e cups065-f cups065-g cups065-h cups065-i cups065-j cups070-a colored wood blocks070-b colored wood blocks071 nine hole peg test072-a toy airplane072-b toy airplane072-c toy airplane072-d toy airplane072-e toy airplane072-f toy airplane072-g toy airplane072-h toy airplane072-i toy airplane072-j toy airplane072-k toy airplane073-a lego duplo073-b lego duplo073-c lego duplo073-d lego duplo073-e lego duplo073-f lego duplo073-g lego duplo073-h lego duplo073-i lego duplo073-j lego duplo073-k lego duplo073-l lego duplo073-m lego duplo076 timer077 rubiks cube078 tshirtBallBallBad reconstructionBlockBad reconstructionBad reconstructionMissing objectsMissing objectsMissing objectsMissing objectsMissing objectsBad reconstruction, looks like a blobBad reconstructionBad reconstructionCupCupCupCupBad reconstructionCupCupBad reconstructionMissing objectsBlockSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartSmall Object PartOnly 1 shapeBlockNo obj model9

3Relation Plotspositivenegativepositivenegativealigned topositivenegativearound332211y0 1 2 330323 3 2 10x 3 2123positivenegative3y332110 z0 z 3 1 2 10x 3 21231 3 3310 z 3 1 2 10x 3 2123 1 2 3 1(j) in front of0x 3 2123 1 2 3(k) inpositivenegativenear3 2123 3on32211y00x(l) inside2y 1positivenegativeon top of1 1321y0 1 1 2 2 2 2 3 3 333232211110 z0 z0 z0 z 3 1 2 10x 3 2123(n) near 10x 3 2123 1 2 3 1(o) on top ofpositivenegativeover333221y00x 2123 3(p) onpositivenegativepassing through2points away321y01y0 1 1 1 1 2 2 2 2 3 3 33 10x 3 2123210 z 1 23210 z 3 33210 z 1 1 2 3y0 33(q) outside210 z 3 23210 z 3 3 33210 z23y0 1positivenegative21y0 2 21321y 110inside 21x 3 10 13 2outside 22 1(m) leaning against 3 21 2 1positivenegative0x(h) far from30 1positivenegativein2y 1 2 333332positivenegative22321 3 21(g) faces towards 100x20x 1positivenegativein front of3 1 1 2 3(f) faces awayleaning against 2210 z(i) in front of (wrt you) 33210 z 3y 1 2 1positivenegative 1 3positivenegative33210 2 22far from 3 31(d) behindy0 130 3 22x3 31 12 2in front of (wrt you) 2 21 30 30x3y0 1positivenegativefaces towards2(e) coveringpositivenegative 2 31 1233 212230 3 2112x0x2 1 1 110 2 2 1(c) behind (wrt you)faces away3 3210 z 1 3(b) aroundpositivenegativecovering3210 z 1 3(a) aligned topositivenegative3210 z 21 1 2 32x 1 2 31y0 20 z 1321y0 3 1 2behind1y0 1 3positivenegativebehind (wrt you)32 3(r) over 1 2 10x 2123 3(s) passing through100 z 3 1 2 10x 2123 3(t) points awayy

positivenegativepositivenegativepositivenegativeto the left of (wrt you)points towards0 1 20 z 3 10x23 20x0 z 2123 32210y0 z 1 2123positivenegative0x 2123 3(x) to the right of (wrtyou)positivenegativeto the side of0x230 1 1 2 23210 z 3 1 3 2 1(z) to the side of (wrtyou)positivenegative0x 2123 3(aa) to the side ofy 3321 211y 321 13210 1 2touching320 z 3 1 3y 3(y) to the right of33 33202 31x 2 21 2 1 10x 1 2 2 1(w) to the left of331 3 2 1to the side of (wrt you)to the right of0 3 1 3(v) to the left of (wrtyou)positivenegative10 z1 3(u) points towardspositivenegative 1232 1 213 3210 z1 1 2 3 23 33 10y0y 1 32121 2 1 2323y0yto the right of (wrt you)to the left of121 3positivenegative3230 z 3 1 2 10x 2123 3(ab) touchingunder3210y 1 2 33210 z 3 1 2 10x 2123(ac) under 3Figure 2: Each dot represents a scene in our dataset (blue means positive examples and red fornegative). The location of the dot represent the relative position of the object w.r.t. to the subject inthe frame of reference of the observer.11

4Model HyperparametersTable 4: All the hyper-parameters we used to tune the baseline models in Rel3D. The values in boldrepresent the hyperparameter choice that performed the best on the validation set.Modell2 regularizationFeature dimRoi size Back-bone0, 1e-4, 1e-30, 1e-6, 1e-4, 1e-20, 1e-6, 1e-40, 1e-6, 1e-4, 1e-30, 1e-6, 1e-42DVtranEVipCNNDRNetPPFRCN1, 3, 53, 5, 7, 9resnet18resnet18, resnet1013resnet18, resnet10164, 128, 256, neaonontopoutsidoverpointtotothelert ytotheou)rightoftotherightof(wtort ytheou)sideoftothesideof (wrt 230.7520.625 0.6060.8440.7770.7930.5130.5670.830.7520.633 010.8170.9130.5580.9390.690.9030.667MLP (Aligned .674offt of (wlefts towardsthethroughawaypointspassingeofrningagainstf (wrt yoffronttosms towardfrofacengsawayndnedundto(wrt you)ou)ModelaroResults Predicate-wisealig564, 128, 256, 512128, 256, 512, 1024, 2048, 4096MLP (Raw 70.9550.938 0.9860.950.9880.9580.9830.9510.9860.93212

Controller Knife Cap Animal Clock Ladder Car Washer Bus Boat Fish Ring Truck Bird Teapot Airplane Bowl Fork Spoon . Picture,Painting Yes Picture Airplane Yes Airplane Toilet Yes Toilet Chair,AccentChair Yes Chair . Gamecube,VideoGameConsole Yes Controller StaplerWithStaples No Yes Cabinet,ChestOfDrawers Yes Media Storage