Introduction To PyTorch Lecture 4 - Forsiden

IN5400 Machine learning for image analysis, 2020 springTorch.tensor summaryVery similar to numpy (indexing, main functions)Every tensor has a device, a type, and a required grad attributeConversion and/or device transfer might be needed.In-place operations end in underscore, e.g. .fill ()Some operations create new tensors, some share data.Careful with the broadcasting semantics.Remark: not just tensors are similar to ndarrays,but torch functions are also similar to numpy functions.XPage 30 / 84

IN5400 Machine learning for image analysis, 2020 springProgressDeep learning frameworksPyTorchtorch.tensorComputational graph (reminder from last week)Automatic differentiation (torch.autograd)Data loading and preprocessing (torch.utils)Useful functions (torch.nn.functional)Creating the model (torch.nn)Optimizers (torch.optim)Save/load modelsMiscellaneousXPage 31 / 84

IN5400 Machine learning for image analysis, 2020 springWhat is a computational graph?XPage 32 / 84

IN5400 Machine learning for image analysis, 2020 springForward propagationXPage 33 / 84

IN5400 Machine learning for image analysis, 2020 springBackward propagationWhat if we want to get the derivative of f with respect to the x1?XPage 34 / 84

IN5400 Machine learning for image analysis, 2020 springProgressDeep learning frameworksPyTorchtorch.tensorComputational graphAutomatic differentiation ( torch.autograd )Data loading and preprocessing (torch.utils)Useful functions (torch.nn.functional)Creating the model (torch.nn)Optimizers (torch.optim)Save/load modelsMiscellaneousXPage 35 / 84

IN5400 Machine learning for image analysis, 2020 springAutogradAutograd - Automatic differentiation for all operations on TensorsStatic computational graph (TensorFlow 1.0)Dynamic computational graph (PyTorch)The backward graph is defined by the forward run!XPage 36 / 84

IN5400 Machine learning for image analysis, 2020 springExample 1 (autograd) import torch from torch import autograd x1 torch.tensor(2, requires grad True, dtype torch.float32) x2 torch.tensor(3, requires grad True, dtype torch.float32) x3 torch.tensor(1, requires grad True, dtype torch.float32) x4 torch.tensor(4, requires grad True, dtype torch.float32) # Forward propagation z1 x1 * x2 z2 x3 * x4 f z1 z2 df dx grad(outputs f, inputs [x1, x2, x3, x4]) df dx(tensor(3.), tensor(2.), tensor(4.), tensor(1.))XPage 37 / 84

IN5400 Machine learning for image analysis, 2020 springExample 1 (autograd) df dx grad(outputs f, inputs [x1, x2, x3, x4]) df dx(tensor(3.), tensor(2.), tensor(4.), tensor(1.))XPage 38 / 84

IN5400 Machine learning for image analysis, 2020 springLeaf tensorA «leaf tensor» is a tensor you created directly, not as the result of an operation. x torch.tensor(2) # x is a leaf tensor y x 1# y is not a leaf tensorRemember the computation graph:x1, x2, x3, x4 are the leaf tensors.XPage 39 / 84

IN5400 Machine learning for image analysis, 2020 springAutogradThe need for specifying all tensors is inconvenient. import torch from torch import autograd x1 torch.tensor(2, requires grad True, dtype torch.float32) x2 torch.tensor(3, requires grad True, dtype torch.float32) x3 torch.tensor(1, requires grad True, dtype torch.float32) x4 torch.tensor(4, requires grad True, dtype torch.float32) # Forward propagation z1 x1 * x2 z2 x3 * x4 f z1 z2 # df dx grad(outputs f, inputs [x1, x2, x3, x4]) # inconvenient f.backward()# that is better! print(f" f's derivative w.r.t. x1 is {x.grad}")tensor(3.)Chain rule is applied back to all the leaf tensors with requires grad Trueattribute.XPage 40 / 84

IN5400 Machine learning for image analysis, 2020 springXPage 41 / 84

IN5400 Machine learning for image analysis, 2020 springContext managers, decoratorsWe can locally disable/enable gradient calculation withtorch.no grad()torch.enable grad()or using the @torch.no grad @torch.enable grad decorators x torch.tensor([1], requires grad True) with torch.no grad():. y x * 2 y.requires gradFalse with torch.no grad():. with torch.enable grad():.y x * 2 y.requires gradTrueNote: Use «torch.no grad()» during inferenceXPage 42 / 84

IN5400 Machine learning for image analysis, 2020 springAutograd in depth (optional)https://www.youtube.com/watch?v d.htmlXPage 43 / 84

IN5400 Machine learning for image analysis, 2020 springExample 2 - Solving a linear problemGenerating data: a ref -1.5b ref 8noise 0.2 * np.random.randn(50)x np.linspace(1, 4, 50)y a ref * x b ref noiseDefining loss function: def MSE loss(prediction, target):.return (prediction-target).pow(2).mean()XPage 44 / 84

IN5400 Machine learning for image analysis, 2020 springExample 2 - Solving a linear problemData as torch tensors and the unknown variables:xx torch.tensor(x, dtype torch.float32)yy torch.tensor(y, dtype torch.float32)a torch.tensors(0, requires grad True, dtype torch.float32)b torch.tensors(5, requires grad True, dtype torch.float32)XPage 45 / 84

IN5400 Machine learning for image analysis, 2020 springExample 2 - Solving a linear problemTraining loop:number of epochs 1000learning rate 0.01for iteration in range(number of epochs):y pred a * xx bloss MSE loss(y pred, yy)loss.backward()with torch.no grad():a a - learning rate * a.gradb b - learning rate * b.grada.requires grad Trueb.requires grad Trueprint(a)print(b)XPage 46 / 84

IN5400 Machine learning for image analysis, 2020 springExample 2 - Solving a linear problemResult:tensor(-1.5061, requires grad True)tensor(8.0354, requires grad True)XPage 47 / 84

IN5400 Machine learning for image analysis, 2020 springOther useful torch.tensor functionsIf you want to detach a tensor from the graph, you can use « detach() »If you want to get a python number from a tensor, you can use « item() »But if you just take an element, it still will be part of the computational graph! x torch.tensor([2.5,3.5], requires grad True)tensor([2.5000, 3.5000], requires grad True) x.detach()tensor([2.5000, 3.5000]) x[0]# still part of the graph!tensor(2.5000, grad fn SelectBackward ) x[0].item()2.5 # a frequent line when you go back to numpy: x.detach().cpu().numpy()array([2.5, 3.5], dtype float32)XPage 48 / 84

IN5400 Machine learning for image analysis, 2020 springRemember our example workflowcreating datasetcreating a neural network (model)defining a loss functionloading samples (data loader)predicting with the modelcomparison of the prediction and the target (loss)backpropagation: calculating gradients from the errorupdating the model (optimizer)checking the loss: if low enough, stop trainingXPage 49 / 84

IN5400 Machine learning for image analysis, 2020 springProgressDeep learning frameworksPyTorchtorch.tensorComputational graphAutomatic differentiation (torch.autograd)Data loading and preprocessing ( torch.utils )Useful functions (torch.nn.functional)Creating the model (torch.nn)Optimizers (torch.optim)Save/load modelsMiscellaneousXPage 50 / 84

IN5400 Machine learning for image analysis, 2020 springData loading and preprocessingThe «torch.utils.data» package have two useful classes for loading and ils.data.DataLoaderFor more information visit:https://pytorch.org/tutorials/beginner/data loading tutorial.htmlXPage 51 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.utils.data.DatasetTypical structure of the Dataset classimport torchclass ExampleDataset(torch.utils.data.Dataset):def init (self,params, *args,**kwargs):super(). init (*args,**kwargs)# do initalization based on the params,# e.g. load images, etc.self.data .def getitem (self, idx):# return sample indexed with 'idx'# must be tensor or dictionary of tensors!self.data[idx]def len (self):# return the number of samplesreturn self.data.shape[0]XPage 52 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.utils.data.Dataset: regressionimport torchclass def init (self,N 50, m -3, b 2, *args,**kwargs):# N: number of samples, e.g. 50# m: slope# b: offsetsuper(). init (*args,**kwargs)self.x torch.rand(N)self.noise torch.rand(N)*0.2self.m mself.b bdef getitem (self, idx):y self.x[idx] * self.m self.b self.noise[idx]return {'input': self.x[idx], 'target': y}def len (self):return len(self.x)XPage 53 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.utils.data.Dataset: imagesimport torchimport imageioclass ImageDataset(torch.utils.data.Dataset):def init (self, root, N, *args,**kwargs):super(). init (*args,**kwargs)self.input, self.target [], []for i in range(N):t imageio.imread(f'{root}/train {i}.png')t torch.from numpy(t).permute(2,0,1)l imageio.imread(f'target {i}.png')l torch.from rget.append(l)def getitem (self, idx):return {'input': self.input[idx], 'target': self.target[idx]}def len (self):return len(self.input)XPage 54 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.utils.data.Datasetimport torchimport ImageDatasetdatapath 'data directory'myImageDataset ImageDataset(dataPath, 50)# iterating through the samplesfor sample in myImageDataset:input sample['input'].cpu() # or .cuda()target sample['target'].cpu() # or .to(device).Never ever use .cuda() in the dataset or data loaders!XPage 55 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.utils.data.DataLoaderimport torchimport ImageDatasetdatapath 'data directory'myImageDataset ImageDataset(dataPath, 50)# iterating through the samplestrain loader DataLoader(dataset myImageDataset, batch size 32,shuffle False, num workers 2)for sample in train loader:.«DataLoader» is used to:Batching the datasetShuffling the datasetUtilizing multiple CPU cores/ threadsXPage 56 / 84

IN5400 Machine learning for image analysis, 2020 springData augmentetion(forward reference, not for today)modifying the dataset for better training(more robust, etc.)data set can have a a transform parameterDetails here:https://pytorch.org/tutorials/beginner/data loading tutorial.htmlXPage 57 / 84

IN5400 Machine learning for image analysis, 2020 springData augmentetion(forward reference, not for today)import torchimport imageioclass ImageDataset(torch.utils.data.Dataset):def init (self, root, N, transform None, *args,**kwargs):super(). init (*args,**kwargs)self.transform transform.def getitem (self, idx):sample {'input': self.input[idx], 'target': self.target[idx]}if self.transform:sample self.transform(sample)return sampledef len (self):return len(self.input)XPage 58 / 84

IN5400 Machine learning for image analysis, 2020 springData augmentetion(forward reference, not for today)class ImageDataset(torch.utils.data.Dataset):def init (self, root, N, transform None, *args,**kwargs):super(). init (*args,**kwargs)self.transform transform.def getitem (self, idx):sample {'input': self.input[idx], 'target': self.target[idx]}if self.transform:sample self.transform(sample)return sampledef len (self):return len(self.input)XPage 59 / 84

IN5400 Machine learning for image analysis, 2020 springData transformations(forward reference, not for today)import torchvision.transforms as Tcomposed 4),T.ToTensor()]).dataset Mydataset(., transform composed)# another version, needs different datasetdataset Mydataset(., transform {'input' : composed,'target' : None} )XPage 60 / 84

IN5400 Machine learning for image analysis, 2020 springProgressDeep learning frameworksPyTorchtorch.tensorComputational graphAutomatic differentiation (torch.autograd)Data loading and preprocessing (torch.utils)Useful functions ( torch.nn.functional )Creating the model (torch.nn)Optimizers (torch.optim)Save/load modelsMiscellaneousXPage 61 / 84

IN5400 Machine learning for image analysis, 2020 springtorch.nn.functionalThe «torch.nn.functional» package is the functional interface for Pytorch features.Most feature exist both as a function and as a class.Structural parts, or objects with internal state usually used as objectsStateless or simple expressions are usually used in functional form.Activation functions, losses, convolutions, etc. It is a huge module.import torchimport torch.nn as nnimport torch.nn.functional as Fx torch.rand(2,2)y F.relu(x)relu nn.ReLU() # creating the object firstz relu(x)# then using ity z# they should be the same# Similarly:mseloss nn.MSELoss()F.mseloss(.) mseloss(.)XPage 62 / 84

IN5400 Machine learning for image analysis, 2020 springProgressDeep learning frameworksPyTorchtorch.tensorComputational graphAutomatic differentiation (torch.autograd)Data loading and preprocessing (torch.utils)Useful functions (torch.nn.functional)Creating the model ( torch.nn )Optimizers (torch.optim)Save/load modelsMiscellaneousXPage 63 / 84

IN5400 Machine learning for image analysis, 2020 springCreating the modelA model is of a nn.Module class type. A model can contain other models. E.g. we cancreate the class “Model” based on the stacking nn.Modules of type nn.Linear()The nn.Module’s weights as called “Parameters”, and are similar to tensors with“requires grad True”.A nn.Module consists of an initialization of the Parameters and a forward function.class Model(nn.Module):def init (self):super(). init ()# structure definition and initializationdef forward(self, x):# actual forward propagationresult processing(x)return resultXPage 64 / 84

IN5400 Machine learning for image analysis, 2020 springCreating the modelclass Model(nn.Module):def init (self):super(). init ()# let's assume 28x28 input images,self.fc1 nn.Linear(in features self.fc2 nn.Linear(in features self.fc3 nn.Linear(in features e.g. MNIST characters28 * 28, out features 128, bias True)128, out features 64, bias True)64, out features 10, bias True)def forward(self, x):x F.relu(self.fc1(x))x F.relu(self.fc2(x))x self.fc3(x)return xXPage 65 / 84

IN5400 Machine learning for image analysis, 2020 springCreating the model, alternative wayclass Model2(nn.Module):def init (self):super(). init ()# let's assume 28x28 input images,self.fc1 nn.Linear(in features self.activation1 nn.ReLU()self.fc2 nn.Linear(in features self.activation2 nn.ReLU()self.fc3 nn.Linear(in features self.activation3 nn.ReLU()e.g. MNIST characters28 * 28, out features 128, bias True)128, out features 64, bias True)64, out features 10, bias True)def forward(self, x):x self.activation1(self.fc1(x))x self.activation2(self.fc2(x))x self.activation3(self.fc3(x))return xWhat is the difference?XPage 66 / 84

IN5400 Machine learning for image analysis, 2020 springnn.Module’s member functionsAccess information of a model: model Model() model.eval() # see below list(model.children())[Linear(in features 784, out features 128, bias True),Linear(in features 128, out features 64, bias True),Linear(in features 64, out features 10, bias True)]Children: the parameters and modules / layers defined in the constructor.Parts defined in the forward method will not be listed.Forward is called many times, expensive objects should not be recreated.Some layers as e.g. "dropout" and "batch norm" should operate differently during trainingand evaluation of the model. We can set the model in different state by the .train() and.eval() functions.XPage 67 / 84

IN5400 Machine learning for image analysis, 2020 springModel parameters for key, value in model.state dict().items():.print(f'layer {key:10s} feature shape {value.shape}')layerlayerlayerlayerlayerlayer bias peshapeshapeshapeshape torch.Size([128, 784])torch.Size([128])torch.Size([64, 128])torch.Size([64])torch.Size([10, 64])torch.Size([10])The .state dict() contains all the trainable parameters of the model,this is used for optimization and saving/restoring the model. (See later.)XPage 68 / 84

IN5400 Machine learning for image analysis, 2020 springAdvanced examples(Not part of the mandatory curriculum for today)Naive implementation of ReLU:class ReLU(nn.Module):def init (self):super(). init ()def forward(self, x):x x.clone()x[x 0] 0# implements max(x,0)return xYou can implement any activation function, any transformation,and autograd tracks everything.XPage 69 / 84

IN5400 Machine learning for image analysis, 2020 springAdvanced examples, part 2(Not part of the mandatory curriculum for today)Skip connections and residual connections:class SkipResBlock(nn.Module):def init (self):super(). init ()self.convolution nn.ConvX(.)self.convolution2 nn.ConvX(.)def forward(self, x):y self.convolution(x) x # residual blocky F.relu(y)z torch.cat(y,x, dim .) # skip connectionz self.convolut

Some tutorials use Python 2. Stick to Python 3.6 . IN5400 Machine learning for image analysis, 2020 spring . import numpy as np import torch import sys import matplotlib . 32-bit floating point torch.float32 or torch.float torch.FloatTensor torch.cuda.FloatTensor