Transfer Meets Hybrid: A Synthetic Approach For Cross .

knowledge, we propose a novel neural model, TMH, for crossdomain recommendation with unstructured text. TMH can not onlyattentively extract useful content via a memory network but alsoselectively transfer knowledge across domains by a novel transfernetwork. A shared layer of feature interactions is stacked on the topto couple high-level representations learned from both networks.On real-world datasets, TMH shows the better performance invarious settings. We conduct thorough analyses to understand howthe content and transferred knowledge help the proposed model.Our contributions are summarized as follows: The proposed model is a novel deep model that transferscross-domain knowledge for recommendation with unstructured text by using an attention based neural network. (Sec. 4) We interpret the memory networks to attentively exploit thetext content to match word semantics with user preferences. The transfer network can selectively transfer source itemswith the guidance of target user-item interactions by theattentive weights. Our model alleviates cold-user and cold-item start issues,and outperforms various baselines on real-world datasets.(Sec. 5)2RELATED WORKSWe review related works on three topics: collaborative filtering,hybrid methods, and cross-domain recommendation.Collaborative Filtering Recommender systems aim at learninguser preferences on unknown items from their past history. Contentbased recommendations are based on the matching between userprofiles and item descriptions. It is difficult to build the profile foreach user when there is no/few content. CF alleviates this issueby predicting user preferences based on the user-item interactionbehavior, agnostic to the content [9]. Latent factor models learnfeature vectors for users and items mainly based on MF [26] whichhas probabilistic interpretations [35]. Factorization machines (FM)can mimic MF [40]. To address the data sparsity, an item-item matrix called Shifted Positive Pointiest Mutual Information (SPPMI) isconstructed from the user-item interaction matrix in the CoFactormodel [28]. It then simultaneously factorizes the interaction matrix and the SPPMI matrix in a shared item latent space, enablingthe usage of co-click information to regularize the learning of theuser-item matrix. In contrast, we use independent unstructured textand source domain information to alleviate the data sparsity issuein the user-item matrix. Neural networks are proposed to pushthe learning of feature vectors towards non-linear representations,including the Neural Network Matrix Factorization (NNMF) andMultiLayer Perceptron (MLP) [10, 17]. The basic MLP architectureis extended to regularize the factors of users and items via socialand geographical information [51]. Other neural approaches learnfrom the explicit feedback for the rating prediction task [5, 56]. Wefocus on learning from the implicit feedback for top-N recommendation [50].Hybrid Filtering Items are usually associated with the contentinformation such as unstructured text (e.g., abstracts of articles andreviews of products). CF approaches can be extended to exploitthe content information [1, 47, 48] and user reviews [16, 20, 33].Combining matrix factorization and topic modelling technique (e.g.,2823Topic MF) is an effective way to integrate ratings with item contents [2, 29, 33]. Item reviews justify the rating behavior of a user,and item ratings are associated with their attributes hidden in reviews [14]. Topic MF methods combine latent item factors in ratingswith latent topics in reviews [2, 33]. The behavior factors and topicfactors are aligned with a link function such as softmax transformation in the Hidden Factors and hidden Topics (HFT) model [33]or an offset deviation in the Collaborative Topic Regression (CTR)model [47]. The CTR model assumes the latent item vector learntfrom the interaction data is close to the corresponding topic proportions learnt from the text content, but allows them to be divergentfrom each other if necessary. Additional sources of informationare integrated into CF to alleviate the data sparsity issues including knowledge graph [49, 55]. Convolutional Networks (CNNs)have been used to extract features from audio signals for musicrecommendation [45] and from image for product and multimediarecommendation [6, 16]. Autoencoders are used to learn an intermediate representations from text [48, 53]. Recurrent networks [1] andconvolutional networks [5, 24, 56] can exploit the word order whenlearning the text representations. Memory networks can reasonwith an external memory [43]. Due to the capability of neurallylearnt word embeddings to address the problems of word sparseness and semantic gap, a memory module can be used to modelitem content [19] or the neighborhood of users [11]. Memory networks can learn to match word semantics with the specific user. Wefollow this thread by using neural networks to attentively extractimportant information from text content.Cross-domain Recommendation Cross-domain recommendation [3] is an effective technique to alleviate the data sparsity issue.A class of MF-based methods has been applied to cross-domain recommendation. Typical methods include the CMF approach whichjointly factorizes two rating matrices by sharing the latent userfactors and hence it enables knowledge transfer. CMF has its heterogeneous variants [37], and codebook transfer [27]. The coordinatesystem transfer can exploit heterogeneous feedbacks [38, 52]. Multiple source domains [32] and multi-view learning [13] are alsoproposed for integrating information from several domains. Transfer Learning (TL) aims at improving the performance of the targetdomain by exploiting knowledge from source domains [36]. Similarto TL, Multitask Learning (MTL) is to leverage useful knowledge inmultiple related tasks to help each other [4, 54]. The cross-stitchnetwork [34] and its sparse variant [22] enable information sharingbetween two base networks for each domain in a deep way. Robustlearning is also considered during knowledge transfer [15]. Thesemethods treat knowledge transfer as a global process with sharedglobal parameters and do not match source items with the specifictarget item given a user. We follow this research thread by usingneural networks to selectively transfer knowledge from the sourceitems. We introduce a transfer network to exploit the source domainknowledge.3A BASIC NEURAL CF NETWORKWe adopt a Feedforward Neural Network (FFNN) as the base neuralCF model to parameterize the interaction function [7, 8, 17]:f (xui P, Q, θ f ) ϕo (.(ϕ 1 (xui )).),(1)

𝑟𝑢𝑖ƸLoss 𝑟𝑢𝑖SoftmaxShared mapDot prod.Dot iven a document dui (w 1 , w 2 , ., wl ) corresponding to the(u, i) interaction, we form the memory slots mk R2d by mappingeach word w k into an embedding vector with matrix A, wherek 1, ., l and the length of the longest document is equal to thememory size. We form a preference vector q (ui ) corresponding tothe given document dui and the user-item interaction (u, i) whereeach element encodes the relevance of user u to these words given(ui )(u )(i )item i as: qk xuT mk x iT mk , k 1, ., l, where we split theReLUα1Sum𝒙𝑗1 αs𝒙𝑗s(u )H(i )(u )(i )mk [mk , mk ] into the user part mk and the item part mk .Then, we compute the attentive weights over words for a givenuser-item interaction to infer the importance of each word’s uniqueH 𝑗1𝑗𝑠Source items𝒙𝑢𝒙𝑖QPu Useri Target item(ui )contribution: pk(ui )exp(βq(ui ))k Softmax(qk ) P(ui ) , where pak ′ exp(βq k ′ )rameter β is introduced to stabilize the numerical computationand can amplify or attenuate the precision of the attention like1a temperature [18]. We set β d 2 by scaling along with thedimensionality [46].We construct the high-level representations by interpolating theexternal memories with the attentive weights as the output:X(ui )oui pk c k ,(2)Figure 1: The architecture of TMH.where input xui [P T xu , Q T x i ] R2d is concatenated fromembeddings of user u and item i and is a projection of their one-hotencodings xu {0, 1}m and x i {0, 1}n via embedding matricesP Rm d and Q Rn d , respectively. The output and hiddenlayers are computed by ϕo and {ϕl } in FFNN. The sizes of users anditems are m and n, respectively. The rating rui is 1 if user u has ankinteraction with item i and 0 otherwise.The base network consists of four modules with the informationwhere the external memory slot c k Rd is another embeddingflow from the input (u, i) to the output rˆui as follows.vector for word w k by mapping it with matrix C.Input: (u, i) u , i This module encodes user-item interactionSelecting Source Items to Transfer We propose a novel transferindices. We adopt the one-hot encoding. It takes user u and itemnetwork (TNet) which can selectively transfer source knowledge fori, and maps them into one-hot encodings u {0, 1}m and i specific target item in a coarse-to-fine way. Given the source items{0, 1}n where only the element corresponding to that index is 1 and[j]u (j 1 , j 2 , ., js ) with which the user u has interacted in theall others are 0.source domain, TNet learns a transfer vector cui Rd to captureEmbedding: u , i xui This module firstly embeds one-hotthe relations between the target item i and source items given theencodings into continuous representations xu P T u and x i user u. The similarities between target item i and source items can(i )Q T i by embedding matrices P and Q respectively, and then conbe computed by their dot products: a j x iT x j , j 1, ., s, wherecatenates them as xui [xu , x i ], to be the input of followingx j Rd is the embedding for the source item j by an embeddingbuilding blocks.matrix H Rn S d . This score computes the compatibility betweenHidden layers: xui zui . This module takes the continuousthe target item and the source items consumed by the user.representations from the embedding module and then transformsscores to be a probability distrithrough several layers to a final latent representation zui (.(ϕ 1 (xui ).). Then, we normalize similarity(i )(i )butionoversourceitems:α Softmax(aThis module consists of hidden layers to learn nonlinear interactionjj ). Finally the transfervector is a weighted sum of the corresponding source item embedbetween users and items.dings:ˆˆOutput : zui rui . This module predicts the score rui for theX(i )given user-item pair based on the representation zui from the lastcui ReLU(α x j ),(3)j jlayer of multi-hop module. Since we focus on one-class collaborative filtering, the output is the probability that the input pair iswhere we introduce non-linearity on the transfer vector by thea positive interaction. This can be achieved by a softmax layer:rectified linear unit ReLU(x ) max (0, x ).1rˆui ϕo (zui ) 1 exp( hPutting It All Together We firstly use a simple neural CF modelT z ) , where h is the parameter.ui(CFNet) which has one hidden layer to learn a nonlinear representation for the user-item interaction:4 THE PROPOSED TMH MODELThe architecture of the TMH model is illustrated in Fig. 1.Matching Word Semantics with User Preferences We adapt amemory network (MNet) to integrate unstructured text since itcan learn to match word semantics with user preferences [12, 19,23, 44]. The MNet consists of one internal memory matrix A RL 2d , where L is the vocabulary size (typically L 8, 000 afterprocessing [47]) and 2d is the dimension of each memory slot, andone external memory matrix C with the same dimensions as A. Thefunction of the two memory matrices works as follows.2824zui ReLU(W xui b),(4)where W and b are the weight and bias parameters in the hiddenlayer. Usually the dimension of zui is half of that xui in a typicaltower-pattern architecture.The outputs from the three individual networks can be viewedhigh-level features of the content text, source domain knowledge,and the user-item interaction. They come from different featurespace learned by different networks. Thus, we use a shared layer on

1the top of the all features: rˆui 1 exp( hT y ) , where h is the pauirameter. The joint representation, yui [Wo oui ,Wz zui ,Wc cui ],is concatenated from the linear mapped outputs of individual networks where matrices Wo ,Wz ,Wc are the corresponding linearmapping transformations.Learning Due to the nature of the implicit feedback and the taskof item recommendation, the squared loss (rˆui rui ) 2 may be notsuitable since it is usually for rating prediction. Instead, we adoptPthe binary cross-entropy loss: L (u,i ) S rui log rˆui (1 rui ) log(1 rˆui ), where the training samples S RT RT are theunion of observed target interaction matrix and randomly samplednegative pairs. Usually, RT RT and we do not perform a predefined negative sampling in advance since this can only generate afixed training set of negative samples. Instead, we generate negativesamples during each epoch, enabling diverse and augmented training sets of negative examples to be used. The objective function canbe optimized by stochastic gradient descent (SGD) and its variantslike adaptive moment(Adam) method [25].Complexity In the model parameters Θ, the embedding matrices P, Q and H contain a large number of parameters since theydepend on the input size of users and (target and source) items,and their scale is hundreds of thousands. Typically, the number ofwords, i.e., the vocabulary size is L 8, 000 [47]. The dimension ofembeddings is typically d 100. Since the architecture follows atower pattern, the dimension of the outputs of the three individualnetworks is also limited within hundreds. In total, the size of modelparameters is linear with the input size and is close to the size oftypical latent factors models [42] and neural CF approaches [17]with a hidden layer. During training, we compute the outputs of thethree individual networks in parallel using mini-batch stochasticoptimization which can be trained efficiently by back-propagation.TMH is scalable to the number of the training data. It can easilyupdate when new data examples come by just feeding them intothe training mini-batch. Thus, TMH c

A Synthetic Approach for Cross-Domain Collaborative Filtering with Text. In Proceedings of the 2019 World Wide Web Conference (WWW’19), May . recommendation [45] and from image for product and multimedia recommendation [6, 16]. Autoencoders are used to learn an interme-