基于邻域的算法是个性化推荐系统中最基本的算法，该算法不仅在学术界得到了深入研究，而且在业界得到了广泛应用。基于邻域的算法分为两大类，一类是基于用户的协同过滤算法，另一类是基于物品的协同过滤算法。本文主要研究基于物品的协同过滤算法和基于ALS协同过滤算法。
一、基于物品的协同过滤算法
1.基本思想
ItemCF算法通过计算用户的历史行为记录，来分析物品之间的相似度：如果喜欢物品A的用户大多数也喜欢物品B，那么认为物品A与物品B具有一定的相似度。这就很容易为推荐结果做出合理的解释。例如，如果你购买过《数据挖掘导论》，会向你推荐《机器学习》。
2.相似度度量
如何度量物品间相似度，常用的相似度度量有同现相似度、欧几里得距离、皮尔逊相关系数、余弦相似度、jaccard距离等，具体如下所示。
2.1 同现相似度
同现相似度计算公式如下：
公式中分母是喜欢物品x的用户数，而分子则是同时对物品x和物品y感兴趣的用户数。因此，上述公式可用理解为对物品x感兴趣的用户有多大概率也对y感兴趣 (和关联规则类似)
但上述的公式存在一个问题，如果物品y是热门物品，有很多人都喜欢，则会导致W(x, y)很大，接近于1。因此会造成任何物品都和热门物品交有很大的相似度。
2.2 改进的同现相似度
针对热门物品对同现相似度影响，引入惩罚了物品y的权重，因此减轻了热门物品和很多物品相似的可能性。改进的计算公式如下：
2.3 欧几里得距离
在数学中，欧几里得距离或欧几里得度量是欧几里得空间中两点间“普通”（即直线）距离。使用这个距离，欧氏空间成为度量空间。相关联的范数称为欧几里得范数。较早的文献称之为毕达哥拉斯度量。计算公式如下：
2.4 皮尔逊相关系数
皮尔逊相关系数，即概率论中的相关系数，取值范围[-1，+1]。当大于零时，两个变量正相关，当小于零时表示两个向量负相关。计算公式如下：
2.5 余弦相似度
利用多维空间两点与所设定的点形成夹角的余弦值范围为[-1,1]，值越大，说明夹角越大，两点相距就越远，相似度就越小。计算公式如下：
该公式只考虑到了用户的评分，很可能评分较高的物品会排在前面而不管物品的其它信息。
2.6 改进的余弦相似度
考虑到了两个向量相同个体个数、X向量大小、Y向量大小，改进的余弦相似度计算公式如下：
2.7 Jaccard距离
此相似度不考虑评价值，只考虑两个集合共同个体数量。计算公式如下：
3. 预测用户评分公式
通过相似度度量可以得到物品间相似度矩阵Score(i,p)，则用户u对物品预测评分计算公式如下：
其中，u为用户，p为物品，ratedItems为用户评价过物品集，r为用户对物品评价分集合。
4. 代码实现
4.1 环境及依赖
java 1.8.0_172+scala 2.11.8+spark.2.3.1
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.3.1</version>
</dependency>4.2 物品相似度度量代码
4.2.1 改进共现相似度
/**
* 改进的共现相似度
* 共现相似度=numRatersPairs/numRaterPair，同时对A和B感兴趣的用户数/对A感兴趣的用户数
* 描述喜欢A(numRaters)的用户有多大概率对B(numRatersPair)感兴趣，但B是热门物品，导致共现相似度为1
* 改进的共现相似度=numRatersPairs/sqrt(numRaters * numRatersPair)，惩罚物品B权重，减轻热门物品和很多相似物品的可能性
* @param numRatersPairs
* @param numRaters
* @param numRatersPair
* @return
*/
def cooccurrence(numRatersPairs:Long,numRaters:Long,numRatersPair:Long):Double = {
numRatersPairs / math.sqrt(numRaters * numRatersPair)
}4.2.2 皮尔逊相关系数
/**
* 皮尔逊相关系数=变量协方差/标准差
* @param size
* @param dotProduct
* @param ratingSum
* @param ratingPairSum
* @param ratingNorm
* @param ratingNormPair
* @return
*/
def correlation(size:Double,dotProduct:Double,ratingSum:Double,ratingPairSum:Double,ratingNorm:Double,ratingNormPair:Double):Double = {
val numerator = size * dotProduct - ratingSum * ratingPairSum
val denomiator = math.sqrt(size * ratingNorm - ratingSum * ratingSum) * math.sqrt(size * ratingNormPair - ratingPairSum * ratingPairSum)
if(denomiator==0) 0 else numerator/denomiator
}4.2.3 改进皮尔逊相关系数
/**
* 正则化相关系数
* @param size
* @param dotProduct
* @param ratingSum
* @param ratingPairSum
* @param ratingNorm
* @param ratingNormPair
* @param virtualCount
* @param priorCorrelation
* @return
*/
def regularCorrelation(size:Double,dotProduct:Double,ratingSum:Double,ratingPairSum:Double,ratingNorm:Double,ratingNormPair:Double,virtualCount:Double,priorCorrelation:Double):Double = {
val unregularizedCorrelation = correlation(size, dotProduct, ratingSum, ratingPairSum, ratingNorm, ratingNormPair)
val w = size/(size + virtualCount)
w * unregularizedCorrelation + (1 - w) * priorCorrelation
}4.2.4 余弦相似度
/**
* 余弦相似度
* @param dotProduct
* @param ratingNorm
* @param ratingNormPair
* @return
*/
def cosineSimilarity(dotProduct:Double, ratingNorm:Double, ratingNormPair:Double):Double = {
dotProduct/(ratingNorm * ratingNormPair)
}4.2.5 改进的余弦相似度
/**
* 改进的余弦相似度
* 考虑两个向量相同个体个数，A向量大小和B向量大小
* @param dotProduct
* @param ratingNorm
* @param ratingNormPair
* @param numPairs
* @param num
* @param numPair
* @return
*/
def improvedCosineSimilarity(dotProduct:Double,ratingNorm:Double,ratingNormPair:Double,numPairs:Long,num:Long,numPair:Long):Double = {
dotProduct * numPairs / (ratingNorm * ratingNormPair * num * math.log10(10 + numPair))
}4.2.6 Jaccard距离
/**
* Jaccard相似度
* @param size
* @param numRaters
* @param numRatersPair
* @return
*/
def jaccardSimilarity(size:Double,numRaters:Double,numRatersPair:Double):Double = {
size/(numRaters + numRatersPair - size)
}4.3 预测用户评分代码
4.3.1 计算物品集相似度矩阵
依次计算改进共现相似度、皮尔逊系数、改进皮尔逊系数、余弦相似度、改进余弦相似度、Jaccard距离，并引入以上度量加权系数和度量，这里的加权系数自定义为coef = (0.1,0.1,0.1,0.2,0.3,0.1)。其中余弦相似度和改进余弦相似度越小，描述物品的相似度越大，因此加权系数和度量将余弦相似度和改进余弦相似度作取负处理。
import spark.implicits._
val rating = spark.read.textFile(path).map(parseRating(_)).toDF()
rating.show(10, false)
// 每个用户评分最高的top10物品
val userRecs = rating.select($"userId", $"movieId", $"rating", functions.row_number().over(Window.partitionBy("userId").orderBy("rating")).alias("rank"))
.filter($"rank" <= 10)
userRecs.show(10, false)
// 获取每个物品评分用户数,item2manyUser格式如下(movieId,numRaters)
// rating.groupBy($"movieId").pivot("rating").count().show(false)
val item2manyUser = rating.groupBy($"movieId").count().toDF("movieId", "numRaters").coalesce(defaultParallelism)
item2manyUser.show(10, false)
// 获取用户对物品评分及评分物品数量,ratingWithSize和ratingWithSizePair格式如下(userId,movieId,rating,timestamp,numRaters)
val ratingsWithSize = rating.join(item2manyUser, "movieId").coalesce(defaultParallelism)
ratingsWithSize.show(10, false)
val ratingsWithSizePair = ratingsWithSize.toDF("movieIdPair", "userId", "ratingPair", "timestampPair", "numRatersPair")
// 获取用户对不同物品的评分的矩阵，并过滤相同item pairs
val ratingPairs = ratingsWithSize.join(ratingsWithSizePair, "userId").where($"movieId" < $"movieIdPair")
.selectExpr("userId", "movieId", "rating", "numRaters", "movieIdPair", "ratingPair", "numRatersPair", "rating * ratingPair as product", "pow(rating,2) as ratingPow", "pow(ratingPair,2) as ratingPairPow")
.coalesce(defaultParallelism)
ratingPairs.show(10, false)
// 计算item pairs的相似度统计量
val vectorCals = ratingPairs.groupBy("movieId", "movieIdPair")
.agg(functions.count("userId").alias("size"),
functions.sum("product").alias("dotProduct"),
functions.sum("rating").alias("ratingSum"),
functions.sum("ratingPair").alias("ratingPairSum"),
functions.sum("ratingPow").alias("ratingPowSum"),
functions.sum("ratingPairPow").alias("ratingPairPowSum"),
functions.first("numRaters").alias("numRaters"),
functions.first("numRatersPair").alias("numRatersPair"))
// .agg(Map("userId"->"count","product"->"sum","rating"->"sum","ratingPair"->"sum","ratingPow"->"sum","ratingPairPow"->"sum","numRaters"->"first","numRatersPair"->"first"))
// .toDF("movieId","movieIdPair","size","dotProduct","ratingSum","ratingPairSum","ratingPowSum","ratingPairPowSum","numRaters","numRatersPair")
.coalesce(defaultParallelism)
vectorCals.show(10, false)
// 计算item pairs的相似度度量(包括:共现相似度、改进共现相似度、皮尔逊系数、改进皮尔逊系数、余弦相似度、改进余弦相似度、Jaccard距离)
val similar = vectorCals.map(row => {
val movieId = row.getAs[Int]("movieId")
val movieIdPair = row.getAs[Int]("movieIdPair")
val size = row.getAs[Long]("size")
val dotProduct = row.getAs[Double]("dotProduct")
val ratingSum = row.getAs[Double]("ratingSum")
val ratingPairSum = row.getAs[Double]("ratingPairSum")
val ratingPowSum = row.getAs[Double]("ratingPowSum")
val ratingPairPowSum = row.getAs[Double]("ratingPairPowSum")
val numRaters = row.getAs[Long]("numRaters")
val numRatersPair = row.getAs[Long]("numRatersPair")
val cooc = cooccurrence(size, numRaters, numRatersPair)
val corr = correlation(size, dotProduct, ratingSum, ratingPairSum, ratingPowSum, ratingPairPowSum)
val regCorr = regularCorrelation(size, dotProduct, ratingSum, ratingPairSum, ratingPowSum, ratingPairPowSum, PRIOR_COUNT, PRIOR_CORRELATION)
val cos = cosineSimilarity(dotProduct, math.sqrt(ratingPowSum), math.sqrt(ratingPairPowSum))
val impCos = improvedCosineSimilarity(dotProduct, math.sqrt(ratingPowSum), math.sqrt(ratingPairPowSum), size, numRaters, numRatersPair)
val jac = jaccardSimilarity(size, numRaters, numRatersPair)
val score = coef(0)*cooc + coef(1)*corr + coef(2)*regCorr - coef(3)*cos - coef(4)*impCos + coef(5)*jac
(movieId, movieIdPair, cooc, corr, regCorr, cos, impCos, jac, score)
}).toDF("movieId", "movieIdPair", "cooc", "corr", "regCorr", "cos", "impCos", "jac", "score")
similar.show(10, false)
// 半角矩阵反转，计算所有item pairs相似度度量
val similarities = similar.withColumnRenamed("movieId", "movieIdRe")
.withColumnRenamed("movieIdPair", "movieId")
.withColumnRenamed("movieIdRe","movieIdPair")
.union(similar)
.repartition(defaultParallelism)
similarities.show(10, false)
val ItemPairSim = similarities.groupBy("movieId","movieIdPair").agg(
functions.sum("cooc").alias("coocSim"),
functions.sum("corr").alias("corrSim"),
functions.sum("regCorr").alias("regCorrSim"),
functions.sum("cos").alias("cosSim"),
functions.sum("impCos").alias("impCosSim"),
functions.sum("jac").alias("jacSim"),
functions.sum("score").alias("scores")
)
val simCols = Array("coocSim","corrSim","regCorrSim","cosSim","impCosSim","jacSim","scores")
simCols.map(simCol =>{
simCol match{
case "coocSim" => println("共现相似度:")
case "corrSim" => println("皮尔逊相关系数:")
case "regCorrSim" => println("改进皮尔逊相关系数:")
case "cosSim" => println("余弦相似度:")
case "impCosSim" => println("改进的余弦相似度:")
case "jacSim" => println("Jaccard相似度:")
case _ => println("加权相似度:")
}
val itemPairsCol = if(simCol.equals("cosSim")||simCol.equals("impCosSim")){
ItemPairSim.select( $"movieId", $"movieIdPair", functions.row_number().over(Window.partitionBy("movieId").orderBy(simCol)).alias("rank"))
.filter($"rank" <= 10)
}else{
ItemPairSim.select( $"movieId", $"movieIdPair", functions.row_number().over(Window.partitionBy("movieId").orderBy(functions.desc(simCol))).alias("rank"))
.filter($"rank" <= 10)
}
println(itemPairsCol.select("movieId", "movieIdPair").where($"movieId" === 15).collectAsList())
})
ItemPairSim.where($"movieId" === 15).show(10,false)4.3.2 计算用户对物品预测评分
在3 预测用户评分中，选取用户对所有物品评分最高的top10作为最终的推荐结果。
// 用户评分与item pairs连接
val userRating = rating.join(similar, "movieId")
.selectExpr("userId", "movieId", "movieIdPair", "cooc", "cooc * rating as coocMeasure",
"corr", "corr * rating as corrMeasure", "regCorr", "regCorr * rating as regCorrMeasure",
"cos", "cos * rating as cosMeasure", "impCos", "impCos * rating as impCosMeasure", "jac", "jac * rating as jacMeasure", "score", "score * rating as scoreMeasure")
.coalesce(defaultParallelism)
userRating.show(10, false)
// 用户对所有物品评分预测
val userScore = userRating.groupBy("userId", "movieIdPair")
.agg(functions.sum("cooc").alias("coocSum"),
functions.sum("coocMeasure").alias("coocMeasureSum"),
functions.sum("corr").alias("corrSum"),
functions.sum("corrMeasure").alias("corrMeasureSum"),
functions.sum("regCorr").alias("regCorrSum"),
functions.sum("regCorrMeasure").alias("regCorrMeasureSum"),
functions.sum("cos").alias("cosSum"),
functions.sum("cosMeasure").alias("cosMeasureSum"),
functions.sum("impCos").alias("impCosSum"),
functions.sum("impCosMeasure").alias("impCosMeasureSum"),
functions.sum("jac").alias("jacSum"),
functions.sum("jacMeasure").alias("jacMeasureSum"),
functions.sum("score").alias("score"),
functions.sum("scoreMeasure").alias("scoreMeasure")
)
.selectExpr("userId", "movieIdPair", "coocSum/coocMeasureSum as coocScore",
"corrSum/corrMeasureSum as corrScore", "regCorrSum/regCorrMeasureSum as regCorrScore",
"cosSum/cosMeasureSum as cosScore", "impCosSum/impCosMeasureSum as impCosScore",
"jacSum/jacMeasureSum as jacScore","score/scoreMeasure as scores")
.coalesce(defaultParallelism)
// 选取每个用户评分最高的10个商品
val userRanks = userScore
.select($"userId", $"movieIdPair", $"scores", functions.row_number().over(Window.partitionBy("userId").orderBy(functions.desc("scores"))).alias("rank"))
.filter($"rank" <= RANKS)
val userRecommend = userRanks.select($"userId", functions.concat_ws(":", $"movieIdPair", $"scores").alias("recommend"))
.groupBy("userId")
.agg(functions.collect_set("recommend"))
userRecommend.show(10, false)4.4 运行结果
4.4.1 计算物品相似度矩阵
+------+-------+------+----------+
|userId|movieId|rating|timestamp |
+------+-------+------+----------+
|0 |2 |3.0 |1424380312|
|0 |3 |1.0 |1424380312|
|0 |5 |2.0 |1424380312|
|0 |9 |4.0 |1424380312|
|0 |11 |1.0 |1424380312|
|0 |12 |2.0 |1424380312|
|0 |15 |1.0 |1424380312|
|0 |17 |1.0 |1424380312|
|0 |19 |1.0 |1424380312|
|0 |21 |1.0 |1424380312|
+------+-------+------+----------+
only showing top 10 rows+------+-------+------+----+
|userId|movieId|rating|rank|
+------+-------+------+----+
|28 |1 |1.0 |1 |
|28 |3 |1.0 |2 |
|28 |6 |1.0 |3 |
|28 |7 |1.0 |4 |
|28 |14 |1.0 |5 |
|28 |15 |1.0 |6 |
|28 |17 |1.0 |7 |
|28 |20 |1.0 |8 |
|28 |27 |1.0 |9 |
|28 |29 |1.0 |10 |
+------+-------+------+----+
only showing top 10 rows+-------+---------+
|movieId|numRaters|
+-------+---------+
|31 |15 |
|85 |18 |
|65 |11 |
|53 |12 |
|78 |14 |
|34 |11 |
|81 |16 |
|28 |12 |
|76 |11 |
|26 |14 |
+-------+---------+
only showing top 10 rows+-------+------+------+----------+---------+
|movieId|userId|rating|timestamp |numRaters|
+-------+------+------+----------+---------+
|2 |0 |3.0 |1424380312|19 |
|3 |0 |1.0 |1424380312|13 |
|5 |0 |2.0 |1424380312|13 |
|9 |0 |4.0 |1424380312|16 |
|11 |0 |1.0 |1424380312|12 |
|12 |0 |2.0 |1424380312|17 |
|15 |0 |1.0 |1424380312|19 |
|17 |0 |1.0 |1424380312|13 |
|19 |0 |1.0 |1424380312|17 |
|21 |0 |1.0 |1424380312|17 |
+-------+------+------+----------+---------+
only showing top 10 rows+------+-------+------+---------+-----------+----------+-------------+-------+---------+-------------+
|userId|movieId|rating|numRaters|movieIdPair|ratingPair|numRatersPair|product|ratingPow|ratingPairPow|
+------+-------+------+---------+-----------+----------+-------------+-------+---------+-------------+
|28 |0 |3.0 |16 |1 |1.0 |13 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |2 |4.0 |19 |12.0 |9.0 |16.0 |
|28 |0 |3.0 |16 |3 |1.0 |13 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |6 |1.0 |20 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |7 |1.0 |16 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |12 |5.0 |17 |15.0 |9.0 |25.0 |
|28 |0 |3.0 |16 |13 |2.0 |16 |6.0 |9.0 |4.0 |
|28 |0 |3.0 |16 |14 |1.0 |18 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |15 |1.0 |19 |3.0 |9.0 |1.0 |
|28 |0 |3.0 |16 |17 |1.0 |13 |3.0 |9.0 |1.0 |
+------+-------+------+---------+-----------+----------+-------------+-------+---------+-------------+
only showing top 10 rows+-------+-----------+----+----------+---------+-------------+------------+----------------+---------+-------------+
|movieId|movieIdPair|size|dotProduct|ratingSum|ratingPairSum|ratingPowSum|ratingPairPowSum|numRaters|numRatersPair|
+-------+-----------+----+----------+---------+-------------+------------+----------------+---------+-------------+
|3 |57 |4 |8.0 |4.0 |8.0 |4.0 |20.0 |13 |12 |
|3 |89 |3 |8.0 |3.0 |8.0 |3.0 |24.0 |13 |11 |
|27 |65 |5 |33.0 |11.0 |11.0 |37.0 |35.0 |15 |11 |
|36 |83 |8 |18.0 |14.0 |12.0 |30.0 |32.0 |18 |14 |
|52 |58 |8 |32.0 |15.0 |12.0 |43.0 |26.0 |14 |15 |
|58 |81 |10 |23.0 |15.0 |18.0 |33.0 |48.0 |15 |16 |
|63 |81 |9 |24.0 |15.0 |16.0 |31.0 |44.0 |16 |16 |
|7 |55 |8 |14.0 |13.0 |9.0 |35.0 |11.0 |16 |19 |
|18 |68 |10 |56.0 |26.0 |19.0 |82.0 |51.0 |15 |19 |
|18 |95 |8 |32.0 |19.0 |15.0 |57.0 |35.0 |15 |17 |
+-------+-----------+----+----------+---------+-------------+------------+----------------+---------+-------------+
only showing top 10 rows+-------+-----------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
|movieId|movieIdPair|cooc |corr |regCorr |cos |impCos |jac |score |
+-------+-----------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
|3 |57 |0.32025630761017426|0.0 |0.0 |0.8944271909999159|0.20500872816969473|0.19047619047619047|-0.1702671877946361 |
|3 |89 |0.2508726030021272 |0.0 |0.0 |0.9428090415820635|0.1645501000890719 |0.14285714285714285|-0.18426814947149298|
|27 |65 |0.3892494720807615 |0.7484551991837488 |0.24948506639458293 |0.9170205237216019|0.23118215648843166|0.23809523809523808|-0.06642073030589295|
|36 |83 |0.5039526306789696 |-0.3418817293789138 |-0.15194743527951723|0.5809475019311126|0.18707200893898498|0.3333333333333333 |-0.10463208979919748|
|52 |58 |0.5520524474738834 |0.8708635721768008 |0.38705047652302255 |0.9570377672873267|0.3912032853853935 |0.38095238095238093|-0.05158141326523652|
|58 |81 |0.6454972243679028 |-0.3125381539589969 |-0.15626907697949846|0.577896743774047 |0.2722768569470682 |0.47619047619047616|-0.08435531125789389|
|63 |81 |0.5625 |-0.27602622373694163|-0.1307492638753934 |0.6498364332886588|0.25833206982242696|0.391304347826087 |-0.11363358680047596|
|7 |55 |0.4588314677411235 |-0.17937400083354382|-0.07972177814824169|0.7135060680126758|0.2439507128147666 |0.2962962962962963 |-0.1366535993117721 |
|18 |68 |0.5923488777590923 |0.45057755628547236 |0.22528877814273618 |0.8659563730239938|0.3947654807460637 |0.4166666666666667 |-0.08146606427655445|
|18 |95 |0.5009794328681196 |-0.40119438904232335|-0.1783086173521437 |0.7164378605434321|0.26694834804228634|0.3333333333333333 |-0.1645577672073404 |
+-------+-----------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
only showing top 10 rows+-----------+-------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
|movieIdPair|movieId|cooc |corr |regCorr |cos |impCos |jac |score |
+-----------+-------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
|25 |51 |0.5976143046671968 |-0.45014069095231984|-0.22507034547615992|0.6819309069874763|0.32975864603850563|0.4166666666666667 |-0.1597401150518419 |
|47 |87 |0.3450327796711771 |0.0 |0.0 |0.9233805168766386|0.2359033677994821 |0.20833333333333334|-0.17927716908138797|
|7 |96 |0.5809475019311126 |-0.48176685558046484|-0.22820535264337807|0.5507685172519937|0.22161701434423176|0.4090909090909091 |-0.1077230965647595 |
|56 |76 |0.5118906968889915 |-0.6454972243679029 |-0.26579297473972474|0.7105597124064275|0.22128206127090347|0.3333333333333333 |-0.1817698444177535 |
|32 |43 |0.4472135954999579 |0.090075469822209 |0.03377830118332838 |0.8549090976340066|0.30577460131716266|0.2857142857142857 |-0.14846460612854348|
|10 |54 |0.3757345746510897 |0.0 |0.0 |0.9525793444156805|0.2662018889308347 |0.23076923076923078|-0.18664913194343136|
|12 |75 |0.3500700210070024 |-0.5103103630798287 |-0.1701034543599429 |0.5011148285857957|0.10979158531474496|0.20833333333333334|-0.12452815428819287|
|9 |66 |0.375 |0.0 |0.0 |0.8725028717782317|0.23123303162285763|0.23076923076923078|-0.1602166376886575 |
|32 |34 |0.26111648393354675|0.0 |0.0 |0.8164965809277261|0.15437994744510897|0.15 |-0.15350165202572327|
|86 |92 |0.44095855184409843|0.16666666666666669 |0.06862745098039216 |0.903696114115064 |0.2546257899447296 |0.28 |-0.13350169285731595|
+-----------+-------+-------------------+--------------------+--------------------+------------------+-------------------+-------------------+--------------------+
only showing top 10 rows共现相似度:
[[15,5], [15,7], [15,4], [15,6], [15,14], [15,9], [15,12], [15,10], [15,2], [15,1]]
皮尔逊相关系数:
[[15,8], [15,12], [15,7], [15,4], [15,11], [15,3], [15,2], [15,1], [15,0], [15,9]]
改进皮尔逊相关系数:
[[15,12], [15,7], [15,4], [15,8], [15,11], [15,3], [15,2], [15,1], [15,0], [15,9]]
余弦相似度:
[[15,9], [15,10], [15,1], [15,13], [15,6], [15,2], [15,0], [15,11], [15,7], [15,12]]
改进的余弦相似度:
[[15,0], [15,13], [15,2], [15,11], [15,10], [15,9], [15,1], [15,6], [15,12], [15,3]]
Jaccard相似度:
[[15,5], [15,7], [15,6], [15,4], [15,14], [15,9], [15,12], [15,10], [15,2], [15,1]]
加权相似度:
[[15,7], [15,12], [15,4], [15,9], [15,2], [15,1], [15,8], [15,6], [15,14], [15,11]]
+-------+-----------+------------------+--------------------+--------------------+------------------+------------------+-------------------+--------------------+
|movieId|movieIdPair|coocSim |corrSim |regCorrSim |cosSim |impCosSim |jacSim |scores |
+-------+-----------+------------------+--------------------+--------------------+------------------+------------------+-------------------+--------------------+
|15 |14 |1.2977713690461004|-0.4633323746250466 |-0.25272674979547993|1.6979054399120357|0.7740279743049593|0.96 |-0.32161825581133757|
|15 |12 |1.224112744964246 |0.663663648395968 |0.34763333963598325 |1.6799278063066676|0.7433079855995841|0.88 |-0.159436983641589 |
|15 |1 |1.1453125733564 |-0.25 |-0.11842105263157894|1.5320646925708532|0.725288309546159 |0.782608695652174 |-0.28978854017510147|
|15 |5 |1.399826478546711 |-0.43852900965351466|-0.22970567172326958|1.7172593257387583|0.993618416740892 |1.0476190476190477 |-0.35885440092921705|
|15 |6 |1.3337718577107005|-0.5564866749122019 |-0.3145359466895054 |1.6412198797244364|0.7294819353921144|1.0 |-0.3008136329516223 |
|15 |2 |1.1578947368421053|-0.1437770309379179 |-0.07531177811033796|1.6470588235294117|0.6520525690591884|0.8148148148148148 |-0.26818397968129093|
|15 |8 |0.8671099695241199|0.8164965809277261 |0.2721655269759087 |1.6853174301284732|0.8231672678073898|0.47619047619047616|-0.2931983633870409 |
|15 |11 |0.9271726499455306|0.0 |0.0 |1.6630436812405998|0.6633685326776863|0.5833333333333334 |-0.32223536439020606|
|15 |10 |1.1846977555181846|-0.6965260331469925 |-0.34826301657349623|1.5212174611483278|0.6934808277609235|0.8333333333333334 |-0.33163020331150633|
|15 |7 |1.3764944032233704|0.6112274566280462 |0.3333967945243888 |1.6711454971746993|0.8570574663543986|1.0434782608695652 |-0.15053882172976585|
+-------+-----------+------------------+--------------------+--------------------+------------------+------------------+-------------------+--------------------+
only showing top 10 rows
4.4.2 计算用户对物品预测评分
+------+-------+-----------+-------------------+-------------------+----+-----------+-------+--------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
|userId|movieId|movieIdPair|cooc |coocMeasure |corr|corrMeasure|regCorr|regCorrMeasure|cos |cosMeasure |impCos |impCosMeasure |jac |jacMeasure |score |scoreMeasure |
+------+-------+-----------+-------------------+-------------------+----+-----------+-------+--------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
|29 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|28 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|26 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|22 |3 |57 |0.32025630761017426|0.6405126152203485 |0.0 |0.0 |0.0 |0.0 |0.8944271909999159|1.7888543819998317|0.20500872816969473|0.41001745633938946|0.19047619047619047|0.38095238095238093|-0.1702671877946361|-0.3405343755892722|
|21 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|17 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|14 |3 |57 |0.32025630761017426|0.9607689228305227 |0.0 |0.0 |0.0 |0.0 |0.8944271909999159|2.6832815729997477|0.20500872816969473|0.6150261845090842 |0.19047619047619047|0.5714285714285714 |-0.1702671877946361|-0.5108015633839083|
|13 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|9 |3 |57 |0.32025630761017426|0.32025630761017426|0.0 |0.0 |0.0 |0.0 |0.8944271909999159|0.8944271909999159|0.20500872816969473|0.20500872816969473|0.19047619047619047|0.19047619047619047|-0.1702671877946361|-0.1702671877946361|
|8 |3 |57 |0.32025630761017426|0.6405126152203485 |0.0 |0.0 |0.0 |0.0 |0.8944271909999159|1.7888543819998317|0.20500872816969473|0.41001745633938946|0.19047619047619047|0.38095238095238093|-0.1702671877946361|-0.3405343755892722|
+------+-------+-----------+-------------------+-------------------+----+-----------+-------+--------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
only showing top 10 rows+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|userId|collect_set(recommend) |
+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|28 |[57:0.6823095186850562, 81:0.7074190948255319, 82:0.672283640715388, 12:0.636297470883653, 92:0.647256075399604, 38:0.6347904579454706, 49:0.6254644038108939, 89:0.6941355224210544, 80:0.6154978795267612, 40:0.6156027460005857]|
|26 |[3:1.0, 73:0.5968684282344038, 6:0.6647065655763444, 18:0.5799619983007535, 1:1.0, 2:1.0, 21:0.5799845694216877, 5:0.8498976850090274, 7:0.5942352082433142, 4:1.0] |
|27 |[9:1.0, 18:1.0, 10:1.0, 3:1.0, 13:1.0, 11:1.0, 8:1.0, 6:1.0, 2:1.0, 4:1.0] |
|12 |[7:1.0, 14:0.8059733660850584, 5:1.0, 3:1.0, 10:0.8293788719613415, 15:0.8349960681296311, 16:0.7700888581918046, 6:1.0, 4:1.0, 13:0.8092037847999061] |
|22 |[21:0.7304053132054968, 14:0.7261359695697873, 22:0.7789954217249625, 3:1.0, 15:0.7439260230313952, 24:0.7289206702134335, 18:0.7695442506930248, 2:1.0, 1:1.0, 17:0.7492600967934236] |
|1 |[56:0.7346282753379684, 21:0.761510986584304, 68:0.7437219860490558, 20:0.7619575246355602, 77:0.7191257119486413, 4:0.7228603942815985, 19:0.7429016915005017, 28:0.7400829561035526, 86:0.717171677819262, 62:0.7218697862031331]|
|13 |[14:0.8650749933539622, 7:0.8879325873718314, 3:1.0, 8:0.904542655291971, 5:0.903713357971985, 12:0.893690744991781, 2:1.0, 1:1.0, 11:0.9145180452679533, 4:1.0] |
|6 |[40:0.8284063212375322, 43:0.8278553890046102, 12:0.8405217298206186, 25:0.8331053495744748, 14:0.9206849112675594, 42:0.856004729785749, 61:0.8413345199040396, 1:1.0, 2:1.0, 58:0.8419910885809283] |
|16 |[28:0.7777501297286809, 5:1.0, 3:1.0, 22:0.7936015217150886, 51:0.767085271204479, 45:0.7314911580677125, 18:0.767613647131584, 21:0.8171576034323195, 24:0.8137831859252895, 4:1.0] |
|3 |[22:0.7082578422647425, 7:1.0, 5:1.0, 3:1.0, 8:0.7233338418664426, 6:1.0, 1:1.0, 2:1.0, 4:1.0, 18:0.7594820850748755] |
+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
only showing top 10 rows二、基于ALS的协同过滤算法
1.基本思想
通过观察所有用户给产品的打分，来推断每个用户的喜好并向用户推荐合适产品。不像基于用户或者物品的协同过滤算法，通过计算相似度来对评分预测和推荐，而是通过矩阵分解方法来进行预测。
2.交替最小二乘求解ALS
用户评分矩阵的每一行代表一个用户
，每一列代表一个物品
，矩阵中的每一个元素代表用户对物品的评分。ALS的核心假设为：打分矩阵A是近似低秩的，即一个m * n的打分矩阵A可以用两个小矩阵
和
乘积来近似：
打分矩阵
就可以由用户喜好特征矩阵
和产品特征矩阵
表示。
为了找到使低秩矩阵U和V尽可能逼近A，需要最小化平方误差损失函数：
损失函数一般需要加入正则化项来避免过拟合问题，使用L2正则化，则改造为：
把协同过滤问题转化为优化问题，求解采用交替最小二乘(ALS)。
3. 代码实现
环境和依赖同上。
3.1 建立ALS协同过滤模型及预测
val als = new ALS()
.setMaxIter(5)
.setRegParam(0.01)
.setUserCol("userId")
.setItemCol("movieId")
.setRatingCol("rating")
val model = als.fit(training)
model.setColdStartStrategy("drop")
val predictions = model.transform(test)3.2 模型评估
val evaluator = new RegressionEvaluator()
.setMetricName("rmse")
.setLabelCol("rating")
.setPredictionCol("prediction")
val rmse = evaluator.evaluate(predictions)
println(s"Root-mean-square error = $rmse")3.3 推荐列表
// 为每个用户生成前10个电影推荐
val userRecs = model.recommendForAllUsers(10)
userRecs.show(10, false)
// 为每部电影生成前10个用户推荐
val movieRecs = model.recommendForAllItems(10)
movieRecs.show(10,false)
// 为指定的一组用户生成前10个电影推荐
val users = ratings.select(als.getUserCol).distinct().limit(3)
val userSubsetRecs = model.recommendForUserSubset(users, 10)
userSubsetRecs.show(10, false)
// 为指定的一组电影生成前10个用户推荐
val movies = ratings.select(als.getItemCol).distinct().limit(3)
val movieSubSetRecs = model.recommendForItemSubset(movies, 10)
movieSubSetRecs.show(5, false)
}
case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
def parseRating(str: String): Rating = {
val fields = str.split("::")
assert(fields.size == 4)
Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong)4 运行结果
Root-mean-square error = 1.927040678568387
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|userId|recommendations |
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|28 |[[92, 5.190725], [81, 4.86317], [4, 4.7198944], [69, 4.33059], [29, 4.2829266], [89, 4.2616525], [76, 4.1987777], [96, 4.184493], [7, 4.1226907], [2, 4.0815644]] |
|26 |[[51, 6.058535], [30, 5.7161875], [94, 5.0318503], [88, 4.9334908], [7, 4.8731565], [24, 4.6401963], [55, 4.472246], [53, 4.236663], [77, 4.1913238], [68, 4.061715]] |
|27 |[[38, 4.2577467], [46, 4.063789], [30, 3.945347], [18, 3.7809763], [23, 3.7044308], [17, 3.3986986], [69, 3.2548237], [27, 3.2114499], [1, 3.1994212], [83, 3.1642542]] |
|12 |[[25, 5.5946026], [46, 5.5124335], [17, 5.11423], [35, 5.0823307], [64, 5.0337462], [27, 4.959539], [43, 4.428263], [1, 4.240394], [94, 3.9515357], [31, 3.9412215]] |
|22 |[[53, 5.5407586], [75, 5.1068535], [46, 5.0826797], [22, 5.047842], [74, 4.9705014], [52, 4.8736644], [88, 4.8529797], [87, 4.850308], [30, 4.643672], [51, 4.5106797]] |
|1 |[[62, 3.5494595], [10, 3.4318256], [68, 3.4186132], [49, 3.3213418], [92, 3.2935987], [85, 3.0727763], [77, 2.9407728], [9, 2.8703325], [39, 2.7454283], [55, 2.435999]]|
|13 |[[32, 4.720694], [69, 4.131916], [93, 3.9207232], [96, 3.6651685], [62, 3.439321], [4, 3.4145846], [74, 3.3329194], [53, 3.194853], [30, 2.9827442], [92, 2.9585018]] |
|6 |[[25, 4.8636093], [58, 3.809049], [62, 3.5082598], [43, 3.4347136], [40, 3.1993797], [37, 3.172602], [92, 3.0829635], [64, 3.0454423], [52, 2.998352], [95, 2.9680624]] |
|16 |[[90, 5.2693963], [85, 4.881312], [54, 4.7646246], [51, 4.623986], [1, 4.4009867], [33, 3.4036496], [68, 3.381634], [94, 3.1365643], [47, 3.060517], [10, 2.9913652]] |
|3 |[[51, 5.0008397], [88, 3.9850216], [24, 3.3814678], [57, 3.186227], [97, 3.106503], [94, 3.0213935], [74, 3.0056891], [76, 2.9751751], [29, 2.9741306], [87, 2.9737644]]|
+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
only showing top 10 rows+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|movieId|recommendations |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|31 |[[12, 3.9412215], [8, 3.2569442], [7, 2.8316283], [6, 2.5838141], [15, 2.1348748], [22, 2.0737545], [21, 2.0215204], [25, 1.8854598], [29, 1.4713752], [14, 1.3172331]]|
|85 |[[16, 4.881312], [17, 4.2706757], [14, 4.2134066], [7, 3.6982865], [1, 3.0727763], [15, 2.7946656], [19, 2.5757656], [6, 2.2734714], [20, 2.2645016], [3, 2.1871538]] |
|65 |[[23, 4.6813827], [20, 4.5017395], [25, 3.6871593], [14, 3.652615], [22, 3.0776486], [7, 2.9361606], [6, 2.4191651], [5, 2.20444], [0, 2.1689785], [3, 2.026729]] |
|53 |[[22, 5.5407586], [21, 4.9554963], [8, 4.9435635], [24, 4.7219305], [26, 4.236663], [13, 3.194853], [5, 2.8868222], [20, 2.8620806], [27, 2.6194685], [28, 2.595715]] |
|78 |[[5, 1.3975992], [23, 1.3725746], [25, 1.2985932], [18, 1.2398205], [6, 1.1964377], [29, 1.1187494], [7, 1.1098578], [2, 1.0951111], [24, 1.0589423], [13, 1.04663]] |
|34 |[[14, 4.7571654], [23, 4.189182], [2, 4.0316715], [18, 3.776127], [28, 3.2931669], [25, 2.9498177], [20, 2.885182], [3, 2.7920961], [13, 2.5485854], [0, 2.4903808]] |
|81 |[[28, 4.86317], [11, 4.0060267], [23, 3.2592351], [18, 3.1237376], [14, 2.7114706], [13, 2.612024], [10, 2.527891], [2, 2.2649016], [9, 2.2159412], [24, 2.1401815]] |
|28 |[[12, 2.1206188], [7, 2.0479355], [6, 1.9925008], [15, 1.5918586], [25, 1.5583891], [8, 1.5504173], [14, 1.2316033], [0, 1.2297935], [29, 1.2055482], [5, 1.1394241]] |
|76 |[[28, 4.1987777], [14, 3.4017448], [10, 3.2845893], [3, 2.9751751], [0, 2.9261422], [12, 2.8340385], [18, 2.7923949], [7, 2.764282], [6, 2.289724], [16, 2.189875]] |
|26 |[[12, 3.3199286], [11, 3.3151531], [15, 2.6134777], [29, 2.49589], [0, 2.153885], [25, 2.1172814], [27, 1.9949272], [18, 1.3374854], [20, 1.3024286], [7, 1.1732591]] |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
only showing top 10 rows+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|userId|recommendations |
+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|28 |[[92, 5.190725], [81, 4.86317], [4, 4.7198944], [69, 4.33059], [29, 4.2829266], [89, 4.2616525], [76, 4.1987777], [96, 4.184493], [7, 4.1226907], [2, 4.0815644]] |
|26 |[[51, 6.058535], [30, 5.7161875], [94, 5.0318503], [88, 4.9334908], [7, 4.8731565], [24, 4.6401963], [55, 4.472246], [53, 4.236663], [77, 4.1913238], [68, 4.061715]] |
|27 |[[38, 4.2577467], [46, 4.063789], [30, 3.945347], [18, 3.7809763], [23, 3.7044308], [17, 3.3986986], [69, 3.2548237], [27, 3.2114499], [1, 3.1994212], [83, 3.1642542]]|
+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------++-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|movieId|recommendations |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|31 |[[12, 3.9412215], [8, 3.2569442], [7, 2.8316283], [6, 2.5838141], [15, 2.1348748], [22, 2.0737545], [21, 2.0215204], [25, 1.8854598], [29, 1.4713752], [14, 1.3172331]]|
|85 |[[16, 4.881312], [17, 4.2706757], [14, 4.2134066], [7, 3.6982865], [1, 3.0727763], [15, 2.7946656], [19, 2.5757656], [6, 2.2734714], [20, 2.2645016], [3, 2.1871538]] |
|65 |[[23, 4.6813827], [20, 4.5017395], [25, 3.6871593], [14, 3.652615], [22, 3.0776486], [7, 2.9361606], [6, 2.4191651], [5, 2.20444], [0, 2.1689785], [3, 2.026729]] |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+参考文献
https://glassywing.github.io/2018/04/10/spark-itemcf/
本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。
