python数据挖掘——亲和性分析

1. 亲和性分析定义

亲和性分析指的是根据样本个体之间的相似度来确定它们之间关系的亲疏

2. 亲和性分析的应用

• 向网站用户提供多样性化的服务或投放定向广告
• 为了向用户推荐电影或商品，而卖给他们一些与之相关的商品

3. 亲和性分析实例

我们通过计算购买不同商品之间的相关性来分析商品之间的亲和性，比如说某个用户“在购买苹果之后，再购买来香蕉”，那么对于该用户来说，“苹果”和“香蕉”这两个样本具有一定的亲和性。当然这个只是一种简单的假设，并没有大数据的统计和分析。

• 定义规则：“如果顾客购买来商品X，那么他们可能购买商品Y”
• 支持度：指数据集中规则应验的次数，有时候可能需要对支持度进行规范化
• 置信度：指的是规则准确率，即符合给定条件的所有规则中，跟当前规则结论一致的比例

4. 构建数据集

我们先假设有5中商品：["bread", "milk", "cheese", "apples", "bananas"]，然后构建一个字典，用来存取两种商品是否存在关联，比如dict[1,3] = 1就表示“顾客在购买milk之后，又购买了apples”

import numpy as np


X = np.zeros((100, 5), dtype='bool')
features = ["bread", "milk", "cheese", "apples", "bananas"]

for i in range(X.shape[0]):
    if np.random.random() < 0.3:
        # A bread winner
        X[i][0] = 1
        if np.random.random() < 0.5:
            # Who likes milk
            X[i][1] = 1
        if np.random.random() < 0.2:
            # Who likes cheese
            X[i][2] = 1
        if np.random.random() < 0.25:
            # Who likes apples
            X[i][3] = 1
        if np.random.random() < 0.5:
            # Who likes bananas
            X[i][4] = 1
    else:
        # Not a bread winner
        X[i][0] = 0
        if np.random.random() < 0.5:
            # Who like milk
            X[i][1] = 1
            if np.random.random() < 0.2:
                # Who likes cheese
                X[i][2] = 1
            if np.random.random() < 0.25:
                # Who likes apples
                X[i][3] = 1
            if np.random.random() < 0.5:
                # Who likes bananas
                X[i][4] = 1
        else:
            if np.random.random() < 0.8:
                # Who likes cheese
                X[i][2] = 1
            if np.random.random() < 0.6:
                # Who likes apples
                X[i][3] = 1
            if np.random.random() < 0.7:
                # Who likes bananas
                X[i][4] = 1
    if X[i].sum() == 0:
        X[i][4] = 1; # Must buy something, so gets bananas

print(X[:5])

np.savetxt("affinity_dataset.txt", X, fmt="%d")

5. 对数据进行计算分析

import numpy as np
dataset_filename = "affinity_dataset.txt"
X = np.loadtxt(dataset_filename)
n_samples, n_features = X.shape
print("This dataset has {0} samples and {1} features".format(n_samples, n_features))

print(X[:5])

features = ["bread", "milk", "cheese", "apples", "bananas"]
# First, how many rows contain our premise: that a person is buying apples
num_apple_purchases = 0
for sample in X:
    if sample[3] == 1:
        num_apple_purchases +=1
print("{0} people bought Apples".format(num_apple_purchases))


rule_valid = 0
rule_invalid = 0
for sample in X:
    if sample[3] == 1:
        if sample[4] == 1:
            rule_valid +=1
        else:
            rule_invalid +=1
print("{0} cases of the rule being valid were discovered".format(rule_valid))
print("{0} cases of the rule being invalide where discovered".format(rule_invalid))

from collections import defaultdict
valid_rules = defaultdict(int)
invalid_rules = defaultdict(int)
num_occurences = defaultdict(int)

for sample in X:
    for premise in range(n_features):
        if sample[premise] == 0: continue
        # Record that the premise was bought in another transaction
        num_occurences[premise] +=1
        for conclusion in range(n_features):
            if premise == conclusion: continue
            if sample[conclusion] == 1:
                valid_rules[(premise, conclusion)] +=1
            else:
                invalid_rules[(premise, conclusion)] +=1

support = valid_rules # 支持度
confidence = defaultdict(float)  # 置信度
for premise,conclusion in valid_rules.keys():
    confidence[(premise, conclusion)] = valid_rules[(premise, conclusion)] / num_occurences[premise]

for premise, conclusion in confidence:
    premise_name = features[premise]
    conclusion_name = features[conclusion]
    print("Rule: If a person buys {0} they will alse buy {1}.".format(premise_name, conclusion_name))
    print(" - Confidence: {0:.3f}".format(confidence[(premise, conclusion)]))
    print(" - Support: {0}".format(support[premise, conclusion]))
    print("")