{ "cells": [ { "cell_type": "markdown", "id": "a07da69b-4328-420a-81f7-8ea7a78748e6", "metadata": {}, "source": [ "

研究生《深度学习》课程
实验报告

\n", "
\n", "
课程名称:深度学习 M502019B
\n", "
实验题目:Pytorch基本操作实验
\n", "
学号:25120323
\n", "
姓名:柯劲帆
\n", "
授课老师:原继东
\n", "
报告日期:2025年7月28日
\n", "
" ] }, { "cell_type": "code", "execution_count": 1, "id": "a4e12268-bad4-44c4-92d5-883624d93e25", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pytorch version: 2.7.1+cu118\n", "CUDA version: 11.8\n", "CUDA device count: 1\n", "CUDA device name: NVIDIA TITAN Xp\n", "CUDA device capability: (6, 1)\n", "CUDA device memory: 11.90 GB\n", "CPU count: 8\n" ] } ], "source": [ "import numpy as np\n", "import torch\n", "from torch.autograd import Variable\n", "from torch.utils.data import Dataset, DataLoader, random_split\n", "from torch import nn\n", "from torchvision import datasets, transforms\n", "from multiprocessing import cpu_count\n", "import matplotlib.pyplot as plt\n", "from tqdm.notebook import tqdm\n", "from typing import Literal, Union\n", "\n", "print('Pytorch version:',torch.__version__)\n", "if not torch.cuda.is_available():\n", " print('CUDA is_available:', torch.cuda.is_available())\n", "else:\n", " print('CUDA version:', torch.version.cuda)\n", " print('CUDA device count:', torch.cuda.device_count())\n", " print('CUDA device name:', torch.cuda.get_device_name())\n", " print('CUDA device capability:', torch.cuda.get_device_capability())\n", " print('CUDA device memory:', f'{torch.cuda.get_device_properties(0).total_memory/1024/1024/1024:.2f}', 'GB')\n", "print('CPU count:', cpu_count())\n", "\n", "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", "seed = 42\n", "np.random.seed(seed)\n", "torch.manual_seed(seed)\n", "torch.cuda.manual_seed(seed)" ] }, { "cell_type": "markdown", "id": "59a43d35-56ac-4ade-995d-1c6fcbcd1262", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# 一、Pytorch基本操作考察\n", "## 题目1\n", "**使用 𝐓𝐞𝐧𝐬𝐨𝐫 初始化一个 𝟏×𝟑 的矩阵 𝑴 和一个 𝟐×𝟏 的矩阵 𝑵，对两矩阵进行减法操作（要求实现三种不同的形式），给出结果并分析三种方式的不同（如果出现报错，分析报错的原因），同时需要指出在计算过程中发生了什么。**" ] }, { "cell_type": "code", "execution_count": 2, "id": "79ea46db-cf49-436c-9b5b-c6562d0da9e2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "M矩阵:\n", "tensor([[1., 2., 3.]], device='cuda:0')\n", "N矩阵:\n", "tensor([[4.],\n", " [5.]], device='cuda:0')\n", "运行结果:\n", "方法1 - 使用PyTorch的减法操作符:\n", "tensor([[-3., -2., -1.],\n", " [-4., -3., -2.]], device='cuda:0')\n", "方法2 - 使用PyTorch的sub函数:\n", "tensor([[-3., -2., -1.],\n", " [-4., -3., -2.]], device='cuda:0')\n", "方法3 - 手动实现广播机制并作差:\n", "tensor([[-3., -2., -1.],\n", " [-4., -3., -2.]], device='cuda:0')\n" ] } ], "source": [ "M = torch.tensor([[1, 2, 3]], dtype=torch.float32, device=device)\n", "N = torch.tensor([[4], [5]], dtype=torch.float32, device=device)\n", "\n", "# 方法1: 使用PyTorch的减法操作符\n", "result1 = M - N\n", "\n", "# 方法2: 使用PyTorch的sub函数\n", "result2 = torch.sub(M, N)\n", "\n", "# 方法3: 手动实现广播机制并作差\n", "def my_sub(a: torch.Tensor, b: torch.Tensor):\n", " if not ((a.size(0) == 1 and b.size(1) == 1) or (a.size(1) == 1 and b.size(0) == 1)):\n", " raise ValueError(\"输入的张量大小无法满足广播机制的条件。\")\n", " else:\n", " target_shape = torch.Size([max(a.size(0), b.size(0)), max(a.size(1), b.size(1))])\n", " a_broadcasted = a.expand(target_shape)\n", " b_broadcasted = b.expand(target_shape)\n", " result = torch.zeros(target_shape, dtype=a_broadcasted.dtype, device=a_broadcasted.device)\n", " for i in range(target_shape[0]):\n", " for j in range(target_shape[1]):\n", " result[i, j] = a_broadcasted[i, j] - b_broadcasted[i, j]\n", " return result\n", "result3 = my_sub(M, N)\n", "\n", "print(f\"M矩阵:\\n{M}\")\n", "print(f\"N矩阵:\\n{N}\")\n", "print(\"运行结果:\")\n", "print(f\"方法1 - 使用PyTorch的减法操作符:\\n{result1}\")\n", "print(f\"方法2 - 使用PyTorch的sub函数:\\n{result2}\")\n", "print(f\"方法3 - 手动实现广播机制并作差:\\n{result3}\")" ] }, { "cell_type": "markdown", "id": "bd9bd5cc-b6da-4dd6-a599-76498bc5247d", "metadata": {}, "source": [ "第1、2、3种减法形式实质是一样的。\n", "\n", "步骤如下：\n", "1. 对A、B两个张量进行广播，将A、B向广播的方向复制，得到两个$\\max(A.size(0), B.size(0))\\times \\max(A.size(1), B.size(1))$的张量；\n", "2. 对广播后的两个张量作差，尺寸不变。\n", "\n", "第1种减法形式和第2种是等价的，前者是后者的符号化表示。\n", "\n", "第3种形式是手动实现的，将上述两个步骤分别手动实现了。但是torch.Tensor还内置了其他机制，这里仅模拟了广播和作差。" ] }, { "cell_type": "markdown", "id": "2489a3ad-f6ff-4561-bb26-e02654090b98", "metadata": {}, "source": [ "## 题目2\n", "1. **利用Tensor创建两个大小分别$3\\times 2$和$4\\times 2$的随机数矩阵$P$和$Q$，要求服从均值为$0$，标准差$0.01$为的正态分布；**\n", "2. **对第二步得到的矩阵$Q$进行形状变换得到$Q$的转置$Q^T$；**\n", "3. **对上述得到的矩阵$P$和矩阵$Q^T$求矩阵相乘。**" ] }, { "cell_type": "code", "execution_count": 3, "id": "41e4ee02-1d05-4101-b3f0-477bac0277fb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "矩阵 P:\n", "tensor([[ 0.0019, 0.0216],\n", " [-0.0017, 0.0085],\n", " [-0.0192, 0.0065]], device='cuda:0')\n", "矩阵 Q:\n", "tensor([[ 1.3914e-03, -1.0822e-03],\n", " [-7.1742e-03, 7.5665e-03],\n", " [ 3.7149e-03, -1.0049e-02],\n", " [ 8.2947e-05, 3.2766e-03]], device='cuda:0')\n", "矩阵 Q^T:\n", "tensor([[ 1.3914e-03, -7.1742e-03, 3.7149e-03, 8.2947e-05],\n", " [-1.0822e-03, 7.5665e-03, -1.0049e-02, 3.2766e-03]], device='cuda:0')\n", "矩阵相乘的结果:\n", "tensor([[-2.0690e-05, 1.4962e-04, -2.1000e-04, 7.0980e-05],\n", " [-1.1582e-05, 7.6587e-05, -9.1717e-05, 2.7677e-05],\n", " [-3.3842e-05, 1.8747e-04, -1.3711e-04, 1.9799e-05]], device='cuda:0')\n" ] } ], "source": [ "P = torch.normal(mean=0, std=0.01, size=(3, 2), device=device)\n", "Q = torch.normal(mean=0, std=0.01, size=(4, 2), device=device)\n", "\n", "print(\"矩阵 P:\")\n", "print(P)\n", "print(\"矩阵 Q:\")\n", "print(Q)\n", "\n", "# 对矩阵Q进行转置操作，得到矩阵Q的转置Q^T\n", "QT = Q.T\n", "print(f\"矩阵 Q^T:\\n{QT}\")\n", "\n", "# 计算矩阵P和矩阵Q^T的矩阵相乘\n", "print(f\"矩阵相乘的结果:\\n{torch.matmul(P, QT)}\")" ] }, { "cell_type": "markdown", "id": "cea9cb6d-adde-4e08-b9f2-8c417abf4231", "metadata": {}, "source": [ "## 题目3\n", "**给定公式$ y_3=y_1+y_2=𝑥^2+𝑥^3$，且$x=1$。利用学习所得到的Tensor的相关知识，求$y_3$对$x$的梯度，即$\\frac{dy_3}{dx}$。**" ] }, { "cell_type": "code", "execution_count": 4, "id": "951512cd-d915-4d04-959f-eb99d1971e2d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "仅通过y_1传递的梯度: 2.0\n", "仅通过y_2传递的梯度: 3.0\n", "dy_3/dx: 5.0\n" ] } ], "source": [ "x = torch.tensor(1.0, requires_grad=True, device=device)\n", "\n", "y_1 = x ** 2\n", "with torch.no_grad():\n", " y_2 = x ** 3\n", "y_3 = y_1 + y_2\n", "y_3.backward()\n", "print(\"仅通过y_1传递的梯度: \", x.grad.item())\n", "\n", "x.grad.data.zero_()\n", "with torch.no_grad():\n", " y_1 = x ** 2\n", "y_2 = x ** 3\n", "y_3 = y_1 + y_2\n", "y_3.backward()\n", "print(\"仅通过y_2传递的梯度: \", x.grad.item())\n", "\n", "x.grad.data.zero_()\n", "y_1 = x ** 2\n", "y_2 = x ** 3\n", "y_3 = y_1 + y_2\n", "y_3.backward()\n", "\n", "print(\"dy_3/dx: \", x.grad.item())" ] }, { "cell_type": "markdown", "id": "3269dbf6-889a-49eb-8094-1e588e1a6c30", "metadata": {}, "source": [ "# 二、动手实现logistic回归\n", "## 题目1\n", "**要求动手从0实现 logistic 回归（只借助Tensor和Numpy相关的库）在人工构造的数据集上进行训练和测试，并从loss以及训练集上的准确率等多个角度对结果进行分析（可借助nn.BCELoss或nn.BCEWithLogitsLoss作为损失函数，从零实现二元交叉熵为选作）**" ] }, { "cell_type": "markdown", "id": "bcd12aa9-f187-4d88-8c59-af6d16107edb", "metadata": {}, "source": [ "给定预测输出$ \\hat{y} $和目标标签$ y$（通常是0或1），BCELoss的计算公式如下：\n", "$$\n", " \\text{BCELoss}(\\hat{y}, y) = -\\frac{1}{N} \\sum_{i=1}^{N} \\left(y_i \\cdot \\log(\\hat{y}_i) + (1 - y_i) \\cdot \\log(1 - \\hat{y}_i)\\right) \n", "$$\n", "其中，$N $是样本数量，$\\hat{y}_i $表示模型的预测概率向量中的第$ i $个元素，$y_i $表示实际的目标标签中的第$ i $个元素。在二分类问题中，$y_i $通常是0或1。这个公式表示对所有样本的二分类交叉熵损失进行了求和并取平均。\n", "\n", "因此BCELoss的手动实现如下。" ] }, { "cell_type": "code", "execution_count": 5, "id": "e31b86ec-4114-48dd-8d73-fe4e0686419a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([0.6900], device='cuda:0')\n", "标签：\n", "tensor([1.], device='cuda:0')\n", "My_BCELoss损失值: 0.37110066413879395\n", "nn.BCELoss损失值: 0.37110066413879395\n" ] } ], "source": [ "class My_BCELoss:\n", " def __call__(self, prediction: torch.Tensor, target: torch.Tensor):\n", " eps = 1e-9\n", " loss = -torch.mean(target * torch.log(prediction + eps) + (1 - target) * torch.log(1 - prediction + eps))\n", " return loss\n", "\n", "\n", "# 测试\n", "prediction = torch.sigmoid(torch.tensor([0.8], device=device))\n", "target = torch.tensor([1.0], device=device)\n", "print(f\"输入：\\n{prediction}\")\n", "print(f\"标签：\\n{target}\")\n", "\n", "my_bce_loss = My_BCELoss()\n", "my_loss = my_bce_loss(prediction, target)\n", "print(\"My_BCELoss损失值:\", my_loss.item())\n", "\n", "nn_bce_loss = nn.BCELoss()\n", "nn_loss = nn_bce_loss(prediction, target)\n", "print(\"nn.BCELoss损失值:\", nn_loss.item())" ] }, { "cell_type": "markdown", "id": "345b0300-8808-4c43-9bf9-05a7e6e1f5af", "metadata": {}, "source": [ "Optimizer的实现较为简单。\n", "\n", "主要实现：\n", "- 传入参数：`__init__()`\n", "- 对传入的参数进行更新：`step()`\n", "- 清空传入参数存储的梯度：`zero_grad()`\n", "\n", "但是有一点需要注意，就是需要将传进来的`params`参数转化为`list`类型。因为`nn.Module`的`parameters()`方法会以``的类型返回模型的参数，但是该类型变量无法像`list`一样使用`for`循环遍历。" ] }, { "cell_type": "code", "execution_count": 6, "id": "0297066c-9fc1-448d-bdcb-29a6f1519117", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "x的初始值: 1.0\n", "学习率: 0.1\n", "y.backward()之后，x的梯度: 2.0\n", "optimizer_test.step()之后，x的值: 0.800000011920929\n", "optimizer_test.zero_grad()之后，x的梯度: 0.0\n" ] } ], "source": [ "class My_Optimizer:\n", " def __init__(self, params: list[torch.Tensor], lr: float):\n", " self.params = list(params)\n", " self.lr = lr\n", "\n", " def step(self):\n", " for param in self.params:\n", " if param.grad is not None:\n", " param.data = param.data - self.lr * param.grad.data\n", "\n", " def zero_grad(self):\n", " for param in self.params:\n", " if param.grad is not None:\n", " param.grad.data.zero_()\n", "\n", "\n", "# 测试\n", "x = torch.tensor(1.0, requires_grad=True, device=device)\n", "print(\"x的初始值: \", x.item())\n", "\n", "optimizer_test = My_Optimizer([x], lr=0.1)\n", "print(\"学习率: \", optimizer_test.lr)\n", "\n", "y = x ** 2\n", "y.backward()\n", "print(\"y.backward()之后，x的梯度: \", x.grad.item())\n", "\n", "optimizer_test.step()\n", "print(\"optimizer_test.step()之后，x的值: \", x.item())\n", "\n", "optimizer_test.zero_grad()\n", "print(\"optimizer_test.zero_grad()之后，x的梯度: \", x.grad.item())" ] }, { "cell_type": "markdown", "id": "8cbc476a-2438-4d0d-854a-4cdd2f726363", "metadata": {}, "source": [ "接下来实现Logistic回归的Trainer，包括训练流程和画图。\n", "\n", "训练进行如下步骤：\n", "1. 定义模型、数据集、损失函数、优化器和其他超参数\n", "2. 训练\n", " 1. 从训练dataloader中获取批量数据\n", " 2. 传入模型\n", " 3. 使用损失函数计算与ground_truth的损失\n", " 4. 使用优化器进行反向传播\n", " 5. 循环以上步骤" ] }, { "cell_type": "code", "execution_count": 7, "id": "d28d5245-bb60-4baf-be54-8c4944ec9180", "metadata": {}, "outputs": [], "source": [ "class LogisticTrainer():\n", " def __init__(\n", " self,\n", " model,\n", " dataset: Union[Dataset, DataLoader],\n", " optimizer: Literal['torch', 'manual'],\n", " criterion: Literal['torch', 'manual'],\n", " learning_rate: float,\n", " num_epochs: int,\n", " batch_size: int,\n", " ):\n", " self.model = model\n", " self.learning_rate = learning_rate\n", " self.num_epochs = num_epochs\n", " self.batch_size = batch_size\n", "\n", " if isinstance(dataset, Dataset):\n", " self.dataloader = DataLoader(\n", " dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=cpu_count()\n", " )\n", " else:\n", " self.dataloader = dataset\n", "\n", " if optimizer == 'torch':\n", " self.optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n", " else:\n", " self.optimizer = My_Optimizer(model.parameters(), lr=learning_rate)\n", "\n", " if criterion == 'torch':\n", " self.criterion = nn.BCELoss()\n", " else:\n", " self.criterion = My_BCELoss()\n", "\n", " def train(self):\n", " loss_curve = []\n", " step = 0\n", " total_train_steps = self.num_epochs * len(self.dataloader)\n", " num_sample_per_epoch = len(self.dataloader) * self.batch_size\n", " with tqdm(total=total_train_steps) as pbar:\n", " for epoch in range(self.num_epochs):\n", " total_epoch_loss = 0\n", " total_epoch_acc = 0\n", " for x, targets in self.dataloader:\n", " x = x.to(device=device, dtype=torch.float32)\n", " targets = targets.to(device=device, dtype=torch.float32)\n", "\n", " self.optimizer.zero_grad()\n", " output = self.model(x)\n", " loss = self.criterion(output, targets)\n", " total_epoch_loss += loss.item()\n", " loss_curve.append(loss.item())\n", " \n", " preds = (output >= 0.5).float()\n", " total_epoch_acc += (preds == targets).float().sum().item()\n", " \n", " loss.backward()\n", " self.optimizer.step()\n", "\n", " step += 1\n", " pbar.update(1)\n", "\n", " log_info = {\n", " 'Epoch': f'{epoch + 1}/{self.num_epochs}',\n", " 'Total Loss': f'{total_epoch_loss:.2f}',\n", " 'Avg Acc': f'{total_epoch_acc / num_sample_per_epoch:.2%}'\n", " }\n", " print(log_info)\n", " \n", " self.plot_results(loss_curve)\n", " \n", " def plot_results(self, loss_curve):\n", " fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n", "\n", " axes[0].plot(loss_curve, label='Training Loss')\n", " axes[0].set_xlabel('Step')\n", " axes[0].set_ylabel('Loss')\n", " axes[0].set_title('Loss Curve')\n", " axes[0].legend()\n", " axes[0].grid(True)\n", "\n", " x, label = next(iter(self.dataloader))\n", " inputs = x.cpu().numpy()\n", " labels = label.cpu().numpy()\n", " x_data = inputs[:, 0]\n", " y_data = inputs[:, 1]\n", " \n", " w = self.model.linear.weight.detach().cpu().numpy()[0]\n", " w_x, w_y = w[0], w[1]\n", " b = self.model.linear.bias.detach().cpu().numpy()[0]\n", " x_vals = np.linspace(-1, 1, 100)\n", " y_model = - (w_x * x_vals + b) / w_y\n", " y_target = 4 - 3 * x_vals\n", " \n", " axes[1].plot(x_vals, y_target, label='Target Line: y=4-3x', linestyle='--', color='green')\n", " axes[1].plot(x_vals, y_model, label='Model Decision Boundary', color='red')\n", "\n", " label_0_shown, label_1_shown = False, False\n", " for i in range(min(100, len(x_data))):\n", " label_val = int(labels[i].item())\n", " if label_val == 1:\n", " color = 'blue'\n", " label_name = 'Label=1' if not label_1_shown else \"\"\n", " label_1_shown = True\n", " else:\n", " color = 'orange'\n", " label_name = 'Label=0' if not label_0_shown else \"\"\n", " label_0_shown = True\n", " axes[1].scatter(x_data[i], y_data[i], color=color, label=label_name)\n", " \n", " axes[1].set_xlabel('x')\n", " axes[1].set_ylabel('y')\n", " axes[1].set_title('Fitted Line vs Target Line')\n", " axes[1].legend()\n", " axes[1].grid(True)\n", " \n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "6ab83528-a88b-4d66-b0c9-b1315cf75c22", "metadata": {}, "source": [ "线性层主要有一个权重（weight）和一个偏置（bias）。\n", "线性层的数学公式如下：\n", "$$\n", "x:=x \\times weight^T+bias\n", "$$\n", "因此代码实现如下：" ] }, { "cell_type": "code", "execution_count": 8, "id": "8e18695a-d8c5-4f77-8b5c-de40d9240fb9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([[1.],\n", " [2.]], device='cuda:0', requires_grad=True)\n", "权重：\n", "tensor([[ 0.8815],\n", " [-0.7336],\n", " [ 0.8692]], device='cuda:0')\n", "偏置：\n", "tensor([0.1872, 0.7388, 0.1354], device='cuda:0')\n", "My_Linear输出：\n", "tensor([[ 1.0687, 0.0052, 1.0046],\n", " [ 1.9502, -0.7284, 1.8738]], device='cuda:0', grad_fn=)\n", "nn.Linear输出：\n", "tensor([[ 1.0687, 0.0052, 1.0046],\n", " [ 1.9502, -0.7284, 1.8738]], device='cuda:0',\n", " grad_fn=)\n" ] } ], "source": [ "class My_Linear:\n", " def __init__(self, input_feature: int, output_feature: int):\n", " self.weight = torch.randn((output_feature, input_feature), requires_grad=True, dtype=torch.float32)\n", " self.bias = torch.zeros(1, requires_grad=True, dtype=torch.float32)\n", " self.params = [self.weight, self.bias]\n", "\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = torch.matmul(x, self.weight.T) + self.bias\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params\n", "\n", " \n", "# 测试\n", "my_linear = My_Linear(1, 3).to(device)\n", "nn_linear = nn.Linear(1, 3).to(device)\n", "my_linear.weight = nn_linear.weight.clone().requires_grad_()\n", "my_linear.bias = nn_linear.bias.clone().requires_grad_()\n", "x = torch.tensor([[1.], [2.]], requires_grad=True, device=device)\n", "print(f\"输入：\\n{x}\")\n", "print(f\"权重：\\n{my_linear.weight.data}\")\n", "print(f\"偏置：\\n{my_linear.bias.data}\")\n", "y_my_linear = my_linear(x)\n", "print(f\"My_Linear输出：\\n{y_my_linear}\")\n", "y_nn_linear = nn_linear(x)\n", "print(f\"nn.Linear输出：\\n{y_nn_linear}\")" ] }, { "cell_type": "markdown", "id": "5ff813cc-c1f0-4c73-a3e8-d6796ef5d366", "metadata": {}, "source": [ "手动实现logistic回归模型。\n", "\n", "模型很简单，主要由一个线性层和一个sigmoid层组成。\n", "\n", "Sigmoid函数（又称为 Logistic函数）是一种常用的激活函数，通常用于神经网络的输出层或隐藏层，其作用是将输入的实数值压缩到一个范围在0和1之间的数值：\n", "\n", "$$\n", "\\sigma(x) = {(1 + e^{-x})}^{-1}\n", "$$\n", "\n", "由于当$x << 0$时，$e^{-x}$较大，进而导致${(1 + e^{-x})}^{-1}$产生数值下溢。因此对Sigmoid函数公式进行优化：\n", "$$\n", "\\sigma(x) = \n", "\\begin{cases}\n", "\\frac{1}{1 + e^{-x}}, & \\text{if } x \\geq 0 \\\\\n", "\\frac{e^{x}}{1 + e^{x}}, & \\text{if } x < 0\n", "\\end{cases}\n", "$$" ] }, { "cell_type": "code", "execution_count": 9, "id": "e7de7e4b-a084-4793-812e-46e8550ecd8d", "metadata": {}, "outputs": [], "source": [ "def my_sigmoid(x: torch.Tensor):\n", " z = torch.exp(-x.abs())\n", " return torch.where(x >= 0, 1 / (1 + z), z / (1 + z))\n", "\n", "\n", "class Model_2_1():\n", " def __init__(self):\n", " self.linear = My_Linear(2, 1)\n", " self.params = self.linear.params\n", "\n", " def __call__(self, x):\n", " return self.forward(x)\n", "\n", " def forward(self, x):\n", " x = self.linear(x)\n", " x = my_sigmoid(x)\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params" ] }, { "cell_type": "markdown", "id": "e14acea9-e5ef-4c24-aea9-329647224ce1", "metadata": {}, "source": [ "人工随机构造数据集。\n", "\n", "我的y设置为$4-3\\times x + noise$，noise为随机噪声。\n", "\n", "生成完x和y后判断给出ground truth，并写好DataLoader访问数据集的接口`__getitem__()`。" ] }, { "cell_type": "code", "execution_count": 10, "id": "c39fbafb-62e4-4b8c-9d65-6718d25f2970", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "测试数据集大小：1000000\n", "测试数据集第0对数据：\n", "x_0 = tensor([-0.2509, 3.0322])\n", "y_0 = tensor([0.])\n" ] } ], "source": [ "class My_Dataset(Dataset):\n", " def __init__(self, data_size=1000000):\n", " x = np.random.uniform(low=-1, high=1, size=(data_size, 1))\n", " noise = np.random.normal(loc=0, scale=1, size=(data_size, 1))\n", " y = 4 - 3 * x + noise\n", " labels = (y > 4 - 3 * x).astype(np.float32)\n", " self.inputs = torch.tensor(np.concatenate([x, y], axis=1), dtype=torch.float32)\n", " self.labels = torch.tensor(labels, dtype=torch.float32)\n", "\n", " def __len__(self):\n", " return self.inputs.shape[0]\n", "\n", " def __getitem__(self, index):\n", " return self.inputs[index], self.labels[index]\n", "\n", "\n", "# 测试，并后面的训练创建变量\n", "dataset = My_Dataset()\n", "dataset_size = len(dataset)\n", "print(f\"测试数据集大小：{dataset_size}\")\n", "x0, y0 = dataset[0]\n", "print(f\"测试数据集第0对数据：\")\n", "print(f\"x_0 = {x0}\")\n", "print(f\"y_0 = {y0}\")" ] }, { "cell_type": "markdown", "id": "957a76a2-b306-47a8-912e-8fbf00cdfd42", "metadata": {}, "source": [ "训练Logistic回归模型。" ] }, { "cell_type": "code", "execution_count": 11, "id": "5612661e-2809-4d46-96c2-33ee9f44116d", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "131f3f0073f247b6901a50dde366d2c2", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4885 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "hyper_params = {\n", " 'learning_rate': 5.0e-2,\n", " 'num_epochs': 5,\n", " 'batch_size': 1024,\n", "}\n", "\n", "model = Model_2_1().to(device)\n", "trainer = LogisticTrainer(model=model, dataset=dataset, optimizer='manual', criterion='manual', **hyper_params)\n", "trainer.train()" ] }, { "cell_type": "markdown", "id": "9e416582-a30d-4084-acc6-6e05f80a6aff", "metadata": {}, "source": [ "## 题目2\n", "**利用 torch.nn 实现 logistic 回归在人工构造的数据集上进行训练和测试，并对结果进行分析，并从loss以及训练集上的准确率等多个角度对结果进行分析**" ] }, { "cell_type": "markdown", "id": "0460d125-7d03-44fe-845c-c4d13792e241", "metadata": {}, "source": [ "使用torch.nn实现模型。\n", "\n", "将之前的Model_2_1中的手动实现函数改为torch.nn内置函数即可，再加上继承nn.Module以使用torch.nn内置模型模板特性。" ] }, { "cell_type": "code", "execution_count": 12, "id": "fa121afd-a1af-4193-9b54-68041e0ed068", "metadata": {}, "outputs": [], "source": [ "class Model_2_2(nn.Module):\n", " def __init__(self):\n", " super(Model_2_2, self).__init__()\n", " self.linear = nn.Linear(2, 1, dtype=torch.float32)\n", "\n", " def forward(self, x):\n", " x = self.linear(x)\n", " x = torch.sigmoid(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "176eee7e-4e3d-470e-8af2-8761bca039f8", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的一致。" ] }, { "cell_type": "code", "execution_count": 13, "id": "93b0fdb6-be8b-4663-b59e-05ed19a9ea09", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9520af64d1cd4867850624e0605f3745", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4885 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "hyper_params = {\n", " 'learning_rate': 5.0e-2,\n", " 'num_epochs': 5,\n", " 'batch_size': 1024,\n", "}\n", "\n", "model = Model_2_2().to(device)\n", "trainer = LogisticTrainer(model=model, dataset=dataset, optimizer='torch', criterion='torch', **hyper_params)\n", "trainer.train()" ] }, { "cell_type": "markdown", "id": "e6bff679-f8d2-46cc-bdcb-82af7dab38b3", "metadata": {}, "source": [ "对比发现，手动实现的损失函数和优化器与torch.nn的内置损失函数和优化器相比，表现差不多。" ] }, { "cell_type": "markdown", "id": "ef41d7fa-c2bf-4024-833b-60af0a87043a", "metadata": {}, "source": [ "# 三、动手实现softmax回归\n", "\n", "## 问题1\n", "\n", "**要求动手从0实现softmax回归（只借助Tensor和Numpy相关的库）在Fashion-MNIST数据集上进行训练和测试，并从loss、训练集以及测试集上的准确率等多个角度对结果进行分析（要求从零实现交叉熵损失函数）**" ] }, { "cell_type": "markdown", "id": "902603a6-bfb9-4ce3-bd0d-b00cebb1d3cb", "metadata": {}, "source": [ "手动实现CrossEntropyLoss。\n", "\n", "CrossEntropyLoss由一个log_softmax和一个nll_loss组成。\n", "\n", "softmax的数学表达式如下：\n", "$$\n", "\\text{softmax}(x_i) = \\frac{e^{x_i}}{\\sum_{j=1}^{N} e^{x_j}} = \\frac{e^{x_i - \\text{max}(x)}}{\\sum_{j=1}^{N} e^{x_j - \\text{max}(x)}} \n", "$$\n", "log_softmax即为$\\log(\\text{softmax}(x))$，但可以进一步优化：\n", "$$\n", "\\text{logsoftmax}(x_i) = \\log{\\frac{e^{x_i - \\text{max}(x)}}{\\sum_{j=1}^{N} e^{x_j - \\text{max}(x)}}} = x_i - \\text{max}(x) - \\log{\\sum_{j=1}^{N} e^{x_j - \\text{max}(x)}}\n", "$$\n", "\n", "CrossEntropyLoss的数学表达式如下：\n", "$$\n", "\\text{CrossEntropyLoss}(x, \\hat{x}) = -\\frac{1}{N} \\sum_{i=1}^{N} \\hat{x}_i \\cdot \\log(\\text{softmax}(x_i)) \n", "$$\n", "\n", "故代码如下：" ] }, { "cell_type": "code", "execution_count": 14, "id": "759a3bb2-b5f4-4ea5-a2d7-15f0c4cdd14b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "输入：\n", "tensor([[-0.1808, -0.6778, -0.5920, -0.6382, -1.9187],\n", " [-0.6441, -0.6061, -0.1425, 0.9727, 2.0038],\n", " [ 0.6622, 0.5332, 2.7489, -0.3841, -1.9623]], requires_grad=True)\n", "标签：\n", "tensor([2, 0, 1])\n", "My_CrossEntropyLoss损失值: 2.377387762069702\n", "nn.CrossEntropyLoss损失值: 2.377387762069702\n" ] } ], "source": [ "class My_Softmax:\n", " def __init__(self, dim: int):\n", " self.dim = dim\n", " def __call__(self, x: torch.Tensor):\n", " max_x = torch.max(x, dim=self.dim, keepdim=True).values\n", " exp_x = torch.exp(x - max_x)\n", " return exp_x / torch.sum(exp_x, dim=self.dim, keepdim=True)\n", "\n", "def my_logsoftmax(x: torch.Tensor):\n", " max_x = torch.max(x, dim=1, keepdim=True).values\n", " exp_x = torch.exp(x - max_x)\n", " return x - max_x - torch.log(torch.sum(exp_x, dim=1, keepdim=True))\n", "\n", "class My_CrossEntropyLoss:\n", " def __call__(\n", " self, \n", " predictions: torch.Tensor, \n", " targets: torch.Tensor, \n", " reduction: Literal[\"mean\", \"sum\"] = \"mean\"\n", " ):\n", " log_probs = my_logsoftmax(predictions)\n", " \n", " if len(predictions.shape) == len(targets.shape) + 1:\n", " nll_loss = -log_probs.gather(1, targets.unsqueeze(-1)).squeeze()\n", " else:\n", " nll_loss = -torch.sum(targets * log_probs, dim=1)\n", " \n", " if reduction == \"mean\": \n", " return torch.mean(nll_loss)\n", " else: \n", " return torch.sum(nll_loss)\n", "\n", " \n", "# 测试\n", "input = torch.randn(3, 5, requires_grad=True)\n", "target = torch.randn(3, 5).softmax(dim=1).argmax(1)\n", "print(f\"输入：\\n{input}\")\n", "print(f\"标签：\\n{target}\")\n", "\n", "my_crossentropyloss = My_CrossEntropyLoss()\n", "my_loss = my_crossentropyloss(input, target)\n", "print(\"My_CrossEntropyLoss损失值:\", my_loss.item())\n", "\n", "nn_crossentropyloss = nn.CrossEntropyLoss()\n", "nn_loss = nn_crossentropyloss(input, target)\n", "print(\"nn.CrossEntropyLoss损失值:\", nn_loss.item())" ] }, { "cell_type": "markdown", "id": "92c224a3-8c27-4392-9017-aa526030a0a6", "metadata": {}, "source": [ "接下来实现Softmax回归的Trainer，包括训练流程、测试和画图。\n", "\n", "训练softmax回归模型，进行如下步骤：\n", "1. 定义模型、数据集、损失函数、优化器和其他超参数\n", "2. 训练\n", " 1. 从训练dataloader中获取批量数据\n", " 2. 传入模型\n", " 3. 使用损失函数计算与ground_truth的损失\n", " 4. 使用优化器进行反向传播\n", " 5. 循环以上步骤\n", "3. 验证及测试\n", " 1. 从验证或测试dataloader中获取批量数据\n", " 2. 传入模型，验证时需要将模型输出与ground_truth进行比较得计算loss\n", " 3. 将预测值与ground_truth进行比较，得出正确率\n", " 4. 对整个训练集统计正确率，从而分析训练效果" ] }, { "cell_type": "code", "execution_count": 15, "id": "159fc93c-fa21-4a94-b460-dda9e8557a43", "metadata": {}, "outputs": [], "source": [ "class SoftmaxTrainer():\n", " def __init__(\n", " self,\n", " model,\n", " train_dataset: Union[Dataset, DataLoader],\n", " eval_dataset: Union[Dataset, DataLoader],\n", " test_dataset: Union[Dataset, DataLoader],\n", " optimizer: Literal['torch', 'manual'],\n", " criterion: Literal['torch', 'manual'],\n", " learning_rate: float,\n", " num_epochs: int,\n", " batch_size: int,\n", " ):\n", " self.model = model\n", " self.learning_rate = learning_rate\n", " self.num_epochs = num_epochs\n", " self.batch_size = batch_size\n", "\n", " if isinstance(train_dataset, Dataset):\n", " self.train_dataloader = DataLoader(\n", " dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=cpu_count()\n", " )\n", " else:\n", " self.train_dataloader = train_dataset\n", " if isinstance(eval_dataset, Dataset):\n", " self.eval_dataloader = DataLoader(\n", " dataset=eval_dataset, batch_size=batch_size, shuffle=True, num_workers=cpu_count()\n", " )\n", " else:\n", " self.eval_dataloader = eval_dataset\n", " if isinstance(test_dataset, Dataset):\n", " self.test_dataloader = DataLoader(\n", " dataset=test_dataset, batch_size=batch_size, shuffle=True, num_workers=cpu_count()\n", " )\n", " else:\n", " self.test_dataloader = test_dataset\n", "\n", " if optimizer == 'torch':\n", " self.optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n", " else:\n", " self.optimizer = My_Optimizer(model.parameters(), lr=learning_rate)\n", "\n", " if criterion == 'torch':\n", " self.criterion = nn.CrossEntropyLoss()\n", " self.softmax = nn.Softmax(dim=1)\n", " else:\n", " self.criterion = My_CrossEntropyLoss()\n", " self.softmax = My_Softmax(dim=1)\n", "\n", " def train(self):\n", " train_loss_curve = []\n", " eval_loss_curve = []\n", " eval_acc_curve = []\n", " step = 0\n", " total_train_steps = self.num_epochs * len(self.train_dataloader)\n", " with tqdm(total=total_train_steps) as pbar:\n", " for epoch in range(self.num_epochs):\n", " total_train_loss = 0\n", " for x, targets in self.train_dataloader:\n", " x = x.to(device=device, dtype=torch.float32)\n", " targets = targets.to(device=device, dtype=torch.long)\n", "\n", " self.optimizer.zero_grad()\n", " output = self.model(x)\n", " loss = self.criterion(output, targets)\n", " total_train_loss += loss.item()\n", " train_loss_curve.append(loss.item())\n", " \n", " loss.backward()\n", " self.optimizer.step()\n", " step += 1\n", " pbar.update(1)\n", "\n", " avg_eval_loss, avg_eval_acc = self.eval()\n", " eval_loss_curve.append(avg_eval_loss)\n", " eval_acc_curve.append(avg_eval_acc)\n", " log_info = {\n", " 'Epoch': f'{epoch + 1}/{self.num_epochs}',\n", " 'Total Train Loss': f'{total_train_loss:.2f}',\n", " 'Scaled Total Valid Loss': f'{avg_eval_loss * len(self.train_dataloader):.2f}',\n", " 'Avg Valid Acc': f'{avg_eval_acc:.2%}'\n", " }\n", " print(log_info)\n", "\n", " print('Avg Test Acc:', f'{self.test():.2%}')\n", " self.plot_results(train_loss_curve, eval_loss_curve, eval_acc_curve)\n", "\n", " def eval(self):\n", " total_eval_loss = 0\n", " total_eval_acc = 0\n", " with torch.inference_mode():\n", " for x, targets in self.eval_dataloader:\n", " x = x.to(device=device, dtype=torch.float32)\n", " targets = targets.to(device=device, dtype=torch.long)\n", " output = self.model(x)\n", " loss = self.criterion(output, targets)\n", " total_eval_loss += loss.item()\n", " preds = self.softmax(output).argmax(dim=1)\n", " total_eval_acc += (preds == targets).float().sum().item()\n", " \n", " avg_eval_loss = total_eval_loss / len(self.eval_dataloader)\n", " num_eval_sample = len(self.eval_dataloader) * self.batch_size\n", " avg_eval_acc = total_eval_acc / num_eval_sample\n", " return avg_eval_loss, avg_eval_acc\n", "\n", " def test(self):\n", " total_test_acc = 0\n", " with torch.inference_mode():\n", " for x, targets in self.test_dataloader:\n", " x = x.to(device=device, dtype=torch.float32)\n", " targets = targets.to(device=device, dtype=torch.long)\n", " output = self.model(x)\n", " preds = self.softmax(output).argmax(dim=1)\n", " total_test_acc += (preds == targets).float().sum().item()\n", " num_test_sample = len(self.test_dataloader) * self.batch_size\n", " avg_test_acc = total_test_acc / num_test_sample\n", " return avg_test_acc\n", " \n", " def plot_results(self, train_loss_curve, eval_loss_curve, eval_acc_curve):\n", " fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n", " \n", " axes[0].plot(train_loss_curve, label='Training Loss', color='blue')\n", " axes[0].plot(\n", " np.linspace(len(self.train_dataloader), len(train_loss_curve), len(eval_loss_curve), endpoint=True),\n", " eval_loss_curve, label='Validation Loss', color='orange'\n", " )\n", " axes[0].set_xlabel('Step')\n", " axes[0].set_ylabel('Loss')\n", " axes[0].set_title('Loss Curve')\n", " axes[0].legend()\n", " axes[0].grid(True)\n", " \n", " axes[1].plot(eval_acc_curve, label='Validation Accuracy', color='green', marker='o')\n", " axes[1].set_xlabel('Epoch')\n", " axes[1].set_ylabel('Accuracy')\n", " axes[1].set_title('Validation Accuracy Curve')\n", " axes[1].legend()\n", " axes[1].grid(True)\n", " \n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "dbf78501-f5be-4008-986c-d331d531491f", "metadata": {}, "source": [ "手动实现Flatten。\n", "\n", "原理很简单，就是把多维的张量拉直成一个向量。" ] }, { "cell_type": "code", "execution_count": 16, "id": "74322629-8325-4823-b80f-f28182d577c1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Flatten之前的x：\n", "tensor([[[1., 2.],\n", " [3., 4.]],\n", "\n", " [[5., 6.],\n", " [7., 8.]]])\n", "My_Flatten之后的x：\n", "tensor([[1., 2., 3., 4.],\n", " [5., 6., 7., 8.]])\n", "nn.Flatten之后的x：\n", "tensor([[1., 2., 3., 4.],\n", " [5., 6., 7., 8.]])\n" ] } ], "source": [ "class My_Flatten:\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = x.view(x.shape[0], -1)\n", " return x\n", "\n", "\n", "# 测试\n", "my_flatten = My_Flatten()\n", "nn_flatten = nn.Flatten()\n", "x = torch.tensor(\n", " [[[1., 2.], [3., 4.]],\n", " [[5., 6.], [7., 8.]]]\n", ")\n", "print(f\"Flatten之前的x：\\n{x}\")\n", "x_my_flatten = my_flatten(x)\n", "print(f\"My_Flatten之后的x：\\n{x_my_flatten}\")\n", "x_nn_flatten = nn_flatten(x)\n", "print(f\"nn.Flatten之后的x：\\n{x_nn_flatten}\")" ] }, { "cell_type": "markdown", "id": "35aee905-ae37-4faa-a7f1-a04cd8579f78", "metadata": {}, "source": [ "手动实现softmax回归模型。\n", "\n", "模型很简单，主要由一个Flatten层和一个线性层组成。\n", "\n", "Flatten层主要用于将2维的图像展开，直接作为1维的特征量输入网络。" ] }, { "cell_type": "code", "execution_count": 17, "id": "bb31a75e-464c-4b94-b927-b219a765e35d", "metadata": {}, "outputs": [], "source": [ "class Model_3_1:\n", " def __init__(self, num_classes):\n", " self.flatten = My_Flatten()\n", " self.linear = My_Linear(28 * 28, num_classes)\n", " self.params = self.linear.params\n", "\n", " def __call__(self, x: torch.Tensor):\n", " return self.forward(x)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = self.flatten(x)\n", " x = self.linear(x)\n", " return x\n", "\n", " def to(self, device: str):\n", " for param in self.params:\n", " param.data = param.data.to(device=device)\n", " return self\n", "\n", " def parameters(self):\n", " return self.params" ] }, { "cell_type": "markdown", "id": "17e686d1-9c9a-4727-8fdc-9990d348c523", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的几乎一致。由于数据集的变化，对应超参数也进行了调整。\n", "\n", "数据集也使用了现成的FashionMNIST数据集，且划分了训练集和测试集。\n", "\n", "FashionMNIST数据集直接调用API获取。数据集的image为28*28的单通道灰白图片，label为单个数值标签。" ] }, { "cell_type": "code", "execution_count": 18, "id": "02f7d7dc-e2a8-4127-b505-f31993a75131", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train Dataset Size: 59000\n", "Valid Dataset Size: 1000\n", "Test Dataset Size: 10000\n", "A Train Sample:\n", "\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIcAAACdCAYAAACeqmv3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjUsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvWftoOwAAAAlwSFlzAAAPYQAAD2EBqD+naQAADiFJREFUeJztnWtsVFX3xp+h1rZURUFbpEJtLWKRJpJWRAJpEaE1ImkToyZeMN4So4kahOgXygejMUo0BhNJvDRGjR8m1XhpMKJFgmKLNlbA1tZykaogpSgoUqjd74d/Zv7dzzl7zkxnOi3v+/wSPqzTc87e57By9jNrr712yBhjIIQPE8a6A2L8IucQTuQcwomcQziRcwgncg7hRM4hnMg5hBM5h3DyX+0c+/btQygUwvPPP5+ye27ZsgWhUAhbtmxJ2T3HK+POORoaGhAKhfDNN9+MdVdGhcbGRtx6660oLi7GxIkTMWvWLKxatQp//PHHWHfNw1lj3YH/NR544AFMmzYNd9xxB2bMmIGdO3diw4YNaGpqQltbG3Jycsa6i1HkHGkmHA6jqqrKOlZeXo6VK1fi7bffxn333Tc2HfNh3A0r8XDq1CmsXbsW5eXlmDRpEnJzc7Fo0SI0Nzc7r3nhhRdQWFiInJwcVFZWYteuXZ5zOjs7cfPNN2Py5MnIzs5GRUUFPvjgg8D+nDhxAp2dnejr6ws8lx0DAOrq6gAAHR0dgdenkzPSOY4dO4ZXX30VVVVVePbZZ7Fu3TocPnwY1dXV+O677zznv/nmm3jppZfw0EMP4cknn8SuXbtw3XXX4dChQ9Fzdu/ejfnz56OjowNPPPEE1q9fj9zcXNTW1uK9996L2Z/W1laUlpZiw4YNI3qegwcPAgAuvPDCEV0/aphxxhtvvGEAmB07djjPGRwcNAMDA9axo0ePmvz8fHPPPfdEj+3du9cAMDk5Oaa3tzd6vKWlxQAwjz32WPTYkiVLTFlZmTl58mT02NDQkFmwYIGZOXNm9Fhzc7MBYJqbmz3H6uvrR/LI5t577zUZGRmmq6trRNePFmfklyMjIwNnn302AGBoaAj9/f0YHBxERUUF2traPOfX1taioKAgas+bNw/XXHMNmpqaAAD9/f34/PPPccstt+D48ePo6+tDX18fjhw5gurqanR3d+OXX35x9qeqqgrGGKxbty7hZ3nnnXfw2muvYdWqVZg5c2bC148qY+2dTDxfDmOMaWhoMGVlZSYzM9MAiP4rKiqKnhP5cqxdu9Zz/Z133mmysrKMMf//JYn1r62tzRjj/+UYKVu3bjXZ2dmmurranD59Oun7pZoz8tfKW2+9hbvvvhu1tbVYvXo18vLykJGRgWeeeQY9PT0J329oaAgA8Pjjj6O6utr3nJKSkqT6zLS3t2PFihWYM2cOwuEwzjpr/P1XjL8exUE4HEZxcTEaGxsRCoWix+vr633P7+7u9hzr6urCpZdeCgAoLi4GAGRmZuL6669PfYeJnp4e1NTUIC8vD01NTTjnnHNGvc2RcMZqDgAww3KjW1pasH37dt/z33//fUsztLa2oqWlBTfccAMAIC8vD1VVVdi4cSN+++03z/WHDx+O2Z9EfsoePHgQy5Ytw4QJE/DJJ5/goosuCrxmrBi3X47XX38dmzZt8hx/5JFHsHz5cjQ2NqKurg433ngj9u7di1deeQWzZ8/GX3/95bmmpKQECxcuxIMPPoiBgQG8+OKLmDJlCtasWRM95+WXX8bChQtRVlaG+++/H8XFxTh06BC2b9+O3t5etLe3O/va2tqKxYsXo76+PlCU1tTUYM+ePVizZg22bduGbdu2Rf+Wn5+PpUuXxvF20sRYix4mIkhd/w4cOGCGhobM008/bQoLC01WVpaZO3eu+eijj8zKlStNYWFh9F4RQfrcc8+Z9evXm+nTp5usrCyzaNEi097e7mm7p6fH3HXXXWbq1KkmMzPTFBQUmOXLl5twOBw9J9mfsrGerbKyMok3l3pCxmjdivDnjNQcIj3IOYQTOYdwIucQTuQcwomcQziRcwgncUdIh89hjBcicyMRli1b5jmHE5X37Nlj2UGJvTwhxm2yDQCnT5+2bE4jOH78eMw200E84S19OYQTOYdwkrKJtwkTbD+L5EgkwlVXXWXZU6dOtWz+HA8MDFj24sWLPfd89NFHLfvbb7+17NraWsvm6fNwOGzZPGvr12ZDQ4NlT5w4MWYbg4ODls1DX6xJv9FEXw7hRM4hnMg5hJO4p+yDfsqORHMUFhZadn9/v2WXlpbG/Pvff/9t2X5ZXLfffrtlz54927IrKiosm5+Tx39+XV988YWnzXfffdey8/PzLZvfFb+HSKZbhC+//NLTBsP9Dvpv1U9ZkRRyDuFEziGcpExzJDrmAcCKFSssm3/P79+/37KnTJli2Tw2n3vuuZ42Tpw4YdnTpk2L2SeOOUyePNmyh6+vBf5v3S6Tl5cXsw1OgmZdM2nSJMueM2eOZfvpnESR5hBJIecQTuQcwomcQzhJmSAdyT3YjixPjMCBNA5y8fJDzqMAvAKT+fPPPy07aN0qt3Hq1CnPORzk4jpf559/vmWzgJ0+fbpl//jjj5btV5UoUSRIRVLIOYQTOYdwklbNwcEcDgZx0IsDUCdPnrTsoqIiy/Yb//kYT3JxwlBWVpZl//PPP5YdpCf8rmHtxBOIrGN+/fVXy87MzLTsf//919MmP0cQ0hwiKeQcwomcQzhJq+bIzs627GuvvdayOebgN7YOh8s/sh4AgKNHj8bsA0/W8WQeJzXzOhfWKEDwu+Lkn4svvtiyWZNcffXVlr1161bPPQ8cOBCzTUaaQySFnEM4kXMIJ6OmOfi3ORCcOFNeXm7Zvb29ls3jf6R+aIRIyevhcJyDNQXDf+c4Bscw4nkv/C4ihfAj8HPNnTvXsjdv3hzYRqJIc4ikkHMIJ3IO4SStcY6bbrrJsnnOgWtpcJyDx3+Oc4xk8TbX3wjK/4iHoGRrrunBsRcup82xlBkzZnja5AXifrktsfrkh74cwomcQziRcwgnadUcDM8p8CKnBQsWWPbq1astm+McfrAOYR3DGoPjHDxfw5v0cdzDrw3WFBy3qKmpsWzWFD/88INlc5wEAH766SfPsVhIc4ikkHMIJ3IO4WTUdmrifFHAu0B49+7dlt3Z2WnZQbkXvG7Fr3gLawqOlbCW4rkYtllPxBNbYZ1y2WWXWTavY/nqq68sm9ep+G0WyO+K821Hgr4cwomcQziRcwgncg7hZNSCYCyQAO+EE+/dfvnll1s2V9H7+uuvLZuFGQteIDhJmeFqw5xAxPfzqybEIpYXKfE9r7jiCstesmSJZbNQ5yqKgFeMB1U9VhBMJIWcQziRcwgnKdMcPP7zjgeAN2GYF/ewZujq6rLsefPmWfYFF1xg2X5jMSfvcuIMB8l4cTfrA9YTfgupWLdw4I0Tc3gR9MMPP2zZH374oWXv27fP02aiSHOIpJBzCCdyDuEkZZqDx/L58+d7zuHYR9COBU899ZRlc3KQ30TbeCQ3N9eyWSuxFrvtttti3q+jo8NzjAvdtbS0WDZrK2kOkRRyDuFEziGcpCzZh3+rx1PZnwuOcBwjCN51yW8hNY/FnATDcyMcx+D4DT9n0MJswDsf8/3338c8n+M9rFF4jgnwT3ROFn05hBM5h3Ai5xBOUqY5OFeDd5cGvON3ZWWlZfM8CMMF41mz+CXecgyB5054boR3duLCK7xA2a9NnlvhBGPWFNzmjh07LJuTtf0WNXE/g+aQ4kFfDuFEziGcyDmEk5RpDs4x4GKugLcoPdtBm8xwfkdQgRLAO5/D4z3/nXMvghYtsV4AvLEPLpQbNP6zDuKcUr/n5jgHx1J4B8p40JdDOJFzCCdyDuEkZZqDC6lwTAIALrnkEsu+8sorLTtoPoaLt/LYHk8hFf79z8XZmKDiLxzDALw6prS01LJZn/G6FP77pk2bLJsXWgPegr9aSC1GFTmHcCLnEE7kHMJJygQpJ8n4BXo4YPTzzz9bNu+8zLAQS3RHRD844MR2ULKP305NLAY5cdpvR8nhsMhdunSpZfMO1n5t8kJqvwShIPTlEE7kHMKJnEM4SZnm4MklTuwFgqsDc7IPj63xVCxmeHxnjcB6gINevDibn8GvSA3rLw7OcVLzp59+atkcFPv4448t229R0++//27ZXGlxJOjLIZzIOYQTOYdwkjLNwRNtXPwN8C6EDloQxOMma5AjR45Ytl8SDMdbEq1QzAuOWaP4PQMnGLPmmDVrlmUHJTFznIMrIAPeatDd3d2WzYlS8aAvh3Ai5xBO5BzCSco0By8w8vudzXMGvKiJ4QXEPPfC471fbIXHfy5kF1SUNiiJOZ75HU625qIzQW189tlnlu2nH1h/9ff3B/YrCH05hBM5h3Ai5xBOUqY5ysrKLJuL3gNAUVGRZfMC4Y0bN1o2JxTzjkY8zvoVb+E8B45jcAyCz+fEXS7+5je3EqRzuA3uAxex5ev9CuMeO3bMsjnOkejukYC+HCIGcg7hRM4hnKR1R+rzzjvPslkjpCIHIQjOCeX8Du4jF3sJyikFvBoizlfspK6uzrI5dwPwaqn9+/dbNsdWVKRWJIWcQziRcwgnadUcQXAOCMdKOO+BiadgLMMxA87/4IXWHOfw2wCQXylfw3MtvJ5n8+bN7g6nCGkOkRRyDuFEziGcyDmEk7QKUr4HN82Ckye1OCDFE1x+gpQDUnxO0I7TQa/Hr01eCBVUkZBFLT9nskE0PyRIRVLIOYQTOYdwkrJkn3gIGucKCgosu6SkJOb5PDb7jf+sKdgOWqTEeoGTgVmjAP4FXYYTtAv2zp07Ldtvp+10oC+HcCLnEE7kHMJJWjVHEEHVhFkPcKINxz0Ar8aIp7DdcIIqAfsl+wbBGiI/Pz+hPqULfTmEEzmHcCLnEE7i1hyjEd8X4xt9OYQTOYdwIucQTuQcwomcQziRcwgncg7hRM4hnMg5hJP/ACZzb00BuFjAAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "{'Image Type': , 'Image Shape': torch.Size([1, 28, 28]), 'Label Type': , 'Label Value': 2}\n" ] } ], "source": [ "transform = transforms.Compose(\n", " [\n", " transforms.ToTensor(),\n", " transforms.Normalize((0.5,), (0.5,)),\n", " ]\n", ")\n", "train_dataset = datasets.FashionMNIST(root=\"./dataset\", train=True, transform=transform, download=True)\n", "eval_size = min(int(len(train_dataset) * 0.1), 1000)\n", "train_dataset, eval_dataset = random_split(train_dataset, [len(train_dataset) - eval_size, eval_size])\n", "test_dataset = datasets.FashionMNIST(root=\"./dataset\", train=False, transform=transform, download=True)\n", "print('Train Dataset Size:', len(train_dataset))\n", "print('Valid Dataset Size:', len(eval_dataset))\n", "print('Test Dataset Size:', len(test_dataset))\n", "\n", "image, label = train_dataset[0]\n", "sample = {\n", " 'Image Type': type(image),\n", " 'Image Shape': image.shape,\n", " 'Label Type': type(label),\n", " 'Label Value': label\n", "}\n", "print('A Train Sample:\\n')\n", "image = image * 0.5 + 0.5 # 将图像从 [-1, 1] 还原到 [0, 1] 以便更好地可视化\n", "plt.figure(figsize=(1.5, 1.5))\n", "plt.imshow(image.squeeze(), cmap='gray')\n", "plt.title(f\"Label: {label}\")\n", "plt.axis('off')\n", "plt.show()\n", "print(sample)\n", "\n", "num_classes = 10" ] }, { "cell_type": "markdown", "id": "24594cbc-18b2-47eb-a526-c2ab37facf63", "metadata": {}, "source": [ "开始训练。" ] }, { "cell_type": "code", "execution_count": 19, "id": "d816dae1-5fbe-4c29-9597-19d66b5eb6b4", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2cd0298a4a254c018e497a76ccfd0246", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/1160 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "hyper_params = {\n", " 'learning_rate': 2.0e-1,\n", " 'num_epochs': 20,\n", " 'batch_size': 1024,\n", "}\n", "\n", "model = Model_3_1(num_classes).to(device)\n", "\n", "trainer = SoftmaxTrainer(\n", " model=model, \n", " train_dataset=train_dataset, eval_dataset=eval_dataset, test_dataset=test_dataset, \n", " optimizer='manual', criterion='manual', **hyper_params\n", ")\n", "trainer.train()" ] }, { "cell_type": "markdown", "id": "a49d0165-aeb7-48c0-9b67-956bb08cb356", "metadata": {}, "source": [ "模型正常收敛。" ] }, { "cell_type": "markdown", "id": "3ef5240f-8a11-4678-bfce-f1cbc7e71b77", "metadata": {}, "source": [ "## 问题2\n", "\n", "**利用torch.nn实现softmax回归在Fashion-MNIST数据集上进行训练和测试，并从loss，训练集以及测试集上的准确率等多个角度对结果进行分析**" ] }, { "cell_type": "markdown", "id": "5c4a88c6-637e-4af5-bed5-f644685dcabc", "metadata": {}, "source": [ "使用torch.nn实现模型。\n", "\n", "将之前的Model_3_1中的手动实现函数改为torch.nn内置函数即可，再加上继承nn.Module以使用torch.nn内置模型模板特性。" ] }, { "cell_type": "code", "execution_count": 20, "id": "0163b9f7-1019-429c-8c29-06436d0a4c98", "metadata": {}, "outputs": [], "source": [ "class Model_3_2(nn.Module):\n", " def __init__(self, num_classes):\n", " super(Model_3_2, self).__init__()\n", " self.flatten = nn.Flatten()\n", " self.linear = nn.Linear(28 * 28, num_classes)\n", "\n", " def forward(self, x: torch.Tensor):\n", " x = self.flatten(x)\n", " x = self.linear(x)\n", " return x" ] }, { "cell_type": "markdown", "id": "6e765ad7-c1c6-4166-bd7f-361666bd4016", "metadata": {}, "source": [ "训练与测试过程与之前手动实现的几乎一致。" ] }, { "cell_type": "code", "execution_count": 21, "id": "6d241c05-b153-4f56-a845-0f2362f6459b", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "36bd868142c14b278e0c64868d513a84", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/1160 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "hyper_params = {\n", " 'learning_rate': 2.0e-2,\n", " 'num_epochs': 20,\n", " 'batch_size': 1024,\n", "}\n", "\n", "model = Model_3_2(num_classes).to(device)\n", "\n", "trainer = SoftmaxTrainer(\n", " model=model, \n", " train_dataset=train_dataset, eval_dataset=eval_dataset, test_dataset=test_dataset, \n", " optimizer='manual', criterion='manual', **hyper_params\n", ")\n", "trainer.train()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 5 }