6.自动微分.ipynb

{"cells":[{"cell_type":"markdown","source":["<a href=\"https://github.com/xuehangcang/DeepLearning/blob/main/docs/PyTorch/%E5%9F%BA%E7%A1%80%E7%9F%A5%E8%AF%86/6.自动微分.ipynb\" download=\"\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"110\" height=\"20\" role=\"img\" aria-label=\"jupyter: notebook\"><title>jupyter: notebook</title><linearGradient id=\"s\" x2=\"0\" y2=\"100%\"><stop offset=\"0\" stop-color=\"#bbb\" stop-opacity=\".1\"/><stop offset=\"1\" stop-opacity=\".1\"/></linearGradient><clipPath id=\"r\"><rect width=\"110\" height=\"20\" rx=\"3\" fill=\"#fff\"/></clipPath><g clip-path=\"url(#r)\"><rect width=\"49\" height=\"20\" fill=\"#555\"/><rect x=\"49\" width=\"61\" height=\"20\" fill=\"#fe7d37\"/><rect width=\"110\" height=\"20\" fill=\"url(#s)\"/></g><g fill=\"#fff\" text-anchor=\"middle\" font-family=\"Verdana,Geneva,DejaVu Sans,sans-serif\" text-rendering=\"geometricPrecision\" font-size=\"110\"><text aria-hidden=\"true\" x=\"255\" y=\"150\" fill=\"#010101\" fill-opacity=\".3\" transform=\"scale(.1)\" textLength=\"390\">jupyter</text><text x=\"255\" y=\"140\" transform=\"scale(.1)\" fill=\"#fff\" textLength=\"390\">jupyter</text><text aria-hidden=\"true\" x=\"785\" y=\"150\" fill=\"#010101\" fill-opacity=\".3\" transform=\"scale(.1)\" textLength=\"510\">notebook</text><text x=\"785\" y=\"140\" transform=\"scale(.1)\" fill=\"#fff\" textLength=\"510\">notebook</text></g></svg></a>"],"metadata":{"collapsed":false,"id":"15F37EE0890346A8B8D1B8A6FE08E2D8"}},{"cell_type":"markdown","source":["## 6.1 使用`torch.autograd`进行自动微分\n","\n","\n","在训练神经网络时，最常用的算法是反向传播。在这个算法中，根据损失函数相对于给定参数的梯度来调整参数（模型权重）。\n","\n","为了计算这些梯度，PyTorch具有一个内置的微分引擎，称为`torch.autograd`。它支持计算任何计算图的梯度。\n","\n","考虑最简单的单层神经网络，具有输入 `x`，参数 `w` 和 `b`，以及一些损失函数。\n","\n","它可以用以下方式在PyTorch中定义："],"metadata":{"collapsed":false,"id":"F2CEEE9A613341E689226F73BE9A3B7B"}},{"cell_type":"code","execution_count":1,"outputs":[{"data":{"text/plain":"tensor(0.7622, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>)"},"execution_count":1,"metadata":{},"output_type":"execute_result"}],"source":["import torch\n","\n","x = torch.ones(5)  # input tensor\n","y = torch.zeros(3)  # expected output\n","w = torch.randn(5, 3, requires_grad=True)\n","b = torch.randn(3, requires_grad=True)\n","z = torch.matmul(x, w)+b\n","loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)\n","loss"],"metadata":{"collapsed":false,"ExecuteTime":{"start_time":"2023-04-22T12:35:28.325929Z","end_time":"2023-04-22T12:35:30.022609Z"},"id":"5A96A56FE1DF4933B5F73025246C0953"}},{"cell_type":"markdown","source":["## 6.2 张量、函数和计算图\n","\n","本代码定义了以下**计算图**：\n","\n","![](https://raw.githubusercontent.com/xuehangcang/DeepLearning/main/static/comp-graph.png)\n","\n","在这个网络中，`w` 和 `b` 是 **参数**，我们需要优化它们。\n","\n","因此，我们需要能够计算损失函数对这些变量的梯度。为了做到这一点，我们需要将这些张量的 `requires_grad` 属性设置为 `True` 。\n","\n","你可以在创建张量时设置 `requires_grad` 的值，也可以使用 `x.requires_grad_(True)` 方法来设置它。\n","\n","我们应用于张量以构建计算图的函数实际上是 `Function` 类的一个对象。\n","\n","这个对象知道如何在 *正向传播* 过程中计算函数，也知道如何在 *反向传播* 步骤中计算它的导数。\n","\n","反向传播函数的引用存储在张量的 `grad_fn` 属性中。你可以在 `Function` 的文档中找到更多信息。"],"metadata":{"collapsed":false,"id":"BAA2D42F35374C0CAE649F689CE6103D"}},{"cell_type":"code","execution_count":2,"outputs":[{"name":"stdout","output_type":"stream","text":["Gradient function for z = <AddBackward0 object at 0x0000012A6BF386A0>\n","Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward0 object at 0x0000012A6BF38D30>\n"]}],"source":["print(f\"Gradient function for z = {z.grad_fn}\")\n","print(f\"Gradient function for loss = {loss.grad_fn}\")"],"metadata":{"collapsed":false,"ExecuteTime":{"start_time":"2023-04-22T12:42:05.056671Z","end_time":"2023-04-22T12:42:05.069997Z"},"id":"BE71BEC8EC84449C9202107A53FAE306"}},{"cell_type":"markdown","source":["## 6.3 计算梯度\n","\n","为了优化神经网络参数的权重，我们需要计算损失函数对参数的导数，即我们需要在某些固定的`x`和`y`值下计算。\n","\n","为了计算这些导数，我们调用 `loss.backward()`，然后从 `w.grad` 和 `b.grad` 中检索这些值："],"metadata":{"collapsed":false,"id":"DBE868B092DA495BAE9E33331BD594C5"}},{"cell_type":"code","execution_count":3,"outputs":[{"name":"stdout","output_type":"stream","text":["tensor([[0.1110, 0.2633, 0.0917],\n","        [0.1110, 0.2633, 0.0917],\n","        [0.1110, 0.2633, 0.0917],\n","        [0.1110, 0.2633, 0.0917],\n","        [0.1110, 0.2633, 0.0917]])\n","tensor([0.1110, 0.2633, 0.0917])\n"]}],"source":["loss.backward()\n","print(w.grad)\n","print(b.grad)"],"metadata":{"collapsed":false,"ExecuteTime":{"start_time":"2023-04-22T12:43:40.164066Z","end_time":"2023-04-22T12:43:40.219757Z"},"id":"B7D64D91FDF84E29BE6762937346351E"}},{"cell_type":"markdown","source":["- 我们只能获得计算图中具有 `requires_grad` 属性设置为 `True` 的叶子节点的 `grad` 属性。对于图中的其他节点，梯度将不可用。\n","- 由于性能原因，我们只能在给定图上执行一次 `backward` 梯度计算。如果我们需要在同一图上进行多次 `backward` 调用，我们需要在`backward`调用中传递 `retain_graph=True`。"],"metadata":{"collapsed":false,"id":"A0A6E1A863A04A2787C23CA9411798C5"}},{"cell_type":"markdown","source":["## 6.4 关闭梯度跟踪\n","\n","默认情况下，所有具有 `requires_grad=True` 的张量都会跟踪它们的计算历史并支持梯度计算。然而，在某些情况下，我们不需要这样做，例如当我们已经训练好模型并只想将其应用于某些输入数据时，即我们只想通过网络进行*前向*计算。我们可以通过在计算代码周围使用`torch.no_grad()`块来停止跟踪计算："],"metadata":{"collapsed":false,"id":"2B24A45BA5BF48B4B3D798E1652BB6B8"}},{"cell_type":"code","execution_count":4,"outputs":[{"name":"stdout","output_type":"stream","text":["True\n","False\n"]}],"source":["z = torch.matmul(x, w)+b\n","print(z.requires_grad)\n","\n","with torch.no_grad():\n","    z = torch.matmul(x, w)+b\n","print(z.requires_grad)"],"metadata":{"collapsed":false,"ExecuteTime":{"start_time":"2023-04-22T12:46:21.698861Z","end_time":"2023-04-22T12:46:21.713572Z"},"id":"F032A70E5B1B44CE80E93F50B09F414A"}},{"cell_type":"markdown","source":["另一种实现相同结果的方法是在张量上使用 `detach()` 方法："],"metadata":{"collapsed":false,"id":"78CF4474104745D8BD6DEA24612A8AD6"}},{"cell_type":"code","execution_count":5,"outputs":[{"name":"stdout","output_type":"stream","text":["False\n"]}],"source":["z = torch.matmul(x, w)+b\n","z_det = z.detach()\n","print(z_det.requires_grad)"],"metadata":{"collapsed":false,"ExecuteTime":{"start_time":"2023-04-22T12:46:49.009613Z","end_time":"2023-04-22T12:46:49.048571Z"},"id":"BAB1EE3A6B09418B89A49F4EB2484BBA"}},{"cell_type":"markdown","source":["有几个原因可能会让你想要禁用梯度跟踪：\n","\n","- 将神经网络中的某些参数标记为`冻结参数`\n","- 在仅进行正向传递时，为了**加快计算速度**，因为不跟踪梯度的张量计算更高效"],"metadata":{"collapsed":false,"id":"B5A8EE9CB13742D0A25A3AB1E7623DD5"}}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python","nbconvert_exporter":"python","file_extension":".py","version":"3.5.2","pygments_lexer":"ipython3"}},"nbformat":4,"nbformat_minor":0}