A Gentle Tutorial on Vector Calculus

Published on Thursday, 21-08-2025

#Tutorials

(Adopted from CS221) (../../images/basics/calculus.png)

🚀 A Gentle Tutorial on Vector Calculus

With Real Functions, Vector Inputs, and Matrix Inputs

Vector calculus is the mathematical backbone of modern machine learning, optimization, and physics. In this tutorial, we’ll explore how calculus extends beyond single-variable functions to vectors and matrices. We’ll cover:

Real Functions of Real Inputs
Functions of Vector Inputs
Functions of Matrix Inputs

And along the way, we’ll implement these ideas with Python and NumPy, plus some visualizations using Matplotlib.

1. 🔢 Real Functions of Real Inputs

This is the classic case you learn in calculus: A function $f : \mathbb{R} \to \mathbb{R}$ .

Example:

f(x) = x^2 + 3x + 2

Its derivative:

f'(x) = 2x + 3

Python Implementation

import numpy as np
import matplotlib.pyplot as plt

# Define function and derivative
f = lambda x: x**2 + 3*x + 2
f_prime = lambda x: 2*x + 3

# Plot
x = np.linspace(-5, 5, 200)
plt.figure(figsize=(6,4))
plt.plot(x, f(x), label="f(x) = x² + 3x + 2")
plt.plot(x, f_prime(x), label="f'(x) = 2x + 3", linestyle="--")
plt.legend()
plt.title("Real Function and Its Derivative")
plt.xlabel("x")
plt.ylabel("Value")
plt.grid(True)
plt.show()

👉 This simple case generalizes naturally to vectors.

2. 🧭 Functions of Vector Inputs

Now, consider a function $f: \mathbb{R}^n \to \mathbb{R}$ . For example:

f(\mathbf{x}) = x_1^2 + 2x_2^2 + 3x_3^2

Here, $\mathbf{x} = (x_1, x_2, x_3)$ .

The gradient is the vector of partial derivatives:

\nabla f(\mathbf{x}) = \left[ \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \frac{\partial f}{\partial x_3} \right] = [2x_1, 4x_2, 6x_3]

Python Implementation

# Vector function: f(x1, x2, x3) = x1^2 + 2x2^2 + 3x3^2
def f_vec(x):
    return x[0]**2 + 2*x[1]**2 + 3*x[2]**2

def grad_f_vec(x):
    return np.array([2*x[0], 4*x[1], 6*x[2]])

x = np.array([1.0, 2.0, -1.0])
print("f(x) =", f_vec(x))
print("∇f(x) =", grad_f_vec(x))

✅ This is the foundation of gradient descent in machine learning.

Visualization in 2D

For a function $f(x,y) = x^2 + y^2$ , the gradient points outward radially.

X, Y = np.meshgrid(np.linspace(-2, 2, 20), np.linspace(-2, 2, 20))
U = 2*X
V = 2*Y

plt.figure(figsize=(6,6))
plt.quiver(X, Y, U, V, color="blue")
plt.title("Gradient Field of f(x,y) = x² + y²")
plt.xlabel("x")
plt.ylabel("y")
plt.show()

3. 🏗️ Functions of Matrix Inputs

Now let’s extend further: $f: \mathbb{R}^{m \times n} \to \mathbb{R}$ .

Example:

f(A) = \text{Tr}(A^T A) = \sum_{i,j} a_{ij}^2

This is the squared Frobenius norm of a matrix.

The derivative (gradient w.r.t. $A$ ) is:

\nabla_A f = 2A

Python Implementation

# Function of a matrix: Frobenius norm squared
def f_matrix(A):
    return np.trace(A.T @ A)

def grad_f_matrix(A):
    return 2*A

A = np.array([[1., 2.],
              [3., -1.]])
print("f(A) =", f_matrix(A))
print("∇f(A) =\n", grad_f_matrix(A))

4. 🧮 Jacobian Matrix

If we have a vector-valued function:

\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m

with

\mathbf{f}(\mathbf{x}) = \begin{bmatrix} f_1(\mathbf{x}) \\ f_2(\mathbf{x}) \\ \vdots \\ f_m(\mathbf{x}) \end{bmatrix}

the Jacobian is the matrix of partial derivatives:

J_{\mathbf{f}}(\mathbf{x}) = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \frac{\partial f_2}{\partial x_1} & \cdots & \frac{\partial f_2}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}

Example

\mathbf{f}(x,y) = \begin{bmatrix} f_1(x,y) = x^2 + y \\ f_2(x,y) = e^x \cdot y \end{bmatrix}

Jacobian:

J_{\mathbf{f}}(x,y) = \begin{bmatrix} 2x & 1 \\ e^x y & e^x \end{bmatrix}

Python Implementation

import sympy as sp

# Define variables
x, y = sp.symbols('x y')

# Define vector-valued function
f1 = x**2 + y
f2 = sp.exp(x) * y
f = sp.Matrix([f1, f2])

# Jacobian
J = f.jacobian([x, y])
J

✅ The Jacobian is crucial in neural networks (backpropagation) and multivariable optimization.

5. 📐 Hessian Matrix

The Hessian generalizes the second derivative to multiple dimensions.

For a scalar function $f: \mathbb{R}^n \to \mathbb{R}$ , the Hessian is:

H(f)(\mathbf{x}) = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots \\ \vdots & \vdots & \ddots \end{bmatrix}

Example

f(x,y) = x^2 + xy + y^2

Gradient: $\nabla f = [2x + y, \, x + 2y]$
Hessian:

H(f) = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}

Python Implementation

# Define scalar function
f = x**2 + x*y + y**2

# Gradient
grad_f = [sp.diff(f, var) for var in (x, y)]
print("Gradient:", grad_f)

# Hessian
H = sp.hessian(f, (x, y))
H

✅ The Hessian tells us about curvature:

Positive definite → local minimum
Negative definite → local maximum
Mixed signs → saddle point

6. 🌐 Visualization: Hessian as Curvature

Let’s visualize $f(x,y) = x^2 + xy + y^2$ .

from mpl_toolkits.mplot3d import Axes3D

X = np.linspace(-2, 2, 50)
Y = np.linspace(-2, 2, 50)
X, Y = np.meshgrid(X, Y)
Z = X**2 + X*Y + Y**2

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap="viridis", alpha=0.8)
ax.set_title("Surface of f(x,y) = x² + xy + y²")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_zlabel("f(x,y)")
plt.show()

The Hessian explains the bowl-shaped curvature we see in the plot.