Why Neural
Networks?

Good [morning/afternoon], everyone. Before we dive in, I want to start with a very human question: Why does your phone recognize your face? Why can you talk to a speaker on your kitchen counter and it understands you? Why can a program beat world champions at chess, Go, and StarCraft?

The answer isn't magic. It's a set of ideas that started in the 1940s — ideas borrowed directly from how your brain works. By the end of today, you'll understand those ideas from the ground up. We start incredibly small — with one fake neuron — and build piece by piece until we can see why the massive networks powering modern AI are designed the way they are. Every step will make logical sense.

Let's go back to 1943. Two scientists — Warren McCulloch, a neuroscientist, and Walter Pitts, a mathematician — asked a wild question: Can we model what a brain cell does using pure math?

A neuron in your brain receives signals through dendrites, collects them in the soma, and if the total signal is strong enough, it fires — it sends a signal down the axon to the next neuron. McCulloch and Pitts built a math version of that.

The model: inputs are either 0 or 1. We add them all up. If the sum crosses a threshold (θ), the neuron fires (output = 1). Otherwise it stays silent (output = 0).

Frank Rosenblatt (1957) makes one crucial upgrade: weights. Each input gets a number — a weight — saying how much to trust it. When deciding whether to carry an umbrella, "is it already raining?" matters far more than "is the sky slightly cloudy?"

The second upgrade: the perceptron can learn those weights by itself. It looks at examples, makes a prediction, sees if it's wrong, and nudges the weights to do better. This is the core idea of machine learning.

The perceptron draws a decision line — everything on one side is YES, the other NO. Try dragging the weights below and watch the line move.

In 1969, Minsky and Papert dropped a bombshell: a single perceptron cannot solve XOR. XOR outputs 1 if inputs are different, 0 if they're the same: (0,1)→1, (1,0)→1, but (0,0)→0 and (1,1)→0.

Plot those four points. Can you draw one straight line separating the 1s from the 0s? Try below — you literally can't. The 1s are diagonally opposite. This killed research funding for a decade — the "AI winter."

But the conclusion wasn't that neural networks were bad. It was that one layer isn't enough. The solution was right there: use more layers.

Stack perceptrons in layers — that's an ANN. Three layer types: Input (raw data, no computation), Hidden (where the thinking happens), Output (final answer).

But there's one crucial ingredient: activation functions. The step function — fire or don't fire — is terrible for learning. We replace it with smoother functions. Without non-linearity, 100 stacked layers would mathematically collapse into one. Non-linearity is what gives deep networks their power.

Explore the four main activation functions below, then drag the point to feel how each one responds.

Let's trace data through an ANN — this is forward propagation. Our example: predict if a student will pass an exam, given hours studied and hours slept. Watch the signal pulse through the network below.

We predicted 83% chance of passing — but the student actually failed. Now we need to go back and fix the weights. But in a network with millions of weights, which ones caused the mistake, and by how much?

Backpropagation solves this with calculus (the chain rule): it figures out exactly how much each weight contributed to the error, then nudges every weight in the direction that would reduce it.

Think of it like hiking to find the lowest valley blindfolded — feel which way is downhill, take a step, repeat. The visualization below shows this "loss landscape."

Our examples used simple tabular data. But a 256×256 image has ~200,000 numbers — a plain ANN treats every pixel independently, ignoring that pixel (100,100) is next to (101,100). It can't "see" spatial relationships.

Text? "I love this movie" vs "I hate this movie" — one word flips the meaning. Word order and context are everything. Speech? Audio is a sequence over time — timing matters enormously.

Different data types need different inductive biases — architectural assumptions baked into the network that match the structure of the data.

CNNs use a small filter (e.g. 3×3 pixels) that slides across the image, detecting local patterns. The same filter reused everywhere = weight sharing = far fewer parameters. Early layers detect edges. Middle: shapes/textures. Deep: faces, objects.

RNNs process one element at a time (one word, one audio frame), maintaining a hidden state — short-term memory carrying info from previous steps. Click words in the sentence below and watch memory accumulate.

These architectures — CNN, RNN, and modern Transformers — are all built on the same foundation we've covered. The Transformer replaced RNN's sequential memory with attention: every word looks at every other word simultaneously. That's what made large language models possible.

But you now understand the foundation. A single artificial neuron → weights → activation functions → layers → forward propagation → backpropagation. The rest is evolution of these same ideas.