COVID-19 is one of those events that will likely not just define the way people will interact with each other, but likely entire socieities. I wouldn't even be surprised to see this pop up in an AP US History textbook in a decade just for how long the pandemic has been drawn out for. So it should be no surprise that from the first month of the pandemic, a vaccine was the only thing on people's minds. I mean, just look at this graph depicting the number of total coronavirus cases in the U.S. alone.

It took around 2 months to hit a million cases in the U.S., and another 3 months to reach 5 million cases. That's the issue with pandemics: they explode at a rate faster than we can realize. So, today, I want to talk a little bit about different types of disease models, and how these can educate ourselves on the right preventative courses of action.

The SIR Model 2 Ways

The Susceptible Infected Recovered Model of disease spread groups individuals into 3 boxes and relates them to show how each box grows or diminishes over time: susceptible means that you are currently healthy but are vulnerable to the infection; infected means just that and indicates a current illness; and while recovered doesn't necessarily mean you overcame the disease, it means you are unable to spread the disease anymore (sobeit proper recovery and developed immunity, but also the unfortunate case if you die since both are no longer disease vectors). This sounds like the perfect use for a Markov chain (here's a refresher if you need one)! Our Markov chain will have 3 states being the aforementioned susceptible, infected, and recovered, and will follow a transition model as such:

Let's say everyone starts out as susceptible; we can write our initial population as $N$, and the initial distribution of those $N$ people as $P = \begin{bmatrix} N & 0 & 0 \end{bmatrix}$ for $N$ susceptible, 0 infected, and 0 recovered.

With that out of the way, let's think about the above Markov chain. If you are currently susceptible, each day there's a chance you might become infected. Let's call this probability $\beta$ as the average chance of getting infected. At the same time though, if you are smart and act carefully, you might not become infected, and this will be $1-\beta$. Similarly if you're infected, there's a chance you might (positively or otherwise) overcome the disease! We'll call this probability $\gamma$, and the chance of not recovering $1-\gamma$ (you can think of $\gamma$ by taking its inverse: $\frac{1}{\gamma}$ is the average number of days for a recovery to occur). Finally, if you're recovered, that's the end of your journey, as once recovered you're always recovered. This can be condensed into the simple transition matrix below.

With that, let's run a trial with infectivity $\beta=.4$ and $\gamma=\frac{1}{5}$ (we expect an infection to take 5 days to recover). So on Day 0, everyone is healthy and (ironically) susceptible to our disease. The following Day 1, people are interacting and enjoying themselves, unbeknown to them that they are spreading a new contagion. Day 1 results in $PM = \begin{bmatrix} 800 & 200 & 0 \end{bmatrix}$, which makes sense as we expected 20% to become infected. We can track each state of S, I, and R and plot them accordingly across the span of a month.

For such a simple model, it's not bad, but there are some obvious flaws. For starters, the infected population doesn't really infect others; all infections stem from random appearances in the susceptible population. This is why by the end of the 30 days, we have 0 susceptible and only recovered, since infections don't require infected to be present which is kind of odd (notice how our initial population was only susceptible and yet infected pop up). We can do better.

While Markov chains were appealing due to the nature of having 3 states, what we really want to focus on is the relationships between the 3 states: if there are lots of infected people, that should increase the rate at which others get infected. Since we're looking at how the value of one box (infected people) affect the rate at which the other boxes change (how fast susceptible and recovery decrease/increase), perhaps we should try a system of differential equations. Instead of states, we'll now have 3 different functions: $S(t)$, $I(t)$, and $R(t)$, which returns the number of susceptible, infected, and recovered at a time $t$. If you're not familiar with differential equations, don't worry, this section will be brief. The idea behind differential equations is that sometimes it's hard to exactly quantify a function or value, but we know how the function changes relative to another value. See the first few minutes of this video if this idea intrigues you and for some nice opening examples. Besides, you already know more or less what the equations look like we know what the function should look like.

That's really all there is to them. The specific math behind it looks more complicated than it is, but keeping the above in mind will make it much clearer.

I like to think of it that of that the fraction of the vulnerable people $\frac{S}{N}$ have a $\beta$ chance of being infected, which is scaled up by the number of infected people $I$. That makes sense for number of new infected people. Similarly, if the average chance of recovery is $\gamma$, then you should expected a proportion of $\gamma$ infected people to recover ($\gamma I$). Last important fact to note is what happens when we add these equations together:

No matter how the 3 different categories of people evolve over time, the net change between all of them will be 0, meaning our total population remains the same over time (which is good since we don't want people appearing and disappearing out of nowhere). Let's watch the scenario unfold once more with $N=1000$ with a distribution of $P = \begin{bmatrix} 999 & 1 & 0 \end{bmatrix}$ (someone has to start the pandemic), $\beta=.4$, and $\gamma=\frac{1}{5}$.

That's more like that famous curve. It may not be as drastic as the 10 day everyone-is-infected model as the Markov chain we tried earlier, but it's definitely much more realistic. A single person was able to infect about 800 people across only a couple of months, which is still scary fast.

$R$ and Disease Containment

Just as famous as a curve this may be, you likely have heard of the idea of $R$ and $R_0$ too. $R$ is the reproduction number of a disease/virus that tells you on average, how many people an infected person will spread the disease to. If $R>1$, then you get the epidemic issue, where the disease is spreading exponentially as each person is giving more people than just themselves the disease. When $R=1$, you have an endemic where the disease is neither spreading nor being contained. When $R<1$, then you have a contained virus that is decaying throughout a community. $R_0$ is just the $R$ value at the start of the outbreak, and can be found with the formula $R = \frac{\beta}{\gamma}$. In our simulation, $R_0 = \frac{.4}{.2} = 2$. So if we wanted to contain this disease, we want to find how many infections we need to contain for $R<1$. Since $R_0 \cdot .5 = 1$, we only need to contain 50% of infections to contain the outbreak!

You can see as soon as there is less than 50% of the population left to infect, infections start to decline—not end, but decline. You can also reduce infectivity by wearing preventative measures (i.e. masks) or getting the appropriate immunization against the disease (i.e. vaccines). If you want to read more about this model and more of its intricacies, Nicky Case wrote a very nice interactive article about it.

Small Scale Vaccinations and Graph Theory

While we did find a nice model to represent disease spread, the SIR model only is really nice for analyzing very big communities. With the SIR model, we assume everyone interacts with enough people so that infected people can always target a susceptible person if they need to in this very dense network of people.

Most people, though, only regularly interact and really care about those immediately in their social circles. In this small scale, such as say, a neighborhood or even a friend group, people do not interact with others equally. Some are more introverted and interact with only a few people, and some are extroverted beacons that interact with most of the group. This changes the dynamics of disease spread a lot as obviously those who are more outgoing and meet more people will be more at risk of being infected than someone who only talks to a few people. As such, we'll need a different approach than the SIR model.

In order to model connections between people we'll use a graph. A graph in this case is not the standard parabola of $f(x) = x^2$, but rather it is a visual tool consisting of two parts: there are nodes which are dots (to represent people here), and edges that are lines that connect nodes (to represent interactions or friendships).

As in most groups, we have a couple of people at the center connected to quite a few people, and some on the edge of the circle regularly interacting with as few as 2 people. This is a really helpful representation as now not only can we watch where the virus spreads, but we also have a direct way to implement small scale prevention tactics as we can see specifically who is infected. Ideally, we would vaccinate everyone in this group and make sure (consequential) disease spread can occur, but that might not happen. So, if we could only vaccinate, say, 25% of this group, who would you vaccinate to minimize a disease vector? There are some obvious candidates such as the nodes with the most connections, but what is the best way Let's leave our susceptible people in green and infected in red, but we'll add a new purple node to indicate vaccinated.

Before we talk strategies, let's be clear about some of the assumptions made for simplicity sake.

• Vaccinations are 100% effective both ways, completely negating the possibility of infection of and transmission from the vaccinated person as if fully immune.
• Edge lengths do not matter; if someone is connected to another person, we assume that that edge will act the exact same way as all other interactions and edges. This ensures a contant $\beta$.
• Dying and recovering from the disease will be again treated the same under an overarching "Recovered" state, in which the affected person becomes fully immune.
• An infected person "recovers" after only a single day ($\gamma = 1$) since a) it will ensure our simulation can spit out some numbers at the end as it's hard to completely protect an entire network of people, and b) our program wasn't written the most efficiently.

As before, we want to find strategies ways to reduce $R_0$. But, what is $R_0$ in this case? Well, remember $R_0$ is just the average number of transmissions a single infected person will instigate. So, we can approximate this with the average number of edges an infected node has, multiplied by the infectivity. Since we can't change the infectivity rate ($\beta$), the only way we can lower it is by removing possible edges for infected people to transmit across. So, let's look at different strategies that are both really good and really bad at removing edges from our graph.

1. Random: Realistically, vaccinating people randomly seems like a bad idea. In a simulation, though, it's never a bad starting point.
2. Random then neighbors: Here, we start with a single random person to vaccinate, then only vaccinate people connected to an already vaccinated person. If your friend got vaccinated, why shouldn't you?
3. Most connected: This is the obvious solution. If we want to reduce edges efficiently, why don't we vaccinate the nodes with the most edges connected to the most people?
4. Most connected non-neighbors: This is a spin on the last one. We want to vaccinate the most connected people, but why not spread them out a bit? We pick the highest connected node to vaccinate, and then proceed to vaccinate the next highest node that is not connected to a vaccinated person.
5. Least connected: This one is more of a why not scenario. It's like the previous two strategies but reversed: find the loners of the group and vaccinate them.

Again, we'll randomly vaccinate $\approx$25% of our population (50/100, 100/200, 200/400, and 400/800 people/total connections) according to our strategies above. We'll also infect 10% of our population with the contagion of your choosing with the same infectivity at $\beta = .4$ as before. We'll do 10 trials per population size, and average them to get a rough estimate at how each strategy faired.

Unsurprisingly, vacciating the most connected nodes without restriction faired the best; closing off as many routes of infection as fast as possible led to only about 10% of non-vaccinated people getting infected. Following close behind is most connected non-neighbors strategy. Even though we tried spreading out vaccinations since the edge between two vaccinated people doesn't require two does of the same vaccine, by nature of being a popular node, it is likely connected to other popular nodes too. So in reality, we are removing fewer edges than we could be just vaccinating the most popular nodes. It's for this reason why our orange strategy of randomly vaccinating one person then its neighbors was such a bad strategy: not only do we not know if we're vaccinating a well connected person, but we're wasting vaccines but doubly protecting edges between neighbors. Since reducing the edge count is our only form of reducing $R_0$, this is a very bad strategy. What is surprising though, is that vaccinating the least connected people was almost as good as random. Since unpopular people can only be connected to so many people, they tend to be pretty spread out removing a fair number of edges between all the vaccinated people.

What next?

Honestly, our graph model isn't all that bad considering how simple it is, but let's look at some nice upgrades we can hand over to it.

Graph Generation

This was originally made as a school project, so we generated our graph by taking some number of edges and placing them randomly between nodes for simplicity in presentation. However, these obviously aren't the only types of networks. A prominent one for modelling interactions between people is the Erdös-Rényi graph. For $n$ nodes, there are $n \choose 2$ possible pairings of nodes for edges. In an ER graph, we flip a weighted coin to accept an individual edge, and the final result is just the collection of edges we added. If our acceptance probability is $p$, then we would expect the graph to have a total of ${n \choose 2}p$ edges, each node to have on average $p(n-1)$ connections. In this network, people are approximately as popular as others (since our coin flip method leads to a binomial distribution with few super popular or super unpopular people).

Another type of graph is the Barabási-Albert type graph. The idea with this network is that people tend to be connected to a distinct, central "hub" of a few people that make up most of a network. The idea is that you start with a small network of $m$ people/nodes to begin with. At each step, you add a new node and connect it to another person $i$ with probability $p_i = \frac{E(i)}{\sum_jE(j)}$ where $E(x)$ is the number of edges for node $x$ and $j$ represents all other nodes. The idea is that if $E(i)$ is really big (i.e. a super popular person), then you'll have a much bigger probability to connect to that person over someone who only has a few connections. Wouldn't you want to hang out with the cool kids too? You can give $i$ as many connections as you want, so long as it's less than $m$ (think about the first new node: how can someone have 10 different friends in a group of 5?).

If you want to explore this more, the inspiring article that led to this project analyzes these 2 exact graphs in the exact problem we discussed above, going a bit more in depth with the idea of the counterintuitive friendship paradox.

New Vaccination Schemes

We looked at a very specific set of vaccination parameters, namely relying on all edges are treated equally and that the number of edges a node has is constant. In reality, people don't interact with each other equally, so instead of having a generic edge, we could implement an edge weighting. This quality would add a number between 0 and 1 to each edge to represent how "friendly" two people are with each other. We could then simulate the difference between best friends and mere colleagues interacting that may have more or less of a chance to spread a virus. The other aspect to consider was updating edge counts. Ideally we would only want to count a vaccine's candidate's number of edges to vulnerable people, not vaccinated onr infected people too, with counts updating after every vaccination. Then we could focus on protecting people specifically.

The final problem I wanted to talk about what the goal of vaccines are in a network. As we said, it's about removing edges efficiently, but some edges are definitely worth more than others. Imagine you have two friend circles, with exactly one mutual person between them. Even if that mutual friend has only two edges, vaccinating him would close that bridge between the two friend circles, isolating the virus into only one of them.

This searching for a way to split our graph in two parts with the fewest number of edges removed is called the sparsest cut problem. If we can find the (usually approximate) sparsest cut, we can divide our graph in two and attempt to isolate the virus in only a singular bubble. Before, we were looking to remove something known as Hamiltonian paths, where you can't connect any one node to another. Here, we are trying to isolate the virus into a smaller bubble.

So here we would want to vaccinate everyone on the border of the yellow-purple divide to make the sparsest cut a reality. Of course, this only works if you have enough vaccines AND if your sparsest cut splits the graph in half evenly. This strategy can be used recursively on the two sub-graphs, so the more vaccines you have, the better this strategy works.

Conclusion

I hope this brought some light to the epidemic math that goes on behind the scenes of so many news articles, but more importantly showed you the power of modelling situations not just in different ways, but creative ones as well. Graph theory in particular has such widespread applications, often just thinking of something simply with connections can allow you to borrow from its variety of tools and techniques.

As with all puzzles, drawing something always helps.

The key to this puzzle is to use the triangle inequality: no side of a triangle can be longer than the sum of the other two.

So, that means that no segment can be longer than half the length of our stick. Now, we can start solving the puzzle. Without loss of generality, make the first break at point $x$ between 0 and .5 to ensure our first break is on the left side (if it's not within this range, it'll be some symmetric case on the right half of the stick). Our left half of the stick is already less than .5, so that's good, but that means our second break must be on the right half, as otherwise that will be a leg that will be greater than .5, and that's no good. The probability our second break will be on the right half of the leg is equal to the length of that segment over the total line (since it's uniformly random): $\frac{1-x}{1}$. Let's analyze that longer leg now.

We can't make a cut at or further than the .5 mark on the segment for the obvious reason: it would make a leg longer than .5, breaking the triangle inequality. For a similar reason, we can't make a cut at $1-x-.5$ or earlier, as that will make a leg on the right side longer than .5 too. We need a cut in between those two boundaries. Just like we found the probability for making a cut on this longer leg to begin with, if we can find the length of that feasibile region and divide that by the length of the leg, that will give us the probability for making a good cut here. The length of the red feasible region is $|.5-(1-x-.5)| = |1-1+x| = x$, and the total length is just $1-x$. So the probability our second cut is valid is $\frac{x}{1-x}$.

Putting it all together, we get the probability that our second cut is valid given a first cut at $x$ is $\frac{1-x}{1} \cdot \frac{x}{1-x} = x$. Kind of neat that the probability is exactly proportional to the length of the first break. But of course, $x$ isn't constant. It too is a random variable, so we can average it to get an expected probability across a large number of trials.

As a gut check, this should make sense as .25 is exactly the midpoint between 0 and .5, the boundaries we set at the start of the solution.

So, if you break a stick at 2 uniformly random points into 3 segments, the probability those 3 segments can form a triangle is exactly $\frac{1}{4}$.

Obviously, we can't just make infinite of each wearable. We have some constraints between available cloth, printing, and effort. If you have 20 sheets of cloth, and each shirt requires $\frac{1}{2}$ of one and the hats only require $\frac{1}{5}$ of one. For printing, maybe you only have enough ink for 100 prints total. Finally, hats are hard to make, so you cap yourself at only making 70 and no more. We have a list of constraints which can be nicely added as a set of inequalities:

This isn't an easy system to solve as, well, we're not solving anything in the traditional sense by plugging in equations to one another, nor is it a standard maximization problem where we can take some form of a gradient since every equation here is a line, so we would only get constants that give us no new information.

This type of problem where we want to maximize a linear equation that is also bound by a set of linear restraints is called a linear program. These types of problems are called specifically programs as you'll see that we have analytic ways to find solutions, but to find the best solution requires some well-crafted algorithms to make solving them faster and more efficient. Before we can understand the 2, and later the general $n$, variable set up, let's try an easier case.

1D Linear Program

As with all problems, let's try and cut back on degrees of freedom so we can really focus on what's going on here. Instead of maximizing a function of 2 variables, let's do a function of 1 variable.

This isn't all that interesting, as I'm sure the answer is popping out almost immediately: $\max(3x)=30$ from since we can only have at most 10 of variable $x$ from the first constraint. What will become an important concept later is how we visualize these solutions to linear programs, so let's graph our inequalities on a number line to represent the quantity of variable $x$.

In blue, we have our first constraint of showing all values of $x$ that satisfy $5x \leq 50$, and in red we have our second constraint showing all $x$ that satisfy $x \geq 6$. Where those two regions overlap in the purple-ish area is the feasibility region, as well, it's the area where all of our constraints are satisfied and where feasible values of $x$ lie. The feasibility region defines a polytope, a generalization of the 2D polygon and 3D polyhedron. The key takeaway is that our solution of $x=10$ is an edge case: it lies on the furthest possible boundary of our feasibility region.

This makes sense for two reasons. 1) We want to always be looking to maximize or minimize our objective function ($3x$, in this case) so if we can add more or less of $x$ to our solution, we want to do that depending on our goal. This is a more intuitive way to show reason 2) our function is linear, so as long as we go in one direction, it will always be increasing or decreasing. Let's add the $y$-axis into this graph and let $y=3x$ to see the value of the function we're maximizing at different values of $x$.

Since our objective function is linear, it is directly proportional to $x$, so as $x$ increases or decreases, $y$ also has to only increase or decrease with no chance of weird curves or bends in its function. So if we want to maximize our function, we just want to walk in the direction of $x$ that does so until we hit the edge of our feasiblity region. Similarly, if we wanted to minimize it, we'd walk the other way along $x$ that decreases $y$.

With this in mind, we can now go back to our original 2 variable case.

2D Linear Program

Recalling what our original setup was

Let's plot our feasibility region like we did before. Only now, it'll be a 2-dimensional region instead of just the number line. We'll put $s$ for shirts on the $x$-axis and $h$ for hats on the $y$-axis.

Just as in the 1-variable case, our solution for maximizing our objective function should lie at the edge of our feasible region. I've marked the feasible region's defining vertices in purple. It's worth pointing out that I have added an additional vertex not defined by our original constraints, and that's the vertex $(0,0)$. This is aptly called the non-negativity constraints as it, well, constrains our variables to be non-negative (shocking, right). These tend to be a requirement for some minimization cases just so that we don't end up with problem of running into a pair of negaively infinite numbers as our solution, which obviously doesn't make sense.

Although what I said about our solution being on the edge of the region isn't true, I can tell you more about our maximum solution: it will (usually) be one of the vertices of our feasible region. If you want a more math-based explanation, check this out, but there's a very intuitive way to view think of this, especially with a 2-variable linear program.

Remember how in the 1D case, our objective function $y=3x$ could be represent as a line? We can do a similar idea and add a third $z$-axis to our 2D problem and get $z=15s+10h$. This equation gives us a 3-dimensional plane as we encode the number of shirts, hats, and the amount of money they profit. Our constraints are also planes, acting as curtain-like walls extending forever vertically in the $z$-axis (as all values of $z$ will satisfy whatever values of $s$ and $h$ that lie on the line). Now, imagine our objective function plane of $z=15s+10h$ to be like a tilted table, and we place a ball on it to roll, what does the ball do? Well, the ball will just roll down the incline to the lowest point gravity will push it down. Normally, the ball will roll down forever and ever as our table extends infinitely in all directions, but we have our constraining walls to stop the ball. As soon as the ball hits a wall, it'll continue to slide down that wall as long as the ground tilts downward. It'll only stop if another wall pushes it into a corner, which is where our constraints form a vertex. Isn't that neat? The only times this won't be true is if one of our constraints form a wall that is exactly perpendicular to the direction the ball is rolling, in which case you will have a line segment worth of solutions (which includes two vertices at the end of it). This specific logic applies to finding minimums of our objective function, but you can also think of the reverse for maximums with a tilted ceiling and a balloon instead of a ball. It's basically a streamlined version of gradient descent since the gradient of the plane is always constant.

So if you manually check all 5 vertices, you'll find that a combination of $(50 \textrm{ shirts}, 50 \textrm{ hats})$ gives the optimal combination of $1250 profit. Not bad.

The Simplex Algorithm

Not to dismiss the wonders of guess-and-check, but it only worked well for us due to the small number of variables and constraints we had. As we tackle more complex and intricate problems, we want to be able to solve linear programs much quicker and more efficiently. There are many algorithms that have been developed for LP optimization with varying motives, but the first one was George Dantzig's simplex algorithm.

The simplex algorithm is essentially a systematic way for us to narrow our guess-and-check quickly. We start at some vertex, and then travel edges of the feasbile region between vertices until it lands on the optimal solution. The idea is by turning our constraints into a matrix, we can use Gaussian elimination to move between possible candidate solutions until no improvement to our objective function can be made. Let's use our original 2-variable problem to try it out.

To start, we're going to turn these inequalities into equations by adding slack variables. To avoid confusing variable names later, I have replaced $s$ with $x$ and $h$ with $y$. The idea is that if our inequalities are anything less than the right-hand side, we should be able to cover that excess by adding an extra variable to make it equal. Rewriting all of our inequalities (including the objective function with a new objective variable $z$) into equations, we get

This is a system of linear equations we can encode as an augmented matrix!

Here are some things to identify in our simplex tableau above. The first row is more a convenience than anything, keeping track of which column corresponds to each variable. The second, third, and fourth rows all correspond to some constraint we have rewritten as equations with slack. The fifth and final row is our objective function which we are trying to maximize. So, by default, we assume we start at the point $(0,0)$ in our feasibility region. Yes, it is a vertex and therefore a candidate solution, but obviously it's not the maximum solution we want, giving us a whopping $z=0$. How do we find the next candidate point? Well we first want to identify which of our variables would increase $z$ the quickest. Well, notice that $x$ has a coefficient of $-15$ in the objective function final row, while $y$ has only a value of $-10$ (we call these numbers indicators). This means for every unit of $x$, we gain an additional \$15 contrasting an additional \$10 from a singular unit of $y$. Since $x$ increases $z$ faster than $y$ does, let's focus on column 1.

If we're going to increase $x$ to increase $z$, how do we know how much to increase it by? We can use our handy constraints to tell us exactly that. What we do is we take each value in the $x$ column and divide its associated row's constraint by that value. So, for the $x$ variable we have

Now why is this helpful? These divisions tell us the maximum amount of $x$ we can have according to each constraint (as we assume $y$ can be 0). So, if $\frac{1}{2}x + \frac{1}{5}y \leq 35$, then $x$ can be at most 70 without breaking that constraint. Similarly, $x$ can be at most 100 for the second constraint, and there's no limit for $x$ on the third constraint since it doesn't impact that inequality. So, this tells us we should use the first row as our guiding row! This is because if we used the second row with a maximum of 100, we would be violating our first constraint that $x \leq 70$. So, we call $\frac{1}{2}$ our pivot as we use that value to shift our focus from one vertex solution to another. We use our pivot to do Gaussian elimination, and create 0s in that column's other rows to land us at a new vertex.

This new vertex we have arrived at is exactly the solution you get only focusing on achieving a maximum $x$ at the point $(70,0)$, which is in fact, the solution with the most units of $x$.

\$1050 is definitely much better than netting \$0, but how do we know if we can make more? Going back to our objective function in the last row of our tableau, there's a $-4$ in the $y$ column, and as before, should mean there's a potential to increase profit by adding some $y$ component to our solution. Before we do that though, notice how some of our constraints have changed. $R_2$ used to denote $x+y+s_2=100$, but now it shows $\frac{3}{5}y - 2s_1 + s_2 = 30$, or as an inequality, $\frac{3}{5}y \leq 30$. What caused our constraint to change? It has to do with the fact we added rows together. Let's analyze the two original constraints in question of $R_1$ and $R_2$.

We can then treat these as a system of equations like we did with Guassian elimination, and perform the same operation as we did before with $R_2 - 2R_1$.

The important part to notice is that we're subtracting from the blue region $\color{blue}{R_2}$ and expecting a positive result (since both $x$ and $y$ must be greater than 0). This can only happen for our feasible region where the blue area $\color{blue}{R_2}$ is strictly greater than the red area $\color{red}{R_1}$. This appears only when $\color{red}{R_1}$ is contained within $\color{blue}{R_2}$. The $y$-value we solved for of $\frac{3}{5}y \leq 30$ is that maximum $y$ value of which that holds true. Anything above $(50,50)$ and suddenly the blue region is contained by the red, but anything below is fair game.

Our constraints haven't actually changed, they actually narrowed in given new information of which boundaries we are at with our current candidate solutions.

With that justified, we can now move on to repeating our pivoting process like before. In our new tableau, the $y$ column has a value of $-4$ in the indicators, meaning we have a possibility to increase $z$ by changing increasing $y$. Going through our process all over again to find and use the pivot:

With a new pivot found...

There are no more negative numbers in the objective function row's indicators, so this should be our optimal solution! This is because of the actual equation that row represents:

Since every variable, including our slack variables $s_1$ and $s_2$, is non-negative, the maximum our objective function can be is $3z=3750$ with both $s_1$ and $s_2$ equal to 0. So, while we may be done doing computing our solution, we should simplify our tableau to make it more readable. Let's make $x$, $y$, $z$, and $s_3$—our non-zero variables—have coefficients of 1 so we can easily find their values.

Our second step in the simplex algorithm brought us to our optimal solution of $x=50$, $y=50$, $s_1=0$, $s_2=0$, $s_3=20$, and $z=1250$.

Here's a summary of the simplex algorithm to solve linear programs:

1. Rewrite the objective function and constraints into equations with slack variables.
2. Create the initial simplex tableau using the newly written equations.
3. Identify the most negative indicator to find the pivot variable.
4. Calculate quotients to find upper bounds on the pivot variable, and select the smallest quotient; this is the pivot for this iteration.
5. Using Gaussian elimination and row operations to turn all other values in the column to 0 using the pivot.
6. If any negative indicators remain after all row operations, repeat steps 3–5.
7. If no negative indicators remain, we are done and at the optimal solution$^*$!

While we only looked at 1- and 2-dimensional linear programs, remember that this can work for as many variables as you'd like, which is nice since I don't want to deal with imagining a ball rolling down a 12-dimensional tabletop to find my function minimum. There is a small asterisk though, since sometimes the simplex algorithm can "stall" or "cycle", resulting in no net improvements of the objective function. Fortunately other algorithms based on other concepts have been built to not just be quicker, but avoid degenerate cases stalling and cycling.

Sensitivity and Slack Analysis

What the values of $x$, $y$, and $z$ should be clear as those correspond to the point on the feasible region and maximum of the function, but what do the values of $s_1$, $s_2$, and $s_3$ mean? Recall that these are our slack variables as these are the variables that turned our constraining inequalities into equations by accounting for any slack in the constraints themselves. So, if we have 0 slack for one of our constraints, it means that we are using up as much of that constraint as we can; there is no slack to account for, and the corresponding slack variable is 0. If there is slack, it means we are not using up a constraint to its fullest potential. Recall our third constraint was

Also remember that our optimal solution included that $y=50$. We set a cap of $y=70$, but we are only using 50 of those possible 70 units. $s_3$ tells us that in the third constraint, we have an excess of 20 unused constraining units. In that same sense, $s_1$ and $s_2$ tell us we have 0 wasted resources for the first and second constraints. This also tells us that we are precisely at the vertex of where the first and second constraints graphically meet, since our solution is on the edge of both inequalities.

That's not the only information our slack variables tell us, though. You can also find how sensitive our result is in the final row of our tableau. The $\frac{50}{3}$ and $\frac{20}{3}$ tells us for every additional unit we add to constraints of $R_1$ or $R_2$, we will make an additional \$ $\frac{50}{3}$ or \$ $\frac{20}{3}$ respectively since $\frac{50}{3}(s_1 + 1) = \frac{50}{3}s_1 + \frac{50}{3}$. These are called shadow prices of our objective function. If you want to read more about shadow prices and other marginal analysis such as reduced costs, MIT OpenCourseWare has you covered, but for now there are a few cool extensions of linear programming I want to cover.

Duality and Dual Problems

Even though we have a systematic way to find our ideal solution to a linear program, we could have quickly found some facts about our solution before we started. Again, here is our previous, 2-variable linear program.

In our 1st constraint, we could have equivalently rewritten it as

by multiplying both sides by 60. Furthermore, since both $x$ and $y$ are greater than 0, we can also compare it to our objective function.

So we now have an upper bound on our objective function: we know for sure it has to be less than 2100. We can be even smarter about this and do a similar process with our second constraint. By multiplying both sides of the second constraint by 15, we can say

which gives us an even tighter upper bound on our optimal solution. This is the heart of duality: instead of trying to directly solve our maximization problem, we can turn it into an equivalent minimization problem to find the lowest upper bound of our objective function, indirectly solving it.

We can generalize this by multiplying all of our constraints by a scalar $a_i$ for each constraint and adding them all together.

Adding these all together, we get a unifying inequality of

Lastly, remember this is supposed to be an upper bound on our objective function, so we set this all greater than our objective function which I've highlighted in red. In our first few tries to bound the problem, we had $(a_1, a_2, a_3)$ equal $(60,0,0)$ and $(0,15,0)$ respectively. We can summarize our goal of minimizing the right-hand side while maintaining all these inequalities as true as

This looks like another linear program! We originally started with a primal problem with 2 variables and 3 constraints and used it to formulate its dual problem with 3 variables and 2 constraints!

But, what even is the purpose of this dual formulation? We already have a way to solve for an optimal solution, why needlessly copmlicate it with an extra intermediary step? The dual problem is useful as it allows us to readily access a lot of the sensitivity analysis. Each variable $a_i$ we're solving in the dual problem corresponds to the optimal shadow price (marginal utility of a resource) of the $i^\textrm{th}$ constraint in the primal problem.

Let's take a quick look back at our primal problem's solution. Recall that it had shadow prices of \$ $\frac{50}{3}$ for the first constraint of $\frac{1}{2}x + \frac{1}{5}y \leq \color{blue}{35}$, a shadow price of \$ $\frac{20}{3}$ for the second constraint of $x + y \leq \color{blue}{100}$, and a final shadow price of $0$ for the third constraint of $y \leq \color{blue}{70}$. If we value our total resources (highlighted in blue) at their marginal costs, we get that

which is precisely the optimal profit we got originally from the primal problem. The leading principle of a dual problem is that if we can solve for the optimal marginal profits of each resource, then we know implicitly how much of that item we should buy given the total selling price of each primal variable.

We try to minimize the cost of our primal constraints, while trying to ensure our dual constraints satisfy the coefficients of the primal objective function. In the case of shirt and hat selling, we want our marginal profits to at least equal the price we want to sell our products at. Even more interestingly, just like how our dual variables are equal to the primal shadow prices, the dual shadow prices are equal to the primal variables! Everything has been switched around!

Curiosities aside, we still haven't talked about many of the reasons why analyzing dual problems is useful. Here's a rundown of some of the benefits of duality:

1. Sometimes it's just easier: As you just saw, we turned a problem of 2 variables and 3 constraints into one of 3 variables and 2 constraints, and turned it from a question of maximization to one of minimization. For problems with few variables and lots of constraints, it's usually much easier to turn it into a problem with lots of variables and fewer constraints as every constraint in the problem will add some number of extra vertices for algorithms like the simplex to check, making it less efficient to check every case.
2. Feasibility and boundedness: Sometimes a linear program will have no solutions whatsoever to check (imagine constraints like $x \geq 0$ and $x \leq 0$; can't be both at the same time). If the primal is unbounded (think no non-negativity constraints), then the dual is infeasible, and vice versa.
3. Specialized algorithms and theorems: Beyond optimizing business plans, duality has found its way into combinatorics, graph theory, and even into the fame of game theory as a means to prove the minimax theorem for zero-sum games. Many more results like these tend to stem out of the Weak and Strong Duality Theorems (Weak Duality talks about how a dual problem can set an upper bound to a primal soultion, and Strong Duality says it can find the optimal primal solution).

Duality is a powerful concept not just in linear programming, but frequently pops up in other areas of math and recognizing when you can represent one problem with another is a great tool to have in your back pocket.

Conclusion

Linear programming is only one small niche of optimization study, but the depth and applicability in its simple premises is wildly effective. Starting out in the 40s with Dantzig's original simplex algorith, it has grown to affect much of computer science and math to systematically solve otherwise impossibly long computations. Even now, variations of the original linear programming formulation is still being researched with new methods to not only solve them, but also adding new fundamental restrictions to the problem. If you were a car manufacturer that wanted to know the optimal distribution of models to produce, you can't just make 41237.7963 cars; you would only care about integer solutions, and thus integer linear programming (ILP) was born. Linear programming's innate utility in optimization has lent itself kindly to modern applications, but from ILP, to the even crazier mixed-integer linear programming with a combination of integer and non-integer variables, as well as duality, LP has found itself touching every corner of math from combinatorics to graph theory as a simple multidimensional geometric encoding of a constraint function.

Today's post is one that's been months in the making. It originally started as one that only covers a single problem, but quickly branched off as I delved deep into dozens of papers and videos, just with more and more questions coming up. We're going to be discussing one of the oldest mixed studies of algebra and geometry: dynamical systems. Today's post is going to be a long one, but should have a fair number of fun visuals to keep it worth the scroll. This is really two, maybe three posts in one, so I recommend reading this with breaks at each header as to make it less overwhelming.

To begin, let's look at a type of problem you might have experienced before, rather than have read formally: billiard problems.

For the best experience, avoid reading this in Safari; most other browsers should work and load the visuals correctly, but Safari breaks rendering a few of them.

Ricochets and Rebounds

If you have ever played pool, or even laser tag, you might already be familiar with billiard problems. If you have a billiard ball you want to land in a pocket, but there are other balls in your way, where on the side of the pool table do you want to bounce your ball to land in the pocket? Alternatively, if you have a laser gun and an opponent by a mirror, where do you want to aim your laser to hit your opponent? To reduce this problem further (and for future reference), if you have a laser, a target, and a wall, where do you want to aim the laser to hit the target?

We covered how to solve this exact problem in a previous post, which also shows how light reflects and bounces (which is what we'll be using today). To recap that post:

• Our laser/billiards bounces must follow the Law of Reflection: the angle the laser strikes the mirror is the same angle it reflects.
• To solve for where to bounce the light off the wall, we create a "mirror world" to find the reflection point.

The idea is to reflect our target over the wall, draw a straight line from our laser to the reflected target, and the intersection point is the point of reflection. This seems arbitrary, but there's a good reason for it: the angle that our straight line creates in the "mirror world" is equal to the incident angle, and therefore ensures the angle the light bounces off at in the normal world is equal. I recommend following through with the previous post for a more on this, but the following demonstration should suffice.

Simple enough, but this reflection technique is an invaluable tool for solving these types of problems, so put a pin in that for later. But now, let's look at a problem that throws that very easy technique out of the window.

Alhazen's Circular Mirror

Dynamical systems have been studied since forever at this point. One of the oldest (hard) billiard problems comes from Ptolemy 150 AD:

If you have a candle and a circular mirror, where on the wall do you have to aim to hit a target?

This problem plagued the Greeks for centuries, primarily since they tried to solve this with their typical ruler-and-compass constructions. Dr. Peter Neumann proved that this is an impossible task, but other solutions and proofs have arisen over the years by some of the most famous mathematicians including Huygens and l'Hôpital. The problem was named after Abu Ali al Hassan ibn al Hassan ibn Alhaitham, later retroactively given the mononym Alhazen, who discussed this in book Optics around the 10th century.

First, let's clear some details up.

• To make sense of a curved mirror, the reflection acts according to the tangent line at the point of reflection. So, you can think of the curved mirror as made up of infinite, infinitesimal straight-walled mirrors.
• The light is a pure ray like a laser. No point source or beams to get cheap answers like, "If I stand about here, some of the light will reflect on the target." This is a laser beam that can only blind us if it directly hits us.
• In a similar manner to the light assumption, the target has been reduced a to 0-dimensional point. The light must hit the target exactly in our solution.
• And, importantly, we assume a solution exists. If the light and target are on opposite sides of the circle, clearly no reflection will make it. So, to simplify, we'll just assume that they are in positions with a possible reflection point.

Before I show you my solution to the problem, I suggest you try this problem for yourself. This is one of the few, very simple-to-state problems that has caused me a lot of trouble deciphering a clean solution for it. Mathematicians have developed quite the dictionary to solve this one problem which we'll discuss a bit at the end, but only one way has appeared the most elegant (in the most stretched definition) to me.

Clutch Complex Contours

The way I solved this problem was with the magnificence of complex numbers. Let's center our circular mirror as the unit circle $Ø$ at the origin $O$, and let's call the light and target $\color{red}{a}$ and $\color{green}{b}$ (if your circular mirror is not of radius 1, just scale all coordinates appropriately to make it so). We want to solve for the point $\color{purple}{z}$ on the mirror such that $\angle azO = \angle Ozb$.

The path our light takes can be modelled in two segments: from the candle $a$ to the mirror $z$ as $a-z$, and then from the mirror to the target as $b-z$ (you'll see why we pick these directions later). To make our lives easier, let's rotate our whole setup so that the mirror reflection point occurs at $z=1+0i$ by dividing our whole setup by $z$. So right now, we have two vectors representing our reflected ray of light as $\frac{a-z}{z}$ and $\frac{b-z}{z}$ (if you're not completely familiar with the geometry of complex numbers and why this division tactic works, here's a good introductory video).

Since our vectors are now symmetrical about the real axis, we know that $\arg(\frac{a-z}{z}) = -\arg(\frac{b-z}{z})$ as the angle of reflection must be equal angle to the angle of incidence (see this previous post for a proof).

Ok, so why is any of this helpful? Just as we divided by two complex numbers to subtract the angles of their vectors, we can multiply them to add them together. If we multiply $(\frac{a-z}{z})(\frac{b-z}{z})$, we get that their angles sum to 0 since their arguments are opposite! If the argument of the product is 0, that means that it lies on the real axis, and therefore is a real number! We can then extract that

where $\operatorname{Im}(c)$ denotes the imaginary part of a complex number $c$. Noting that $\color{purple}{z} \cdot \overline{\color{purple}{z}} = 1$ we can further simplify this equation.

This might not seem like much, this describes all possible points $\color{purple}{z}$ our reflection point can lie on! Let $\color{purple}{z} = x + yi$, $\color{red}{a} \color{green}{b} = p + qi$, and $\color{red}{a} + \color{green}{b} = r + si$, and we can rewrite our equation in terms of cartesian coordinates $(x,y)$.

This is an equation for a hyperbola! And since we want $\color{purple}{z}$ to lie on the circumference of our spherical mirror, the point $(x,y)$ must also be a solution to $x^2+y^2 = 1$ to lie on the unit circle as well. To find where our desired $\color{purple}{z}$ is, we just need to find the intersection between this hyperbola and the unit circle.

It's no coincidence that this problem involves finding the intersection between a circle and a hyperbola. While yes, all of our complex number algebra does the job, there's a purely geometrical way of coming to the same conclusion involving isogonal conjugates. I wasn't very familiar with them, so I decided to present the complex number approach instead.

You could try and solve this system of equations, but the nature of the conic sections make it a pretty tedious and gross task. Fortunately, we can actually reduce this sytem of 2 simultaneous equations to a single polynomial! Going back to one of our previous equations:

The important thing to note is that we have a complex number whose imaginary component is equal to 0. This means that this expression is equal to its conjugate, since there is no imaginary component to flip the sign of: $x + 0i = x - 0i = x$.

Again noting that $\color{purple}{z} \cdot \overline{\color{purple}{z}} = 1$, we can multiply both sides by $\color{purple}{z}^2$ to get that

All we need are the complex solutions $\color{purple}{z}$, and since this a quartic equation, we technically have a closed form solution. Once we have the coordinates of our reflection points, all we do is graph $(\operatorname{Re}(\color{purple}{z}), \operatorname{Im}(\color{purple}{z}))$, and we are done.

If these points look familiar, they should: they are precisely the points our hyperbola predicted before!

Also, notice how we only used information about where the supposed solution $\color{purple}{z}$ to find our quartic; we never specified any conditions for where the light $\color{red}{a}$ or target $\color{green}{b}$ had to be! This means we can have our light and target on the inside of our circular mirror and find points that satisfy the Law of Reflection.

The best part is, all 4 of the possible solutions our quartic and hyperbola find are valid! No need to worry about the laser clipping through the mirror randomly; it all works out.

While we solved the problem, there are a few details to address that are not completely obvious about this approach using complex numbers.

The Core 4

Why are there 4 "solutions" according to our polynomial? Being a quartic equation, 4 complex solutions isn't unexpected, but they don't seem to have any physical significance for our bouncing laser. Obviously, only one looks like it can reflect our points correctly; how can this be considered a viable spot to aim your laser?

No mirror bounces like that, let alone allow the laser to move straight through it. Moreover, our Law of Reflection looks completely broken, too. What's happening here?

It lies in the direction of our vectors. Watch what happens as I extend the line segment from the target to the "solution".

If we extend the ray, then it's clear that the Law of Reflection is satisfied, and this is true for the 3 other supposed "solutions": every "bad" reflection point is correct if the rays are extended far enough. So the 4 "solutions" correspond with how are vectors are lined up, since if we change the direction of our light bouncing we can get different points where the Law of Reflection is satisfied.

That doesn't answer, though, how we know which point to pick as the "correct" reflection point? If you read the previous post on retroreflectors, then you know light takes the fastest and (only in this case) shortest path. So, we just pick the point where the total distance of the light's path is minimized: $\min(|\color{purple}{z} - \color{red}{a}| + |\color{purple}{z} - \color{green}{b}|)$

To be (on the circle), or not to be (on the circle)

Some of you might be wondering why this should produce any solutions on the unit circle—let alone 4 at that. That is in part by the property we have mentioned a few times: $\color{purple}{z} \cdot \overline{\color{purple}{z}} = 1$. If the geometry of this isn't obvious to you with the rotations and scalings, we can turn this into Cartesian coordinates by setting $\color{purple}{z}=x+yi$, we get that

Which is precisely the equation for the unit circle. The property that $\color{purple}{z} \cdot \overline{\color{purple}{z}} = 1$ forces $z$ to be on the unit circle.

…for the most part. The proof I've highlighted is adapted from this paper. As it shows, if you have the light at $\color{red}{a}=.5+.5i$ and a target at $\color{green}{b}=.5+0i$, two of the supposed solutions for $\color{purple}{z}$ are completely off the unit circle.

I'm not totally sure why this happens, but the previously linked paper proves that at least two of the generated solutions must be on the unit circle.

If you want to look at this problem more, there is also an algebraic solution, here there are ideas involving tangent ellipses in this paper, and there's even an approach discussed in Dorrie's 100 Great Problems of Elementary Mathematics. These would be my recommended starting points. For even more depth, this is a solution involving origami (yes, the paper folding) and here's the same problem in hyperbolic space.

Now that we've seen where billiard problems started, let's see how far they've come with the main focus of today's post.

Get Down Mr. President!

You and an assassin are trapped in a square room. With a single bullet, the assassin wants to do everything he can to take you out without wasting his shot. You, however, came prepared and hired a bodyguard to prevent direct line of sight between you and the assassin. But remember, you're trapped in a room. Without hesitation, the assassin flicks a shot to the side and ricochets off the wall and grazes your arm, avoiding the bodyguard completely. You might have been lucky this time, but who knows what happens next.

If you (target) and an assassin are placed in a square room, can you hire a finite number of bodyguards to prevent any shot from hitting you (including ricochets)?

At first glance, this might seem absurd. There are an infinite number of ways for the assassin to line up and bounce his shot, so how can anything less than an infinite number of bodyguards suffice? As one might anticipate, this wouldn't be a blog post if it didn't have an incredible answer.

It's no coincidence I placed the bodyguards where I did in the above widget; not only does a finite number of bodyguards make do, you can prove you only need to hire 16 to ensure 100% protection!

I first found this problem through Tai-Danae Bradley's video and post, where she writes up the proof very well on her own. It was this problem that inspired me to look for other problems to extend this post, and moreover it was a fun programming challenge. Here, I want to outline the proof with the key insights Bradley utilizes, as well as pose a few other questions of my own.

Just as we did before, let's clarify some problem details:

• Just like before, the assassin's bullet is a pure ray, and the target has been reduced a to 0-dimensional point.
• Now, though, we have bodyguards, which are 0-dimensional points as well; if they are going to protect the target, they have to fully take the hit.
• Lastly, just to make it clear, this is a perfectly square room and the bullet ricochets at exact angles off the walls, so the assassin's shot can bounce forever if needed to hit its mark.

The Tiling Torus

Let's look at an easy case: what if the assassin can only reflect off the left wall? Since this is a square room with straight walls, we can use our reflection trick from before. Now, though, we have to analyze the target's position in space relative to the room.

I've color-coded the left, right, top, and bottom walls to be yellow, gray, magenta, and cyan respectively. The reason if we want to track the target's location within the room even after the reflection, we have to reflect the room itself too, creating an actual "mirror world" that I referred to earlier.

Ok, so that isn't too different than what we've already been doing, so what's the point? The magic lies in modelling multiple bounces. Remember, we reflected over the yellow wall to say we wanted our bounce to be off that wall, but we can chain these reflections to give multiple instructions to our bounces. If we first reflect over the yellow wall and then the magenta wall, we create a doubly mirrored world with our straight line showing the path of what 2 bounces looks like.

If you want to convince yourself this trick works for multiple bounces, I recommend finding the congruent angles within the mirrored world's straight line and the actual bounces within the room in question.

An important part of finding this mirrored world's straight line though is the fact that it intersects the colored walls in the order the assassin's bullet bounces. In the initial setup provided, the straight line hits the yellow wall first before intersecting the magenta wall, just as the beam's bounce path reflects off the yellow then magenta walls in that order. If you move the assassin and target, you can see this idea holds for a magenta then yellow wall bounce too.

So, if we wanted to model the assassin's hitting the target in more bounces, we just reflect our room more times and draw the straight line between the assassin and mirrored target.

Moreover, since squares can tile the plane maintain the same "silhouette" under reflection, we can infinitely tile the plane with reflected copies of our room. Since a line through this plane can represent any bounce shot from the assassin in the original room, we have successfully simplified our problem setup. Why? With straight lines, we can now use coordinate geometry to place our bodyguards and not worry about annoying reflectedl light patterns within our square.

In more math-y terms, we have turned our original room into what is known as a flat torus (yes, the thing that's equal to a coffee cup). Essentially, all this means is that our problem sort of exists in a world similar to the game of Asteroids: as you exit the top or left of one flat torus, you enter through the bottom or right of another one (and vice versa). This fact is what allows us to tile the plane consistently with our problem setup. If you look back at the 2-by-2 grid setup from before modelling the 2-bounce paths, you'll see that our top/bottom edges are both cyan and our left/right edges are both gray, showing that exact relationship we'd expect in a flat torus.

This is the first key insight to solving this problem: turning our bouncing shots into straight lines in an infinitely tiled plane. Working with straight lines makes life so much easier than bent ones. With that, we can move on to the second epiphany to prove our result.

The (Different) Core 4

Even though we tile the plane infinitely, there aren't an infinite type of rooms. Just looking at our 2-by-2 grid that makes our flat torus shows us everything we need to know: there are exactly 4 types of rooms that build our tiling: the original one, the one reflected over the yellow wall, the one reflected over the magenta wall, and the one reflected over both walls. This regularity is clear visually: watch what happens if I reflect the target into every mirrored room.

Each one of the reflected rooms generates a lattice of that reflected target! I've colored the 4 different lattices in green, yellow, magenta, and cyan. Now, since each dot represents a way of hitting our target, we just need to block every line from the assassin to any one of these colored dots.

This is the second critical idea to finish out this proof: every reflected target falls into 1 of 4 possible lattices (each represented by a color). Dividing the mirrored targets into lattices is nice since it places all dots in a given lattice to be the same distance away from each other.

At this point, you have everything you need to finish this proof using the flat torus tiling and the 4 lattices. If you want to try and finish it through, I recommend doing so as it has some pretty satisfying reasoning throughout it. If you just want to keep reading, I'd recommend visiting Bradley's post where she completes the proof there.

Once you reach the end of the proof, you'll find that you need exactly 4 bodyguards to protect any given lattice, and since there are 4 lattices, we need $4 \cdot 4 = 16$ bodyguards total to completely protect the target.

Since it is possible to protect the target from the assassin with a finite number of bodyguards, we can say that the square is a secure polygon.

What Next?

This is one of the most surprising facts I've come across in a long time. But, there's more places to take these billiard problems and dynamical systems. One of the first extensions I thought of upon seeing this was other grids. As we know, there are also hexagonal and triangular grids in addition to the square one we analyzed today.

Last post, we looked at different types of billiard problems, a class of math problems analyzing how light bounces with different setups of mirrors. Notably, we saw how straight lines make for very simple, easy to compute mirrors, while others like circular ones, can be incredibly frustrating.

A large portion of last post's content, though, was made up of interactive graphics. While I went over much of the math that goes into solving these types of problems, we skipped over a large part of the math that goes into simulating them. Math is very nice in that many problems can be solved with nothing more than a pen, paper, and your mind, but oftentimes, that's only helpful if you are confident in how to approach the problem. What computer's can do is help build our intuition to solve a problem by calculating, drawing, and modelling scenarios with precision and speed we can only wish to achieve.

So, today, we'll look at some of the clever math that goes into computer graphics (that we'll later extend), and to introduce such a topic, we'll look at a simple, fundamental problem in graphics: how do you find the intersection between a line and a circle?

Languishing Lines and Confounding Circles

Before we can even attempt this problem, we're going to have to start from scratch, since we have one slight issue: a computer has no idea what a line or a circle is! So before we can do anything, let's teach our computer how to draw a line.

Perfect Parameterizations

At its core, computer graphics is displaying a set of pixels with certain colors. If we want to visualize anything on a computer screen, we just need to find all the relevant pixels (coordinates) to light up and color. Because we want to compute these individual coordinates of, say, a line or circle very quickly and easily, almost always we will use vectors. These can be typical column or row vectors you see in linear algebra, or it can even take the form of complex numbers. The reason why these tend to be helpful is that they give very easy ways to compute coordinates for lines, circles, and other shapes.

If we want to draw a line with slope, say 2, we need to ensure that it is constructed by a vector of slope 2. An easy one to find is the vector $v=\small{\begin{bmatrix} 1 \\ 2 \end{bmatrix}}$ since we know that will pass through the point $(1,2)$. So, to get other points beyond this vector, we can scale $v$ by a factor of $t$ to get other vectors (i.e. points) with the same slope. If $t=2$, we get the point $(2,4)$. If $t=1.5$, we get the point $(1.5,3)$. If $t=239470$, we get the point $(239470,478940)$. Whatever you choose $t$ to be, our vector $v$ will give us a point on the line $y=2x$.

However, this isn't super helpful, since we are still only restricted to lines that go through the origin at $(0,0)$. So, we can add a starting point $\color{red}{p}$ to our vector equation to offset the line by $\color{red}{p}$, guaranteeing our line goes through the point $\color{red}{p}$ (since that's the coordinate generated by $t=0$).

Now we just plot every point for $t \in (-\infty, \infty)$, and we get a line with $v$ dictating the slope of our line (negative $t$ values gives us coordinates behind $\color{red}{p}$)!

We can do a similar process for a circle. To parameterize a circle, we'll have to pull from trigonometry. We know that a circle is defined by $x^2 + y^2 = r^2$. The Pythagorean identity tells us that $\cos^2(\theta) + \sin^2(\theta) = 1$, so we can quickly make the connection that $x=r\cos(\theta)$ and $y=r\sin(\theta)$ (which the geometry justifies). This precisely defines $x$ and $y$ in terms of the parameter $\theta$! Again, though, this is centered at the origin, so we can center the circle around a point $\color{blue}{q}$ by adding it to our parameterization.

where $r$ is some real number for the radius of the circle, and $\theta \in [0, 2\pi)$. We can now easily draw both lines and circles!

Collisions and Intersections

Now that we have defined our line and circle for our computer to interpret, we can start thinking about how to detect collisions between a line and a circle.

Discerning Distances

A good place to start is by looking at how far away the line $l$ is from the center of the circle $\color{blue}{q}$. For reference, the distance from a point to a line is the shortest (i.e. perpendicular) distance from the point to the line. If $l$ is more than a distance of $r$ away from $\color{blue}{q}$, then we know that it's outside the circle and doesn't intersect, and if $l$ is less than a distance $r$ away from $\color{blue}{q}$, then we know it's inside the circle and does intersect.

Let's look at an individual line and see if we can draw any useful conclusions about this distance.

From a given point $\color{red}{p}$ on our line $l$, we can find a new vector between $\color{red}{p}$ and the circle's center $\color{blue}{q}$ as $\overrightarrow{\color{blue}{q} - \color{red}{p}}$. This will form some angle $\theta$ with $l$, more specifically its vector $v$. Recalling that $\color{green}{d}$ is the perpendicular distance between $\color{blue}{q}$ and $l$, we have a right triangle that gives us that $\color{green}{d} = |\overrightarrow{\color{blue}{q} - \color{red}{p}}| \sin \theta$.

If you're familiar with your linear algebra, this almost looks like the formula for the magnitude of the cross product: $|v \times u| = |v||u|\sin \theta$. So, writing our two relevant vectors and rearranging we can see that…

So all we need to do to see if our line intersects our circle is if that cross product is less than or equal to the radius of our circle (if you're concerned about the dimensionality of our vectors—cross products only exist in dimensions 3 and 7—we can treat them as 3D vectors with z-component 0, which makes the calculation easier and equivalent to the determinant).

If this isn't totally apparent why this is true, it has to do with the geometrical interpretation for the cross product: we're finding the area of the parallelogram that the two vectors span, and since the area of a parallelogram is $A=\textrm{base}\cdot\textrm{height}$, we're essentially finding the height of that parallelogram by dividing by its base.

We have a working condition! Using the cross product, we can identify point-circle intersections with a single line of computation. However, this simple solution does have its limitations. Mainly, this is only a boolean condition; this method only tells us whether or not an intersection occurs, but nothing else. We don't know where on the line it intersects, nor how many times. Sometimes, this doesn't really matter like when you want to approximate lines intersecting points (since then you can treat points as small circles). But for more complex tasks and graphics like raytracing, this won't cut it.

Fancy Vector Operations

If we have a point $x$ on our circle, then the distance between $x$ and the center of the circle $\color{blue}{q}$ should be equal to the radius $r$. As an equation, the magnitude of the vector from $x$ to $\color{blue}{q}$ equals $r$.

Moreover, we want this point $x$ on our circle to also be on our line $l$. So, $x = \color{red}{p} + tv$ for some value of $t$. With this in mind, we can substitute $x$ in our previous equation.

Now, let's square both sides.

This may seem pointless, but it helps us rewrite that left side of the equation. Generally, working with the magnitude of a vector as an operator isn't super helpful, but we can quickly rewrite the square of the magnitude in terms of the dot product, since for any vector $v \cdot v = |v|^2$.

Expanding this out and collecting like terms gives us…

Which is just a quadratic equation in $t$! With coefficients…

…we can solve for $t$ using our trusted quadratic formula (note that $a$, $b$, and $c$ are all outputs of dot products, ensuring they are valid scalars to plug in).

Remember, $t$ is the scalar that tells us where on our line we are, so if there are real solutions to $t$, then we will have the exact intersection points for our line and the circle!

We can analyze this quadratic like any other to give us insight into our intersection points. Specifically, using the discriminant. When $b^2 - 4ac > 0$, then we get two solutions/intersection points. If $b^2 - 4ac < 0$, then we get no real solutions and therefore no intersection points. Finally, if $b^2 - 4ac = 0$, then we have exactly one intersection point, and can conclude our line is tangent to the circle.

Also, this quadratic can straight up replace our closest-distance method from before, since the point at which our line is closest to the circle corresponds to the vertex of the parabola at $t=\frac{-b}{2a}$.

Not to mention, notice how everything we did here was independent of the fact our line and circle exist in two dimensions; we can easily use this for 3D graphics, and even higher dimensions as well to find the intersections between lines and hyperspheres! Below is a raytraced scene I drew of 3 balls using this exact quadratic to compute lighting with shadows and reflections (a.k.a. my formal application to Pixar).

And to think that we'd never use the quadratic formula in real life.

I Don't Know Where Else to Put This

Before I end off this post, I want to include some other interesting circle facts since I don't know where else to put them.

Squaring the Circle (Bounces)

If you have a ray of light start from the circumference of the circle, after a total of $n$ reflections within the circle, the sum of all the angles of reflection will be $n^2$ times the initial angle.

Between this and the Basel problem, circles and squares are just weirdly intertwined. The reason this particular statement is true is because of how much the angle with the horizontal increases after a single bounce. If your light starts at an angle $\alpha$, we can show that every additional bounce will add $2\alpha$ to the angle with respect to the horizontal.

With the help of some auxiliary lines, I hope the above picture makes this clear. Then by symmetry, of the circle, we can see that each subsequent bounce will also add $2\alpha$ to the angle. Moreover, since our initial angle itself is $\alpha$, every bounce will just be the odd multiples of $\alpha$ (since odd numbers can be thought of as a multiple of 2 plus 1, which is precisely what our angle bounces mimic)! So, for a series of $n$ bounces, the sum of the angles of each reflection is equal to

(Yes I am aware there is a formula for an arithmetic sequence with with any initial term but this is how I remember to solve them okay) I didn't know how to fit it in last post with the mention of circular mirrors there, but here seems like a good spot to mention it.

Perpendicular Parabolas

The set of intersection points between two orthogonal parabolas lie on a common circle.

To show this is true, we just need to crank out the algebra. To find our intersection points, we need to solve the system of equations

If these individual equations are true for our intersection points, then so is their sum.

While that last line may seem a bit unruly, note that $\color{red}{x_1}$, $\color{red}{y_1}$, $\color{blue}{x_2}$, and $\color{blue}{y_2}$ are all constants, so the right-hand side of that last equation can be summarized as one big constant.

That's precisely the equation of a circle with a center at $(\color{red}{x_1} + \frac{1}{2}, \color{blue}{y_2} + \frac{1}{2})$ and radius $\sqrt{C}$, and that's exactly what is plotted above.

I have a few more circle tidbits to share, but they have more to expand on in their own posts for another day.

Until then, hopefully you found this slight detour into the world of graphics interesting. There are (as you could imagine) a lot more to graphics I want to share. From image homography, to video textures, to even a more in-depth look into raytracing and rasterization, but we'll save those for later.

First, let's take a loot at the integration by parts shortcut.

IBP, but You Forgot +C

This is by far my new favorite trick to pull out of my back pocket whenever I can. It leverages the fact of the innate products built into integration by parts, and the nature of antiderivatives. Though, I'm sure some of us could use a refresher on integration by parts.

To Undo the Product Rule

When given the product of two functions $f(x)g(x)$, the standard formula to compute its derivative is

Now if you integrate both sides and rearrange a little bit we can conclude that

Let $u = f(x)$ and $v = g(x)$ to get that

…which is precisely the integration by parts formula we've come to know. It really just is the opposite of the product rule. The main reason why it's such a useful technique is because if you have a function that's really hard or you don't know how to integrate, you can use it as your function $u$ and express its integral purely in terms of its derivative. Let's try this with an example:

This doesn't seem like a particularly product-y integral that can leverage integration by parts, but it really is!

Since we don't want to deal with integrating $\ln x$ (besides, that's what we're trying to find anyway), we can set $u = \ln x$ and $dv = 1 \ dx$. Then working it through we get that

That's pretty neat! We were able to reduce a relatively hard integral into one that was much simpler by thinking of it in terms of what product of functions when differentiated would include our original integral.

However, this doesn't always work by itself, and this is where our integral trick comes into play.

Now, Remember Your +C

Let's try a very similar integral from before:

This doesn't look too bad, right? It's basically the same as before. Let's try the same choice of $u = \ln(x+1)$ and $dv = 1 \ dx$ again and see where it takes us.

Aaand there's our problem. Our supposed simplified integral of $\int v \ du$ ended up with something also annoying: $\int x \cdot \frac{1}{x+1} \ dx$. This isn't too bad if you're okay with polynomial division (with this one being relatively easy, too), but it isn't necessarily trivial. Since we want to avoid doing more work, we can do much better by realizing an overlooked aspect of integration by parts.

We rewrote our original integral in the form of $\int u \ dv$, and later found an antiderivative $v$ from that differential. In our case, we let $dv = 1 \ dx$ and deduced that $v = x$ by undoing the power rule of differentiation. This isn't wrong, per se, but it is incomplete. The antiderivative of $1\ dx$ isn't $x$, but $x \ \mathbf{+ \ C}$. Since, remember, the derivative of any constant goes to 0, we can add whatever constant we want to the end of our antiderivative and it'll still remain valid.

So how can this help us? Well, let's do the same integral with the same choice of $u$ and $dv$, but instead of letting $v = x$, let's make $v = x+1$.

Look at that! That last, previously annoying integral, has become much simpler! Instead of getting two polynomials dividing each other, our new choice of $v$ reduces it to $\int (x+1) \cdot \frac{1}{x+1} \ dx = \int 1 \ dx = x + C$. So, finally, we can conclude that

In fact, for any choice of a constant $\alpha$, we can see that

It's such a simple trick, but an important reminder to remember the basics and fundamentals when attempting a problem. Besides, $+C$ being more than a formality is at least a little bit satisfying.

Polynomial Predictions

The last integral trick is a bit niche—it relies on that second integral when doing IBP to give a quotient two polynomials of matching degrees. This next shortcut, though, uses this idea of polynomial degree a bit more cleverly, but does not always work. When it does, though, it's certainly satisfying.

Let's find the following antiderivative:

This doesn't look particularly friendly, but we can make some observations about this function. The numerator of our function is a cubic, or a polynomial of degree 3. Similarly, our denominator is the square root of a quadratic, or polynomial degree 2. In the very loose sense of "degree" we can say that asymptotically, the denominator is closer to a degree 1 or linear polynomial (yes, I know that that $\sqrt{x^2} = |x| \neq x$, but just play along for now). So, if we were to carry out all of the polynomial long division, we'd expect our original function to behave like a degree $3 - \frac{2}{2} = 2$ polynomial.

Ok, so what? With basic integration, antidifferentiating a polynomial increases its degree by one. This is just the power rule.

This fact implies that if our polynomial is loosely of "degree" 2, then integrating it should give us a function of degree $2+1=3$. So, let's make a guess at what our integral might look like.

This guess should look somewhat reasonable, since we have a quadratic multiplied by square root of another quadratic, which we loosely said was degree 1. And a polynomial of degree 2 multiplied by a polynomial of degree 1 gives us a polynomial of degree 3, which we wanted. However, you might wonder why we even wrote this as a product; why not just write this out as a pure cubic of $ax^3 + bx^2 + cx + d$? The main reason is expecting the chain rule of some kind to occur. When composing functions, the derivative—and therefore the integral—tend to include the structure of these compositions, so it's not unreasonable to make a guess with the denominator in the result.

Now here's the trick: let's differentiate both sides.

If we simplify this expression and expand the right side…

For this last equation to hold, we need the coefficients to match.

Therefore,

Putting it all together, we can go back to our original guess of the antiderivative and find that

There we go! A succcessful antiderivative found.

This is trick is a great first attempt at integrating rational functions, but it is also extremely sensitive to minute changes in the integrand. For example, if we change our integral to

our algorithm breaks. It's the cost associated with what makes this algorithm so convenient: we don't touch the numerator of the integrand at all. Our antiderivative guess only depended on the denominator, and as a result, the coefficients we tried to match at the end had no intrinsic tie to the numerator and thus polynomial we were matching.

This integral shortcut's convenience is definitely a double-edged sword, but the method behind making these educated guesses is a useful idea in its own right to take away. For more on this type of integration, I recommend reading up on the Risch algorithm, a standard in computing indefinite integrals. Here's also a very thorough synopsis on evaluating integrals on the Wolfram Blog.

Feynman's At It Again

This last integration stratagem comes from none other than the celebrated Richard Feynman of physics fame, and thus has been aptly coined as Feynman's Intregral Trick.

…as it has been popularized. What I'm about to show has historically been known as the Leibniz Integral Rule, or differentiation under the integral sign. Not quite the same ring to it, but nonetheless good to know for accuracy.

Let's try the following integral:

What we're about to do might seem insane, but it will be immensely helpful in a second. What we're going to do is generalize this integral. Let

We've replaced the exponent of 2 with a $t$. So, in our new, generalized problem, we want to find $f(2)$. A useful fact also to note is that we know some values of $f(t)$. For example, we know that $f(0)=0$. How does this help? Well, now we can what the name of this trick alludes to—moreso outright says: we'll differentiate under the integral sign. Let's take the derivative of $f(t)$ with respect to $t$.

That last integral is super easy, only reversing the power rule to calculate.

Now that we know the derivative of $f(t)$, we can now integrate this simpler function in terms of known values and use the Fundamental Theorem of Calculus to find $f(2)$.

Note how I used the Fundamental Theorem of Calculus with clever bounds for our integral. You could instead solve the differential equation generally, but the FTC skips shortcuts a few steps. So, after all of that, we can conclude

As counterintuitive as it may seem, solving a general problem can sometimes actually be easier to solve than its individual cases. The best part about this technique, though, is that we haven't just solved one integral, but a whole family of integrals. For any exponent $\alpha$, we can conclude that

You might have noticed something different with this integral compared to our previous approaches: this applies to definite integrals as opposed to indefinite integrals (or antiderivatives). Namely, because of the fact we have to integrate not once but twice in this method. So, at the following step,

if this was not a definite integral, we would end up with a $+C$ attached to the end of it that we would not be able to solve for.

Here's another application of Feynman's trick:

Knowing Feynman's trick wins you the battle, but knowing how to use it wins you the war. Many times, you have to be creative in your choice of parameter when wanting to differentiate under the integral sign, so don't be discouraged if it doesn't work the first time. For this particular integral, we'll want to consider

Now, we want to find $f(1)$, and we know that $f(0)=0$. Now let's differentiate both sides with respect to $t$.

Decomposing that last integral into its partial fractions yields

Now, with more elementary calculus, we can evaluate that integral.

Now, we want $f(1)$, and know that $f(0)=0$, so let's integrate this function of $t$ from 0 to 1.

Again, almost magically, by generalizing a hard integral, it became a much easier one to tackle.

To give you an idea how powerful this technique is, the above integral comes from the 2005 Putnam Exam. Not only does it come from one of the most difficult math tests, it's also the 5th problem of the first set of problems (with problem 1 being the "easiest" and 6 being near impossible). And, in only a few lines, Feynman and Leibniz had it beat.

Conclusion

These are the three most recent integration techniques I have picked up and tucked away in my problem solving toolbox, but if you're interested in more advanced integral shortcuts and tricks, take a look at this MathStackExchange post I came across while doing this write-up. There are some genuinely mesmerizing ideas showcased there that just are out of the scope of my ability to explain, so do browse the forum if you're interested.

But I have to admit, I sort of lied to you. While, yes, that image does use the quadratic formula millions of times, it doesn't only do that. To render shadows and reflections, the scene also had to compute lighting and the physics you'd expect with mirror-like objects. Without any of this, our scene would just look like, well, uh, this:

Whichever one you think is better looking is up to the eye of the beholder, but what can't be argued, is that the second image is much cheaper to render; I'm sure you could guess, no shadows and reflections causes the scene to be rendered in a fifth of the time. A fifth.

Quizzing Quotients

Intuitively, more stuff to compute should take a computer a longer amount of time to go through, but can we pinpoint this bottleneck? Let's quickly look at what it takes to compute some of these reflections. When light bounces off, say, a mirror, these calculations become much easier when we use the mirror or surface's normal vector: the vector perpendicular to the surface (or the point at the surface) in question.

The above formula for reflecting a ray works in general for reflecting any ray $\vec{R}$ over another vector $\vec{N}$ (even if they're not normal)… Under a small assumption: the vector $\vec{N}$ is normalized (yes, the naming scheme isn't ideal), or of unit length (denoted by a little hat $\hat{N}$). We can do this by just scaling the vector down by its own length:

Recalling that the length of a vector 3D $\lVert N \rVert = \lVert <x,y,z> \rVert = \sqrt{x^2 + y^2 + z^2}$, we end up with

And here lies our bottleneck. While we, as humans, treat division not too differently from multiplication in theory, computers can't work with "just in theory"; computers have to actually compute this arithmetic somehow. It turns out, while multiplication is a bit more complicated than addition, we've been able to make algorithms for decades to accelerate the computation. Division, on the other hand, has been such a difficult endeavor to match other operations speed, major companies like Intel have lots of research dedicated to this alone.

So, what do we do?

The Fast Inverse Square Root

Under pressure, people can do some amazing things. You can imagine if someone was making a game or anything that required lots of lighting calculations, say, in a video game, calculating $\frac{1}{\sqrt{x}}$ millions of times, therefore also computing millions of divisions won't really cut it.

The developers of the video game Quaker III, an incredibly fast-paced shooter that definitely needed these optimizations, used a now infamous algorithm aptly called the fast inverse square root, because, well, it computes the inverse square root $\frac{1}{\sqrt{x}}$, fast and avoids dreaded division. The history of the algorithm has been found to predate the game that made it so infamous, but pop culture assigns value to whatever it latches onto first. Without further ado, the original source code (along with all the original comments and annotations) for Quake III was released in 2005, and the program is right there for us to learn from:

float Q_rsqrt( float number )
{
      long i;
      float x2, y;
      const float threehalfs = 1.5F;

      x2 = number * 0.5F;
      y  = number;
      i  = * ( long * ) &y;                     // evil floating point bit level hacking
      i  = 0x5f3759df - ( i >> 1 );             // what the fuck? 
      y  = * ( float * ) &i;
      y  = y * ( threehalfs - ( x2 * y * y ) ); // 1st iteration
//    y  = y * ( threehalfs - ( x2 * y * y ) ); // 2nd iteration, this can be removed

      return y;
}

It's not a long algorithm by any means, but I think the comments themselves explain just how crazy this is; even the developers who USED it are impressed, but let's break it down line by line.

      long i;
      float x2, y;
      const float threehalfs = 1.5F;

Here we define four different numbers: i, x2, y, and threehalfs. But not all of these numbers are treated the same.

Binary and Floating Point

In our day-to-day routine, we (at least, most of us) use base 10, decimal, to represent our numbers. What this means is that each digit in a number corresponds to some power of 10 we add together. For example, the number 1409, can be grouped as

with 1 thousands, 4 hundred, 0 tens, and 9 ones. You add these all together to get $1(10^3) + 4(10^2) + 0(10^1) + 9(10^0) = 1409$. This may seem obvious, but this is a really important idea in how we write numbers. Each digit represents a condensed shorthand for how many of a specific power of 10 is in our number. Computers do it similarly, but instead of base 10, they use base 2, or binary. If we wanted to represent 1409 in binary, we'd have

Now if you went and added these together you could verify that $2^{10} + 2^{8} + 2^{7} + 2^{0} = 1409$. Now each digit—or decimal-igit—represents a power of 2. We call these binary-igits bits. That first line of code that defines a long i means we define a number with 32 bits that looks like

With 32 bits, we can write any number from 0 to $2^{32} - 1 = 4294967295$. Let's notice some nice properties with this format. In decimal, if we wanted to add a 0 to the end of our number like $1409 \rightarrow 14090$, this is the same as multiplying our number by 10, because now every digit has moved up into one bucket higher than before.

In the same way, we can remove zeroes on the right $14090 \rightarrow 1409$ by dividing by 10 since every digit will be shifted into one power lower. Binary works the same. If we want to add a 0 to the right end of our number, we now multiply our number by 2, but if we wanted to remove a 0, we divide by 2. This is known as bit shifting, and serves as one of the nice workarounds of division: if you want to divide specifically by a power of 2, just bit shift the number in binary by however many zeroes you need.

But that leads to an issue: only even numbers can be wholly divided by 2, so what do we do if we want to divide an odd number? How would we write a decimal like .5? How would we write any rational number? We currently can only represent 32-bit integers, since we have no way of writing fractional parts. If we wanted to add decimals, why don't we just throw in a decimal point then?

(Remember this decimal point is just for our convenience; the computer doesn't actually see anything here but the 32 bits) If we use the left 16 bits for the integer part, and the right 16 bits for the decimal, we can now in fact write rational numbers, but all of a sudden the range of numbers we can represent has shrunk immensely, only to just under 65535.9999847, but the upside is now we can do much more precise, rational numbers. This is okay, but we can do better.

If you have ever taken a chemistry or physics class, you're probably all too familiar with scientific notation. We can write really big numbers and really small numbers in a much more condensed way by turning everything into a product of a power of ten:

This is how our calculators can do arithmetic beyond just the digits shown on the screen. By slapping on a power of 10, we can now represent a much wider range of values using the same number of digits previously. Similarly, binary works just the same except with powers of 2.

So, let's do just that. Let's allocate 8 bits of our previous 32 to representing the exponent, and another 23 to our actual number.

With 8 bits for the exponent, we can represent anything from 0 to 255, but we also want negative exponents, so we just subtract 127 to get the new range of exponents from -127 to 128. With our fractional number and its 1 whole bit and 22 decimal bits, we can represent numbers from 0 to 1.999999761581. We call this part of the number the mantissa. However, this is actually not the full extent of the potential precision we can get. In all of our examples of scientific notation, there was always a non-zero number before the decimal point, since if there was a leading 0, that's another power of our exponent we could factor out. In binary, there's only two values: 0 and 1. If we know the first digit is non-zero, then we know it has to be a 1! So we can actually shift our decimal point over and gain an extra bit.

Now all we have to do is affix a leading 1 and we're good to go. So, the number

would represent $1\color{blue}{.453125} \cdot \color{green}{2^{202-127}}$. In general, if we're given a 23 bit number representing the mantissa and an 8 bit number representing the exponent, we can write our number we're expressing as:

We divide our mantissa by $2^{23}$ so that it is only the fractional part of our number like we want. However, doesn't part of this just feel wrong? Like, when we were defining an integer as a 32 bit long, we used every bit to denote a new power of 2. Here, we really are writing two different binary numbers side-by-side. If we wrote our number as

would it really make a difference? So, just for the sake of consistency, we'll put the exponent first, and we can then represent our number's bit representation as a sum:

We just added 23 filler zeroes to our exponent to make sure it landed where we wanted to in the final bit representation. That sounds like a bit shift! We can thus multiply our exponent by $2^{23}$ to give us our 23 extra zeroes So, this final sum—our exponent and mantissa together—can be written as

Now, there are some flaws that do need to be addressed. If we assume our leading bit is non-zero, how _do_ we represent 0? That actually doesn't matter in the grand scheme of our intended use in lighting (i.e. we only call the fast inverse square root when we have to use it), and when we're not, this is a single edge case that can be put in later. Yeah, I'll admit, it's not ideal, but it gives us more precision where we need it. The other issue is we haven't used all of our 32 bits! $8+23=31$, so where did our last bit go? In our set-up, we have it such that we can only represent positive numbers. We can attach an additional sign bit at the front, where if it's a 0, we say the number is positive, and if it's a 1, the number is negative.

However, we want to know how to find positive square roots, not enter the complex world, so we always just assume it's positive. So if we don't use the bit, why don't we repurpose it? Conventions. The standard for binary fractional-part representation and arithmetic is known as IEEE 754, and for that reason we just have to abide by it.

I've been calling this with terms like "decimal points" and "fractional-parts", but a decimal point seems wrong when we're doing it in binary. The type of number we've just formatted is called a floating point or a float as we see in lines 2 and 3. While floats give us nice ways to represent a lot of numbers, they are a bit annoying compared to longs in the sense that we can't bit shift or manipulate a float since the bits in a float represent multiple, different parts of the number in question, namely the exponent and the mantissa.

Bit Approximations… With Themselves?

This may seem like a gross, unnecessary dive into how computers understand numbers, but understanding what binary, bits, and floats are will help us greatly in understanding the ingenuity behind the fast inverse square root. To recap, we've found that we can represent a binary number as a float with two parts, an exponent and a mantissa, as if we were using scientific notation. To find the actual number our float represents, we use the formula

Since we're working with two different binary numbers together, we combine them into one sequence of bits that to as a shorthand to represent our float with the formula

We can now perform some mathematical magic. Let's take the logarithm of the actual number of our float (note by $\log(x)$, we assume it to be $\log_2(x)$ since we're working in binary).

This may not seem that useful, but there's an important detail here: we're looking to optimize a program, not get exact results. So, a useful fact to note is that for $x$ between 0 and 1, $\log_2(1+x)\approx x$.

We can get an even better approximation by slightly offseting our estimate; $\log_2(1+x)\approx x + \delta$ is a better approximation than $\log_2(1+x)\approx x$

It turns out the best value for $\delta = 0.0430357$ (as in minimizing the average error). By definition, our mantissa is between 0 and 1, so we can use this approximation ourselves.

If we rearrange this a bit,

Okay, why did we do any of this? This definitely is kinda random to not only take the $\log$ of our float, but also do all these approximations to then get rid of that $\log$ too? Why?

Look inside the parantheses in the above equation.

That's precisely the bit representation of our float! So, in a way, the $\log$ of our number is equal to the bit representation of our float, up to some scaling and shifting.

With this under our belt, we can finally start looking at the steps of the fast inverse square root algorithm.

Evil Floating Point Bit Level Hacking

First, we assign our number we want to find the inverse square root of into a float (a.k.a. scientific notation-type decimal number).

      y  = number;

Now, recall that a float isn't that compatible with bit shifting or that many operations, so here's the first clever part of the algorithm.

      i  = * ( long * ) &y;

What this does is we take the exact bits of our number as a float and copies it into a long. That's it. Under the hood, it takes the number at the memory address of y and exactly transfers the bits over to i. This will make our life easier here in the next step.

Since we have now put our number that we're trying to find the inverse square root to, y, as its bit representation, we have effectively stored approximately $\log({y})$ into i.

What the F#@k?

The fabled step that makes this algorithm so smart.

      i  = 0x5f3759df - ( i >> 1 );

Remember, at the end of all of this, we want to find a number, $\alpha = \frac{1}{\sqrt{{y}}}$, but we have been working almost exclusively in logarithms. So, let's take the $\log$ of both sides.

Wait, but we have a division in there! On quite the contrary, it's a division by 2, and since i is a long, we can just bit shift to the right 1 to divide by 2! That's precisely what i >> 1 does: it bit shifts i once to the right.

But what is the deal with that 0x5f3759df? Well, remember that i is only an approximation for the $\log(y)$ up to some constants. So, we have to account for those constants somehow. Let's go back to $\alpha$. We know that

In terms of floats…

Fortunately we already know how to expand this from before.

This looks pretty bad, but after some simplifying and rearranging…

We know that anything of the form $E\cdot 2^{23} + M$ is just the bit representation of the number, and we know the bits of $y$ is just i, so

That magic constant 0x5f3759df is the hexadecimal (not totally sure why there is so many changes of bases) approximation of that constant $\frac{3}{2}2^{23}(127 - \delta)$. So what we do in this line of code is we bit shift i once to the right to halve it, and take that result and subtract it from 0x5f3759df to correct for all the constants that came with our approximations of $\log(y)$. Not totally sure why the developers felt the need to write a variable for threehalfs and not this number, but what can we do.

But now note we are storing this value in i. So, from here on i no longer refers to the bits of $y$, but the bits of $\alpha$, our desired number. The bits, though, aren't particularly helpful since we want the float and decimal representation of $\alpha$, so we do just that:

      y  = * ( float * ) &i;

Just like how we casted the bits of a float $y$ into a long i, we now do the reverse and cast the bits of i into a float $y$.

At this point, we're technically done: $y$ currently stores an approximation of $\frac{1}{\sqrt{\texttt{number}}}$, using 0 steps of slow division! But we can do better for a marginal amount of extra computation.

1st Iteration

Say we wanted to solve for the zeroes of the function

where $C$ is any arbitrary constant. Solving for $y$…

If we could find a way to approximate the roots of this function, we'd then in turn have a way to approximate the inverse square root of any number!

In a previous post, we discussed a technique to precisely do that: the Newton-Raphson Method (sometimes just called Newton's Method).

Let's say we have a random function $g(x)$. To find a solution, what can we do? Well, not a good idea, but an idea, we could just guess a random number $x_0$ as a solution. If $x_0$ is a solution, then obviously $g(x_0)=0$.

As you'd imagine, the chances of guessing a root of $g(x)$ immediately is slim. So, the next step in Newton's Method tells us to draw the tangent line at our first guess $(x_0, g(x_0))$ to get our next guess $x_1$.

Now we're getting pretty close. That's the whole premise of the Newton-Raphson Method:

1. Pick an initial guess $x_n$
2. Draw the tangent line at $(x_n, g(x_n))$ and find where it intersects the $x$-axis
3. Use that as your new guess $x_{n+1}$
4. Repeat steps 1–3 as needed

So, if we do another iteration of our example above…

There are some edge cases though where this obviously won't work, such as if our guess happens to hit an extremum.

We could even get loops where we just continuously bounce back and forth between two guesses. Fortunately, we don't have to worry about that. If our first guess is already really accurate and near the actual solution, then our graph $g(x)$ begins to look like this:

$g(x)$ starts to look like a line! And when a function locally looks like a line, it also locally looks like its tangent line.

This is important to us since we already have a good estimate from all of our bit manipulation from earlier, so we do one iteration of Newton's method to get an even better approximation.

To put this in terms of some equations to compute, we want to estimate the root of

Given an initial guess $y_n$, our next guess $y_{n+1}$ is the solution to

since this describes where our tangent line generates our next solution. Solving for $y$ we get that

Now it's just a matter of plugging everything in.

With a small substitution of $x_2 = \frac{C}{2}$,

If we look at the line of code that entails this "1st iteration",

      y  = y * ( threehalfs - ( x2 * y * y ) );

That's precisely the formula they have right there. You might wonder if that $\frac{3}{2}$ poses an issue at all in terms of division, but it is of no concern since we know its decimal expansion to be 1.5 so we can just use floating point arithmetic from the start; division becomes an increasingly hard problem when we don't know what the decimal representation of the quotient in question is.

Conclusion

Let's quickly recap what we've learned about the fast inverse square root algorithm and how it works:

float Q_rsqrt( float number )
{
      long i;
      float x2, y;
      const float threehalfs = 1.5F;

      x2 = number * 0.5F;
      y  = number;
      i  = * ( long * ) &y;                     // evil floating point bit level hacking
      i  = 0x5f3759df - ( i >> 1 );             // what the fuck? 
      y  = * ( float * ) &i;
      y  = y * ( threehalfs - ( x2 * y * y ) ); // 1st iteration
//    y  = y * ( threehalfs - ( x2 * y * y ) ); // 2nd iteration, this can be removed

      return y;
}

1. We first take our number as a float $y$ and store its bits in a long i.
2. Noting that $\log_2(y) \approx \texttt{i}$, we approximate $\frac{1}{\sqrt{y}}$ using bit shifting and the magic number 0x5f3759df.
3. We then perform one iteration of Newton's method to get an even better approximation.

These three steps required a fair bit of knowledge to properly unpack, but it's incredibly insightful when thinking about the lenghts programmers go to optimize their code. Remember: this is all to avoid using any division! What we consider such a simple operation is almost never such for a computer, and when it comes to teaching computers to do these things, they are as blind as a bat. But it's this difficulty and blank slate of a circuit board that makes computers be able to teach us, just as much as we teach them.

Something about the extreme wide angle capturing more than what your eye can behold all at once makes it truly magical. And, if you have an Android phone, you can get even more incredible types of otherwise impossible photos with Google's staple Photo Spheres:

While these are cool, it does beg the question: how are these made? It's not like your phone is able to take them in a single snapshot; usually you have to spin or take multiple photos. How is your phone able to tie multiple images together into a single cohesive one? We'll explore a little bit of linear algebra to manipulate our photos exactly how we want to, and indulge in stylistic photos your others can only dream of.

The Problem

Before anything else, let's first define what a panorama is. In computer graphics, a panorama is a type of mosaic, that is, a unification of two or more images. A panorama in particular, though, is a mosaic in which all the photos to be stitched together are all taken from the same camera position. When you take a panorama on your phone, all you're doing is spinning in place, so that's what we want to recreate.

So, if we're given two images from different angles,

how do we combine them into a single unified image? Thinking about it this way is not super helpful, since, well, that's what we already know about panoramas. What do we want to get out of a panorama? If our panorama is done well, then objects in one image should overlap properly with the same objects in the other image.

For example, take a look at the top left of my computer monitor:

In our panorama, these two points should overlap, since we obviously recognize these to be the same object in the real world. But to the two pictures, they are wildly different! In the first picture, the corner of my monitor is closer to the left side of the frame, while in the second picture, it is almost on the top-right edge of the image. So, if we try to manually align these two corners so that they overlap, we get:

While our single corner of the monitor is aligned between the two images, I don't think I have to try very hard to convince you that this isn't a great panorama. I mean, just look at the rest of the overlap.

The skew angles of the resting laptop and the monitor itself don't align at all, and while my cable management is bad, it's not that bad. This is the real challenge at the heart of making panoramas: images are flat, while the motion of a panorama is cylindrical. Ideally we would take a "cylindrical" photo and unwrap that into a rectangle, but we can't. Undoubtedly, we will have to warp our images somehow to align.

How do we find the right way to warp our image? The first thing we will need to do is get more data! Having one point line up between the photos is not great, but, say, 10 different data points might not be bad.

So, our final panorama would like the same numbered red and blue points to overlap. With data to use, the second thing we will need is a way to actually warp our image; how do we actually make our points between photos line up? For that, we turn to linear algebra.

Rethinking Coordinates

If you had a random ordinary point, how might you describe its position to someone?

A common choice we're all familiar with in some way is by using a coordinate system. That is, we define a place to be $(0,0)$ and locate every point relative to that origin in terms of its $x$ and $y$ coordinates $(x,y)$.

In the above coordiante space choice, we might say the point is at $(3,2)$. But what exactly do we mean by the point being at $(3,2)$? What this really implies is that the point is 3 steps to the right of the origin, and 2 steps above the origin.

So, instead of thinking of this point in terms of separate coordinates, we can think of it in terms of these two basis vectors. Let's use $\color{blue}{i}$ to represent the blue, horizontal vector, and $\color{red}{j}$ to represent the red, vertical vector. So, our point is really the combination of $\color{blue}{3i}$ and $\color{red}{2j}$, or simply, $\color{blue}{3i} + \color{red}{2j}$, which itself repsents another vector (the one pointing from the origin to the point $(3,2)$).

This might seem extra and unnecessary, since we just rewrote a vector as the sum of its horizontal and vertical components, which is what coordinates literally do in the first place. But the useful insight here is that there is nothing that says our basis vectors have to be in the unit directions! We can now rewrite points in multiple ways depending on our choice of $\color{blue}{i}$ and $\color{red}{j}$.

With a new choice of basis vectors, the vector $\color{blue}{3i} + \color{red}{2j}$ has a totally new position as that now encodes the coordinate $(5,3)$ since neither $\color{blue}{i}$ nor $\color{red}{j}$ represents horizontal or vertical steps anymore, but rather skew, diagonal steps.

But look at that! We've basically accomplished our goal of warping points! We've managed to transform the point $(3,2) \rightarrow (5,3)$ by manipulating $\color{blue}{i}$ and $\color{red}{j}$; both points are techincally at $\color{blue}{3i} + \color{red}{2j}$, just for different basis vectors.

This is what linear algebra and matrices encode geometrically. If we write our basis vectors $\color{blue}{i}$ and $\color{red}{j}$ in a matrix and multiply that by the vector representing our initial point, we will get a new point representing our transformation (a.k.a., our warp). How do we write our basis vectors in a matrix? Each vector implicitly has coordinates associated with themselves! In the above picture, $\color{blue}{i}$ points at $\color{blue}{(1,-1)}$, since from its tail to its tip it moves one step to the right and one step down. $\color{red}{j}$, on the other hand, points at $\color{red}{(1,3)}$, and these are precisely the vectors we see in our matrix.

For the unit basis vectors that point at $(1,0)$ and $(0,1)$, to differentiate them from any old pair of basis vectors, we call them $\color{blue}{\hat{\imath}}$ and $\color{red}{\hat{\jmath}}$ wearing a little hat, and their respective matrix the identity matrix, since it leaves vectors unchanged after multiplication (since that's what we used to define coordinates in the first place).

These $2 \times 2$ matrices represent linear transformations. They're transformations in they way that they transform points from one coordinate to another (well, most of them at least), and they are linear in the sense that keep all grid lines parallel, evenly spaced and, well, linear after the transformation. This is best seen through video and not stills. For this post you don't need to understand the mechanics of matrix-vector multiplication, but just understand that it represents some transformation on a point.

Warping Images

So why should we care? Why is this helpful in any way? If we think of each pixel on our images as a coordinate, we can just apply our transformation to all pixels on that image, find where they land and color them, and generate a new image. Let's take this picture as an example.

We can scale it, rotate it, or even shear it by applying the same transformation matrix to every pixel by changing $\color{blue}{i}$ and $\color{red}{j}$ like before.

But we have a big problem here: the typical linear transformation does not allow for translations. By the qualities of linear transformations, the origin cannot move, therefore forcing the bottom left corner of our images to always overlap! That's pretty restrictive in terms of the panoramas we can make—and for practical purposes—a complete nonstarter. If we want to continue through with making a panorama, we'll need to find a way around translations.

Homogenous Coordinates and Affine Transformations

There's a very sneaky workaround being confined to the origin. To do so, we'll need to do something that might seem a bit weird to do translations. Let's rewrite our 2D points with a 3rd coordinate. For a given $(x,y)$, let's rewrite that with a $z$-coordinate $(x,y,1)$. If $z \neq 1$, then we can just divide all the other coordinates by $z$ to make it equal to 1: $(x,y,w) \rightarrow (\frac{x}{w}, \frac{y}{w}, 1)$ (we generally use $w$ to represent the $z$-coordinate to indicate that there is no "real" $z$ value since everything is projected into 2D; we use $w$ as a "weight" to say how much we scale our projections down to). This means we have multiple coordinates represent the same point. In this way, $(2,5,1)$ and $(4,10,2)$ and $(-3,-7.5,-1.5)$ and $(2w,5w,w)$ all represent the same point (we don't include points when $z=0$ as it represents a point at infinity).

This might seem arbitrary, but what we're doing here is not too different than our original, 2-coordinate system. When we look at a cross-section of the $xyz$ coordinate space, it looks exactly like the $xy$ plane. What we are doing here is projecting all of $xyz$ space onto the plane $z=1$.

In fact, many of you are already familiar with homogeneous coordinates (representing 2D points with a 3rd scalar coordinate) and projective planes! When you take a photo on your phone, how does the camera know what's drawn in its frame? How does it take a 3-dimensional world and put it into a 2-dimensional picture? The many light rays that enter the camera lens (the origin) will intersect a plane ($z=1$) based on its focal length, and colors the pixel based on the projection.

Using this analogy with photos, clearly translations should be possible! If you've seen any cat videos on the internet, clearly it is possible for the cat to enter and exit the frame freely without the camera necessarily moving, and that is precisely possible due to the fact the origin $(0,0,0)$ is not contained in our projective plane $z=1$, since all of our basis vectors have to stem out of the origin! (For those interested a translation matrix is equivalent to a shear along the $z$-axis.)

Think about what we're doing here: we're turning a linear transformation in 3-dimensions to create special non-linear affine transformations in 2-dimensions! When I first learned this geometrically, awe can't encapsulate the total shock I felt. So, if we're given a point $p$, we can transform it with a matrix $M$ to get its image $p'$.

(Notice how $M$ is now a $3 \times 3$ matrix as we are working with 3D coordinates now) If our matrix-vector product results in a $w$ value not equal to 1, then we just divide everything by $w$ to make it so, and get our coordinate in terms of our 2D plane.

These projections with homogeneous coordinates are known as homographies. When we take one picture, and reproject it according to a matrix like this but keeping the same camera center (like the origin), we call it a homography. Again, like homogeneous coordinates, people have been leveraging homographies for a while now. You know how on roads, whenever there's text written like, "STOP AHEAD", it's always a bit vertically stretched? That's because when you read it at the angle of a driver in a car, it looks more regular and readable. At least, that's my theory.

More creatively, that weird, perspective street art you might have seen before? That's the most manual you can get to using homographies—literally warping images with the angle you look at them to make them appear at a normal proportion.

What we're doing is computing a homography to build a mosaic. Just like the decorative tile art of the same name, we are taking tiles of photos that we transform to overlap, and them stitching them all together into one, broader image.

Moreover, our homographies have a really funny interpretation to them. Since we are reprojecting pictures, what it geometrically looks like is that we're taking two photos which should be rotated in space (as you would spin taking the panorama), and taking a photo of the two photos. Photo-ception.

There are other ways to reproject images to make other mosaics with the own benefits and downsides, but this is what we'll use for now. Benefits with this type of mosaics? They are (relatively) easy and fast(er) to compute. Downsides? Since we are projecting onto a plane like this, we can only take panoramas up to 180° wide (can you see why?).

One Slight Issue…

While it's great that we are able to transform points with matrices, let me remind us what our goal is.

We have these two photos, where we want to transform one image's points to overlap with the other's. In terms of our matrix arithmetic from before, we have $p$ and $p'$, but no matrix $M$... Up to this point we have been finding our image points using our own matrices, but how do we find that intermediate matrix given a point and its image?

Regressions and 4-Dimensional Lines

Like homogeneous coordinates, many of you will already be familiar with solving for the intermediate matrix given a $p$ and $p'$. Let's do it with a simpler example.

If you had the points $(1,2)$ and $(6,4)$, and I needed you to find the line $y=mx+b$ that went through them, most of you would be able to do that. We'd set up a system of linear equations

and solve for $m$ and $b$ respectively. In this case, $m=\frac{2}{5}$ and $b=\frac{8}{5}$. Simple algebra with little to worry about here. What is important to note here is that we could solve for a unique $m$ and $b$, since two points define one unique line.

But what if I introduced a third point $(p_3,p_3')$? Or even a 4th point $(p_4,p_4')$? How do we draw a line through those 4 points? There might be a line that goes through all 4 points, but it's highly unlikely.

We may not have exact values for $m$ and $b$, but what's the best value for both to get the closest solution to this system of equations?

That's a task as simple as plugging it into a spreadsheet and doing a linear regression. More specifically, we can use the common least-squares regression where we want to minimize not the sum of the errors, but the sum of the square of the errors (as the name would suggest). For those a little more comfortable working with matrices and linear algebra, here's a more in-depth explanation of what we're doing with our data when finding a regression.

To many, this might seem like an obvious thing to do; everyone from middle schoolers to office workers have been finding trend lines forever. But what we did here is pretty useful when we think more abstractly: given a system of linear equations that correlated independent data $p$ with their dependent data $p'$, we were able to solve for the best coefficients of that system of linear equations that most closely solved the system (in the previous case was $m$ and $b$). Finding a line was a nice byproduct, but what we're really doing here is solving that system of linear equations.

Now I promise this will be helpful. Let's look at our original expanded matrix equation of $Mp = p'$.

Remember, we're working in homogeneous coordinates, so $p'$ might not land on the plane $z=1$, and we account for that with $w$ here. I also set $i=1$, since a) that corresponds with a certain scaling and is not necessarily unique vector in the land of homogeneous coordinates, and as a result, more importantly b) gives us one less variable to solve for.

Here, we will need to actually do the matrix-vector multiplication, and carrying it out nets a system of linear equations! (I know I said you won't need to know the mechanics of these operations, but it's hard to avoid it now. If you can accept this fact, that's great, but I'd recommend looking here if you are unfamiliar.)

Using the third equation in tandem with the first two…

Just like before, we can solve for $a$, $b$, $c$, $d$, $e$, $f$, $g$, and $h$ with a least-squares regression! Since we have 8 variables, at minimum we need 8 equations, or 4 pairs of $p$ and $p'$ (since each pair contains two equations: one for $x'$ and one for $y'$). Though, just like we have 10 points, generally it is better to have more data and overfit than less (we'd rather have an overall average fit, than just 4 points be exatly where we want them to be). It's weird to think of this geometrically, since what we're doing here is not finding the line between one independent variable and one dependent variable, but rather two independent variables $(x,y)$ with two corresponding dependent variables $(x',y')$; our regression exists in 4-dimensions!

Putting It All Together

Let's quickly reflect on what we've covered thus far.

1. We've redefined coordinates purely with vectors, allowing us to nicely compact our image-warping transformations in matrices.
2. Our original definition of coordinates failed to include translations—a key transformation. We described 2-dimensional points in 3-dimensions with homogeneous coordinates, resolving our worries.
3. We then ran into ANOTHER problem in that while we knew how to warp images given the transformation matrix, we really wanted to be able to find the matrix given a starting point and an end point to map to.
4. Using a least-squares regression, we were able to turn our unknown matrix equation into a system of linear equations that were much easier to work with to compute our homography (sort of, see the aside below).

Let's use this first photo to give our list of points $p$.

And we'll try to match those red points to these blue points on the second photo: our list of $p'$.

Having the computer compute the transformation matrix, we take that matrix and multiply every pixel (remember, treating them as coordinates/vectors), and warping the first image. Then, we can overlay them to see how close our points line up! If our points were well selected, and our computed homography—with the least-squares regression—has minimal error, we should get a pretty decent attempt at a panorama.

Sure, the blending isn't great, and it didn't completely fix the overlap issue, but the seams and photo stitching definitely is much nicer! And honestly, it's pretty cool seeing how the image was transformed and finding the outline of the images cross like that.

With some simple masking and basic filtering (basically averaging every pixel's color with the pixels around it), suddenly it really begins to look clean.

While this is cool, it does reveal another unfortunate downside of our choice of mosaic: if we want a uniform picture, we have to sacrifice a lot of data.

Even so, it doesn't even look that bad. All in all, though, not a bad first attempt at building a panorama.

What Next?

While we have a working prototype, we can do signficantly better. For one, I used only 10 labelled points to compute our homography, but if you use even more, it's not hard to get a better, and closer fit. With algorithms like LoFTR, finding lots of corresponding labelled points between multiple images is quick and easy.

Also, since we are manually constructing our panorama, we can stitch and blend photos that have no right being together in a panorama.

In a similar manner, we only conjoined two photos together, but we can easily extend this to as many photos as we want (but I can't say how well the photos towards the end will necessarily stretch).

We never really touched on our homographies, either. When we decided 10 initial points $p$ and 10 warped points $p'$, our $p'$ was decided as a result of lining up 2 photos. What if we didn't want to line up multiple photos, but rather just creatively warp a single photo?

This is know as rectification, as it is a means to correct for mistakes we might have had in our photo.

Finally, the last improvement we can make to our mosaics is trying new projections and warpings. If we want something even as simple as just wider, up to 360°, full views, we'll need to find something more robust than our previous approach. Or what if we wanted to make something akin to a full photosphere like from before?

What we did today was simply planar projection, or just reprojection onto a plane. We did that with homogenous coordinates. For wider, more complete mosaics, we'll need either cylindrical or spherical projection, which is exactly what it sounds like. These have their own benefits like wider field of view, but because of the nature of projecting onto a curved surface, the images being stitched together do tend to, well, curve. The type of mosaic one uses comes down to preference and artistic need.

And lastly, there are many optimizations and polishing details we could add to make our panoramas cleaner, and run faster. For instance, we never mentioned the discontinuities that could be present in warping our images with matrix multiplication. While linear transformations keep lines before the warp as lines afterwards as well, that's only helpful if our line is continuous. Pictures are not continuous! They are discrete points! So, forward warping with our matrix multiplication and finding where pixels lands can sometimes create (albeit, usually imperceptible) holes in our images, but they are there nonetheless. Instead, we can reverse warp by applying the inverse of our transformation matrix, and find what coordinates land on our original image! Not to mention different blending and masking techniques, or even just algorithmic improvements to make the code run faster. Check out the Python notebook below for more details.

For more like this and additional resources, I recommend reading these slides from UC Berkeley's introductory computer vision and computational photography class.

I hope this gave an interesting peak at the intersection of linear algebra and photography, and more over, I hope this gave you an appreciation for the math your phone goes through every time you take a panorama.

If you're interested, here's a link to a Python notebook where you can see some of my experiments during my struggle and exploration with panoramas and homographies.

Aside: Least-Squares with Linear Algebra

Okay, this previous section is really hard to describe without already knowing a fair amount of linear algebra, and it felt a little flat without having a more methodical procedure of solving a least-squares regression. I wasn't planning on including this section, but it felt incomplete otherwise. For those interested, feel free to peer over it, but this is not necessary within the scope of this post; all you need to understand is what our regression is accomplishing, thinking of that "line of best fit" idea giving rise to optimal coefficients in a overfitted system of linear equations.

Let's go back to when we were trying to find a line between two points. If you have 2 points, $(p_1, p_1')$ and $(p_2, p_2')$ being fit to the line $y=mx+b$, we have a system of linear equations like before.

We can solve this just like we did before to find $m$ and $b$, but there's another, sly way we can approach this. If we look carefully at the structure of these equations, there's actually a secret matrix relationship embedded into this system.

In a sense, that's what a matrix is: a system of linear equations, and you can freely go between either a system of linear equations or a matrix via matrix multiplication. (I know I said you won't need to know the mechanics of these operations, but it's hard to avoid it now. If you can accept this fact, that's great, but I'd recommend looking here for more details.)

If we write this in general terms, we are basically solving the equation

where $A$ is a matrix, and $b$ and $x$ are vectors, and we are solving for the latter. It might seem pointless to rewrite it, but what we're actually solving is

Since $Ax$ is exactly equal to $b$ in the 2-point case, we can solve this matrix equation fairly directly; when there's a unique, perfect solution $Ax$ is the same vector as to $b$. We were able to find a unique line with $m$ and $b$ through them, no? Just as we were able to solve the system of linear equations before, we can easily solve this with matrix inverses:

Now, let's add more points.

Now we turn this into a matrix equation like before.

We know that there's a good chance our four points don't all lie on the same line. So it's unlikely that $Ax - b = 0$. Moreover, now that our matrix $A$ isn't square, we can't just use inverses to solve for $x$. So instead, we want to get a line that gets as close to 0 (a.k.a. being a perfect fit). So our goal is to

Here, the $||x||_2$ means we're looking at the Euclidean distance (a.k.a. straight line distance) as our error for our line of best fit, and we're squaring it to get a tighter fit since small errors are kept relatively small, while large errors are weighed heavier. We know $A$ and $b$ with $x$ as our unknown—this sort of looks like a parabola-y equation! When we minimize a single variable function, we do so with the derivative. We can do the same thing here except with the multivariable equivalent: the gradient. So, we know the minimum occurs where the gradient of this function is 0.

Even if you're not familiar with multivariable calculus, much of the following should still look vaguely familiar to the chain and power rules of single-variable calculus.

All finally simplifying to the very nice formula of

I like this approach for it's intuitive roots in the geometry of single-variable calculus, but if you want a more strictly linear algebra approach, here's this excerpt from Georgia Tech that explains another proof for the same formula:

Theorem. Let $A$ be a $m \times n$ matrix and let $b$ be a vector in $\mathbb{R}^m$. The following are equivalent:

1. $Ax=b$ has a unique least-squares solution.
2. The columns of $A$ are linearly independent.
3. $A^TA$ is invertible.

In this case, the least-squares solution is

Proof. The set of least-squares solutions of $Ax = b$ is the solution set of the consistent equation $A^TAx = A^Tb$, which is a translate of the solution set of the homogeneous equation $A^TAx = 0$. Since $A^TAx$ is a square matrix, the equivalence of [facts] 1 and 3 follows from the invertible matrix theorem. The set of least squares-solutions is also the solution set of the consistent equation $Ax=b_{\textrm{Col}(A)}$, which has a unique solution if and only if the columns of A are linearly independent.

Basically, it says if our system of linear equations contain only unique equations (i.e. no one equation is a multiple of another), we can turn our non-square matrix $A$ into a square one by multiplying by its transpose $A^T$, and solve our least squares the way we'd solve it before with inverses. In other words, if our matrix follows the criteria listed above, our minimizing solution comes from creating an equivalent equation with an invertible matrix:

Netting precisely the same formula as before.

Now, let's recall the our matrix equation from before of the homography we wanted to solve.

Then, we expanded this into 3 linear equations, and further simplified them to the following two:

This, can be rewritten as another, secret matrix equation:

Wait, we turned our original matrix equation into another one? As awful as that may look, this is much more useful than our original equation since now, all of our unknowns are in a vector instead of a matrix; it really is no different than our previous least-squares examples, and we're still solving for the vector $x$ in

So, we can still solve it like before finding

And with that, we now have also gone through what our program is doing under the hood, and have gone through some of the tedium of justifying what a least-squares regression is from a linear algebra perspective.

Claim: $\sqrt{2}$ is irrational.

Proof: Suppose that $\sqrt{2}$ is not irrational. Therefore it is rational, and can be written as $\sqrt{2} = \frac{p}{q}$ where $p$ and $q$ are distinct coprime numbers (i.e. they don't share any factors in common; this is a way of sayiing that there is a unique way of writing $\sqrt{2}$ as a fraction in lowest terms). Squaring both sides and moving terms around, this equation becomes $2q^2 = p^2$. Therefore, $p^2$ is even, since it will always have a factor of 2 according to the lefthand-side of the equation. If $p^2$ is even, then $p$ is also even (if you're unsure of this fact, try some test cases and proving it!). Since $p$ is even, then let's rewrite it as $p=2n$. Plugging this into our equation yields: $2q^2 = p^2 = (2n)^2 = 4n^2$. Dividing both sides by 2 gives us $q^2 = 2n^2$. Similar to before, the equation implies that $q^2$ and hence $q$ is even due to that factor of 2 on the righthand-side. But this is a contradiction! $p$ and $q$ cannot simultaneously both be coprime and even (share a factor of 2). Hence, our initial assumption that the $\sqrt{2}$ is rational is wrong, and allows us to conclude that $\sqrt{2}$ is irrational. $\blacksquare$

It's not a long proof at all, and in the grand scheme of the history of math, pretty important, too. Supposedly, this was the first instance of irrationality being proven as a possible quality. More specifically, legend has it that Hippasus of Metapontum was credited with showing that the $\sqrt{2}$ is not rational (note the word choice) in the 5th century BCE while at sea, and upon sharing his amazing discovery, was instantly thrown overboard and left to drown.

Arguably of more significance, the proof highlights a general technique we use not just in math, but everyday logic and reasoning: proof by contradiction.

Proof by Contradiction

I'm sure you have all heard someone say, "For the sake of argument…," at some point in your life before. The idea is to disprove any hypothetical argument by showing it leads to an impossibility.

Let's say someone was arguing that Euclid was still alive today. Instead of remaining in utter shock at such a claim, here's a simple reply to show otherwise. For the sake of argument, let's assume Euclid is still alive. It is generall agreed that Euclid lived around 300 BCE, so if he is alive now, that would make him around 2300 years old now, which is crazy! Only a handful of people live past 100, let alone 1000 years. So it is safe to say, it is not the case that Euclid is still alive today.

In fact, the Latin phrase that describes proof by contradiction, "Reductio ad absurdum," literally means, "Reduced to the absurd/impossible".

Even if you haven't realized it before, you've probably used this same principle a lot in school. When taking a test, process of elimination can be an immensely helpful tactic to find the right answer to a question: if you can eliminate what you definitely know to be false, the rest of the options must include the truth.

At its core, we're using double negatives to prove a "positive" statement; we assume something to be false, show that it can't be, and therefore conclude, it's true since something that is not false must be true. If a number is not rational, it's irrational. If someone is not alive, they're dead. If someone's not guilty, they're innocent.

That last one feels weird, right? Just because there's no evidence to show someone is involved in a crime does not mean we always feel them to have been completely removed, right? In the previous examples, there's a strong sense of complements. "Rational" means "not irrational", so "not not irrational" is the same as "irrational". "Alive" and "not dead" are the same, so "not alive" is "not not dead" is "dead". But is "not guilty" really the same as "innocent"? Semantically, that's what it means. If you Google it, that's what you get.

From a casual, conversational point, it might, but from a legal standpoint, there's a reason why we make a distinction. We don't live in a binary world. So, it's not totally clear that we can say "not guilty" is the same as "innocent", even if that's what their definitions may mean. Even if two things are definitionally polar opposites, we, as the humans behind the words' significance to begin with, add subtlety and nuance.

"Not guilty" still insinuates that there's a feeling that there's something missing; we think they are guilty, but are unable to prove it. So even if we define "guilty" to be "not innocent", we might not want to say "not guilty" is the same as "innocent".

Which should come with a weird realization: if "guilty" is "not innocent", then "not guilty" is "not not innocent", which we said to not be equal to "innocent". Double negatives don't always cancel out.

Maybe our original definitions are wrong then? But it would be hard to define either "guilty" or "innocent" external to each other.

This may just seem like nitpicking, but it's important to consider this, since it puts into doubt such an intuitive, clear cut idea. If we can't use double negatives in one place, who's to say where else we can't use it? It might seem obvious at first that "not alive" must mean "dead", but have we really shown that? Just because it's absurd to think about Euclid being alive right now, does our proof really guarantee that? Is there something more to rational and irrational numbers that we might be missing? Just like with differentiating "not guilty" and "innocent", just because we have not been able to show someone guilty does not mean that they aren't.

Yet, this is precisely the concept that proof by contradiction relies on. We assume the opposite of what we actually want to prove, show it is impossible, then negate it to get to our original statement. But we don't actually show anything directly about our claim, only that it is impossible that it can't be true. If that's not valid—like it is the case between "guilty" and "innocent"—this no longer becomes a valid proof technique!

Above, we've only demonstrated that $\sqrt{2}$ is not rational, when we wanted to show that it is irrational. Did we really do that?

The Law of Excluded Middle

The easy answer to that is yes. We state a strict definition, and we can pretty clearly figure out when something is binary and when something has wiggle room. We can define a number to either strictly be rational or irrational. If it's not one, it has to be the other.

Technically, fine, yeah, we can do that. But that's kind of unsatisfying. But, there's a more fundamental question we should ask, since we're not just concerned with a number being rational or not, or someone's mortality, but proofs in general; we want to know if we can use proof by contradiction consistently. In general, we're trying to prove the truth of claims, so what we really care about is the following question: if something is not true, is it false (and vice versa)?

Again, obviously yeah, right? What else could it be? All declarative sentences (sentences make a claim about something, i.e. not questions) have to be true or false. It's that simple. It either is or it isn't. "Cats are animals," is true, and, "The Earth is flat," is false. Easy.

This is known as the Law of Excluded Middle, for, well, we exclude any type of middle value in between true or false. For those who like their logical formalities, if $\Phi$ is our claim about the world, then we say $(\Phi \lor \neg \Phi)$ is a tautology (that is, something that is always true). In natural language, this literally means "either our claim of the world is true or our claim of the world is not true", which just makes sense.

This forms one of the, and I can't make this up, Three Laws of Thought upon which "reasonable arguments" are based off of. These principles include:

1. The Law of Excluded Middle: every claim or its negation is true.
2. The Law of Non-Contradiction: nothing can simultaneously be true and false.
3. The Law of Identity: everything is identical to itself.

These three are usually taken as "common sense" axioms, and along with a few other assumptions build the basis of inference and deduction. Some might note that the first two laws might mirror each other through De Morgan's laws, but remember these don't actually explicitly state De Morgan's laws and we have no way of inferring that with these three principles alone. These three laws come up in many contexts, and have been refomalized many times over across history. Even the last Law of Identity, which seems almost pointless, has been taken by none other than Leibniz (yes, the same one from calculus) and been turned into two laws. That's for another day, though, and we will continue to only focus on the first law.

Ok, then how about the following claim?

Is this true or false? If it's true, then it declares itself to be false. If it's false, then it must be giving the wrong information, and therefore it actually is declaring itself to be true. And if it's true…

Suddenly binary truth values don't seem to be as reliable as they first seemed. This sentence doesn't neatly fit into one cateogry. If you consider one side, it flips to the other. It's been aptly named the Liar's paradox for its deceptive nature, and there are many attempted solutions, of my favorites being the fuzzy logic answer for its clarity and funny name. The number of attempts at resolving such an annoying sentence, though, should highlight that binary truth values are not always the right approach.

At this point, it's time we might have to reconsider the Law of Excluded Middle. Maybe this is just a dumb paradox that should be ignored as an edge case, but if you leave it as it is, then are we really going to be happy with what we have, knowing that there are some problems out there we just can't address?

Intuitionism

Perhaps, we don't have to reject the Law of Excluded Middle, but instead come up with a stronger type of reasoning. As we saw, the Law of Excluded Middle lines up pretty well with most things we observe in general, so we shouldn't discount it for that. Though, we might find a way that can avoid this issue we ran into with "not guilty" versus "innocent" altogether, and that would be foolproof from the start.

Our proof by contradiction is known as a non-constructive proof, in that it reasons about a claim without ever demonstrating the claim itself; there's no actual evidence presented for the thing we want to show true, since we try to make it appear "obvious" that there is no other way for such a claim to exist.

Our proof by contradiction for $\sqrt{2}$ being irrational is one example, but here's another one of my favorites.

Claim: There exists irrational numbers $a$ and $b$ such that $a^b$ is rational.

Proof: We know (and continue later to show) that $\sqrt{2}$ is irrational, so consider the number $\sqrt{2}^\sqrt{2}$. If this is rational, we're done. If this is irrational, then consider the number $(\sqrt{2}^\sqrt{2})^\sqrt{2} = \sqrt{2}^2 = 2$, which is rational, and hence we are done.

Never once did we actually deduce the rationality of $\sqrt{2}^\sqrt{2}$, but by exhausting the possible cases, it doesn't matter since every possibility leads to a conclusion that proves our claim.

If we can find a constructive proof and actually manifest our claim in some way, then we don't have to worry about double negatives or anything like that since we directly showed what we wanted to.

We're now dabbling along lines of intuitionistic logic where, ironically, we are breaking our natural sense of double negation and relying solely on these constructive proofs to validate truths. Since, in a way, non-constructive proofs are just left in abstraction and are no more than just a construct of our mind. Intuitionism tries to separate ourselves in the proof by showing an objective way of realizing a truth.

Let's go back to our original proof by contradiction above:

Claim: $\sqrt{2}$ is irrational.

Remember, we don't trust double negatives, so we want to avoid saying "irrational" is the same as "not rational". What does it really mean for a number to be irrational?

Well, even though we don't want to say "irrational" is "not rational", we can draw inspiration from what the latter to find a better definition. We say a number $x$ is rational if it can be written in the form $x=\frac{m}{n}$ where $m$ and $n$ are integers. Or, equivalently, $x$ is rational if there exists integers $m$ and $n$ such that $x - \frac{m}{n} = 0$. Since we're using subtraction here, there's a natural way to think of rationality in terms of distance; a number is rational if it is distance 0 away from some integer fraction. This gives us a nice way to think of irrationality.

Definition: A number $x$ irrational if for all integers $m$ and $n$, $|x-\frac{m}{n}| \neq 0$. In other words, a number $x$ is irrational if it is some distance away from every rational number.

We add the asbolute value signs since we're quanitifying distance, and we don't care if a number is less than or greater than any rational number, just that it is not exactly equal to one of them.

Some of you may be skeptical of what I've just laid out, since if you're familiar with mathematical quantifiers, it does look like there is some negation work that is not explicit has been used under the cover of my words. Even so, the point is we have arrived at a distinct, specific definition of the property we set out to prove, allowing us to try and prove a positive claim as opposed to a negative one (like in proof by contradiction).

The New Proof

We can rewrite our claim now as such:

Claim: $\forall m,n \in \mathbb{Z}, \ |\sqrt{2} - \frac{m}{n}| \neq 0$

That first part just means $m$ and $n$ are integers. We can now do our proof fairly quickly.

Proof: Note that $1<2<\frac{9}{4}$, therefore $1<\sqrt{2}<3/2$. So the only rationals we care about comparing $\sqrt{2}$ are the ones in between $1$ and $\frac{3}{2}$, since it's apparent that any number outside this range will have a non-zero distance away from $\sqrt{2}$. Therefore, let's take our integers $m$ and $n$ such that they are positive, and $1<\frac{m}{n}<\frac{3}{2}$. Let's also take a second to note we can add these inequalities, giving us that $\sqrt{2} + \frac{m}{n} < 3 \rightarrow n\sqrt{2} + m < 3n$.

Now we can do some simple algebra on our claim to show it true.

Let's examine the numerator. Every integer, $x$, has some number of factors of 2 in its prime factorization i.e. $x = 2^\alpha \cdot C$ where C is the rest of its factors. When you square that integer, the number of 2s in the denominator doubles $x^2 = 2^{2\alpha}\cdot C^2$. So, we can write $2n^2 = 2 \cdot 2^{2\alpha} \cdot C_1^2 = 2^{2\alpha + 1} \cdot C_1^2$. Similarly, $m^2 = 2^{2\beta} \cdot C_2^2$. The key detail here is that $2n^2$ has an odd number of factors of 2, while $m^2$ has an even number of factors of 2, and therefore cannot be the same number! And since they are integers, then $|2n^2 - m^2| \geq 1$.* Continuing on,

We simplified the denominator with the inequality we found at the start. But look at what we have here!

This means for any rational number with denominator $n$, $\sqrt{2}$ is at least a distance $\frac{1}{3n^2}$ away from that rational! Therefore, $|\sqrt{2} - \frac{m}{n}|$ can never equal 0, and we have thus proved our claim and that $\sqrt{2}$ is irrational.

Conclusion

A lot of this can seem like nitpicking definitions and being overly pedantic, and to a certain extent I agree. I mean, even in the proof above we sort of implicitly assume $\sqrt{2}$ is irrational at (*)! When we said $2n^2 \neq m^2$, you could rearrange that to get $\sqrt{2} \neq \frac{m}{n}$. In our definition above, this would only constitute as being "not rational" as opposed to "irrational", but we are so subtely tweaking our interpretation of this fact to make it useful. When differences between proofs are this minute in terms of what we are saying, it seems hard to take any of what we talked about today as practical. We even turned to the definition of rationality to come up with a "better" definition of irrationality, so what's even the point in trying to discern minutiae of our constructs.

Even so, it's a good exercise in thinking about how we arrive at "facts" or "knowledge". When can we prove something by contradiction, and even if we can, is there a more convincing argument? Are there possible cases we might be leaving out and forgetting? Are there some things that we cam only prove directly or indirectly (answer is yes)?

These questions are what underpin so much that we naturally gravitate towards, and there's a reason why they've been examined so harshly. These questions literally uprooted math and led to some of the most famous, mind-blowing results the field has ever seen. So always ask yourself, "What can I )assume, and what do I know?"

What shape has the largest area to perimeter ratio?

Some might have a guess, or an intuitition for what the answer should be, but it's a surprisingly difficult question to pin down as a proof. In fact, a rigorous proof wasn't given until around 1840 (J. Steiner)!

Some Observations

There are some facts about our ideal shape we can observe quickly. First, it has to be convex_**. For if our shape was concave, we could always add more area with simple reflections.

We can get free area by reflecting over the blue line.

This can also be seen to work for not just polygons but curves as well, where instead of reflecting over vertices we can reflect over tangents.

Further, in a similar argument, there has to be a type of perimeter-symmetry to our shape. For any way of dividing our shape into two equal perimeters, the areas enclosed by this dividing line must be equal. If they were not, one half would be bigger than the other, and we could just reflect that bigger half over the line to get a shape with the same perimeter but greater area (as we would have two copies of the bigger half glued together, as opposed to the bigger half attached to the smaller half).

This then also gives us a way to half our problem. Literally: instead of looking for the curve that encloses the most area for the same perimeter, we can instead look for the arc that encloses the most area with its endpoints connected by a fixed straight line. Then we can just reflect over it and we're good. So let's look at an arc.

Is there any way we can increase the area of this arc? With the above diagram, we have divided the area into 3 sections. Since we want to keep the arc's perimeter fixed, we can't really affect areas 1 and 2. But 3 we can maximize easily. For a triangle with 2 fixed side-lengths and the angle in between them, the area of the triangle is $ab\sin\theta$, which is maximized when $\theta$ is a right angle. So, we can increase the arc's overall area by picking a point along the curve, and having it such that it always forms a right triangle with the base of the arc.

So, to maximize the area of the arc, we want this to be true for any point picked along the curve. What curve is defined by forming a right triangle with its base at every point? It's a semi-circle! By fixing the base-length as well as the right angle restriction, it happens to also fix all radial lines from the mid point of the base. So finally reflecting back over the line as our second observation suggests, the shape that encloses the most area for a given perimeter is a circle.

…that is if such a shape exists. What we have really shown is that IF there is a shape that encloses the most area for a given arc length, THEN it is a circle. We have not necessarily shown that there is a shape. Intuitively, it seems a bit frivolous to worry about it since it just makes sense that there is only so much you can do to spread a curve out, but it is worth taking a second to do so. For, if the problem was, "What shape encloses the least area for a given perimeter?", there is no such shape we can give. But this is a nit-picking detail that can be shown with not much difficulty using some calculus and limiting processes, but is worth keeping in mind.

Some might know the answer off the top of their head, and some might be able to guess, but the answer is unsurprisingly a circle. And we see that this is realized all throughout nature. Bubbles are circular since for a fixed amount of gas (i.e. volume), the sphere requires the least surface area to accomodate it (and therefore put the least amount of stress on the bubble). Rain drops, too, would be perfectly spherical if not for gravity.

The Analytic Approach

With more robust tools, we can prove this more directly—albeit in a more overkill way. Our problem of finding the greatest area per perimeter is a byproduct of the isoperimetric inequality: for a given perimeter/arc length of a curve $L$ that encloses an area $A$, then $4 \pi A \leq L^2$. The proof isn't long, but requires some less accessible calculus machinery. The below is adapted from Erhard Schmidt's 1938 proof.

PROOF: Let's parameterize a general simple closed curve $C$ (i.e. does not cross itself and ends where it starts), say $c(t) = (x(t),y(t))$. Since we care about arc length, we will parameterize $c(t)$ with arc length i.e. $\forall t \ |c'(t)| = 1$ so that $c(L)$ is our end point of the curve. Now as it is a simple closed curve, $y(t)$ must be bounded as continuous functions on a bounded interval are bounded. So we can enclose our curve by two parallel lines, say, $l$ and $l'$ that meet the curve $c(t)$ at say $t=0, t_{l}$.

Then, the area bounded by our curve $C$ is $A = \int_C x \ dy = \int_0^L x(t)y'(t) \ dt$ by Green's theorem.

Now we can construct an "approximate" circle to our curve, that is parameterized by $r(t) = (x(t), z(t))$ and is also tangent to $l$ and $l'$ (note that they are parameterized by the same x-vector). Call the radius of the circle $R$ and let its center be the origin of our coordinate system. Using our previous parameters, we will further fix this circle by

to ensure it lies in the $l$ and $l'$ and is, well, a circle. Then, the area bounded by our circle is $\pi R^2 = - \int_\textrm{circle} y \ dx = - \int_0^L z(t)x'(t) \ dt$.

Adding our areas together, we get,

as $x^2 + z^2 = R^2$ and $y'^2 + x'^2 = 1$. The first inequality is by the arithmetic-geometric mean inequality (AM-GM), and the second one is by the Cauchy-Shwarz inequality applied to $(x, z)$ and $(y',x')$. So putting it all together, $2R\sqrt{A\pi} \leq LR$, or $4\pi A \leq L^2$.

To get equality, we need to satisfy the first and last inequalities. To get equality in AM-GM, we need $A = \pi R^2$ and thus $L = 2\pi R$. To get equality in Cauchy-Scharwz, we want our vectors to be parallel i.e. $\frac{z}{x} = \frac{-x'}{y'} \rightarrow zy = -xx'$. Substituting this into

from above nets that $x = \pm Ry'$ and $y = \pm Rx'$, or in other words by squaring and summing both equations, $x^2 + y^2 = R^2$, which is precisely the definition of a circle.

So?

This problem generalizes to higher dimensions with higher dimension balls being the solution. But what I find most ineteresting is how quickly this problem changes with a slight tweak. We now know for sure circles are the best shape for a single container; an orange grows in the way for me to have the most amount of fruit in that single orange. But what if I had multiple oranges? Circles are awful to pack together as unlike, say, squares that nestle nicely together, circles leave so much extra room.

So, if we wanted to create the optimal shape for stacking together using the least materials, what shape would that be? Clearly the answer is not a circle since it doesn't stack efficiently, and we have at the very least a square as our starting point. It turns out the answer, as nature has found its answer through bees, are hexagons. Perhaps another day we will go through the proof, but Thomas Hale's original proof of this Honeycomb conjecture is dense enough to leave alone for now.

What is even more interesting, is that unlike the isoperimetric problem from above, the 3D Honeycomb conjecture is still unsolved: what is the shape with the best area/surface area ratio to tesselate space? The Weaire-Phelan structure gives a non-obvious polyhedron that's the current record holder for known shapes, but it has not been proven to be the best.

The study of these minimal surfaces all tend to follow similar problems like the ones above, and just how deceptively difficult they are are what make them so interesting. Inspiration from physics and observed phenomena in nature are so helpful to guide us towards intuitions, the rigorous proof math demands is so often buried by our own self-defeating intuitions. If you're interested in more like this, I'd suggest skimming over one of our previous discussions on the calculus of variations.

Pythagorean Theorem: the sum of the squares of the length of the legs of a right triangle is equal to the square of the hypotenuse.

The Pythagorean theorem is perhaps the most famous statement in all of mathematics. $a^2 + b^2 = c^2$ is practically drilled into the minds of everyone that went through any form of secondary school education.

But let me ask you: how do you know it is true? It's not like you've looked at every right triangle and checked it follows the theorem. There are many, many proofs of the Pythagorean theorem, but what makes you know that those fancy symbol manipulation is really legit? This might seem like an inane question, for math follows strict, rigorous, logical argument. But what makes this logical argument valid? Why do we trust logic so readily? There are many ideas that are intuitive that are false, and unintutive that are true, yet logic seems to get a pass from everyone? Why?

Over the past year, I took my university's introductory sequence for logic, which were probably the most enlightening classes I've ever taken. It forced me to reflect on not only what mathematics meant to me, but also what knowledge and deduction as a whole does too. This post is a summary collecting the highlights of the courses, so to act as a checkpoint to refer back to from future posts when I explore some of the most powerful ideas that have eluded me in what feels like a forgotten field of study.

As this will be a (quite a) bit of a longer post, the table of contents below will guide each section in the sequence they should be read, and all context needed for one section will be given in anything before it. I'll be following the conventions and terminology as per V. Halbach's The Logic Manual and A. Eagle's Elements of Deductive Logic. Brief examples will be given, but for more examples and exercises, see the texts above. For the best viewing experience, read this on a computer or wider-screen device for the equations to format as they were intended to be.

Table of Contents

1. The Language of Logic
   • What even is logic?
   • $\mathcal{L}_1$: A First Attempt
     • Connectives
     • Syntax and Semantics of $\mathcal{L}_1$
     • Validity of an Argument
     • Proof Systems and Natural Deduction
   • $\mathcal{L}_2$: The Logical Quantifiers
     • Predicates and Constants
     • Quantifiers
     • Domains
     • Satisfaction and Truth
     • Natural Deduction 2.0
   • $\mathcal{L}_=$: Identity and Definite Descriptions
     • Natural Deduction 2.5
     • Numerical Claims
     • Definite Descriptions
     • An Aside on Identity
2. Introduction to Metatheory
   • Principle of Mathematical Induction
   • Truth Functions
   • Substitution
   • Disjunctive Normal Form and Expressive Adequacy
   • Duality
3. Soundness and Completeness
   • The Soundness Theorem
   • The Completeness Theorem
     • Deductive Completeness
     • Maximally Consistent Sets
       • Constructing Maximally Complete Sets
       • Satisfying Maximally Complete Sets
     • Proof of the Completeness Theorem (of $\mathcal{L}_1$)
     • Completeness of Other Logics
4. What's Next?
   • More To Be Studied!
   • Natural Language and Grice's Maxims
   • Extensions of Logic
     • Higher-Order Logic
     • "Universal" Logics
     • Multivalue and Fuzzy Logics
     • Altering the Axioms
5. Conclusion

Part 1: The Language of Logic

What Even is Logic?

English is bad. Notoriously so. Try interpreting the following sentence: "The old man the boat." This might not even read as a correct sentence to some of you. The lexical ambiguity in that "man" is being used as a verb, even if it is more commonly a noun, is what makes this sentence lead one astray. Or, what about, "The professor said there would be an exam on Monday." Does the professor mean there will be an exam to be taken on the upcoming Monday, or was it that this past Monday he claimed there would be an exam in the indeterminate future? The structural ambiguity inherent to what "on Monday" qualifies makes the sentence on its own unclear.

English—and natural language as a whole—relies on expectations being met by both the speaker and listener for effective communication. We'll touch on this more later, but the inherent vagueness of our language, while gives it the nuance that we praise in art and literature, makes it really hard to analyze the truth of what people claim when they talk, especially in writing. Inflection and tone may give cues to what is precisely being said, but sometimes we want the content of our statement to be independent of any necessary human interpretation or judgement.

Take the following claim: "If Socrates is a man, then he is mortal. Socrates is a man. Therefore, Socrates is mortal." Few would dispute this argument, but there is very little content we needed to be okay with this argument. We don't care who Socrates is, or what it means to be a mortal or a man; there is something embedded into the nature of the argument that makes it good. "The Earth is flat if 2+2=5. 2+2=5. Thus the Earth is flat," is a perfectly good argument assuming that 2+2=5, so while it might not be true, it certainly is still a good inference given those facts. "If $P$, then $Q$. $P$. Therefore, $Q$," seems to always be a sound claim.

Logic, in this way, is the study of valid arguments. Given a set of premises, what conclusions can we deduce?

$\mathcal{L}_1$: A First Attempt

Above we already were able to begin generalizing a little bit. We came up with the general form, "If $P$, then $Q$. $P$. Therefore, $Q$." $P$ and $Q$ could be almost any sentence, and that argument would always hold. Almost any sentence, in that our sentences have to be truth evaluable; parsing questions or exclamations does not elicit any new information for our claims, so it does not really make sense to consider them. Even if we do not know if $P$ and $Q$ are actually true or false, we can still consider the different cases to check our argument structure.

Here we have the beginning of our first formal language: the logic of $\mathcal{L}_1$. We have already been able to characterize sentences in the form of sentence letters i.e. $P$ and $Q$, but if we need more we can add $R$, $P_2$, $Q_{3984}$, and others if we need more basic sentences (by convention, only $P,Q,R$ are the allowable sentence letters, with additional ones being subscripted). But this is a bit primitive.

Connectives

What makes arguments in English interesting is not the basic sentences, but how we are able to link them together via "If… then…", "…and…", "…or…", and other conjunctions. Can we formalize these in any way?

Let's think about a simple one: "…and…". Just like before, we can only care about sentences that are true or false, so a natural question will be when is " $P$ and $Q$ " true or false? "…and…" unifies two disparate sentences, so if the whole sentence was true, we would expect what ever is on either side of the "and" to also be true. To say that, "Grass is green and the Earth is flat," is a true sentence seems to ignore what the sentence is claiming. "…and…" states both subsentences "Grass is green" and "The Earth is flat" together at once, so if we think it is true, then both necessarily seem to have to be true, too. We can represent this as a truth table.

where $T$ means the sentence is true and $F$ false. Whenever $P$ and $Q$ are not both true, there conjunction is not either. As seen in the table above, we don't write "…and…" in $\mathcal{L}_1$, but rather use the symbol $\wedge$.

We call $\wedge$ a connective, for it connects two sentence letters together. Below are some other commonly used English connectives formalized in $\mathcal{L}_1$ with their corresponding truth tables.

It's worth taking a second to make sure you can get behind that these formalizations are reasonable. I won't go through all of them as we did with "…and…", but there are some things to take note of that do differ a bit in English.

• $\vee$ is specifically the truth table of inclusive or. In English, if one says "I'll either buy a pair of sneakers or sandals," we expect that person to usually just buy one option or the other, not both. $\vee$ allows the option for both, but does not require it to be true.
• $\rightarrow$ is technically different from "If…then…", even though I put them in the same row in the table. When we say, "If a butterfly lands in North America, then there will be a earthquake in Australia tomorrow", we can definitely say that is false if the butterfly does land in North America and no earthquake occurs, since they are definitely not causally related then. But if the butterfly does NOT land in North America, then we are in no position to say whether or not there is a relationship then since we have nothing to observe. Similarly, if an earthquake happens, we can't say if that was because of the butterfly or not regardless of what happens since correlation and causation are different. So really, the truth table for "If…then…" is $ \begin{array}{c|c|c} P & Q & \textrm{If} \ P \ \textrm{then} \ Q \\ \hline T & T & ? \\ T & F & F \\ F & T & ? \\ F & F & ? \\ \end{array} $ We say it is not truth functional for those question marks there since we cannot evaluate whether that sentence is true or not, but for the sake of convenience, we use $\rightarrow$ with the completed table.

The Syntax and Semantics of $\mathcal{L}_1$

Now, we can upgrade from sentence letters to complex sentences in our logic. We will define a sentence in $\mathcal{L}_1$ as follows:

1. All sentence letters $P, Q, R, P_1,…$ are sentences.
2. If $\Phi$ and $\Psi$ are sentences, then $\neg\Phi$, $(\Phi \wedge \Psi)$, $(\Phi \vee \Psi)$, $(\Phi \rightarrow \Psi)$, and $(\Phi \leftrightarrow \Psi)$ are sentences.
3. Nothing else is a sentence.

Note the use of parantheses to bracket the sentences. This is our way to remove all structural ambiguity by clearly stating the scope of the connectives; there is no question that $(P \wedge (Q \vee R))$ is different from $((P \wedge Q) \vee R)$ as the parantheses explicitly denote the scope of the connectives to what subsentences they are linking together. There are conventions to remove these brackets for convenience, which I may default to later, but as a whole, are not important.

The last detail we are missing are interpretations. Before, we were talking about how logic does not care about human judgement, but we can consider the judgement of the universe. What is the truth value of the sentence $\neg(P \wedge Q \rightarrow P \vee R)$? We could make a truth table, but clearly the truth of the sentence depends on the truth values of $P$, $Q$, and $R$. So under an $\mathcal{L}_1$-interpretation or $\mathcal{L}_1$-structure that assigns a truth value to every sentence letter, we can assign truth values to sentences. For example, if we let a structure $A$ be such that $|P|_A = T$, $|Q|_A = T$, and $|R|_A = F$, then $|\neg(P \wedge Q \rightarrow P \vee R)|_A = F$ by its truth table. The idea of a structure is that it is a model of a universe, where we designate certain sentences/facts to be true or false, and see how the rest of the other complex facts (that are linked by connectives) change in accordance to that possible universe.

Notation: $|\cdot|_A$ will stand for the meaning or semantics under a structure $A$.

Now most logical sentences one encounters will be true sometimes, and false sometimes depending on the structure its evaluated in. But consider the sentence $(P \vee \neg P)$. Looking at its truth table,

Every structure assigns a truth value to every sentence letter, so $P$ must be true or false in every structure, but regardless, $(P \vee \neg P)$ is always true. Tautologies or logical truths are in a sense, facts of nature, for no matter what universe we are looking at, "Either the sky is blue or it is not" will be true. At least, usually.

On the other hand, contradictions like $(P \wedge \neg P)$ are false in every structure.

For future reference, the following notation will be helpful. For a given structure $A$ and sentences $\Phi$ and $\Psi$ :

• If $|\Phi|_A = T$, we say $A$ satisfies $\Phi$, written $A \vDash \Phi$
• If $|\Phi|_A = F$, we say $A$ does not satisfy $\Phi$, written $A \nvDash \Phi$
• If $\Phi$ is a tautology, i.e. $A \vDash \Phi$ for all structures $A$, we just write $\vDash \Phi$
• If $\Phi$ is a contradiction, i.e. $A \nvDash \Phi$ for all structures $A$, we just write $\Phi \vDash$
• If $|\Phi|_A = |\Psi|_A$ for all strucutres $A$, we say $\Phi$ and $\Psi$ are logically equivalent, written $\Phi \equiv \Psi$

The last bullet point is just to say that there are lots of sentences that express the exact same truth table despite being written differently. For example, $P \rightarrow Q \equiv \neg P \vee Q$, so it's useful to say when to sentences are actually essentially the same.

Sets of Sentences

In addition to the semantics of individual sentences, we can discuss sets of sentences too. Consider $\Gamma = \{\Phi, \Psi,…\}$.

• A structure $A$ satisfies $\Gamma$ if and only if $|\gamma|_A = T \ \ \ \forall \gamma \in \Gamma$, written $A \vDash \Gamma$
• We say $\Gamma$ is consistent or satisfiable if and only if there is a structure $A$ such that $A \vDash \Gamma$

Validity of an Argument

We are finally in a position to start talking concretely and specifically what makes a good argument.

When someone has argued a claim (conclusion) from a series of facts and assumptions (premises), we want there to be a substantive connection between their premises and the conclusion. Just like with our truth table for "If…then…", we know there is no connection if there is a scenario in which our premises are true and our conclusion isn't. So if an argument is valid, then whenever the premises are true, the conclusion must also be true.

We can formalize this a little bit with the notion of semantic entailment:

Definition: Given a set of sentences (premises) $\Gamma = \{\Phi, \Psi,…\}$ and a single sentence $\varphi$ (conclusion), we say that $\Gamma$ semantically entails $\varphi$ if and only if for all structures $A$, if $A \vDash \Gamma$, then $A \vDash \varphi$.

In other words, whenever all of our premises are true, our conclusion is true.

When $\Gamma$ semantically entails a sentence $\varphi$, denoted $\Gamma \vDash \varphi$, then $\varphi$ is a valid conclusion from premises $\Gamma$. It's called semantic entailment since we are working with the semantics of the individual sentences, namely their truth values. In the language of consistency, we can also say that $\Gamma \vDash \varphi$ if and only if $\Gamma \cup \{\neg \varphi\}$ is inconsistent.

Claim: $\Gamma \vDash \varphi$ if and only if $\Gamma \cup \{\neg \varphi\}$ is inconsistent.

Proof: Suppose that $\Gamma \vDash \varphi$. Say that $A$ is a structure that satisfies $\Gamma$ i.e. $|\gamma|_A = T \ \ \ \forall \gamma \in \Gamma$. Then by our assumption, $|\varphi|_A = T$. By the rules for $\neg$, then $|\neg \varphi|_A = F$. For any structure $B$ that does not satisfy $\Gamma$, $\exists \gamma \in \Gamma \ \ \ |\gamma|_B = F$. So, for all structures, at least one sentence in $\Gamma \cup \{\neg \varphi\}$ is false, so it is inconsistent. $ \ \blacksquare$

It's worth pointing out that we have two uses for the symbol $\vDash$, one relating structures to sentences, and another relating sentences to other sentences. The context will specify which use we mean, but it is useful to keep in mind that we might double up on symbols when their related meanings are so similar.

Let's look at the example we've been holding this whole time: "If $P$, then $Q$. $P$. Therefore, $Q$." If this is a valid argument, then the formalization would say that $P \rightarrow Q, P \vDash Q$ is correct. We can verify this just by truth tables:

Whenever the premises are true, so is the conclusion. While technically an okay way to check, it is inefficient. Imagine we had an argument with not 2, but even just 3, 4, or more sentence letters, since for every additional sentence letter in our premises, it doubles the number of rows in our truth table to check. Instead, we can show it just as easily with a proof by contradiction. Suppose our argument is invalid i.e. there is a structure where our premises are true and our conclusion is false.

But, if $P$ is true and $Q$ is false, then by the rules for $\rightarrow$, we have $P\rightarrow Q$ is false.

That contradicts our assumption that $P\rightarrow Q$ was true as a premise. So, it can't be the case that our argument is invalid.

Quicks of Validity

Keep in mind our definition of a valid argument: $\Gamma \vDash \varphi$ if and only if whenever $\Gamma$ is satisfied, so is $\varphi$. Now consider the argument $P \wedge \neg P \vDash Q$. Is this valid? Even though the premises have nothing to do with the conclusion, surprisingly, this is valid. $P \wedge \neg P$ technically satisfies the condition that whenever it is true (never), so is $Q$; it never has the chance to fail our definition. For this we can write $P \wedge \neg P \vDash$ since it will entail everything by our definition, and anything can go on the right hand side of the turnstile, and this is why we shorten all contradicitions accordingly.

Similarly, we can do the same for tautologies, $\vDash P \vee \neg P$ since they will always be true, so certainly whenever whatever on the left is true, so is the tautology.

So we can have weird technically valid arguments like, "The sky is blue and not blue, therefore the Earth is flat," or similarly that "Squares have 5 corners, so either it will rain today or it won't." These both read as absurd, but they are technnically valid with our definition. These might seem like a flaw for our definition, but for the convenience of all the other arguments, I think it is worth accepting as an oddity of the system. You can also think of them as just sort of fact-of-the-matter statements; tautologies are entailed by everything since they don't care what makes you think they are true, they just are. Similarly with contradictions, your starting set of premises are just inconsistent, and it is just a bad argument; if you're willing to accept bad premises, you can deduce bad conclusions.

Proof Systems and Natural Deduction

We have established quite a bit already in $\mathcal{L}_1$, but it is still not that elegant of a system. We now have a way to formalize English arguments, but our ways to validate them are still unwieldy. The only two ways we really have to verify statements is, in essence, brute force checking $\mathcal{L}_1$-structures to see if the argument fails anywhere. This is no better than checking every right-angled triangle to see if it satisfies the Pythagorean theorem. We want a systematic way to go about our arguments.

We want a proof system: given a set of premises $\Gamma$, we want to be able to manipulate them in some way to conclude $\varphi$.

For example, consider the argument $P \wedge Q, R \vDash P \wedge R$. Without checking any structures and evaluating truth tables, this argument should make sense. If we know $P \wedge Q$, i.e. $P$ and $Q$, we certainly know both $P$ and $Q$ are true individually, so we know $P$. Since we also know $R$ is true, then again it seems obvious we know $P$ and $R$ is also true together i.e. $P \wedge R$. We did not touch any semantics at all of our premises or conclusion (I used the word "true" as a means for accepting a premise), but rather just moved our sentence letters around in a sensible way to get the conclusion.

We just found two rules that make sense to have in our proof system: $\wedge$-Introduction and $\wedge$-Elimination.

So our above proof of $P \wedge Q, R \vdash P \wedge R$ would look like

Thinking through what the other connectives mean can give some intuitive rules for them too:

The square brackets are assumed premises. They are not explicitly given, but, for the sake of argument, you can take a "fake" premise to demonstrate a rule, and discard that assumption after you've used your rule. Some other comments on the rules:

• $\vee$-Intro basically says if you know one sentence, appending any other sentence via $\vee$ is certainly fine since you already have a known fact. $\vee$-Elim says if you know one or the other (or both) of $\Phi$ and $\Psi$ is true, but do not know which, you can deduce a new sentence $\Delta$ if it follows from either of the sentences.
• $\neg$-Rules are basically proofs by contradiction. If $\Phi$ results in both $\Psi$ and $\neg\Psi$, $\Phi$ can't be held true since it would prove contradictory results. Similarly with assuming $\neg\Phi$.
• $\rightarrow$-Intro follows in line with for-the-sake-of-argument. If we assume $\Phi$, and can deduce $\Psi$, then regardless if we know $\Phi$ or not, we can say that it would logically imply $\Psi$ as well.

This specific proof system is known as natural deduction. These proof trees give us a direct way to interact with our premises in a way that we might normally do in English. However, some of these proofs do become huge quite quickly. Consider the proof of $P\rightarrow (Q\leftrightarrow R) \vdash P \wedge Q \leftrightarrow P \wedge R$:

Some of you might have noticed a small change in notation; I've been using $\vdash$ as opposed to $\vDash$. In proofs, since we are not considering structures and the truth values of our sentences, semantic entailment does not seem like the right way to describe the relationship between our premises and the conclusion. We now say $\Gamma \vdash \varphi$ if $\Gamma$ syntactically entails, or simply proves via our proof system, $\varphi$, since we are strictly only concerned with the symbols and the actual writing of the sentence as opposed to its truth.

But we should not forget our original goal of logic: to demonstrate arguments are valid, which involves $\Gamma \vDash \varphi$, which seems to have a very different meaning to $\Gamma \vdash \varphi$. It happens that our choice of rules above was not random. As we will see later, natural deduction is a sound system:

Soundness Theorem: If $\Gamma \vdash \varphi$, then $\Gamma \vDash \varphi$.

Soundness can be thought of as the quality of preserving truth; we can rely on our proofs to deduce meaningful, accurate conclusions in our formal language. Given our rules, we would hope that these are sound, since they are based on how people typically argue in real life, too. You might come across other proof systems, but I can guarantee you, they are designed to be sound and useful.

It's worth mentioning at this point we have similar notions for consistency in a proof based sense just as we have in a semantic sense. We say a set of sentences $\Gamma$ is ND-consistent (ND for natural deduction) if there exists a sentence $\Phi$ that $\Gamma$ does not prove; i.e. $\exists \Phi$ s.t. $\Gamma \nvdash_{ND} \Phi$. In general, we say $\Gamma$ is D-consistent if there is a sentence $\Gamma \nvdash_{D} \Phi$ in the deductive system $D$. A good way of thinking about this is if we have bad premises, we can deduce anything since our assumptions were contradictory in some way. For example, $\{P, \neg P\}$ is ND-inconsistent since by $\neg$-Elim, it can prove anything.

Fortunately enough for us, ND-consistency is equivalent to semantic consistency, but we'll see that later.

This concludes our discussion of $\mathcal{L}_1$. The system and rules we've outlined above is sometimes also referred to as propositional logic.

$\mathcal{L}_2$: The Logical Quantifiers

Let's go back to our very first argument we discussed: "If Socrates is a man, then he is mortal. Socrates is a man. Therefore, Socrates is mortal." We were able to show it is valid in $\mathcal{L}_1$ by formalizing it, and either checking its truth table or with our proof system. Let's look at a very similar argument. "All men are mortal. Socrates is a man. Therefore, Socrates is mortal." Perfectly reasonable. If we formalize this with our current logic, though, we end up with $P, Q \vDash R$, where $P$ is "All men are mortal," $Q$ is "Socrates is a man", and $R$ is "Socrates is mortal". Clearly this is not a logical argument in $\mathcal{L}_1$, since we can just specify a structure where $|P|_A = |Q|_A = T$ and $|R|_A = F$ since they are separate sentence letters.

Predicates and Constants

$\mathcal{L}_1$ is good for keeping sentences simple, but it lacks the nuance of the predicates English has; we will never be able to show the relation of men and mortality, as we can't express "…are mortal" nicely; relations between simpler sentences are covered by connectives, but relations between objects are not. We need something to convey properties somehow. Ideally, we want to be able to say Socrates is mortal, by first saying he is a man, and to further show that anything that has the property of being a man also has the property of being mortal.

To formalize "Socrates is a man", we can introduce predicates kind of like connectives. Let's have $P^1$ stand for "…is a man", and for convenience, we'll let the letter $|a|_A$ stand for the constant Socrates (remember, $|\cdot|_A$ denotes meaning). Then $P^1a$ is a nice compact way to write out "Socrates is a man".

Now how do we formalize that this is a true sentence? We don't want to have to explicitly say $|P^1a|_A = T$, since then we would have to specify that for everything that is a man, there is a constant associated with that human and that when put together with $P^1$ forms a true statement.

Let's take a step back for a moment. If you had to explain to an alien precisely what a human is, how would you do it? You might describe some key traits for them, but the easiest way, in a universal sense, is just to show them by example. Give them a list of humans and have them extrapolate on their own. So perhaps a good semantics for predicates is to exactly that: we will let $|P^1|_A = \{\textrm{Socrates}, \textrm{Feynman}, \textrm{Noether},…\}$.

Then a natural way of defining truth would be that $|P^1a|_A = T$ if and only if $|a|_A \in |P^1|_A$. That sentence makes sense, since $|a|_A$ is an object/noun, and $|P^1|_A$ is a set of objects, so we can evaluate that, and since $|P^1|_A$ explicitly states all the objects that have that property, seeing if something has that property (like being a man) is a matter of being in the list of objects said to have it. "Socrates is a man" is true iff "Socrates" has the property "is a man" iff Socrates is in the set of all men.

I've been putting a small superscript in our predicate $P^{\color{red}{1}}$. This is the arity index, and basically says how many objects the predicate takes as an input. So, naturally we can have indices greater than 1 as well.

The predicate "…like…" is a perfectly good predicate, but it now qualifies two objects and relates them to each other. So if Alice likes Bob, we can write this as $P^2ab$ with the relevant meanings for $P^2$ as "…likes…" and the constants referring to Alice and Bob respectively. But note here, order matters. Just because Alice likes Bob, that does not necessarily mean Bob likes Alice and so for some structures it might be the case that $|P^2ab|_A \neq |P^2ba|_A$.

Just as arity 1 predicates had a set of objects as its semantics, arity 2 has a set of ordered pairs i.e. $|P^2|_A = \{\langle \textrm{Alice}, \textrm{Bob}\rangle, \langle \textrm{Davis}, \textrm{Edward} \rangle, \langle \textrm{Jack}, \textrm{Jack} \rangle \}$ would be a reasonable way to denote the predicate. And similarly, we say $|P^2ab|_A = T$ if and only if $\langle |a|_A, |b|_A \rangle \in |P^2|_A$. So 2-ary predicates define binary relations.

In practice, the arity index usually doesn't need to be written, as it will be implied by the number of constants attached to the predicate i.e. $Pabc$ is clearly a ternary predicate, as if it wasn't, it would either be incomplete ("a hates b and c, but likes ?"), or it would accept too many arguments ("If a then b"; where does c go?). So in sentences like $Pa \wedge Pab$, each $P$ stands for a different predicate as implied by their arity.

This is the start of the more robust logic $\mathcal{L}_2$. Just for the formalities, here are some things we need to keep in mind over $\mathcal{L}_1$.

• Sentence letters are the same as 0-ary predicates; they accept no arguments.
• In an $\mathcal{L_2}$-structure, we assign semantics to every predicate (either truth values or sets to predicates), as well as an object to every constant
• Connectives work the same as they have done before
• Truth for predicates are equivalent to membership of a set

Quantifiers

So we now have a good way of storing properties of objects via the predicates they satisfy, so we have a good way of saying "Socrates is a man". But now we need a way of saying, "All men are mortal". It's that "all" that makes this difficult, since it does not refer to a specific man, but a generic one.

Let's generalize the sentence a bit more. What it really is saying, "If one is a man, then they are mortal." Now we have this nice "one" acting a placeholder name for our generic man. Actually, for a generic anything. If we are talking about a turtle, we cannot say if they are mortal by this sentence since they have nothing to do with that claim.

In an even more mathematical way of looking at the sentence, it could read as, "For all $x$: if $x$ is a man, then $x$ is mortal." That "for all" appears enough in arguments, we have formalize it as $\forall$, the universal quantifier. You might have seen this appear in math, and if not, I've already been abusing its convenience in some previous proofs on this post.

So our sentence would look like $\forall x(Px \rightarrow Qx)$ where $P$ and $Q$ are predicates for being a person and mortal respectively.

$x$ is clearly not a constant, since, well, it is not constant; it does not have a fixed meaning. We then aptly call it a variable. But clearly, variables can take on certain meanings, so we also have variable assignments $\alpha, \beta,…$ that allow us to arbitrarily assign variables meanings in a formal way. For example, if we let $x$ mean Socrates in our structures, i.e. $|x|_A^\alpha = \textrm{Socrates} = |a|_A$, then our argument works by the truth rules for $\rightarrow$.

The "opposite" of the universal quantifier is the existential quantifier $\exists$, which instead of saying our sentence applies to all possible objects, $\exists x \Phi$ only claims that there is at least one object that satisfies the sentence $\Phi$.

The duality of the quantifiers can be found even in natural language. "Everything is not a cow," as in $\forall x \neg Px$, is the same as saying, "There does not exist a cow", or $\neg \exists x Px$. Similarly, "There is something that is not a cow," or $\exists x \neg Px$, is equivalent to saying "Not everything is a cow" or $\forall x \neg Px$.

However, not every formula with variables and quantifiers make a sentence. Consider $\forall x \exists y(Px \vee \neg Qxy \rightarrow Rz)$. This doesn't make that much sense since $z$ is just hanging out as a free variable as there is no quantifier claiming anything about $z$. And since $z$ is a variable, it's not like it has any semantic meaning, either. So we wouldn't call this a sentence, but a formula. Variables that are not free are bound.

Definition: A formula is a sentence if and only if it has no free variables.

Similarly, $\forall y Qy \rightarrow Py$ wouldn't be called a sentence either, as depite how it looks, there is a free variable in $Py$ even though there is a $\forall y$ in the formula. You usually need parantheses to specify what the quantifiers bind. $\forall y(Qy \rightarrow Py)$ is a sentence, while the previous expression is not.

Domains

The final extra detail we need is to specify what meanings our variables are allowed to be assigned. So in an $\mathcal{L}_2$-structure, we also specify a domain that specifies all the possible objects in the "universe" that we are discussing. All constants are assigned meanings from the domain, and our quantifiers use variable assignments that range over the domain.

Satisfaction and Truth in $\mathcal{L}_2$

Suppose $\Phi$ and $\Psi$ are $\mathcal{L}_2$ formulas, $v$ is a variable, $A$ is a structure, and $\alpha$ is a variable assignment. We say $\alpha$ satisfies $\varphi$ under $A$, or $|\varphi|_A^\alpha = T$, in the following way:

• $|\Phi v_1v_2…v_n|_A^\alpha = T$ iff $\langle |v_1|_A^\alpha, |v_2|_A^\alpha,…, |v_n|_A^\alpha \rangle \in |\Phi|_A^\alpha$ where $\Phi$ is an n-ary predicate and each $v_i$ is a variable or constant
• $|\neg\Phi|_A^\alpha = T$ iff $|\Phi|_A^\alpha = F$
• $|\Phi \wedge \Psi|_A^\alpha = T$ iff $|\Phi|_A^\alpha = T$ and $|\Psi|_A^\alpha = T$
• $|\Phi \vee \Psi|_A^\alpha = T$ iff $|\Phi|_A^\alpha = T$ or $|\Psi|_A^\alpha = T$
• $|\Phi \rightarrow \Psi|_A^\alpha = T$ iff $|\Phi|_A^\alpha = F$ or $|\Psi|_A^\alpha = T$
• $|\Phi \leftrightarrow \Psi|_A^\alpha = T$ iff $|\Phi|_A^\alpha = |\Psi|_A^\alpha$
• $|\forall v \Phi|_A^\alpha = T$ iff for every variable assignment $\beta$ that differs from $\alpha$ in at most $v$, $|\Phi|_A^\beta = T$; in other words, $|\forall v \Phi|_A^\alpha = T$ iff $\Phi$ is satisfied when every thing except $v$ is fixed
• $|\exists v \Phi|_A^\alpha = T$ iff there is at least one variable assignment $\beta$ that differs from $\alpha$ in at most $v$ where $|\Phi|_A^\beta = T$; in other words, $|\exists v \Phi|_A^\alpha = T$ iff $\Phi$ is satisfied for some value of $v$

The normal connective satisfaction come from the standard truth tables we did before, and the quantifiers are exactly what you would imagine them to mean, just formalized in the new terminology we have to make it precise.

The truth of an $\mathcal{L}_2$ sentence is now easy given the above satisfaction rules:

Definition: An $\mathcal{L}_2$ sentence $\varphi$ is true $|\varphi|_A = T$ iff $|\varphi|_A^\alpha = T$ for every variable assignment $\alpha$ over $A$.

Validity remains the same in $\mathcal{L}_2$, just with our updated definitions for structures and truth:

Definition: Given a set of $\mathcal{L}_2$ sentences $\Gamma = \{\Phi, \Psi,…\}$ and a single sentence $\varphi$, we say that $\Gamma$ semantically entails $\varphi$ if and only if for all $\mathcal{L}_2$-structures $A$, if $A \vDash \Gamma$, then $A \vDash \varphi$.

Natural Deduction 2.0

Rules for our original connectives $\wedge, \vee, \neg, \rightarrow, \leftrightarrow$ remain the same. We only need to add the rules for $\forall$ and $\exists$. It will be easier to write them out then explain them afterwards.

The notation of $\Phi[t/v]$ is to highlight substitution: $\Phi[t/v]$ is the formula $\Phi$ with all occurrences of $v$ in it replaced with $t$. For example, if $\Phi = Px \wedge \forall y (Qxy \rightarrow Rx)$, then $\Phi[t/x] = Pt \wedge \forall y (Qty \rightarrow Rt)$.

Here is the intuition behind the new rules:

• $\forall$-Intro basically says if $t$ is a constant that does not appear anywhere else in our proof, it is in practice a "dummy variable", and allows us to introduce a proper variable bound by $\forall$.
• $\forall$-Elim says if we know $\Phi$ is satisfied by possible $v$, it is certainly true if $v$ is replaced by a specific $t$; if all men are mortal, certainly Socrates is mortal too
• $\exists$-Intro is the formal way of saying if we see an occurence of $\Phi$, then there is some $v$ that satisfies $\Phi$; if you see a cow, you know there is at least one animal
• $\exists$-Elim is a for-the-sake-of-argument style proof. While we may not know exactly what satisfies $\exists v \Phi$, if we can deduce some claim independent of the specific thing that satisfies it, we know that claim must also be true as it only relies on the existence of something. If we know there is at least one plant-eating animal—call it Bob—we can conclude that there are plants, since Bob eats plants, and that's true for all "Bob"s (a.k.a. plant-eating animals).

With these added rules, we can now prove a lot of the things we already knew, such as:

• The mortality argument of $\forall x(Px \rightarrow Qx), Pa \vdash Qa$
• The quantifier interplay $\neg \forall x Px \dashv\vdash \exists x \neg Px$
• Standard tautologies and contradictions; $\vdash \forall x Px \vee \neg \forall x Px$, and $\forall x Px \wedge \neg \forall x Px \vdash$

Here is an example worked proof from the additional exercises of The Logic Manual of $\forall y \exists x(Ryx \vee Qyx) \vdash \forall y (\exists x Ryx \vee \exists x Qyx)$:

Similarly, we will see that these rules are also sound.

$\mathcal{L}_=$: Identity and Definite Descriptions

$\mathcal{L}_2$ gets us most of the way for what we want but there's one slight inconvenience: we don't have a notion of identity. Say we have a constant $a$ with semantic value $|a|_A = \textrm{Socrates}$. Then, if we have another constant $b$ such that $|b|_A = \textrm{Socrates}$, we have no good way of showing these constants are really "the same"; they might look different, but they function in the same way as both standing in for Socrates.

And in a philosophical, sense, identity is an important concept in ontologies and metaphysics, so it's worthwhile having a formal notion that we can refer to when picking apart arguments.

We could just specify the predicate, "…is identical to…" as a binary predicate letter in $\mathcal{L}_2$, but that means that the predicate might receive arbitrary meanings in different structures, and identity seems like a pretty unshakeable concept. It also might mean that we use different predicate letters in different formalizations, and we want this to be consistent for it to be useful.

This is what $\mathcal{L}_=$ is for. $\mathcal{L}_=$ is exactly like $\mathcal{L}_2$ in terms of structures and truth, but it has the one difference of having an extra predicate of $=$ to refer to identity. Some quick notes:

• As $=$ is a binary predicate, it relates two terms that are constants or variables (not formulas or sentences!)
• By convention, we write $a=b$ as per the norm of math; we don't write $=ab$ as we would for a normal predicate/predicate letter
• For variables/constants $s$ and $t$, $|s=t|_A^\alpha = T$ iff $|s|_A^\alpha=|t|_A^\alpha$ i.e. they share the same semantic value and represent the same object

Natural Deduction 2.5

We don't have much else to add alongside $=$, and the new rules we use are among the most intuitive of the lot.

• For all constants $a$, it is clear that $a$ is identical to itself, so the introduction rule allows us to say that whenever we want for any constant
• If $s$ and $t$ are identical, they can be swapped freely in formulas without hurting the underlying semantics, so the elimination rule also makes sense

Using these rules along side our normal natural deduction, try to prove that $=$ is an equivalence relation, that is

• It is reflexive: $\vDash \forall x (x=x)$
• It is symmetric: $\vDash \forall x \forall y (x=y \rightarrow y=x)$
• It is transitive: $\vDash \forall x \forall y \forall z (x = y \wedge y = z \rightarrow x = z)$

just as we would expect of normal identity.

Numerical Quantifiers and Definite Descriptions

With identity in place, we can now establish a surprising amount of additional, useful claims regarding the specifity of the objects we are talking about.

Numerical Claims

If we wanted to say, "There is at least 1 chicken", we can already do that with $\exists$ by letting $|P| = \textrm{"…is a chicken"}$ and the formalization $\exists x Px$. This is by definition of what it takes to satisfy $\exists$: it requires only one variable assignment, i.e. object in the domain, to be in the set of chickens that $P$ expresses.

If we want to say there are at least two chickens, then we can write there are two things that are chickens i.e. $\exists x \exists y(Px \wedge Py)$. But this alone does not guarantee there might be two chickens, since if there is one chicken, then the variable assignment where both $x$ and $y$ both refer to the same chicken would satisfy that sentence. So we add the clause that they are distinct: $\exists x \exists y(Px \wedge Py \wedge \neg x = y)$; the $\neg x = y$ prevents our variable assignments allowing $x$ and $y$ to be the same if we want

At least 3 chickens would be like $\exists x \exists y \exists z(Px \wedge Py \wedge Pz \wedge \neg x = y \wedge \neg x = z \wedge \neg y = z)$. At least $n$ chickens, would be $\exists x_1 \exists x_2 \cdots \exists x_n(\bigwedge_{i=1}^n Px_i \wedge\bigwedge_{1 \leq i < j \leq n} \neg x_i = x_j)$ where $x_i$ are variables and $\bigwedge$ is like $\sum$ where each term instead of being added is linked by the $\wedge$ connective.

Similarly, if we want at most 1 chicken, that is the same as saying the opposite of at least 2 chickens; the complement of $\geq 2$ is $$<2$$ or $\leq 1$. So we can just negate our sentence from before: $\neg \exists x \exists y(Px \wedge Py \wedge \neg x = y)$. A cleaner (and more intuitive way) to formalize this, in my opinion is the following: $\forall x \forall y(Px \wedge Py \rightarrow x = y)$; for every 2 supposedly different chickens, they are actually the same.

At most 2 chickens would be $\forall x \forall y \forall z(Px \wedge Py \wedge Pz \rightarrow x = y \vee x = z \vee y = z)$; if there are 3 supposedly different chickens, actually at least 1 is a duplicate of the other 2. For at most $n$ chickens: $\forall x_1 \forall x_2 \cdots \forall x_{n+1}(\bigwedge_{i=1}^{n+1} Px_i \rightarrow \bigvee_{1\leq i < j \leq n+1} x_i = x_j)$

Definite Descriptions

We can set lower bounds on how many chickens there are, and upper bounds on how many chickens there are, so we are now in place to formalize when there is exactly one chicken. For there to be only one supreme chicken, we just need to specify there is at least one and at most one chicken: $\color{blue}{\exists x Px} \wedge \color{red}{\forall x \forall y(Px \wedge Py \rightarrow x = y)}$. An even nicer expression is $\exists x (Px \wedge \forall y (Py \rightarrow y = x))$; there is something that is a chicken, and all other chickens are really the same as that original, defining chicken.

With this, we can now formalize definite descriptions. When we mention "The president of the United States", we are referring to a general position i.e. many people were president from George Washington to Joe Biden. However, there is only one current president of the U.S., and we are now able to differentiate the two with our formalizations above. "The president is a democrat" would be written as $\exists x (Px \wedge \forall y (Py \rightarrow y = x) \wedge Qx)$ where $P$ expresses "…is the president of the U.S." and $Q$ for "…is a democrat". In general to express, "The $\Phi$ has the property $\Psi$," we could use the following formula:

Russell's theory of definite descriptions: $\exists x (\Phi \wedge \forall y (\Phi[y/x] \rightarrow y = x) \wedge \Psi)$

With definite descriptions in hand, we can now formally prove arguments that naturally make sense in English. "Tim's car is red. Therefore there is a red car." Easy enough to see in English, but requires a bit more work to express in $\mathcal{L}_=$:

Generalized Numerical Quantifiers

We were able to specify exactly 1 thing satisfies a property, but we can generalize to exactly $n$ with a simple recursive definition. To express there are exactly $n$ things satisfying a formula, we will denote it with $\exists!_n$, given by the following definition:

• $\exists!_0 v \Phi = \neg \exists v \Phi$
• $\exists!_{n+1} v \Phi = \exists u (\Phi[u/v] \wedge \exists!_n v(\Phi \wedge \neg u = v))$

Exactly 0 things satisfy the formula $\Phi$ when there does not exist anything that satisfies it. Exactly $n+1$ things satisfy $\Phi$ when there is something that satisfies $\Phi$ and exactly $n$ distinct, other things that satisfy $\Phi$ as well. As a gut check, we can test to see if this agrees with our definition from definite descriptions from before.

By a series of logical equivalences, we can see that we get precisely what we expect.

A Quick Aside on Identity

We've been using identity loosely so far, since what we mean by "identical" actually varies, whether you realize it or not.

If we say my friend and I have identical cars, we don't mean that my friend and I literally own the same car, but rather the make, the model, the color, etc. all look identical. This is an example of (approximate) qualitative identity, where two objects share the same properties. However, if you say the police tell you your car is the same one that was seen at a crime scene, then we mean it is literally the one and the same car. This is numerical identity.

We have to be careful about this since these two types of identities clearly seem disparate. While numerical identity seems to imply qualitative identity (one thing will always share the same properties as itself), it is debated if qualitative identity really implies numerical identity. If two things really share all the same properties, it becomes unclear whether or not that really implies we are talking of the same object.

The language of $\mathcal{L}_=$ focuses on the latter numerical identity, even though the former might be the more convenient one in everyday speech.

Summary of our Logics

We've covered a lot, so let's quickly recap some of the key ideas covered so far:

• Logic studies valid arguments; what conclusions can we draw by nature of the structure of our argument?

For $\mathcal{L}_1$:

• We had basic sentence letters, which we made more complex with the connectives $\{\wedge, \vee, \neg, \rightarrow, \leftrightarrow\}$, each with their own set of truth rules
• We modelled the universe in the manner of structures, saying which sentences we said to be true and which to be false
• Alongside it came its own proof system, natural deduction, that allowed us to validate arguments in a more systematic way beyond brute force checking

For $\mathcal{L}_2$:

• We lacked the nuance of relating objects to each other in $\mathcal{L}_1$, so we introduced constants to formalize those objects with properties stored as predicates
• We updated our model of the universe with specifying constants and predicates (unary, binary, ternary, and n-ary relations)
• The universal $\forall$ and existential $\exists$ quantifiers were a nice shortcut to state claims of how many things in the domain satisfy a formula
• We updated our proof system to account for these quantifiers

For $\mathcal{L}_=$:

• The notion of numerical identity leads to obvious truths that would be convenient to introduce
• We expanded $\mathcal{L}_2$: with the new binary predicate $=$ to show when two constants are essentially the same despite being written with different letters
• Included the final set of rules for natural deduction to accommodate $=$
• Could now make numerical claims

This completes everything you need to know about first-order logic (more on this later), sometimes referred to as predicate calculus.

Part 2: Introduction to Metatheory

What we've covered before gives us a pretty reliable way to analyze English arguments, which philosophers really want to make sure they aren't tripping over themselves when discussing and debating ideas. But there are a whole host of patterns and results within logic itself that arguably have a bigger impact on the rest of science and deductive reasoning.

For example, consider the following theorem:

Deduction Theorem: $\Gamma, \Psi \vDash \Phi$ if and only if $\Gamma \vDash \Psi \rightarrow \Phi$.

By thinking through the definitions and terms we are working with, we can treat this almost as we would a mathematical theorem, and prove it analytically.

Proof: If $\Gamma, \Psi \vDash \Phi$, then by consistency (and definition of entailment), $\Gamma, \Psi, \neg \Phi \vDash $. Note that ony whenever the set $\{\Psi, \neg \Phi\}$ is satisfied, so is $\Psi \wedge \neg \Phi$. So we can rewrite our sequent as an equivalent one as $\Gamma, \Psi \wedge \neg \Phi \vDash$. By logical equivalence, then $\Gamma, \neg(\Psi \rightarrow \Phi) \vDash $. Then by consistency again, we arrive at $\Gamma \vDash \Psi \rightarrow \Phi$. $\blacksquare$

We are able to determine a result about arguments in general. We don't know what our premises and conclusion are, yet we can still make substantive claims regarding logical arguments of that form.

Here is another simple example that is commonly used all throughout science and math:

Contraposition: $\Phi \vDash \Psi$ if and only if $\neg \Psi \vDash \neg \Phi$.

"If it is Monday, then I have math class. So if I don't have math class, it certainly cannot be Monday." That is the general idea of argument by contraposition: if we know what premise entails a conclusion, and we don't observe the conclusion, the premise must not be fulfilled either.

Proof: Suppose $|\neg \Psi|_A = T$. Then $|\Psi|_A = F$. Since $\Phi \vDash \Psi$, then $|\Phi|_A = F$ too since if it didn't we would break our assumption of enatilment. Thus $|\neg \Phi|_A = T$. We made no assumptions about the structure $A$, and thus found whenever $|\neg \Psi|_A = T$, then also $|\neg \Phi|_A = T$. By the definition of entailment, $\neg \Psi \vDash \neg \Phi$.

We've already proven two useful theorems! Now, we will develop and practice using more tools to argue with logic as its own field of study rather than just an auxiliary field to other sciences.

Principle of Mathematical Induction

The major tool we will need is induction. For a hypothesis $\Phi$, the Principle of Mathematical Induction (PMI) is usually stated as:

The common analogy is with dominoes: if you knock over a starting domino (show hypothesis $\Phi$ works for integer like 0), and can prove that each domino will knock over the next domino (i.e. the hypothesis holding for $\Phi(n)$ implies the hypothesis holds for $\Phi(n+1)$), then you can infer that all the dominoes will fall over eventually (the hypothesis holds for all integers $\forall n \Phi(n)$).

Here is a classic example: we will show the formula for the sum of the first $n$ integers is $\sum_{k=1}^n k = \frac{n(n+1)}{2}$.

Base Case: The formula holds for $n=1$ since $\sum_{k=1}^1 k = 1 = \frac{1(1+1)}{2}$.

Inductive Hypothesis: Let's assume our formula works for the first $n$ integers. We want to show that it holds for case $n+1$: $\sum_{k=1}^{n+1} k = \sum_{k=1}^{n} k + (n+1)$. By our assumption, we can reduce this to $\frac{n(n+1)}{2} + n+1$. Simplifying further,

Which is precisely the formula our hypothesis predicted. So by the PMI, for all $n$, $\sum_{k=1}^n k = \frac{n(n+1)}{2}$.

Those who are more familiar with math will have seen examples of induction everywhere, and it is quite good for proving theorems and claims that naturally divide into these cases-by-integers.

One drawback of induction is that it does not tell you how you get your hypothesis to begin with. That usually requires trial-and-error with a strong intuition, but once you have it, induction gives a relatively straightforward way of proving your claim (if it is right).

The formulation above is sometimes referred to as weak induction. The alternative is strong induction:

Sometimes you need more cases besides just $\Phi(n)$ to prove $\Phi(n+1)$, so strong induction is a way to justify that. Despite the names, the weak and strong forms of induction are equivalent, so we can use either whenever.

With induction in hand, we can now start proving results within logic.

Example: Any $\mathcal{L}_1$ sentence $\Phi$ that only uses the connective $\leftrightarrow$ is not a contradiction i.e. there's a structure $A$ in which $|\Phi|_A = T$.

Proof: We will show this by giving a specific structure that $\Phi$ is true, and proving it holds via induction. I propose the structure $A$ where all sentence letters in $\Phi$ are assigned true. That is $\forall \alpha \in \textrm{SenLett}(\Phi) \ |\alpha|_A = T$.

Base Case: The simplest possible sentence using just $\leftrightarrow$ is the one not using it at all: a sentence letter. So if $\Phi$ is a sentence letter, by construction of our structure, $|\Phi|_A = T$.

Inductive Hypothesis: We'll induct on the complexity of $\Phi$. So let's build up $\Phi$ from "simpler" sentences; $\Phi = \Psi_1 \leftrightarrow \Psi_2$ where $\Psi_1, \Psi_2$ are both subsentences only containing $\leftrightarrow$. Remember, we have defined our structure $A$ such that $\forall \alpha \in \textrm{SenLett}(\Phi) \ |\alpha|_A = T$. Since $\Psi_1,\Psi_2$ build up $\Phi$, it should be clear that $\textrm{SenLett}(\Psi_{1,2}) \subseteq \textrm{SenLett}(\Phi)$. Then, by our structure, $\forall \alpha \in \textrm{SenLett}(\Psi_{1,2}) \ |\alpha|_A = T$. By our inductive hypothesis then, $|\Psi_1|_A = |\Psi_2|_A = T$, since they are $\mathcal{L}_1$ sentences that only use $\leftrightarrow$. By the truth table for $\leftrightarrow$, $|\Phi|_A = T$. $\blacksquare$

So for all sentences that only use $\leftrightarrow$, there is a structure in which it is true, and hence it is not a contradiction.

Induction naturally lends itself to proofs that are divided by integer cases, and we leveraged that by ascribing a natural number to a sentence via its complexity. Also note we used the strong form of induction here, since we don't actually know the complexity of $\Psi_1$ or $\Psi_2$, only that they are less complex than the inductive hypothesis necessitates. Here are some more formal definitions of what was used above, and others that will be helpful later:

• $\textrm{NConn}(\Phi)$: The number of occurrences of connectives in $\Phi$. We also define this as the complexity of $\Phi$, as it characterizes how "deep" subsentences go.
• $\textrm{Conn}(\Phi)$: The set of connectives used in $\Phi$; $\textrm{NConn}$ counts tokens, while $\textrm{Conn}$ counts types.
• $\textrm{SenLett}(\Phi)$: The set of sentence letters in $\Phi$.
• $\textrm{Atoms}(\Phi)$: The set of atomic objects in our language.

In general, $\textrm{Atoms}(\Phi) = \textrm{SenLett}(\Phi)$ for $\mathcal{L}_1$, but sometimes we will consider convenient extensions of $\mathcal{L}_1$. For one, it's sometimes useful to have a symbol for tautologies $\top$ and contradictions $\bot$. Just for clarity, for all structures $A$, $|\top|_A = T$ and $|\bot|_A = F$. These are considered useful atoms in the extended language $\mathcal{L}_1^+$.

Here's another example of how induction can be applied to logic:

De Morgan's Laws: For sentences $\varphi_1, \varphi_2, \cdots, \varphi_n$,

For those who are familiar with some set theory, these are identical to how set intersections (equivalent to $\wedge$) and unions (equivalent to $\vee$) relate to each other with complements ($\neg$).

Base Case: It can easily be seen by truth tables, or verified with natural deduction that $\neg(\varphi_1 \wedge \varphi_2) \equiv \neg\varphi_1 \vee \neg\varphi_2$ (even in English, it can be seen to be fairly reasonable).

Inductive Hypothesis: Assume this holds for $n$ sentences. Then:

The first equivalence is given by the case for 2 sentences, and the last is given by the case for $n$ sentences by the inductive hypothesis. The second law follows similarly. $\blacksquare$

The last immediate example we'll look at is a simple, yet important lemma.

Relevance Lemma: Suppose for two structures $A$ and $B$, $\forall \alpha \in \ \textrm{Atoms}(\Phi)$ we have it such that $|\alpha|_A = |\alpha|_B$. Then $|\Phi|_A = |\Phi|_B$.

The idea is that the only information pertinent to a logical sentence is what it looks like at the "lowest level"; only the sentence letters in $\Phi$ are what affect its truth value, and nothing else is necessary.

Proof: We'll follow similarly before by inducting on the complexity of $\Phi$.

Base Case: The simplest sentence $\Phi$ can be is a sentence letter. Then by the definition of structures, clearly $|\Phi|_A = |\Phi|_B$. In the case of $\mathcal{L}_1^+$, $\Phi$ can be $\top$ or $\bot$, but since their semantics are fixed for any structure, the claim still holds.

Inductive Hypothesis: Say this holds for sentences of complexity less than or equal to $n$. Now let $\Phi$ be a sentence of complexity $n+1$ i.e. $\textrm{NConn}(\Phi) = n+1$. Then $\Phi$ is of the form $\neg \Psi_1$, $\Psi_1 \wedge \Psi_2$, $\Psi_1 \vee \Psi_2$, $\Psi_1 \rightarrow \Psi_2$, or $\Psi_1 \leftrightarrow \Psi_2$ where $\textrm{NConn}(\Psi_{1,2}) \leq n$ and $\textrm{NConn}(\Psi_1) + \textrm{NConn}(\Psi_2) = n + 1$. If structures $A$ and $B$ agree on the values of the atoms of $\Phi$, they agree on the atoms of $\Psi_1$ and $\Psi_2$. By the inductive hypothesis, $|\Psi_1|_A = |\Psi_2|_B$. Since the truth value of $\Phi$ is fixed by the truth value of $\Psi_{1,2}$, then in all the cases, by their respective truth tables, $|\Phi|_A = |\Phi|_B$. $\blacksquare$

The Relevance Lemma essentially gives us a way of justifying finite truth tables; the reason why we know the sentence $P \wedge Q$ only depends on the truth values for $P$ and $Q$ is precisely because of the Relevance Lemma; so long as two structures agree on the value of $P$ and $Q$, they will agree on $P \wedge Q$ regardless of what the other sentence letters are assigned. It seems like an obvious fact, but is one that needs to be proven anyway.

Induction, especially on the complexity of sentences, will be a key tool in future results.

Truth Functions

For future convenience, we'll introduce the idea of a truth function. Just as you'd imagine, truth functions are like the typical function in math that takes some amount of inputs and spits out a single output. We thus define a truth function as a function $f: \{T,F\}^n \rightarrow \{T,F\}$. We have already seen some simple truth functions in the form of connectives. For example, $\wedge$ has the truth function such that $f_\wedge(T,F) = F$; if you connect a true sentence letter with a false sentence letter by $\wedge$, we get a more complex sentence that evaluates to false.

This leads to the natural idea of sentences expressing truth functions. We say a sentence $\Phi$ expresses a truth function $f$ iff the sentence letters $P_1,\cdots,P_n$ occur in $\Phi$, and $f(|P_1|_A, \cdots, |P_n|_A) = |\Phi|_A$.

Substitution

Earlier we discussed substitution briefly in the context of constants and variables with $\forall$ and $\exists$ and their proof rules. We can generalize substitution even further:

Definition: For a set of sentences $\Gamma$, $\Gamma[\varphi/S]$ is the uniform subsitution of the sentence $\varphi$ for the sentence letter $S$ in every sentence in $\Gamma$. If $S$ does not occur at all in $\Gamma$, then $\Gamma[\varphi/S] = \Gamma$.

For example, if we have the sentence $\Phi = P \rightarrow (Q \vee R) \wedge P$, then with substitution, $\Phi[(R \leftrightarrow \neg Q) / P] = (R \leftrightarrow \neg Q) \rightarrow (Q \vee R) \wedge (R \leftrightarrow \neg Q)$. This applies for sets of sentences as described above.

We then have the following theorem:

Substitution Theorem: If $\Gamma \vDash \Phi$, then $\Gamma[\varphi/S] \vDash \Phi[\varphi/S]$.

Which should sort of make sense; we've been focusing on the validity of argument as something intrinsic to the structure of the argument, as opposed to having anything to do with the specific sentences. So exchanging sentences with another in an argument should be no issue, even if the sentences are more complicated. We'll first prove a helpful lemma:

Substitution Lemma: For a sentence $\Psi$, sentence letter $X$, and a structure $A$, we define a new substitution structure $A_{\Psi/X}$ as follows:

That is, the structure keeps all sentence letters fixed, UNLESS it is $X$, which we assign the value of $\Psi$. Then for all sentences $\Phi$, we have $|\Phi[\Psi/X]|_A = |\Phi|_{A_{\Psi/X}}$.

Proof: We'll do this by induction on the complexity of $\Phi$ as usual.

Base Case: Simplest sentence is a sentence letter. Then by construction of $A_{\Psi/X}$, $|\Phi|_{A_{\Psi/X}} = |\Phi[\Psi/X]|_A$.

Inductive Hypothesis: Suppose this holds for sentences of complexity $n$. Let $\Phi$ be of complexity $n+1$. Then $\Phi$ is of the form $\neg \delta_1$, $\delta_1 \wedge \delta_2$, $\delta_1 \vee \delta_2$, $\delta_1 \rightarrow \delta_2$, or $\delta_1 \leftrightarrow \delta_2$ where $\textrm{NConn}(\delta_{1,2}) \leq n$ and $\textrm{NConn}(\delta_1) + \textrm{NConn}(\delta_2) = n + 1$. We can then check this by each case.

$\underline{\Phi = \neg \delta}$: $|\Phi|_{A_{\Psi/X}} = |\neg \delta|_{A_{\Psi/X}} = \color{red}{f_{\neg}(|\delta|_{A_{\Psi/X}}) = f_{\neg}(|\delta[\Psi/X]|_A)} = |\neg \delta[\Psi/X]|_A = |\Phi[\Psi/X]|_A$

$\underline{\Phi = \delta_1 \wedge \delta_2}$: $ \begin{align} |\Phi|_{A_{\Psi/X}} = |\delta_1 \wedge \delta_2|_{A_{\Psi/X}} = \color{red}{f_{\wedge}(|\delta_1|_{A_{\Psi/X}}, |\delta_2|_{A_{\Psi/X}})} \ & \color{red}{= f_{\wedge}(|\delta_1[\Psi/X]|_{A}, |\delta_2[\Psi/X]|_{A})} \\ & = |\delta_1[\Psi/X] \wedge \delta_2[\Psi/X]|_A \\ & = |(\delta_1 \wedge \delta_2)[\Psi/X]|_A \\ & = |\Phi[\Psi/X]|_A \end{align} $

The equality in red is given by the inductive hypothesis, and the rest by definitions of the connectives' truth functions and substitution. The other cases are all identical. $\blacksquare$

We can now prove the Substitution Theorem.

Proof: Suppose for contradiction that $\Gamma \vDash \Phi$ while $\Gamma[\varphi/S] \nvDash \Phi[\varphi/S]$. That is, there is a structure where $A$ where $\forall \gamma \in \Gamma \ |\gamma[\varphi/S]|_A = T$ while $|\Phi[\varphi/S]|_A = F$.

By the Substitution Lemma, there is another structure $B$ where $|\gamma|_B = |\gamma[\varphi/S]|_A = T$ and $|\Phi|_B = |\Phi[\varphi/S]|_A = F$. But that contradicts our original assumption that $\Gamma \vDash \Phi$. So it must be the case that $\Gamma[\varphi/S] \vDash \Phi[\varphi/S]$. $\blacksquare$

Disjunctive Normal Form and Expressive Adequacy

While we have the connectives $\wedge,\vee,\neg,\rightarrow,\leftrightarrow$, there's a natural question as to whether or not we need all of them.

Disjunctive Normal Form

Consider the following sentence: $\neg((P \wedge Q \rightarrow R) \leftrightarrow \neg Q)$. Let's write out the truth table for this sentence.

This sentence is fairly compact, but also not very readable. We can instead rewrite this sentence into something slightly longer, but logically equivalent that is indicative of what this sentence is saying.

Notice that this sentence is true if and only if one of the three types of structures are in use:

1. $P,Q,R$ are true
2. $P$ is false (or $\neg P$ is true), and $Q,R$ are true
3. $P$ is false (or $\neg P$ is true), $Q$ is true, and $R$ is false (or $\neg R$ is true)

These correspond to the highlighted rows in the truth table. We can express these truth conditions of the sentence easily then by reading off each condition:

Whenever the red structure is in use, then our original sentence is true. Likewise, the first disjunct in the above sentence is also true, making the whole sentence true by the truth table for $\vee$. Similarly, when the blue or green structures are in use, our original sentence is true and so is our new sentence made of just $\neg$, $\wedge$, and $\vee$. And when none of those structures are used, then our original sentence is false, and since our new sentence is made from only the conditions that make our original sentence true, it too would be false. Since these two sentences are true in the same structures and false in all the same structures, these sentences are logically equivalent.

We call this second sentence the disjunctive normal form of the original sentence.

Definition: A sentence is in disjunctive normal form (DNF) if it only uses connectives from the set $\{\neg, \wedge, \vee \}$, and that $\vee$ never occurs in the scope of $\neg$ or $\wedge$, and $\wedge$ never occurs in the scope of $\neg$. Informally, a sentence is in DNF if it is a conjunction of disjunctions of sentence letters or their negations.

Our above method gives a constructive way on how to convert sentences into DNF using truth tables. A more formal version of the proof specifying structures is given in Eagle's Elements of Deductive Logic, but the idea remains the same even if obscured by notation. A sentence $\Phi$ can be written in DNF by writing

where each Case is the listing of the truth conditions needed given by the sentence letters.

The alternative to DNF is the conjunctive nomal form (CNF), where you can write a logically equivalent sentence using a conjunction of disjunctions of sentence letters and or their negations. The proof that any sentence can be written in CNF can be done similarly to the proof of DNF. The idea is a sentence $\Phi$ is true if and only if none of the structures that make it false are in use.

But we also can use De Morgan's Laws and the existence of DNF to prove the existence of CNF much quicker. First, write $\neg \Phi$ in DNF: $\neg \Phi \equiv \bigvee_{n=1}^m (\bigwedge_{i=1}^k \mathscr{P}_i)$ where $\mathscr{P}_i$ is a sentence letter or its negation. This would say that there are $k$ sentence letters in $\neg \Phi$, and that you only need $m$ disjuncts to express it. But, since $\neg \neg \Phi \equiv \Phi$, we can negate $\neg \Phi$ in DNF to get $\Phi$ in CNF:

which is precisely in CNF, with $m$ conjuncts and all the sentence letters and their negations negated.

Expressive Adequacy

We have seen above that we can convert any sentence into a logically equivalent one using only $\neg$, $\wedge$, and $\vee$. However, note the only thing we really needed was the truth table of a sentence, not the sentence itself. So if we outline any truth function via a truth table, we can find a sentence in DNF that expresses that truth function.

So we say that the set of connectives $\{\neg, \wedge, \vee \}$ is expressively adequate: any truth function can be expressed by a sentence using exclusively the connectives $\{\neg, \wedge, \vee \}$. In other words, the truth functions of $\{\neg, \wedge, \vee \}$ can make any other truth function under composition with each other. We proved that with the existence of DNF.

Then, in a way, we have some redundant connectives in $\mathcal{L_1}$: $\rightarrow$ and $\leftrightarrow$ can be expressed using $\{\neg, \wedge, \vee \}$.

Of course, it's much more convenient to use $\rightarrow$ and $\leftrightarrow$ than their equivalent forms, but in a way, they are just that—a convenience.

But we can do better. We know from De Morgan's Laws:

So even $\wedge$ and $\vee$ is redundant; you only need one of them with negation to express the other.

So we have two small expressively adequate sets already with $\{\neg, \vee\}$ and $\{\neg, \wedge\}$, but there are many others, such as $\{\neg, \rightarrow \}$ and $\{ \rightarrow, \bot \}$. There are even single connectives that are expressively adequate. Consider the connectives with the following truth tables:

$\uparrow$ is sometimes called nand (as in not/negated-and) while $\downarrow$ is nor (not-or). Since we know $\{\neg, \wedge \}$ is expressively adequate, and that

so $\{\uparrow \}$ can express the truth functions of the expressively adequate connective set $\{\neg, \wedge \}$. Thus, $\{\uparrow \} $ is expressively adequate.

Here are some good exercises to try and fully wrap your head around expressive adequacy:

• Show that $\{\downarrow \}$ is expressively adequate as well (hint: try to write another expressively adequate set in terms of just $\downarrow$).
• Show $\uparrow$ and $\downarrow$ are the only 2-place connectives that are expressively adequate by themselves.
• Consider the 3-place connective $\square(\Phi, \Psi, \Delta)$ which is true iff $\Phi,\Psi,\Delta$ are all false. Show $\{\square\}$ is expressively adequate.
• Show $\{\neg\}$ is not expressively adequate (hint: how many inputs can a sentence only using $\neg$ have?)
• Show $\{\vee, \wedge, \rightarrow, \leftrightarrow\}$ is not expressively adequate (hint: find a specific truth function it cannot express).
• Consider the new connective $\leftrightarrow^*$ with the following truth table: $ \begin{array}{c|c|c} P & Q & (P \leftrightarrow^* Q) \\ \hline T & T & F \\ T & F & T \\ F & T & T \\ F & F & F \\ \end{array} $ Show $\{\leftrightarrow, \leftrightarrow^*\}$ is not expressively adequate.

Duality

We have seen many times that there's a natural interplay between between $\wedge$ and $\vee$ with patterns of negation with De Morgan's Laws, and as well as between $\forall$ and $\exists$. Even in English, we see this mixing of "internal" and "external" negations; "already outside" means the same thing as "not still inside", so "already" and "still" act in this "dual" manner, similar to $\wedge$ and $\vee$.

We can formalize logical duality, but we first will introduce a new, convenient extension to $\mathcal{L}_1^+$, appropriately called $\mathcal{L}_1^{++}$. This language is exactly like $\mathcal{L_1}^+$, but with the addition of generalized connectives; for each $n$-place truth function $f$, there is a unique $n$-place connective $c$ that is associated with $f$ i.e. that would express it with $n$ sentence letters.

Something else worth noting that I've avoided for consistency is that sometimes it's useful to represent $T$ with 1 and $F$ with 0, so we can do basic arithmetic to simplify our manipulation of truth values. For example, $|\neg P|_A = 1 - |P|_A$, or with $\wedge$, $|P \wedge Q|_A = |P|_A \cdot |Q|_A$.

Definition (Dual of a Connective): Let $c$ be an $n$-place connective with associated truth function $f_c$. The dual connective $c^*$ is the connective with the associated truth function $f_{c^*}$ defined as:

for all truth values $t_1,t_2,\cdots,t_n$.

In essence, the dual connective is defined by this relation of internal and external negations. In our discussion of expressive adequacy, the final problem I wrote at the end used $\leftrightarrow^{*}$, the dual of $\leftrightarrow$. For a more visual definition, the dual of a connective $c^*$ is obtained by flipping every occurrence of $T$ to $F$ and $F$ to $T$ in the truth table for $c$. If we look at $\wedge$, then

We see then that $\wedge^*$ has the exact same truth table for $\vee$, and conclude they are duals. If we do the same for $\vee$, we would see that $\vee^*$ has the same truth table for $\wedge$, and that leads us to our first observation.

Claim: For all $\mathcal{L}_1^{++}$ connectives, $c^{**} = c$.

Proof:

Since these associated truth functions are unique to each connective, we've shown $c^{**} = c$. $\blacksquare$

Connective duals are the heart of this topic, so with that solidified, we are now in a position to define other forms of duality:

Definition (Generalized Duality):

• The dual of a sentence is given recursively: if $\Phi$ is a sentence letter, then $\Phi^{*} = \Phi$, and if $\Phi = c(\varphi_1, \cdots, \varphi_n)$, then $\Phi^{*} = (c(\varphi_1, \cdots, \varphi_n))^{*} = c^{*}(\varphi_1^{*}, \cdots, \varphi_n^{*})$
• Let $T^* = F$ and $F^* = T$. The dual of a structure $A^{*}$ is defined by flipping the truth values of all sentence letters in $A$, i.e. $|\alpha|_{A^*} = (|\alpha|_A)^* = 1 - |\alpha|_A$ for all sentence letters $\alpha$

We can now prove another adjacent lemma as before.

Claim: For all $\mathcal{L}_1^{++}$ sentences, $\Phi^{**} = \Phi$.

Proof: Induct on the complexity of $\Phi$.

Base Case: If $\Phi$ is a sentence letter, then $\Phi^{**} = (\Phi^*)^* = \Phi^* = \Phi$.

Inductive Hypothesis: Suppose this holds for sentences of complexity $n$. Let $c$ be the highest scope connective of $\Phi$ i.e. $\Phi = c(\varphi_1, \varphi_2, \cdots, \varphi_n)$ with $\textrm{NConn}(\varphi_i) \leq n$. Then

It's worth mentioning that outside of $\mathcal{L}_1^{++}$, it might not be the case that $\Phi^{**} = \Phi$ since we don't have this nice general connective $c$ to work with. But, it will still be logically equivalent; they may not look the same in other languages (as in use the exact same characters in the exact same order), but certainly $\Phi^{**} \equiv \Phi$.

With all of this in place, we now have our first major result:

Duality Lemma: For all sentences, $|\Phi^*|_A + |\Phi|_{A^*} = 1$. In other words, $|\Phi^*|_A = |\neg\Phi|_{A^*}$

Proof: Induct on the complexity of $\Phi$.

Base Case: If $\Phi$ is a sentence letter, then $|\Phi^{*}|_A = |\Phi|_A = 1 - |\Phi|_{A^*}$

Inductive Hypothesis: Suppose this holds for sentences of complexity $n$. Let $c$ be the highest scope connective of $\Phi$ i.e. $\Phi = c(\varphi_1, \varphi_2, \cdots, \varphi_n)$ with $\textrm{NConn}(\varphi_i) \leq n$. Then

This lemma comes with a very nice consequence:

Duality Theorem: If $\Phi \vDash \Psi$, then $\Psi^* \vDash \Phi^*$.

Proof: Say $|\Psi^*|_A = 1$. Then by the Duality Lemma, $|\Psi|_{A^*} = 0$. Since $\Phi \vDash \Psi$, then $|\Phi|_{A^*} = 0$. By the Duality Lemma again, $|\Phi^*|_A = 1$. So in any structure where $|\Psi^*|_A = 1$, $|\Phi^*|_A = 1$. In other words, $\Psi^* \vDash \Phi^*$.

Some other interesting ideas to consider:

• We proved earlier that $c^{**} = c$, but that does not rule out the possibility that there are connectives such that $c^* = c$. For example, $\neg$ is self-dual as $\neg^* = \neg$. Can you find any other self-dual connectives?
• Having shown $\wedge$ and $\vee$ as duals, this gives us a nice informal way to explain why $\forall$ and $\exists$ are duals too (remember, $\forall x Px \equiv \neg \exists x \neg Px$). Informally, $\forall x Px$ sort of expresses $\bigwedge_{\tau \in C} P\tau$ which ranges over all constants $C$ (while accurate, it is not a strictly good way of defining $\forall x Px$ as sentences cannot be of infinite length). Similarly, $\exists x Px$ is sort of equivalent to $\bigvee_{\tau\in C} P\tau$. So given the duality of $\wedge$ and $\vee$, perhaps it is not that surprising that $\forall$ and $\exists$ are duals of each other too.
• The Duality Theorem applies to only single sentences entailing other sentences, but what about sets of sentences? We can make a modified version for entailment between sets: we say $\Gamma \vDash \Sigma$ if when all sentences of $\Gamma$ are true, at least one sentence of $\Sigma$ is true. We can then also define $\Gamma^* = \{\gamma^* \ | \ \gamma \in \Gamma \}$ i.e. the dual of a set of sentences is the set of all dual sentences in the set. Then we can also prove in a similar manner if $\Gamma \vDash \Sigma$, then $\Sigma^* \vDash \Gamma^*$.

Part 3: Soundness and Completeness

The previous section was an exploration into how we are able to analyze logic in a very robust and formal way akin to the way we do so in math. And to be fair, it did start to feel like we were doing math after a certain point; we were manipulating sentences and structures in a fairly abstract way, while interesting, did not always have a direct appeal to why we were cared about those topics outside of curious questions. We now return back to some more concrete motivations, as we now have the methods to finally break down the two key theorems that make logic useful.

Soundness

As mentioned earlier, soundness is basically what makes proof systems and provability useful. As a reminder,

Soundness Theorem: If $\Gamma \vdash \Phi$, then $\Gamma \vDash \Phi$.

That is, for a given proof system, if it is sound, then it "preserves truth"; you wouldn't be able to prove something that is a non-sensical, bad argument. Soundness is arguably one of, if not, the most important quality we can have in a logical system. It's what allows us to know that for any right triangle the square of the shorter legs sum to the hypotenuse without having to check every right triangle. It's what allows us to know that $\sqrt{2}$ is irrational without having to check every possible fraction.

And the way you prove soundness is surprisingly easy.

Proof: We will prove this with induction, but instead of looking at the complexity of sentences, we will induct on the complexity of proofs; we will ascribe an integer to a proof to how many natural deduction rules it uses.

It's also worth noting here that our proof of soundness below only applies to natural deduction; we have to prove soundness for each proof system. We are working with Natural Deduction, so we will show those rules are sound, but if we use another proof system, say, Frege's propositional calculus, we would have to re-prove soundness (since we could just make up arbitrary proof rules that are not sound if we wanted).

Base Case: The simplest proof of "complexity 0" of $\Phi$ is from $\Phi$ itself $\Phi \vdash \Phi$; if you know $\Phi$, you certainly know $\Phi$ without needing to use any proof rules. Clearly $\Phi \vDash \Phi$ as any structure such that $|\Phi|_A = T$, that structure also assigns $|\Phi|_A = T$, establishing the base case.

Inductive Step: Now we will assume from premises $\Gamma$ that our proofs are sound, up until the last step. We will check for each of our Natural Deduction rules that they are sound.

$\underline{\wedge\textrm{-Intro:}}$ Suppose we have a proof of $\Psi_1$ from premises $\Pi_1$ (that is, $\Pi_1 \vdash \Psi_1$) and $\Psi_2$ from premises $\Pi_2$ (similarly $\Pi_2 \vdash \Psi_2$), we obtain a proof $\Psi_1 \wedge \Psi_2$ from an application of $\wedge\textrm{-Intro}$ (where $\Pi_1, \Pi_2 \subseteq \Gamma \ $). By our inductive hypothesis, $\Pi_1 \vDash \Psi_1$ and $\Pi_2 \vDash \Psi_2$. So, for any structure that satisfies $\Gamma$, that structure clearly satisfies $\Pi_1$ and $\Pi_2$ (as they come from $\Gamma \ $), and therefore also satisfies $\Psi_1$ and $\Psi_2$ (by the entailment derived from our inductive hypothesis). So $\Gamma \vDash \Psi_1$ and $\Gamma \vDash \Psi_2$. By the truth table for $\wedge$, we can see that $\Gamma \vDash \Psi_1 \wedge \Psi_2$.

$\underline{\wedge\textrm{-Elim:}}$ From a proof of $\Psi_1 \wedge \Psi_2$ on premises $\Gamma$ ($ \ \Gamma \vdash \Psi_1 \wedge \Psi_2$), we obtain a proof of $\Psi_1$ (or $\Psi_2$) using $\wedge\textrm{-Elim}$. By our inductive hypothesis, $\Gamma \vDash \Psi_1 \wedge \Psi_2$. By the truth table for $\wedge$, $\Psi_1 \wedge \Psi_2$ is true iff $\Psi_1$ and $\Psi_2$ are true, so we can conclude $\Gamma \vDash \Psi_1$ and $\Gamma \vDash \Psi_2$.

This is the basic idea of proving soundness: use our knowledge of semantics and definitions to show our final entailment. Sometimes it is just a matter of writing out what feels obvious, but some might require some more background knowledge or tricks.

$\underline{\textrm{→-Intro:}}$ From a proof of $\Phi$ on premises $\Gamma \cup \{\Psi\}$, get a proof $\Gamma \vdash \Psi \rightarrow \Phi$ with $\textrm{→-Intro}$ (discharging $\Psi$ at the end). From the inductive hypothesis $\Gamma, \Psi \vDash \Phi$. By the Deduction Theorem, $\Gamma \vDash \Psi \rightarrow \Phi$.

$\underline{\textrm{→-Elim:}}$ From a proof of $\Psi$ on premises $\Pi_1$ and another proof of $\Psi \rightarrow \Phi$ from premises $\Pi_2$, obtain a proof of $\Phi$ via $\textrm{→-Elim}$ (where $\Pi_1, \Pi_2 \subseteq \Gamma \ $). By the inductive hypothesis, $\Gamma \vDash \Psi$ and $\Gamma \vDash \Psi \rightarrow \Phi$. By the truth table for $\rightarrow$, whenever $\Gamma$ is satisfied, $\Phi$ must also be satisfied. Thus, $\Gamma \vDash \Phi$.

The other cases for $\neg$, $\vee$, and $\leftrightarrow$ are all very similar (but do know they are in fact sound). $ \ \blacksquare$

Soundness of $\mathcal{L}_2$

The above proof shows the natural deduction proof system is sound in $\mathcal{L}_1$, but we've moved on past to bigger and better things. To show natural deduction is sound in $\mathcal{L}_2$, we only need to show the 4 new rules are sound: $\forall\textrm{-Intro}$, $\forall\textrm{-Elim}$, $\exists\textrm{-Intro}$, and $\exists\textrm{-Elim}$.

I won't go through them in detail (because I this post is long enough and I don't want to type them out), but the proof idea is exactly the same as it was for $\mathcal{L}_1$. Here's a rough sketch of the results one could show to deduce soundness of $\mathcal{L}_2$ (starter proofs can be found in Elements of Deductive Logic):

1. Substitution of Co-Designating Terms: Let $\tau_1$ and $\tau_2$ be constants/variables, and $\varphi[\tau_2/\tau_1]$ is the formula of replacing free occurrences of $\tau_1$ by $\tau_2$ where none of these instances fall under $\forall \tau_2$ or $\exists \tau_2$ (constants occur freely vacuously). Then for any structure $A$ and variable assignment $\alpha$ where $|\tau_1|_A^\alpha = |\tau_2|_A^\alpha$, then $|\varphi|_A^\alpha = |\varphi[\tau_2/\tau_1]|_A^\alpha$ (show with induction on the length of formulae).
   • This basically says that if two constants/variables stand for the same thing, they are interchangeable in that variable assignment.
2. With substitution of co-designating terms and the satisfaction rules for $\forall$ and $\exists$, show the following sequents are correct:
   • $\Phi[t/v] \vDash \exists v \Phi$
   • $\forall v \Phi \vDash \Phi[t/v]$
   • If $\Gamma, \Phi[t/v] \vDash \Psi$, then $\Gamma, \exists v \Phi \vDash \Psi$ where the constant $t$ does not occur in $\Gamma, \Phi, \Psi$
   • If $\Gamma \vDash \Phi[t/v]$, then $\Gamma \vDash \forall v \Phi$ where constant $t$ does not occur in $\Gamma, \Phi$
3. Use those sequents to complete the Soundess Theorem for $\forall$ and $\exists$ in the inductive steps of the proof.

Completeness

We have shown soundness, meaning anything that we do prove is a good argument. But we haven't really given a method of showing how to prove things; proofs in natural deduction—in math even—often rely on having a good intuition of the problem at hand that can be formalized into rigorous logic. For example, look at the following claim and proof:

Claim: There exists irrational numbers $a$ and $b$ such that $a^b$ is rational.

Proof: Consider the number $\sqrt{2}^\sqrt{2}$.

• If this is rational, we're done.
• If this is irrational, then consider the number $(\sqrt{2}^\sqrt{2})^\sqrt{2} = \sqrt{2}^2 = 2$, which is rational, and we are done.

Never once did we actually deduce the rationality of $\sqrt{2}^\sqrt{2}$, but by exhausting the possible cases, it doesn't matter since every possibility leads to a conclusion that proves our claim. $ \ \blacksquare$

To independently think of the above proof, one would have to just be comfortable and familiar with exponents, since it really just abused the fact how exponenets combine to create a possible number that satisfies the claim. Moreover, the above proof used the implicit, extra premise that all real numbers are either rational or irrational. This might be a tautology that can be proven under what one assumes about the real numbers, but that would still need to be shown (since not everyone necessarily takes this for granted). All together, this proof requires a lot of creativiity to not only think of the individual, relevant facts, but to string them together too.

What am I getting at, though? Since there is no algorithm or method to go about proofs, we have no good way to determine if we can prove something until we have proved it. When something has not been proven, though, we don't just give up; mathematicians and scientists keep searching for new techniques and evidence in the hopes of proving or disproving a theorem. But there's a key word there: "hope". We don't actually know if what we are trying to prove, actually is provable! We don't know if we are wasting our time away on a Sisyphean quest with no end.

Ideally, we want to be working in a complete logical system, that is:

Completeness Theorem: If $\Gamma \vDash \Phi$, then $\Gamma \vdash \Phi$.

This is the converse of soundness, and would be a very convenient property of our logic to have, since it would mean there are no "unjustified" truths; all arguments would have a chain of causal reasoning in our deductive system that allow us to show its validity.

Some of you may have heard of completeness over soundness before for a couple of reasons: 1) soundness is a fairly intuitive property many of us just trust, as much of logic can be readily thought out to seem to make sense; but more likely 2) Gödel's incompleteness theorems are among the most celebrated and frustrating results of modern math that have infiltrated the highest ranks of science legends. Ultimately, they culminate many decades of the development of logic and putting to rest, arguably the most fundamental question in math: can we know everything? The ultimate answer is no, leaving us with many theorems in math that might be logically true, but just unprovable from our axioms. As of now, over 10 trillion non-trivial zeroes of the Riemann hypothesis have been verified, yet a concrete, irrefutable proof of this 160 year old problem still eludes us, and incompleteness forces us to wonder if there might not be one at all (but if the Riemann hypothesis is false, then it is provably false according to Robin's theorem!).

But that's going a bit too far off the beaten path. Fortunately for us, $\mathcal{L}_1$, $\mathcal{L}_2$, and $\mathcal{L}_=$ are all complete logics, and the rest of this section will be to show that, starting with $\mathcal{L}_1$.

Having a complete system is desirable for a few reasons. Having all theorems be provable is of course nice, but completeness also allows us to connect a lot of dots between semantics and proofs as we would expect. Earlier we mentioned the idea of ND-consistency. In a sound and complete system (which ND is), it turns out syntactic consistency is equivalent to semantic consistency (or satisfiability):

Claim: $\Gamma$ is ND-consistent iff it is satisfiable.

Proof:

• $(\Rightarrow)$ Suppose $\Gamma$ is ND-consistent. Then there is a sentence $\Phi$ such that $\Gamma \nvdash \Phi$. By completeness, $\Gamma \nvDash \Phi$. The only way an argument or sequent is invalid is if there is a structure $A$ such that $A \vDash \Gamma$ but $A \nvDash \Phi$. But then we have a structure such that $A \vDash \Gamma$, thus showing $\Gamma$ is satisfiable.

• $(\Leftarrow)$ We'll prove the contrapositive i.e. we'll show that if $\Gamma$ is ND-inconsistent, then it is unsatisfiable. So say $\Gamma$ is ND-inconsistent, that is, $\Gamma \vdash \Phi$ for all sentences $\Phi$. By soundness, we have $\Gamma \vDash \Phi$ for all sentences $\Phi$. This includes contradictions, such as $\Phi = P \wedge \neg P$, and the only way $\Gamma$ could entail all sentences, including contradictions, would be if it was unsatisfiable.

From here on, I'll use consistent only to mean ND-consistent unless I specify otherwise, and use satisfiable for semantic consistency.

Deductive Completeness

Our ultimate goal is to prove completeness of a logical language. But first, we'll need to talk about the completeness of a set of sentences:

• Definition (Semantic Completeness): $\Gamma$ is semantically complete when for all sentences $\Phi$, either $\Gamma \vDash \Phi$ or $\Gamma \vDash \neg \Phi$ (or both)
• Definition (Deductive Completeness): $\Gamma$ is deductively complete when for all sentences $\Phi$, either $\Gamma \vdash \Phi$ or $\Gamma \vdash \neg \Phi$ (or both)

Semantically complete sets are like structures, ascribing truth values to all sentences when it is satisfied. Deductively complete sets act likewise but with proofs.

Combining ND-consistency and ND-completeness allows us to prove a very intuitive result about proofs:

Consistency and Completeness Lemma: Suppose $\Gamma \subseteq \textrm{Sen}(\mathcal{L}_1)$ is ND-consistent and -complete. Then for all sentences $\Phi,\Psi$:

1. $\Gamma \vdash \neg \Phi$ iff $\Gamma \nvdash \Phi$
2. $\Gamma \vdash \Phi \wedge \Psi$ iff $\Gamma \vdash \Phi$ and $\Gamma \vdash \Psi$
3. $\Gamma \vdash \Phi \vee \Psi$ iff $\Gamma \vdash \Phi$ or $\Gamma \vdash \Psi$ (or both)
4. $\Gamma \vdash \Phi \rightarrow \Psi$ iff $\Gamma \nvdash \Phi$ or $\Gamma \vdash \Psi$ (or both)
5. $\Gamma \vdash \Phi \leftrightarrow \Psi$ iff $\Gamma \vdash \Phi$ and $\Gamma \vdash \Psi$, or $\Gamma \nvdash \Phi$ and $\Gamma \nvdash \Psi$

This lemma esentially allows us to formalize results about proofs in how we would expect them to act; it gives us a bridge that consistency and completeness are enough to show that proofs and derivability act like truth in a structure.

Proof: We'll just verify each case individually.

Case (i):

• $(\Rightarrow)$ Say $\Gamma \vdash \neg \Phi$. By its ND-consistency, $\Gamma \nvdash \Phi$.
• $(\Leftarrow)$ Say $\Gamma \nvdash \Phi$. By its ND-completeness, $\Gamma \vdash \neg \Phi$.

Case (ii):

• $(\Rightarrow)$ If $\Gamma \vdash \Phi \wedge \Psi$, then by $\wedge$-Elim, $\Gamma \vdash \Phi$ and $\Gamma \vdash \Psi$.
• $(\Leftarrow)$ If $\Gamma \vdash \Phi$ and $\Gamma \vdash \Psi$, we can combine these two proofs into one bigger proof by applying $\wedge$-Intro, netting us a proof of $\Gamma \vdash \Phi \wedge \Psi$.

Cases (iii), (iv), and (v) are all similar: look at the given proof we assume, and extend it via natural deduction to get the conclusion we want. $ \ \blacksquare$

Maximally Consistent Sets

The central idea that we'll use to prove the completeness of $\mathcal{L}_1$ is maximally consistent sets.

Definition (Maximally Consistent Sets): A set $\Gamma$ is maximally consistent (or maximally D-consistent with respect to a proof system $D$) when $\Gamma$ is ND-consistent, and if $\Gamma \cup \{\Phi \}$ is ND-consistent, then $\Phi \in \Gamma$.

We can think of maximally consistent sets as model universes, completely filled with as many rules (sentences) that determine the laws and facts of this universe without contradicting itself. One more sentence and $\Gamma$ would no longer be consistent.

Conveniently enough, maximally consistent sets hold the two properties we just discussed.

Maximally Consistent Sets Are Complete: If $\Gamma \subseteq \textrm{Sen}(\mathcal{L}_1)$ is maximally consistent, then it is ND-consistent and ND-complete.

Proof: By definition of maximal consistency, $\Gamma$ is ND-consistent.

Now suppose for a contradiction that $\Gamma$ was ND-incomplete, i.e. there's a sentence $\Phi$ such that $\Gamma \nvdash \Phi$ and $\Gamma \nvdash \neg \Phi$. Hence, I claim both $\Phi \in \Gamma$ and $\neg \Phi \in \Gamma$, making $\Gamma$ inconsistent and showing our assumption of its incompleteness incorrect. To show a sentence $\Psi \in \Gamma$, we can leverage the definition of maximal consistency that $\Gamma \cup \{\Psi \}$ is consistent.

• Consider for contradiction that $\Gamma \cup \{\neg \Phi \}$ is inconsistent. Then, it would prove everything, and in particular, it would also prove $\Gamma \cup \{\neg \Phi \} \vdash \Phi$. By $\neg$-Elim, we can see that also $\Gamma \vdash \Phi$ since $\neg \Phi$ can be discarded in that proof. But we already know that $\Gamma \nvdash \Phi$, so it must be that $\Gamma \cup \{\neg \Phi \}$ is consistent. By maximal consistency then, $\neg \Phi \in \Gamma$.
• Same argument as before, but with assuming $\Gamma \cup \{\Phi \}$ is inconsistent. Similarly we can conclude $\Phi \in \Gamma$.

Since $\{\Phi, \neg \Phi\} \subseteq \Gamma$, we have $\Gamma \vdash \Phi$ and $\Gamma \vdash \neg \Phi$ trivially, and thus with $\neg$-Elim, $\Gamma$ proves everything. Hence it is inconsistent, which contradicts that it is maximally consistent. Hence $\Gamma$ must be ND-complete. $ \ \blacksquare$

Another nice property of maximally consistent sets is that they provability and membership are equivalent in $\Gamma$.

Membership Lemma: If $\Gamma$ is maximally consistent, then $\Gamma \vdash \Phi$ iff $\Phi \in \Gamma$.

Proof:

• $(\Leftarrow)$ Clearly if $\Phi \in \Gamma$, then $\Gamma \vdash \Phi$ trivially.
• $(\Rightarrow)$ Since $\Gamma \vdash \Phi$, by the consistency of $\Gamma$, we also have $\Gamma \nvdash \neg \Phi$ (since if it proved both, then we would be able to prove any sentence with $\neg$-Elim). We want to show $\Phi \in \Gamma$, which would be true if $\Gamma \cup \{\Phi\}$ was consistent (by definition of a maximally consistent set). Say for a contradiction that $\Gamma \cup \{\Phi\}$ was inconsistent, that is, it proves everything. If it proves everything, in particular, it also proves $\Gamma \cup \{\Phi\} \vdash \neg \Phi$. But then also, $\Gamma \vdash \neg \Phi$, since by $\neg$-Intro, we can discard the premise $\Phi$ as an assumed premise that we do not actually need concretely. This contradicts the fact that $\Gamma \nvdash \neg \Phi$, so our assumption must be wrong. Therefore, $\Gamma \cup \{\Phi\}$ is consistent, and hence $\Phi \in \Gamma$ by definition of a maximally consistent set.

With these two lemmas in mind, we can now generalize membership for $\Gamma$ even broader:

Generalized Membership Lemma: Suppose $\Gamma \subseteq \textrm{Sen}(\mathcal{L}_1)$ is maximally consistent. Then for all sentences $\Phi,\Psi$:

1. $\neg\Phi \in \Gamma$ iff $\Phi \notin \Gamma$
2. $\Phi \wedge \Psi \in \Gamma$ iff $\Phi \in \Gamma$ and $\Psi \in \Gamma$
3. $\Phi \vee \Psi \in \Gamma$ iff $\Phi \in \Gamma$ or $\Psi \in \Gamma$ (or both)
4. $\Phi \rightarrow \Psi \in \Gamma$ iff $\Phi \notin \Gamma$ or $\Psi \in \Gamma$ (or both)
5. $\Phi \leftrightarrow \Psi \in \Gamma$ iff $\Phi \in \Gamma$ and $\Psi \in \Gamma$, or $\Phi \notin \Gamma$ and $\Psi \notin \Gamma$

Proof: We've already done all the work in the previous lemmas. $\Gamma$ is maximally consistent, so it's ND-consistent and ND-complete. By the Consistency and Completeness Lemma, we have all those cases about provability true in $\Gamma$. By the Membership Lemma, we can replace all those proofs with clauses of membership. $ \ \blacksquare$

Now compare each of these cases of the Generalized Membership Lemma to what satisfaction looks in a structure. The relations are identical! This is the key property of maximally consistent sets: membership in $\Gamma$ acts like truth in an $\mathcal{L}_1$-structure. Likewise, we are able to not only make a maximally consistent set $\Gamma$, but also create a structure specifically tailored to satisfy $\Gamma$.

Constructing a Maximally Consistent Set

Maximally consistent sets are filled with as many sentences as possible while remaining consistent. So, if we wanted to make a maximally consistent set, we could just look at every sentence and see if we can add it to our bucket while remaining consistent.

Completeness Lemma 1: Suppose we have an ND-consistent set $\Gamma$. Then there is a maximally consistent set $\Gamma^{+}$ with $\Gamma \subseteq \Gamma^{+}$.

Proof: The idea is that there are a countably infinite number of sentence letters in $\mathcal{L}_1$, and therefore a countably infinite number of sentences in $\mathcal{L}_1$ too (the Cartesian product of two countably infinite sets is countably infinite). So we just go through all the possible sentences, add them to $\Gamma$ if preserves consistency, and at the end we obtain not just a consistent, but maximally consistent set.

Let $\textrm{Sen}(\mathcal{L}_1) = \{\Phi_0, \Phi_1, \Phi_2, \cdots\}$, and let $\Gamma_0 = \Gamma$. We then define the recursion

Then let $\Gamma^{+} = \bigcup_{n} \Gamma_n$. By construction, $\Gamma_n$ is consistent and $\Gamma_n \subseteq \Gamma^+$ for all $n$. Importantly, including $n=0$, so $\Gamma = \Gamma_0 \subseteq \Gamma^+$.

• First we show that $\Gamma^+$ is ND-consistent. Say it wasn't. Then $\Gamma^+$ proves every sentence, so $\Gamma^+ \vdash \varphi$ and $\Gamma^+ \vdash \neg \varphi$ for some sentence $\varphi$. Proofs must be finite (what would an infinite proof look like?), so we only need finite premises to prove these claims: $\{\gamma_1, \gamma_2, \cdots, \gamma_m \} \vdash \varphi$ where $\{\gamma_1, \gamma_2, \cdots, \gamma_n \} \subseteq \Gamma^+$
  $\{\delta_1, \delta_2, \cdots, \delta_n \} \vdash \neg\varphi$ where $\{\delta_1, \delta_2, \cdots, \delta_n \} \subseteq \Gamma^+$ Where $m$ and $n$ are some positive integers. Then, the finitely many sentences of $\{\gamma_1, \gamma_2, \cdots, \gamma_m \} \cup \{\delta_1, \delta_2, \cdots, \delta_n \}$ must have appeared somewhere in our listing of $\textrm{Sen}(\mathcal{L}_1)$ and thus must all have been added to $\Gamma_k$ for some $k$. But then $\Gamma_k$ would be inconsistent as $\Gamma_k \vdash \varphi$ and $\Gamma_k \vdash \neg\varphi$, which cannot happen by construction. So $\Gamma^{+}$ is ND-consistent.
• Now we need to show $\Gamma^{+}$ is maximally consistent. Say $\Gamma^{+} \cup \{\varphi\}$ is consistent. $\varphi$ must have appeared in our enumeration of $\textrm{Sen}(\mathcal{L}_1)$ at some point, i.e. $\varphi = \Phi_k$ for some $k$. $\Gamma^{+} \cup \{\Phi_k\}$ is consistent, and $\Gamma_k \subseteq \Gamma^{+}$, so it follows that $\Gamma_k \cup \{\Phi_k\}$ is consistent (subsets of consistent sets are consistent). So by our recursive construction of $\Gamma_{k+1}$, we have $\varphi = \Phi_k \in \Gamma_{k+1} \subseteq \Gamma^{+}$, so $\varphi \in \Gamma^{+}$. Hence $\Gamma^{+}$ is maximally consistent.

So now we have a way of making a maximally consistent set.

Satisfying a Maximally Consistent Set

We showed earlier maximally consistent sets treat membership almost identically to how sentences are true in a structure. As such, it shouldn't be surprising that we can create a structure that exclusively revolves around a maximally consistent set.

Completeness Lemma 2: If $\Gamma$ is a maximally consistent set, then there's an $\mathcal{L}_1$ structure $A_{\Gamma}$ such that $A_{\Gamma} \vDash \Phi$ iff $\Phi \in \Gamma$.

Proof: We prove this by giving an explicit structure: for a sentence letter $\alpha$, let $A_{\Gamma} \vDash \alpha$ iff $\alpha \in \Gamma$. We now show with induction on the complexity of sentences, that for all sentences $\Phi$ we still have $A_{\Gamma} \vDash \Phi$ iff $\Phi \in \Gamma$.

Base Case: We defined $A_{\Gamma}$ by satisfying sentence letters our condition for sentence letters, so the base case holds.

Inductive Hypothesis: Say this holds up to complexity $n$, and $\Phi$ is of complexity $n+1$. As with so many proofs before, there are 5 cases to consider, one for each connective in $\mathcal{L}_1$. The key tool we'll use is the Generalized Membership Lemma for maximally consistent sets.

$\underline{\Phi = \neg \Psi}:$ $\Gamma$ is maximally consistent by assumption, so $\Phi = \neg \Psi \in \Gamma$ iff $\Psi \notin \Gamma$. By the inductive hypothesis, $A_{\Gamma} \vDash \Psi$ iff $\Psi \in \Gamma$, so we deduce $A_{\Gamma} \nvDash \Psi$. Thus, by the semantic rule for $\neg$, we get $A_{\Gamma} \vDash \neg\Psi = \Phi$.

$\underline{\Phi = \Psi_1 \wedge \Psi_2}:$ $\Phi = \Psi_1 \wedge \Psi_2 \in \Gamma$ iff $\Psi_1 \in \Gamma$ and $\Psi_2 \in \Gamma$. By the inductive hypothesis then, $A_{\Gamma} \vDash \Psi_1$ and $A_{\Gamma} \vDash \Psi_2$. By the truth rules for $\wedge$, $A_{\Gamma} \vDash \Psi_1 \wedge \Psi_2$, and so $A_{\Gamma} \vDash \Phi$.

The other connectives are more of the same of what we've done before. Hence we can create a structure that exactly and only satisfies members of a maximally consistent set.

The Proof of Completeness

We've proven a lot about maximally consistent sets, and how they share a lot of properties to structures, but we've actually done pretty much all the work we need to fully prove the completeness of $\mathcal{L}_1$.

Completeness Theorem: If $\Gamma \vDash \Phi$, then $\Gamma \vdash \Phi$.

Proof: We'll prove the contrapositive: if $\Gamma \nvdash \Phi$, then $\Gamma \nvDash \Phi$. So suppose that $\Gamma \nvdash \Phi$. As we've seen many times before, if $\Gamma \nvdash \Phi$, then $\Gamma \cup \{\neg \Phi\}$ is consistent. By Completeness Lemma 1, we can make a maximally consistent set $\Gamma^{+}$ out of $\Gamma \cup \{\neg\Phi\}$ i.e. $\Gamma \cup \{\neg\Phi\} \subseteq \Gamma^{+}$. By Completeness Lemma 2, we can find a structure $A$ such that $A \vDash \varphi$ iff $\varphi \in \Gamma^{+}$. Since $\Gamma \cup \{\neg\Phi\} \subseteq \Gamma^{+}$, we have $A \vDash \Gamma$ and $A \vDash \neg\Phi$. By the semantic rule for $\neg$, we also have $A \nvDash \Phi$. Thus by the definition of entailment $\Gamma \nvDash \Phi$.

It's a relatively short proof, but that's mostly because we shoved a lot of the proof into the Completeness Lemmas 1 and 2. Not to mention, I don't know about you, but it is definitely not obvious to me that maximally consistent sets would be the tool that paved the path forward to prove completeness, let alone even have the intuition that $\mathcal{L}_1$ is complete. To realize that satisfiability in a structure is like membership in a maximally consistent set is already impressive, but to then think to actually link the two for this proof is very slick.

As a final aside, we can now deem $\mathcal{L}_1$ as having special status:

Adequacy Theorem: $\Gamma \vdash \Phi$ if and only if $\Gamma \vDash \Phi$.

Completeness of Other Logics

As before, I won't go into full detail, but $\mathcal{L}_2$ and $\mathcal{L}_=$ are likewise both complete logics as well. To extend the proof of completeness, it is very similar to how we would extend it for soundness, needing to just account for the quanitifers $\forall$ and $\exists$.

• First we would need to extend the Generalized Membership Lemma:
  • $\exists v \Phi \in \Gamma$ iff $\Phi[t/v] \in \Gamma$ for some constant $t$
  • $\forall v \Phi \in \Gamma$ iff $\Phi[t/v] \in \Gamma$ for all constants $t$
• Next would be to show that Completeness Lemma 2 still holds, that is, every maximally consistent set of $\mathcal{L}_2$ sentences is satisfiable
• Then the proof for completeness remains the same as before with $\mathcal{L}_1$

Part 4: What's Next?

This is where my logic sequence ended this year. But of course that does not mean that this is where logic as a field stops, too.

More To Be Studied!

Even before considering other logics or flaws within our current one, there is still many, extremely important results to be studied within $\mathcal{L}_1$, $\mathcal{L}_2$, and $\mathcal{L}_=$.

Compactness

One particularly strong result that deserves its own post is the Compactness Theorem:

Compactness Theorem: If every finite subset of a set of sentences $\Gamma$ is satisfiable, then the whole set $\Gamma$ is satisfiable.

We've considered finite sets of sentences a lot already (as that's what proofs required of us), and the Compactness Theorem trivially holds for them since if $\Gamma$ is finite, then $\Gamma \subseteq \Gamma$ is a finite subset of itself. But for infinite sets of sentences, it's not entirely clear when they are satisfiable or not, and Compactness gives us a more tractable way of evaluating satisfiability.

The contrapositive gives us the equivalent formulation:

Compactness Theorem: If $\Gamma$ is unsatisfiable, then there is a finite subset $\Gamma^{\textrm{fin}} \subseteq \Gamma$ that is unsatisfiable.

And this is guaranteed to hold for infinite sets, while it isn't for finite! Consider the unsatisfiable set $\{P, \neg P\} \vDash$ with clearly having its subsets (except itself) satisfiable. This has the immediate consequence that every argument $\Gamma \vDash \Phi$ can be captured in a finite argument.

Alternate Form of Compactness: If $\Gamma \vDash \Phi$, then $\Gamma^{\textrm{fin}} \vDash \Phi$.

Proof: If $\Gamma$ is finite already, then the claim holds as it is. So consider $\Gamma$ to be infinite. Recall that $\Gamma \vDash \Phi$ iff $\Gamma \cup \{\neg \Phi\}$ is inconsistent. Then by Compactness, there is a finite subset $\Gamma^{\textrm{fin}} \cup \{\neg\Phi \}$ that is inconsistent. $\neg\Phi$ must be in this set as $\Gamma$ is satisfiable by assumption of $\Gamma \vDash \Phi$, so $\Gamma^{\textrm{fin}}$ is always satisfiable (if $\Gamma$ is unsatisfiable, then it entails everything, and then by Compactness there's a finite susbet that's also unsatisfiable that entails everything). Since $\Gamma^{\textrm{fin}} \cup \{\neg\Phi \}$ is inconsistent, then $\Gamma^{\textrm{fin}} \vDash \Phi$. $ \ \blacksquare$

Finally, it's worth noting that Compactness quickly follows from any logic that is both sound and complete:

Proof of Compactness:

There are more ways to prove and apply Compactness, but that's for another day.

Löwenheim-Skolem Theorem

Löwenheim-Skolem Theorem: If $\Gamma$ is a set of $\mathcal{L}_=$ sentences with an infinite structure with cardinality $\omega$, then

• (Upward) $\Gamma$ has a structure of every cardinality $\omega' > \omega$
• (Downward) $\Gamma$ has a countable structure

The proofs found can be found in Elements of Deductive Logic (which only hold in first-order logic!), but there's an interesting paradox that comes along with it:

Skolem's Paradox: The Löwenheim-Skolem Theorem says that no first-order theory can limit what the size of the structures that satisfy it. Set theory is a first-order theory, so the theorem would say that there is a countable structure that satisfies set theory. But set theory entails that there are uncountable sets as well. How can a structure with only countably many elements satisfy something that is uncountable?

I'll leave you to think about that, but the resolution to this dilemma (also in Eagle) leads to what is now known as the non-absoluteness of set theory; a set may be uncountable relative to one structure, but countable to another.

Lindström's Theorem

We've looked at different ways logic can be limited in some math-y ways, like with expressive adequacy. The previous two theorems above give some more implicit restrictions on what a first-order logic can do. Compactness says that $\mathcal{L}_=$ can't discern finite sets from "pseudo-finite" (infinite sets whose finite subsets are satisfiable), and Löwenheim-Skolem says that it can't differentiate cardinalities of a structure. However, Lindström's theorem tells us that given these two restrictions of satisfying Compactness and (specifically the downward) Löwenheim-Skolem theorem, first-order logic is actually the strongest logic with these restrictions. Different definitions can yield different results, but analyzing the relative strength of logics is an area to still explore. Lindström uses the idea of overlapping "good structures" between two logics to determine their relative strengths.

Natural Language and Grice's Maxims

Remember, we started building logic with the intention to formalize English and natural language arguments. But obviously our formalizations above can only go so far, and one of the first, early signs of the "weirdness" logic might bring with it is the case of expressing, "If… then…" If you might recall we had the following truth tables:

To get rid of those question marks, we decided to use $\rightarrow$, since it was easier to assign the case of vacuous more than anything else. So the only real information $\rightarrow$ carries is when it evaluates to false, telling us when one fact definitely does not result in another fact. For this reason, we call $\rightarrow$ the material implication as it only really cares about the current state of the actual world: if something $\Phi$ is true in our world and another thing $\Psi$ is false, it cannot be the case that in our world $\Phi \rightarrow \Psi$.

But that's not really how we usually use "If… then…" When someone uses "if", it does not necessarily mean that they know that fact is true, but they're supposing that it's true; "if" can denote a hypothetical. Compare the following conditionals:

• If Apple merges with Google, then Samsung does not have a monopoly on smartphones.
• If Apple merged with Google, then Samsung would not have a monopoly on smartphones.

The use of "would" in the second case indicates something that hasn't actually happend, but describes a perfectly understandable situation that we can still parse information from. In the first sentence, $\rightarrow$ would deem that a true sentence since our antecedent is false i.e. Apple and Google have not merged in the actual world. But in the second case, that sentence is false, since it is conceivable that even if Apple and Google merged in an alternate universe, it is possible that Samsung could still be the number one seller of smartphones.

The counterfactual implication, denoted by the fancy symbol $\square \!\! \rightarrow$, captures these hypothetical situations we use all the time that our material implication just fails to render accordingly.

Grice's Maxims

Note what we introduced above with $\square \!\! \rightarrow$ had no concrete semantics attached to it; our idea of hypotheticals, and "would" only arose out of what we understand in natural language, instead of something problematic from within the logic (or even English) itself. As such, this doesn't really give us a new truth table or anything, and unfortunately for logic, there are quite a few of these tacit rules to English.

For example, if I say, "Either I will pass my job interview, or the other candidate will get in," there's an additional piece of information that's communicated beyond just the two possibilities: I don't know which case will occur. And according to our semantic rules for $\vee$ that formalize "…or…", I could entail a disjunction by knowing a disjunct. That is, if I know $\Phi$, I certainly also know $\Phi \vee \Psi$ according to the truth table for $\vee$; $\Phi \vDash \Phi \vee \Psi$. Strictly speaking then, saying "I will pass my job interview" just contains more information as not only does it entail that "or" sentence I said at the beginning, but also an additional sentence (itself).

So if someone was to use any statement involving "…or…", we would expect them to be telling us exactly as much information as we need to get their message across, that being that they just don't know either side of the "…or…" statement since if they did, they would just say it.

Paul Grice (1975) formalized these cooperative principles of communication into what is now know as Grice's maxims that describe these underlying, hidden aspects of language that we always use, but never explicitly write out or explain (since they are just expected). In no particular order, they are:

1. Quantity: contribute as much information as is required without excess or lack of details.
2. Quality: contribute only as much as one knows and has evidence to be truthful.
3. Relation: contribute to the subject matter at hand; only be relevant.
4. Manner: be direct, clear, and avoid obscurity and esoteric constructions and words.

Notably the maxim of quantity is what poses a difficulty for our logical language, as seen with "…or…" and $\vee$. Incorporating these maxims, though, is a possible direction one could look to enhance or strengthen their logic's utility and applicability.

Extensions of Logic

Our logic above has done pretty well for what it needs to do concerning truth, and has found its way into computer science and discrete math as well by its very nature. But there are some arguments that go beyond truth alone.

Higher-Order Logics

A natural—and actually well-practiced—addition to logic are higher-order types. What we've been working with is known as first-order logic, as we can only quantify over objects. That is, $\forall$ and $\exists$ only ranges over possible, single objects that satisfy a formula. But, there are many things we can say beyond just objects. For example, in math we often able to make claims about sets of numbers. The least-upper-bound property of the real numbers states that every nonempty subset of the real numbers has a least upper bound (i.e. the supremum exists).

Or even simpler, for any property $P$ and any object $x$, it must be the case that $Px \vee \neg Px$.

Second-order logic is the upgrade we need that allows us to start quantifying over sets of objects. In first-order logic, it was predicates that took on the semantic value of sets of objects, so naturally second-order quantification looks like quantifying over predicates. So for our above example, it would look like: $\forall P \forall x(Px \vee \neg Px)$ where $P$ is now a variable standing in place for a predicate.

Quantifying over predicates/properties/sets of objects allows us to say quite a bit more as you'd expect. Earlier we mentioned the difference between numerical identity and qualitative identity, but now we are able to formalize how they relate to each other:

The Indiscernibility of Identicals: $\forall x \forall y (x = y \rightarrow \forall P(Px \leftrightarrow Py))$

• If two things are numerically identical, then they certainly have all the same properties

The Identity of Indiscernibles: $\forall x \forall y (\forall P(Px \leftrightarrow Py) \rightarrow x = y)$

• If two things have all the same properties, then they should be the same object

These are known as Leibniz's laws, and as discussed, the first is usually uncontested, while the second is more controversial. Whatever the case may be, second-order logic has given us a way to more clearly pick apart these claims.

But why stop there? First-order logic allowed us to quantify over objects; second-order logic allowed us to quantify over sets of objects (predicates). third-order logic allows us to quantify over sets of sets of objects, and the pattern continues. Second-order logic in particular is the most widely "used" from mathematics to computer science, but it should not be held to be necessarily "better" than first-order logic: second-order logic is actually incomplete.

Higher-order logic has its pulls and drawbacks, but it is a natural step from what we've looked at today, especially if you're interested in the foundations of mathematics and set theory.

"Universal" Logics

We already saw with the counterfactual conditional $\square \!\! \rightarrow$, there are already some obvious cases our logic lacks. Not to mention, the idea of possible and necessarily pops up all the time in philosophy and arguments, that it's a little disconcerting that we can't already formalize them.

Modal logic extends our logic precisely in this way, with two new quantifiers: We express, "It is possible the case that P," as $\square P$, and, "It is necessarily the case that P," as $\lozenge P$. The semantics of $\square$ and $\lozenge$ take our ideas of a structure as a model of the universe a little too literally. We say $\square P$ is true if there is a possible world in which $P$ is true, and we say $\lozenge P$ is true if in all possible worlds $P$ is true. The notion of what a possible world is by nature just vague, but it is an interesting one to consider. Ignoring semantics, we can already deduce a lot with these operators:

• $\neg \lozenge \Phi \equiv \square \neg \Phi$
  • If it is not necessary that $\Phi$ occurs, there is a possible world in which $\Phi$ does not occur (i.e. $\neg \Phi$ occurs)
• $\neg \square \Phi \equiv \lozenge \neg \Phi$
  • If it is not possible that $\Phi$ can occur, then it is necessarily the case that $\Phi$ does not occur (i.e. $\neg \Phi$ occurs)

As might be expected by the way we discussed their semantics, $\square$ and $\lozenge$ act a lot like $\exists$ and $\forall$ respectively.

Other extensions include temporal logics, that allow us to formalize time-sensitivity as well. The statement, "It is raining right now," might have a constant meaning, but whether it is true or not changes based on time, thus motivating the formalization of predicates like "eventually", "always", "until", and others that delineate time.

Free logics allow us to abandon the need to have constants to actually mean anything, or even have a non-empty domain. Just because we can talk about an imaginary, non-real thing, like a unicorn (i.e. has the body of a horse, has one horn, etc.), does not necessarily mean that a unicorn exists. Yet, if we allow unicorns into our domain of a structure to logically discuss them, it would appear that we would be claiming the existence of a mythical creature, which becomes quite useful when discussing metaphysics and any ontology or description of existence (in a way, free logic is all about specifying the conditions of existence.).

Multivalue and Fuzzy Logics

We have only ever considered two possible ways to evaluate a sentence: it is either true or false. But is that really the case? Sounds kind of insane to even try and conceive of something else a sentence could be, but consider the following sentence:

If it is true, then it declares itself false. If it is false, then it declaring that it is false is wrong, so it is true. The cycle continues forever and ever, without ever resolving itself to one of our prescribed truth values.

So some propose a solution that include a third, deviant truth value that allows us to ascribe some sentences as neither true nor false in a 3-value logic. Or another unintuitive solution is to allow degress of truth, i.e. the above sentence might be considered .5 true. Giving truth values between 0 and 1 is the cornerstone of fuzzy logic, which is usually used to describe when there is inexact information. However, don't confuse this number of truth as that of probability; it's not that this sentence is true only half the time, but rather that it's considered only half true since it is vague. Probability usually is associated with ignorance (lack of information; i.e. "the coin will land heads with probability .5" is said since we don't have knowledge of the future) as opposed to a lack of clarity. This notion is similar to the idea of degrees of membership in fuzzy set theory.

Altering the Axioms

The tautology $P \vee \neg P$ is quite a natural one. "Either it is or it is not." What other option could there be? Even in math we use double negatives multiplying to a positive, so positive and negative numbers seems to be the only cases possible. The law of excluded middle just seems to be a fact of the universe, that falls perfectly in line with our proofs and semantics for the classical logic we've been looking at.

But I've alluded throughout the post that some people do not always accept this as a given. Intuitionistic, or constructive logic, rejects the law of excluded middle, and argues that $\neg \neg P \nvDash P$, which come as standard inference rules that we can derive. The name comes from the fact that this doctrine encourages literal demonstrations of proof: if you have a proof of $P \wedge Q$, you literally have a proof of both $P$ and $Q$; a proof of $P \vee Q$ means one has a proof of $P$, or has a proof of $Q$.

Even more (ironically) unintuitive, we lose the ability to use proof by contradiction, as that relies on the ability to claim that if an assumption is wrong, its negation is true. So for classic proofs like the irrationality of $\sqrt{2}$, intuitionists would claim that all that's been proven is that $\sqrt{2}$ is not rational, not necessarily that it is irrational.

Picking and choosing what axioms, entailments, or laws one accepts of course changes what meaning and truth will look like to them, and can lead to some interesting, tenable persepctives that hold real weight and become the focus of entire areas of philosophy.

Conclusion

This post does not replace the two textbooks outlined above. If anything, it is only a simple supplement that gives a high-level overview of all the major conceptst to newcomers of formal logic, and a quick review source for those who have already studied it. At the very least, it'll be a quick reference for me, and future posts, as logic really has forced me to reconsider what I thought to be "staple truths", not just in math, but life in general. We'll definitely revisist logic in the future whether it be from a computer science perspective with decidability and algorithms, certain shortcuts in math, or just more theory in logic itself. The way logic has developed in such an abstract way lends itself to be applied anywhere, it's just a matter of thinking about how, as it almost certainly can be.

Last post, I summarized the basics of formal logic, starting from a philosophical context before building up to more broad and involved results, coming closer to what just looks like math by the end of it. One of those results which was briefly thrown into the conclusion was the Compactness Theorem. You won't need to know a lot of logic to appreciate the power of this theorem, but it would help to either skim that last blog post or brush up on how to read mathematical logic and its symbols. If the following talk of abstract logic seems confusing, don't worry, skip ahead to the Applications of Compactness to see why this theorem is so much more than just abstract logic.

Here a quick review of some concepts from propositional logic (that is, predicate calculus without quantifiers and predicates) we'll be using and discussing:

• Sentence: a string of logical characters, built from sentence letters $\{P, Q, R, \cdots \}$ that are either true or false, and connectives $\{\wedge, \vee, \neg, \rightarrow, \leftrightarrow\}$
• Structure: a "model universe"; a total function that assigns every sentence letter a truth value. Complex sentences are assigned truth values according to the rules of the connectives they use.
• Satisfiability: a set of sentences $\Gamma$ is satisfiable if and only if there is a structure that makes every sentence in $\Gamma$ true
• Finite Satisfiability: a set $\Gamma$ is finitely satisfiable if and only if every finite subset of $\Gamma$ is satisfiable
• Entailment: We say a set $\Gamma$ entails a sentence $\Phi$, written $\Gamma \vDash \Phi$, for whenever a structure satisfies $\Gamma$, then $\Phi$ is also true

Compactness Theorem: If $\Gamma$ is finitely satisfiable, then the whole set $\Gamma$ is satisfiable.

Roughly speaking, this says that if you have a set of logical statements, you can claim your set logical statements $\Gamma$ are not contradictory if every finite subset of logical statements are not contradictory themselves (we call the property of being non-contradictory satisfiability). If $\Gamma$ is already a finite collection of sentences, then Compactness trivially holds since $\Gamma \subseteq \Gamma$ is a finite subset of itself.

This becomes interesting when $\Gamma$ is infinite, since then this gives us a sufficient method for deducing a set of logical sentences is satisfiable. This should sort of make sense since if you can find a "problem set" within $\Gamma$ that are not satisfiable, then of course you can say $\Gamma$ is not satisfiable since if you try to make every sentence in $\Gamma$ not contradict each other, you'll have to find a way around the problem child set. This is essentially the contrapositive of the above formulation:

Compactness Theorem: If $\Gamma$ is unsatisfiable, then there exists a finite subset of $\Gamma$ that is unsatisfiable.

To see the strength of compactness, we can use this final formulation of compactness along with the definition of entailment to get one more that shows a compact logics are able to reduce arguments to only some number of "key" premises:

Compactness Theorem: If $\Gamma \vDash \Phi$, then $\Gamma^{\textrm{fin}} \vDash \Phi$.

This follows from an equivalent statement of entailment: $\Gamma \vDash \Phi$ if and only if $\Gamma \cup \{\neg\Phi\}$ is inconsistent (i.e. contradictory/unsatisfiable).

Even if an argument appears to need infinite premises $\Gamma$, there's an equivalent formulation that only requires a finite subset $\Gamma^{\textrm{fin}} \subseteq \Gamma$ that is what really matters to our argument. This relation between finite satisfiability and actual satisfiability, in a sense, states that there are certain "pseudo-finite" sets of sentences; they may look infinite, but for what you can deduce from them, they are no better than being a finite set of sentences.

We'll first cover some examples on why I think this theorem needs to be more known, where it gets its name from, and finally a proof of the Compactness Theorem for propositional logic.

Applications of Compactness

I've been going on a bit about this random theorem in a very abstract way—as logic tends to do. So, hopefully some examples will show just how useful this idea is; it gives us a way to extend results to infinity, which is not something that is all that common in math. Take proof by induction, for example: that gives us a way of showing a result holds for an arbitrary natural number, even as it approaches infinity. But to say the result holds at infinity is usually meaningless, even though we describe infinity or approaching infinity as that of the concept of an arbitrarily large number or similar.

Graph Theory

Due to the nature of graphs being readily characterized by a set of vertices and edges, they are surprisingly easy to formalize them and their properties in logic, making results fall out much quicker than you might expect.

Graph Coloring

Let's start with a concrete example: graph coloring. We say a graph $G$ of vertices $V$ and edges $E$ (we define a graph as the object $G=(V,E)$ to clearly define its vertices and edges) is $4$-colorable if every vertex can be assigned one of four colors (say, red, blue, green, and purple) where no two vertices of the same color are neighbors (that is, share an edge). We say a graph is $k$-colorable if we can assign every vertex one of $k$ colors with no two of the same color neighboring each other.

Theorem: A graph is $k$-colorable if and only if all of its finite subgraphs are $k$-colorable.

Proof: $(\Rightarrow)$ Clearly if the whole graph is $k$-colorable, then all of its finite subgraphs are $k$-colorable.

$(\Leftarrow)$ For the other direction, we can write some logical sentences that describe the behavior of our graph. For a (undirected) graph $G=(V,E)$ with

• $V = \{1,2,\cdots, m\}$ as a set of verteices (we'll just label with numbers for convenience),
• $E \subseteq V \times V$ as set of edges (that is an irreflexive and symmetric relation),
  • I.e. if the pair $(1,2) \in E$, we mean there is an edge connecting vertices 1 and 2
• $C = \{1,2,\cdots, k\}$ as a set of colors,

we'll define a set of sentence letters $P_{v,c}$ to represent $\textrm{"vertex} \ v \ \textrm{has color} \ c \textrm{"}$. With these simple sentences, we can write more complex descriptions of our graphs.

• Vertex $v$ has at least one color: $F_v := \bigvee_{c=1}^{k} P_{v,c}$
  • Either vertex $v$ has color 1, or color 2, or…, or color $k$
• Vertex $v$ has at most one color: $G_v := \bigwedge_{c_1 = 1}^{k} \bigwedge_{c_2 = c_1 + 1}^{k} \neg(P_{v,c_1} \wedge P_{v,c_2})$
  • For any two different colors $c_1$ and $c_2$, it is not the case that vertex $v$ has both colors
• Vertices $u$ and $v$ don't have the same color: $H_{u,v} := \bigwedge_{c = 1}^{k} \neg(P_{u,c} \wedge P_{v,c})$

Now let $\Gamma = \{F_v \ | \ v \in V\} \cup \{G_v \ | \ v \in V\} \cup \{H_{u,v} \ | \ (u,v) \in E\}$. Now it should be fairly clear that $\Gamma$ is satisfiable and not contradictory if and only if $G$ is $k$-colorable.

• The first two sets of sentences $\{F_v \ | \ v \in V\} \cup \{G_v \ | \ v \in V\}$ says that every vertex has at at least one color and at most one color, that is, every vertex has exactly one color.
• The last set $\{H_{u,v} \ | \ (u,v) \in E\}$ says that no neighboring vertices share the same color. If $G$ is $k$-colorable, then by definition, no two adjacent vertices have the same color.

So an easy structure $A$ to define that satisfies $\Gamma$ is $A(P_{v,c}) = T$; i.e. let $P_{v,c}$ be true if and only if vertex $v$ has color $c$ (exactly as it denotes).

By assumption, every finite subgraph of $G$ is $k$-colorable, so every finite subset of $\Gamma$ is satisfiable. By the Compactness Theorem, then $\Gamma$ must be satisfiable, and hence all of $G$ is $k$-colorable.

We can then combine this with the famous Four Color Theorem:

Four Color Theorem: Every finite graph is 4-colorable.

And by the above, we can extend this to the

Infinite Four Color Theorem: Every infinite graph is 4-colorable.

Proof: Let $G$ be an infinte graph. Every finite subgraph of $G$ is 4-colorable, so by the above theorem, $G$ itself must also be 4-colorable. $\ \blacksquare$

The above method of formalizing and encoding behavior and relationships in math—and graph theory especially—alongside the Compactness Theorem gives us a nice, relatively straightforward strategy to extending finite results to the infinite.

Kőnig's Lemma

Here's another result from graph theory that has its own handful of applications.

Kőnig's Lemma: Every locally-finite infinite tree contains an infinite branch.

Just to be clear, here are some definitions:

• A tree is a special type of (undirected) graph characterized by being connected (there's a path between any two vertices) and that it is acyclic (there is no path from a vertex to itself without passing through a vertex more than once)
• A locally-finite graph is one where every vertex only has a finite number of edges/neighbors (known as the degree of the vertex)
• An infinite graph is a graph with an infinite number of vertices

So a tree can look something like

Doesn't that sort of look like a tree growing downward? Here are some useful terms to describe trees:

• The very top vertex is called the root
• A vertex is at level $n$ if it takes $n$ edges to get to that vertex from the root. So the root can be defined by being the vertex at level 0.
• A vertex $x$ is the parent of another vertex $y$ (the child) if they share an edge, and $x$ is a level lower than $y$
• $x$ is the ancestor of $y$ if there is a path (i.e. a chain of parents) connecting $x$ and $y$, which we denote $x \preceq y$

So Kőnig's Lemma says every infinite tree either has an infinite-degree vertex or an infinite path. This should seem somewhat obvious, but as with anything involving infinity, it needs to be dealt with carefully. Since we can imagine building a tree where we carefully terminate every path to be finite. Sure, there may be arbitrarily long paths, but to show that there is an endless path as opposed to it being a very long finite path is a careful distinction. Moreover, Kőnig's Lemma, has ties to the axiom of choice, which should never be treated carelessly.

Proof of Kőnig's Lemma: As before, we'll do this with the Compactness Theorem. For a tree $T$, we'll define a series of sentence letters $P_{x}$ to denote "we select vertex $x$ in our path". The idea will be that—as the sentence suggests—to find a structure that enumerates specific $P_{x}$ at each level to tell us what our vertices will be in our path. The more complex sentences we will use and satisfy are:

• We use at least one vertex at level $n$: $F_n := \bigvee_{i=1}^k P_{n_i}$ where $\{n_1, \cdots, n_k \}$ is the set of vertices at level $n$
  • At level $n$, it must be the case we select either $n_1$, or $n_2$, or…, or $n_k$.
• We pick at most one vertex at level $n$: $G_n := \bigwedge_{i = 1}^{k} \bigwedge_{j = i + 1}^{k} \neg(P_{n_i} \wedge P_{n_j})$
  • For two different vertices $n_i$ and $n_j$ on level $n$, we do not pick both of them in our path
• If we pick a vertex at some level, then we have to pick its ancestors: $H_{xy} := (P_{y} \rightarrow P_{x})$ where $x \preceq y$
  • Since every vertex (except the root) has one parent, there's only one path from the root to that vertex, so naturally a line of ancestors forms our path

Now let $\Gamma = \{F_n \ | \ n \in \mathbb{N} \} \cup \{G_n \ | \ n \in \mathbb{N} \} \cup \{H_{xy} \ | \ x \preceq y \ \forall x,y \in T\}$. Now say there is a structure $A$ that satisfies $\Gamma$. Then it should be clear that the sequence of vertices $B = \{x \in T \ | \ A(P_x) = T\}$ would form an infinite path through our tree $T$:

• The set $\{F_n \ | \ n \in \mathbb{N} \} \cup \{G_n \ | \ n \in \mathbb{N} \}$ specifies we use exactly one vertex at each level, so there's this natural descent we can follow in our tree
• The last set $\{H_{xy} \ | \ x \preceq y \ \forall x,y \in T\}$ says that if we pick one vertex in our tree to form our path, we also pick its ancestors, so we know there is this path of edges that we can actually follow

To show that there is such a structure, we'll use Compactness. Take any finite subset $\Gamma^\textrm{fin} \subseteq \Gamma$. Let $\{x_1, x_2, \cdots, x_k\}$ be the set of vertices that have a corresponding sentence letter $P_{x_i}$ occurs in some element of $\Gamma^\textrm{fin}$. Since $\Gamma^\textrm{fin}$ only talks about a finite number of sentence letters and thus a finite number of vertices, it is also talking about a finite tree of height $n = \max\{\textrm{Lev}(x_1), \cdots, \textrm{Lev}(x_k)\}$ where $\textrm{Lev}(x_i)$ gives the level of vertex $x_i$. So let's pick some vertex at level $n$, and call it $\alpha$. Now the structure

should satisfy $\Gamma^\textrm{fin}$. Since every vertex has exactly one ancestor per level (i.e. its parent, the parent of the parent, etc.), this structure only satisfies one vertex per level as we'd want, and by construction, obviously ensures we select a chain of ancestors to have a path. Even if our subset of our tree is disconnected, this structure would still satisfy $\Gamma^\textrm{fin}$ as if a vertex $x$ has no ancestors, it would vacuously satisfy $H_{xy}$.

Since $\Gamma^\textrm{fin}$ was an arbitrary finite subset of $\Gamma$ and was satisfiable, by the Compactness Theorem, $\Gamma$ is satisfiable, and so for any locally-finite infinite tree, there is an infinite path through it.

There are direct proofs, too, but Kőnig's Lemma itself can be seen as a weaker version of Compactness, so I think proving it both ways is informative.

Order-Extension Principle

As the previous two examples with graphs have shown, results with Compactness are all about extending results in some way. Here's another one that doesn't necessarily rely on infinity like our previous ones, but first we'll need some definitions.

A partial order on a set $X$ is a relation $\preceq$ satisfying:

• Reflexivity: $\forall x \in X \ x \preceq x$
• Antisymmetry: $\forall x,y \in X \ (x \preceq y \wedge y \preceq x \Rightarrow x = y)$
• Transitivity: $\forall x,y \in X \ (x \preceq y \wedge y \preceq z \Rightarrow x \preceq z)$

A total order is a partial order with the additional restriction of

• Connectedness: $\forall x,y \in X \ (x \preceq y \vee y \preceq x)$

So a total order is something in which all elements are able to be compared with one another, while in a partial order some may not be. So all total orders are partial orders, but not all partial orders are total orders. We denote an a partially ordered set as a pair of a set and a partial order defined on the set: $(X, \preceq)$.

An easy example of a total order is $\leq$ on the set of integers $\mathbb{Z}$: every pair of integers contains a greater element. An example of a partial order would be the relation "divides" on $\mathbb{Z}$: 2 does not divide 3, and 3 does not divide 2, so under the relation of divides, 2 and 3 would be incomparable.

It's worth mentioning that there are also strict partial/total orders that remove the reflexivity requirement (think the difference between $$<$$ and $\leq$ on the integers).

Now there is a natural question of whether or not we can remedy partial orders; is there a way we can maintain the structure of a partial order, but find a way to "fix" the incomparable elements. It turns out, we can do just that.

Order-Extension Principle: Any partial order may be extended to a total order.

For any partial order $\preceq$ on a set $X$, we can find a total order $\leq$ on $X$ such that $\forall x,y \in X \ (x \preceq y \Rightarrow x \leq y)$; we can preserve all the original relationships from the partial order, while adding all the missing relations to make a total order without contradicting the criteria above. This isn't completely obvious to me, specifically for the transitivity rule, since it is totally possible to imagine accidentally creating a chain that circles on itself, coming to the conclusion that $x \preceq y \preceq z \preceq \cdots \preceq x$, getting us that distinct elements should be equal to each other by antisymmetry.

Until now, we've been using the Compactness Theorem for propositional logic. For the following proof, we will use the Compactness Theorem for first-order logic, as this will allow us to talk about binary relations—which partial and total orders are. A relation $R$ on a set $X$ is a subset of $X \times X$, so we say that $xRy$ (or $x \preceq y$ in our case) is true iff the ordered pair $\langle x, y \rangle \in R$. The theorem is the exact same statement, just applies to a broader logic.

Proof: Let $X$ be a set, and $\preceq$ be a partial order on $X$. We then add the binary relation $\leq$ to represent our partial order, and $\forall x \in X$, we add a constant $c_x$ to denote it. The following set of sentences will make up our $\Gamma$ that we'll show satisfiable:

1. $\leq$ is a partial order
   • This just specifies the content of $\leq$ so that it behaves how we want it to
2. Whenever $p \preceq q$, we add the sentence $c_p \leq c_q$
   • Similar to sentences of form 1), this keeps the structure between our original partially ordered set and our formalization
3. For every pair $p, q \in X$, we add the sentence $c_p \leq c_q \vee c_q \leq c_p$
   • This is how we'll extend our partial order to a total order by ensuring every pair of elements is compared

Now we'll show that $\Gamma$ is satisfiable. Take some finite subset $\Gamma^\textrm{fin} \subseteq \Gamma$. If $\Gamma^\textrm{fin}$ only contains sentences from rules (1) and (2), then we know $\Gamma^{\textrm{fin}}$ is satisfiable, since $\preceq$ on $X$—which $\leq$ is based on—is a non-contradictory object (I mean, it exists), so that gives us a valid guide to make our structure. So $\Gamma^\textrm{fin}$ is satisfiable in this case.

Otherwise, $\Gamma^\textrm{fin}$ has a finite number of sentences from rule (3). To satisfy these sentences, we will extend our partial order $\preceq$ to another one $\preceq^{*}$. We can construct $\preceq^{*}$ by induction:

• If there is only 1 sentence from rule (3) which isn't already satisfied by $\preceq$, then, say, $p$ and $q$ are incomparable. So let's just state that $p \leq q$, and add this to our relation. So we let $\preceq^{*} \, = \,\, \preceq \cup \, \{\langle p, q' \rangle \ | \ q \preceq q' \ \forall q' \in X \} \cup \{\langle p', q \rangle \ | \ p \preceq p' \ \forall p' \in X \}$ We can't just add $\langle p, q \rangle$ alone to our relation $\leq$, since we need to ensure transitivity. That's why we add these sets of ordered pairs as opposed to just the incomparable pair.
• Now we do this exact same extension as before for the finite amount of sentences from rule (3), and we're done (we have no issue with the Axiom of Choice since we only need to consider finite numbers of sentences).

So $\preceq^{*}$ gives us the precise model to show that $\Gamma^{\textrm{fin}}$ is satisfiable in this case. So in all cases, $\Gamma^{\textrm{fin}}$ is satisfiable, and by Compactness, $\Gamma$ is satisfiable too. (Note this isn't our total order, since it only acts on finite subsets of $\Gamma$, where we want our total order to act on all of $\Gamma$; $\leq$ is our total order)

So there is a structure in which $\Gamma$ is satisfied, which specifies that there is such a binary relation $\leq$ that acts as like a total order on $X$. We can define this total order $\leq^{*}$ specifically by: $p \leq^{*} q \Leftrightarrow (X, \leq^{*}) \vDash c_p \leq c_q$ (we have to distinguish $\leq$ and $\leq^{*}$ because one acts on our set, and the other acts as a binary relation in our language).

Logical Results

I hope the above gives some more concrete examples of how the Compactness Theorem applies to useful mathematics, despite it being a purely logical result. That being said, it being from logic should make it no surprise there are a number of purely theoretic results that are just interesting to consider.

Weakness of Infinity

First-order logic can readily express the size of a model/structure fairly easily with quantification. The simple sentence $\varphi_{\geq 1} = \exists x (x=x)$ is satisfied if and only if the structure it is implemented in contains at least 1 element. $\varphi_{\geq 2} = \exists x \exists y (\neg x = y)$ is satisfied iff the structure contains at least 2 distinct elements. In general, $\varphi_{\geq n} = \exists x_1 \exists x_2 \cdots \exists x_n (\bigwedge_{1 \leq i < j \leq n} \neg x_i = x_j)$ expresses the size of a structure to be at least of size $n$ elements.

Similarly, we can come up with upper bounds on the size of a structure with $\forall$: $\varphi_{\leq n} = \forall x_1 \forall x_2 \cdots \forall x_{n+1}(\bigvee_{1\leq i < j \leq n+1} x_i = x_j)$.

Combining these can then get us exact sizes of structures: $\varphi_{= n} = \varphi_{\leq n} \wedge \varphi_{\geq n}$.

However, note that these only allow us to express specific finite sizes of structures. As it turns out, it is impossible to express in general the finitude of a model.

Claim: There is no sentence $\Phi$ that can be satisfied in and only in finite models.

Proof: Suppose for a contradiction there was such a $\Phi$. Now consider the following set of sentences:

It should be clear that $\Delta_{\infty}$ is only satisfied in an infinite model, since if the model was finite, then there would be some minimum element $\varphi_i$ that would not be satisfied by expressing the finitude of a structure greater than the finite one in question.

Now take $\{\Phi \} \cup \Delta_{\infty}$. This set of sentences should now be unsatisfiable, since $\Phi$ is only satisfied in finite models by assumption, while $\Delta_{\infty}$ is only satisfied in infinite ones. By the Compactness Theorem, there is a finite subset $\Gamma^{\textrm{fin}} \subseteq \{\Phi \} \cup \Delta_{\infty}$ that is unsatisfiable. If $\Gamma^{\textrm{fin}} \subseteq \Delta_{\infty}$, then let $n$ denote the greatest $\varphi_n \in \Gamma^{\textrm{fin}}$. Any model of size equal to or greater than $n$ clearly then satisfies $\Gamma^{\textrm{fin}}$. But since $\Gamma^{\textrm{fin}}$ is unsatisfiable, it must be the case that $\Phi \in \Gamma^{\textrm{fin}}$ as otherwise it would be satisfiable for the reason just listed.

But then in any such structure of size greater than $n$ that satisfies the finite component of $\Delta_{\infty}$, $\Phi$ must be unsatisfied to keep $\Gamma^{\textrm{fin}}$ unsatisfied. But then we have found a finite structure in which $\Phi$ is not satisfied, contradicting our assumption. $\ \blacksquare$

As it turns out, it is also impossible to express the infinitude of a structure.

Claim: There is no sentence $\Phi$ that cannot be satisfied in and only in infinite models.

Proof: Suppose for contradiction there was such a $\Phi$. Then for all finite structures $A$, it is the case that $A \nvDash \Phi$. Thus also in all such structures, $A \vDash \neg \Phi$. Now take the set of sentences $\Gamma = \{\neg \Phi\} \cup \Delta_{\infty}$ (same $\Delta_{\infty}$ as before). Any finite subset of $\Gamma$ is satisfiable, since any finite subset of $\Delta_{\infty}$ is satisfiable for some finite structure, and $\neg \Phi$ is satisfiable in all finite structures. By the Compactness Theorem, $\Gamma$ is satisfiable, so there is some structure $A^+$ such that $A^+ \vDash \Gamma$ i.e. $A^+ \vDash \neg \Phi$ and $A^+ \vDash \Delta_{\infty}$. Since $A^+ \vDash \Delta_{\infty}$, we must have $A^+$ be an infinite structure. But by assumption, $A^+ \vDash \neg \Phi$ if and only if it is finite. Thus there cannot be any such sentence $\Phi$ that is satisfied in and only in infinite structures. $\ \blacksquare$

Just as propositional logic had the weakness of being able to not use quantifiers, we remedy it in strengthened first-order logic. In this way, this inability to express certain statements like the above might be seen as a flaw in first-order logic that needs some fix as well.

Extending Arithmetic and the Hyperreals

The above proofs on the in-expressability of infinity are not all that interesting on their own, but they highlight a proof strategy using that specific set of $\Delta_{\infty}$ to force certain properties to arise out of satisfying $\Gamma$; by its simple construction, $\Delta_{\infty}$ does not usually affect finite satisfiability and hence satisfiability by Compactness, but forces our set $\Gamma$ to behave in a certain way. Here are some weirder consequence these types of proofs can result in.

We all know the standard model of arithmetic: that's just the natural numbers $\mathbb{N}$ with our normal understanding of addition, multiplication, etc. No more than the basic math we learned in elementary school. More formally, this is the standard model of the Peano axioms, as $\mathbb{N}$ satisfies the axioms in the most "obvious" way that we are most familiar with.

But there are also non-standard models containing other numbers that are less commonly seen.

Claim: There is a non-standard model of arithmetic.

Proof: Let $P$ denote the Peano axioms. Now consider the set of sentences

for some new symbol $x$. If we can show $\Gamma$ is satisfiable, then we'll have shown that there is a model that not only satisfies the Peano axioms, but also includes a "number" $x$ that is greater than all other natural numbers.

If we take any finite subset $\Gamma^{\textrm{fin}} \subseteq \Gamma$, then it is satisfiable by the standard model of arithmetic (as those satisfy the Peano axioms), with the addition that $x$ is a number greater than any number mentioned in $\Gamma^{\textrm{fin}}$ (since it'll only have finitely many sentences of the form $x > n$). By Compactness, since all finite subsets $\Gamma^{\textrm{fin}}$ are satisfiable, $\Gamma$ is satisfiable and has a model.

Since a model of $\Gamma$ is a model of $P$ (as it is just a susbet of $\Gamma$), it will be some model of arithmetic. But also, any model of $\Gamma$ that corresponds to $x$ cannot be any typical natural number, since $\Gamma$ states that it is greater than any natural number. So there is a non-standard model of arithmetic. $ \ \blacksquare$

In other words, there is a way to interpret our standard rules and axioms for finite numbers, and somehow apply them to infinite quantities. I don't know about you, but I was taught never to treat infinity like a number, but always as a concept or a process. Yet clearly it's not always contradictory or even bad logic to use them just like normal numbers.

Other non-standard models can have more complicated properties, like certain theorems failing that would be true in the standard model, but nonetheless it is interesting that Compactness implies that even such a model can exist and justifies our use of otherwise strange concepts.

For example, we can also show there is a non-standard model of analysis.

Claim: There is a non-standard model of (real) analysis.

Proof: Real analysis concerns itself with the ordered field of real numbers, so let's have $T$ be the set of sentences that all hold in this field. Now let's have

Like before, if we can show that $\Gamma$ is satisfiable, we'll find a model that satisfies all everything we would expect of the real numbers (from $T$) while also showing that we can introduce a new positive number $\epsilon$ that is smaller than any other number. Again like before, any finite subset $\Gamma^{\textrm{fin}}$ is satisfiable, since any finite subset of $T$ holds in the original model of the real numbers, and for any finite subset of $\{0 < \epsilon < \frac{1}{n} \ | \ n \in \mathbb{N_{>0}}\}$, we just let $\epsilon$ be a number smaller than $\frac{1}{n}$ for the largest $n$ that appears in that set. Since any finite subset $\Gamma^{\textrm{fin}}$ is satisfiable and has a model, by Compactness, $\Gamma$ also is satisfiable and has a model. But clearly this model is non-standard for $\epsilon$ is a positive number that is not identical to any other positive real number, as it is smaller than all other positive reals. $\ \blacksquare \ $ (This also shows the existence of a non-Archimedean ordered field)

We are often reminded in math that we cannot treat the infinitely large and the infinitely small however we want, but there is a rigorous sense in which we can treat them familiarly as with all the other numbers. There is a sense in which you can do calculus and functional analysis without the need for limits, and just use these hyperreal or non-standard real numbers. If $h$ is one of these infintessimal hyperreal numbers (like $\epsilon$ from above), then we can define the derivative of a function $f$ at a point $x$ as

This looks just like the standard definition of the derivative, but instead of having a limit attached to it, we have this new function $\textrm{st}(\cdot)$ that acts as a "rounding function", of sorts, that turns our hyperreal fraction into a real number we can work with. If this interests you, here are some nice notes on the topic.

Topology and the Axiom of Choice

If it isn't clear by all the proofs we've covered, the Compactness Theorem can lend its hand in many places of mathematics. To see just how powerful this is, we will need to explore its connections to other ideas we already briefly touched on. But, it's worth already pointing out, the name is a bit misleading, isn't it?

Topological Compactness

Compactness, as it stands as a property, is more often found in topology than anything, as a way of generalzing the notion of "having no holes". For example, the set $(0,1)$ is not compact as it is missing its endpoints of 0 and 1. But, the closed interval $[0,1]$ is compact. The set of rational numbers $\mathbb{Q}$ is not compact as there are infinite holes in the irrational numbers: you can get as close as you want to any irrational number $r$ with a finite approximation in $\mathbb{Q}$, but clearly $r \notin \mathbb{Q}$, creating a "hole" of sorts.

Definition 1: A subset $S$ of Euclidean space $\mathbb{R}^n$ is compact if $S$ is closed and bounded.

• Closed, meaning that $S$ contains all of its limit points (like the irrationals in the case of $\mathbb{Q}$: you can always get as close as you want to any irrational in $\mathbb{Q}$, but since they are not in $\mathbb{Q}$, it is not closed)
• Bounded, meaning that every point in $S$ is a finite distance away from each other (so even $\mathbb{R} = (-\infty, \infty)$ is not compact as it is unbounded)

In a way, compactness is the next best generalization of being finite; lots of the properties we'd expect of finite sets generalize to compact ones, even though compact sets can be infinite like $[0,1]$. In fact, a finite set is just a discrete and compact set. For example, if a set $S$ is finite, then any continuous function $f: S \rightarrow \mathbb{R}$ is bounded. This trivially holds since if $S$ is finite, then $f(S)$ is also finite and we can just find its maximum and minimum to bound it. However, if $S$ is compact, this still remains true (Boundedness Theorem).

This should ring a bell, as this sort of sounds like the talk of pseudo-finite sets of sentences from the start:

Compactness Theorem: If $\Gamma \vDash \Phi$, then $\Gamma^{\textrm{fin}} \vDash \Phi$.

If a logic is compact, all infinite sets of sentences mimic the behavior of finite subset of themselves. If a space is compact, then a set from that space acts a lot like a finite set. As it turns out, there is a definition that is much definition of topological compactness that even more closely resembles this idea:

Definition 2: A set $S$ is compact if $S$ every open cover of $S$ has a finite subcover.

• An open cover is a collection of open sets $C$ such that $S = \bigcup_{X \in C} X$; an open cover is a collection of open sets in which every element of $S$ is in one of the open sets. For example, the collection of open intervals $\{(-n,n) \ | \ n \in \mathbb{N}\}$ is an open cover of $\mathbb{R}$.
• According to the Heine-Borel theorem, Definitions 1 and 2 are equivalent
• We usually say topologies/metric spaces are compact, as opposed to generic sets
• This definition applies to general topologies, not just Euclidean space $\mathbb{R}^n$

But this isn't the "real" idea of the Compactness Theorem; in all of our proofs, the version of Compactness we really cared about was with finite satisfiability:

Compactness Theorem: If $\Gamma$ is finitely satisfiable, then the whole set $\Gamma$ is satisfiable.

What we really liked was the ability to turn a problem of an infinite set into a more tractable, arbitrary finite one; we could take a local property and make it a global property. Topological compactness has a similar property:

Finite Intersection Property: Given a collection of sets $X$ in a compact space, if every finite subset of $X$ has a non-empty intersection, then the whole set has a non-empty intersection.

• If we wanted to show a collection of sets shared a common element, the finite intersection property would give us a way to demonstrate that like the Compactness Theorem

Referencing the Boundedness Theorem from above, we can think of it as if a function is locally bounded on a compact set, then it is also globally bounded.

There is, however, an even tighter connection between topology and propositional logic.

Claim: The Compactness Theorem (of propositional logic) is equivalent to the claim that its associated valuation space is compact.

• A valuation is an assignment of True or False to every sentence letter (equivalent to a structure for propostional logic)

Just as a reminder, here are the characteristics that determine a topology over a set $X$.

Definition: A topological space $(X, \mathcal{T})$ consists of a non-empty set $X$ and a family $\mathcal{T}$ of subsets of $X$ with the following properties:

• $X, \emptyset \in \mathcal{T}$
• $U,V \in \mathcal{T} \Rightarrow U \cap V \in \mathcal{T}$
• If $U_i \in \mathcal{T}$ for $i = 1,2,\cdots$, then $\bigcup U_i \in T$

Proof: Let $V$ be the set of all valuations of our propositional logic. For every sentence $\phi$, we will assign it a set of valuations in which it's true: $U(\phi) = \{v \in V \ | \ v(\phi) = 1\}$. Now notice that for sentences $\phi_1$ and $\phi_2$,

• $V = \bigcup U(\phi)$ ranging over all sentences $\phi$
  • This is easier seen by $V = U(P) \cup U(\neg P)$ for a sentence letter $P$
  • So the set $\{U(\phi) \ | \ \phi \textrm{ is a sentence}\}$ is a cover of $V$
• $U(\phi_1) \cap U(\phi_2) = \{v \in V \ | \ v(\phi_1) = 1\} \cap \{v \in V \ | \ v(\phi_2) = 1\} = U(\phi_1 \wedge \phi_2)$
  • This is by definition of the truth table for $\wedge$
  • So $U(\cdot)$ is stable under intersection (we got another $U(\cdot)$ after intersection)

These two properties implies that $\{U(\phi) \ | \ \phi \textrm{ is a sentence}\}$ forms a basis of a topology over $V$.

• A basis of a topology $\mathcal{B}$ is a family of open sets such that for a topology $\mathcal{T}$, every set in $\mathcal{T}$ can be expressed as a union of sets from $\mathcal{B}$
• The set $\{(a,b) \ | \ a,b \in \mathbb{R}\}$ is a basis for a (the standard) topology on $\mathbb{R}$

So we are interested in the topological space $(V, \mathcal{T})$ where $\mathcal{T}$ is the topology generated by the basis $\{U(\phi) \ | \ \phi \textrm{ is a sentence}\}$.

Now lets consider some set of sentences $\Gamma$. $\Gamma$ is unsatisfiable if and only if for all valuations, at least one sentence in $\Gamma$ is false. Or equivalently, $\Gamma$ is unsatisfiable if and only if for all valuations, at least one sentence's negation $\neg \varphi$ is true. In terms of our $U(\cdot)$ function, we can also say $\Gamma$ is unsatisfiable if and only if each valuation is in one of the open sets (i.e. members of the topology) of the form $U(\neg \varphi)$ for $\varphi \in \Gamma$ (since $U(\neg\varphi)$ is precisely the set of valuations that make $\neg\varphi$ true). Equivalently, this also means that $\Gamma$ is unsatisfiable if $\{U(\neg\varphi) \ | \ \varphi \in \Gamma\}$ is an open cover of our topological space $V$ (recall what an open cover is from above if this isn't clear). So this shows the equivalence that

Now recall the Compactness Theorem: if $\Gamma$ is unsatisfiable, then there exists a finite subset $\Gamma^{\textrm{fin}}$ of $\Gamma$ that is unsatisfiable. So this gives us the equivalence:

We can make this an if and only if since the Compactness Theorem gives us the $(\Rightarrow)$ direction, and the $(\Leftarrow)$ direction comes trivially by definition of unsatisfiability: if there is an unsatisfiable subset of $\Gamma$, then of course all of $\Gamma$ is unsatisfiable.

So, given the Compactness of propositional logic, we can combine these biconditionals to deduce that the Compactness Theorem is equivalent to the claim:

And that is precisely the definition of a compact space (see Definition 2 above).

We were working with normal propositional logic above, but note that we didn't really assume anything about our logic in the proof. If we are working with an arbitrary logic with its own rules, symbols, connectives, etc., we could show that the Compactness Theorem holds if $V$ is compact. If you're interested in further reading the connection between logic and topology, Tychonoff's theorem is actually the broader theorem that has the consequence of implying the Compactness Theorem (of propositional logic).

The Axiom of Choice

The above shows, in my opinion, a fairly strong and fundamental use connection between logical compactness, to some more concrete math; it not only gives reason to its name, but also puts it into context of its broader origins.

There is an arguably EVEN MORE FUNDAMENTAL connection to math that the Compactness Theorem appeals to: the Axiom of Choice. We unwittingly saw this earlier when we proved König's Lemma, a weaker form of the Axiom of Choice.

To see this, it's worth considering a non-Compactenss proof of König's lemma. Here's a quick sketch:

Alternative Sketch of König's Lemma: Let $v_0$ be the root of our finitely-branching tree $T$ with children $\{c_1, \cdots, c_k \}$. Now let's define $S_i = \{v \in T \ | \ c_i \textrm{ is the ancestor of } v \}$. These sets encapsulate possible vertices we can reach from a given child of $v_0$. Since each vertex has precisely one parent, it follows that $T \ \backslash \{v_0\} = S_1 \cup \cdots \cup S_k$ (we trivially take each vertex to be one of its own ancestors; you are related to yourself, no?). Since $T$ is an infinite tree, at least one of the $S_i$ has to be infinite, too (if they were all finite, then there would only be a finite number of vertices in our tree; think Pigeonhole Principle). So we pick a child $c_i$ with an infinite associated $S_i$, and let $v_1 = c_i$. Now we list the children of $v_1$ and simlarly construct there associated descendant sets $S_i$. At least one of the children of $v_1$ must have an infinite set of descendants, so we select that child and let it be $v_2$. We repeat this process over and over again, at each step, looking at the children of our previously selected vertex, and see which one has an infinite set of descendants, and let that be our next step in our infinite path $v_0, v_1, v_2, \cdots$ and so on. $\ \blacksquare$

There's a certain level of choice necessary here. At each step, we're picking a vertex with a certain property (infinite descendants), which is fine since we were able to show that there is at least one vertex at each step with that property. However, if there are more than one vertex, then which vertex do you choose? The need for the axiom of dependent choice, while weaker than the full axiom of choice, is nonetheless a choice principle that König's lemma depends on. In turn, you can also use König's lemma to prove Compactness, giving the connection we want.

Spelled out a bit more, the idea is that the Compactness Theorem can be thought of as finding an infinite path in a tree of sentence letters and their negations: say $\Gamma = \{\Phi_1, \Phi_2, \cdots \}$ using sentence letters $\{P_1, P_2, \cdots\}$, and we let our tree be a binary tree, with each vertex containing exactly two children. We will let then find a path through the tree, where each step at level $n$, we can either take $P_n$ or $\neg P_n$ to be true (this can be seen as going on either the left or right branch of the tree). Any finite path of this tree defines a valuation, and in combination with the finite satisfiability of $\Gamma$ and König's lemma, you can find an infinite path that determines a structure that satisfies all of $\Gamma$.

But again, there's a level of choice necessary in picking a sentence letter or its negation. The Compactness Theorem can be seen as that specific choice principle.

Some other choice principles that Compactness is equivalent to the ultrafilter lemma, and the Completeness Theorem, and some others, all of which are strictly weaker than the full axiom of choice. But again, this should not be that surprising given what we've proven with Compactness, and the general nature of infinity itself. The Order-Extension Principle (which strongly resembles the ultrafilter lemma in form), while weaker than the axiom of choice, cannot be proven without it. I sort of even led you astray earlier, as Tychonoff's theorem from topology—that implies the Compactness Theorem—is actually just equivalent to the full axiom of choice.

Something so controversial and so important (literally makes up an entire letter in ZFC—Zermelo-Frankel set theory with the axiom of choice) is inextricably tied to the Compactness Theorem.

Proof of the Compactness Theorem

This last section is to offer a proof of the Compactness Theorem for propositional logic to close. Since this will be a proof of a logical result, this section would definitely benefit from knowing some basic logic, just at the very least to more easily read notation.

Remember, propositional logic does not include the first-order quantifiers $\forall$ and $\exists$. To show Compactness holds in first-order logic (like we used in the Order-Extension Principle), we'll have to amend our proof.

We'll prove the following form of the Compactness Theorem:

Compactness Theorem: If $\Gamma$ is finitely satisfiable, then the whole set $\Gamma$ is satisfiable.

Proof of Compactess: The idea of the proof is we'll extend $\Gamma$ by squeezing in sentence letters in such a way that maintains finite satisfiability. By doing so, that will allow us to determine a structure that will satisfy $\Gamma$ by essentially finding what sentence letters we have to make true from our extended $\Gamma$, allowing us to carry out all the finite "comparisons" that force a structure to converge to one that will satisfy $\Gamma$. We'll break it down step-by-step.

1) Definition of $\Gamma_i$: By our hypothesis, say that $\Gamma$ is finitely satisfiable. The set of sentence letters occurring in $\Gamma$ are enumerable, so let's do that with $P_1, P_2, \cdots$. Now let's define the following series of supersets. Let $\Gamma_0 = \Gamma$ and

2) Each $\Gamma_i$ is finitely satisfiable: We'll prove this by induction. $\Gamma_0 = \Gamma$ is finitely satisfiable by assumption. Now suppose $\Gamma_i$ is satisfiable for $i \leq n$. For the sake of contradiction, assume $\Gamma_{n+1}$ is not i.e. neither $\Gamma_n \cup \{P_{n+1}\}$ or $\Gamma_n \cup \{\neg P_{n+1}\}$ are finitely satisfiable. So there are finite subsets $\Delta$ and $\Sigma$ of $\Gamma_{n}$ such that $\Delta \cup \{P_{n+1}\}$ and $\Sigma \cup \{\neg P_{n+1}\}$ are unsatisfiable. Each of these sets must include $P_{n+1}$ and $\neg P_{n+1}$ since both $\Delta, \Sigma \subseteq \Gamma_{n+1}$ which is finitely satisfiable, so every finite subset of $\Gamma_{n}$ is satisfiable.

Further, clearly if $\Delta \cup \{P_{n+1}\}$ is unsatisfiable, clearly $\Delta \cup \Sigma \cup \{P_{n+1}\}$ is unsatisfiable (adding elements to a contradictory set won't make it less contradictory). Similarly, if $\Sigma \cup \{\neg P_{n+1}\}$ is unsatisfiable, then so is $\Delta \cup \Sigma \cup \{\neg P_{n+1}\}$. Since $\Delta \cup \Sigma \subseteq \Gamma_{n}$ is a finite subset from a finite satisfiable set, $\Delta \cup \Sigma$ is satisfiable. So in any structure/valuation in which $\Delta \cup \Sigma$ is true, it must be the case that either the sentence letter $P_{n+1}$ or its negation $\neg P_{n+1}$ is true. But we just said that both the sets $\Delta \cup \Sigma \cup \{P_{n+1}\}$ and $\Delta \cup \Sigma \cup \{\neg P_{n+1}\}$ are unsatisfiable.

This is a contradiction, so our initial assumption must be wrong, and $\Gamma_{n+1}$ must be finitely satisfiable.

3) Defining a structure from the $\Gamma_i \textrm{s}:$ For every sentence letter $P_i$, we will define a structure $A$ as follows:

In step 2, we showed that for all $n$, $\Gamma_n$ is finitely satisfiable. Since

is a finite subset of $\Gamma_n$, the set $\Phi_n$ (just the set of sentence letters or their negations in $\Gamma_n$) is also satisfiable. By construction of our structure $A$, for every $n$, every sentence letter or its negation $\alpha \in \Phi_n$ is true in $A$. In particular, though, $\Phi_n$ is a finite subset, so we only need a finite amount of sentence letters to be assigned specific truth values to satisfy $\Phi_n$. So any structure which agrees (i.e. assigns the same truth value) to the members of $\Phi_n$ as $A$ does will also make satisfy $\Phi_n$ (this might seem obvious—if we give the same sentence letter values that satisfy $\Phi_n$ to those sentence letters, of course $\Phi_n$ will be satisfied—but this will be important in defining a full structure later).

4) Showing $A$ satisfies $\Gamma$: Suppose for contradiction $A$ does not satisfy $\Gamma$. That is, for some sentence $\psi \in \Gamma$, $|\psi|_A = F$. By definition of a logical sentence, $\psi$ must be of finite length, and hence only have finitely many sentence letters occuring in it. Let $P_k$ be the highest numbered sentence letter (from our previous ordering in step 2) that occurs in $\psi$. As we discued in step 3, every $\alpha \in \Phi_k$ is true in a structure that agrees with $A$. Hence, $\psi$ must be false in every such structure too (since such a structure would fully determine the truth value of $\psi$ in the exact same way as $A$ did).

So we can conclude that $\Phi_k \cup \{\psi \}$ must be unsatisfiable. But $\Phi_k \cup \{\psi \} \subseteq \Gamma_k$ by construction of $\Gamma_k$. So by the finite satisfiability of $\Gamma_k$, it must be that $\Phi_k \cup \{\psi \}$ is also satisfiable. Contradiction. Therefore, our structure $A$ must satisfy $\Gamma$.

Since we found a structure that satisfies $\Gamma$, obviously it is satisfiable. So we've shown that if $\Gamma$ is finitely satisfiable, then itself is satisfiable.

Conclusion

In a way, Compactness should sort of seem obvious. This idea of finite satisfiability basically allows us to make a bunch of finite comparisons of a property within a set, slowly but surely forcing the entire set to also conform to that property. In this case, it was satisfiability, but, say, if it was the property that every finite collection of numbers is prime, you'd expect your whole set to also be prime. Yet, obviously this is still a theorem, and in fact is not always true for all logics. Second-order logic does is not a compact logic. But when it is true, it is an immensely powerful tool, allowing us to extend seemingly local properties to global ones. Compactness has made a name for itself in analysis and topology, but logic has given us a method to take that beyond simple set theory to graphs, computability, and all the way to the foundational axioms that make up modern math.

I've been on a kick for mathematical foundations recently. Maybe it's all that logic I've been looking at. One of those key ideas that finds its way into math is the Principle of Mathematical Induction. It's likely even if you've only taken high school math classes, you've encountered induction in one way or another. The importance of induction stems all the way back to the all-important, characterizing Peano axioms of arithmetic. In fact, any formal system that can perform induction is a critical indicator of not just the strength, but flaws of the system, as induction is the "simplest" cause of a logical system being incomplete (as in Gödel's incompleteness theorems).

Principle of Mathematical Induction

The usual idea of induction is proof by "climbing a ladder", or "knocking over mathematical dominoes". Let's have $\Phi(n)$ stand for "the theorem $\Phi$ for the case of natural number $$n$$". Then if you can show that

• (PMI 1) $\Phi(0)$ is true
• (PMI 2) $\forall k \geq 0$, if $\Phi(k)$ is true, then $\Phi(k+1)$ is true

Then you can conclude that $\Phi(n)$ is true for all natural numbers $n \geq 0$. The idea is that in property (PMI 2), the theorem $\Phi$ is self-fulfilling; there is an inherent quality to our theorem $\Phi$ that has it proves many cases for itself. If we know $\Phi(0)$ is true, then by (PMI 2) we know $\Phi(1)$ is true. Similarly, from $\Phi(1)$ we then know $\Phi(2)$ is true, and then from $\Phi(2)$ we know $\Phi(3)$ and so on. We climb the mathematical ladder starting with a simple case $\Phi(0)$ we know for a fact is true.

This is best seen with an example.

We will show the formula for the sum of the first $n$ integers is $\sum_{k=1}^n k = \frac{n(n+1)}{2}$.

Base Case (PMI 1): The formula holds for $n=1$ since $\sum_{k=1}^1 k = 1 = \frac{1(1+1)}{2}$.

Inductive Hypothesis (PMI 2): Let's assume our formula works for the first $n$ integers. We want to show that it holds for case $n+1$: $\sum_{k=1}^{n+1} k = \frac{(n+1)(n+2)}{2}$. We will show this by first rewriting our sum: $\sum_{k=1}^{n+1} k = \sum_{k=1}^{n} k + (n+1)$. By our assumption, we can reduce this to $\frac{n(n+1)}{2} + n+1$. Simplifying further,

Which is precisely the formula our hypothesis predicted. So by the PMI, for all $n$, $\sum_{k=1}^n k = \frac{n(n+1)}{2}$.

Here, we took a statement that can naturally be indexed by the natural numbers—in this case, a formula—and using the method of induction, proved our claim that the statement held. Note here that we didn't show $\Phi(0)$ is true, but rather $\Phi(1)$. This minor change isn't a big deal, since what we really want to show is a base case to build our inductive claim off of; we don't care where our ladder starts, so long as we can show there is some ground for it to stand on.

Induction is a very strong tool, as it does and can usually be applied to any claim that can be indexed by the integers; if there is a natural way to "count" the cases of your hypothesis, odds are induction could be useful.

However, note that induction didn't give us the arithmetic sum formula to begin with, but rather only a method to verify it. Some amount of intuition or case work needs to be done beforehand to get a hypothesis to follow through with induction.

Let's do a few more examples. Here's a simple one from set theory.

De Morgan's Laws: For sets $A_1, A_2, \cdots, A_n$, we have

Just as a reminder, for sets $A$ and $B$,

• $A \cup B = \{x \ | \ x \in A \textrm{ or } x \in B\}$
• $A \cap B = \{x \ | \ x \in A \textrm{ and } x \in B\}$
• $A^c = \{x \ | \ x \notin A\}$

Notice how we're not working with integers directly, but rather integer amounts of sets. So, we'll then induct on the number of sets as our natural division of cases.

Base Case: We'll show this for $n = 2$. Let $x \in (A \cap B)^c$ be an arbitrary element. Then $x \notin A \cap B$, so we have either $x \notin A$ or $x \notin B$ by the rules for $\cap$. Equivalently, $x \in A^c$ or $x \in B^c$, so regardless of which set $x$ is in, it must be the case that $x \in A^c \cup B^c$. Hence, since every element of $(A \cap B)^c$ is an element of $A^c \cup B^c$, so $(A \cap B)^c \subseteq A^c \cup B^c$. To summarize,

Going the other way, suppose we have an arbitrary element $x \in A^c \cup B^c$. For the sake of contradiction, assume $x \notin (A \cap B)^c$. The rest of the proof is very similar to above.

The only way to sets can both be subsets of each other is if there equal to each other. So by double containment, $(A \cap B)^c = A^c \cup B^c$.

Inductive Hypothesis: Let's assume our formula works for some number of $n$ sets. We want to show this works for $n+1$ sets. Fortunately most of our work was already done in the base case.

Since $(A_1 \cap \cdots \cap A_n)$ is just another set, we can first use the case for 2 sets in the 2nd equality, and the case for $n$ sets in the last equality.

The proof for the second of De Morgan's Law is proved almost identically.

Let's pull another example from linear algebra.

Claim: Let $A$ be a square matrix. The eigenvectors of distinct eigenvalues of $A$ are linearly independent.

As a reminder:

• An eigenvector is a non-zero vector $\textbf{v}$ with corresponding eigenvalue $\lambda$ of matrix $A$ such that $A \textbf{v} = \lambda v$
• We call set of vectors $\{\textbf{v}_1, \cdots, \textbf{v}_k \}$ linearly independent when $a_1 \textbf{v}_1 + \cdots + a_k \textbf{v}_k = 0$ if and only if $a_1 = a_2 = \cdots = a_k = 0$ for scalars $a_1, \cdots, a_k$

We'll induct on the number of eigenvectors.

Base Case: For one eigenvector, it's clearly true, as $\textbf{v}_1 \neq 0$ by definition. So $a_1 \textbf{v}_1 = 0$ if and only if $a_1 = 0$.

Inductive Hypothesis: Now suppose that the claim is true for fewer than $n$ eigenvectors of distinct eigenvalues. So suppose that

We want to show $a_1 = a_2 = \cdots = a_n = 0$. Let's apply $A$ to both sides of $(*)$:

where each $\lambda_i$ are the associated eigenvalues. Now let's also multiply $(*)$ by $\lambda_1$:

Now let's subtract equation (2) from (1).

By the inductive hypothesis, we know the $n-1$ eigenvectors $\textbf{v}_2,\textbf{v}_3,\cdots,\textbf{v}_n$ are linearly independent, so all the coefficients $a_i(\lambda_i - \lambda_1) = 0$. But also remember, each $\lambda_i$ are distinct eigenvalues, so $\lambda_i - \lambda_1 \neq 0$ for $i=2,\cdots,n$. So the only way $a_i(\lambda_i - \lambda_1) = 0$ is if $a_i = 0$ for $i = 2,\cdots, n$. Plugging this back into $(*)$, we get that $a_1 \textbf{v}_1 = 0$, and since $\textbf{v}_1 \neq 0$, it must be that $a_1 = 0$.

Thus, all coefficients $a_1 = a_2 = \cdots = a_n = 0$, so the eigenvectors are linearly independent, having proven the inductive hypothesis.

Now induction doesn't always have to prove a positive claim. In fact, it can be combined with a proof by contradiction to prove a negative claim.

Claim: $\cos(1^{\circ})$ is irrational.

Proof: First, let's note the angle sum formula for cosine.

We can combine these to get the identity

Now let's assume for contradiction that $\cos(1^\circ)$ is rational. Now from our identity we just derived, let's note that

Then, if we let $N = 1$, we can get an expression for $\cos(2^\circ)$.

By our assumption that $\cos(1^\circ)$ is rational, $2\cos^2 (1^\circ) - 1$ is also rational, thus $\cos(2^\circ)$ is rational. Then using our identity

We get $\cos(3^\circ)$ is rational by letting $N=2$. Then from that we get $\cos(4^\circ)$ is rational by letting $N=3$, and so on. Since we've proven our base cases for $\cos(1^\circ)$ and $\cos(2^\circ)$ are rational, we get inductively that $\cos(n^\circ)$ is rational for all natural numbers $n$.

In particular, then according to our induction, $\cos(30^\circ)$ is rational. But $\cos(30^\circ) = \frac{\sqrt{3}}{2}$, which is irrational. Contradiction! Thus, our initial assumption that $\cos(1^\circ)$ is rational must have been wrong, or in other words, it must be that $\cos(1^\circ)$ is irrational.

Choosing a Base Case

In our formulation of induction, we said in particular:

• (PMI 1) $\Phi(0)$ is true

We specified our base case to be $n=0$. But if you look at our example proofs, we used $n=1$ and $n=2$ as base cases sometimes, such as in the proof of De Morgan's Laws. Usually the base case doesn't matter, as induction would just say that for a base case $k$, a claim holds for all $n \geq k$. But we have to be careful of our choice of base case. For example, here's a false inductive proof.

Claim: All horses are the same color.

Base Case: For $n=1$, obviously a horse is the same color as itself.

Inductive Hypothesis: Suppose all sets of $n$ horses are the same color. Suppose we have $n+1$ horses, enumerated in some way (we assign each horse a number $1$ through $n+1$ to identify them). By the inductive hypothesis, the horses $1$ through $n$ are the same color. By the same inductive hypothesis again, horses $2$ through $n+1$ are the same color too. So all the horses are the same color as horses $2$ through $n$, thus showing that all $n+1$ horses are the same color. $\ \blacksquare$

Clearly this proof is wrong. Clearly some horses are brown, some are black, some are white, and some are patterned. Yet our "proof" would suggest otherwise. The issue lies in our base case, since it cannot be used to prove the case for $n=2$ horses. The key idea in the "proof" was to use the set of "intermediary" horses $2$ through $n$ to give some way of comparing horse $1$ with horse $n+1$ indirectly. But if there are only 2 horses, there is no middle horse to compare them to. If you want to prove that all horses are the same color in this way, you'd have to show it initially for 2 horses to have any chance of a real argument (which of course, isn't true).

But as we saw in our proof that $\cos(1^\circ)$ is irrational, choosing a faulty base case can be useful. There, we purposely picked a bad base case (i.e. by assuming $\cos(1^\circ)$ is rational) to get to an observable contradiction to then prove a negative claim. The point still stands, though, you must be careful with how you choose and apply your base cases.

1954237295746 ≠ ∞

Similar to the importance of choosing a base case, we must also be aware of what exactly induction is saying. The conclusion we get from an inductive argument is that $\Phi(n)$ is true for all natural numbers $n$. Note this doesn't mean it's true at a limit to $\infty$, it is only true for any arbitrarily large $n$. Consider the following faulty proof:

Claim: $\pi$ is rational.

Base Case: We'll induct on the number of decimal points in the expansion for $\pi$. For $n=0$, clearly $3$ is rational.

Inductive Hypothesis: Say the truncation of $\pi$ to the $n$th decimal place, call it $x_n$ is rational. Then $\pi$ to the $(n+1)$st decimal can be written as $x_{n+1} = x_n + \frac{m}{10^{n+1}}$ for $m \in {0,1,\cdots,9}$. So $x_{n+1}$ is rational. So all decimal expansions of $\pi$ are rational, so $\pi$ is rational. $\ \blacksquare$

Clearly this is also wrong, and it's just by the misinterpretation we spelled out now: there is a difference between being true in the limit and true at each individual step to the limit.

Weak Induction vs. Strong Induction

There's something else weird in our inductive proof of De Morgan's Laws. Again, let's look at our specific outline of induction:

• (PMI 2) $\forall k \geq 0$, if $\Phi(k)$ is true, then $\Phi(k+1)$ is true

The inductive step says to show $\Phi(k)$ implies $\Phi(k+1)$. Yet, if you look at our proof of De Morgan's Laws, we had to use not only the case that $\Phi(k)$ was true, but also $\Phi(2)$ was true; we needed to assume that the law holds for not only $n$ sets, but for 2 sets as well. I mean, clearly if we assume that $\Phi(k)$ is true and $k \geq 2$, then $\Phi(2)$ should be true, but that's not what we explicitly allowed in our Principle of Mathematical Induction.

This is an instance of strong induction. The formulation of mathematical induction we gave before was weak induction, but the difference between them is minimal. If you can show that a theorem $\Phi$ holds for:

• (SPMI 1) $\Phi(0)$ is true
• (SPMI 2) $\forall k \geq 0$, if $\Phi(0)\wedge\Phi(1)\wedge\cdots\wedge\Phi(k)$ is true, then $\Phi(k+1)$ is true

then you can conclude that $\Phi(n)$ is true for all $n$. The difference between strong induction and weak induction is just that strong induction allows you to use multiple cases in your proof, while weak induction only specifies the previous case.

Here's a nice example of strong induction:

Claim: A country has $n$ cities. Suppose that any two cities are connected by a one-way road. Then there is a route that passes through every city.

Base Case: Clearly it is true for $n=2$: just take the one-way road that leads out from one city and into the other.

Inductive Hypothesis: Now suppose this holds up to and for $n$ cities. We want to show this holds for $n+1$ cities. Let's take the $(n+1)^{\textrm{th}}$ city, and divide the remaining cities into two groups: cities that have a road into city $n+1$ (call this group $A$) and cities that have roads coming out of city $n+1$ (group $B$).

Now obviously group $A$ has $n$ or fewer cities, so by the inductive hypothesis, there is a route that passes through all of the cities in group $A$. Similarly, for the same reason, there is a route that passes through all the cities in group $B$.

Now, take a route that passes through all the cities in $A$, then stop at the $(n+1)^{\textrm{th}}$ city, then finally go to a city in $B$ and complete a route that passes through all those cities. We can do that since every city in $A$ has a road to $n+1$, and every city in $B$ has a road from $n+1$.

Hence, the inductive hypothesis holds, and proves our claim. $\ \blacksquare$

We needed strong induction above since we have no idea how many cities are in groups $A$ and $B$, so we need the claim to hold for all values up to $n$, not just $n$ itself, in the inductive hypothesis. The renewed inductive hypothesis given in strong induction is—obviously so—much more applicable than its weaker counterpart, just by sheer introduction of additional inductive instances.

Despite the naming, though, they are actually equal in strength; anything you can prove with weak induction you can prove with strong induction, and anything you can prove with strong induction you can prove with weak induction.

Strong Induction Implies Weak Induction: It shouldn't be much of a surprise that if strong induction holds, then weak induction also holds, since weak induction is just like strong induction but looser requirements. Let's prove it anyway. Suppose that strong induction is a valid proof technique. We want to show that weak induction is also valid. That is, from assumptions

• (WPMI 1) for some base case $k$ we know $\Phi(k)$ is true, and
• (WPMI 2) $\forall m \geq k$, $\Phi(m)$ is true $\Rightarrow$ $\Phi(m+1)$ is true,

we want to show that $\Phi(n)$ is true for all $n \geq k$. Obviously, whenever (WPMI 1) is true, then so is (SPMI 1) since they are both the same clause. However, let's also note that whenever (WPMI 2) is true, so is (SPMI 2). To see this, let's suppose (WPMI 1) is true. For any $m \geq k$, if $\Phi(k)\wedge\Phi(k+1)\wedge\cdots\wedge\Phi(m)$ is true, then certainly $\Phi(m)$ alone is true. By (WPMI 2), then we also know $\Phi(m+1)$, thus showing (SPMI 2) holds.

Having this fact in hand, we can complete the proof. Let's assume strong induction holds. Then, if the assumptions (WPMI 1) and (WPMI 2) for weak induction are true, we also know the assumptions (SPMI 1) and (SPMI 2) for strong induction are also true. Then by the conclusion for strong induction, we know $\Phi(n)$ holds for all $n$. But this is the exact same conclusion that we would get from weak induction. Hence weak induction holds whenever strong induction does.

Again, this shouldn't be that surprising, as every instance of weak induction is just an instance of strong induction with weaker hypotheses.

Weak Induction Implies Strong Induction: This direction will require a bit more work. Let's assume that weak induction is a valid proof technique. We want to show that strong induction is valid. That is, from the assumptions

• (SPMI 1) for some base case $k$ we know $\Phi(k)$ is true, and
• (SPMI 2) $\forall m \geq k$, $\Phi(k)\wedge\Phi(k+1)\wedge\cdots\wedge\Phi(m)$ is true $\Rightarrow$ $\Phi(m+1)$ is true,

we want to show that $\Phi(n)$ is true for all $n \geq k$. What we will do is induct on a meta-statement of $\Phi$. Let $S(m)$ be the statement "$$\Phi(n)$$ is true for all $$k \leq n \leq m$$". If we can show $S(n)$ is true for all $n$ (with regular induction), then we would also show $\Phi(n)$ is true for all $n$ as well.

From (SPMI 1), we get $\Phi(k)$ is true, and thus $S(k)$ is also true (this will be our base case). From (SPMI 2), we get that for all $m \geq k$, if $S(m)$ is true, then $\Phi(m+1)$ is true (since $S(m)$ is equivalent to the hypothesis of the conditional). But, if $\Phi(m+1)$ is true and $S(m)$ is true, then we also get that $S(m+1)$ is true by definition. Combining these two conditionals, we get that if $S(m)$ is true, then $S(m+1)$ is true (this is our inductive step).

Combining our base case and inductive hypothesis for $S(n)$, we can deduce via weak induction, that $S(n)$ is true for all $n \geq k$. And as described before, it must be the case that $\Phi(n)$ is also true for all $n \geq k$, which is precisely the conclusion that strong induction wants.

Here's a more intuitive version of this proof, but relies on knowing some basic logic to justify a direct argument. The idea is the same, but the above is more "contained" in my opinion, while the link is more readily understandable.

So whenever, one proof looks like it requires strong induction, it could also be done with weak induction. For example, with De Morgan's Laws, you could imagine having shown the case for $n=2$, you could use weak induction to show $n=3$. Similarly,

So, perhaps a more appropriate name would just to say "mathematical induction", for weak and strong induction are both cut from the same cloth.

"Proving" Induction

We showed that weak and strong induction are equivalent, meaning if one holds so does the other. But, we never showed that either form of induction actually holds. Intuitively, I think the analogy of dominos or climbing a ladder makes induction only seem natural, but with all things in math, we do need to justify using induction rigorously somehow. Fortunately, a lot of the work in the proof is given by the axioms that define the natural numbers. We'll first prove weak induction.

Weak Induction: Let $S$ be a set of natural numbers with the following properties:

1. $0 \in S$
2. If $k \in S$, then $k+1 \in S$

Then $S$ is the set of all natural numbers.

Although this looks different to our above applications of induction, all we need to do is define our set such that $S = \{ n \in \mathbb{N} \ | \ \Phi(n) \textrm{ is true} \}$. Hopefully this is clear to be equivalent to our previous uses.

Proof: We will prove this by contradiction. So suppose that there is some set of natural numbers $T \neq \varnothing$ that are not in $S$. By the well-ordering principle, $T$ has a smallest element—any two distinct natural numbers has a lesser one among them two. Let's call this least element $\alpha$.

• Since $0 \in S$ by $(1)$, we have $0 < \alpha \notin S$
• By (the contrapositive) of $(2)$, we also have $\alpha - 1 \notin S$
• Since $\alpha$ is the least element of $T$, we have $\alpha - 1 \notin T$, or in other words, $\alpha - 1 \in S$
• This is a contradiction!

So, our initial assumption that such a non-empty set $T$ exists must be false, and thus it must be that $S$ contains all natural numbers. $ \ \blacksquare$

Even though we've proven the equivalence of weak and strong induction already, we can also prove the validity of strong strong induction independently in an almost identical fashion.

Strong Induction: Let $S$ be a set of natural numbers with the following properties:

1. $0 \in S$
2. If $0,1,\cdots,k \in S$, then $k+1 \in S$

Then $S$ is the set of all natural numbers.

Proof: We'll also prove this by contradiction. Suppose $T$ is the set of integers not in $S$, so $T$—by well-ordering—must have a least element $\alpha$.

• Since $0 \in S$ by $(1)$, we have $0 < \alpha \notin S$.
• By $(2)$, we have at least one element of $\{0,1,\cdots,\alpha -1\}$ that's not in $S$
• But since $\alpha$ is the least element, $0,1,\cdots,\alpha -1 \notin T$ and so $0,1,\cdots,\alpha -1 \in S$
• But then there is at least one element from $\{0,1,\cdots,\alpha -1\}$ that is both in and not in $S$
• Contradiction!

Since the only assumption we made was $T$ was non-empty, it must be wrong, and thus $S$ is the set of all natural numbers. $ \ \blacksquare$

Well-Ordering Principle

In both proofs, we relied on the well-ordering of the natural numbers. A well-order is a relation $\leq$ on a set $X$ with the following properties:

1. Reflexivity: $\forall x \in X \ x \leq x$
2. Antisymmetry: $\forall x,y \in X \ (x \leq y \wedge y \leq x \Rightarrow x = y)$
3. Transitivity: $\forall x,y \in X \ (x \leq y \wedge y \leq z \Rightarrow x \leq z)$
4. Connectedness: $\forall x,y \in X \ (x \leq y \vee y \leq x)$
5. Minimums: Every non-empty subset $\varnothing \neq S \subseteq X$ contains a least element, i.e., $\exists \alpha \in S$ such that $\forall x \in S \ \alpha \leq x$

The set of natural numbers $\mathbb{N}$ is well-ordered under the relation "less than" $\leq$. This is something that is just an intuitive fact that we take for granted when working with the natural numbers, but it is a very special property given by the axioms of the natural numbers, and it will become important to us later on. This simple fact, though, is actually equivalent to the principle of mathematical induction itself!

Induction Implies Well-Ordering: Suppose for contradiction that $\mathbb{N}$ is not well-ordered. That is, there is a subset $\varnothing \neq S \subseteq \mathbb{N}$ with no least element. Then $0 \notin S$ as it would be then be minimal. Then $1 \notin S$ as $1$ would be minimal as $0 \notin S$. Suppose $0,1,\cdots,n \notin S$. Then $n+1 \notin S$ as it would be minimal. By induction, $\forall n \in \mathbb{N} \ n \notin S$. That is, $S = \varnothing$, which is a contradiction. Thus $\mathbb{N}$ is well-ordered. $ \ \blacksquare$

Well-Ordering Implies Induction: Assume $\mathbb{N}$ is well-ordered. We want to show that if a subset $S\subseteq \mathbb{N}$ with the properties that

1. $0 \in S$
2. If $k \in S$, then $k+1 \in S$

then $S = \mathbb{N}$.

So for a contradiction, suppose $S \neq \mathbb{N}$. That is, there is a set of natural numbers $\varnothing \neq T \subseteq \mathbb{N}$ not in $S$. By well-ordering, $T$ has a least element $\alpha$. From assumption $(1)$, we're saying $0 \in S$, so $0 \notin T$. Therefore, $\alpha = k + 1$ for some $k \in \mathbb{N}$. But if $\alpha = k + 1 \in T$, $k + 1 \notin S$. And by assumption $(2)$, if $k + 1 \notin S$, then $k \notin S$ and further $k \in T$. But $k < k+1 = \alpha$, contradicting the minimality of $\alpha$.

So if $T \neq \varnothing$, it contains no minimal element, contradicting the well-ordering of $\mathbb{N}$. Therefore, it must be that $T = \varnothing$, or in other words, $S = \mathbb{N}$, validating induction. $ \ \blacksquare$

That's enough metatheory. Let's get back to some applications.

Variations on Induction

The simple idea of induction is far more flexible than our statement of it may seem. With some clever combinations of inductive arguments, we can prove many things in much simpler ways than typical induction may suggest.

Forward-Backward Induction

Induction hopefully should clearly work in an "forward" direction at this point; we can climb the integer ladder up and up so long as we have one rung to start on. Similarly we can have a "backward" inductive (BI) argument: if you can show that for some claim $\Phi$ that

• (BI 1) For some base case $k$ we know $\Phi(k)$ is true
• (BI 2) $\forall m \geq k$, $\Phi(m)$ is true $\Rightarrow$ $\Phi(m-1)$ is true

then, naturally, we should be able to conclude that $\Phi(n)$ is true for all $n \leq k$. But we can use this idea in tandem with our normal form of induction to get a nice combined approach with forward-backward induction. If you can show that

• (F-B 1) $\Phi(n)$ is true for infinitely many $n$
• (F-B 2) For all $k$, if $\Phi(k)$ is true, then $\Phi(k-1)$ is true

then you can conclude that $\Phi(n)$ is true for all positive $n$. The idea is that you only need to show that $\Phi$ is true for an infinite amount of $n$, not necessarily all $n$. Then using (F-B 2), we can fill in the gaps between our infinite $\Phi(n)$ by working backwards from the proven cases in (F-B 1). Here's a more specific, but common example of forward-backward induction.

• For some base case $k$ we know $\Phi(k)$ is true
• $\forall m \geq k$, $\Phi(m)$ is true $\Rightarrow$ $\Phi(2m)$ is true
• $\forall m \geq k$, $\Phi(m)$ is true $\Rightarrow$ $\Phi(m-1)$ is true

Here we show that all the powers of 2 (scaled by our base case) satisfy $\Phi(n)$, and then fill in all the gaps with our third condition that works backwards.

So really, forward-backward induction is just two inductive arguments being put together, which is more common than it may seem. For example, if you can show that

• $\Phi(0)$ and $\Phi(1)$ are true
• For all $k \geq 0$, $\Phi(k)$ is true $\Rightarrow$ $\Phi(k+2)$ is true

then $\Phi(n)$ is true for all $n$ as we just showed that $\Phi(n)$ is true for all even numbers and odd numbers separately; $\Phi(0)$ with the inductive step gets you the evens, while $\Phi(1)$ gets the odds. Forward-backward induction is the same idea, only combining two other inductive proofs to "fill in the gaps".

Let's put it to practice.

Arithmetic Mean-Geometric Mean (AM-GM) Inequality: For non-negative real numbers $a_1,a_2,\cdots,a_n$,

with equality given if and only if $a_1 = a_2 = \cdots = a_n$.

Base Case: Clearly $n=1$ holds, so we'll also show it for $n=2$. Note that $\left(\sqrt{a_1} - \sqrt{a_2} \right)^2 \geq 0$. So by simple algebra,

Forward-Inductive Hypothesis: Let's assume the AM-GM inequality holds for some $n$. We'll show it also holds for $2n$.

Hence proving the forward inductive claim using cases for $2$ and $n$ non-negative numbers.

Backward-Inductive Hypothesis: Again we suppose AM-GM inequality holds for $n$ numbers. We will show it holds for $n-1$ numbers. First, though, since AM-GM holds for $n$ numbers, it will also hold when we let $a_n = \frac{a_1 + \cdots + a_{n-1}}{n-1}$ the average of the previous $n-1$ numbers.

Thus proving the backward-inductive hypothesis.

By forward-backward induction, we've shown that AM-GM holds any number of non-negative real numbers.

Proof by Infinite Descent

Building on this idea of using backwards induction, Fermat used backward induction in a very neat way. Because, well, there's something weird going on in our forward-backward induction, notably the backwards step:

• (F-B 2) For all $k$, if $\Phi(k)$ is true, then $\Phi(k-1)$ is true

This doesn't seem all that problematic, but while the natural numbers are unbounded upwards, they are bounded downwards by 0. From $\Phi(k)$, we get $\Phi(k-1)$, then further $\Phi(k-2)$, and so on until we get $\Phi(0)$. But from $\Phi(0)$ can we conclude $\Phi(-1)$? The backwards inductive step would certainly suggest so, but that's not how natural numbers work. In the AM-GM inequality above, what would it mean for it to hold for "-1 numbers"? It just doesn't make sense.

The difference here is that we do have an understanding that backwards induction terminates eventually at 0. In particular, what our backwards inductive statement really should say is that

• (F-B 2) For all $k\in \mathbb{N}$, if $\Phi(k+1)$ is true, then $\Phi(k)$ is true

We first assume that $k \in \mathbb{N}$ before we apply it to our claim we're proving. But what if we didn't assume that?

Proof by infinite descent exploits this assumption to make a very nice proof by contradiction. If we remove this condition assuming that one of our numbers is already a natural number, we could prove things by contradiction by showing that is a consequence. That is,

• (Suppose for contradiction that) $\Phi$ is true
• If $m \in \mathbb{N}$ and $\Psi(m)$ is true, then $m-1 \in \mathbb{N}$ and $\Psi(m-1)$ is true

Then you could conclude that $\Phi(n)$ is false for all $n \in \mathbb{N}$ since there is no infinitely descending sequence of natural numbers. $\Psi(n)$ here could be any consequence that would imply an infinitely decreasing sequence of natural numbers.

This is best done with an example.

Claim: If $\sqrt{k}$ is not an integer, then it is irrational.

Proof: Suppose for a contradiction that there is a positive integer $k$ such that $\sqrt{k}$ is not an integer and not irrational, i.e. $\sqrt{k}$ is rational. So there are natural numbers $m,n \in \mathbb{N}$ such that $\sqrt{k} = \frac{m}{n}$. Now we can do some algebra on this expression:

By multiplying by $\frac{\sqrt{k} - \lfloor \sqrt{k} \rfloor}{\sqrt{k} - \lfloor \sqrt{k} \rfloor} = 1$, we ended up with a new rational expression for $\sqrt{k}$. But notice, by definition of the floor function $\lfloor \cdot \rfloor$, we multiplied the numerator and denominator by $\sqrt{k} - \lfloor \sqrt{k} \rfloor < 1$. So $m > m(\sqrt{k} - \lfloor \sqrt{k} \rfloor) = nk - m\lfloor \sqrt{k} \rfloor$ and similarly for the denominator. But the numerator and denominator are both still natural numbers, since $m$, $n$, $k$, and $\lfloor k \rfloor$ are all natural numbers, so their sums and products are too!

So from a rational expression for $\sqrt{k} = \frac{m}{n}$, we found another equivalent rational expression with a strictly smaller numerator and denominator. So we could repeat this process again multiplying through by $\frac{\sqrt{k} - \lfloor \sqrt{k} \rfloor}{\sqrt{k} - \lfloor \sqrt{k} \rfloor}$ to find another smaller rational expression ad infinitum. But there are no infinitely decreasing sequences of natural numbers for the numerator and denominator to be a part of, giving us a contradiction.

So it must be that our initial assumption was wrong that there is a $\sqrt{k}$ that is neither and integer nor irrational. In other words, it for all natural numbers $k$, either $\sqrt{k}$ is an integer or it is irrational.

Here's a geometric application of infinite descent applied not to natural numbers directly, but rather integral multiples of a length of a line.

Multi-Induction

Let's revisit the very first inductive proof we gave. We showed that the formula for the sum of the first $n$ natural numbers is

There's a similar formula for a related double sum:

You might try to prove this with induction, but there's a sizeable difference between our original problems and this one: it's a formula of multiple variables.

As such, there is also a natural extension to what we've been doing in multi-induction. A way to think of it is like the "product" of inductive arguments; if we were "inducting linearly" before, we'll now "induct in a grid".

For our case of double induction, we'll represent our formula is true with $\Phi(m,n)$—just like what we did before with one variable. We'll then use the following formulation for double induction (DI): we'll show that

• (DI 1) $\Phi(1,1)$ is true
• (DI 2) For all $m \geq 1$, show $\Phi(m,1)$ is true
• (DI 3) For each fixed $m_0$ and for all $n \geq 1$, show that $\Phi(m_0, n)$ is true

to prove that $\Phi(m,n)$ is true for all integers $m,n \geq 1$. If you think of $(m,n)$ as coordinates in space, (DI 2) is the "horizontal" induction, proving the claim along the $x$-axis and giving us the base cases for (DI 3), which fills in the columns along the $y$-axis.

And as you can probably guess, we'll show (DI 2) and (DI 3) with induction.

Claim: $\sum_{j=1}^m \sum_{i=1}^n (i + j) = \large{\frac{mn(m+n+2)}{2}}$

Base Case: We'll check it for the case $m = n = 1$. $\sum_{j=1}^1 \sum_{i=1}^1 (i + j) = 2 = \frac{1 \cdot 1(1+1+2)}{2}$.

Horizontal Inductive Hypothesis: We'll induct on $m$ and prove that the formula holds for all $m \geq 1$ while fixing $n=1$. So suppose the formula holds for some $m$, that is, $\sum_{j=1}^{m} \sum_{i=1}^1 (i + j) = \frac{m(m+3)}{2}$. We'll show that it also holds for $m+1$:

Since $\Phi(m,1) \Rightarrow \Phi(m+1,1)$, by induction, $\Phi(m,1)$ is true for all $m \geq 1$.

Vertical Inductive Hypothesis: Horizontal induction now gives us the base cases we need to induct on $n$. We now know that for a fixed $m_0$, $\Phi(m_0, 1)$ is true, so that will act as our base case. Now assume that $\Phi(m_0,n)$ is true for some $n$. That is,

We will show that $\Phi(m_0, n+1)$ is also true.

We get the 3rd line equality by the inductive hypothesis, the 4th line is just an arithmetic series, and the rest follow from algebra. Thus, $\Phi(m_0, n) \Rightarrow \Phi(m_0, n+1)$, and by induction, $\Phi(m_0,n)$ is true for a fixed $m_0$ and all $n \geq 1$.

But $m_0$ was arbitrary. So $\Phi(m,n)$ is true for all $m,n \geq 1$. In other words, for all $m,n \geq 1$,

Transfinite Induction

In our original proofs of weak and strong induction, we relied primarily on the well-ordering of the natural numbers. Actually, that's really about all we cared about. Let's look again at our statement of (strong) induction from that proof:

Strong Induction: Let $S$ be a set of natural numbers with the following properties:

1. $0 \in S$
2. If $0,1,\cdots,k \in S$, then $k+1 \in S$

Then $S$ is the set of all natural numbers.

Let's rewrite this with a bit more notation.

Strong Induction Again: Let $S \subseteq \mathbb{N}$ be a subset with the following properties:

1. $0 \in S$
2. If $\forall m \leq k \ \ m \in S$, then $k+1 \in S$

Then $S = \mathbb{N}$.

If all that really mattered was the idea of $\mathbb{N}$ being well-ordered—this idea that there's a lesser element, or a "next" element in the set—what's stopping us from changing $\mathbb{N}$ with any set $A$ that is well-ordered under a relation $\leq$ (that is not necessarily "less than or equal")? There is one issue though, that comes in the second clause for strong induction:

1. If $\forall m \leq k \ \ m \in S$, then $k+1 \in S$

Sometimes we can have well-ordered sets–that is have this concept of a "next" element—but have it such that there is no "good" definition of having a next, or $+1$ element. I know this sounds contradictory, but this will help if we talk it out from the start. So far we've been working with the natural numbers, so that seems like a good place to work out of. But first, we need to understand what the natural numbers are.

Finite Ordinals

In general, we use the the natural numbers to count things, like the sizes of sets. So oftentimes, we do just that! Let's look at a construction of the natural numbers via sets that do exactly that (due to von Neumann):

von Neumann's Construction of $\mathbb{N}$: We will define them recursively.

• $0 = \varnothing$
• $n+1 = n \cup \{n\}$ (this is the successor of a number)

So the first few natural numbers with this definition would be

If it's easier, you can also think of each number being defined as the set of all previous numbers, meaning that $n+1 = \{0, 1, \cdots, n\}$. So the number, say $4$, could be thought of as how many numbers are in the set $\{0,1,2,3\}$. If you're interested, you can look into how to do all the standard arithmetic with these set operations, but the only point that we care about is how we can compare them.

Since we're working with sets, the idea of "less than or equal" doesn't really make sense, but fortunately, we built into our definition a nice counterpart to represent "less than or equal". Our well-ordering relation is "contains" $\in$: for two numbers, if $a \in b$, then we have $a < b$. This can be seen easiest with the second definition of a number being the set of all previous numbers. We call these numbers that relate to each other in this set-theoretic way as ordinals. In particular,

• Ordinals are well-ordered under the relation $\in$, and
• Ordinals also have the property that whenever $x \in S$, it's also the case that $x \subseteq S$ as well.

Ordinals generalize the way we would normally count things to infinite sizes. The natural numbers are also known as the finite ordinals.

So every natural number can be identified with the set of all natural numbers before it. And the set of all these finite ordinals are well-ordered, and we can do induction on them like we have been doing before. Great!

Infinite Ordinals

But look at our definition of an ordinal. We're defining them in this recursive manner, but there's nothing saying that they have to be of finite size. So what happens if we just make an infinite ordinal? So far all of our ordinals have been of finite natural numbers, what's stopping us from extending one of these sets to infinity?

Well, suppose we did take the limit of the set of natural numbers. That is the set, let's call $\omega$, defined as

This set is certainly greater than any other natural number under the relation $\in$—in fact, it is the supremum of the natural numbers, being the smallest ordinal greater than all of the natural numbers. Also, certainly by the way we defined the natural numbers, every element of $\omega$ is also a subset of $\omega$, so $\omega$ fits the definition of an ordinal too.

But think about that for a second. If we take the set of all natural numbers alongside this infinite ordinal in the set $\mathbb{N} \cup \{\omega\}$, we definitely have a well-ordered set, but how could we ever do induction on this set? Because $\omega$ is greater than all natural numbers, it definitely does not have the form of $k+1$ for some natural number $k$. Yet, according to our definition of induction

• (SPMI 2) If $\forall m \leq k \ \ m \in S$, then $k+1 \in S$

those are the only statements we could build off of. No matter how long you let our induction go for, it will never be able to prove a statement is true for the ordinal $\omega$, while still being in a well-ordered set.

This is the purpose of transfinite induction. When working to prove things for any well-ordered set, we can run into these limiting cases ($\omega$ is aptly called a limiting ordinal) that don't work completely well with the notion of having a "next element" at infinity; you can imagine labelling elements in a well-ordered set as with 0, 1, 2, etc. as we have been doing before, and then eventually running into an element $\omega$ to throw your induction off. It's always infinity that makes things a little more difficult.

The Principle of Transfinite Induction

The principle of transfinite induction will look extremely familiar to what we've already been doing.

Transfinite Induction: Let $A$ be a well-ordered set, and $\Phi(x)$ be a proposition that has domain $A$ with the following properties:

• (TI 1) $\Phi(0)$ is true
• (TI 2) $\Phi(b)$ is true for all $b < a$, then $\Phi(a)$ is true

Then you can conclude that $\Phi(x)$ is true $\forall x \in A$.

$0$ just means the least element of $A$. The only real difference again is in the inductive step. In (TI 2), we replace the idea of having $k+1$ with just a notion of "greater than" as to deal with possible limiting cases. In fact, transfinite induction is presented as normal induction with an additional condition:

Transfinite Induction: Let $A$ be a well-ordered set, and $\Phi(x)$ be a proposition that has domain $A$ with the following properties:

• (TI 1: Base Case) $\Phi(0)$ is true
• (TI 2a: Successor Case) If $\Phi(\beta)$ is true for all $\beta \leq \alpha$, then $\Phi(\alpha+1)$ is true
• (TI 2b: Limiting Case) $\Phi(\beta)$ is true for all $\beta < \alpha$, then $\Phi(\alpha)$ is true

Then you can conclude that $\Phi(x)$ is true $\forall x \in A$.

(TI 2a) is essentially just the normal form of (strong) induction we've come to know and love, with the only difference being that we can have successor ordinals that are not natural numbers, like $\omega + 1$. So it really is just the limiting cases that can throw a wrench into our proofs.

Let's now put it to work.

Examples

Claim: There is a subset $A \subset \mathbb{R}^2$ that intersects every line in exactly two points.

This is not clear to me at all. I mean, just by literally the sheer number of lines there are, it seems hard to make it so that you can nudge points around so that no three are ever collinear, while still somehow covering every line. We can uniquely identify every line with an equation $y = ax + b$, so we can identify every line with an ordered pair $(a,b)$. Thinking of this as coordinates, it should be clear that the number of lines is equivalent to the size of $\mathbb{R}^2$, which is more than the number of natural numbers. So if we want to prove this via induction, we'll need to do so with transfinite induction.

Proof: Let $\{ L_\alpha \}$ be a labelling of every line (i.e. assign every line an ordinal $\alpha$). We'll inductively construct a sequence $\{A_\alpha\}$ of subsets of $\mathbb{R}^2$ with the following properties for each $\alpha$:

• (Instantiation) $A_\alpha$ has at most two points
• (Preservation) $\bigcup_{\beta \leq \alpha} A_\beta$ contains no three points collinear
• (Diagonal) $\bigcup_{\beta \leq \alpha} A_\beta$ contains exactly two points of $L_\alpha$

Then $A = \bigcup A_\alpha$ over all ordinals $\alpha$ will have the desired property. This is because the preservation property will ensure every line has at most two points in $A$, while the diagonal property will ensure that every line has at most two points in $A$.

Base Case: $\alpha = 0$. $L_0$ would be our first line, and all we need to do to satisfy those 3 properties above is let $A_0$ be any two points in $L_0$.

Successor Case: Suppose for any ordinal $\alpha$, the sequence $\{A_\beta\}_{\beta \leq \alpha}$ satisfies the 3 properties above. We'll show that we can construct a satisfactory $A_{\alpha + 1}$ to extend the sequence.

Let $B = \bigcup_{\beta \leq \alpha} A_{\beta}$, and $C$ be the set of all lines that contain two points from $B$. By the inductive hypothesis, $B$ has the preservation property, so line $L_{\alpha + 1}$ has at most 2 points in $B$, i.e. $L_{\alpha + 1} \cap B$ has at most 2 points. Now we have two cases to consider.

Case 1: If $L_{\alpha + 1} \cap B$ has exactly 2 points, we just let $A_{\alpha+1} = \varnothing$ to immediately satisfy the 3 properties.

Case 2: If $L_{\alpha + 1} \cap B$ has less than 2 points, then line $L_{\alpha+1}$ intersects every line in $C$ in at most 1 point (if it intersected a line in more than 1 point, it would be that line itself), i.e. for all lines $L \in C \ \ |L_{\alpha+1} \cap L| \leq 1$. To satisfy the preservation and diagonal property, we obviously want to pick a point in $L_{\alpha + 1} \backslash \bigcup C$ (i.e. a point on our new line but not on any of the previous lines). We can do this for sure since we can show $L_{\alpha + 1} \backslash \bigcup C \neq \varnothing$.

• The set of all points in a line has equivalent cardinality to the reals (it's just a tilted number line, after all). So $|L_{\alpha + 1}| = |\mathbb{R}| = 2^{\aleph_0}$.
• Every line can be uniquely written as $y = ax+b$, and thus also be identified by the ordered pair $(a,b)$. So the number of lines is equivalent to $|\mathbb{R}^2| = 2^{\aleph_0}$
• $C$ is the set of all lines with two points from $B$, which by definition, does not include all lines yet. So $|C| < 2^{\aleph_0}$
• Using the fact that for all lines $L \in C \ \ |L_{\alpha+1} \cap L| \leq 1$: $|L_{\alpha + 1} \cap \bigcup C| = |\bigcup_{L \in C} L_{\alpha + 1} \cap L| \leq |C| < 2^{\aleph_0}$
• So $L_{\alpha + 1}$ and $\bigcup C$ share fewer than $2^{\aleph_0}$ elements, together, meaning that $L_{\alpha + 1} \backslash \bigcup C \neq \varnothing$.

Now we're free to pick a subset from $A_{\alpha + 1} \subset L_{\alpha + 1} \backslash \bigcup C$. If $L_{\alpha + 1} \cap B$ has one point, just pick 1 point for $A_{\alpha + 1}$, and if $L_{\alpha + 1} \cap B$ has no points, then let $A_{\alpha + 1}$ 2 points. This will satisfy our preservation and diagonal properties.

Limit Case: Suppose for any ordinal $\alpha$, and for all $\gamma < \alpha$, the sequence $\{A_\gamma\}_{\beta \leq \gamma}$ satisfies the 3 properties above. We want to show that we can construct a $A_{\alpha}$ to extend the sequence. Notice in our successor argument, we never really needed the fact that we were proving it for $A_{\alpha + 1}$, but just the fact that we had a series of already constructed previous cases of $A_\beta$ with $\beta < \alpha + 1$. This makes our limit case almost identical to the successor case. Here's the outline:

• Let $B = \bigcup_{\gamma < \alpha} A_\gamma$, and $C$ be the set of all lines that contain two points from $B$
• Now consider the set $L_\alpha \cap B$
• $L_\alpha \cap B$ contains at most 2 points because $\{A_\gamma\}_{\beta \leq \gamma}$ has the preservation property
• Case 1: If $L_\alpha \cap B$ contains 2 points, let $A_{\alpha} = \varnothing$
• Case 2: If $L_\alpha \cap B$ contains less than 2 points, then $L_{\alpha}$ intersects with every line in $C$ at most once
  • By the same reasoning as before we can pick a subset $A_{\alpha} \subset L_{\alpha} \backslash \bigcup C \neq \varnothing$
  • If $L_\alpha \cap B$ has 1 point, let $A_{\alpha}$ be 1 point
  • If $L_\alpha \cap B$ has 0 points, let $A_{\alpha}$ be 2 points

Now again let $A = \bigcup A_\alpha$ over all ordinals $\alpha$. By transfinite induction, we've shown there is a subset $A \subset \mathbb{R}^2$ that intersects every line exactly twice.

Outside of set theory, because there are more reals than natural numbers, transfinite induction lends itself to these types of weird geometry problems.

Though, I have to admit I did lie a little bit here. We technically used transfinite recursion as opposed to transfinite induction. We didn't necessarily prove a claim for all ordinals $\alpha$, but rather constructed something inductively/recursively over the ordinals that satisfied some properties. The difference is subtle, but they are technically different methods from one another. Maybe you could argue that we were proving the claim that "there is something constructible" for all ordinals, so I find the difference nitpicky.

Examples of transfinite induction can be tedious (especially if uncomfortable with ordinals and working with infinities), but you can find some in this book for more. Some interesting examples include:

• There is a countable partition of $\mathbb{R^2}$ so that the distance between any two different points in the same set is irrational
• $\mathbb{R^2}$ is not a union of disjoint circles
• $\mathbb{R^3}$ is a union of disjoint circles

Structural Induction

The idea of well-ordering—a key idea in "climbing a ladder"—naturally led us to generalize induction to transfinite induction. But there's something else in our idea of induction that, to me at least, seems more characteristic to the proof technique: recursion.

We've already seen recursion a little bit, but this idea of defining things within themselves is what induction really screams to me. In that second clause of our principle,

• (PMI 2) $\forall k \geq 0$, if $\Phi(k)$ is true, then $\Phi(k+1)$ is true

we rely on proving the validity of $\Phi(k+1)$ by already assuming the "simpler" idea $\Phi(k)$. How do we know $\Phi(k)$ is valid? We show it from the "simpler" $\Phi(k-1)$. And we keep going until we hit the simplest version of our claim with our base case $\Phi(0)$.

Building up examples from a simplest case is what I think of when I use induction. But, there are many other times we build something from a base or simple case, too. Trees, in computer science and graph theory, follow this exact pattern of construction.

We have a root at the top that brances out into different leaves and paths. While we may not have a definite "next" element going down a tree, we certainly have this recursive nature where at each step going down a tree, we have a smaller, "simpler", subtree. So perhaps, if we wanted to prove somthing about trees, there's a type of inductive argument we could use that modifies our original idea. For a propsition (like a property) $\Phi$ about a recursively defined structure like a tree, we could have somthing like

• Show $\Phi$ is true for the base case of the structure
• If $\Phi$ holds for all substructures, then $\Phi$ also holds for the recursively defined structure

then we should be able to conclude that $\Phi$ holds for the entire structure. This type of structural induction is something that should make sense based on our previous, regular versions of induction, but it's something so general we'll have to adapt it to our structures.

Here's a simple example building on trees. First, we need some definitions.

• A vertex is a dot in our tree.
• An edge is a line that connects two vertices

We define a tree recursively:

• (Base Case) A tree has a root vertex with no edges, or
• (Recursive Definition) A tree has a root vertex with edges connecting to some number of other trees

Claim: For our tree, let $V$ be the set of vertices and $E$ the set of edges. Then $|V| = |E| + 1$.

Base Case: Our base case is a tree is a root vertex with no edges. So $|V| = 1$, and $|E| = 0$, so the claim holds.

Inductive Hypothesis: Suppose that we have a tree with $n$ subtrees, and the claim holds for all subtrees. We will use $V_i$ and $E_i$ to denote the set of vertices and edges respectively for tree $i=1,\cdots,n$.

Then by definition, the number of vertices of the whole tree

as the tree is built from a root vertex connected to the subtrees. Similarly,

as the number of edges would be the sum of the edges of the subtrees plus the additional $n$ edges that connect the subtrees to the root. Now it's just a matter of algebra with the inductive hypothesis.

Here's another one with recursively defined sets.

Claim: Define the set $S$ as follows:

• (Base) $6,21 \in S$
• (Recursive) If $x,y \in S$, then $x + y \in S$

Show that every element in $S$ is a multiple of 3.

Base Case: Clearly $6 = 3 \cdot 2$ and $21 = 3 \cdot 7$ are both multiples of 3

Inductive Hypothesis: Assume that $x,y \in S$ and both are multiples of 3 i.e. $x = 3n$ and $y = 3m$ for integers $n,m \in \mathbb{Z}$. Then $x + y = 3(n + m)$ which is also a multiple of 3. So clearly all elements are multiples of 3. $\ \blacksquare$

Finally, one more from propositional logic. In propositional logic, we define sentences recursively with connectives. These and only these are sentences:

• (Base) Sentence letters $P, Q, R, \cdots$ are sentences
• (Recursive) If $\phi$ and $\psi$ are sentences then the following are also sentences:
  • $\neg \phi$ (negation)
  • $(\phi \wedge \psi)$ (and)
  • $(\phi \vee \psi)$ (or)
  • $(\phi \rightarrow \psi)$ (material implication)
  • $(\phi \leftrightarrow \psi)$ (biconditional)

Claim: All sentences have an equal number of left and right parantheses.

Base Case: Sentence letters have no parantheses, and $0=0$, so they have an equal number of left and right parantheses.

Inductive Step: Suppose $\phi$ and $\psi$ are sentences with an equal number of left and right parantheses. We check the claim for each connective:

• $\neg \phi$: Since we add no new parantheses and $\phi$ has an equal amount of left and right, the claim holds.
• $(\phi \wedge \psi)$: If $\phi$ has $n$ left and right parantheses, and $\psi$ has $m$, then $\phi \wedge \psi$ has $n+m$ left and right parantheses, still being equal. Adding parantheses around it keeps the equality, just adding 1 to each side.
• The remaining connectives are identical to the $\wedge$ case

Structural induction is a fairly straightforward adaptation of induction, and, as a proof technique, is helpful to formalize what might otherwise just be obvious facts (like the above).

Induction was something we only limited to well-ordered sets. Structural induction allows us to use it on well-founded partial orderings too. Well-founded is similar to well-ordered, meaning that given a relation $R$ on a set $X$, every non-empty subset has a minimal element. A partial order is a relation $R$ on a set $X$ that relates elements in a set $X$ to one another, but without necessarily comparing all elements to each other. Going back to the example of trees, we can relate elements by being in the same chain of edges, so different branches would be uncomparable to one another.

Continuous Induction

Up until now, we've treated induction as an inherently discrete method. I mean, how can it not be? We started by first describing a method that is iterating over the natural numbers, and then further generalized them to any well-ordered set. But nonetheless, we only applied induction to inherently discrete casees. What would it even mean to climb a "continuous" ladder? What would a "next step" look like? For any real number $x$ and some incremental value $\delta$, there's always a number $y$ in between $x < y < x + \delta$, so the real numbers don't give us this nice "next step" to prove (with this, it shouldn't be hard to see that the real numbers aren't even well-ordered).

Let's think about what we're doing when we use induction. It sort of feels like an almost constructive method, right? We first show there is at least one element in our set with our base case. Then from our base case, we add another element through the inductive step. From that new element, we add another element with another inductive step. And so on and so on. We treat induction like a method in which we add more and more elements to our set.

This is inherently rooted in thinking in terms of an indexed claim $\Phi(n)$. If we want to continue this analogy, we really need a way of thinking that gives us a new "next step". Let's revisit our definition of induction when we were proving it from well-ordering:

Principle of Mathematical Induction: Let $S \subseteq \mathbb{N}$ be a subset with the following properties:

1. $0 \in S$
2. If $k \in S$, then $k+1 \in S$

Then $S = \mathbb{N}$.

Again, we could define $S = \{ n \in \mathbb{N} \ | \ \Phi(n) \ \textrm{is true} \}$, but the clear issue is still our inductive step. What will be our "next step", or $k+1$? Well, if there's no good indexing of the reals and we can't include any particular one inductive step, why don't we just include a bunch? As in, why don't we just include a new subset of numbers at each inductive step? What could be wrong with that? Our induction then would be that if we can show that some number $x \in S$, then we can show that some further interval is also in $S$, as opposed to just a single other number. That way we can just keep adding more and more interval until we reach as many reals as we want!

Real Induction Attempt 1: Let $S \subseteq \mathbb{R}_{\geq 0}$ be a subset with the following properties:

1. $0 \in S$
2. For any $x \geq 0$, if $x \in S$ then $[x, z) \subseteq S$ for some $z > x$

Then $S = \mathbb{R}_{\geq 0}$.

Instead of dominoes, our analogy would be like improvising a rope: so long as you have smaller threads that have some length, you can eventually tie a rope as long as you want.

There is one problem with our formulation, though: it doesn't ensure a certain size for our intervals. So if our intervals are shrinking in length, they might converge onto only some smaller part of the real line. Say we start at 0, but at each step $n$ only add an interval of length $(\frac{1}{2})^n$. Then we'll only cover the reals from $[0,1)$. So we need at least one more requirement for this to work.

And actually, we have run into this issue before. Recall for transfinite induction, we had limit ordinals that blocked our path, and had to prove the limiting cases in addition to the inductive step. We will do the same here.

Real Induction: Let $S \subseteq \mathbb{R}_{\geq 0}$ be a subset with the following properties:

• (RI 1) $0 \in S$
• (RI 2) For any $x \geq 0$, if $[0,x] \subseteq S$ then $[x, z] \subseteq S$ for some $z > x$
• (RI 3) For any $x \geq 0$, if $[0,x) \subseteq S$ then $x \in S$

Then $S = \mathbb{R}_{\geq 0}$.

So now if we run into a limit case like before, we say it must be in our set, and that gives us a new case to continue performing real induction on. One other thing to note is that I tweaked our second condition (RI 2) to be closer to the strong form of induction from before to highlight there is an equivalent analogy in real induction. I also changed (RI 2) to be a closed interval, as our new (RI 3) allows us to do that. If a set $S$ satisfies (RI 1–3), we say $S$ is inductive.

So real induction isn't all that different from what we're used to already. Instead of natural numbers being our cases, intervals are. We start with a base case, and slowly show our claim holds on new intervals until we end up showing that from combining all those intervals, the claim actually holds for all real numbers.

You might be wondering how this is at all useful. Particularly because of (RI 2): we no longer have a concrete $k+1$ we can work with in our proofs. Instead, we're stuck with this weird existence proof we have to find that doesn't seem tangible at all in practice. That's what I, at least, thought at first. Fortunately, much of real analysis has built into their definitions this same ambiguous existence-of-a-number that we can leverage (as you'll see).

Earlier I pointed out how we might run into the "issue" of only proving a claim for the interval $[0,1)$, but sometimes that's exactly what we want as opposed to showing something for all reals. We can slightly modify our method of induction just by limiting our choices of possible elements:

Interval Induction: Let $S \subseteq [a,b]$ be a subset with the following properties:

• (RI 1) $a \in S$
• (RI 2) For any $a \leq x < b$, if $[a,x] \subseteq S$ then $[x, z] \subseteq S$ for some $z > x$
• (RI 3) For any $a < x \leq b$, if $[a,x) \subseteq S$ then $x \in S$

Then $S = [a,b]$.

All we did is change our base case, and add a cap to our inductive step.

While we're here, we might as well check beyond our intuition that this should work.

Proof of Real Induction: Suppose a set $S \subseteq [a,b]$ is inductive, and (RI 1–3) hold. We want to show that $S = [a,b]$. So—just like in our proof of regular induction—suppose for contradiction, $S \neq [a,b]$, so there is a non-empty set $T = [a,b] \backslash S \neq \varnothing$. Since $T$ is bounded, and by the completeness of the reals, $T$ has an infimum (greatest lower bound).

• Case 1: $\mathbf{\inf (T) = a}$. By (RI 1), $a \in S$. By (RI 2), there is a $z > a$ such that $[a,z] \subseteq S$. So $z$ is a lower bound for $T$ (the elements not in $S$). But $z > a = \inf (T)$, contradicting the definition of an infimum.
• Case 2: $\mathbf{a < \inf(T) \in S}$. $\inf(T) \neq b$, since we declared $T$ non-empty. Then by (RI 2), there is a $z > \inf(T)$ such that $[\inf(T), z] \subseteq S$. Like before, this would suggest $z > \inf(T)$ is a lower bound for $T$, contradicting the definition of an infimum.
• Case 3: $\mathbf{a < \inf(T) \in T}$. Then by definition of $T$, we would have $[a,\inf(T)) \in S$. By (RI 3), then $\inf(T) \in S$, which is again a contradiction by definition of $T$.

Since these 3 cases are exhaustive for the infimum of $T$, and all lead to a contradiction, it must be the case that our initial assumption is wrong. That is, that $S \neq [a,b]$. In other words, if (RI 1–3) hold on our desired interval, it must be that $S = [a,b]$.

So now we have this method of showing a claim holds for more and more subintervals that build on each other, until it completes the whole interval, allowing us to prove theorems in a now more familar way than before.

Here's a nice intuitive example.

Intermediate Value Theorem: Suppose $f:[a,b] \rightarrow \mathbb{R} \backslash \{0\}$ is continuous, and $f(a) > 0$. Then $f(b) > 0$.

If $f(x)$ is continuous and only maps numbers to non-zero real numbers, then it should make sense that if we have a point $a$ where $f(a) > 0$, then we should not have any number where $f(x) \leq 0$ without "lifting up our pen"; since $f(x) \neq 0$, we have this barrier where either our function is continuous and positive, or it is not continuous and negative too.

The idea of this proof is to use continuity to inductively string together intervals that are positive on $f$ until we cover the whole length of $[a,b]$. Just for consistency, the set we're secretely working with (as from our statement of real induction) is $S = \{x \in [a,b] \ | \ f(x) > 0\}$.

Base Case: From our assumption, we're already given that $f(a) > 0$.

Inductive Hypothesis: Say that for some $a \leq x < b$ that $f(t) > 0$ for all $t \in [a,x]$. We want to show that there is a further interval $z > x$ where $f(t) > 0$ for all $t \in [x,z]$.

Since $f$ is continuous at $x$, there is a small neighborhood centered at $x$ with also positive outputs i.e. $\exists \delta > 0$ such that for all $t \in (x-\delta, x+\delta)$ we have $f(t)>0$. You can just check this with the definition of continuity. A function $f:E \rightarrow \mathbb{R}$ is continuous at $x$ if:

So if we let $\epsilon < f(x)$, we get some corresponding $\delta$ so that $f(p)>0$ for $p$ in that neighborhood. Thus, if we let $z = x + \delta$, we've proven our inductive step that there is a number $z > x$ such that $f(t) > 0$ for all $t \in [x,z]$. It might be a small interval, but good enough nonetheless.

Limit Case: Now say for for some $a < x \leq b$, we have that $f(t) > 0$ for all $t \in [a,x)$. We need to show that $f(x) > 0$ as well. Suppose it wasn't, i.e. $f(x) \leq 0$. By definition of our function, $f(x) \neq 0$, so the only remaining case is $f(x) < 0$. But by continuity—like in the inductive step—there would be a neighborhood such that $f(t) < 0$ for all $t \in (x - \delta, x + \delta)$. But that contradicts our assumption that $f(t) > 0$ for all $t \in [a,x)$ as $x - \delta < x$. So it must be the case that $f(x) > 0$.

By real induction, we have proven that if $f:[a,b] \rightarrow \mathbb{R} \backslash \{0\}$ is continuous and $f(a) > 0$, then $f(x) > 0$ for all $x \in [a,b]$. And in particular, $f(b) > 0$.

For more examples, I'd read the instructional paper I linked above that goes quite in-depth in proving many famous theorems with real induction. Most of the proofs are relatively straightforward once you figure out how to state your set in a good way to induct on it.

Real induction is one of those tools that just doesn't seem like it should work by the way we treat normal induction. It's not a method one would probably think of, and given how little it's used, it's not surprising it's not even that well-known.

Conclusion

Induction is one of the most fundamental tools in math. It's inextricably linked to our modern conception of the natural numbers, and in many ways, is what sets apart our contemporary understanding of math from other logical systems. So, I'm not surprised it's as powerful a proof tool as it is, but there is so much more to it than the typical weak and strong induction that is primarily taught. How we apply induction gives us so many new and creative ways to prove claims, but more importantly how thinking inductively naturally guides our intuition for alternate approaches. We saw it first with transfinite induction, then real induction, but who knows what other powerful variations on induction there may be.

But in the day-to-day, we rarely care for such rigor. If the weather man says it's going to rain today, I'm inclined to believe it since he's mostly been right before, and that's good enough for me. "Proof", to most people, usually just refers to any kind of persuasive evidence, as opposed to air-tight arguments. You want to be convinced with proof—nothing more, nothing less. If someone tells me they're a Michelin star chef, I won't need to see the certificate, but rather I'll probably be convinced if they even make one meal for me up to that standard.

Indirect proof to validate something, like with the case of the Michelin star chef, is what we usually rely on. We can't always provide hard evidence all the time, so convincing others with a demonstration of sorts is a nice alternative. But sometimes, it is the only option. If you had to prove citizenship to a country, sending over all your legal documents in one package, to one person, seems like a risky idea. If that package is lost or intercepted, your identity is gone. Or perhaps you are voting in an election, and want to prove that you did in fact vote. But ideally you could do that without having to disclose exactly who you voted for.

In these higher-stake situations with more sensitive information, any type of "direct" proof is a bit of a terrifying idea, for if something goes wrong between you and who you are interacting with, the consequences can be devastating. We want some way to prove something, without necessarily having to reveal any information about the proof claim.

Zero-Knowledge Proofs

Think about what we are asking for a second. We want to prove something to someone, without giving any particular facts; we want to endow someone with new information, necesssarily without giving any new information.

These zero-knowledge proofs (ZKPs) that validate a claim without imparting new knowledge sounds like an impossible feat. If someone has access to all the information already, and can't come to the conclusion themself, then how will they ever come to agree with the claim? Remarkably, there are strategies to create such a proof. An example does best here.

The Colorblind Man

Say your friend is extremely red-green colorblind. You give them two balls, one red and one green. Besides their color, they are completely identical (size, weight, mass distribution, etc.) to each other. To your friend, the balls do in fact look identical (due to the colors), but you want to prove to them they are truly not. Here's how you do it:

1. Hand them a ball in each hand.
2. Tell them to put it behind their back and choose to either swap the balls or leave them in the same hands.
3. Have them reveal to the balls to you and ask if he swapped them or not.

You, who's not colorblind, can answer him no problem. Getting it right once may be a fluke, but the more times you get it right, the more confident your friend should become that there is something differentiating the balls, even if he can't see it. If you're friend thinks you're only guessing randomly, even just getting it right 10 times in a row has probability $(\frac{1}{2})^{10} = 0.00098$, or a less than .1% chance of happening. You can keep repeating the test as long as your friend wants, but soon enough, he'll have to accept that you aren't just guessing, but rather seeing something he cannot.

In this, clearly we didn't change our friend's perception in anyway—he's still colorblind. Moreover, we didn't do any additional tests, like using an RBG monitor to digitally record values for your friend to read. Yet, we were able to convince them with a probabilistic guarantee that he is lacking information, even if we can't give him that information directly. In essence, nothing was changed of the scenario—no new knowledge was given—yet we were still able to prove to the colorblind friend that he in fact holding two differently colored balls, even if he can't verify that directly.

Inequitable Income

You recently joined a small company—no more than 5 people including yourself—and have recently begun to feel unvalued. You think you might be paid less than average, and you bring this up to the others. They try to explain to you that you are being paid fairly, despite the amount of work you put in. They seem confident enough, but they won't tell you their salary. How can you prove to them that you are paid less than average, without getting the data from your colleagues?

Let's say you, as person $A$, have a monthly salary of $ \$ a$, and the other four have salaries of $ \$ b, \$ c, \$ d, \$ e $. Now this is what we do:

1. You give a random, placeholder number $x$.
2. Then person $B$ computes the sum $x + b$, and tells person $C$ this value.
3. Person $C$ then adds their own salary to the sum to get $x + b + c$, and tells $D$ this number
4. People $D$ and $E$ do the same thing, until $E$ tells you the final number $x + b + c + d + e$.
5. Finally, since you know the value $x$, you can subtract it and recompute the value of $a + b + c + d + e$
6. You can then announce the average to the group $\large{\frac{a + b + c + d + e}{5}}$

Then, without any other individual having to know any other person's salary, can confidently agree that this is the group average, and enjoy the ensuing chaos of wealth mismanagement. This all works because of the value $x$: no one except you knows the value, so at each step, no other person will have enough information to deduce others' salaries from just their own. So without the need for any specific piece of new information, we are still able to deduce averages from a set of secret numbers.

These last two examples, particularly the former with the colorblind man, encapsulates the idea of zero-knowledge proofs. You, and possibly other parties, have secret information you want (or have) to keep secret, and somehow manipulate whatever knowledge you already have to deduce something meaningful. Even after I saw these two examples, it still was a surprising fact to me that not only such an concept exists, but also has super practical applications.

But we haven't actually specified exactly what makes a zero-knowledge proof, well, zero-knowledge. If we (the prover) are trying to prove some statement to a verifier with a ZKP, there are 3 properties we'd want to satisfy:

1. Completeness: if the statement is true, then an honest verifier will be convinced of it by an honest prover
   • This ensures we can prove the statement
2. Soundness: if the statement is false, then there is no proof that can convince the verifier (up to some small probability)
   • This ensures our proof method is trustworthy
3. Zero-Knowledge: if the statement is true, then no verifier learns anything except that the statement is true
   • This protects our secret

The first two qualities are that of a general interactive proof system, which is a type of proof that relies on a conversation-esque exchange between the prover and verifier (like in the colorblindness example). These interactive proofs follow a protocol that dictates the flow of conversation, and in the context of our qualities, someone is honest when they legitimately follow this protocol in good faith.

Let's do a more practical example.

The Discrete Logarithm Problem

You and your friends set-up your own little secret network to talk to each other privately. To make sure no one else can spy on you guys, you implement a password system: everyone picks a username that consists of 3 numbers, two integers $A$ and $B$ as well as a (usually large) prime $p$. Your password $x$ will be a number such that $B = A^{x} \bmod p$. Obviously, if you wanted to log-in, you could just give your password everytime, but that puts you at risk for eavesdroppers or anyone listening in. Ideally, we want to be able to prove to our network that we do actually know our password, without having to give the password. This way, if anyone wants to break into our account, they would have to brute force it, which isn't the best use of one's time.

One issue, for example, is that you could have multiple solutions to a modular arithmetic problem. The equation $13 = 3^x \bmod 17$. You can check that $x = 4$ is a solution. But, due to Fermat's little theorem, we also know that $1 = 3^{16} \bmod 17$. Then for any integer $k$, we know that

so even if you have a solution, it doesn't necessarily mean you found the particular solution that is our password.

Finding $x$ given $A,B,p$ is known as the discrete logarithm problem, and due to the difficulty of brute forcing, can actually be used as a method of identification (like a password). An additional thing to note, to maximize the search space for people brute forcing your $x$ value, we choose $A$ to be a generator, meaning that for every value $m = 1,2,\cdots,p-1$, there is a value $n$ such that $m = A^{n} \bmod p$. You tell everyone your $A,B,p$, and no one but you ever has to know your $x$. But you can still confidently prove to people that you do know it, and that you are in fact actually you.

After we (the prover) have distributed our values of $A,B,p$ (to any potential verifier) such that $B = A^x \bmod p$, this will be our protocol:

1. The prover picks a random number $r$ and calculates the value $C = A^r \bmod p$
2. The verifier now flips a coin. If the coin is heads they give the prover $c = 0$, and if tails $c=1$.
3. The prover sends the verifier the value of $cx + r$
4. The verifier now checks they get the expected value: $A^{(cx + r)} \bmod p = (A^{x})^c \cdot A^{r} \bmod p = B^c \cdot C$
5. Verifier repeats steps 1–3 until they are satisfied.

We see parts of our previous ZKPs appear here: 1) we let the verifier do a random action to test the prover's knowledge just like in the colorblind friend example, and 2) we do a pseudo-encryption of our password $x$ by adding a random number $r$ to obscure it in a sum.

Now we want to verify this acutally works, both as a password-method and a zero-knowledge proof.

So say someone was trying to hack into our account. If they knew what our verifier was going to give for $c$, then theoretically they could cheat their way into our account. There are two possibilities they could guess, being the value of $c$:

• $c=0$. Then the attacker just pick a random value $r$ and the verification process works as normal.
• $c=1$. Then the attacker pick a random value $r'$ and instead of giving the value $C$ that the verifier requests, they instead give the value $C{'} = (A^{r{'}} \bmod p) \cdot B^{-1}$. Then when the verifier wants the value $x+r$, the attacker instead gives $r'$, so that $A^{r'} \bmod p = (A^{r'} \bmod p) \cdot B^{-1} \cdot B = C' \cdot B$ which would be match the swapped values the verifier would expect.

But, if the attacker is wrong, because of how difficult it is to compute the discrete logarithm, they would not be able to reverse any of the new values they need with the value of $c$ they did not expect. So someone who doesn't know the password $x$ would have a 50% chance of passing the test, so the verifier just does this enough times until they are convinced. So at the very least, this does work as an effective identity/password check. So, also, this shows our proof is both sound and complete: if we don't, then it's near impossible for someone to cheat, and if we do know our password, a verifier should be convinced eventually.

This proof is also zero-knowledge since, well, we only worked with the public information. You can imagine building a simulator—like a computer—that can mimic exchanges between a prover and verifier (as if following the procedure of an attacker in our proofs of soundness and completeness) that operates entirely on its own. And I don't know about you, but a computer program is only working with what it's given, so if this interactive proof works within the confines of a computer, no new information is ever given.

While this is a good proof of concept, it does require a lot of work in practice. If you want to try and wrap your head around these types of proofs before moving on, here's another simple one about proving knowledge of a vector. It's an interactive proof with a focus on succinctness, and I found it to be quite helpful to think about some of the related concepts in cryptography and zero-knowledge proofs. The important concept to take away from this article and our above example with the discrete logarithm, randomness is an extremely useful way to instantly bound the error of our proofs; we don't need the proof to be 100% confident, we just need it to be confident enough such that the prover can't cheat if they tried.

Non-Interactive Zero-Knowledge Proofs

Before, in the example with the discrete logarithm problem and the colorblind man, we had this verifier act as an arbiter: they make a random decision that should prevent hacks (i.e. swapping the balls, sending $c=0$ or $c=1$), and after some arbitrary number of successes, they accept that the prover is genuine. Now imagine that you are the verifier, or are implementing this. Do you really want to sit through doing the same test over and over again, until you feel convinced? Do you really want to have to rely on some random mechanism every time that you carry out? Even if you make a computer to do this for you, imagine how annoying that must be to work with every time you want to log in with a password. Seems like a pain.

What we would ideally use is a non-interactive protocol: something that removes our need to have to always make a random choice and carry it out until some subjective level of confidence. But again, we don't want to compromise on zero-knowledge; we like our privacy. Fortunately, others' have come to the same issues and have come up with zero-knowledge succinct non-interactive arguments for knowledge (zk-SNARKs) to solve many of our qualms. To keep it clear, let's go over exactly what each part of this obscenely long name actually means:

• zero-knowledge: this is the same as before; we leak no new information and only prove that we know a claim is true without divulging anything specific about it.
• succinct: our proof takes a constant amount of space, i.e. the proof takes the same amount of time every time you go through it; we don't need to do the protocol until we are (subjectively) confident as we did with the previous proofs.
• non-interactive: our verifier doesn't give any intermediate input, i.e. no need to decide swapping balls or picking $c=0$ or $c=1$.

So here's a super simple zk-SNARK:

Claim: I want to prove to my friend that two entrances to a building are connected.

Proof: All I need to do is have my friend watch from outside that I can in fact enter the one door and exit the other.

• This is zero-knowledge since all my friend gets is knowledge that my claim was true and nothing more; they don't actually get to see how they are connected or anything more about the buliding.
• It's succinct since I only need to do it once, since there is no room for doubt once I've shown it once (how do you prove something like this by a fluke?).
• Finally, it's non-interactive since my friend doesn't need to actually do anything but observe.

That's the idea of zk-SNARK, at least. We'll build up to a pretty surprising zk-SNARK by the end of this, but to get there, we'll have to cover some more ground.

Now the remainder of this post will be pretty mathematically heavy in the worst way possible. The ideas are important and useful, and the overarching concept of a zero-knowledge proof is extremely interesting, but the mechanics behind them are fairly tedious—and that's just how it goes sometimes with cryptography. To ensure security, often times the best (and first thought about) ideas are to obscure the data with as many impossibly hard operations to undo. The notation might get dense, but hopefully we build up to it in a comprehensible manner.

Honesty Checks

Our discrete logarithm example, while not only gave us a nice demonstration of zero-knowledge proofs, they also now have given us an additional encryption method. The entire problem hinged on the fact that calculating discrete logarithms are hard to begin with. So we can use exponents, actually, to hide and encrypt lots of other pieces of data. And moreoever, can act as a safeguard to make sure no one is cheating us.

For example, say we're a verifier, and want to make sure that our prover follows the protocol of multiplying a number we give them. They can multiply whatever they want, but they have to multiply the number we give them. We can force this with exponents because the discrete logarithm is hard.

1. The verifier picks a generator $g$ (like from the discrete logarithm), our secret number $s$, and our encrypted shift $\alpha$.
2. the verifier gives the prover the values $g^s$ and $g^{\alpha \cdot s}$.
3. The prover with some number $c$, then gives back the values $(g^s)^c = g^{cs}$ and $(g^{\alpha \cdot s})^c = g^{c\alpha s}$.
4. The verifier checks that $(g^{cs})^\alpha = g^{c\alpha s}$.

Because of how hard it is to cheat this system by finding another exponent to get equal values that the verifier would expect, the prover is forced to use the encrypted value of $g^s$, and the shift of $g^{\alpha \cdot s}$ gives us insurance to check the prover's honesty at the end. And in the same way the prover doesn't learn the values of $s$ or $\alpha$, we don't learn the value of $c$ they want to multiply by. We only checked that they were honest.

Polynomial Knowledge

Now the reason why that's important, is that exponents give us all the methods we need to check more complicated values, like that of polynomials. The question we'll aim to answer is:

In other words, can we prove to a verifier that for some polynomial $h(x)$, we know that $p(x) = (x-r_1)(x-r_2)\cdots (x-r_n)\cdot h(x)$? For brevity and convenience, we'll call the target polynomial $t(x) = (x-r_1)(x-r_2)\cdots (x-r_n)$.

The reason why polynomials are interesting is because they are incredibly hard to cheat. Say the prover has a polynomial $f(x)$, and claims to know the exact polynomial that the verifier has $g(x)$, that is the prover claims $f(x) \equiv g(x)$. If both polynomials are of degree $d$, then either $f(x)=g(x)$ has at most $d$ solutions by the Fundamental Theorem of Algebra, or has infinite solutions if and only if $f(x) \equiv g(x)$. So if the verifier picks a random value $s$ for the prover to evaluate, it is extremely unlikely that $f(s) = g(s)$ if they are not genuinely the same. If the prover does not know $g(x)$ and just guessed a random polynomial $f(x)$, if the verifier pick a random integer from the range $s \in [1,10000]$, there is at most a $\frac{d}{10000}$ chance that $f(s) = g(s)$.

First Attempt

Here's a naive way to prove to a verifier they know such a polynomial:

Protocol Attempt 1:

1. The verifier picks a random number $s$ and gives this to the prover along with $t(s)$.
2. The prover calculates $h(s) = \frac{p(s)}{t(s)}$ and passes that along to the verifier with $p(s)$.
3. The verifier checks that $h(s) \cdot t(s) = p(s)$.

This, though, has the issues of:

• The prover can just pick an arbitrary number $h$ and calculate $p = h \cdot t(s)$ and that will satisfy the verifier.
• Since we give the prover access to the number $s$, the prover can just make a new polynomial that happens to have the value of $p(s) = t(s) \cdot h$.
• There's no verification of the degree of the polynomial.

The first two issues are purely because we don't encrypt anything; we just give the values of $s$ and $t(s)$. We need a way to obscure them so that computations with them are still feasible, but the prover can't use them to cheat the protocol.

Homomorphic Encryption

The way we can do this is in line with our honesty checks and the discrete logarithm, and exploiting the properties of exponents. Say our polynomial was a quadratic $f(x) = x^2 - 3x + 2$. We can encrypt $f(x)$ the same way we have been doing before by using a generator $g$ taken modulo $n$ (ideally a prime):

This type of encryption is known as homomorphic encryption, as our encryption method has a nice structure to it that allows us to do our arithmetic operations on them without having to decrypt it. In particular, if we let $E(s) = g^s \bmod n$ be our encryption as we have been, the structure we get is that $E(n + m) = E(n) \cdot E(m)$ by exponent rules.

One issue, though, is that we cannot multiply two encrypted values together with this homomorphic encryption scheme. If you have two numbers $a,b$ that have been encrypted as $E(a) = g^a$ and $E(b) = g^b$, we can easily find $E(a + b) = E(a)E(b)$ as we stated above. But only given $E(a), E(b)$, you cannot find an expression for $E(ab)$. Similarly, we also can't find an expression for $E(a^b)$. We'll address this later.

As we've discussed, reverse engineering an exponent modulo a number is especially hard because of how the modulo operation cycles, leaving room for many options to satisfy the equation and thus making finding a specific solution hard. So we can update our protocol:

Protocol Attempt 2:

1. The verifier picks a random number $s$, and sends encrypted powers of $s$ to the prover $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$.
2. The prover calculates $h(x) = \frac{p(x)}{t(x)}$. With the encrypted values, they calculate and send to the verifier $g^{p(s)}$ and $g^{h(s)}$.
   • i.e. $g^{p(s)} = g^{c_d s^d + \cdots c_1 s^1 + c_0 s^0} = (g^{s^d})^{c_d} \cdots (g^{s^1})^{c_1} \cdot (g^{s^0})^{c_0}$ for the coefficients $c_d, \cdots, c_0$
3. The verifier checks that $g^{p(s)} = (g^{h(s)})^{t(s)} = g^{h(s) \cdot t(s)}$.

Although I've left it out for convenience, remember that all these encrypted values $g^x$ are taken to mean $g^x \bmod n$. It's the discrete logarithm that is difficult, not the normal logarithm.

So by encrypting $s$ and just not giving the value of $t(s)$, we've fixed our issues, right?

The first two are in fact fixed, but we still need a way to enforce the degree requirement. In a way, we already do have a type of restriction since the verifier only gives the prover powers of $s$ encrypted with $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$. But this restriction is only in place if the prover uses these and only these values. There's nothing stopping the prover from just not using those encrypted values of $s$, and using their own constructed numbers to cheat. Here's a more concrete way of seeing this:

What the verifier wants at the end of the day is for the equation $g^{p(s)} = (g^{h(s)})^{t(s)}$ to hold, and is trusting that $g^{p(s)}$ and $g^{h(s)}$ are provided truthfully by the prover. But if we have a prover that does not know a polynomial, all they need to do to fool the verifier is to give values $z_p = g^{p(s)}$ and $z_h = g^{h(s)}$. In other words, the prover just needs to find a solution to $z_p = (z_h)^{t(s)}$. Which, unfortunately for our protocol, is quite easy.

• Let $z_h = g^r$ for some chosen random $r$
• Then we just need $z_p = (g^r)^{t(s)} = (g^{t(s)})^r$
• Since the target polynomial $t(x)$ is public, the prover can solve $g^{t(s)}$ with the given values $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$
• So finding $z_p$ is done

So finding a solution is no harder than essentially picking a random number. We want some way to ensure that the prover uses and only uses the values the verifier gives them from $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$ in finding their $z_p,z_h$. If we can do that, the only way they'll be able to calculate $z_p,z_h$ is probably with a polynomial they have.

Readily enough, we have done this with our honesty checks above: we provide some arbitrary shift $\alpha$ only the verifier knows, to make sure the prover didn't leave anything out or cheat.

Protocol Attempt 3:

1. The verifier picks random number $s$ and $\alpha$. They send the encrypted powers of $s$ and the shifts to the prover: $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$ and $\{g^{\alpha s^0}, g^{\alpha s^1}, g^{\alpha s^2}, \cdots, g^{\alpha s^d}\}$.
2. The prover calculates $h(x) = \frac{p(x)}{t(x)}$. With the encrypted values, they calculate and send to the verifier $g^{p(s)}$, $g^{\alpha p(s)}$, and $g^{h(s)}$.
3. The verifier checks that $g^{p(s)} = (g^{h(s)})^{t(s)} = g^{h(s) \cdot t(s)}$, and that $(g^{p(s)})^\alpha = g^{\alpha p(s)}$.

The first check the verifier does is to see the values match like before, and the second check is to make sure there was no cheating and only the encrypted values of $s$ were used; the prover had to give a polynomial of degree $d$ and had to evaluate it at the $s$ the verifier gives them, as that is the only way to preserve the $\alpha$ shift.

For example, if the prover claimed they knew a quadratic with $d=2$, they claim they have a polynomial looking like $p(x)=c_2x^2+c_1x^1 + c_0$. But if they were really trying to sneak in that they had a cubic or quartic or any polynomial of degree higher than 2, they would not be able to calculate the terms needing $x^3$ or $x^4$ since the encrypted and shifted values could not be calculated and thus preserved. But that is only true if the prover used values from $\{g^{s^0},g^{s^1},g^{s^2}\}$. Now by using the $\alpha$ shift, the prover has no choice but to use these values since any deviation from them will appear comparing to the $\alpha$ values.

Further, we can do more than just ensure the degree of the polynomial we're checking, but also which specific powers are used in the polynomial. If, say, the polynomial was claimed to be a cubic, but only used powers $x^3$ and $x^1$, the verifier can choose to only send in the encrypted and shifted values of $s^3$ and $s^1$. If the prover needed the other powers to evaluate their polynomial, they would be stuck since they can only evaluate the terms with power 3 and 1.

Making It Zero-Knowledge

So far, our protocol has gone under a few iterations—and to be fair it's actually pretty robust. But we kind of forgot about making it zero-knowledge. I mean, yes we've encrypted the data to prevent any cheating from the prover, but the verifier can theoretically use the values the prover gives to brute force their way to finding the polynomial since they are the ones that generate the secret values $s$ and $\alpha$ from the beginning. For example, ideally our protocol should be secure for even a 1-degree polynomial, and even brute forcing that is just a matter of iterating through a series of numbers.

This is easily enough done in the same way that we've been doing before: we have the prover introduce a random parameter $\delta$ that obscures the data.

Zero-Knowledge Protocol:

1. The verifier picks random number $s$ and $\alpha$. They send the encrypted powers of $s$ and the shifts to the prover: $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$ and $\{g^{\alpha s^0}, g^{\alpha s^1}, g^{\alpha s^2}, \cdots, g^{\alpha s^d}\}$.
2. The prover calculates $h(x) = \frac{p(x)}{t(x)}$. With the encrypted values, they calculate $g^{p(s)}$, $g^{\alpha p(s)}$, and $g^{h(s)}$. With a random number $\delta$, they send the verifier $(g^{p(s)})^\delta$, $(g^{\alpha p(s)})^\delta$, and $(g^{h(s)})^\delta$
3. The verifier checks that $(g^{p(s)})^\delta = ((g^{h(s)})^\delta)^{t(s)} = g^{\delta \cdot h(s) \cdot t(s)}$, and that $((g^{p(s)})^\delta)^\alpha = (g^{\alpha p(s)})^\delta$.

Making it Non-Interactive

Our protocol is very similar to our discrete logarithm problem. I mean, we based it off of what we did there, only generalizing what we did with polynomials instead of specific numbers. So this doesn't really show us anything we haven't already seen before. We want to try and make it non-interactive so our verifier doesn't have to constantly monitor our verification. And more importantly, so our protocol is trustworthy: due to the nature of the interactive parts, verifiers could collude with provers, making each protocol use a one-time check. Even better would be if we could make it also succinct, and have each call of the protocol takes a consistent amount of time.

One way we can remove the need for interactivity is by, well, replacing the interactive parts with some constant, reliable parameters to always use as oppose to the ones suggested by the verifier. In this case, the verifier has to pick a value $t(s_0)$ (since the target polynomial $t(x)$ is known, really we just need to fix an $s_0$) as well as a fixed shift $\alpha_0$. But we need these to be trustworthy, and unable to be leaked.

We could just try encrypting these values like before by exponenitating modulo $n$ like before with $g^{t(s_0)}$ and $g^{\alpha_0}$. But the problem is, we've also encrypted the other values like $p(s_0)$ and $h(s_0)$, and as we said, we can't multiply two encrypted values together, which is exactly what the checks the verifier needs at the end of the protocol; with our encryption, if we have $E(\alpha_0)$ and $E(p(s_0))$, we have no way of getting $E(\alpha_0 p(s_0))$.

There's one extra piece of machinery we'll need: elliptic curves. Elliptic curves are a class of implicit functions of the form $y^2 = x^3 + ax + b$ and their relation to the whole of cryptography is a bit too wide to encapsulate in this post, so perhaps we'll come back to them another day. The important thing about them though is that we can establish cryptographic pairings with these curves that can get around our multiplication issue.

A cryptographic pairing is a function $e(\cdot, \cdot)$ that take two encrypted numbers, and outputs the product of those two numbers in a different representation (i.e. outputs the product of the numbers encrypted in a new way). Because the output space is a different "encryption scheme" from before, it makes it irreversible, and a "one-time operation"; you can't use the output of $e(\cdot, \cdot)$ in another cryptographic pairing in a meaningful way. The key properties of this pairing are:

I know this is a very rough sketch as to how we are overcoming the problem of multiplying encrypted values, but for the sake of brevity, we'll come back to it sometime later (if you can't wait, check this out). The important idea to keep in mind is that we have a way of multiplying encrypted values of $t(s)$ and $\alpha$ in a useable way.

So now, to make our proof non-interactive, we fix our values $\alpha, t(s)$, and then encrypt them $g^\alpha, g^{t(s)}$. These values will be the ones used by every prover and verifier moving forward, and to carry out our operations, we will use our cryptographic pairing function, which we'll see below.

Also, as an aside, we no longer need multiplicative group generators like we have used before. $g^n$ can now mean adding the generator of the elliptic curve $g$ to itself $n$ times. It's essentially the same (and acts the same for our purposes), but removes an extra component from our process.

The Final Protocol

We are now ready to put together our final protocol for knowledge of a polynomial of degree $d$ with the same roots as $t(x)$. The keys below are the necessary elements each party needs on their side of the proof beforehand.

Set-Up:

• Fix random values $s$ and $\alpha$
• Establish a cryptographic pairing $e(\cdot,\cdot)$ and a generator $g$
• Find encryptions $g^\alpha$, $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$, $\{g^{\alpha s^0}, g^{\alpha s^1}, g^{\alpha s^2}, \cdots, g^{\alpha s^d}\}$

Now we distribute to the prover and verifier their respective information they are allowed to work with:

• Proof key: $\left(\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}, \{g^{\alpha s^0}, g^{\alpha s^1}, g^{\alpha s^2}, \cdots, g^{\alpha s^d}\}\right)$
• Verification key: $(g^\alpha, g^{t(s)})$

Proof:

• Wants to prove knowledge of $p(x) = t(x) \cdot h(x) = c_d x^d + \cdots + c_0 x^0$
• Let $h(x) = \frac{p(x)}{t(x)}$
• Calculate $g^{p(s)}, g^{h(s)}$ using $\{g^{s^0}, g^{s^1}, g^{s^2}, \cdots, g^{s^d}\}$
• Calculate $g^{\alpha p(s)}$ using shifted values $\{g^{\alpha s^0}, g^{\alpha s^1}, g^{\alpha s^2}, \cdots, g^{\alpha s^d}\}$
• Pick a random shift $\delta$
• Send to the verifier our proof $\pi = \left(g^{\delta p(s)}, g^{\delta h(s)}, g^{\delta \alpha p(s)} \right)$

Verification:

• With $\pi = \left(g^{\delta p(s)}, g^{\delta h(s)}, g^{\delta \alpha p(s)} \right)$, we do our two checks for satisfiability and degree of the polynomial
• Does the polynomial work? Check that $e(g^{\delta p(s)}, g) = e(g^{t(s)}, g^{\delta h(s)})$
• Did the prover cheat? Check that $e(g^{\delta \alpha p(s)}, g) = e(g^{\alpha}, g^{\delta p(s)})$

And that's our protocol from start to finish. Without ever needing to reveal what $p(x)$, we can confirm with high probability that our prover's polynomial has the desired roots matching $t(x)$. We can add other requirements to the polynomial, like only including certain powers as we discussed, or others such as it must be a square polynomial.

General zk-SNARKs

So far, we've spent roughly 6000 words talking about polynomials and homomorphic encryption, and that's with skipping explanations of elliptic curve cryptography thrown in there too. But in practice, when was the last time you saw someone work with a polynomial directly? If I had to think of when someone would want to prove knowledge of something in practice, it would probably be something less direct, like knowing the output of a program, or a secret input (like a password). Those don't seem related to polynomials at all. For example, if I had a computer program and a given output, I'd like to show I have the corresponding input without revealing that input.

But of course, in the weirdest ways, polynomials are the current backbone of zk-SNARKs. And unfortunately, like with elliptic curves from before, require an already huge amount of literature to even crack the surface of how they work. Ultimately, the idea is that given any program, we can make the problem of proving knowledge-of-input into a question of knowledge-of-polynomial. Here are the rough steps (as outlined in this more in-depth post):

1. Computation: The actual code for the program itself.
2. Flattening: We turn the code into a combination involving expression only involving $=+,-,\times,\div$. These arithmetic operations essentially correspond to different types of gates in a circuit. This is probably the hardest, least obvious step, and even now it is not clear to me what are the restrictions for programs that can be converted to these arithmetic circuits. I'd recommend reading this.
3. R1CS: With the arithmetic circuit, we now convert it to a rank-1 constraint system. An R1CS is a set of groups of 3 vectors $(a,b,c)$, that has a solution $x$ such that $(a \cdot x) \times (b\cdot x) = c\cdot x$ where $\cdot$ is the standard dot product. Each group of these 3 vectors $(a,b,c)$ represents some kind of constraint on our solution. In our case, we will have a triple of vectors $(a,b,c)$ for each gate/operation we have in our arithmetic circuit. So if our arithmetic circuit has 4 steps in it, we'll have 4 triplets of vectors constraining our solution $x$ that it must all satisfy. The length of the vectors will be equivalent to the number of variables needed in the circuit. Here's an example conversion of an arithmetic gate to R1CS constraint vectors.
4. QAP: We do yet another conversion from R1CS to a quadratic arithmetic program. The idea is to encode our vector constraints in polynomials. If, for example, we had 5 constraints with vectors of length 7, we would have 5 pairs of $(a,b,c)$ constraints where $a,b,c$ are all length 7 vectors. Then, we could encode these in 3 groups of 7 polynomials (in this case, they would be of degree $5-1=4$). Each polynomial represents a coordinate in the vector, and each group represents whether that vector is the $a$, $b$, or $c$ vector in the constraint. Since we have 5 constraints, we then retrieve our each constraint vector by plugging in $x=1,2,3,4,5$ to each polynomial and read off the values by group and coordinate. We can create these polynomials via Lagrange interpolation, or your favorite way to fit polynomials to specific values. This is why the degree of the polynomials are 1 less than the number of constraints: you need exactly $d+1$ points to determine a polynomial of degree $d$.

The point of putting our program into a QAP is that we can now do our R1CS verification a lot more compactly. Instead of checking all the dot products individually between our solution vector $x$ and the constraint vectors, we can dot product the solution once with our polynomials. Dot products are just combinations of addition and multiplication, so the result of $(x \cdot A(t)) \times (x \cdot B(t)) - x\cdot C(t) = F(t)$ will just be another polynomial in $t$. Our solution vector is genuine if $F(t) = 0$ for $t=1,2,3,4,5$ in our above example, since plugging in those values corresponds to checking a different R1CS constraint (that represents one of our arithmetic logic gates).

But now, we're mostly at the point in which our zk-SNARK protocol for polynomials is starting to look like a more useable tool for general programs. There are a few additional steps outlined in some of the links above, but the core idea is here, of encoding the computation of the program in a polynomial that we later can check the knowledge of via a zk-SNARK.

Conclusion

There are a lot of uses that zero-knowledge proofs can find themselves in. Basically, whenever you want any level of privacy. From passwords, to graphs, to polynomials, or to nuclear disarmament. And its most popular use, blockchains and cryptocurrencies (good way of checking valid transactions without needing to reveal people's currency balances). The underlying theory of zero-knowledge proofs, though, is simultaneously easy to understand, and difficult to implement. The theory is rich, but dense, so pehaps one of these days we'll fill in the gaps of elliptic curves (which definitely deserves its own post) and fully fleshing out how we can convert computer programs to polynomials.

For more reading and types of proofs, here's a nice simple example for proving knowledge of coloring a graph that can be found here or here (and a nice little demo to go with it). Interestingly, as an aside, we can since reduce any NP problem to the 3-coloring problem, this actually gives us a way of generating zero-knowledge proofs for any NP hard problem.

For even more details on zk-SNARKs and zero-knowledge ideas on the whole, see this article that informed much of this post.

As we now know, this clearly isn't the case. We've talked about irrational numbers a little bit before, specifically on how $\sqrt{2}$ is irrational. It might be worthwhile going over the proof again:

Claim: $\sqrt{2}$ is irrational.

Proof: Suppose that $\sqrt{2}$ is not irrational. Therefore it is rational, and can be written as $\sqrt{2} = \frac{p}{q}$ where $p$ and $q$ are distinct coprime numbers (i.e. they don't share any factors in common; this is a way of sayiing that there is a unique way of writing $\sqrt{2}$ as a fraction in lowest terms). Squaring both sides and moving terms around, this equation becomes $2q^2 = p^2$. Therefore, $p^2$ is even, since it will always have a factor of 2 according to the lefthand-side of the equation. If $p^2$ is even, then $p$ is also even (if you're unsure of this fact, try some test cases and proving it!). Since $p$ is even, then let's rewrite it as $p=2n$. Plugging this into our equation yields: $2q^2 = p^2 = (2n)^2 = 4n^2$. Dividing both sides by 2 gives us $q^2 = 2n^2$. Similar to before, the equation implies that $q^2$ and hence $q$ is even due to that factor of 2 on the righthand-side. But this is a contradiction! $p$ and $q$ cannot simultaneously both be coprime and even (share a factor of 2). Hence, our initial assumption that the $\sqrt{2}$ is rational is wrong, and allows us to conclude that $\sqrt{2}$ is irrational. $\blacksquare$

This proof is pretty well-known, but to me, it's not necessarily obvious. Sure, in retrospect after knowing the proof, it might seem like a good direction to go in and get a contradiction within that rational assumption. But even then, it's a pretty clever proof.

I mention this, since I was recently thinking about an arguably more famous irrational number: $\pi$. A number studied forever that manages to appear in random sums, integrals, and expressions it has no right being in, it's no surprise it is one of the few symbols of math that transcends pop culture. And despite that, before writing this, I couldn't explain or prove why it is irrational. $\pi$ is geometric at heart, and translating to number-theoretic contexts just didn't seem to make sense. But along the way of finding out its proof (that stumped many before me), there is much to unpack about not just about $\pi$, but irrational numbers on the whole.

If you care for only the proof that $e$ and $\pi$ are irrational, skip here. Otherwise, follow through the table of contents for whatever you're looking for.

Decimal Expansions

The existence of irrational numbers actually implies a lot. For one, there are infinitely many more irrational numbers than rationals. With the classic diagonal argument, we can see that there are not any "more" rationals than integers or naturals. The irrationals are what make the real numbers uncountable. A way we can see that is through decimal expansions: all irrational numbers have non-repeating decimal expansions. So now if you imagine constructing a random number $0.142346\dots$ by picking a random number 0–9 at each digit, the chance of landing a repeating decimal is extremely unlikely.

Claim: A number is rational if and only if it has a terminating or repeating decimal expansion.

Proof: $(\Rightarrow)$ The easiest way to see this is through long division. If we have a rational number $\frac{p}{q}$ with $q \neq 0$, then at every step in the long division, we have $q$ possible remainders (being $0,1,2,\cdots,q-1$). If the remainder is 0 at any point, then we are done with the divison algorithm and the decimal expansion terminates (this it the case of 0 being the repeating portion of the decimal). If the remainder is never 0, after $q+1$ steps in the division algorithm, we will get a remainder we have already seen before, and so the division algorithm will keep producing the same digits in order we have seen before from that remainder. Hence the decimal expansion repeats.

$(\Leftarrow)$ Say we have a number $x = a.d_{1}d_{2}\cdots d_{n} \overline{d_{n+1} \cdots d_{n+m}}$ that has integer part $a$, $n$ pre-repeating decimals (each $d_i$ is a digit of the number), and $m$ repeating decimals (indicated by the overline). Then we can left-shift the non-repeating number by multiplying by a power of 10:

Since the decimals repeat after an additional $m$ digits, we can shift it further to get another similar looking number:

Now we can subtract equation $(1)$ from $(2)$ to eliminate the repeating decimal portion:

Now, see that since we have removed the decimal/fractional part of the numbers in that subtraction, the righthand side of the equation is an integer, call it $N$. Then we can solve for $x$ and see that

is a quotient of two integers, and so rational.

So if we want to use real numbers with decimals as we normally do, we need our non-repeating decimals to correspond to something. These are our of course our irrationals. But let's be clear exactly what non-repeating means. It's frequently unjustifiably said that since $\pi$ is irrational, it contains every single string of numbers ever conceived, and hence contains exactly the time and date you were born, you will die, and a copy of every book ever written. This is unknown. Just consider the non-repeating decimal that only contains 0s and 1s

Clearly this does not repeat, but certainly also does not contain all possible strings of numbers; it does not even contain all strings of 0s and 1s.

In a sense, too, the irrationals are also necessary for us. Without the irrational numbers, obviously we do not have all the real numbers, but the irrationals are necessary to fill in the gaps and holes left by the rationals to make the real numbers $\mathbb{R}$ as useful as they are. For example, a key property of the reals is that every subset of the reals that has an upper bound, in fact has a least upper bound. That is, for a subset $A \subset \mathbb{R}$ such that $\exists x \in \mathbb{R} \ \forall a \in A \ a \leq x$ (i.e. $x$ is an upper bound), there exists $\ell \in \mathbb{R}$ that is less than or equal to all possible upper bounds $x$. The rationals don't have this property: $\{ x \in \mathbb{Q} \ | \ x^2 < 2 \}$ has upper bounds in $\mathbb{Q}$, but not a least upper bound; we need irrationals and $\sqrt{2} \in \mathbb{R}$ to do this.

Completeness and the Variety of Irrationals

The irrationals, in this way, are integral to the real numbers. Not just in the sense that we have to include them, but in that they complete the real numbers. That least upper bound property we discussed above is what is referred to as the completeness of the real numbers (which can be an axiom or property derivative of other axioms, but either way is what characterizes the reals).

And yet, despite their utility, irrational numbers are still pretty mysterious. We know all rational numbers take on a certain form of $\frac{p}{q}$. Even complex numbers all reduce to some $a + bi$. But irrational numbers can take on any number of forms.

• Taking the square root of any non-square number is irrational
• The golden ratio $\varphi$ is irrational
• $e = \lim\limits_{n\to\infty} \left(1+\frac{1}{n} \right)^n = \sum_{n = 0}^{\infty} \frac{1}{n!} \approx 2.71828\cdots \ $ is irrational
• $\pi$ as the ratio of a circle's circumference to its diameter is irrational

Irrational numbers can pop up almost anywhere. Though, we might be getting a bit ahead of ourselves. We have a sort of "standard form" of irrational numbers given by our decimal expansions and our above claim: a number is irrational if and only if it has a non-repeating decimal part. But it's not like we can a priori check a decimal expansion is infinite to prove a number is irrational; it's usually shown that infinite decimal expansions are a consequence of irrationality.

Continued, Infinite, and Reasonable Fractions

So, while we showed that we in theory need irrational numbers, in practice representing them and using them seem far more tedious and annoying to work with.

And further, if you're just an engineer, astronomer, or even a mathematician that needs some concrete number to work with, rational numbers get us most of the way there for what we need. Why not just take some arbitrary decimal cut off? Most people only know $\pi$ up to $3.14$ anyway.

$3.14$ is pretty good, but an arguably better approximation is the famous $\frac{22}{7} \approx 3.14286$. Sure, it might be further off from $\pi$ than $3.14$, but it is a rather simple fraction that can make calculations easier and more convenient. You could go even further and get $\frac{355}{113} \approx 3.1415929$ which is good up to 6 decimals. Can we get any better?

So before we get any further in directly discussing irrationals, let's take a minute to talk more about rational numbers and their relationship with irrationals. From this, we'll see another nice property of irrational numbers that will help us along our way in a similar, but stronger way compared to decimal expansions.

Repeating Decimals $\Leftrightarrow$ Rationals

We alread proved that repeating and terminating decimals correspond to some rational number. So as a first step, it might be worth thinking about how we might go to and from fractions and these nice decimals before considering how we might do something with the weird and infinite decimals of irrationals.

Finite decimals are easy: just take the least power of 10 in it's decimal expansion and put it in the denominator of the digits. $.5 = \frac{5}{10^1} = \frac{1}{2}$, and $1.562782 = \frac{1562782}{10^6}$. Not particularly helpful.

If we have a repeating decimal, it's slightly trickier, but our proof above showing they are rational essentially tells us how to do the conversion.

• Consider the number $x = 2.642857 \overline{142857}$
• Using the above notation, $a=2$, $n=6$, $m=6$
• So $N = 10^{6+6}\cdot 2 + 642857142857 - (10^6\cdot 2 + 642857)$
• All together, we get that $x = \frac{N}{(10^6 - 1)10^6} = \frac{37}{14}$

You can also use geometric series to get the same answer, using the fact that the repetitions in the decimal are all separated by a ratio of $10^{-m}$ (for example, $0.\overline{142857} = 10^{-6} \cdot \sum_{n=0}^{\infty}142857\cdot 10^{-6n}$, and since $10^{-6}<1$, we get that it's equal to $10^{-6} \cdot 142857\cdot \frac{1}{1 - 10^{-6n}} = \frac{142857}{999999} = \frac{1}{7}$).

Though, we should expect that we can get these exact rationals since we already proved that finite and repeating decimals all are rational.

Rational Approximations

Now irrational numbers, on the other hand, by definition, will not have a rational form. So now we will actually need rational approximations. But finding those approximations, is not particularly easily. In particular, for irrational $x$, we want to find a rational $\frac{p}{q}$ such that $\left| x - \frac{p}{q} \right| < \epsilon$. At the very least, we know such rationals exist for any $\epsilon > 0$, since the rationals are dense in $\mathbb{R}$; there is always a(n infinite amount of) rational(s) in the interval $\left( x - \epsilon, x + \epsilon \right)$.

But we don't want to waste our time with "bad" approximations. Notice that for any denominator $q$, we can find a $p$ such that $\left| x - \frac{p}{q} \right| \leq \frac{1}{2q}$. This is just a result that if we divide the real numbers into intervals of size $\frac{1}{q}$, our number $x$ will fall in one of them and be closer to one side of the interval than the other.

It turns out though, we can find approximations that are far more "efficient" and get us much closer to our irrational number than just $\frac{1}{2q}$ for their denominator $q$.

Dirichlet's Approximation Theorem: $\forall x \in \mathbb{R}$ and $\forall N \in \mathbb{N}$, there are integers $p$ and $q$ such that $0 < q \leq N$ and $\left|qx - p \right| < \frac{1}{N}$.

As a consequence, we then have rational approximations $\frac{p}{q}$ that get us within $\frac{1}{q^2}$ of our irrational $x$ since $q\leq N$ implies

Proof: Pick $x\in \mathbb{R}$ and $N \in \mathbb{N}$. We will prove this using the Pigeonhole Principle. Note that any integer $n$ can be written as a fraction with any denominator $k$ as $\frac{nk}{k}$. So all we care about is approximating the fractional part of $x$, call it $\{x\} \in [0,1)$. Now divide the interval $[0,1)$ into $N$ equal subintervals of length $\frac{1}{N}$ i.e.

Now consider the set of $N+1$ numbers $kx$ for $0\leq k \leq N$, and in particular their fractional parts $\{kx\}$. By the Pigeonhole Principle, at least two of these fractional parts $\{k_1 x\}, \{k_2 x\}$ lie in the same $\frac{1}{N}$-subinterval of $[0,1)$ (without loss of generality, we can pick $k_1 > k_2$).

Note, we can write $\{kx\} = kx - \lfloor kx \rfloor = kx - j$ for some interger $j$. So if two fractional parts lie in the same subinterval, then what we have is

So if we let $q = k_1 - k_2$, and $p = j_1 - j_2$, then we have $0 < q \leq N$ and

which is exactly what we wanted to show.

Also, it's worth noting that there are infinitely many of these approximations, despite this just being an existence proof of at least one such pair $p,q$ that bound $\left|x - \frac{p}{q} \right| < \frac{1}{q^2}$. This can be quickly noticed since for a given $N_0$, Dirichlet's theorem says that there is at least one pair of $p_0,q_0$ such that $\left| q_0x - p_0 \right| < \frac{1}{N_0}$. But then, if we increase $N_0$ to $N_1$ large enough such that $\left| q_0x - p_0 \right| \geq \frac{1}{N_1}$, then Dirichlet's theorem ensures that there is some new pair of numbers such that $\left| q_1x - p_1 \right| < \frac{1}{N_1}$. And we can continue this forever.

As an interesting aside, this can also be used to show that rationals are in some ways worse than irrationals at being approximated by other rationals. Of course, if we "approximate" a rational with itself, it trivially holds that it is the "best" approximation. But concretely, if we have a rational $x = \frac{a}{b}$ and another rational $\frac{p}{q} \neq x$, then we get that

In particular, since $p,q,a,b$ are all integers, so would that difference above be, and so we can conclude that $|pb - qa| \geq 1$. So all together,

which is bounded away from 0, regardless of $q$. Which shouldn't be that surprising thinking through that $p,q$ are integers so that difference in distance should only have a non-integer part that is exactly proportional to the denominator $\frac{1}{b}$.

Best Rational Approximations

Dirichlet's Approximation Theorem tells us that we have stronger candidates than others for rationally approximating irrationals. Which is great and all, but can we give a more specific criterion to single out these "better" approximations? Or at the very least, approximations that we would prefer using?

We won't use the approximation that Dirichlet's Theorem would suggest above, as in using something "efficient" that gets us closer to the number we are approximating more than we'd expect it to. And besides, our proof above doesn't actually tell us how to find these good approximations since we have no control for our denominator $q$, but rather only an upper bound for $q$ with our choice of $N$. In particular, it also doesn't tell us how good our approximations are to other Dirichlet approximations; it could be the case that one approximation from Dirichlet's theorem is better than another despite having a smaller denominator (and so is in some way more preferable).

Probably the easiest metric of a best rational approximation of a number in this case would be that it is the best approximation for the smallest denominator found yet. Ideally, we want to work with "simple" fractions (i.e. with small denominators). So we say a rational approximation $\frac{n}{d}$ for a real number $x$ is best if $\frac{n}{d}$ is closer to $x$ than any other fraction with denominator less than $d$. If you want something more concrete, $\frac{n}{d}$ is a best approximation for $x$ if

To really get the point across, $\frac{p}{q}$ is a best rational approximation if it is not possible to get a better approximation using a smaller denominator. This is part of the reason why $\frac{22}{7}$ is so convenient to approximate $\pi$ as it is still a rather simple and easy-to-work with fraction, but is in a sense better than any possibly simpler fraction. Again, this is great to have, but we still don't have a way of finding these approximations any better than just brute force checking every other simpler fraction.

To find these approximations, we'll have to take a slight detour that perhaps is the closest link between representing any real number with rationals that you might have seen before.

Continued Fractions

Consider one of our favorite irrational numbers that we might want to approximate: $\sqrt{2}$. Before trying to find approximations or anything, notice that $\sqrt{2} - 1 = \frac{1}{1 + \sqrt{2}}$.

Now hold on. We just found that $\sqrt{2} = 1 + \frac{1}{1 + \sqrt{2}}$. If $\sqrt{2}$ equals that fraction on the righthand side, then we can just replace an instance of $\sqrt{2}$ with that same fraction:

And again:

And can keep doing this forever:

This is a continued fraction for $\sqrt{2}$. This particular one is the canonical or simple continued fraction for $\sqrt{2}$ since all the numerators are 1. The generalized continued fraction allows for any numerator. Perhaps an even more famous continued fraction is that for the golden ratio $\varphi$:

As shorthand, simple continued fractions are sometimes written as $\sqrt{2} = [1;2,2,2,2,\ldots]$ to indicate the coefficients. So likewise, $\varphi = [1;1,1,1,1,\ldots]$.

Properties of Continued Fractions

Working with things that look like rational numbers should key us into that this is closer to where we want to be looking. And like decimals, continued fractions gives us a nice dichotomy between the rational and irrational.

Claim: A number is rational if and only if its simple continued fraction representation is finite.

Proof: $(\Leftarrow)$ If the continued fraction is finite i.e.

Then we can just collapse the continued fraction by creating common denominators and simplifying with normal fraction rules, showing $x$ is rational.

$(\Rightarrow)$ The idea is we'll continuously separate our fraction into its integer and fractional parts, and continue to reduce the fractional part. As an example,

If $x = \frac{p}{q}$ is rational, then we can repeatedly apply the division algorithm to create our continued fraction. So write $p = a_1 q + r_1$ for $0\leq r_1 < q$. Hence, we can write $\frac{p}{q} = a_1 + \frac{r_1}{q} = a_1 + \frac{1}{\frac{q}{r_1}}$. Again, we can repeat the division algorithm and write $q = a_2 r_1 + r_2$ for $0 \leq r_2 < r_1$.

To put it more neatly, write the following division algorithm steps:

Note since $r_1 > r_2 > \cdots > r_{n-1}$ form a sequence strictly decreasing and non-negative integers, this process must eventually terminate with all $r_m = 0$ after a certain point in a finite number of steps (this is what justifies using the division algorithm to find the GCD of two numbers). Once our algorithm terminates in a finite number of steps (this is what keeps our continued fraction finite), it's just a matter of writing out the continued fraction:

and we're done.

So this actually gives us a direct proof that $\sqrt{2}$ is irrational with its continued fraction: its simple continued fraction is infinite, and so it must be irrational.

More on Irrational Continued Fractions

Let's go back to our proof that all rational numbers have finite continued fraction expansions. Another way we can see this fact is by constructing continued fractinos for irrational numbers, and we do so in almost the exact same way we did for rationals. The idea in using the division algorithm is that we separate our number $x$ into its integer and fractional component, and then invert the fractional component (which is less than 1, so inverting it gives us a number greater than one to keep separating into integer and fractional parts). I.e. first write

and then repeat this exact writing process on $x - \lfloor x \rfloor$. But since $x - \lfloor x \rfloor$ is also irrational, and so will all future denominators at each step. The only way this process terminates is if our step $x - \lfloor x \rfloor = 0$ which is rational, and hence can't happen.

Claim: Let $x$ be irrational. Let $x_0 = x, \ a_k = \lfloor x_k \rfloor, \ x_{k+1} = \frac{1}{x_k - a_k}$. Then for all $k$

Proof: This follows immediately from our construction above: separate the integer part and invert the fractional. It clearly holds for $k=0$, since $x = x_0$. Say this holds for $k$. Then $x_{k+1} = \frac{1}{x_k - a_k} \Rightarrow x_k = a_k + \frac{1}{x_{k+1}}$ and so

Then it is not too hard to show that we can extend this to the infinite continued fraction in the way we'd want to.

Claim: $[a_0; a_1, a_2, \ldots] = \lim\limits_{n\to\infty} [a_0; a_1, a_2, \ldots, a_n] = x$

Before we continue into the infinite, we'll need some tools about the finite first that we will extend with limits. The truncated continued fractions of $x$ called the convergents of the continued fraction. The nth convergent of a continued fraction is

A convergent is just the initial segment of the continued fraction up to the $nth$ denominator. One useful fact we will need is the recursive formula for the numerator $p_n$ and denominator $q_n$ of these convergents (since the direct formula can get very complicated in simplifying it).

Lemma: For an infinite continued fraction $[a_0; a_1, a_2, \ldots]$ with $a_i > 0 \ \forall i > 1$, let $c_n = \frac{p_n}{q_n}$ be the nth convergent. We then have the following recurrence:

Proof: We'll prove this inductively. Note when I equate two fractions here, I mean to literally equate the numerators and denominators just as shorthand to show the two recursions at once (and simultaneously link it back to the convergent directly).

       Base Case: We just quickly check this $c_0 = \frac{a_0}{1} = \frac{p_0}{q_0}$. $c_1 = a_0 + \frac{1}{a_1} = \frac{a_1 a_0 + 1}{a_1} = \frac{p_1}{q_1}$.

       Inductive Step: Say this is true for $c_n = \frac{p_n}{q_n} = \frac{a_n p_{n-1} + p_{n-2}}{a_n q_{n-1} + q_{n-2}}$. Now consider $c_{n+1} = [a_0, a_1, \ldots, a_n, a_{n+1}]$. Note we can obtain $c_{n+1}$ from $c_n$ by replacing $a_n$ with $a_n + \frac{1}{a_{n+1}}$. That is, we can do a substitution trick like we did before, and write $c_{n+1} = [a_0, a_1, \ldots, a_n + \frac{1}{a_{n+1}}]$. Fortunately, by the inductive hypothesis, we have our numerator and denominator of $c_n$ in a form that is dependent on $a_n$ and makes this substitution possible (note that $p_{n-1}$, $p_{n-2}$, $q_{n-1}$, $q_{n-2}$ don't depend on $a_n$, so they remain unaffected by this substitution).

Giving us exactly what we wanted.

These recursions give us some very useful ways of characterizing convergents, since without them, it isn't hard to imagine how complicated the numerators and denominators might become when trying to collapse the continued fraction.

Claim: $p_{n-1} q_n - q_{n-1} p_n = (-1)^n$

Proof: As expected, we'll use induction:

       Base Case: For $n=1$, we get $p_0 q_1 - q_0 p_1 = a_0 a_1 - 1 \cdot (a_1 a_0 + 1) = -1$.        Inductive Step: Suppose this is true for $n$. Then using our recursions, we can show this holds for $n+1$:

This fact also then immediately tells us that $\gcd (p_n, q_n) = 1 $ and that they are coprime, so $c_n = \frac{p_n}{q_n}$ is always in lowest terms. It also gives us a nice relation between adjacent convergents:

This fact that $c_n - c_{n-1} = \frac{(-1)^{n+1}}{q_n q_{n-1}}$ will get arbitrarily small also shows they form a Cauchy sequence (the formal proof is just using this bound with the triangle inequality) and hence converge $\lim\limits_{n\to\infty} c_n$ exists. Below we'll show that they converge to the limit we expect.

Taking a minute to give some terminology and the above recurrence will help us in the future.

Claim: Using the $a_i$, $x_i$ defined above, $[a_0; a_1, a_2, \ldots] = \lim\limits_{n\to\infty} [a_0; a_1, a_2, \ldots, a_n] = x$

Proof: As with any question about infinity, we look at the partial, in-between steps and consider the formal limit. That means our convergents $c_n$ i.e. we want to show $\left| x - c_n \right| = \left| x - \frac{p_n}{q_n} \right| \rightarrow 0$.

First, note clearly the $x_n$ is irrational for all $n$. Second, we'll show $a_n > 0$ for all $n$ so we can use our recursive formula of convergents above. This follows because our algorithm essentially takes the fractional part of our number (which is less than 1), and inverts it (making it greater than 1) at the next step: since $a_k = \lfloor x_k \rfloor$, we have $a_k < x_k < a_k + 1$ (we get a strict lower inequality by the irrationality of $x_k$). So $0 < x_k - a_k < 1$. Hence, $x_{k+1} = \frac{1}{x_k - a_k} > 1$, and therefore $a_{k + 1} = \lfloor x_{k+1} \rfloor \geq 1$.

So we can use our recursive formulas for convergents we proved above. Also using our "finite" continued fraction from before, we can write

Therefore,

The last equality we showed above with our recursions. Taking absolute values,

Also, $x_{n+1} > \lfloor x_{n+1} \rfloor = a_{n+1}$, so $x_{n+1} q_n + q_{n-1} > a_{n+1} q_n + q_{n-1} = q_{n+1}$:

Next, note that $q_n \geq n$ for $n \geq 1$:

      Lemma: $q_n \geq n$ for $n \geq 1$.

      Proof: $q_1 = a_1 \geq 1$ as shown earlier, so we have a base case established. Say $q_k \geq k$ for $k\leq n$. Then $q_{n+1} = a_{n+1}q_n + q_{n-1} \geq 1 \cdot n + n-1 \geq n+1$. $\blacksquare$

Finally,

Giving the limit

So we're actually justified in using these infinite continued fractions in the way we want to, and generate them in a semi-algorithmic way:

I say semi-algorithmic since the way I presented above already relies on having the exact value of $\pi$ in decimal, which can be arguably harder to find. It's good enough to get us as many coefficients as we'll practically need, but we'll address this problem later.

Best Rational Approximations

Now the reason we went through all this trouble is because continued fractions have the surprising ability to generate best rational approximations systematically. Remember, we say a rational approximation $\frac{n}{d}$ for a real number $x$ is best if $\frac{n}{d}$ is closer to $x$ than any other fraction with denominator less than $d$.

Definition: We say $\frac{p}{q}$ is a best rational approximation for $x$ if

Now here's the remarkable fact:

Theorem: Write an irrational number $x = [a_0; a_1, a_2, \ldots]$ as its continued fraction expansion. Then for any convergent $c_n = \frac{p_n}{q_n}$ and any rational $\frac{a}{b}$ with $b < q_{n+1}$, then $\left| q_n x - p_n \right| \leq \left| bx - a \right|$.

This is actually a stronger statement, since it's not only a best rational approximation, but it is also a better rational approximation than some denominators greater than $q_n$. It should not be surprising that $q_{n+1} > q_n$ for all $n$ (just think about the process involved in collapsing a continued fraction; or show by induction with the recursion formulas). But what our claim says is that for not only rational approximations of denominator less than $q_n$, but also for some denominators greater than $q_n$, $\frac{p_n}{q_n}$ is still a better approximation for $x$. Which is surprising, since that would mean even if we divide the real numbers into smaller boxes, sometimes our numbers will be closer to the edges of the bigger box. It's like saying that a steak knife can get you a more precise cut than some X-Acto knives.

Proof: Let $\frac{a}{b}$ be a rational number in lowest terms such that $b < q_{n+1}$. We'll also assume $a \neq p_n$ and $b \neq q_n$ (since then we clearly have equality). We'll rewrite $a,b$ in terms of new numbers $r,s \in \mathbb{R}$ that we define by the following system of equations:

$r,s$ exist since that system of equations is equivalent to the matrix system

and we showed that the determinant $p_n q_{n+1} - q_n p_{n+1} = (-1)^{n+1} \neq 0$ and so is invertible. Now if we solve for $r,s$ (either with matrices or directly), we see that

So $r,s$ are integers.

      Claim: $r,s \neq 0$.

      Proof: If $r = 0$, then $q_{n+1}a = p_{n+1}b$. But since $p_{n+1}$ and $q_{n+1}$ don't divide each other (we proved earlier they were coprime), we have $q_{n+1}$ divides $b$, contradicting the assumption that $b < q_{n+1}$. If $s=0$, then we have $\frac{a}{b} = \frac{p_n}{q_n}$ which we already excluded in our assumptions. $\blacksquare$

      Claim: $r$ and $s$ have opposite signs.

      Proof: We'll just check the two cases.

• If $s > 0$, then $s \geq 1$. Therefore $sq_{n+1} \geq q_{n+1} > b$, and thus $b = rq_n + sq_{n+1}$ implies that $rq_n = b - sq_{n+1} < 0$. Since $q_n > 0$, we must have $r<0$.
• If $s < 0$, then $-sq_{n+1} > 0$. Hence we have $rq_n = b - sq_{n+1} > 0$. Thus $r > 0$. $\blacksquare$

      Claim: $q_n x - p_n$ and $q_{n+1} x - p_{n+1}$ have opposite signs.

      Proof: Note we saw earlier that the difference between consecutive convergents alternate in sign:

That is, the convergents take turns being greater or lesser than the previous convergent. What this means is that our convergents alternately overshoot and undershoot its limit $x$ (in particular, knowing that $c_0 = \lfloor x \rfloor < x$, the even convergents undershoot and the odd ones overshoot). So in particular, for all $n$, we have

If we take the first case, then the first inequality gives $0 < q_n x - p_n$ while the second one gives $q_{n+1} x - p_{n+1} < 0$, so they're of opposing signs. Similarly the second case gives the same. $\blacksquare$

      Claim: $r(q_n x - p_n)$ and $s(q_{n+1} x - p_{n+1})$ have the same signs.

      Proof: This is just something you can check by cases. The opposing parity of $r,s$ and of $(q_n x - p_n), (q_{n+1} x - p_{n+1})$ cancel each other out. $\blacksquare$

Finally, putting this last fact to work, we get

We can separate the absolute values in the 3rd equality since they have the same sign, and the last inequality comes from the fact that we showed the integer $r \geq 1$.

For completeness sake,

Corollary: Each convergent $c_n = \frac{p_n}{q_n}$ is a best rational approximation for $x$.

Proof: Suppose otherwise, so that there $\exists \frac{a}{b}$ such that $b \leq q_n$ with

Multiplying these inequalities together gives

But this contradicts our theorem as $b \leq q_n < q_{n+1}$. $\ \blacksquare$

The Most Irrational Number

If you want to approximate $\pi$, we now have a good way of doing so: take some convergent of the continued fraction. Looking at the first couple,

which are some of the more famous approximations of $\pi$. Note, if you do want to get really good approximations, making the last term in a convergent large is ideal. Notice how much closer we got with $c_2$ compared to $c_1$, and that is precisely because 15 is (relatively) much bigger than 7. These large coefficients get multiplicatively bigger as we simplify the convergent, giving us a rather large denominator $q_n$ that more finely dices the real numbers for us to approximate with. $\pi$ has a lot of these large numbers in its continued fraction expansion, giving us good convergents to create these surprisingly accurate best rational approximations.

This also means, if you care about small denominators, it's worth stopping one before a large number (i.e. at 7 instead of 15, or 1 instead of 292); if you need such a large number to generate a new best rational approximation (and get a really big denominator), then it must have been that the approximation before it must have also been really good if you had to zoom in that much on the next step.

NOTE: we proved that all convergents are rational approximations, but the converse is not true! $\frac{13}{4}$ is also a best rational approximation for $\pi$, but is not generated through our convergents.

So, what if we minimize all the coefficients then? If we give no large numbers to stop at for our convergents, we'll be left with a number that is not easy to approximate at all. We know all the numbers are positive integers, so let's just set them all to 1 to minimize them:

We know this converges as we showed that the difference between consecutive convergents gets small (and so forms a Cauchy sequence), and as we saw before this is the Golden Ratio $\varphi$. We can also solve this directly if you wanted: if we set the value of the fraction to $x$

we can see that the boxed portion of the fraction is equivalent to the original, whole fraction, so we can substitute $x$ for the box.

Is precisely the quadratic that has roots at $\frac{1 \pm \sqrt{5}}{2}$ (we only admit the positive solution by since all our convergents are positive, and limits preserve weak inequalities, so the limit is non-negative).

So if we look at some of the convergets of $\varphi$, they're quite bad at approximating it (for reference, $\varphi \approx 1.618034$):

In fact, takes until $c_9$ to get 3 decimals correct. Remember, the first convergent for $\pi$ was already within 3 decimals.

Since $\varphi$ is so bad—in fact as we've demonstrated, the worst—to approximate rationally, it is often referred to as the most irrational number.

Relation to Dirichlet's Approximation Theorem

Recall, in our first little discussion about approximating irrational numbers, we spoke about Dirichlet's Approximation Theorem.

Dirichlet's Approximation Theorem: $\forall x \in \mathbb{R}$ and $\forall N \in \mathbb{N}$, there are integers $p$ and $q$ such that $0 < q \leq N$ and $\left|qx - p \right| < \frac{1}{N}$.

The more concrete result we got out of it was that we could have rational approximations $\frac{p}{q}$ that get us within $\frac{1}{q^2}$ of our irrational $x$, which is much better than our initial bound of being within $\frac{1}{2q}$ of $x$.

In our proof on the convergence of continued fractions, we actually say that our convergents actually satisfy this bound too. Look above in the proof, and we deduced that

and the final inequality comes from the fact that $q_{n+1} > q_n$. So, as we'd hope, our best rational approximations from continued fractions are also just outright good and efficient approximations.

In fact, we can usually do a little bit better.

Corollary: For irrational $x$ and for all $n \in \mathbb{N}$, at least one of the convergents $\frac{p_n}{q_n}$ or $\frac{p_{n+1}}{q_{n+1}}$ satisfy the inequality

Proof: Suppose this isn't true. Then

The second equality follows since $\frac{p_n}{q_n} < x < \frac{p_{n+1}}{q_{n+1}}$ or $\frac{p_{n+1}}{q_{n+1}} < x < \frac{p_n}{q_n}$. Say we're in the first case, then

and the $x$ terms cancel (similarly for the second case). The last equality is precisely the difference between two adjacent convergents we found earlier.

But we already know that $q_{n+1} > q_n$, so this can't hold, giving us our contradiction. $ \ \blacksquare$

It turns out the converse to this is in fact true:

Legendre's Theorem: If $x \in \mathbb{R}$, and $\frac{p}{q} \in \mathbb{Q}$ such that $\left| x - \frac{p}{q} \right| < \frac{1}{2q^2}$, then $\frac{p}{q} = c_n$ is a convergent of the continued fraction for $x$.

We won't prove this, but the proof can be found on the Wikipedia page.

Finding Continued Fractions

One problem I had with continued fractions originally was that there didn't seem to be a particularly nice way of actually finding them. It's nice that we have this very straightforward way of doing it by separating integer parts and inverting the fractional, but that already requires us to have calculated the decimal expansion of whatever number we're expanding. This makes it seem not as elegant and a bit clunky, so when I found this paper outlining another method, I had to look through it.

The accuracy of this method will still rely on having some level of floating point precision, but otherwise, I think it is a much nicer of going about finding continued fractions in a way that weaves in another natural approximation method. The idea lies within finding roots of functions in a way similar to Newton's method.

Say we want to find the continued fraction of the number $\alpha$. Further, assume that there is a twice differentiable function $f(x)$ such that $f(\alpha) = 0$. To get a rough idea of the algorithm, there are two facts we first must take note of.

1) The Mean Value Theorem: Given a function $f(x)$ that is differentiable on the interval $(a,b)$, the Mean Value Theorem states that $\exists c \in (a,b)$ such that

That is, there is a point that has a tangent with the same slope as the secant line between endpoints of the graph. So if we consider the interval between $(\alpha, t)$ where $f(\alpha) = 0$, we get that

Rearranging a little bit and letting $t= \frac{p_n}{q_n}$ be a convergent of $\alpha$, we can see that

2) Relating convergents and remainder coefficients: Recall in our proof that simple infinite continued fractions converge, we found that

where $\alpha_{n+1}$ is the "remainder" coefficient we get when calculating continued fractions i.e.

If we combine these two equalities, we get that

If we solve for $\alpha_{n+1}$:

This is great, since now we have a direct way of computing the coefficients $a_{n+1} = \lfloor \alpha_{n+1} \rfloor$. Further, the computation only requires knowing the evaluating the function $f(x)$ and the previous two convergents! This is the basis for the algorithm: we repeatedly compute values of $a_{n+1}$ by computing the remainder $\alpha_{n+1}$ with our convergents.

However, this only maintains equality for the value fo $c \in (a, \frac{p_n}{q_n})$. So what we do instead is approximate $\alpha_{n+1}$ with a reasonably close value. One value that works just fine is $\frac{p_n}{q_n}$ itself. So what originally was could be a hard problem of finding decimal expansions of $\alpha$ to do our separate-and-invert algorithm, now becomes a much more tractable problem in computing decimals in evaluating a function. Also this process is memoryless: it does not need anything more than the previous state two convergents to compute the next. Whereas if we wanted to refine our continued fraction with the old method, we would have to start from the very beginning and recompute everything.

Again, this still requires calculating something else to an arbitrary precision to get the correct continued fraction (and we still need to get the first two convergents manually as seed values). So in the case of $\pi$, we are calculating instances of $\sin(x)$, which in some ways easier. This gives a wide range of numbers we can now compute continued fractions for:

• For $\sqrt{2}$, we use $f(x) = x^2 - 2$
• For $\ln(n)$, we use $f(x) = e^x - n$.
• For $\pi^2$, we use $f(x) = \sin(\sqrt{x})$
• For $e^\pi$, we use $f(x) = \sin(\log x)$

Many other function compositions can lend to many other numbers that would seem otherwise hard to get continued fractions of.

For more details and a coded demo, check out this Python notebook that is set up to calculate the first 50 coefficients (as well as the convergents and their numerators and denominators) of the simple continued fraction for $\pi$.

Applications of Best Rational Approximations

Music Tuning

Approximating irrationals are convenient for nothing more than just that in most cases: convenience. But here's something that requires this irrational approximation.

We here sound and music intervals on an exponential scale. Given a note with frequency $f$, it's octave is the note with frequency $2f$. So obviously, they have a frequency ratio of 2:1. Since this, in a sense, is the simplest ratio we can have between notes' frequencies, the octave forms the basic ratio for end points in our musical scale. The more interesting notes are the ones between an octave with more complex ratios to the base frequency. Common intervals include a major third with a frequence ratio of 6:5 to the base note, or perfect fourth with 4:3. But perhaps the most famous and most recognizable is the perfect fifth with a ratio of 3:2. The theme to Star Wars famously opens with a perfect fifth.

As nice as these rational intervals are, they pose a slightly annoying problem: they are not (geometrically) evenly spaced, so they don't divide an octave into even steps. If you wanted to build a piano, this would make tuning each note very tedious, and make our notes feel a bit random without this consistent spacing.

If we wanted to divide an octave evenly into $n$ notes, that means we want to be able to find the corresponding frequency ratio of $2^\frac{1}{n}$ (remember, we hear sound exponentially, so hearing the next note up is equivalent to multiplying our base frequency by our desired ratio). But this number is irrational (proof is identical to our why $\sqrt{2} = 2^\frac{1}{2}$ is irrational)! So this runs into a different problem: we'll never be able to exactly get our rational frequency ratios for our intervals like the fourth or the fifth.

So we're at a crossroads: either we keep our rational frequency ratios for nice intervals and the octave isn't evenly divided, or we divide our octave evenly and never have a rational interval. So we compromise: we find a value of $2^\frac{m}{n}$ that closely approximates our ratios, and that value of $n$ tells us how many notes we divide our octave into.

Here's where we can use continued fractions and our best rational approximations. Since the perfect fifth is probably the most univerally harmonious interval, we'll try and approximate that first. Then, we note that

So if we can approximate $\log_2 (3/2)$ the best we can, we'll get a reasonable rational exponent that closely solves $2^\frac{m}{n} = \frac{3}{2}$. We can write out the first few numbers of the continued fraction by hand using our algorithm from before:

Looking at the first few convergents, we see that

Looking at our denominators, we get options of $n = 1,2,5,12,41,\cdots$. Using $n \leq 5$ gives too few notes in a scale, and $n = 41$ might be too many. So $n=12$ notes feels like a good in-between, and in fact is what we actually use on most pianos today: 7 white keys and 5 black per octave. (One interesting best approximation not found here is also $n=19$, which some might find a reasonable choice too)

Cryptography and Wiener's Attack

RSA is one of the most widely used cryptographic protocols used, protecting most of the internet's traffic today.

Here's a brief rundown how two people, Alice and Bob, would share secret messages (i.e. can only be read by the recipient) using RSA:

• Bob chooses two (ideally large) prime numbers $p,q$ and calculates $N = pq$, and an integer $e$ (called the encryption exponent). He then releases to everyone the public key $(N, e)$. This is how people will hide and encode their messages to Bob.
• To decrypt messages, we need we need the decryption exponent $d$ that satisfies $ed \equiv 1 \bmod \varphi(N)$. $\varphi(n)$ is Euler's totient function and returns the total number of integers less than $n$ that are coprime to $n$. We also need $\gcd(e, \varphi(N)) = 1$.
• The factorization of $N$ and $d$ are kept secret, and form the private key. This is what allows Bob and only Bob to decrypt messages.
• To encrypt a message $M$, Alice sends the ciphertext $C = M^e \bmod N$
• To decrypt it, Bob computes $C^d \equiv (M^e)^d \equiv M^{ed} \equiv M \bmod N$

The decryption equivalence follows by Fermat's little theorem, and is not too hard to verify (you can also check the Wikipedia page). Also, note that sometimes instead of $\varphi(n)$ people will also use the Carmichael function $\lambda(n)$. In our case, $\lambda(pq) = \mathrm{lcm}(p-1,q-1) \leq (p-1)(q-1)$ which maintains all the properties needed.

The security of RSA relies on the surprising asymmetry in factorization: given $p$ and $q$, calculating $N = pq$ isn't hard, but given an $N$ finding two numbers such that $N = pq$ is difficult. Without being able to factor $N$, the private key remains private and hence messages can't be stolen. At least, that's our idea—it's secure because we don't know how to factor numbers efficiently (at least, not without quantum computers). No one does. So we use RSA everywhere for that reason.

So if we could break it, that would be quite bad.

Wiener's attack exploits one thing that we didn't put much effort specifying: the decryption exponent $d$. If $d$ is chosen in such a way that it's small enough (specifically when $d < \frac{1}{3} N^{\frac{1}{4}}$) Wiener's attack can break through RSA. This, though, is a really small range of $d$. If we let $N$ be a number with 20 digits, $d$ can at most have 5 digits.

Some Preliminaries

First, note for (co)prime $p,q$, we have

This follows from a simple counting argument: the numbers that are not coprime to $pq$ are the multiples of $p$ and the multiples of $q$ less than $pq$, i.e. $1p, 2p, 3p, \cdots, (q-1)p$ and $1q,2q,3q,\cdots(p-1)q$. So there are $(p-1) + (q-1) = p + q - 2$ numbers not coprime to $pq$. There are $pq-1$ numbers less than $pq$. Hence there are

numbers coprime to $pq$. (also follows because $\varphi$ is multiplicative, so $\varphi(pq) = \varphi(p)\varphi(q)$ if $p,q$ coprime; think Chinese Remainder Theorem).

So if we know $p+q$ then we know $\varphi(N)$ and vice versa. If you're familiar with Vieta's fomrulas, these look a lot like expressions that appear in quadratics:

By the quadratic formula,

So if we know $N$ and $\varphi(N)$, we can recover $p,q$ without ever needing to go through the problem of factoring. $N$ is already public information in RSA. To find $\varphi(N)$, all we know about $\varphi(N)$ is $ed \equiv 1 \bmod \varphi(N)$. In other words, $ed = 1 + k \cdot \varphi(N) = 1 + k \cdot (p-1)(q-1)$ for some integer $k$. Therefore,

If we want to find $\varphi(N)$, what we really want to find is $k$ and $d$.

Second, we make the following observation:

• Let $N=pq$, $e$, and $d$ be given.
• Note $\varphi(N) = (p-1)(q-1)$.
• Also, $ed = 1 + k \cdot \varphi(N) = 1 + k \cdot (p-1)(q-1)$ for some integer $k$
• Dividing by $dpq$, we get $\frac{e}{pq} = \frac{k}{d} \cdot \frac{pq - p - q + 1 + \frac{1}{k}}{pq} = \frac{k}{d}(1-\delta)$ where $\delta = \frac{p + q - 1 - \frac{1}{k}}{pq}$
• Note $\delta < 1$ since $pq > p+q$ for all $p,q > 1$. So we get $\frac{e}{pq} < \frac{k}{d}$ by a small amount.

So we have $\frac{e}{N} \approx \frac{k}{d}$, so if we can guess approximations (should start ringing a bell) of $\frac{e}{N}$, we might be able to guess $\frac{k}{d}$.

Wiener's idea then is to use the convergents of the continued fraction for $\frac{e}{N}$ to get potential values of $\frac{k}{d}$, as those would give us nice rational approximations.

Example: Let $(N,e) = (90581, 17993)$ be our public key.

The first convergent $\frac{0}{1}$ does not give $k,d$ values to factor $N$ ($k=0$ is the real problem). But it can be checked that the next convergent $\frac{1}{5} = \frac{k}{d}$ does work, and gives the correct guess for $\varphi(N)$:

If we plug this into our quadratic from before, we get $p,q = 379, 239$ which correctly factors $N$.

Here are all the conditions necessary for this attack.

Wiener's theorem: Let $N=pq$ with $q < p < 2q$, and $d < \frac{1}{3} N^{\frac{1}{4}}$ such that $ed \equiv 1 \bmod \varphi(N)$. Given $(N,e)$, one can recover $d$.

Proof: First note that since $q<p<2q$, we have that $p+q - 1 < 3\sqrt{pq}$:

Recall that $\varphi(N) = (p-1)(q-1) = N - p - q + 1$. Combining this with our above inequality gives $\left| N - \varphi(N) \right| < 3\sqrt{pq} = 3\sqrt{N}$. Now also remember that $ed - k\varphi(N) = 1$, so we get that

$k\varphi(N) = ed - 1 < ed$, so since $e < \varphi(N)$ (the public key can and usually is chosen as such because modular arithmetic ensures there is one), we then must have $k < d$ for that equality to have any hope in working out. Therefore,

Since $d < \frac{1}{3} N^{\frac{1}{4}}$, we have that $9d^2 < N^{\frac{1}{2}}$. Hence,

By Legendre's theorem that we mentioned above, $\frac{k}{d}$ is equivalent to a convergent of the continued fraction for $\frac{e}{N}$. Also, $ed - k\varphi(N) = 1$, so $\gcd(k,d) = 1$, so $\frac{k}{d}$ is already in lowest terms. So not only is $\frac{k}{d}$ equivalent to a convergent, it is in fact the convergent with the same numerator and denominator.

Note that the bound on $d < \frac{1}{3}N^{\frac{1}{4}}$ was picked precisely to satisfy Legendre's theorem, to ensure it would be a convergent. Others have improved on this bound, and one we can clearly see that if we want $\frac{3}{\sqrt{N}} < \frac{1}{2d^2}$, we can let $d < \frac{1}{\sqrt{6}}N^{\frac{1}{4}}$.

Solving Pell's Equation

Pell's equation is the Diophantine equation (that is, we want integer solutions for it)

where $D$ is a non-square positive integer (for negative or perfect square $D$ since there are only the finitely many solutions $(\pm 1, 0)$ by considering the sign of the equation or factoring a difference of squares). Continued fractions and solutions to Pell's equations are surprisingly intertwined.

The key lies in the fact that there is a 1-to-1 correspondence between quadratic irrationals and repeating continued fractions. That is, if the coefficients in a continued fraction ever cycle, then the continued fraction can be written as $\frac{p + \sqrt{q}}{r}$ where $q$ is a non-square.

We won't get into it here, but here are some notes and a paper that fill in the details for quadratic irrationals and Pell's equation.

Pell's equation seems innocuous at first—just a genereic equation that people historically studied for one reason or another, but it is because of its general form that lends itself to the necessity of studying Pell's equation.

Question: The triangular numbers are $1, 3, 6, 10, \cdots, \frac{n(n+1)}{2}, \cdots $. Are any of these a perfect square?

Solution: We are essentially solving $\frac{1}{2}n(n+1) = m^2$. Rewriting a little bit, we want to solve $(2n+1)^2 - 8m^2 = 1$. Let $x = 2n+1$ and $y = m$, we just need to find solutions to $x^2 - 8y^2 =1$. From the above results (that we did not cover), we can do that systematically with the continued fraction for $\sqrt{8}$.

For an absurd historical example of Pell's equation, here's one attributed to Archimedes.

The Original Proof of the Irrationality of $\pi$

We've gone through a few properties and interesting tidbits that characterize irrational numbers especially as it comes to computing and approximating them. These tools, especially of continued fractions, give a tool that can be used to deduce irrationality of numbers. In fact, the first proofs that $e$ and $\pi$ were irrational originally came from them. But I wanted to talk about them specifically because I want it to be clear jsut how little we actually know of irrational numbers and how difficult it can actually be to show that numbers are irrational (as we'll discuss below). Continued fractions, though, seem to be a more universal way out. We've already shown how $\sqrt{2}$ is irrational, but the actual proof is quite unique; we didn't use any of the characteristics of irrational numbers. But then we showed it was irrational again purely from its continued fraction, and that almost completely streamlined the proof to not only be direct, but essentially shorten the proof to one line: just show the continued fraction.

Before we go onto some of the more modern proofs $\pi$ is irrational, we're now somewhat familiar with continued fractions, so let's look at J. H. Lambert's (1761) original proof using them.

Theorem: $\pi$ is irrational.

Proof: I'm adapting the proof found on this Stack Exchange post, as it only uses what's necessary. For some more rigor, the Wikipedia page or this paper provide good details.

There are a few steps we'll take along the way.

      Step 1: A continued fraction for $\tan(x)$.

This is by far the most tedious step, and I'll only sketch it out for actually how annoying it is. The way to derive it is to essentially consider the quotient of the power series for $\sin(x)$ and $\cos(x)$.

We'll now do a series of manipulations on this last fraction:

Let's look at the new sub-fraction of power series we have in the denominator. We'll now add and subtract the denominator of the sub-fraction as a special form of 0 to its numerator:

Combining the terms in red:

So, currently, we have

We can then continue to play the same game and adding and subtracting the denominator of our sub-fraction to its numerator:

We multiplied them by 3 to clear the units. Combining the terms in red again:

So then our expansion is

If we continue doing this, we see the pattern

Obviously, this isn't fully rigorous as we're playing with infinite series pretty carelessly, so see the above links for more details.

      Step 2: A condition for irrationality.

Lemma: For a continued fraction

assume that $1 + a_n \leq b_n$. If $1+a_n < b_n$ infinitely often, then the fraction is irrational.

Proof: For contradiction, say it is rational.

The conditions $1 + a_n \leq b_n$ and $1+a_n < b_n$ infinitely often ensure that our continued fraction is between 0 and 1 (analyze the convergents), so we have $\lambda_1 < \lambda_0$. To make our life simpler, let's call a lower portion of the fraction $\rho_1$:

Then we have

So $\rho_1 = \frac{\lambda_2}{\lambda_1}$ is rational. But also, since $\rho_1$ is essentially the same "type" of continued fraction as the original, it too is between 0 and 1. Hence $\rho_1 < 1$ and therefore $\lambda_2 < \lambda_1$. We can keep repeating this process, producing a sequence of strictly descending positive integers $\cdots < \lambda_2 < \lambda_1 < \lambda_0$. But clearly this is impossible! So our continued fraction is irrational. $\ \blacksquare$

      Step 3: Showing $\pi$ is irrational.

Say $\pi$ is rational. Then $\frac{\pi}{4} = \frac{a}{b}$ is also rational, and $\tan(\frac{\pi}{4}) = 1$. But we also have our continued fraction for $\tan(x)$ :

But clearly, $a^2$ is constant, so for a large enough $n$, we'll have $1 + a^2 < nb$. So at some point, we'll have a sub-fraction

that satisfies our Step 2 claim, and hence is irrational. If that subfraction is irrational, then clearly the whole fraction is irrational (just by thinking about how that continued fraction would collapse and simplify up). But clearly $\tan(\frac{\pi}{4}) = 1$ is rational. Contradiction!

So it must be that $\pi$ is irrational.

In case you were interested, since we spent all that time with approximation theorems from before, it's interesting to note that for all integers $p,q$ we have

so $\pi$ gets pretty close to some rational numbers (as we've seen with our convergents before), but it's interesting seeing the lower bound given. As always, paper attached.

The Simpler Irrationality Proofs

This proof is important for its history alone: it was the first sound proof that $\pi$ is irrational. In fact, $e$ was also first proven irrational in a similar way (Euler, 1737) by writing out the continued fraction for $e$ and showing it was infinite.

But Fourier had a much better proof that $e$ is irrational:

Claim: $e$ is irrational.

Proof: Suppose $e = \frac{a}{N}$ was rational. Now consider $e\cdot N!$ which is clearly an integer by assuming $e = \frac{a}{N}$. We can evaluate this using the series expansion $e = \sum_{k=0}^\infty \frac{1}{k!}$.

Now the terms in blue clearly add to an integer. But the parts in red:

We bounded the red series by a geometric series, and were able to show that it must be less than 1. But that means the series in red is not an integer. Therein lies our contradiction: if we assume $e$ is rational, we have $e \cdot N!$ must be an integer, but the series expansion for $e$ would suggest that it's not.

Continued fractions are still useful today, but for irrationality proofs, the above is so much more attractive compared to finding infinite continued fractions. Which makes me want to think that Lambert's proof that $\pi$ is irrational—albeit important—is an unwieldy relic to what else is there.

Claim: $\pi$ is irrational.

I. Niven's (1947) proof that $\pi$ is irrational is actually quite short, and only relies on some calculus and a helper function. Notably, the key fact that it relies on is that $\pi$ is the smallest positive 0 of $\sin(x)$.

Lemma: For all $n \geq 1$, consider the function

Then the following are true:

1. $f(x)$ is a polynomial of the form $\frac{1}{n!} \sum_{k=n}^{2n} c_k x^k$ with integer coefficients $c_k$
2. For $0 < x < \frac{a}{b}$, we have $0 < f(x) < \frac{1}{n!}$
3. For all $k \geq 0$, the derivatives $f^{(k)}(0)$ and $f^{(k)}(\frac{a}{b})$ are integers.

Proof: Claims 1. and 2. are straightforward (consider binomial expansion, and the factors in the numerator). For 3., note by 1., we've shown $f(x)$ is a polynomial consisting of terms with degree between $n$ and $2n$. So $f^{(k)}(0) = 0$ unless $n\leq k\leq 2n$. When $n\leq k\leq 2n$, then $f^{(k)}(0) = \frac{k!}{n!} c_k$. Since $k \geq n$ and $c_k$ is an integer, then $\frac{k!}{n!} c_k$ is also an integer. By the symmetry in $f(x) = f(\frac{a}{b}-x)$, we get $f^{(k)}(x) = (-1)^k f^{(k)}(\frac{a}{b}-x)$. Therefore $f^{(k)}(\frac{a}{b}) = (-1)^k f^{(k)}(0)$ is also an integer. $ \ \blacksquare$

Now we can prove $\pi$ is irrational.

Proof: Assume $\pi = \frac{a}{b}$ is rational, and consider $f(x)$ from the above lemma. Now define the new function

If we take the derivative of this, we find

$F''(x)$ looks a lot like $F(x)$. In particular, we have this relation:

Note this holds precisely because $f^{(2n+2)}(x) = 0$. With this in mind, we can see that the following derivative is

Thus, if we integrate the righthand-side,

By part 3. of our lemma $f^{(k)}(\pi)$ and $f^{(k)}(0)$ are integers for all $k$, so $F(\pi) + F(0)$ is an integer.

Note now for $0<x<\pi = \frac{a}{b}$ and that $0 < \sin(x) < 1$ in this range, we have

But the RHS can be made arbitrarily small, i.e. $\frac{\pi^n a^n}{n!} < \frac{1}{\pi}$ for a big enough $n$. So we can bound our integral by

However, we claimed that $\int_{0}^{\pi} f(x)\sin(x) = F(\pi) + F(0)$ is an integer. But there is no integer between 0 and 1! So our assumption that $\pi$ was rational must be wrong, and so $\pi$ is irrational.

This proof is "simpler" in that the mathematical machinery required is relatively low-level. It might be that the actual contradiction we were looking for is less obvious, but we didn't have to prove that many obscure, auxilliary lemmas like we did with continued fractions in Lambert's proof.

Proofs from THE BOOK generalizes this proof strategy to show a few more interesting irrationality results.

Lemma: For all $n \geq 1$, consider the function

Then the following are true:

1. $f(x)$ is a polynomial of the form $\frac{1}{n!} \sum_{k=n}^{2n} c_k x^k$ with integer coefficients $c_k$
2. For $0 < x < 1$, we have $0 < f(x) < \frac{1}{n!}$
3. For all $k \geq 0$, the derivatives $f^{(k)}(0)$ and $f^{(k)}(1)$ are integers.

Proof follows exactly as you'd expect from before.

Theorem: $e^r$ is irrational for all rational $r \neq 0$.

Proof: We can reduce this by considering $e^p$ is irratraional for all integers $p$, since if $e^{\frac{p}{q}}$ was rational, then $(e^{\frac{p}{q}})^q = e^p$ would also be rational. The key idea here is using that $\frac{\mathrm{d}}{\mathrm{d}x} e^x = e^x$.

Let $f(x) = \frac{x^n(1-x)^n}{n!}$ refer to the one from the lemma above. As usual, assume the contrary and that $e^p = \frac{a}{b}$. Now consider the new function

This looks familiar to the one Niven used to show $\pi$ is irrational. Taking its derivative:

Hence, we get the relation

Now we take the derivative of a particularly nice function:

Integrating, we then get

Recalling that $e^p = \frac{a}{b}$, we conclude that

is an integer. As before, we can bound this integral from above:

We get the last inequality by just taking a large enough $n$. There are no integers between 0 and 1, giving us our contradiction and completing the proof.

Open Questions and More Strangeness

Irrational Powers

But not much about irrationals are that well understood. It's quite simple to show that combining an rational with another rational number via addition, multiplication, division, and subtraction, only results in more rational numbers. Further, it's not too bad to show that combinations in the same way with an irrational and rational only lead to more irrationals. But combining irrationals are a little strange: obviously $(1 - \sqrt{2}) + \sqrt{2} = 1$ is the sum of two irrational numbers that result in a rational. You can even find two irrational numbers $a,b$ such that $a^b$ is rational.

Claim: There exists irrational numbers $a$ and $b$ such that $a^b$ is rational.

Proof: We know that $\sqrt{2}$ is irrational, so consider the number $\sqrt{2}^\sqrt{2}$. If this is rational, we're done. If this is irrational, then consider the number $(\sqrt{2}^\sqrt{2})^\sqrt{2} = \sqrt{2}^2 = 2$, which is rational, and hence we are done.

In fact, it is possible to show for a lot of positive rational numbers $r$, there exists an irrational $a$ such that $a^a = r$.

Theorem: For every rational number $r \in \left( (1/e)^{1/e}, \infty \right)$, either $r = a^a$ for an irrational $a$, or $r \in \{1, 4, 27, 256, \cdots, n^n, \cdots \}$.

Proof: Consider the function $f(x) = x^x$ on the interval $I = (1/e, \infty)$. $f(x)$ is continuous on $I$, since it is differentiable, and injective as it is monotonic (check derivative is strictly positive), so $f(I) = \left( (1/e)^{1/e}, \infty \right)$. So given a rational $r \in f(I)$, let $a \in I$ be the corresponding value such that $a^a = r$.

We'll now go and prove the contrapositive: if $a$ is rational and not an integer, then $a^a$ is irrational. In other words, if $r = \frac{p}{q}$, and $a = \frac{n}{m}$ are in lowest terms i.e. $\gcd(p,q) = \gcd(n,m) = 1$, and

then $m = 1$ and $a$ must be an integer. So, for a contradiction, assume that $m > 1$. Rearranging our equation, we get that

Since $\gcd(p,q) = 1$, a prime divides the factor $q^m$ on the left-hand side if and only if it divides the factor $m^n$ on the right-hand side. So $q^m = m^n$. Since $m > 1$, we can write $q = \alpha^i k$ and $m = \alpha^j l$ for some prime $\alpha$ and integers $k,l$. Since $q^m = m^n$, we must also have $im = jn$ for the exponent of $\alpha$ to match. Thus $i(\alpha^j l) = jn$, and $\alpha^j$ divides $jn$.

But $\gcd(m,n) = 1$, and $\alpha^j$ is a factor of $m$, hence $\alpha^j$ divides $j$. So, $\alpha^j \leq j$. Since $\alpha$ is a prime, $2 \leq \alpha$, and so $2^j \leq j$. But it can easily be shown that $2^j > j$ for all integers $j$, giving us the contradiction.

Transcendental Numbers

$\pi$ being irrational is a well-known fact among everyone. A more surprising fact to more mathematically inclined is the fact that $\pi$ is transcendental.

Not all irrational numbers are equal. $\sqrt{2}$ is, in some ways, defined by the fact that it is the (positive) root to $x^2 - 2 = 0$. Despite being, well, just a number that seems to exist as its own object, $\sqrt{2}$ has a decidedly algebraic quality to it which allows us to precisely specify it in much simpler terms (i.e. integers and arithmetic). Lots of numbers can be characterized as solutions to polynomials. Even imaginary numbers, which many find difficult to grasp, can be simply defined in terms of simple polynomials: $\pm i$ are the solutions to $x^2 + 1 = 0$.

But not every number can be defined via a polynomial. In fact, it's not too hard to show that there are only countably many of these algebraic numbers, so certainly the unconutable $\mathbb{R}$ contains some non-algebraic or transcendental numbers. Two of which we are already familiar with: $\pi$ and $e$.

Proving numbers, especially $\pi$ and $e$, are not straightforward by any means, and simple statements of them were huge discussion points throughout math's history (see Hilbert's 7th Problem), resulting in major theorems like the Lindemann–Weierstrass theorem. The first number to be discovered to be transcendental was one by Liouville, aptly called Liouville's constant:

and you can find a proof of its transcendence here.

Maybe we'll come back to all this later, but for now, we've done enough and can save it for another day.

There is no "simpler" way to break down the number 2024. But then there's a very natural next question to ask upon learning about prime numbers for the first time: how do we know which numbers are prime? Unfortunately, there is not that a good of an answer to give to anyone, from the elementary school to college students. The only strategy you really have until college is just to try and divide the number in question, and see if there are any factors. You can make small optimizations by checking numbers less than $\sqrt{n}$, and skipping even numbers obviously, but at the end of the day, primes seem kind of random with little to no patterns available. This is one of the first real mysteries young mathematicians encounter.

Though, math has come a long way, and now slowly but surely we are starting to unravel the primes, revealing much about the simple numbers we count with to much more.

Table of Contents

• What is a Prime Number?
• Some Foundational Results
• Don't Take These For Granted
  • Rings and Algebraic Abstractions
  • Integral Domains and Zero Divisors
  • Classifying Elements
• Some Simple Patterns and Useful Facts
• Finding Primes
• Primality Tests
  • Trial Division
  • Wilson's Theorem
  • Fermat's Test
  • Other Tests
  • Different Types of Primes
  • Prime Generating Formula
• Sieves and Filters
  • Probability of Being Prime
• Distributions and the Prime Gap
  • Bertrand's Postulate
• Prime Number Theorem
  • Proof of the Prime Number Theorem
  • Do We Need the Primes at All?
• Consequences of the Prime Number Theorem
  • Analyzing the nth Prime
  • Revisiting Bertrand's Postulate
  • A Quick Application of the PNT
• An Analogue to the Prime Number Theorem in $K[x]$
• Conclusion

What is a Prime Number?

This isn't a trick question. Primes are still exactly what you (probably) think they are:

Definition 1: A positive integer is said to be prime if it is not the product of two smaller natural numbers (excluding 1), i.e. $p \neq ab$ with both $a, b \neq 1$.

We cannot factor primes. This is partly why finding primes is so tricky—we define them not by what they are, but what they are not. This also gives a nice reason for why we should not consider $1$ prime: it just cannot be factored into smaller numbers.

It turns out, there is another, equivalent notion of being a prime number though.

Definition 2: A positive integer (excluding 1) $p$ is prime if when $p \mid ab$, then $p \mid a$ or $p \mid b$.

Here we are using $x \mid y$ to mean $x$ divides $y$. This notion of primality is kind of similar to the definition we are already familiar with: it basically characterizes the idea of prime $p$ being unable to be "split up" into smaller parts that can be distributed between $a$ and $b$. The factor must have contributed wholly to either $a$ or $b$, and did not magically appear by multiplying the two.

Theorem: Definition 1 and Definition 2 are equivalent.

Proof: Although these definitions do look and feel similar, actually arguing the equivalence rigorously is kind of difficult (as it turns out for much of math).

(Definition 2 $\Rightarrow$ Definition 1): Let's say $p \in \mathbb{N}$ satisfies Definition 2. Now suppose for contradiction that $p = ab$ with $a, b \neq 1$. Then, since all numbers divide themselves, $p \mid p = ab$. By the property in Definition 2, $p \mid a$ or $p \mid b$. Let's say $p \mid a$, so $a = pn$ for some $n$. Combining our equalities, we get $p = ab = pnb$. Rearranging, $p(1-nb) = 0$. Since $p \neq 0$, we get $1 = nb$. The only way this can happen is if $n = b = 1$ since they have to be positive integers, giving us our contradiction. Therefore, it must be the case that $p$ also satisfies Definition 1. A similar argument can be used for when $p \mid b$.

(Definition 1 $\Rightarrow$ Definition 2): This is also known as Euclid's Lemma. Say $p$ is prime according to Definition 1. Now suppose that $p \mid ab$, but $p \nmid a$. We'll show that $p \mid b$. The simplest proof I've seen (without presupposing some results we'll later discuss) uses Bézout's Lemma. Since $p \nmid a$ and $p$ has no factors itself, $\gcd(a,p) = 1$. By Bézout's, there are integers, $n$ and $m$ such that

Multiplying both sides by $b$, we get

Clearly, $p \mid npb$. By our assumption, $p \mid ab$ so $p \mid mab$. Therefore, $p \mid npb + mab = b$, giving us our conclusion.

Some Foundational Results

Even with just these definitions and basic results, we can already start to make some headway. The following of which, is probably the most important theorem regarding the integers.

The Fundamental Theorem of Arithmetic: Every integer has a unique* prime factorization.

Proof: We'll prove this in two steps.

(Existence) Showing every integer has a prime factorization is exactly as you'd expect intuitively: we'll break up a number into its factors as many times as possible until we hit its prime factorization. We'll do this by induction.

       Base Case: $2 = 1\cdot 2$ has a trivial prime factorization being prime itself.

       Inductive Step: Say every number greater than 1 and less than $n$ has a prime factorization. If $n$ is prime, we're done. If $n$ is not prime, then it is possible to write $n=ab$ with $1<a\leq b<n$. By the inductive hypothesis,

have prime factorizations. Combining these, we see that

is also a product of primes.

(Uniqueness) Say there are integers that don't have a unique prime factorization. Let

be the least integer with two such factorizations. Therefore, $p_1 \mid q_1 q_2 \cdots q_\ell$. By Definition 2 and Euclid's Lemma, that means $p_1 \mid q_i$ for some $i=1,\cdots,\ell$. Let's suppose (without loss for generality) that $p_1 \mid q_1$. Since $q_1$ is prime, we must have $p_1 = q_1$. That means we can divide both of these out and get that

But then we have found some integer $m < n$ that has two distinct prime factorizations, contradicting the minimality of $n$.

Proving factorization for the negative integers is essentially the same, just noting the extra inclusion of $-1$ as a factor.

*the uniqueness of factorization seems like a proper fact now that we've proved it, but there are a few hiccups we need to address. For one, this factorization is unique up to reordering of the primes; $10 = 2 \cdot 5 = 5 \cdot 2$. This is not that problematic, and can be pretty well understood since we're working with integers and know multiplication is commutative. However, only that we haven't fully explained one small bit away: we can include extra factors of $1$. This seems kind of redundant since we have already said $1$ isn't prime, and it's just kind of obvious to ignore the fact that $2 = 1\cdot 2 = 1\cdot 1 \cdot 2 = \cdots$. But this is an issue if we're going to allow $-1$ in the factorizations of negative numbers. After all, $6 = 2 \cdot 3 = (-2) \cdot (-3)$. So what we can do is combine and push to the front all these extra factors of $1$ and $-1$, and make that sign-determining factor included as a non-prime unique divisor. Let this reinforce that 1 is not a prime number.

Last post, we discussed how the irrational numbers were the key to the real numbers; the irrationals are what give the distinguishing properties of the reals. Even if the names are suggestive, it really should be noted how important the irrationals are to separating the reals and the rationals.

The prime numbers serve a similar purpose to the integers. In fact, because of the above theorem and unique factorization, primes are basically all you need to discuss results about the integers (and that is reflected in many theorems, which we'll mention later).

From this we immediately get another major, fundamental result about the primes.

Theorem: There are infinitely many prime numbers.

Proof (Euclid): Say there were only finitely many primes $p_1, p_2, \cdots, p_n$. Consider the following number:

So all of our primes divide this number. Now consider the number $P = p_1 p_2 \cdots p_n + 1$. We know $P$ has a unique prime factorization by the above. Since all of our primes come from the list $p_1, p_2, \cdots, p_n$, some $p_i \mid P$. But then

So $p_i = 1$, which is impossible given $p_i$ is prime. So in reality, $P$ must have a prime divisor not in our original list, giving us the contradiction that there were only finitely many primes.

Note exactly what this proof is saying: this doesn't say that $p_1 p_2 \cdots p_n + 1$ is prime, it only says that it contains a new prime divisor not found in the list $\{p_1, p_2, \cdots, p_n \}$.

This is a great result, but at the same time, this only widens the mystery and makes finding a pattern—if there is one—all the more unlikely. There is definitely something interesting to be said if there were only finitely many primes, but I think there being so many with such simple-to-state questions about them being unanswered kind of annoying.

Don't Take These For Granted

Before delving further into common tools and patterns found in the primes, it is worth taking a second to note just how nice some of the properties they have—even though everyone likes to make they seem super mysterious.

Rings and Algebraic Abstractions

The integers, in the broad perspective of higher math, is the simplest example of what is known as a ring.

Definition: A ring $(R,+,\times,0,1)$ is a set $R$ given the following structure:

• $(R,+)$ is an abelian group with identity 0. In other words, $(R,+)$ has the properties
  • Commutativity: $a+b = b+a$
  • Associativity: $a + (b + c) = (a + b) + c$
  • Identity: $ a + 0 = 0 + a = a$
  • Inverses: $\forall a \ \exists b \ (a + b = 0)$
• $\times: R\times R\rightarrow R$ has the following properties:
  • Associativity: $a \times (b \times c) = (a \times b) \times c$
  • Identity: $a \times 1 = 1 \times a = a$
  • (Left) Distributivity: $a\times (b+c) = (a\times b) + (a\times c)$
  • (Right) Distributivity: $(b+c) \times a = (b\times a) + (c\times a)$

This definition does look arbitrary and pretty complicated at first glance—and to be honest, it kind of is—but there's no denying its resemblance to the properties of addition and multiplication of the integers. In fact, you could just let $R = \mathbb{Z}$ and interpret $+$ and $\times$ as normal, and you would just get all of our normal arithmetic. As long as you keep those in mind, the definition should become more tractable. The hope is essentially to take some results and intuitions from the integers and extend them to other mathematical objects.

It turns out, with such a broad definition, a lot of algebraic work gets done that has seemingly nothing to do with the "structure of the integers". Some major theorems include:

• The Chinese Remainder Theorem: existence of unique solution to system of modulo remainders
• Eisenstein's Criterion: a sufficient condition on when integer polynomials can be factored
• The classification of finitely generated abelian groups
• Proving the existence of Jordan and Rational Canonical Forms for linear maps

Rings are kind of magical in that way in that they are by far one of the least-obvious tools to employ, but when used correctly, the generality of the theory can lead to a lot. They are unsurprisingly the basis for lots of higher level algebra, especially number theory.

Integral Domains and Zero-Divisors

But not all rings act as nicely as $\mathbb{Z}$. For example, we tend to overlook how convenient it is that $\mathbb{Z}$ is an integral domain. That is,

This is a trick we use when solving for roots of polynomials (which is another ring). If you wanted to solve the quadratic,

That last step is only justified in integral domains. If you're working in the ring, say, the integers modulo 10

then we can very much have zero divisors that break the integral domain property.

If you look back at some of our proofs, like in showing the equivalence of Definitions 1 and 2 of prime numbers, having zero divisors would not allow us to conclude our results.

Classifying Elements

In the same way we classify integers as prime or not, we can do a similar organization of ring elements:

• Unit: $x \in R$ is a unit if it has a multiplicative inverse i.e. $\exists y \ \textrm{s.t.} \ xy=1$
• Irreducible: A (non-unit) $x \in R$ is irreducible if it cannot be factored into non-units
• Prime: A (non-unit) $x \in R$ is prime if when $x \mid ab$, then $x \mid a$ or $x \mid b$

This is partly why I said there were 2 definitions of prime numbers at the top of this post: they are truly separate definitions of two different types of elements in a general ring. It just happens that the notion of being prime and irreducible are the same in the integers. Even though we think of these notions as intertwined by our intuitions of the prime, they can turn out separate.

If every element in the ring (except 0) is a unit, then we have a special type of ring called a field, which give us especially nice properties that we see most commonnly used for vector spaces. And although we haven't mentioned them yet by name, we've run into units already in the form of $1$ and $-1$: they're the reason we need to clarify the Fundamental Theorem of Arithmetic, since everything is a multiple of a unit and made it unclear if we actually had unique prime factorization.

In some rings, though, we might genuinely lose unique factorization. Consider the ring

Then, in this ring, we get that

As such, the integers are known as a unique factorization domain (UFD) since it doesn't suffer from such a problem.

Those are some examples and a peak into abstract algebra. Hopefully it gives some appreciation for why the integers and primes numbers are special in their own way, but we've covered enough ring theory for today and perhaps will revisit it later down the road.

Some Simple Patterns and Uses of the Primes

To build up a bit more familiarity with the primes, its worth perusing some of the simple patterns and ways the primes can be leveraged. Here's a simple one that isn't too hard to notice.

Claim: Every prime is of the form $6n \pm 1$.

Proof: Just check the other cases:

• $6n \pm 2 = 2(6n \pm 1)$ so these can't be prime
• $6n \pm 3 = 3(6n \pm 1)$ so these can't be prime

Another simple pattern.

Claim: For $p > 2$ prime, $24 \mid p^2 - 1$.

Proof: $p^2 - 1 = (p-1)(p+1)$. Now $p$ is odd, so $(p-1)$ and $(p+1)$ are even. Moreover, these are consecutive even numbers, so while one is only divisble by 2 the other is actually divisble by 4. Finally, $p$ being prime means it isn't a multiple of 3, but $(p-1),p,(p+1)$ are 3 consecutive numbers. So again, one of $(p-1)$ and $(p+1)$ is divisible by 3. All together, we have collected that $p^2 - 1 = (p-1)(p+1)$ has at least the factors of $2 \cdot 4 \cdot 3 = 24$. $\blacksquare$

Here's a much more famous one.

Fermat's Little Theorem: For any integer $a$ and $p$ prime, $a^{p} \equiv a \bmod p$.

Proof: Note if $a \equiv 0 \bmod p$, then the result follows. So we just need to show $a^{p-1} \equiv 1 \bmod p$ for $a \neq kp$. Consider the set of numbers $\{a,2a,3a,\cdots,(p-1)a\}$. Note that these are all distinct residues modulo $p$; consider when two residues of this type are equal $ia \equiv ja \bmod p$. So $(i-j)a \equiv 0 \bmod p$. Since $p$ is prime and doesn't divide $a$, by Euclid's Lemma, $p \mid i-j$ i.e. $i \equiv j \bmod p$. So, the set $\{a,2a,3a,\cdots,(p-1)a\}$ is really just a reshuffling of the $p-1$ residues $\{1,2,3,\cdots,p-1\}$ (since there are $p-1$ numbers in the set, and they are all distinct modulo $p$). Therefore,

Since $p \nmid (p-1)!$, we can divide this out with Euclid's Lemma, and we arrive at $a^{p-1} \equiv 1 \bmod p$.

Just as we talked about how ring theory fits into number theory briefly, it might not be surprising to see that its cousin group theory plays a big role as well. The proof of Fermat's Little Theorem simplifies quite a bit with the tools of groups, and much of what we bothered proving above is group theory in disguise.

Proof with Groups: Note that $\{1,2,\cdots,p-1\} = (\mathbb{Z}/p\mathbb{Z})^{\times}$ is a group under multiplication. Consider some $a \in (\mathbb{Z}/p\mathbb{Z})^{\times}$, and its cyclic subgroup $A = \{1,a,a^2,\cdots,a^{k-1}\}$. $|A| = k$, and by Lagrange's Theorem, $|A|$ divides $|(\mathbb{Z}/p\mathbb{Z})^{\times}| = p-1$. So $p-1 = k \ell$ for some $\ell$, and $a^{p-1} = a^{k \ell} = (a^k)^\ell = 1^\ell = 1$. $\blacksquare$

Given how much interest there is in them, it shouldn't be surprising that there are so many number theory results about the primes.

Wilson's Theorem: For $p$ prime, $(p-1)! \equiv -1 \bmod p$.

Proof: Note for all numbers $x \in \{1,2,3,\cdots,p-1\}$, they each have a multiplicative inverse i.e. $\exists y \ \textrm{s.t.} \ xy = 1$. This is best seen by using Bézout's Lemma: since $\gcd(x,p)=1$ as $p$ is prime, there exists $n,m$ such that $nx+mp=1$. Thus this reduces to $nx \equiv 1 \bmod p$, showing the existence of a multiplicative inverse $n$.

Further, note that $1$ and $p-1$ are the only numbers to be their own inverse. To see this, we essentially want to solve $a \equiv a^{-1} \bmod p$ or equivalently $a^2 \equiv 1 \bmod p$. Further, we're solving $a^2 - 1 = (a-1)(a+1) \equiv 0 \bmod p$. So $p \mid (a-1)(a+1)$, and since $p$ is prime, $p \mid (a-1)$ or $p \mid (a+1)$. In the language of residues, this means that $a \equiv \pm 1 \bmod p$ which reduces to either $1$ or $p-1$.

Now back to the theorem at hand, if we consider $(p-1)! = 1 \cdot 2 \cdot 3 \cdots (p-1)$, all the numbers except $1$ and $p-1$ pair off with their inverses, cancelling each other out. Thus $(p-1)! \equiv 1 \cdot (p-1) \equiv -1 \bmod p$.

Here's another result that illustrates in particular how useful unique factorization of the primes is.

Theorem: A finite product of countable sets is countable.

Proof: We'll show this for the case of infinitely countable sets, since 1) the product of finite sets is obvious, and 2) the definition of countability relies on injections from a set to $\mathbb{N}$, so countability of products of sets will follow if we can show $\mathbb{N}^{k}$ is countable. If we have countable sets $A_1, A_2, \cdots, A_k$, we then have injections $f_i : A_i \rightarrow \mathbb{N}$. Then we have the injection $g: A_1 \times \cdots \times A_k \rightarrow \mathbb{N}^k$

So the countability of $A_1 \times \cdots \times A_k$ will follow from the countability of $\mathbb{N}^k$. So consider the injection $F: \mathbb{N}^k \rightarrow \mathbb{N}$.

where $p_i$ is the $i^\textrm{th}$ prime number. By the Uniqueness of Factorization, $F$ is injective. For completeness, $F \circ g : A_1 \times \cdots \times A_k \rightarrow \mathbb{N}$ shows $A_1 \times \cdots \times A_k$ is countable.

This ability to uniquely encode integers by their prime factorization is a nice trick to hold onto, especially for problems that involve all the natural numbers, giving us a path to break the problems up into the more "fundamental", smaller parts.

Euler's Product Formula: $\begin{align} \sum_{n=1}^\infty \frac{1}{n^s} = \prod_{p \ \textrm{prime}} \frac{1}{1-p^{-s}} \end{align}$

Proof: The trick is essentially just the use of the Fundametal Theorem of Arithmetic in a generating function-style way. If you write the sum out with prime factors, the relation becomes clear:

The 4th and 5th equalities follow from being products of geometric series. The key step is the 3rd equality. If you consider any term generated by that infinite product, you're essentially picking out which prime factors the denominator gets; when you consider each factor, you're asking, "How many of this prime factor does my denominator get?" By the Fundamental Theorem of Arithmetic, prime factorizations are unique and thus this process of choosing prime factors means each term $\frac{1}{n^s}$ appears once and only once. At least, that's the idea for why this formula should work. To actually prove the sum and product are the same, you have to give a bit more effort using the absolute convergence of that sum on the left. $\blacksquare$

The sum we discussed above is famously known as the Riemann zeta function $\zeta(s)$, the center of the even more famously open million-dollar Riemann Hypothesis problem. Even without name, you probably have seen individual instances of the zeta function before, as individual instances of the function have been studied for hundreds of years.

• $\zeta(1) = \sum_{n=1}^\infty \frac{1}{n}$ is the divergernt harmonic series
• $\zeta(2) = \sum_{n=1}^\infty \frac{1}{n^2} = \frac{\pi^2}{6}$ is the Basel problem
• $\zeta(3) = \sum_{n=1}^\infty \frac{1}{n^3} \approx 1.2020578$ is Apéry's constant, named after the man who proved it irrational

The most useful of which to keep in mind for most people is probably the harmonic series $\zeta(1)$, as it appears as an incredibly useful example in bounding series and proving contradictions, being one of the simplest divergent series there is. Here's a neat example proving a fact we already know.

Theorem: There are infinitely many primes.

Proof (Euler): Say there were only finitely many primes $p_1,p_2,\cdots,p_n$. Then consider the product

The righthand side is obtained since each $p_i > 1$ i.e. $\frac{1}{p_i} < 1$, thus making each geometric series converge. So this finite product of geometric series converges to some value.

Note, however, that since every integer has a unique prime factorization, we can see that this product is equivalent to

(think Euler's Product Formula). But we know that the harmonic series diverges, giving us our contradiction. So there must be infinitely many primes.

The nice thing about this proof for me is that it begins to illustrate some of the more calculus-y tools we have at our disposal to prove facts about the integers. The field of analytic number theory and using results commonly associated with continuous functions or the real numbers to show facts of the integers always feels a bit surprsing to me, but this proof is a nice introduction to the topic.

Finding Primes

The above results are interesting and actually quite useful, but to be used in practice requires actually knowing a number is prime. If you don't mind just having existence of a solution or proof, like we did in the countability proofs, then you have most of what you want already. But having an actual list of prime numbers, or at least knowing where to find prime numbers is often needed to apply some of our powerful results on primes. If we didn't look for them, cryptography, for one, would completely break down along side all of online security.

Primality Tests

Trial Division

One way we can find primes is by having some criterion $\Phi(n)$ that can tell us if $n$ is prime or not. The simplest test is to just check a number satisfies one of our definitions of being prime. This is probably the original test we all tried at some point: sitting down and just manually checking if a number has any divisors besides itself. Of course, we can do some simple optimizations to avoid needless division.

• If the number is even, it's not prime
• If the number ends in 0 or 5, then it's divisible by 5 and not prime
• We only need to check numbers of the form $6k \pm 1$
• We only need to check the prime divisors up to $\sqrt{n}$

and a whole host of other simple division rules that we can use to simplify this process. Though, as I'm sure we've all felt doing this process, it's not great and there must be a better way. In fact, we should be glad that trial division and factoring directly is hard, as again, much of modern cryptography relies on the assumption that determining prime factorization is difficult.

Wilson's Theorem

You may not have realized it yet, but we've already seen a more sophisticated primality test with Wilson's Theorem. If $n$ is not prime, then there are two cases:

1. $n$ can be written as $n=ab$ with $a \neq b$. Since $a,b<n$, it must be that $a,b$ appear somewhere in the list $\{1,2,\cdots,(n-1)\}$, meaning that $(n-1)! = 1 \cdot 2 \cdots (n-1)$ is a multiple of $n$ i.e. $(n-1)! \equiv 0 \bmod n$.
2. If $n$ cannot be written like in Case 1, then $n = p^2$ for some prime $p$ (if $n$ had 2+ distinct prime factors, we can find $a,b$ to be in Case 1). But $2p < p^2 = n$ for $p>2$. So the numbers $p, 2p$ appear somewhere in $1 \cdot 2 \cdots (n-1)$ i.e. $n = p^2 \mid (n-1)!$ and thus $(n-1)! \equiv 0 \bmod n$.

So if $n$ is composite, $(n-1)! \equiv 0 \bmod n$. The only exception is for $n = 4$ where $(4-1)! = 6 \equiv 2 \bmod 4$, but that's fine since $2 \not\equiv -1 \bmod 4$.

Wilson's Theorem: $n$ is prime if and only if $(n-1)! \equiv -1 \bmod n$.

But this test is not great, as you can imagine it grows exponentially worse to evaluate $(n-1)!$ as $n$ grows larger, even despite advances in multiplication.

Fermat's Test

Inspired by Wilson's Theorem, we might be inclined to see if our other theorems about prime numbers are sufficient tests. How about Fermat's Little Theorem? While we have only shown it's a necessary condition to being prime, there is a formulation for it being sufficient as well.

Fermat's Little Theorem: $p$ is prime if and only if $a^{p-1} \equiv 1 \bmod p$ for all $a \in \{1,2,\cdots,n-1\}$.

Proof of Converse: We'll show the contrapositive. Say $n$ is composite i.e. $n$ has a divisor $\exists d \ \textrm{s.t.} \ d \mid n$. Suppose that $d^{n-1} \equiv 1 \bmod n$ i.e. $d^{n-1} = 1 + kn$ for some $k$, or more helpfully, $d^{n-1} - kn = 1$. $d$ divides $n$, so $d \mid d^{n-1} - kn = 1$, so $d = 1$, giving us our contradiction. Hence if $n$ is composite, we have found a number such that $d^{n-1} \not\equiv 1 \bmod n$. $\blacksquare$

There are a couple reasons why using FLT as a criterion so much better than what Wilson's theorem provides.

1) Exponentiation: Computing multiplication—especially factorials—is slow, but exponentiation is relatively easy due it being repeated multiplication of the same number. For example, here's a common trick using repeated squaring to compute exponents $a^n \bmod p$.

1. Write $n$ in binary i.e. $n = c_0 2^0 + c_1 2^1 + \cdots + c_k 2^k$ where $c_i \in \{0,1 \}$
2. Iteratively compute $a^{2^i} = (a^{2^{i-1}})^2 \bmod p$ i.e. Find $a^{2}$, then $a^{2^2} = (a^2)^2$, then $a^{2^3} = (a^{2^3})^2$ reduced $\textrm{mod} \ p$
3. Calculate $a^n = a^{c_0 2^0 + c_1 2^1 + \cdots + c_k 2^k} = (a^{2^0})^{c_0} \cdot (a^{2^1})^{c_1} \cdots (a^{2^k})^{c_k} \bmod p$

Expanding $n$ into binary caps the size of the total number of multiplications and divisions we compute to be proportional to $\log(n)$ as opposed to $n$, making this fairly efficient to compute $\textrm{mod} \ p$.

2) Multiple Computations: While if we want a definitive proof that a number is prime, we will have to compute $a^{p-1} \bmod p$ for every value of $a$, we can stop our test whenever we want. We can only check some values, even just a single one if we wanted, and expedite the calculations. These sped up tests aren't definitive, but they can give us probable prime numbers.

The second benefit is what leads us to make this prime-testing algorithm:

Fermat Primality Test: To check if $p$ is prime, repeat the following $k$ times:

1. Pick a number uniformly $a \in \{2,\cdots,p-1\}$
2. Compute $a^{p-1} \bmod p$
   • If $a^{p-1} \not\equiv 1 \bmod p$, return "$p$ is composite"

Here, we let $k$ be a type of threshold if we accept $p$ to be prime if we're okay not being 100% certain with our test result to cut down computational cost. It must be reiterated, this really is a probable test if we don't check every number. For example,

but $341 = 11 \cdot 31$. We can have false positives if we rely on this criterion. But if we do another test,

we can find a Fermat witness to show 341 is composite. False positives might seem to preclude us from using this test, but there's a nice property that gives us reason to use this as an efficient test.

Claim: If a composite number $n$ has a Fermat witness $a$ with $\gcd(a,n)=1$, then at least half of the numbers $\{1,2,3,\cdots, n-1\}$ are Fermat witnesses.

Proof: Let's call $S = \{1,2,3,\cdots, n-1\}$, and we'll consider the following 3 sets:

Clearly all these sets are disjoint.

       Claim: $C$ is the trivial Fermat witnesses.

       Proof: Essentially the same proof why divisors are Fermat witnesses. Say $x \in S$ with $\gcd(x,n) = d > 1$. Suppose that $x^{n-1} \equiv 1 \bmod n$, so $x^{n-1} = 1 + kn$. $d \mid x$ and $d \mid n$, so $d \mid x^{n-1} - kn = 1$, contradicting $d>1$. So $x^{n-1} \not\equiv 1 \bmod n$. $\blacksquare$

       Claim: $A,B,C \neq \varnothing$

       Proof: $1 \in A$ so $A\neq \varnothing$. $B \neq \varnothing$ by assumption. $C \neq \varnothing$ since $n$ is composite and its divisors are in $D$.

What we are interested in showing is that $B$, the non-trivial Fermat witnesses, has a lot of elements. What we'll show is that $|B| + |C| \geq \frac{1}{2} S$, and we'll do this in a similar way we proved Fermat's Little Theorem by "permuting" elements with multiplication.

Pick any $b \in B$, and consider the set $Ab = \{ab \bmod n \ : \ a \in A\}$.

       Claim: $Ab \subseteq B$, and $|Ab| = |A|$

       Proof: Pick any $a \in A$, and consider $ab \in Ab$. Since $\gcd(a,n) = \gcd(b,n) = 1$, we get that $\gcd(ab,n) = 1$. Also, note that $(ab)^{n-1} \equiv a^{n-1} b^{n-1} \equiv b^{n-1} \not\equiv 1 \bmod n$. Hence $ab \in B$ for all $a \in A$, i.e. $Ab \subseteq B$. For the second half, note that since $\gcd(b,n) = 1$, we can imply that $(a_1 b \equiv a_2 b \bmod n \Rightarrow a_1 \equiv a_2 \bmod n)$. So the elements of $Ab$ are distinct, and hence $|Ab| = |A|$. $\blacksquare$

So we have that $|A| = |Ab| \leq |B|$. Therefore, since $A \cup B \cup C = \{1,2,\cdots,n-1\}$ and they are distinct, as well as $|A|,|B|,|C| > 0$ by being non-empty, we have

Hence, $|A| < \frac{1}{2} (n-1)$, and so its complement $|A^c| = |B| + |C| > \frac{1}{2}(n-1)$.

It's worth mentioning, like with Fermat's Little Theorem, there is a group theoretic proof of this. If you're familiar with groups, the above looks a lot like we're working with cosets indirectly. The idea is that we look at the group $G = (\mathbb{Z}/n\mathbb{Z})^{\times}$ (since $n$ is composite, we look only at the invertible elements), and note that $A$ above as the set of numbers that pass the test form a subgroup. If there is a non-trivial Fermat witness i.e. $B \neq \varnothing$, then $A$ is a proper subgroup so $|G/A| \geq 2$, and hence $|A|$ can only contain at most half of the elements of $G$ (Lagrange's Theorem), and hence further that $|A|$ contains less than half of the full set $\{1,2,\cdots,n-1\}$.

So if over half the numbers $\{1,2,\cdots,n-1\}$ are Fermat witnesses for composite $n$, we can say that the chance of $n$ being returned prime after $k$ Fermat Primality Tests is usually at most $\left(\frac{1}{2}\right)^k \rightarrow 0$ as we increase $k$. Or better, $n$ is returned accurately composite with probability at least $1 - \left(\frac{1}{2}\right)^k \rightarrow 1$.

Some things to note here:

1. We're using the phrase "...with probability..." loosely here, since either a number is composite or not; primality is deterministic. Probability might be better replaced with "confidence" or "our belief that...". After $k$ tests, we believe with confidence $1 -\left(\frac{1}{2}\right)^k$ that $n$ is prime. If a number is actually prime, it will always pass Fermat's test, no matter how many times you run it, but we don't have that knowledge beforehand when running the tests.
2. Our bound on the number of Fermat witnesses relied on there being at least 1 Fermat witness $a$ with $\gcd(a,n) = 1$. There are unfortunately numbers that are not prime that also do not have Fermat witnesses. These are called Carmichael numbers, and gives us either a need to supplement this test further, or just leave a bit of room for doubt with these strong false positives.

Other Tests

We've gone in-depth on one test, but it's worth mentioning that primality tests are still important and relevant topics in number theory. A more updated version of a Fermat-type test is found in the Miller-Rabin test which uses theories of quadratic reciprocity, and a polynomial-time algorithm was found a little while ago with the AKS primality test.

Different Types of Primes

Another approach that is perhaps the most common one today is instead of checking and calculating every odd number's primality, we check only certain forms of primes. This is usually not an exhaustive method and will skip over some primes, but we get the benefit of finding the primes much quicker with specialized tests. Some types of primes include:

• Mersenne primes are of the form $2^n - 1$ for an integer $n$
• Fermat primes are of the form $2^{2^n} - 1$ for an integer $n$
• $p$ is a Sophie Germain prime if $2p+1$ is also prime.

Mersenne primes have been of particular interest throughout history, with odd facts about them even involving perfect numbers. There of modern interest because we can efficiently test their primality, and as such this class of primes has held many records for being the largest known prime (contrasted with the faster growing Fermat primes, of which only 5 are known).

One of the faster tests available for Mersenne primes appears in a bit of a surprising way.

Lucas-Lehmer Primality Test: Let $M_p = 2^p - 1$ be the Mersenne number we want to test, with $p$ an odd prime*. Define the sequence $s_n = s_{n-1}^2 - 2$ with $s_1 = 4$. Then $M_p$ is prime if and only if $s_{p-1} \equiv 0 \bmod M_p$.

Proof: Many proofs of the correctness of this test are fairly lengthy and tedious. So, I'm going to default to linking the group-theoretic proof which I find more enlightening than the others (a proof, I might add, which uses group theory in the loosest sense, using results which can be learned in 5-10 minutes from scratch).

*we let $p$ be an odd prime since if $p=ab$ is composite, then $2^p - 1$ is composite:

Prime Generating Formula

In fact, it is possible to write a closed formula for such a prime test $\Phi(n)$ using some of our criteria above. Here's one using Wilson's theorem:

This isn't anything we haven't talked about: the floor of that division is essentially just using our inefficient Wilson's theorem based test to return 1 or 0 if $n$ is prime or not, then multiplying by $n$ so that the function actually returns $n$ if it is prime.

But we can do a bit better. We can remove the modular arithmetic, and make it such that instead of $\Phi(n)$ checking if $n$ is prime, make it such that $\Phi(n)$ returns the $n^\textrm{th}$ prime. The below is called Willans' Formula:

As you can probably guess just by looking at it, this isn't all that much more efficient, and the instance of $\frac{(j-1)! + 1}{j}$ should tell you that this is still just Wilson's theorem doing the work under the hood. There's a few more tricks invovled that we'll discuss later, but if you want the whole explanation of the mechanisms behind the formula, see this video.

Sieves and Filters

So far, we've gone through and talked about simultaneously very interesting yet shallow results on prime numbers. We've barely scratched the surface in just how important primes are, as we've mostly stuck to readily calculable, number theory results, even of which we've only touched a handful. Even in this post, I've tried avoiding group theory for the sake of accessibility. But I hope all so far has illuminated that we shouldn't shy away from the mysteries of the primes, for there is much we can say about them.

But that's isn't to say that these problems and questions are easily solved (if they can be solved at all). I mean, just take a look at some of our previous prime tests. You would never come up with some of these tests just by investigating numbers—I wouldn't have. They all managed to be byproducts of more dense, abstract results that hide their applications unless one to be studying the right things at the right time. Even now, the Lucas-Lehmer test's group theory proof is merely a verification of correctness; the proof I like and understand best gives no insight into how one might have actually discovered that test for Mersenee primes.

So, let's take a step back in our quest to discover more prime numbers, and begin from the very start. How would you, say in elementary or middle school, try to test if a number is prime?

It's exactly as we suggested first: let's just try and divide the number and see if we can factorize it; we take a big number and see if we can reduce it to a product of smaller numbers. But what if we instead did the reverse? What if instead of checking big numbers, why don't we instead build up from smaller numbers, and eliminate composite numbers that way? We know that 2 is prime, but anything that is a multiple of 2 definitely isn't prime. Same with 3. 4 is a multiple of 2, so we can skip that.

This inspires a nice little algorithm to filter primes from a list:

Sieve of Eratosthenes: To find the prime numbers less than $n$, do the following:

1. Keep track of 3 different list of numbers: prime, composite, unmarked. Everything (except 1) starts as unmarked.
2. Find the least unmarked number $p$.
3. Add every multiple of $p$ less than $n$ to the composite list, and remove them from unmarked.
4. Add $p$ to prime and remove it from unmarked.
5. Repeat Steps 2–4 until unmarked is empty.

This can sound more complicated than it actually is, but it really is just the reverse idea of our trial divsion primality check. If a number $p$ is not a multiple of some prime less than it, it must be prime. We then can use that prime $p$ to find its multiples $pn$ and eliminate those as prime candidates. So first we find 2, and mark it prime, and then eliminate all the even numbers. The next number we find is 3, and eliminate all the multiples of 3. We eliminated 4 earlier with the evens, but then find 5 and remove the multiples of 5. We eliminated 6 with the evens (and again with the 3s), but get the chance to add 7 and remove the multiples of 7. (It turns out to eliminate all the composite numbers less than 100, we only need to eliminate all the multiples of $2,3,5,7$ since $11^2 > 100$)

Using sieves is a pretty common idea in number theory and combinatorics, since they give a concrete method to count, or at the very least give an upper bound for sizes of sets. They don't necessarily need to count primes like we did before, since they just need to be able to "sift out" or "filter" the elements we don't care, and this forms the basis for many proofs. But we'll come back to this point later.

For now, what the old Sieve of Eratosthenes should help visualize is why it's so hard to find primes. Only 25 numbers between 1–100 are prime. Only 168 are prime between 1–1000. As we increase the amount of numbers we look at, it becomes increasingly harder for a given $n$ to be prime; $n$ has to be able to dodge being a multiple of so many previous primes before it. Even just 2 being a prime reduces our potential candidate list in half. 3 being prime removes another sixth of our candidates, and so on. Primes just naturally get rarer, and so it shouldn't be that strange searching for larger and larger primes become incerasingly challenging. Can we quantify this rarity in primes?

Probability of Being Prime

One intuitive notion I think the Sieve of Eratosthenes shows really well is the idea of how divisibility translates to "probability". What is the probability a number is even? Well, every other number is divisible by 2, and we eliminate those in our sieve. How about being divisble by 3? Well, that seems like that should be with probability $\frac{1}{3}$. What about being divisble by 7789, for large enough $n$, you'd expect $\frac{1}{7789}$ to be divisible.

This thought can give us a way to think about the probability of being prime too. What does it mean for $n$ to be prime? Well, as our sieve describes, $n$ being prime should be equivalent to $n$ not being divisible by any prime less than it. We'll let $f(n) = \mathbb{P}(n \ \textrm{is prime})$, so we want to find

If the probability of being divisible by $p$ is $\frac{1}{p}$, then being not divisible by $p$ should have probability $1-\frac{1}{p}$.

This estimate is nice enough, but what we care more about this function is how it changes; can we relate the primality of $n$ to $n+1$? Well, since our product only includes primes, the only way the product changes is if $n$ is prime or not. If $n$ is prime, well that's a new prime to consider $n+1$ not being divisible by, and if $n$ is not prime, then that doesn't affect our product (ignore the fact that $n \nmid n+1$, we're only looking for a heuristic argument right now).

Rearranging a little bit, we get that

If $f(n)$ is a nice enough function, we can approximate further that

We can solve this differentiable equation, giving us

Tying things together, note by our Euler Product Formula for the Riemann zeta function, we could also have done this rough derivation with

Perhaps to make it clearer, if the probability of being prime is roughly $\frac{1}{\ln(n)}$, then there are roughly $\pi(n) \approx \frac{n}{\ln(n)}$ many primes less than $n$ ($\pi(n)$ is often used to denote the prime-counting function, and give the number of primes less than $n$).

Although we only used rough estimates, we ended up with a result that formalizes our intuition that it becomes more difficult and unlikely for larger numbers to be prime. But also note that by our treatment of what "the probability of being prime" means,

all numbers less than $n$ should have the same probability of not having a prime divisor less than $p<n$—after all, we treated our probability of not having a divisor $p$ as $\frac{1}{p}$ which only depends on $p$, not $n$, formalizing the idea of "dodging" being in the fraction of numbers that are a multiple of $p$. So a more accurate statement is that the fraction of numbers that are prime from 1–$n$ is proportional to $\frac{1}{\ln(n)}$. If this is unclear why we specify this, just imagine we include prime divisors $p>n$. Even though it's impossible for $p \mid n$ with $p>n$, our intuition of not being in the fraction of numbers that are mulitiples of $p$ would have us include an additional factor of $1-\frac{1}{p}$ in our product. This obviously lowers our probability just by considering a wider range of numbers that "miss" dividng $n$, but in actuality have no impact on the actual prime-ness of $n$ (what we've stumbled upon for formalizing probability is known as natural density).

With this in mind, let's keep moving forward where we'll later do this with the rigor it deserves.

It should be noted that weirdly enough, as nice as this intuitive argument is, it is actually just techincally, barely wrong. In reality, the asymptotics we've been studying actually look like

where $\gamma$ is the Euler–Mascheroni constant. I don't have a good answer why this is the case and what our argument misses besides the fact that we worked with estimates and assumed independence of of divisibility.

Distributions and the Prime Number Theorem

With our first little idea in the works that primes are approximately logarithmically distributed, we now look for some more precise results on how the primes are distributed.

Bertrand's Postulate

Perhaps one of the earliest and most well-known distribution results we have is Bertrand's Postulate. Due to the logarithmic nature we've found, it shouldn't be surprising that this result involves a multiplicative range instead of an arithmetic one:

Bertrand's Postulate: $\forall n \geq 2, \ \exists p \ \textrm{prime s.t.} \ n<p<2n$

This doesn't necessarily say this is the smallest guaranteed interval to find a prime, nor does it say this interval generates a unique prime (for example, 7 satisfies the Postulate for both $n=4,5$). Nonetheless, it characterizes that feeling of having to expand our search for primes to greater ranges of integers. But what about the range $[n,2n]$ necessitates a prime being there?

Proof (Erdős): Before we get into the proof, we are first going to want to play around with what will become our key guide towards the proof. Consider

If you try factoring this for a few values of $n$, something apparent appears. For example, here's the factorization for $n=22$.

It seems that $2n \choose n$ contains all prime factors between $n$ and $2n$ once, and no prime factor greater $2n$. We're going to try and use that fact to bound $2n \choose n$ appropriately to show that there needs to be at least one prime factor in $[n,2n]$ for it to be not too small and not too large. If we're going to talk about its factorization via how large it is, let's start to find some inequalities.

Lemma 1: $ \ \displaystyle \frac{4^n}{2n} \leq {2n \choose n}$

Proof: $4^n = (1+1)^{2n} = \sum_{k=0}^{2n} {2n \choose k} = 2 + \sum_{k=1}^{2n-1} {2n \choose k} \leq 2n \cdot {2n \choose n}$. The last inequality comes from the middle binomial coefficient being the greatest. $\ \blacksquare$

Next, it seems natural we should understand the prime factorization of factorials. Let $R_p(n)$ be the largest exponent such that for a prime $p$, $p^R \mid n$ (note for any prime $p$; it might be that $R_p(n) = 0$). We are interested in $R_p({2n \choose n})$.

Lemma 2: For any prime $p$ and $R = R_p({2n \choose n})$, $ \ \displaystyle p^R \leq 2n$.

Proof: There's a nice formula (due to Legendre) that gives

If this is not obvious, the idea is that you count how many multiples of $p,p^2,p^3,\cdots$ appear from $1$ to $n$, and each occurrence contributes 1 to the exponent $R$ (it's only one contribution even for $p^2,p^3,\cdots$ since we already count each of the factors previously i.e. $p^2$ is a multiple of $p$ as well). Further, if $p^i > n$, then the $\left\lfloor \frac{n}{p^i} \right\rfloor$ reduces to 0 so we don't count powers of primes that don't divide $n!$.

Back to the problem at hand, since we're considering the division of two factorials, $R = R_p({2n \choose n})$ gains contributions from powers of $p$ in $(2n)!$ and loses powers from the division of powers of $p$ in $(n!)^2$. Hence,

It's not hard to see (if you want, just graph it) that the only values of this final sum's terms are

(This essentially checks if doubling $n$ to $2n$ has room for another multiple of $p^i$ beyond the expected contributions in the ranges of length $n$). Alternatively, notice that each term is an integer, and

Further, this term is all 0 when $p^i > 2n$ i.e. $i > \log_p(2n)$ by the floor function. So, $R \leq \log_p(2n)$ and thus $p^R \leq p^{\log_p(2n)} = 2n$. $ \ \blacksquare$

Already, we can deduce quite a lot of information about the number of primes that divide $2n \choose n$. Using our two lemmas, let $p_1,\cdots,p_\ell$ be the primes dividing $2n \choose n$. Then

Taking logarithms, we get that $\ell \geq \frac{2n \log(2)}{\log(2n)} - 1$, giving a lower bound on how many primes must divide $2n \choose n$. Also, it is clear that no prime dividing ${2n \choose n}$ can be greater than $2n$ (check Legendre's formula if needed) and so in particular this gives us an upper bound on $\pi(2n)$ the number of primes less than or equal to $2n$.

Lemma 3: If $p$ is an odd prime and $\frac{2n}{3} < p \leq n$, then $R_p({2n \choose n}) = 0$.

Proof: $(2n)!$ contains two factors of $p$ via $p$ and $2p$. $(n!)^2$ also contains two factors of $p$. These factors cancel in ${2n \choose n}$. The bound ensures that $3p > 2n$ cannot appear in the numerator to contribute more factors (and it being odd makes it so that $2p$ doesn't contribute more than expected). $\blacksquare$

It's also easy to see by similar reasoning that for prime $p$:

• if $n<p\leq 2n$, then $R_p({2n \choose n}) = 1$
• if $p > 2n$, then $R_p({2n \choose n}) = 0$

These will be useful later. Finally, we need one more simpler bound before proving the Postulate. We'll introduce one last piece of notation for the primorial.

It's essentially $n!$ but only looking at primes less than $n$ as opposed to all integers less than $n$.

Lemma 4: For $n\geq 1$, $n\# < 4^n$.

Proof: We'll do this by induction, checking even and odd cases (the base cases are easy to check). Supoose this holds for up to $n-1$.

       (Even) Say $n$ is even i.e. $n=2k$. Since $2k$ is composite, then by the inductive hypothesis,

       (Odd) Say $n$ is odd i.e. $n=2k+1$, and consider

We get the inequality since the binomial coefficient is an integer, with all the prime factors between $k+2$ and $2k+1$ dividing it since they divide the numerator but not the denominator. These relations get us close to allowing use of our inductive hypothesis.

We get the last inequality by noting that $\sum_{i=0}^{2k+1} {2k+1 \choose i} = 2^{2k+1}$ and ${2k+1 \choose k} = {2k+1 \choose k+1}$ since $2k+1$ is odd. So ${2k+1 \choose k} < \frac{1}{2} 2^{2k+1} = 2^{2k}$. $\ \blacksquare$

Proof of Bertrand's Postulate: Say this is false, and let $n \geq 3$ be such a counterexample with no prime $p$ in the interval $n < p < 2n$. Note by our lemmas, there are no prime factors $p$ of $2n \choose n$ such that:

• $2n < p$ as such a factor must divide $(2n)!$
• $p = 2n$ since $2n$ is even
• $n<p<2n$ by assumption
• $\frac{2n}{3} < p \leq n$ by Lemma 3

So all prime factors of $2n \choose n$ must be $p \leq \frac{2n}{3}$. So we can simplify the prime factorization to be

But also, if $p > \sqrt{2n}$, then ${2n \choose n}$ has at most one factor of $p$ (since $p^2 > 2n$ and can't divide $(2n)!$ i.e. see Lemma 2). So, if we can further simplify our prime factorization for that interval

Using Lemma 2, we can bound $p^R \leq 2n$. Also, obviously, there can only be at most $\sqrt{2n} - 1$ distinct prime numbers less than or equal to $\sqrt{2n}$ (there are only that many integers less than $\sqrt{2n}$ excluding 1, after all). Starting with Lemma 1 and ending with Lemma 4, we can combine all of this into one final inequality.

Rearranging a little bit more, we end with the inequality

This inequality fails for $n>427$, giving us our contradiction that we needed. Hence, we have shown Bertrand's Postulate is true for $n>427$.

To show it for $n\leq 426$, this is small enough that we don't need to do anything hard; we can just find a set of primes that work for all desired intervals. Consider the list

This sequence of primes are chosen to be such that the next prime in the sequence is the largest prime less than twice the previous. We don't need to know all the primes less than 427 and check every case of $(n,2n)$, since these give us all the examples we need!

Despite its simplicity, the impact of Bertrand's Postulate can be found in many results. For example, it gives a simple bound on prime gaps: if you have a prime $p$, you know you only have to search up to $2p$ before you find your next prime. However, these types of bounds are rarely obvious.

Legendre's Conjecture (Unsolved): $\forall n \geq 2, \ \exists p \ \textrm{prime s.t.} \ n^2<p<(n+1)^2$.

Note that $(n+1)^2 \leq 2n^2$ for $n>2$, so we can't solve this using Bertrand's postulate. This is not to say there have not been improvements, however:

Nagura (1952): $\forall n \geq 25, \ \exists p \ \textrm{prime s.t.} \ n<p<\frac{6}{5}n$

Getting hard, concrete bounds like this are rarely tangible, as the insight necessary to get an object to analyze is quite demanding. Even for the above proof, it is not totally obvious that the point of investigation begins at binomial coefficients. With the major result of the Prime Number Theorem at the end, we will obtain asymptotic bounds to shrink this interval as much as we want.

Here's another interesting consequence of Bertrand's postulate:

Claim: The set $\{1,2,3,\cdots,2n\}$ can be partitioned into pairs $\{a_1,b_1\}$, $\{a_2,b_2\}$, $\cdots$, $\{a_n,b_n\}$ so that for all pairs $a_i + b_i$ is prime.

Proof: We use induction.

       Base Case: $n = 1$ works as $1+2=3$ is prime.

       Inductive Step: Say we can find such a partitioning for any set of even size less than $2n$. By Bertrand's Postulate, there is a prime $2n < p < 4n$, i.e. there is a prime of the form $p = 2n + m$ for $1 \leq m < 2n$ (note $m$ must be odd). We can now pair off some numbers in the set by

with every pair summing to $p$. These pairs take care of the numbers $\{m,m+1,\cdots,2n\}$. The remaining set of $\{1,2,\cdots,m-1\}$ can be partitioned by the inductive hypothesis (since $m-1 < 2n$ is even). $\ \blacksquare$

Also, we can chain intervals together if we want to find lots of distinct primes. I.e. if we look in the intervals $[1,2]$, $[2,4]$, $[4,8]$, etc., because each interval picks up exactly where the last one left off, we can find as many distinct primes as intervals we're willing to check. In fact, we've already briefly seen this trick used once in Willan's formula:

The $2^n$ ensures we find at least $n$ distinct primes as we check the intervals $[1,2]$, $[2,4]$, $[4,8]$, etc. This guarantee of finding new primes in these intervals lends itself quite nicely to prime generation, even if in unexpected ways:

Wright's Prime Generating Formula: There's a value of $\alpha \in (1,2)$ such that

are all prime. (The value Wright originally gave was $\alpha \approx 1.929$)

Proof: Just to be clear, what we're exactly saying is that there's a value $\alpha$ such that for the recurrence

we have $\left\lfloor g_n \right\rfloor$ is prime. The idea is that we want to bound $p_n < g_n < p_n + 1$, which leads naturally to trying to find our constant $\alpha$ in between the repeated logarithms of $p_n$ and $p_n + 1$. We can ensure such a constant exists by picking primes that are spaced far enough away—hence Bertrand's Postulate—that will help create a sequence that converges to $\alpha$.

So, first we choose a sequence of primes $p_n$ such that

• $p_1 = 2$ or $3$,
• $2^{p_n} < p_{n+1} < p_{n+1} + 1 < 2^{p_{n} + 1}$

We know we can choose such a sequence of primes thanks to Bertrand's Postulate. These will be the primes our magic sequence $\lfloor g_n \rfloor$ will generate. Taking the $\log_2$ (from here on, $\log$ will refer to $\log_2$) of this inequality, we find that

Now we define two more sequences:

where $\log^{(n)} (x) = \overset{\textrm{n times}}{\overline{\log(\log(\cdots \log}}(x) \cdots ))$ i.e. the $n$-fold composition of $\log$ with itself. By taking further $\log$ of our inequalities $n$ times, we get that

$u_{n}$ is a bounded (by $v_1$) monotonically increasing sequence, and so it converges $u_n \rightarrow \alpha$. So we obtain that $u_n < \alpha < v_n$ ($v_n$ might converge to something else, but we know at least that all terms of $v_n$ must be greater than $\alpha$ or else it'd contradict our inequalities above).

By letting $g_0 = \alpha$, and $g_{n+1} = 2^{g_n}$ as described, and repeatedly raising our inequalities to the power 2, we then get

Therefore, $\lfloor g_n \rfloor = p_n$ and follows our sequence of primes. $\ \blacksquare$

Actually computing such an $\alpha$ is not easy though. We've shown the existence of such a number as the limit of a sequence, but finding that limit is quite hard. Even calculating approximations of $\alpha$ via $u_n$ is hard, since we need to already have a sequence of primes $p_n$ predetermined (so unfortunately, this isn't that useful for actually generating more primes, but at best generating numbers close to primes outside of our seqeuence).

But because it's sequence dependent, there are many such constants that serve our purpose. For instance, if we let

we can find that it's corresponding $\alpha$ (which is also maximal) is about

If we instead choose a different sequence, like

we can find a different approximation:

Also, if you like these power-towers and primes, we're quite lucky Bertrand's postulate finds a prime in $n < p < 2n$, as that also immediately means we can find primes in $n < p < 3n$, or in any range $n < p < Bn$. So really, we can generalize our theorem and proof quite easily to instead say:

Claim: For all $B \geq 2$, there's a value of $\alpha$ such that

are all prime.

Prime Number Theorem

Bertrand's Postulate gave us a nice first step towards making precise how primes are distributed by giving us an upper bound on how frequently primes arise. Bringing things back around, Bertrand's Postulate also suggests that the number of primes grows at least logarithmically. Again, consider the intervals

We know there is at least 1 prime in each of these disjoint intervals, so we know that $p_n \leq 2^n$ where $p_n$ is the nth prime. Thus, if we let $\pi(x)$ be the prime-counting function from before, then we have that $\pi(x) \geq \log_2(x)$. Of course, this is not that close to our previous estimate with $\pi(x) \approx \frac{x}{\ln(x)}$, but it is an initial, definitive start beyond our heuristics to precisely describe the distribution of primes.

We can, however, bound $\pi(x)$ with just Bertrand's postulate in a promising way. From here on, $\log$ refers to the natural logarithm $\ln$. Recall from before that we found that

This, however, suffers from the fact that our proof only makes sense for integers $n$. If we want to extend this to all real numbers $x\geq 2$, we will need to work around it a bit more. Let $n=\lfloor x/2 \rfloor$, so $2n \leq x < 2n+2$ and in particular $x - 2 < 2n$.

The function $\frac{\log(x)}{x}$ is maximized at $x=e$ with value $\frac{1}{e}$. Also, for $x\geq 4$, $x-2 \geq \frac{x}{2}$

So we have bounded from below $\pi(x)$ by $\frac{x}{\log(x)}$ up to a positive constant (one can just check that this also holds for $4 > x\geq 2$). This bound is nowhere near optimal, but I tried giving a constant that was most readily calculable with less additional verification needed; we could give better bounds but have to check for smaller values manually, and or have seemingly arbitrary constants pop out of nowhere to verify later. We can also give a similar upper bound, but it will take a bit more work. First notice that

Again taking logarithms, we have

This is close to what we want, in that $\pi(2n) - \pi(n)$ counts the number of primes in the interval $(n,2n]$ of length $n$, but we want the number of primes in the interval $(0,n]$. And as before, to make this work for real numbers, our bounds have to a be a bit more careful. Let $n = \lfloor x \rfloor$ i.e. $n \leq x < n+1$.

The only main point of interest is to recall that for $x\geq 3$, we have $n \geq x - 1 \geq \sqrt{x}$. We can now chain together exponentially decreasing intervals with a suitable cut-off to bound $\pi(x)$. Let $\ell$ be the greatest integer such that $x/2^\ell > \sqrt{x}$.

where $c$ is the constant found before. Now for all $k \leq \ell$, we have $x/2^k > \sqrt{x}$ i.e. $\log\left(x/2^k \right) > \frac{1}{2} \log(x)$.

Recalling by construction that $x/2^{\ell+1} \leq \sqrt{x}$, and the trivial bound $\pi\left(x/2^{\ell+1}\right) \leq x/2^{\ell+1}$. We also used that $\log(x) \leq \sqrt{x}$ i.e. $\sqrt{x} \leq x/\log(x)$. All together, we have

This gives some real, concrete motivation to pursue the Prime Number Theorem.

Proof of the Prime Number Theorem

For the sake of time, we give up some elementary-ness for concision in the following proof of the Prime Number Theorem. In the spirit of analytic number theory as we tasted before with Euler's proof of the infinitude of primes, we will borrow some (in the grand scheme simple) tools from complex analysis, investigating the properties of some specific functions to get at the result we want. The original paper can be found here. If you aren't already familiar with holomorphic functions, types of convergence, complex integrals, and the like, here's a good time to catch up on it. Not much is totally needed, but we need to be careful of the details.

The ultimate idea of the proof is to analyze analytic properties of some functions to deduce results about their asymptotics.

Prime Number Theorem: $\pi(x) \sim \frac{x}{\log(x)}$, that is, $\lim_{x \to \infty} \frac{\pi(x)}{x/\log(x)} = 1$.

From here on out, to simplify notation, $p$ will always refer to indexing primes, and as often as possible I'll try to keep $s \in \mathbb{C}$ reserved for complex variables and $x \in \mathbb{R}$ for real ones. The three functions of interest to us for this proof are:

• $\displaystyle \zeta(s) = \sum_{n=1}^{\infty} \frac{1}{n^s}$ (the Riemann Zeta Function)
• $\displaystyle \vartheta(x) = \sum_{p \leq x} \log p$ (the first Chebyshev function)
• $\displaystyle \Phi(s) = \sum_{p} \frac{\log p}{p^s}$ (I couldn't find a name for this one)

The main goal of this proof is to prove the following equivalent statement.

Theorem: The Prime Number Theorem is equivalent to $\vartheta(x) \sim x$.

This shouldn't be too surprising given that $π(x) = \sum_{p \leq x} 1$, so the logarithmic factor might be expected.

Proof: We will appropriately bound the limits by each other to determine equality. First, note

Dividing by $x$,

To get the corresponding lower bound, let $\epsilon \in (0,1)$.

Again dividing by $x$,

Combining everything,

Since we can take $\epsilon \rightarrow 0$ and the term $x^{-\epsilon} \log(x) \rightarrow 0$ as $x \rightarrow \infty$, we get the sandwiched bounds (this can be explicitly written assuming either limit exists and taking limits and limit inferior/superior). $\blacksquare$

We'll consider this equivalent statement with $\vartheta(x)$ since the "jumps" at each prime $p$ of $\log(p)$ instead of 1 as in $\pi(x)$ makes it slightly nicer to work with. We have already met the Riemann zeta function before and seen how it relates to primes directly with its product form. Let's now justify it a bit more concretely.

Proposition (I): $\zeta(s)$ absolutely converges for $\Re(s)>1$.

Proof: $\displaystyle \sum_{n=1}^{\infty} \left| \frac{1}{n^s} \right| = \sum_{n=1}^{\infty} \left| \frac{1}{n^{\Re(s)}} \right| \left| \frac{1}{e^{i \Im(s) \log(n)}} \right| = \sum_{n=1}^{\infty} \frac{1}{n^{\Re(s)}}$. This converges since $\Re(s)>1$. $\blacksquare$

Corollary (Euler's Product Form): $\zeta(s) = \prod_{p}\left(1-p^{-s}\right)^{-1}$ for $\Re(s) > 1$.

Proof: We derived the above earlier assuming we could rearrange terms however we wanted, and absolute convergence gives us that prerequisite to do it safely (see Riemann's rearrangment theorem). $\blacksquare$

Proposition (II): $\zeta(s) - \frac{1}{s-1}$ extends holomorphically to $\Re(s)>0$.

Proof: The idea of looking at this function is to note that the right term $\frac{1}{s-1}$ is the integral of the infinite sum that defines $\zeta(s)$. While neither may be defined for $\Re(s) > 0$ individually, their difference will cancel out just right to give us the extension we want. If this isn't clear, think about how the harmonic series and its related integral both diverge, but their difference gives rise to the Euler-Mascheroni Constant. So, with this in mind, we use a clever rewriting of this difference, and bound it appropriately to show it converges uniformly (on compact sets) via the Weierstrass M-test, hence showing such a limiting function exists and by uniform convergence is also holomorphic. First, note

with the Fundamental Theorem of Calculus, we can further rewrite the integral terms:

Now let's see if we can bound this to apply the M-test:

We evaluate the modulus as we did before, and the maximum is given since the fraction is decreasing in $u$. Now for any compact i.e. closed and bounded set $K \subseteq \{ s \in \mathbb{C} \ : \ \Re(s) > 0 \}$, since $|s|$ and $\Re(s)$ are continuous, they achieve their bounds so there are constants $C = \max \{ |s| \ : \ s \in K \}$ and $\delta = \min \{ \Re(s) \ : \ s \in K \}$. Since $\delta > 0$, the sum of $M_n$ converge and so by the M-test get that our sum defining the function converges uniformly and absolutely giving us existence of such a function. Since it also converges uniformly on compacts in the region $\Re(s)>0$ and the finite sums $f_k(s) = \sum_{n=1}^{k} \int_n^{n+1} \left( \frac{1}{n^s} - \frac{1}{x^s} \right) \mathrm{d}x$ are holomorphic, the limit $\zeta(s) - \frac{1}{s-1}$ is holomorphic as well. $\blacksquare$

Proposition (III): $\vartheta(x) = O(x)$ i.e. there are constants $C,x_0$ such that for all $x\geq x_0$, we have $\left| \vartheta(x) \right| \leq C \left| x \right| $.

Proof: Using similar methods from before,

Hence $\vartheta(2n) - \vartheta(n) \leq 2n \log(2)$. We can now extend this to all $x$ instead of just even $x = 2n$ also in a similar method from before. Let $n = \lfloor x/2 \rfloor$:

The first few inequalties are just writing out the sums and checking indices correctly. We obtain the second-to-last inequality by noting by choice $2n \leq x$ and $\log(x) \leq x$. Let $C = 2+\log(2)$. Now let $r$ be the least integer such that $1>x/2^r$.

This works for all $x>1$, hence showing $\vartheta(x) = O(x)$. (Note: this bound is quite bad, but we used it to get concrete values to see clearly $\vartheta(x) = O(x) $. In particular, if you use the fact that $\log(x) = o(x)$, you can make $C$ as close to $\log(2)$ as you want for sufficiently large $x \geq x_0$, and this is what the original proof does) $\blacksquare$

Proposition (IV): For $\Re(s)\geq 1$, $\zeta(s) \neq 0$ and $\Phi(s) - \frac{1}{s-1}$ is holomorphic.

Proof: By Proposition (I) and the Euler Product Form, we can show $\zeta(s) \neq 0$. It might seem intuitively obvious that $\zeta(s) \neq 0$ since $1-p^{-s} \neq 0$ and so we are taking the product of non-zero terms, but since our product is infinite, we have to check that the product doesn't shrink too fast. For example, even though $\frac{n}{n+1} \neq 0$, the product

Note that $\left(1-p^{-s} \right)^{-1} - 1 = \left( p^s - 1 \right)^{-1}$ is absolutely convergent for $\Re(s) > 1$ since $\left| \left( p^s - 1 \right)^{-1} \right| = \left| p^{-s} / \left( 1 - p^{-s} \right) \right| = p^{-\Re(s)} / \left| 1 - p^{-s} \right| \leq p^{-\Re(s)}/2$. So in particular, this means that $\left(1-p^{-s} \right)^{-1} \rightarrow 1$ as we run through increasing primes. Let $N$ be the integer such that $\left(1-p^{-s} \right)^{-1} \in B(1,1/2)$ for $p \geq N$, so we can write the product

Since the second product's terms are bounded away from 0, we can use the (standard branch of the) complex logarithm $\log(z) = \ln(|z|) + i\arg(z)$.

getting the last inequality via the Taylor series for $\log(z+1)$. We showed above this last sum converges, so the $\log$ sum converges, and in particular by the continuity of $e^x$ the product converges:

Since the individual terms $\frac{1}{1-p^{-s}}$ are non-zero, we get on the whole that for $\Re(s) > 1$,

To start on the second part, we build off a bit of what we just talked about noting that $\log(\zeta(s)) = -\sum_p \log\left(1-p^{-s} \right)$. Differentiating,

So in particular,

The sum on the right converges for $\Re(s) > \frac{1}{2}$. So the only possibilities for problematic poles for $\Phi(s)$ come from the term $-\zeta'(s)/\zeta(s)$. By Proposition (II), we can extend $\zeta(s)$ to a meromorphic function on $\Re(s)>0$ with a simple pole at $s=1$. Thus, $-\zeta'(s)/\zeta(s)$ has simple poles at $s=1$ and the zeroes of $\zeta(s)$. We already showed $\zeta(s) \neq 0$ for $\Re(s) > 1$, so if we want to show there are no problems for $\Phi(s)$, we just need to show that there are no zeroes for $\zeta(s)$ on $\Re(s)=1$. We will do so by contradiction by considering multiple zeroes on the line $\Re(s)=1$.

Suppose that $\zeta(s)$ has a zero of order $\mu$ at $s=1+i\alpha$ and a zero of order $\nu$ at $s=1+2i\alpha$ for $\alpha \in \mathbb{R} \setminus \{ 0 \}$. Since $\Phi$ has a simple pole at $s=1$, it has associated residue $1$ (note the sign is flipped since we have $-\zeta'(s)/\zeta(s)$ instead of $\zeta'(s)/\zeta(s)$), and for a zero of $\zeta(s)$ of order $k$ it also has residue $-k$.

We include both zeroes $1 \pm i\alpha$ since $\zeta(\bar{s}) = \overline{\zeta(s)}$. Now, consider the sum

The first equality is by direct computation, and we get the inequality since sums of conjugates are real and hence their 4th powers are non-negative real numbers. Now mulitplying through by $\epsilon>0$ and taking the limit $\epsilon \to 0$:

Since the only pole of $\zeta(s)$ for $\Re(s)>0$ is at $s=1$, $1+\epsilon+ri\alpha$ must be zeroes of positive order i.e. $\mu, \nu \geq 0$. Therefore, $2\nu + 8\mu \leq 6$ can only hold if $\mu=0$ i.e. $\zeta(1+i\alpha) \neq 0$.

Hence $\Phi(s)$ only has the simple pole at $s=1$ for $\Re(s) \geq 1$, and hence $\Phi(s) - \frac{1}{s-1}$ is holomorphic on that half-plane. $\blacksquare$

Next we strengthen our result from (III) to show that $\vartheta(x) \sim x$.

Proposition (V): $\displaystyle \int_1^\infty \frac{\vartheta(x) - x}{x^2} \ \mathrm{d}x$ is a convergent integral.

The idea is that if $\vartheta(x) \approx \lambda x$ for $\lambda > 1$, then the integral would be approximately $\int_1^\infty \frac{\lambda x - x}{x^2} \ \mathrm{d}x = \int_1^\infty (\lambda-1) \frac{1}{x} \ \mathrm{d}x$, which would diverge; the convergence of the integral forces the integrand to decay at the rate we want. We will do this with what Newman calls the Analytic Theorem.

Analytic Theorem. Let $f(t)$ $(t \geq 0)$ be a bounded and locally integrable function and suppose that the function $g(z) = \int_0^{\infty} f(t) e^{-zt} \ \mathrm{d}t$ $(\Re(z) > 0)$ extends holomorphically to $\Re(z) \geq 0$. Then $\int_0^{\infty} f(t) \ \mathrm{d}t$ exists (and equals $g(0)$).

The idea of this theorem is to take the Laplace transform of $f(t)$ analyze its "spectrum". If the spectrum analysis is nice and differentiable, it means that around $s=0$, $g$ can't have singularities that blow up and so make the integral of $f$ blow up; the behavior of $g$ limits the behavior of $f$. I'll leave the proof of the theorem to the original paper, but the idea above is enough (especially if you have seen the Laplace transform before).

Proof of V: The function $\Phi(s)$ looks a lot like $\vartheta(x)$. For $\Re(s)>1$, we can interchange the sum in $\Phi(s)$ with an integral:

Note the differential is with respect to $\mathrm{d}\vartheta(x)$ as opposed to just $\mathrm{d}x$. If you aren't familiar with measure theory and the Lebesgue integral, the idea is that we are "stretching" the x-axis according to $\vartheta(x)$, giving weights to the intervals we're integrating over with respect to $\vartheta(x)$. If $g(x)$ was differentiable, it might be more familiar to write this transformed interval as $\mathrm{d}g(x) = g'(x) \ \mathrm{d}x$. Here are more details on the specific case we're working with. We get the sum equivalence since $\vartheta(x)$ is a step function, so we only evaluate at the step discontinuities at the primes. Most usefully for us, these types of integrals admit a type of integration by parts.

Note since $\vartheta(x) = O(x)$ and $\Re(s)>1$, we have $\left| \vartheta(x)/x^s \right| \leq C/x^{\Re(s)-1} \rightarrow 0$ as $x\to\infty$. With the exact same bounding, we see the integral term is also convergent. We get the last equality substituting $x=e^t$. Now we apply the Analytic Theorem to $f(t) = \vartheta(e^t)e^{-t} - 1$ and $g(z) = \Phi(z+1)/(z+1) - 1/z$. To verify, note that from the above derivation we have

Note by (III), $\vartheta(e^t) = O(e^t)$, so $f(t) = \vartheta(e^t)e^{-t} - 1 = O(1)$ and so is bounded, and only has countably many discontinuities at $\log p$ for primes $p$ where $\vartheta(x)$ has its jump discontinuities at the primes and thus is locally integrable (i.e. integrable on every closed subinterval of $(0,\infty)$). By (IV), $g(z) = \frac{\Phi(z+1)}{z+1} - \frac{1}{z} = \frac{\Phi(z+1) - 1/z - 1}{z+1}$ extends holomorphically to $\Re(z) \geq 0$ since $z+1 \neq 0$ and $\Phi(z+1) - 1/z$ is holomorphic for $\Re(z) \geq 0$. So we use the Analaytic Theorem and conclude that

is a convergent integral. $\blacksquare$

Prime Number Theorem: $\vartheta(x) \sim x$.

Proof: For contradiction, assume that $\lim\limits_{x \to \infty} \vartheta(x)/x \neq 1$. We check two cases. First, say for some $\lambda > 1$, $\limsup\limits_{x\rightarrow \infty} \vartheta(x)/x \geq \lambda$ i.e. there are arbitrarily large $x_n$ such that $\vartheta(x_n) \geq \lambda x_n$ (we drift too far above $x$). Also, since $\vartheta(x)$ is non-decreasing, we have $\min\limits_{t \in [x_n, \lambda x_n]} \vartheta(t) = \vartheta(x_n) \geq \lambda x_n$. Hence integrating over this interval,

Picking the $x_n$ such that the intervals $[x_n, \lambda x_n]$ are disjoint, we get

This contradicts the convergence established in (V), and so $\limsup\limits_{x\rightarrow \infty} \vartheta(x)/x \leq 1$. Similarly, if we assume $\liminf\limits_{x\rightarrow \infty} \vartheta(x)/x \leq \lambda < 1$, i.e. there are abitrarily large $x_n$ such that $\vartheta(x_n) \leq \lambda x_n$, we get a similar divergence looking at the interval $[\lambda x_n, x_n]$.

Therefore $1 \leq \liminf\limits_{x\rightarrow \infty} \vartheta(x)/x \leq \limsup\limits_{x\rightarrow \infty} \vartheta(x)/x \leq 1$ and so $\lim\limits_{x\to\infty} \vartheta(x)/x=1$. $\blacksquare$

Thus, we have proven one of the most fundamental results in analytic number theory.

Do We Need the Primes At All?

I want to note not much of this proof actually used any key properties of the primes. The only time we ever actually used "primeness" was in deducing the Euler product form to show that $\zeta(s) \neq 0$ for $\Re(s) \geq 1$. The rest followed from analytic properties of the functions $\pi(x)$, $\Phi(s)$, and $\vartheta(x)$. If you can find another subset of $\mathbb{N}$ that implied similar non-vanishing properties, this proof could be easily adapted. For instance, for any subset $S \subseteq \mathbb{N}$, we can let $\pi_S(x) = \left| S \cap \mathbb{N}_{\leq x} \right| = \# \{ n \leq x \ : \ n \in S \}$ i.e. $\pi_S(x)$ is the $S$ counting function instead of prime counting function. Similarly let $\vartheta_S(x) = \sum_{s \in S, s \leq x} \log s$, we have the equivalence from before with the same proof.

Theorem: $\pi_S(x) \sim \frac{x}{\log x}$ if and only if $\vartheta_S(x) \sim x$.

What was hard was establishing either of these asymptotics, which is where the analytic properties became important, and in particular needed the primes and fundamental theorem of arithmetic to establish the non-vanishing of $\zeta(s)$.

Consequences of the Prime Number Theorem

The Prime Number Theorem allows us to extend and strengthen many of the results we have already spent time combing through. Quite possibly the most immediate allows us to estimate the number of primes less than or equal to $n$.

Corollary: $\displaystyle \pi(n) \approx \frac{n}{\log n}$.

Of course, this is only useful for large $n$ as we approach the asymptotics. The actual estimate given by $n/\log n$ isn't great, and often in practice it is better to use the Logarithmic integral.

Proposition: The PNT is equivalent to $\pi(x) \sim \mathrm{Li}(x) = \int_2^x \frac{\mathrm{d}t}{\log t}$.

Proof: This follows from the fact that $\mathrm{Li}(x) \sim \frac{x}{\log x}$ as well, and that can be obtained with integration by parts. If interested, you can read the whole induction to obtain the asymptotics here. $\blacksquare$

This roughly makes sense to be a better estimate of $\pi(n)$, as it is a more "dynamic" estimate by integrating over the "density" of the primes suggested by the PNT. For a small interval $(t, t + \epsilon]$ around $t$, the number of primes in that interval is approximately

More crudely, just consider $\pi(t+dt) - \pi(t) = \frac{t+dt}{\log(t+dt)} - \frac{t}{\log t} \approx \frac{t+dt}{\log t} - \frac{t}{\log t} \approx \frac{dt}{\log t}$. The point is that this integral uses the estimate of the PNT to create tighter bounds in accordance to how far along we have counted, as opposed to uniformly treating the number of primes in an interval the same everywhere. You can read more on this specific estimate here.

This estimate should hopefully look familiar, as it and the Prime Number Theorem affirm one of our big suspicians originally! Remember we found heuristically, by considering the Sieve of Eratosthenes, that the probability of $n$ being prime should be about the probability of $n$ not being divisible by any number $x<n$ which resulted in a probability of $1/\log n$. The derivative should make this even more concrete, and even more simply, the number of primes less than $n$ is $\pi(n)$, so the probability of a number up to $n$ being prime should be about $\pi(n)/n \approx 1/\log n$. This also then gives the interpretation that the average gap between primes less than or equal to $n$ is about $\log n$.

So, if you need a prime, let's say around $2^{512}$ for cryptographic purposes, the odds of a number around there being prime is about $1/\log(2^{512}) \approx 1/356$. Since even numbers are obviously not prime, we need to check on average about $356/2 = 178$ numbers to find a prime of that magnitude (and this is without any optimizations, or fault tolerances in allowing probable primes like before).

Estimating the nth Prime Number

The Prime Number Theorem implies a similar type of inverse relation. As before, let $p_n$ be the $n^\textrm{th}$ prime number.

Proposition: $p_n \sim n \log n$.

Proof: This follows since $n = \pi(p_n) \sim \frac{p_n}{\log p_n}$ i.e. $p_n \sim n \log p_n$. This is almost what we want, so resubstituting the relation back into itself one more time gives

This works in particular because logarithms preserve asymptotics for functions increasing to $\infty$, and the PNT implies $\log p_n \sim \log n$. So we can rewrite these substitutions of $p_n$ as substitutions of $\log p_n$. $\blacksquare$

We saw before a few attempts to find prime generating formulae, and while the PNT doesn't give us one directly, it gives us a method to at least estimate the nth prime. Like the original Prime Number Theorem estimate, this isn't that great of an approximation. Since $n = \pi(p_n) \sim \mathrm{Li(p_n)}$, we get the better approximation that $p_n \sim \mathrm{Li}^{-1}(n)$ (note this works due to monotonicity). A better second order estimation given by analyzing the Logarithmic integral is

It seems we get close to something like this directly from our asymptotics above, but we need more careful analysis to get here. Cipolla found this originally in 1902 (of reference I cannot find), but there have been more elaborate methods that result from looking at $\mathrm{Li}^{-1}$. One such developed for more general inversion problems can be found here (helpfully, also in English). Here is one possible, more numeric idea to get this better approximation. We start with the better estimate of $n = \pi(p_n) = \mathrm{Li}(p_n)$. By using the asymptotic expansion given by the integration by parts, we obtain the better estimate

Again using $\pi(p_n) = n$, we get

Hence, we want to approximate solutions to $f(x) = x$. Now, note that $f'(x) = \frac{n}{x} < 1$ since $p_n > n$. Hence, by the Contraction Mapping Theorem, this equation has a (stable) fixed point which we know to be $p_n$, so we can iteratively give guesses to get better and better approximations of the fixed point $p_n$. We start with our initial guess we got before of $p_n \approx n \log n$.

which was the better estimate from before! We can plug this in again to get one more estimate, but since we are adding terms together, it won't simplify as cleanly.

Heuristically, though, this approximation performs worse, likely due to the accumulated errors ignored in our estimations and asymptotics; what we really want to solve is $\pi(p_n) = n$, but we can only estimate $\pi(n)$. This results in a fixed point solution that may be close to $p_n$, but not exactly.

Revisiting Bertrand's Postulate

The Prime Number Theorem allows us to strengthen and show some of the results we have already discussed. For instance, Bertrand's postulate falls out almost immediately from the estimates given by the PNT.

So for large $n$, there is not only at least one prime, but many primes in the interval $(n, 2n]$. Of course, our original proof of Bertrand's Postulate has its benefits, one being that its concrete and establishes at least one prime for all intervals like this, rather than the slightly more vague asymptotic notion. Perhaps a bit more interestingly is that this shows that there are roughly the same number of primes in the interval $(0,n]$ as there is from $(n,2n]$, which is a bit surprising given that $\pi(n)$ isn't linear. But because $\log n$ grows slow enough, the asymptotics balance out. Similarly, we can strengthen Bertrand's Postulate to any non-zero interval $(n, (1+\epsilon)n]$.

So all we need is $n$ to be large enough to overcome the scaling of $\epsilon$; we can take a "small" enough interval given that $n$ is big enough to make that interval have enough room for primes to occur. How big that $n$ must be is not super easy to get given that we haven't done any of the (also difficult) error analysis of the Prime Number Theorem, but a good estimate might just start with getting this asymptotic term greater than 1.

A Quick Application of the Prime Number Theorem

To round things off, here's a quick application of the PNT. While it is nonetheless an interesting and key result about the behavior and distribution of primes, the PNT is not just a novelty and can be used to give bounds on other prime related problems. In particular because it tells us that the average gap of primes around $n$ is $\log n$, they occur just frequently enough for their harmonic style sum to grow logarithmically slower than the typical harmonic sum, and so still diverge.

Proposition: $\sum_{p \textrm{ prime}} \frac{1}{p}$ diverges.

Proof: We can use a Riemann-Stieltjes style integral rewriting like we did before, and use integration by parts.

Applying the PNT, we get

$\log \log t \to \infty$ as $t \to \infty$, hence the sum $\sum_p \frac{1}{p}$ diverges. $\blacksquare$

Alternatively, here's a direct proof of the above result that avoids using the PNT.

An Analogue to the Prime Number Theorem in $K[x]$

Before finshing this post off, I want to remind us where we started. Before treating primes in this abstract, removed way, we started with how we all learned what a prime number is: a number that has no other factors other than 1 and itself. We then recalled the slighlty more useful, still simple and well-known fact that every natural number can be factored as a product of primes, a fact which we summarized with the natural numbers are a unique factorization domain (UFD). But since the natural numbers aren't the only UFD, this leads to the obvious question: is there a Prime Number Theorem for other UFDs? It turns out, there is.

One of the most common and useful UFDs is the ring of polynomials over a field $K[x]$. That is, if $K$ is a field, then the set $K[x]$ of one-variable polynomials in $x$ with coefficients in $K$ is a UFD. For a quick refresher for why if you know some ring theory, $K[x]$ is a principal ideal domain (PID) due to the division algorithm (which works since $K$ is a field), and every PID is a UFD. Our "primes" in $K[x]$ are not numbers, but rather irreducible polynomials i.e. polynomials that cannot be factored into a product of non-constant polynomials.

Example: In $\mathbb{R}[x]$, $x^2 + 1$ is irreducible since it has no roots, but in $\mathbb{C}[x]$ factors into irreducibles as $x^2+1 = (x-i)(x+i)$.

For any infinite field, there are an infinite number of irreducible polynomials: just take the linear polynomials $f(x) = x - \alpha$ for $\alpha \in K$, just like how there are an infinite number of primes in $\mathbb{N}$. Since there is no obvious ordering on polynomials, we can't make a prime counting style function $\pi(x)$ easily for $K[x]$.

However, if we let $K = \mathbb{F}_q$ be the finite field of (prime power) order $q$, and $P_n$ denote the set of monic irreducible polynomials of degree $n$, then it turns out

where we make the substitution $x = q^n$ to highlight the analogy. The proof is not super long, but requires knowing some field/Galois theory. On the whole, it is not difficult and the prerequisites can be learned relatively quickly, but are still necessary. In particular for finite fields, this is a great refresher.

Theorem: The number of monic irreducible polynomials of degree $n$ over the finite field $\mathbb{F}_q$ is $P_n = \frac{1}{n} \sum_{d \mid n} \mu(n/d) q^d$ where $\mu$ is the Möbius function.

Proof: We proceed combinatorially, counting the number of elements of finite fields in two different ways that relate to the size of the set $P_n$. The case for $n=1$ is clear since $| P_1 | = q = \frac{1}{1} \sum_{d \mid 1} \mu(d/1) q^d$ since there are $q$ monic linear polynomials (that are all irreducible) and the only divisor of 1 is 1 itself. So for the remainder of the proof, we assume $n>1$.

Let $R_n = \bigcup \{\textrm{roots of } f(x) \ \mid \ f(x) \in P_n \}$ be the collection of all the roots in $\overline{\mathbb{F}_q}$ of polynomials in $P_n$. Since these irreducible polynomials are of degree $n$, the splitting field for all of these polynomials are $\mathbb{F}_{q^n}$, so we can restrict our attention there. Since irreducible polynomials are minimal polynomials of their roots, we can make use of two properties to characterize $R_n$.

• Roots of minimal (and so irreducible) polynomials are distinct: This can be seen by $\mathbb{F}_q$ being the splitting field of the separable polynomial $x^q - x$ and so Galois. More directly since every element satisfies $x^q - x$ by Lagrange's theorem, take irreducible $f$ with root $\alpha \in \mathbb{F}_q$, so $f(x) \mid x^q - x$ and so is separable.
• Distinct irreducible polynomial have distinct roots: Since each irreducible $f$ is the minimal polynomials of its roots, and minimal polynomials are unique, distinct irreducibles have distinct roots.

So $R_n$ is actually the union of disjoint $n$-element sets, one for each polynomial $f(x) \in P_n$ i.e. $\left| R_n \right| = n \left| P_n \right|$. Therefore it is sufficient to count $R_n$ to count $P_n$. Building off the definition of $R_n$:

The second equality comes from the degree of simple field extensions. The third is for degree reasons from the Tower Law. The fourth follows from $\mathbb{F}_{q^n}/\mathbb{F}_q$ having a finite number of subfields (or more generally from an application of Zorn's lemma that allows us to extend subfields to maximal ones; also, as a result of the Primitive Element Theorem or Fundamental Theorem of Galois Theory, a finite separable field extension can only have finitely many intermediate fields). Note the subfields of $\mathbb{F}_{q^n}$ are $\mathbb{F}_{q^k}$ such that $k \mid n$ (most quickly seen by the Fundamental Theorem of Galois Theory). So writing $n = p_1^{e_1}p_2^{e_2}p_3^{e_3}\cdots p_r^{e_r}$ as its prime factorization, we find that the maximal subfields of $\mathbb{F}_{q^n}$ are

Hence, by our final equivalence above, $| R_n | = \left| \mathbb{F}_{q^n} \setminus \left( F_{p_1} \cup F_{p_2} \cup \cdots \cup F_{p_r} \right) \right|$. Also, the lattice of subfields of $\mathbb{F}_{q^n}$ correspond to the lattice of divisors of $n$ (again by the Fundamental Theorem of Galois Theory), so we get the intersections of fields as we would expect with $F_{p_i} \cap F_{p_j} = \mathbb{F}_{q^{n/p_i p_j}}$ (and similarly for intersections of more than 2 fields). So using inclusion-exclusion counting,

And so in particular,

So dividing by $n$ and applying the Möbius function (noting that $\mu(d) = 0$ for non square-free divisors), we have

We get the last equality since as $d$ runs over all divisors of $n$, so does $n/d$. $\blacksquare$

Corollary: $|P_n| \sim \frac{q^n}{n}$.

Proof: The dominant term in the expression is $q^n/n$, since the next largest power of $q$ in the formula is at most $q^{n/2}$ since the smallest non-trivial divisor of $n$ is at least 2, and thus bound the remaining terms by $o(q^n/n)$. $\blacksquare$

Conclusion

This post, while dense, has only scratched the surface of what primes and the Prime Number Theorem actually has to offer. From other bounds in combinatorics and analysis, to important computational heuristics in cryptography, primes, while random, find their ways into many facets of math. There's always more to revisit, but that's all for another day.

• Written Report
• Presentation Slides