Hey everyone, I'm the author. I'm seeing a lot of the same comments here, so I want to address them.
I teach math by leading with examples. I try to show the intuition behind an idea, and why it is interesting. For this series, my reader is someone who knows algebra, and likes learning new things, especially when a teacher shows what is interesting about a topic.
## You didn't cover x about the dot product.
I try to only teach as much as is necessary to get the student to the next point, which is matrix multiplication. I usually end up cutting a lot of material out of my chapters to keep them simple. In this case, I cut out a whole section on the properties of a dot product, as well as a discussion about inner and outer products, because those weren't necessary to get to matrix multiplication. I think this context was lost while posting to HN.
## 3B1B already has a series on this.
I love 3B1B, but his style of teaching and mine are quite different. Even though we both teach visually, his videos are densely packed with information and his expectation is that you will watch the video a few times till you understand the topic. He also leads with math more than I do. My posts are written more like stories. My goal is they should be easy to get into, and by the time you have finished reading, you should understand more about the topic. I don't expect readers to read through multiple times. I personally learned linear algebra through Strang's videos and textbook, and those videos are awesome, but can be confusing. If you found the Strang or 3b1b videos confusing, hopefully my posts will make it easier for you to follow them. I think comment is spot on: https://news.ycombinator.com/item?id=45800657
If these ideas resonate with you, I think you'll like this post, and if not, there are plenty of guides that go the more traditional route. You can also read the first post in the series and see if you like it: https://www.ducktyped.org/p/an-illustrated-introduction-to-l...
When I see the word "illustrated," I expect to see graphs or something that would help me visualize how linear algebra works. The only thing "illustrated" about this post is that he hand drew some table which could have been easily with some basic HTML+CSS.
I think this is a reasonable expectation. I'm writing about the dot product and there is a geometric visualization for dot products, so it's fair to expect that. As I said in this comment (https://news.ycombinator.com/item?id=45802969) I focused on teaching the reader the bare minimum they would need to know to get to matrix multiplication.
I use illustrations for a lot of other things though. I use them to pace my posts, to call out things that are especially important, to reinforce written ideas with images, and to give the reader's eye a place to rest.
What graphical illustration do you think this is missing? How would that make things better? Have you ever seen http://matrixmultiplication.xyz/? Great graphical illustration. Also: a really unhelpful way to understand matrix multiplication.
This is part 2 of a series, all under the same name; the first part is extensively illustrated (and I'm not sure the part 1 illustrations are all that helpful).
I kind of hate all these, because matrix multiplication is easiest to think of as simply repeated matrix-vector multiplication, which you need anyways (and earlier).
Illustrating the dot product using the projection of one vector on another conveys the geometric idea. Then it's transparent why "orthogonal" means dot product = 0.
The author seems to be taking a different tack though, and maybe doesn't want to be too tied to this particular geometric picture
That's your preference. However "To illustrate is to make something more clear or visible. Children's books are illustrated with pictures. An example can illustrate an abstract idea. "illustrate" comes from the Latin illustrare 'to light up or enlighten.'"
A great resource that isn't mentioned often is the linear algebra chapters in Birkhoff and Mac Lane's Survey of Modern Algebra. Chapters 7,8,9, and 10 (in the 4th and 5th editions anyway) are a self-contained book-within-a-book of about 200 pages on both the computational and theoretical aspects of vector spaces, matrices, linear transformations, and determinants.
Many times I've been puzzled by a concept just to go there and find it made simple and obvious. It's a real golden nuggett... Plus if you then want to go further into groups, rings, fields, and Galois theory, that's also there.
The comments here are a case study in conflict drives engagement. And not in a good way.
Try not to drive more conflict if you respond.
Facts:
The target audience of this post is someone who does not know anything about the dot product. It provides a mechanical/computational introduction. It does not include any geometric interpretation of the dot product.
Critique:
After you read the post, you should have seen something to connect to a matrix-vector product (imo), but the example matrices need to be transposed to look like a traditional matrix-vector product. Maybe the author is intending to multiply on the left by the transpose of the weight vector. The weights don’t seem to consistently appear on either side.
If you actually want to learn linear algebra, don't use this blogpost. It's real weaksauce compared to the wealth of free information and resources available online.
Firstly, the real illustrated guide to linear algebra is the youtube series "The Essence of linear algebra" by 3blue1brown[1]. It has fantastic visualisations for building intuition and in general is wildly superior to this, which seems fine but extremely superficial.
If you're done with 3b1b and want to take things further, then the go-to is the excellent 18.06SC course by the late and legendary Gilbert Strang. It's amazing, it's free. [2]
Still want more? OK now you're talking my language. If you are serious about linear algebra (Up to graduate level, after that you need something else) then you want the book "Linear Algebra Done Right" by Sheldon Axler. It's available for free from the author's website[3] and he has made a bunch of videos to supplement the book. There's also an RTD Math full lecture series[4] that follows the book and he explains each thing in a lot of detail (because Axler goes fast, so it's beneficial to unpack the concepts a bit sometimes).
Yeah agree. I first picked it up before I self-studied the 18.06SC course and bounced off pretty hard, then I'm going through it again now and it's an absolute joy, but is really packed.
Every time this subject comes up, or really any math subject comes up, someone recommends 3Blue1Brown. I love 3Blue1Brown. Just like when The Shawshank Redemption plays on TBS for the 389248th time, I will stop what I'm doing and rewatch any 3Blue1Brown video as soon as it appears in my feed.
But I'm not sure I've ever really learned anything from one of those videos. Appreciated something more? Absolutely. And maybe, sure, that's a kind of learning. But I cringe every time eigenvalues come up and people point to the 3B1B evector video.
In fact, if your goal is to actually get any kind of facility with the concept, this "weaksauce" blog post probably has a better didactic strategy than 3B1B. It strips the concept down, provides specific, minimized worked examples, and provides a useful framing for the concept (something basically at the core of 3B1B's process).
I learned linear algebra from Strang's 18.06. I later did a bunch of Axler helping my daughter through UIUC linear algebra. I like both. Strang is much closer to what the median HN person probably wants. In both cases though: don't do what I did at first, and just watch the videos and read the book. If you're not doing problems, you're probably not learning anything.
This blog post comes closer to "actually doing problems" than 3B1B. Ergo: its sauce is stronger, not weaker.
I came to the blog post expecting to roll my eyes. No discussion of inner product spaces? Not even a mention of conjugate symmetry? I was pleasantly surprised.
It's not easy to come up with a simple, accessible framing for a topic like this, and, maybe, the dot product is particularly tricky to give an intuition for (I'll go out on a limb and say that neither Strang nor Axler do a particularly great job at it --- "it" being, explaining the "why" of the dot product to someone who doesn't really even know what a vector is). The post doesn't purport to teach all of linear algebra. It's just an exercise in trying to explain one small part of it.
I'm not asking you to give the author a break, so much as suggesting that you're closing yourself off from appreciating different strategies for explaining complex topics, which is a valuable skill to have.
But at the very least, surely a dot product explainer should talk about the two main ways of looking at a dot product! This article leaves out the "angle and norms" (||a||·||b||·cos(θ)) interpretation entirely. It's like if someone gave you the formula for complex multiplication, without also showing you how it's all about rotations.
And maybe I'm being a pedant, but the dot product should be between two things that are in the same space. In the Minnesota lottery example, the probabilities should be a row vector instead. It's the exact same calculation, so again, maybe a bit too pedantic.
So does Strang! (I just checked, Linear Algebra & Applications 4E).
Also: sir, this is a blog post. It's wild seeing people say "if you really want to understand this topic, pick up Axler". I mean, yeah, also if you were serious you could just enroll in your local community college's Linear Algebra course.
My feeling is that a lot of the critique here is really signaling. For whatever reason, linear algebra is super high-status in this community, and people want to communicate that they've done something serious with it. (I'm sure I'm guilty of that too.)
> It's wild seeing people say "if you really want to understand this topic, pick up Axler"
I agree. I don't think "Linear Algebra Done Right" is a good fit for most people. It's way too dry, and I don't think his crusade against determinants helps the book. I don't know what a good book suggestion would be, though. Maybe just nab the course notes/slides/exercises off some university's website?
> this is a blog post
Blog posts can be and often are amazing.
> My feeling is that a lot of the critique here is really signaling.
Weird accusation, so just to be clear, I haven't done anything serious with anything, ever.
I wasn't referring to anybody in particular. But, like: what is the point of calling out a blog post for not presenting the angle interpretation of the dot product? How would that have fit with the goals of this post? You presented it as a defect, but that logic also suggests Strang's explanation is defective.
> that logic also suggests Strang's explanation is defective
I haven't read Strang's book, so I can't comment on that. But yeah, if it never mentions the formula ||a||·||b||·cos(θ) or at least talks about how the dot product relates to parallel and orthogonal vectors, I would consider that a big hole in an introduction to the dot product.
> How would that have fit with the goals of this post?
Because the post is titled "an introduction to linear algebra... the dot product", and this is something that I believe should be in anything that considers itself an introduction to the dot product.
You seem to disagree, and I'd like to ask: why? I think this a fundamental aspect of the dot product, again, just as fundamental as the relationship between complex multiplication and rotation. I think my view is common.
> calling out a blog post
I didn't intend to do anything as strong as "call out" the blog post. I just wanted to express surprise at someone so strongly praising ("its sauce is stronger [than 3B1B's video series]") an alright post.
Well, he's one of the most famous and best-respected educators of linear algebra, and this is an unusually basic piece of linear algebra to be taking aim at him for, so one answer is: if you have to ask whether his approach is defective, you should first evaluate how strong your own understanding is. That's argument from authority, but then: Strang is an authority. The typical push-pull on a message board is between Strang and Axler, and, if you want to find out if Axler is going to save your argument, flip to 6.A in LADR.
The direct response to your question is: the centrality of the angular interpretation of the inner product is the kind of thing I feel like you'd say if your primary purpose for learning linear algebra is to program video games. Linear algebra isn't "about" geometry, and, in particular, Strang's teaching goal centers vector spaces and the relationship between spaces. You need inner products to apply linear transformations with matrices. You basically don't need the angular interpretation... at all? if your goal is to teach linear algebra as a tool for data manipulation and, as Strang did in his last course, as a vehicle for deep learning.
(You need orthogonality, but you didn't say `a⟂b=0`, you said `|a||b|cosθ`).
I'm not rating this blog post "higher" than 3B1B; the comparison doesn't even make sense. The blog post and the video series simply have different objectives.
You don't need inner products for linear transformations. You just need the idea of a basis and linearity. You define your transformation on a basis (which is all a matrix is: the list of where the map sends each basis element), and it is automatically defined everywhere else via linearity. The textbook my undergrad class used (Curtis) doesn't define inner products until after linear transformations and matrices, for example.
The angular interpretation and geometry are basically the entire point of inner products (inner products are how you define a large chunk of geometry). Angles and projections are the entire intuition behind talking about orthogonality, which is super important practically to basically every field.
Re the more abstract approach to transformations, fair point, and I feel like that describes Axler well too. I'd soften my argument to just that being Strang's approach to bringing in the subject.
I agree orthogonality is important. But Strang doesn't get to `a⟂b=0` by means of `cosθ`. You're halfway into the book before he's even defined the Euclidean norm. He derives orthogonality mostly algebraicaly; the only angle he talks about is π/2.
Eh, I don't even think it's more abstract. Like if you're in intro engineering/physics math, and you have R^3 with unit vectors i,j,k, and R^2 with unit vectors I,J, then a function f: R^3 -> R^2 is linear exactly when you can calculate all f(v) = f(ai + bj + ck) = af(i) + bf(j) + cf(k). Then you can define f by "what is f(i)? (Some AI+BJ). What is f(j)? (Some CI+DJ). What is f(k)? (Some EI+FJ)", and then a matrix is just a tabulation of those things. Basically, linear functions let you pick out just n points to define them everywhere, kind of like polynomials. Matrices are that information. Perfectly concrete even at the super intro level. Matrix "multiplication" becomes automatic and trivial; it's just function composition tabulated.
Actually it seems way more concrete to me than mystery rules for "multiplying" arrays.
> That's argument from authority, but then: Strang is an authority.
Yes, it's an argument from authority.
> The typical push-pull on a message board is between Strang and Axler
We're not playing Pokemon with linear algebra textbooks here...
> the centrality of the angular interpretation
It's not central. It's one of the ways to think of it. I think 3B1B actually did a good job emphasizing this in his video series: there are many ways to look at vectors, and all of them are sometimes useful. They can be arrows in space, or sequences of numbers, or black boxes that obey the vector space axioms, or polynomials, and so on.
> if your primary purpose for learning linear algebra is to program video games.
What an odd guess. Seriously, I would be surprised if most math teachers didn't mention angles and norms, scalar projections, etc. An important part of math is being able to see things from multiple angles (no pun intended), and this is a useful angle to view the dot product from.
> Linear algebra isn't "about" geometry
Sort of. Linear algebra proper, the study of vector spaces and linear maps between them, is not about geometry, but geometry comes in almost precisely once you equip the vector space with an inner product, such as the dot product. A bare vector space has almost no geometric content, but the inner product gives you lengths, angles, isometries, orthogonality, and all that jazz.
> Strang's teaching goal centers vector spaces and the relationship between spaces.
That's a good goal.
> You need inner products to apply linear transformations with matrices.
I think you have a fundamental misunderstanding of linear algebra here. You do not need an inner product to "apply linear transformations with matrices".
> You basically don't need the angular interpretation... at all? if your goal is to teach linear algebra as a tool for data manipulation
Off the top of my head, cosine similarity? It's not uncommonly used.
But if you're teaching linear algebra for data manipulation, you rarely need the dot product, either. Most "dot products" in data manipulation, like the ones in this article, would be better expressed as row-vector-column-vector matrix products.
> (You need orthogonality, but you didn't say `a⟂b=0`, you said `|a||b|cosθ`).
I said "||a||·||b||·cos(θ) or at least talks about how the dot product relates to parallel and orthogonal vectors". Anyway, it's hard for a person not to interpret orthogonality as a statement about angles, so I'm not sure what distinction you're trying to draw here.
> I'm not rating this blog post "higher" than 3B1B
I guess I misinterpreted "its sauce is stronger", then.
Sorry for any mistakes; this took way too long to type.
We're indeed not playing Pokemon with textbooks here since neither of the two textbooks we're discussing (2 of the 3 most famous for the subject) agree with you. I haven't read Lay; is that the one you read, and does it support this argument?
I take exception to the idea that I was somehow signalling, and frankly that's a pretty weird thing to say. It happens that I love maths and am studying it part-time alongside my work. I posted some links to things that I have found useful on my journey so far in the hope that they are useful to others.
I don't think "taking exception" is helpful. You opened with "If you actually want to learn linear algebra, don't use this blogpost." Do you feel like that really needed to be said?
Strang does include that. I just checked the fourth edition, like you say you did. Scrolling down two pages to get to the first page of the table of contents, I see the heading “cosines and projections onto lines”. I navigate to that section and it explains all the logic, proof, and intuition behind the connection between angles and dot products. Please don’t spread misinformation…
That's page 171, 13 dense pages into the section on orthogonality, long after he's introduced the dot product, in a section where he says, in italics, "the orthogonal case is the most important"; he gives cosines about half a page before going back to a perp b.
Nobody said Strang never mentioned cos θ --- that would be weird --- only that his sequencing doesn't treat the angle formula as fundamental, even in the section introducing it. And nobody has ever read Strang and thought projections didn't matter.
When I was in school, my teachers and professors all insisted that the only way to learn a subject was to _practice_ it. If you want to learn math, work the problems. I was skeptical when I was young, but I have yet to find any other way to learn math (or anything else).
For most people going into science and engineering as opposed to pure mathematics, Poole's "Linear Algebra: A Modern Introduction" is probably more suitable as it's heavy on applications, such as Markov chains, error-correcting codes, spatiel orientation in robotics, GPS calculations, etc.
Comparing this blog post to a 500-page book or a multi-hour course and calling it “weaksauce” misses the point. This post is meant as an introduction to the dot product, and it does that really well. The formal definition (6.1) and explanation in Axler’s book wouldn’t make a good starting point for most people, it isn't even a good next step in my opinion. It’s great that you’re passionate about the topic, really, but helping more people discover math means meeting them where they are and appreciating content like this for what it’s trying to do.
of course. dot products are a symmetric form on vector spaces. they let you compute the spheres of radius r.
given the sphere of radius r, for any pair of vectors v,w in the sphere
-r^2 <= dot(v,w)=dot(w,v) <= r^2
as w varies from v to -v the value moves from r^2 through 0 to -r^2
this is how we define parallel perpendicular and antiparallel.
the dot product is only meaningful in a geometric context. by definition it projects vectors down to scalars. fixing the scalar value finds the spheres, and for a sphere we can vary the vectors to compute cosines.
Find a copy of Yousef Saad’s books (Iterative Methods, and the Eigenvalue Problems). Find a problem you want to solve, and implement one of the solutions he describes. If you don’t understand something, that’s what the first chapter is for.
Some hand-written (not AI-generated) prompts to consider:
"An expert in university-level linear algebra, including solving systems of equations, matrices, determinants, eigenvalues and eigenvectors, symmetry calculations, etc. - is asked the following question by a student: "This is all great, professor, and linearity is also at the heart of calculus, eg the derivative as a linear transformation, but I would now like you to explain what distinguishes linear from non-linear algebra."
"What kind of trouble can the student of physics and engineering and computation get into if they start assuming that their linear models are exact representations of reality?"
"A student new to the machine learning field states confidently, 'machine learning is based on linear models' - but is that statement correct in general? Where do these models fail?"
The point is that even though it takes a lot of time and effort to grasp the inner workings of linear models and the tools and techniques of linear algebra used to build such models, understanding their failure modes and limits is even more important. Many historical engineering disasters (and economic collapses, ahem) were due to over-extrapolation of and excessive faith in linear models.
mathacademy has a course on linear algebra. currently working my way back up from nothign to get to it. easily the best resource for learning math on the internet.
I do love Math Academy (I signed up 9 months ago in the hopes of replacing my NYT Crossword habit with something more productive, and 9 months later I'm gearing up for multivariable calc, which is neat given that except for linear algebra, which I self-studied out of necessity for cryptography work, all my math education stopped in sophomore year of high school).
It has a very different purpose than a post like this though! Also: there's probably more effort at exposition in this blog post than in all of Math Academy's coverage (that's not a dunk on Math Academy).
> here's probably more effort at exposition in this blog post than in all of Math Academy's coverage (that's not a dunk on Math Academy).
haha definitely agree. a lot of these blog posts are great if you want to read about math. mathacademy is pretty much all the exposition chopped out and you spend 90% of your time doing math. I can see how some wouldn't like it but I think the problem solving aspect makes it was more useful for bruteforcing your way towards building intuition
I like comparing it to Lingua Latina Familia Romana, a book that teaches Latin basically without any English; it opens in Latin and just keeps going that way and somehow you're able to follow along. Both are kind of trippy experiences.
> Summary: A dot product is a weighted sum of two vectors.
Nope. This is incorrect. The dot product is a weighted sum of a vector's elements, where the weights are the elements of the other vector. Weighted sum of two vectors would require a third entity to provide the weights.
A dot product is a weighted sum of two vectors, but not in the way the author suggested. The author's use is that one of the vectors is the weights and the other is 'the' vector, so the dot product is the weighted sum of ONE vector. It just so happens that because the author is not interested in the geometric interpretation of the dot product that they forgo the metric.
On the other hand, it is common to need a metric, which is actually the set of weights in the dot product. If `g` is the metric,
dot(a, g, b) = np.einsum('x,xy,y->', np.conj(a), g, b)
g doesn't have to be diagonal, but if you want the dot product to be symmetric in a and b it ought to be self-adjoint. Then you can find a basis where g is diagonal with real diagonal elements, which you can interpret as the weights.
It's an interesting callout; if you go Google "weighted sum of two vectors", it's not too hard to find more authoritative sources (nothing as authoritative as Axler or Strang, of course) describing either a dot product or a linear combination in those terms.
Hey everyone, I'm the author. I'm seeing a lot of the same comments here, so I want to address them.
I teach math by leading with examples. I try to show the intuition behind an idea, and why it is interesting. For this series, my reader is someone who knows algebra, and likes learning new things, especially when a teacher shows what is interesting about a topic.
## You didn't cover x about the dot product.
I try to only teach as much as is necessary to get the student to the next point, which is matrix multiplication. I usually end up cutting a lot of material out of my chapters to keep them simple. In this case, I cut out a whole section on the properties of a dot product, as well as a discussion about inner and outer products, because those weren't necessary to get to matrix multiplication. I think this context was lost while posting to HN.
## 3B1B already has a series on this.
I love 3B1B, but his style of teaching and mine are quite different. Even though we both teach visually, his videos are densely packed with information and his expectation is that you will watch the video a few times till you understand the topic. He also leads with math more than I do. My posts are written more like stories. My goal is they should be easy to get into, and by the time you have finished reading, you should understand more about the topic. I don't expect readers to read through multiple times. I personally learned linear algebra through Strang's videos and textbook, and those videos are awesome, but can be confusing. If you found the Strang or 3b1b videos confusing, hopefully my posts will make it easier for you to follow them. I think comment is spot on: https://news.ycombinator.com/item?id=45800657
If these ideas resonate with you, I think you'll like this post, and if not, there are plenty of guides that go the more traditional route. You can also read the first post in the series and see if you like it: https://www.ducktyped.org/p/an-illustrated-introduction-to-l...
For another example of my writing, see my series on AWS: https://www.ducktyped.org/p/a-mini-book-on-aws-networking-in...
> I teach math by leading with examples
This the only way to teach to humans.
Any time linear algebra comes up, I make it a point to drop Pavel's videos: https://www.youtube.com/playlist?list=PLlXfTHzgMRUKXD88IdzS1....
I try to only teach as much as is necessary to get the student to the next point, which is matrix multiplication
Preemptively noting: this is also Strang's strategy.
When I see the word "illustrated," I expect to see graphs or something that would help me visualize how linear algebra works. The only thing "illustrated" about this post is that he hand drew some table which could have been easily with some basic HTML+CSS.
I think this is a reasonable expectation. I'm writing about the dot product and there is a geometric visualization for dot products, so it's fair to expect that. As I said in this comment (https://news.ycombinator.com/item?id=45802969) I focused on teaching the reader the bare minimum they would need to know to get to matrix multiplication.
I use illustrations for a lot of other things though. I use them to pace my posts, to call out things that are especially important, to reinforce written ideas with images, and to give the reader's eye a place to rest.
If you are looking for graphs, this video about the dot product from 3B1B may be more your speed: https://www.youtube.com/watch?v=LyGKycYT2v0
What graphical illustration do you think this is missing? How would that make things better? Have you ever seen http://matrixmultiplication.xyz/? Great graphical illustration. Also: a really unhelpful way to understand matrix multiplication.
This is part 2 of a series, all under the same name; the first part is extensively illustrated (and I'm not sure the part 1 illustrations are all that helpful).
I agree that's a poor way to teach matrix multiplication. This topic arose last year. [0]
Don't introduce any clumsy nonstandard transformations like 'rotating' a matrix, just highlight the relevant row and column. [1]
[0] https://news.ycombinator.com/item?id=41402224
[1] https://www.mathsisfun.com/algebra/images/matrix-multiply-a.... (from https://www.mathsisfun.com/algebra/matrix-multiplying.html )
I kind of hate all these, because matrix multiplication is easiest to think of as simply repeated matrix-vector multiplication, which you need anyways (and earlier).
Illustrating the dot product using the projection of one vector on another conveys the geometric idea. Then it's transparent why "orthogonal" means dot product = 0.
The author seems to be taking a different tack though, and maybe doesn't want to be too tied to this particular geometric picture
I don't understand the down votes, I had the same reaction. Other posters have suggested some better resources, check those out
That's your preference. However "To illustrate is to make something more clear or visible. Children's books are illustrated with pictures. An example can illustrate an abstract idea. "illustrate" comes from the Latin illustrare 'to light up or enlighten.'"
Quote from https://www.vocabulary.com/dictionary/illustrate
It's extremely obvious that the sense of "illustrated" meant here is "containing illustrations."
In the context of books or internet books illustrated almost exclusively means “with pictures”.
A great resource that isn't mentioned often is the linear algebra chapters in Birkhoff and Mac Lane's Survey of Modern Algebra. Chapters 7,8,9, and 10 (in the 4th and 5th editions anyway) are a self-contained book-within-a-book of about 200 pages on both the computational and theoretical aspects of vector spaces, matrices, linear transformations, and determinants.
Many times I've been puzzled by a concept just to go there and find it made simple and obvious. It's a real golden nuggett... Plus if you then want to go further into groups, rings, fields, and Galois theory, that's also there.
The comments here are a case study in conflict drives engagement. And not in a good way.
Try not to drive more conflict if you respond.
Facts:
The target audience of this post is someone who does not know anything about the dot product. It provides a mechanical/computational introduction. It does not include any geometric interpretation of the dot product.
Critique:
After you read the post, you should have seen something to connect to a matrix-vector product (imo), but the example matrices need to be transposed to look like a traditional matrix-vector product. Maybe the author is intending to multiply on the left by the transpose of the weight vector. The weights don’t seem to consistently appear on either side.
If you actually want to learn linear algebra, don't use this blogpost. It's real weaksauce compared to the wealth of free information and resources available online.
Firstly, the real illustrated guide to linear algebra is the youtube series "The Essence of linear algebra" by 3blue1brown[1]. It has fantastic visualisations for building intuition and in general is wildly superior to this, which seems fine but extremely superficial.
If you're done with 3b1b and want to take things further, then the go-to is the excellent 18.06SC course by the late and legendary Gilbert Strang. It's amazing, it's free. [2]
Still want more? OK now you're talking my language. If you are serious about linear algebra (Up to graduate level, after that you need something else) then you want the book "Linear Algebra Done Right" by Sheldon Axler. It's available for free from the author's website[3] and he has made a bunch of videos to supplement the book. There's also an RTD Math full lecture series[4] that follows the book and he explains each thing in a lot of detail (because Axler goes fast, so it's beneficial to unpack the concepts a bit sometimes).
[1] https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQ...
[2] https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011...
[3] https://linear.axler.net/ and https://www.youtube.com/watch?v=lkx2BJcnyxk&list=PLGAnmvB9m7...
[4] https://www.youtube.com/watch?v=7eggsIan2Y4&list=PLd-yyEHYtI...
Second Axler's book! (probably as a second exposure after a first course, to really understand what's going on in Linear Algebra)
Yeah agree. I first picked it up before I self-studied the 18.06SC course and bounced off pretty hard, then I'm going through it again now and it's an absolute joy, but is really packed.
I also recommend Robert Beezer's "A First Course in Linear Algebra". Great for self-studying.
http://linear.ups.edu/
Thank you, that's a new one.
Every time this subject comes up, or really any math subject comes up, someone recommends 3Blue1Brown. I love 3Blue1Brown. Just like when The Shawshank Redemption plays on TBS for the 389248th time, I will stop what I'm doing and rewatch any 3Blue1Brown video as soon as it appears in my feed.
But I'm not sure I've ever really learned anything from one of those videos. Appreciated something more? Absolutely. And maybe, sure, that's a kind of learning. But I cringe every time eigenvalues come up and people point to the 3B1B evector video.
In fact, if your goal is to actually get any kind of facility with the concept, this "weaksauce" blog post probably has a better didactic strategy than 3B1B. It strips the concept down, provides specific, minimized worked examples, and provides a useful framing for the concept (something basically at the core of 3B1B's process).
I learned linear algebra from Strang's 18.06. I later did a bunch of Axler helping my daughter through UIUC linear algebra. I like both. Strang is much closer to what the median HN person probably wants. In both cases though: don't do what I did at first, and just watch the videos and read the book. If you're not doing problems, you're probably not learning anything.
This blog post comes closer to "actually doing problems" than 3B1B. Ergo: its sauce is stronger, not weaker.
I came to the blog post expecting to roll my eyes. No discussion of inner product spaces? Not even a mention of conjugate symmetry? I was pleasantly surprised.
It's not easy to come up with a simple, accessible framing for a topic like this, and, maybe, the dot product is particularly tricky to give an intuition for (I'll go out on a limb and say that neither Strang nor Axler do a particularly great job at it --- "it" being, explaining the "why" of the dot product to someone who doesn't really even know what a vector is). The post doesn't purport to teach all of linear algebra. It's just an exercise in trying to explain one small part of it.
I'm not asking you to give the author a break, so much as suggesting that you're closing yourself off from appreciating different strategies for explaining complex topics, which is a valuable skill to have.
But at the very least, surely a dot product explainer should talk about the two main ways of looking at a dot product! This article leaves out the "angle and norms" (||a||·||b||·cos(θ)) interpretation entirely. It's like if someone gave you the formula for complex multiplication, without also showing you how it's all about rotations.
And maybe I'm being a pedant, but the dot product should be between two things that are in the same space. In the Minnesota lottery example, the probabilities should be a row vector instead. It's the exact same calculation, so again, maybe a bit too pedantic.
So does Strang! (I just checked, Linear Algebra & Applications 4E).
Also: sir, this is a blog post. It's wild seeing people say "if you really want to understand this topic, pick up Axler". I mean, yeah, also if you were serious you could just enroll in your local community college's Linear Algebra course.
My feeling is that a lot of the critique here is really signaling. For whatever reason, linear algebra is super high-status in this community, and people want to communicate that they've done something serious with it. (I'm sure I'm guilty of that too.)
> It's wild seeing people say "if you really want to understand this topic, pick up Axler"
I agree. I don't think "Linear Algebra Done Right" is a good fit for most people. It's way too dry, and I don't think his crusade against determinants helps the book. I don't know what a good book suggestion would be, though. Maybe just nab the course notes/slides/exercises off some university's website?
> this is a blog post
Blog posts can be and often are amazing.
> My feeling is that a lot of the critique here is really signaling.
Weird accusation, so just to be clear, I haven't done anything serious with anything, ever.
I wasn't referring to anybody in particular. But, like: what is the point of calling out a blog post for not presenting the angle interpretation of the dot product? How would that have fit with the goals of this post? You presented it as a defect, but that logic also suggests Strang's explanation is defective.
> that logic also suggests Strang's explanation is defective
I haven't read Strang's book, so I can't comment on that. But yeah, if it never mentions the formula ||a||·||b||·cos(θ) or at least talks about how the dot product relates to parallel and orthogonal vectors, I would consider that a big hole in an introduction to the dot product.
> How would that have fit with the goals of this post?
Because the post is titled "an introduction to linear algebra... the dot product", and this is something that I believe should be in anything that considers itself an introduction to the dot product.
You seem to disagree, and I'd like to ask: why? I think this a fundamental aspect of the dot product, again, just as fundamental as the relationship between complex multiplication and rotation. I think my view is common.
> calling out a blog post
I didn't intend to do anything as strong as "call out" the blog post. I just wanted to express surprise at someone so strongly praising ("its sauce is stronger [than 3B1B's video series]") an alright post.
Well, he's one of the most famous and best-respected educators of linear algebra, and this is an unusually basic piece of linear algebra to be taking aim at him for, so one answer is: if you have to ask whether his approach is defective, you should first evaluate how strong your own understanding is. That's argument from authority, but then: Strang is an authority. The typical push-pull on a message board is between Strang and Axler, and, if you want to find out if Axler is going to save your argument, flip to 6.A in LADR.
The direct response to your question is: the centrality of the angular interpretation of the inner product is the kind of thing I feel like you'd say if your primary purpose for learning linear algebra is to program video games. Linear algebra isn't "about" geometry, and, in particular, Strang's teaching goal centers vector spaces and the relationship between spaces. You need inner products to apply linear transformations with matrices. You basically don't need the angular interpretation... at all? if your goal is to teach linear algebra as a tool for data manipulation and, as Strang did in his last course, as a vehicle for deep learning.
(You need orthogonality, but you didn't say `a⟂b=0`, you said `|a||b|cosθ`).
I'm not rating this blog post "higher" than 3B1B; the comparison doesn't even make sense. The blog post and the video series simply have different objectives.
You don't need inner products for linear transformations. You just need the idea of a basis and linearity. You define your transformation on a basis (which is all a matrix is: the list of where the map sends each basis element), and it is automatically defined everywhere else via linearity. The textbook my undergrad class used (Curtis) doesn't define inner products until after linear transformations and matrices, for example.
The angular interpretation and geometry are basically the entire point of inner products (inner products are how you define a large chunk of geometry). Angles and projections are the entire intuition behind talking about orthogonality, which is super important practically to basically every field.
Re the more abstract approach to transformations, fair point, and I feel like that describes Axler well too. I'd soften my argument to just that being Strang's approach to bringing in the subject.
I agree orthogonality is important. But Strang doesn't get to `a⟂b=0` by means of `cosθ`. You're halfway into the book before he's even defined the Euclidean norm. He derives orthogonality mostly algebraicaly; the only angle he talks about is π/2.
Eh, I don't even think it's more abstract. Like if you're in intro engineering/physics math, and you have R^3 with unit vectors i,j,k, and R^2 with unit vectors I,J, then a function f: R^3 -> R^2 is linear exactly when you can calculate all f(v) = f(ai + bj + ck) = af(i) + bf(j) + cf(k). Then you can define f by "what is f(i)? (Some AI+BJ). What is f(j)? (Some CI+DJ). What is f(k)? (Some EI+FJ)", and then a matrix is just a tabulation of those things. Basically, linear functions let you pick out just n points to define them everywhere, kind of like polynomials. Matrices are that information. Perfectly concrete even at the super intro level. Matrix "multiplication" becomes automatic and trivial; it's just function composition tabulated.
Actually it seems way more concrete to me than mystery rules for "multiplying" arrays.
> That's argument from authority, but then: Strang is an authority.
Yes, it's an argument from authority.
> The typical push-pull on a message board is between Strang and Axler
We're not playing Pokemon with linear algebra textbooks here...
> the centrality of the angular interpretation
It's not central. It's one of the ways to think of it. I think 3B1B actually did a good job emphasizing this in his video series: there are many ways to look at vectors, and all of them are sometimes useful. They can be arrows in space, or sequences of numbers, or black boxes that obey the vector space axioms, or polynomials, and so on.
> if your primary purpose for learning linear algebra is to program video games.
What an odd guess. Seriously, I would be surprised if most math teachers didn't mention angles and norms, scalar projections, etc. An important part of math is being able to see things from multiple angles (no pun intended), and this is a useful angle to view the dot product from.
> Linear algebra isn't "about" geometry
Sort of. Linear algebra proper, the study of vector spaces and linear maps between them, is not about geometry, but geometry comes in almost precisely once you equip the vector space with an inner product, such as the dot product. A bare vector space has almost no geometric content, but the inner product gives you lengths, angles, isometries, orthogonality, and all that jazz.
> Strang's teaching goal centers vector spaces and the relationship between spaces.
That's a good goal.
> You need inner products to apply linear transformations with matrices.
I think you have a fundamental misunderstanding of linear algebra here. You do not need an inner product to "apply linear transformations with matrices".
> You basically don't need the angular interpretation... at all? if your goal is to teach linear algebra as a tool for data manipulation
Off the top of my head, cosine similarity? It's not uncommonly used.
But if you're teaching linear algebra for data manipulation, you rarely need the dot product, either. Most "dot products" in data manipulation, like the ones in this article, would be better expressed as row-vector-column-vector matrix products.
> (You need orthogonality, but you didn't say `a⟂b=0`, you said `|a||b|cosθ`).
I said "||a||·||b||·cos(θ) or at least talks about how the dot product relates to parallel and orthogonal vectors". Anyway, it's hard for a person not to interpret orthogonality as a statement about angles, so I'm not sure what distinction you're trying to draw here.
> I'm not rating this blog post "higher" than 3B1B
I guess I misinterpreted "its sauce is stronger", then.
Sorry for any mistakes; this took way too long to type.
We're indeed not playing Pokemon with textbooks here since neither of the two textbooks we're discussing (2 of the 3 most famous for the subject) agree with you. I haven't read Lay; is that the one you read, and does it support this argument?
I didn't read a linear algebra textbook; I studied pure math at university.
I take exception to the idea that I was somehow signalling, and frankly that's a pretty weird thing to say. It happens that I love maths and am studying it part-time alongside my work. I posted some links to things that I have found useful on my journey so far in the hope that they are useful to others.
I don't think "taking exception" is helpful. You opened with "If you actually want to learn linear algebra, don't use this blogpost." Do you feel like that really needed to be said?
Strang does include that. I just checked the fourth edition, like you say you did. Scrolling down two pages to get to the first page of the table of contents, I see the heading “cosines and projections onto lines”. I navigate to that section and it explains all the logic, proof, and intuition behind the connection between angles and dot products. Please don’t spread misinformation…
That's page 171, 13 dense pages into the section on orthogonality, long after he's introduced the dot product, in a section where he says, in italics, "the orthogonal case is the most important"; he gives cosines about half a page before going back to a perp b.
Nobody said Strang never mentioned cos θ --- that would be weird --- only that his sequencing doesn't treat the angle formula as fundamental, even in the section introducing it. And nobody has ever read Strang and thought projections didn't matter.
When I was in school, my teachers and professors all insisted that the only way to learn a subject was to _practice_ it. If you want to learn math, work the problems. I was skeptical when I was young, but I have yet to find any other way to learn math (or anything else).
Strang is still very much alive AFAIK
Yep. He retired, is all.
The Axler text was discussed here (631 pts 295 comments):
https://news.ycombinator.com/item?id=38060159
For most people going into science and engineering as opposed to pure mathematics, Poole's "Linear Algebra: A Modern Introduction" is probably more suitable as it's heavy on applications, such as Markov chains, error-correcting codes, spatiel orientation in robotics, GPS calculations, etc.
https://www.physicsforums.com/threads/linear-algebra-a-moder...
Comparing this blog post to a 500-page book or a multi-hour course and calling it “weaksauce” misses the point. This post is meant as an introduction to the dot product, and it does that really well. The formal definition (6.1) and explanation in Axler’s book wouldn’t make a good starting point for most people, it isn't even a good next step in my opinion. It’s great that you’re passionate about the topic, really, but helping more people discover math means meeting them where they are and appreciating content like this for what it’s trying to do.
The post contains no geometry. Which is the only worthwhile content of dot products.
Explaining the dot product by its implementation over R^n is pointless. Conflating 1-forms and vectors is pointless.
The only worthwhile content of dot products is geometry?
of course. dot products are a symmetric form on vector spaces. they let you compute the spheres of radius r.
given the sphere of radius r, for any pair of vectors v,w in the sphere
as w varies from v to -v the value moves from r^2 through 0 to -r^2this is how we define parallel perpendicular and antiparallel.
the dot product is only meaningful in a geometric context. by definition it projects vectors down to scalars. fixing the scalar value finds the spheres, and for a sphere we can vary the vectors to compute cosines.
Thanks for these resources
Find a copy of Yousef Saad’s books (Iterative Methods, and the Eigenvalue Problems). Find a problem you want to solve, and implement one of the solutions he describes. If you don’t understand something, that’s what the first chapter is for.
Some hand-written (not AI-generated) prompts to consider:
"An expert in university-level linear algebra, including solving systems of equations, matrices, determinants, eigenvalues and eigenvectors, symmetry calculations, etc. - is asked the following question by a student: "This is all great, professor, and linearity is also at the heart of calculus, eg the derivative as a linear transformation, but I would now like you to explain what distinguishes linear from non-linear algebra."
"What kind of trouble can the student of physics and engineering and computation get into if they start assuming that their linear models are exact representations of reality?"
"A student new to the machine learning field states confidently, 'machine learning is based on linear models' - but is that statement correct in general? Where do these models fail?"
The point is that even though it takes a lot of time and effort to grasp the inner workings of linear models and the tools and techniques of linear algebra used to build such models, understanding their failure modes and limits is even more important. Many historical engineering disasters (and economic collapses, ahem) were due to over-extrapolation of and excessive faith in linear models.
mathacademy has a course on linear algebra. currently working my way back up from nothign to get to it. easily the best resource for learning math on the internet.
I do love Math Academy (I signed up 9 months ago in the hopes of replacing my NYT Crossword habit with something more productive, and 9 months later I'm gearing up for multivariable calc, which is neat given that except for linear algebra, which I self-studied out of necessity for cryptography work, all my math education stopped in sophomore year of high school).
It has a very different purpose than a post like this though! Also: there's probably more effort at exposition in this blog post than in all of Math Academy's coverage (that's not a dunk on Math Academy).
> here's probably more effort at exposition in this blog post than in all of Math Academy's coverage (that's not a dunk on Math Academy).
haha definitely agree. a lot of these blog posts are great if you want to read about math. mathacademy is pretty much all the exposition chopped out and you spend 90% of your time doing math. I can see how some wouldn't like it but I think the problem solving aspect makes it was more useful for bruteforcing your way towards building intuition
I like comparing it to Lingua Latina Familia Romana, a book that teaches Latin basically without any English; it opens in Latin and just keeps going that way and somehow you're able to follow along. Both are kind of trippy experiences.
> Summary: A dot product is a weighted sum of two vectors.
Nope. This is incorrect. The dot product is a weighted sum of a vector's elements, where the weights are the elements of the other vector. Weighted sum of two vectors would require a third entity to provide the weights.
A dot product is a weighted sum of two vectors, but not in the way the author suggested. The author's use is that one of the vectors is the weights and the other is 'the' vector, so the dot product is the weighted sum of ONE vector. It just so happens that because the author is not interested in the geometric interpretation of the dot product that they forgo the metric.
On the other hand, it is common to need a metric, which is actually the set of weights in the dot product. If `g` is the metric,
g doesn't have to be diagonal, but if you want the dot product to be symmetric in a and b it ought to be self-adjoint. Then you can find a basis where g is diagonal with real diagonal elements, which you can interpret as the weights.It's an interesting callout; if you go Google "weighted sum of two vectors", it's not too hard to find more authoritative sources (nothing as authoritative as Axler or Strang, of course) describing either a dot product or a linear combination in those terms.