The jump from spherical harmonics to eigenfunctions on a general mesh, and the specific example mesh chosen, might be the finest mathematical joke I've seen this decade.
>If you’re alarmed by the fact that the set of all real functions does not form a HILBERT SPACE, you’re probably not in the target audience of this post."
If you're wondering what a Hilbert space, know that you're in good company.
> Dr. von Neumann, ich möchte gerne wissen, was ist denn eigentlich ein Hilbertscher Raum?
(Dr. von Neumann, I'd would really like to know, just what exactly is a Hilbert space?)
Asked to John von Neumann to David Hilbert at a lecture.
I'd like to add, as a physicist by training, that anything can be a Hilbert space if you wish hard enough. You can even use results about countable vector spaces if you need them!
It's quietly reversing the traditional "We approximate the cow to be a sphere" and showing how the spherical math can, in fact, be generalized to solutions on the cow.
Isn't this the opposite way ? Vectors are functions whose input space are discrete dimensions. Let's not pretend going from natural numbers to real is "simple", reals numbers are a fascinating non-obvious math discovery. And also the passage from a few numbers to all natural numbers (aleph0) is non obvious. So basically we have two alephs passages to transforms N-D vectors as functions over reals.
That's not really the assumption of the article, that's the assumption the article makes of the reader's knowledge of vectors as they begin reading. The article is trying to make the point that functions are also vectors, strictly speaking, and give some intuition as to what that means.
Basically it's an introduction to "functional analysis", the field of math that looks at functions like vectors. The term is in there sneakily, but this is really what the article is all about.
I had classes on this subject and we were encouraged to build upon linear algebra intuition (much like the author does), with some caveats (such as a bounded closed set is not necessarily compact in infinite dimension). Things break down, as the author hints at when they mention they didn't prove the Laplacian to be self-adjoint but only symmetric (this has to do with the operator's domain, a concern that doesn't exist for linear operators on finite dimensional spaces)... but it's still a very convenient way to think and talk about things, conceptually.
He does not assume anything! Any assumption is in your head only. Of course he starts from the specific type of vector spaces that's the most familiar to readers. But then he shows that there's nothing that requires a vector space to have a finite, or even countably infinite, dimension. What matters are the axioms.
You may recall that this representation is only one example of an abstract vector space. There are many other types of vectors, such as lists of complex numbers, graph cycles, and even magic squares.
However, all of these vector spaces have one thing in common: a finite number of dimensions. That is, each kind of vector can be represented as a collection of N N numbers, though the definition of “number” varies.
So the goal is to impress the reader by letting him believe we will easily apply our easy linear algebra to real -> real functions. But we can't.
> So the goal is to impress the reader by letting him believe we will easily apply our easy linear algebra to real -> real functions. But we can't.
When you only have a finite number of functions (and the space they span), then you can apply your finite dimensional linear algebra, since n dimensional vector spaces over a field are isomorphic. You absolutely can gain intuition for real functions from arrows on a paper type vectors.
E.g. the article mentions the Cauchy-Schwartz inequalities for functions, that's something you can intuit when you imagine our functions being little arrows on a paper.
What intuition can you gain ? This applies to any field, not just functions. You're gaining intuition for a more general concept and not what makes real->real functions different from others maths object
Vectors are an abstract notion. If you have two sets and two operations that satisfy the definition of a vector space, then you have a vector space; and we refer to elements of the vector set as "vectors" within that vector space.
The observation here is that set of real value functions, combined with the set of real numbers, and the natural notion of function addition and multiplication by a real number satisfies the definition of a vector space. As a result all the results of linear algebra can be applied to real valued functions.
It is true that any vector space is isomorphic to a vector space whose vectors are functions. Linear algebra does make a lot of usage of that result, but it is different from what the article is discussing.
I agree but we're using functions for different things here. Yes some specific families of functions can be treated as vector spaces. In this article it seems like the author is pretending to take all real->real functions and treating them as if they are a vector space, whatever the content of the functions, quote :
we’ve built a vector space of functions
and later he admits it is impossible
Ideally, we could express an arbitrary function f as a linear combination of these basis functions. However, there are uncountably many of them—and we can’t simply write down a sum over the reals. Still, considering their linear combination is illustrative:
It is not required for vector spaces to have a basis. As it turns out, the claim that every vector space has a basis is equivalent to the axiom of choice, which seems well beyond the scope of the article.
However, the particular vector space in question (functions from R to R) does have a basis, which the author describes. That basis is not as useful as a basis typically is for finite dimensional (or even countably unfitine dimensional) vector spaces, but it still exists.
But it's the article talking about vectors as a sequence of reals and having a basis, then extending that to infinite sequences of reals. The author is playing on multiple definitions of vector to produce a "woaw that's cool" effect, and that's bad maths
There is only one definition of "vector space" (up to isomorphism anyway), and that's what the author uses. You'll note that he doesn't talk about bases at all, the assumption of a basis is entirely in your mind. The entire point of the article is that the ℝ→ℝ function space is a vector space. A vector space is not required to have a basis, but assuming the axiom of choice, every vector space does have (at least) one, including that of ℝ→ℝ functions.
The choice of basis is very important for the applications. It doesnt matter that function could be vector spaces in theory without constructing a base, the article is about "hey look at functions they're as easy to operate in as this thing you already know with 3D vectors"
You are misreading the article. At a guess, you have come into it "knowing what a vector is" from physics or computer science. Specifically, a vector is not "a sequence of reals", it is any object that obeys a certain set of properties. He is not "extending" your definition, he's trying to teach you the actual definition, which is more general than the examples you are used to.
(Imagine/remember what it feels like when children first learn that the integers aren't the only number, there are also fractions, then irrationals, then complex numbers...this is a very similar situation).
With that in mind, you may want to reread the text and pay attention to the definitions he is using, and not assume that your definitions are the whole story.
I learned vector spaces in maths and I'm aware of your definition, that's how I learned it. I'm also aware that the same word can be used with different meanings depending on the context.
(Imagine/remember what it feels like when children first learn that the integers aren't the only number, there are also fractions, then irrationals, then complex numbers...this is a very similar situation).
Imagine now an article saying "complex numbers are just natural numbers, you can apply euclidean division to them the exact same way"
The set of all real->real functions is still a vector space.
This vector space also has a basis (even if it is not as useful): there is a (uncountably infinite) subset of real->real functions such that every function can be expressed as a linear combination of a finite number of these basis functions, in exactly one way.
There isn't a clean way to write down this basis, though, as you need to use Zorn's lemma or equivalent to construct it.
If you restrict yourself to Lebesque-integrable functions, can’t you take the complex Fourier transform of the function and call the terms of the Fourier series a basis, with the coefficients being the components of the vector of the function? This is a bit above my current mathematical paygrade, so forgive me if I’m not expressing the idea accurately but I’m learning a lot both from the article and the ensuing discussion - hopefully you understand what I’m getting at.
I think what I may be asking is “Does the complex Fourier transform make a Hilbert space?” but I might be wrong both about that and about that being the right question.
Another example is the eigenvectors of linear operators like the Laplacian. Recall how, in finite dimension, the eigenvectors of a full rank operator (matrix) form an orthonormal basis of the vector space. There is a similar notion in infinite dimension. I can't find an English page that covers this very well, but there's a couple of paragraphs in the Spectral Theorem page (https://en.wikipedia.org/wiki/Spectral_theorem#Unbounded_sel... ). The article linked here also touches on this.
Regarding your last sentence, one thing to note is that having a basis is not what makes you a Hilbert space, but rather having an inner product! In fact, to get the Fourier coefficients, you need to use that inner product.
That's awesome info thank you so much. Reading it, a Hilbert basis is exactly what I am talking about. It's always exciting when my intuition guides me on the right path. I'll check out the Spectral theorem page also.
for some coefficients a_k and b_k as long as f is sufficiently nice (I don't remember the exact condition, sorry).
This is very useful, but the functions sin(x), sin(2x), ... , cos(x), cos(2x), ... don't constitute a basis in the formal sense I mentioned above as you need an infinite sum to represent most functions. It is still often called a basis though.
Thanks for that. That’s the trigonometric Fourier series. A complex Fourier series (which is what I mentioned) is equivalent and works similarly except that the terms are all uniform, so
f(x) = sum n=-infty to +infty C_n e^{i n x}
You can derive one from the other by using the identities
sin x = (e^(ix) - e^(-ix))/2i
cos x = (e^(ix) + e^(-ix))/2
I specifically mentioned the complex series because I didn’t like the fact that the alternating terms use a different trig function and it seemed weird to me to have every second dimension in a space be different in that way but they are equivalent.
The convergence criteria for Fourier series vary depending on how strongly you need convergence but I think basically if a function is differentiable on the interval you care about then the Fourier series provably converges on that interval and otherwise if it has jump discontinuities and that sort of thing, then depending on whether it is square-integrable or a bunch of other properties) you can prove weaker forms of convergence (absolute, pointwise etc).
To address your comment I don’t see why an infinite sum prevents something being a basis. In fact I would specifically say that can’t be true because then there would never be a basis for any infinite-dimensional space- any time you want to take an inner product in such a space you need an infinite sum, and you need such an inner product to construct the basis. A sibling comment pointed me in the direction of a Hilbert basis, which seems to be what I was thinking of.
If you're familiar with Zorn's Lemma, the construction is just to order bases by inclusion and to consider chains created by noting that there must be an independent dimension and adding it inductively. You can upper bound each of these chains by unioning the members of the chain (which preserves linear independence). By Zorn's Lemma that means there is a maximal linearly independent system and if an element existed outside of that system's span it would contradict that maximality.
He doesn't need to talk about it (though you might like to look up the notorious Hilbert's Basis Theorem); it happens to be the case that any vector space has a basis, but even if you don't know that, a vector space is still a vector space and its elements are still vectors.
> Vectors are functions whose input space are discrete dimensions.
This is _also_ true. But the fact that finite dimensional real vectors can be viewed as a special case of set-theoretic functions or as a special case of real functions on a discrete finite space is probably less useful than the opposite: the set of real functions have a vector-space structure, and you can use all the neat theorems about (finite or infinite dimensional) vector spaces.
Well, at least the article is about the latter direction.
A few questions occur to me while reading this, which I am far from qualified to answer:
- How much of this structure survives if you work on "fuzzy" real numbers? Can you make it work? Where I don't necessarily mean "fuzzy" in the specific technical sense, but in any sense in which a number is defined only up to a margin of error/length scale, which in my mind is similar to "finitism", or "automatic differentiation" in ML, or a "UV cutoff" in physics. I imagine the exact definition will determine how much vectorial structure survives. The obvious answer is that it works like a regular Fourier transform but with a low-pass filter applied, but I imagine this might not be the only answer.
- Then if this is possible, can you carry it across the analogy in the other direction? What would be the equivalent of "fuzzy vectors"?
- If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
- The argument arrives at fourier analysis by considering an orthonormal diagonalization of the Laplacian. In linear algebra, SVD applies more generally than diagonalizations—is there an "SVD" for functions?
Convolution with dirac delta will give you an exact sample of f(0), and in principle a whole signal could be constructed as a combination of delayed delta signals - but we can't realize an exact delta signal in most spaces, only approximations.
As a result we get finite resolution and truncation of the spectrum. So "Fourier analysis with pre-applied lowpass filter" would be analysis of sampled signals, the filter determined by the sampling kernel (delta approximator) and properties of the DFT.
But so long as the sampling kernel is good (that is the actual terminology), we can form f exactly as the limit of these fuzzy interpolations.
The term "resolution of the identity" is associated with the fact that delta doesn't exist in most function spaces and instead has to be approximated. A good sampling kernel "resolves" the missing (convolutional) identity. I like thinking of the term also in the sense that these operators behave like the identity if it were only good up to some resolution.
1. Numerical methods for solving differential and integral equations are algorithms for solving algebraic equations (vector solutions) that arise from discretizing infinite-dimensional operator equations (function solutions). When we talk about whether these methods work, we usually do so in terms of their consistency and stability. There is a multistage things that happens here: we start by talking about the well-posedness of the original equation (e.g. the PDE), then the convergence of the mathematical discretization, and then examine what happens when we try to program this thing on a computer. Usually what happens is these algorithms will get implemented "on top" of numerical linear algebra, where algorithms like Gaussian elimination, and different iterative solvers, have been studied very carefully from the perspective of floating point rounding errors etc. This kind of subsumes your concern about "fuzzy" real numbers. Remember that in double precision, if the number "1.0" represents "1 meter", then machin epsilon is atomic scale. So, frequently, you can kind of assume the whole process "just works"...
2/3. I'm not really sure what you mean by these questions... But if you want to do "fourier analysis with a filter preapplied", you'd probably just work with within some space of bandlimited functions. If you only care around N Fourier modes, any time you do an operation which exceeds that number of modes, you need to chop the result back to down to size.
4. In this context, it's really the SVD of an operator you're interested in. In that regard, you can consider trying to extend the various definitions of the SVD to your operator, provided that you carefully think about all spaces involved. I assume at least one "operator SVD" exists and has been studied extensively... For instance, I can imagine trying to extend the variational definition of the SVD... and the algorithms for computing the SVD probably make good sense in a function space, too...
Your can replace the real numbers with the rational numbers and maintain all of the vector structure.
If you wanted something more quantized, you can pick some length unit, d, and replace the real numbers with {... -2d, -d, 0, d, 2d,... }. This forms a structure known as a "ring" with the standard notion of addition, subtraction, and multiplication (but no notion of division. Using this instead of R does lose the vector structure, but is still an example of a slightly more general notion of a "module". Many of the linear algebra results for vector spaces apply to modules as well.
> If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
If that is where you want to end up, you could pretty much start there. If you take all real value functions and apply a courier analysis with a low pass filter to each of them, the resulting set still forms a vector space. Although I don't see any particular way of arriving at this vector space by manipulating functions pre Fourier transform.
I'm not sure I quite understand what you're aiming at with these questions, but there are certainly techniques in ML based on thinking of functions as vectors. The first one that comes to mind is Anyboost [1], which views boosting as doing "gradient descent" in function space - where each "gradient step" is not a typical vector (as you'd see e.g in neural nets) but a function, corresponding in practice to a base classifier. Another that is very popular are gaussian processes - one way to think about them is as modeling functions as samples from an infinite-dimensional gaussian (at least some of them).
I’d guess that it would be factored as “nonlinearity”, which might be characterized as some form of harmonic distortion, analogous to clipping nonlinearity of finite-ranged systems?
Perhaps some conjugate relation could be established between finite-range in one domain and finite-resolution in another, in terms of the effect such nonlinearities have on the spectral response.
I see this a lot with math concepts as they begin to get more abstract: strange visualizations to try to build intuition. I think this is ultimately a dead-end approach which misleads rather than enlightens.
To me, the proper way of continuing to develop intuition is to abandon visualization entirely and start thinking about the math in a linguistic mode. Thus, continuous functions (perhaps on the closed interval [0,1] for example) are vectors precisely because this space of functions meet the criteria for a vector space:
* (+) vector addition where adding two continuous functions on a domain yields another continuous function on that domain
* (.) scalar multiplication where multiplying a continuous function by a real number yields another continuous function with the same domain
* (0) the existence of the zero vector which is simply the function that maps its entire domain of [0,1] to 0 (and we can easily verify that this function is continuous)
We can further verify the other properties of this vector space which are:
* associativity of vector addition
* commutativity of vector addition
* identity element for vector addition (just the zero vector)
* additive inverse elements (just multiply f by -1 to get -f)
* compatibility of scalar multiplication with field multiplication (i.e a(bf) = (ab)f, where a and b are real numbers and f is a function)
* identity element for scalar multiplication (just the number 1)
* distributivity of scalar multiplication over vector addition (so a(f + g) = af + ag)
* distributivity of scalar multiplication over scalar addition (so (a + b)f = af + bf)
So in other words, instead of trying to visualize an infinite-dimensional space, we’re just doing high school algebra with which we should already be familiar. We’re just manipulating symbols on paper and seeing how far the rules take us. This approach can take us much further when we continue on to the ideas of normed vector spaces (abstracting the idea of length), sequences of vectors (a sequence of functions), and Banach spaces (giving us convergence and the existence of limits of sequences of functions).
> I see this a lot with math concepts as they begin to get more abstract: strange visualizations to try to build intuition. I think this is ultimately a dead-end approach which misleads rather than enlightens.
Isn't this how people arrived at most of these concepts historically, how the intuition arose that these are meaningful concepts at all?
For example, the notion of a continuous function arose from a desire to explicitly classify functions whose graph "looks smooth and unbroken". People started with the visual representation, and then started to build a formalism that explains it. Once they found a formalism that was satisfying for regular cases, they could now apply it to cases where the visual intuition fails, such as functions on infinite-dimensional spaces. But the concept of a continuous function remains tied to the visual idea, fundamentally that's where it comes from.
Similalrly with vectors, you have to first develop an intuition of the visual representation of what vector operations mean in a simple to understand vector space like Newtonian two-dimensional or three-dimensional space. Only after you build this clean and visual intuition can you really start understanding the formalization of vectors, and then start extending the same concepts to spaces that are much harder or impossible to visualize. But that doesn't mean that vector addition is an arbitrary operation labeled + - vector addition is a meaningful concept for spatial vectors, one that you can formally extend to other operations if they follow certain rules while retaining many properties of the two-dimensional case.
the notion of a continuous function arose from a desire to explicitly classify functions whose graph "looks smooth and unbroken"
This notion falls down when you get to topology where you have continuous functions on topological spaces (which need not have any concept of distance nor even "smoothness"), since a topology can be defined on a finite (or infinite) set (of objects which may not even be numbers).
But that doesn't mean that vector addition is an arbitrary operation labeled + - vector addition is a meaningful concept for spatial vectors, one that you can formally extend to other operations if they follow certain rules while retaining many properties of the two-dimensional case
Vector addition need not even look like addition. For example, the positive real numbers can be defined as a vector space over the real numbers, with:
* (+) vector addition: u + v = uv (adding two vectors by multiplying two positive real numbers)
* (.) scalar multiplication: av = v^a
* (0) zero vector: 1 (the identity for multiplication)
Now we can verify some of the vector space axioms:
* let a be a scalar and u, v be vectors, then: a(u + v) = a(uv) = (uv)^a = (u^a)(v^a) = au + av, thus distributivity of scalar multiplication over vector addition holds
* let a, b be scalars and v be a vector, then: (a + b)v = v^(a + b) = (v^a)(v^b) = av + bv, thus distributivity of scalar multiplication over scalar addition holds
The rest can also be similarly verified but you get the picture.
Another weird vector space is the set of spanning subgraphs of a finite, simple, undirected graph over the finite field F[2] (which yields only 0 and 1 as scalars). In this one the idea of vector addition between subgraphs G + H is about determining whether two vertices are adjacent in one or the other of G or H, or adjacent in neither or both. This isn't really like addition at all, so none of the intuitions you might develop for vector addition in a Euclidean two-dimensional space would apply at all.
None of these examples contradict what I've said. The historical fact is that continuous functions and vector addition were discovered as interesting mathematical properties based on certain visual intuitions. Only later were did they come to be extended to more general notions applicable to wildly different conditions.
What I claim in addition is that it's still useful for your intuition to understand this history and the leaps that were made by extremely talented mathematicians of the past who came up with these formalizations of intuitive properties.
I'd also claim that they couldn't have ever arrived at the current formal systems if they hadn't started with certain intuitions for simple systems.
You're arguing for including historical motivation in order to make a definition seem more immediately useful and intuitive. I agree with that.
What the person you replied to is suggesting, however, is that this can only get you so far. At some you have to sit down and study the formal definition and understand how it relates to the examples you've seen so far. Otherwise it's very easy to have your intuition lead you astray.
I'd that's what they're arguing, then I fully agree. My fear was that they're arguing that the formal definition is the only thing that you should spend time on, that the original motivation is irrelevant - this is what I was trying to argue against.
Funny, I agree that visualizations aren't that useful after a point, but when you said "start thinking about the math in a linguistic mode" I thought you were going to describe what I do, but then you described an entirely different thing! I can't learn math the way you described at all: when things are described by definitions, my eyes glaze over, and nothing is retained. I think the way you are describing filters out a large percentage of people who would enjoy knowing the concepts, leaving only the people whose minds work in that certain way, a fairly small subset of the interested population.
My third way is that I learn math by learning to "talk" in the concepts, which is I think much more common in physics than pure mathematics (and I gravitated to physics because I loved math but can't stand learning it the way math classes wanted me to). For example, thinking of functions as vectors went kinda like this:
* first I learned about vectors in physics and multivariable calculus, where they were arrows in space
* at some point in a differential equations class (while calculating inner products of orthogonal hermite polynomials, iirc) I realized that integrals were like giant dot products of infinite-dimensional vectors, and I was annoyed that nobody had just told me that because I would have gotten it instantly.
* then I had to repair my understanding of the word "vector" (and grumble about the people who had overloaded it). I began to think of vectors as the N=3 case and functions as the N=infinity case of the same concept. Around this time I also learned quantum mechanics where thinking about a list of binary values as a vector ( |000> + |001> + |010> + etc, for example) was common, which made this easier. It also helped that in mechanics we created larger vectors out of tuples of smaller ones: spatial vector always has N=3 dimensions, a pair of spatial vectors is a single 2N = 6-dimensional vector (albeit with different properties under transformations), and that is much easier to think about than a single vector in R^6. It was also easy to compare it to programming, where there was little difference between an array with 3 elements, an array with 100 elements, and a function that computed a value on every positive integer on request.
* once this is the case, the Fourier transform, Laplace transform, etc are trivial consequences of the model. Give me a basis of orthogonal functions and of course I'll write a function in that basis, no problem, no proofs necessary. I'm vaguely aware there are analytic limitations on when it works but they seem like failures of the formalism, not failures of the technique (as evidenced by how most of them fall away when you switch to doing everything on distributions).
* eventually I learned some differential geometry and Lie theory and learned that addition is actually a pretty weird concept; in most geometries you can't "add" vectors that are far apart; only things that are locally linear can be added. So I had to repair my intuition again: a vector is a local linearization of something that might be macroscopically, and the linearity is what makes it possible to add and scalar-multiply it. And also that there is functionally no difference between composing vectors with addition or multiplication, they're just notations.
At no point in this were the axioms of vector spaces (or normed vector spaces, Banach spaces, etc) useful at all for understanding. I still find them completely unhelpful and would love to read books on higher mathematics that omit all of the axiomatizations in favor of intuition. Unfortunately the more advanced the mathematics, the more formalized the texts on it get, which makes me very sad. It seems very clear that there are two (or more) distinct ways of thinking that are at odds here; the mathematical tradition heavily favors one (especially since Bourbaki, in my impression) and physics is where everyone who can't stand it ends up.
I can't learn math the way you described at all: when things are described by definitions, my eyes glaze over, and nothing is retained. I think the way you are describing filters out a large percentage of people who would enjoy knowing the concepts, leaving only the people whose minds work in that certain way, a fairly small subset of the interested population.
If you told me this in the first year of my math degree I would have included myself in that group. I think you’re right that a lot of people are filtered out by higher math’s focus on definitions and theorems, although I think there’s an argument to be made that many people filter themselves out before really giving themselves the chance to learn it. It took me another year or two to begin to get comfortable working that way. Then at some point it started to click.
I think it’s similar to learning to program. When I’m trying to write a proof, I think of the definitions and theorems as my standard library. I look at the conclusion of the theorem to prove as the result I need to obtain and then think about how to build it using my library.
So for me it’s a linguistic approach but not a natural language one. It’s like a programming language and the proofs are programs. Believe it or not, this isn’t a hand-wavey concept either, it’s a rigorous one [1].
> When I’m trying to write a proof, I think of the definitions and theorems as my standard library. I look at the conclusion of the theorem to prove as the result I need to obtain and then think about how to build it using my library.
fwiw, this is exactly the thing that you when you're trying to formally prove some theorem in a language like Lean.
> So I had to repair my intuition again: a vector is a local linearization of something that might be macroscopically, and the linearity is what makes it possible to add and scalar-multiply it. And also that there is functionally no difference between composing vectors with addition or multiplication, they're just notations.
Except none of this is true of vectors in general, although it might be true of very specific vector spaces in physics that you may have looked at. Matrices or continuous functions form vector spaces where you can add any vectors, no matter how far apart. Maybe what you're referring to is that differentiability allows us to locally approximate nonlinear problems with linear methods but that doesn't mean that other things aren't globally linear.
I also don't understand what you mean by "no difference between composing vectors with addition or multiplication", there's obviously a difference between adding and multiplying functions, for example (and vector spaces in which you can also multiply are another interesting structure called an algebra).
That's the problem if you just go from intuition to intuition without caring about the formalism. You may end up with the wrong understanding.
The vector space axioms in the end are nothing more than saying: here's a set of objects that you can add and scale and here's a set of rules that makes sure these operations behave like they're supposed to.
You have unfortunately written off everything I said by assuming the most naive understanding of it.
The general theme is that I am interested in the metaphysical concept of vectors, not the thing that human mathematicians have labeled vectors. The universe doesn't care if you write ax+by or x^a y^b, hence addition vs multiplication is just a choice of coordinate system. And matrices and functions are vector spaces sure, but out in the world, when they show up in modeling things, they are local linearizations of curved things. Every linear algebra is (inevitably) a local point in a nonlinear one, as far as I can tell. Not in a formal sense, but in the sense that when you go out into the world and find them, it turns out to be the case.
The general theme is: I don't want to spend my life mastering the rigor of these simplistic models so that I can do it intuitively (in Tao's sense); I want to use them to learn intuition of the things that they are simplistic models of, and then master that.
> The general theme is that I am interested in the metaphysical concept of vectors, not the thing that human mathematicians have labeled vectors.
That's fine but then you shouldn't be surprised that you can't read higher level mathematics textbooks, because those are not about metaphysics.
The rest of what you wrote is... just not true. Matrices are used in plenty of areas where they are not mere approximations (e.g. cryptography), and spaces of functions remain vector spaces even when the functions themselves are not linear (because there's a difference between requiring that f(x+y)=f(x)+f(y) and that (f+g)(x) = f(x)+g(x)).
> and I was annoyed that nobody had just told me that because I would have gotten it instantly.
Right?! In my path through the physics curriculum, this whole area was presented in one of two ways. It went straight from "You don't need to worry about the details of this yet, so we'll just present a few conclusions that you will take on faith for now" to "You've already deeply and thoroughly learned the details of this, so we trust that you can trivially extend it to new problems." More time in the math department would have been awfully useful, but somehow that was never suggested by the prerequisites or advisors.
oh, my point was the opposite of that. The math department was totally useless for learning how anything made sense. I only understood linear algebra when I took quantum mechanics for instance. The math department couldn't be bothered to explain anything in any sort of useful way; you were supposed to prove pointless theorems about things you didn't understand.
I did get a lot of that in the lower level math courses, where it kinda felt like the math faculty were grudgingly letting in the unwashed masses to learn some primitive skills to apply [spit] to their various fields, and didn't really give a shit if anybody understood anything as long as the morons could repeat some rituals for moving x around on the page. I didn't really understand integrals until the intermediate classical mechanics prof took an hour or two to explain what the hell we had been doing for three semesters of calculus.
But when I did go past the required courses and into math for math majors, things got a lot better. I just didn't find that out until I was about to graduate.
Let's see: Let A be a non-empty set, N
the set of positive whole numbers, and X
the set of all functions f
f: A --> N
with usual notation.
Assume as is common, the scalars are the
set of real numbers, but the set of complex
numbers will also do.
So, is X a vector space and, thus, each
f in X a vector?
No, since -f is not in X. Neither is
(1/2)f.
Some references (with TeX markup):
Paul R.\ Halmos,
{\it Finite-Dimensional Vector Spaces, Second Edition\/}
linear algebra treated as functional analysis.
Walter Rudin,
{\it Real and Complex Analysis\/}
with Lebesgue integration and, then,
Banach and Hilbert vector spaces.
Walter Rudin,
{\it Functional Analysis\/}
with Fourier theory.
Jacques Neveu,
{\it Mathematical Foundations of the Calculus of Probability\/}
with random variables, that is,
functions from a probability space to,
usually, the set of real numbers with
convergence results, building on the
work A. Kolmogorov building on the work of
H. Lebesgue.
Functions with arbitrary codomains are not vectors, but when your codomain is some field, they are.
This is what the article is very explicitly about. I guess you can quibble that the title is imprecise, but it's just a title and the article makes it clear.
In all those books I listed, and more, e.g., on axiomatic set theory and other foundations, never saw definition or mention of codomain. So, the term is obscure. Readers are supposed to guess at the meaning of obscure terms?
What I did was follow, as in the references, long established convention, that for a function to be a vector at least it had to be in a vector space where (1) can multiply a function by a number (e.g., reals or complex) and (2) add two functions and still get a function in the vector space. To be general, I omitted metrics, inner products, topologies, convergence, probability spaces, and more.
Or, as in the references I gave, math talks about vector spaces and vectors, and each vector is in a vector space. The references are awash in definitions of vector spaces with (1) and (2) and much more.
Computing is awash in indexes for data, e.g., B-trees, SQL (structured query language) operations on relational data bases, addressing in central processors, collection classes in Microsoft's .NET, REDIS, and calling all such also functions confuses established material, conventions, and understanding.
Not just when the codomain is a field, but more generally when the codomain is itself a vector space. The former is a special case of the latter where you construct a 1D vector space from a field.
An engineering, signal processing extension/perspective:
An infinite sequence approximates a general function, as described in the article (see the slider bar example). In signal processing applications, functions can be considered (or forced) to be bandlimited so a much lower-order representation (i.e. vector) suffices:
- The subspace of bandlimited functions is much smaller than the full L^2 space
- It has a countable orthonormal basis (e.g., shifted sinc functions)
- The function can be written as (with sinc functions):
This idea is also at the core of the Finite Element Method (and other Galerkin methods, such as DG) which are probably the most widespread methods to approximate the solutions of partial differential equations (PDEs) used everywhere in physics and engineering.
Basically, define a finite dimensional function space V_N of dimension N, in such a way that you could grow N to be arbitrarily large. Solve not the PDE (originally defined over an infinite dimensional function space V, such as H^1), but its discretization as if it dealt only with functions in V_N rather than all functions. The PDE is then simply a linear system, easy to solve. And you can prove, for instance in the case of elliptic PDEs, that the solution to the discrete problem is the orthogonal projection of the true solution of the PDE (in V) onto V_N (Céa's Lemma). Finally, you can produce estimations of the error this projection incurs as a function of N, and thus give theoretical guarantees that the algorithm converges to the true solution as N goes to infinity.
(N in this case is the number of vertices in a mesh that is used to define the basis functions of V_N)
Yeah, and it's such a boring thing to write about.
Given an vector space V with (+, ), you can define the vector space over functions F whose codomain is V and where F.+ and F. both take two functions as argument and return another function applying V.+ or V.* on the result. All the linear algebra properties come from the original vector space. Hence it is boring.
I will need to read through the rest of the article later, but the initial intuition building is a bit sloppy. None of those vectors drawn in the initial examples belong to the same vector space. Vectors need to emanate from the same origin to be considered as part of the same vector space.
Can you elaborate? Which vector space(s) do these vectors belong to, as drawn?
If anything, this just highlights the sloppiness of these types of illustrations. Things aren't precise enough, in my opinion, for the illustrations to do anything except confuse.
In the mathematical sense, a vector is nothing but an object that can be added to other objects and scaled by some field (with some reasonable properties attached).
In physics, a vector is often more specifically something with magnitude and direction. This still doesn't mean that it needs to be anchored at the origin. Vectors that are anchored at the origin are IIRC called position vectors, but mathematically, if you translate them away from the origin they're still the same vector.
The graph of a function f: X -> Y is the set {(x, f(x)) | x in X}. It is much more clear and precise to associate elements of the graph with vectors such that the 0 vector is identified with the R^2 origin, and then points in R^2 are identified with vectors. Then there is a mapping between vectors in this vector space to the graph, i.e., to points (x, f(x)).
> In physics, a vector is often more specifically something with magnitude and direction.
Physics is sloppy. :) This is not a general description of a vector, where vector is an element of a general vector space. Not all vector spaces have a norm, which is required for magnitude to make any sense.
> but mathematically, if you translate them away from the origin they're still the same vector.
Right, and you cannot always translate vectors without more machinery, such as parallel transport.
> Physics is sloppy. :) This is not a general description of a vector, where vector is an element of a general vector space. Not all vector spaces have a norm, which is required for magnitude to make any sense.
Fine, then let's be precise. An element of the vector space R^n is nothing more than a function from {1,...,n} to R. And every n-dimensional R-vector space is isomorphic to it.
How you wanna draw this in a coordinate system is up to you. It is customary to identify vectors with points on that coordinate system and then equivalently with an arrow pointing from the origin to that point. In that case the zero vector is the origin, or an arrow from the origin to itself.
It is equally customary to use vectors as displacement, i.e. a directed difference between two points. In that case, the vector that you can't actually draw because it has zero length is your zero element. Now your arrows don't have to be anchored at the origin anymore.
Both of these spaces are of course isomorphic.
If your vector space is infinite-dimensional it's of course not gonna be really possible to draw anymore.
Vector spaces have an addition identity normally referred to as the 0 vector.
When drawing vectors, such as on a 2D space, the origin is identified with this 0 vector, and then the points in the 2D space can be associated with the vectors in the vector space.
The author seems to be a great educator and computer scientist, much respect to his work. But from what I can gather, although I'd love to study more infinite sized matrices, he proved / showed nothing in this article. What he wrote is not true at all, they are only analogies and not rigorous maths. Functions are not vectors. But finite polynomials are vectors yes, this is trivial.
It's not a particularly interesting proof, but the author does prove that real valued functions are vectors. The bulk of the article is less about proofs, and more about showing how the above result is useful.
Vectors in the way he talks about in the beginning. With indices (and then extending to "In higher dimensions, vectors start to look more like functions!"). Of course if you use the general meaning of every word, vectors are functions and functions are vectors, and this article shouldn't then have anything interesting to talk about.
how the above result is useful
It doesn't seem useful at all to me, the examples in the article are not that interesting. On the contrary it is more confusing than anything to apply linear algebra to real valued functions.
I understand that. The crux of the article is supposedly being able to apply what we learned with easy small vector space to infinite function vector space : we can't. But he's taking the example of polynomials and analytical functions which are MUCH smaller spaces than functions in general.
Maybe you could envision that I learned maths a similar way than you and that's why I have problem with this material. It's trying to portray complicated space as simple, they're not
Finite degree polynomials are vectors yes. Polynomials is a typical example you study when learning about linear algebra. Doesn't say anything about real functions in general though, I don't think any linear algebra course should make the analogies made in this article, that'd be confusing
I think it is "vector" taken in the way the author wrote about it / showed illustrations in the article.
For the second sentence, he's right, we could also write (wrongly) an article titled "Functions are Sequences" and (try to) apply what we know about dealing with countable sequences to functions
The genre of this article is not pedagogical, really. One usually learns these techniques in the course of a particular field like physics, electrical engineering, or theoretical chemistry. This article is best thought of as "a story you've seen before, but told from the beginning / ground up, with a lot of the connections to other topics and examples laid out for you". For that purpose, it's excellent, perhaps the best I've ever seen. It might also whet the appetite of a novice, but it's not really for that.
This perspective is crucial for understanding signal processing, machine learning, and quantum mechanics. Viewing functions as vectors enables practical techniques like Fourier transforms and kernel methods that underlie many modern technologies.
> "The material is not motivated." Not motivated? Judas just stick a dagger in my heart. This material needs no motivation. Just do it. Faith will come. He's teaching you analysis. Not selling you a used car. By the time you are ready to read this book you should not need motivation from the author as to why you need to know analysis. You should just feel a burning in you chest that can only be quenched by arguments involving an arbitrary sequence {x_n} that converges to x in X.
If you need "practical applications" for some part of math to have value to you, then large parts of math will not be for you. That's fine, but that's also something you should accept and internalize: math is already its own application, we dig through it in order to better understand it, and that understanding will (with rather advanced higher education) be applicable to other fields, which in turn may have practical uses.
Those practical uses are someone else's problem to solve (even if they rely on math to solve them), and they can write their own web pages on how functions as vectors help solve specific problems in a way that's more insightful than using "traditional" calculus, and get those upvoted on HN.
But this link has a "you must be this math to ride" gate, it's not for everyone, and that's fine. It's a world wide web, there's room for all levels of information. You need to already appreciate the problems that you encountered in non-trivial calculus to appreciate this interpretation of what a function even is and how to exploit the new power that gives you.
I don't see any such "math gate" on this link. Also, this math does have practical applications, but they're not mentioned until very late in the article.
My suggestion is that briefly mentioning them up front might be nice. I didn't mean to start a big argument about it.
Yet some parts of math are 'preferred' over others, in that most 'serious' mathematicians would rather read 100 pages about functional analysis than 100 pages of meandering definitions from some rando trying to solve the Collatz conjecture.
Some people would like to have a filter for what to spend their time on, better than "your elders before you have deemed these ideas deeply important". One such filter is "Can these ideas tell us nontrivial things about other areas of math?" That is, "Do they have applications?"
Short of the strawman of immediate economic value, I don't think it's wrong to view a subject with light skepticism if it seemingly ventures off into its own ivory tower without relating back to anything else. A few well-designed examples can defuse this skepticism.
you're supposed to get that the cynical lens you're applying here doesn't fit - if you aren't intrinsically motivated to read this stuff then it's not for you. which is fine btw because (functional) analysis isn't a required class.
The author asserts vectors are functions, specifically a function that takes an index and returns a value. He notes that as you increase the number of indices, a vector can contain an arbitary function (he focuses on continuous, real-valued functions).
It's fun to simulate one thing with another, but there is a deeper and more profound sense in which vectors are functions in Clifford Algebra, or Geometric Algebra. In that system, vectors (and bi-vectors...k-vectors) are themselves meaningful operators on other k-vectors. Even better, the entire system generalizes to n-dimensions, and decribes complex numbers, 2-d vectors, quaternions, and more, essentially for free. (Interestingly, the primary operation in GA is "reflection", the same operation you get in quantum computing with the Hadamard gate)
Thanks for mentioning this. I think I encountered Geometric Algebra in a video related to making video games. It's this one (Why you can't multiply vectors): https://www.youtube.com/watch?v=htYh-Tq7ZBI
This previous thread was also good: Functions are vectors - https://news.ycombinator.com/item?id=36921446 - July 2023 (120 comments)
The jump from spherical harmonics to eigenfunctions on a general mesh, and the specific example mesh chosen, might be the finest mathematical joke I've seen this decade.
Related to this footnote in TFA?
>If you’re alarmed by the fact that the set of all real functions does not form a HILBERT SPACE, you’re probably not in the target audience of this post."
Video: https://youtu.be/q8gng_2gn70?t=8m3s
Thanks to
https://news.ycombinator.com/item?id=44481933
If you're wondering what a Hilbert space, know that you're in good company.
> Dr. von Neumann, ich möchte gerne wissen, was ist denn eigentlich ein Hilbertscher Raum? (Dr. von Neumann, I'd would really like to know, just what exactly is a Hilbert space?)
Asked to John von Neumann to David Hilbert at a lecture.
https://ncatlab.org/nlab/show/Hilbert+space#fn:1
I'd like to add, as a physicist by training, that anything can be a Hilbert space if you wish hard enough. You can even use results about countable vector spaces if you need them!
Would you explain the joke for the rest of us?
It's quietly reversing the traditional "We approximate the cow to be a sphere" and showing how the spherical math can, in fact, be generalized to solutions on the cow.
oh. I did not interpret that blob as a cow. Thanks.
In the Geometry Processing paragraph (second last, above Further Reading), there are clearly cow shapes: four legs, head with ears and horns.
Spherical Haromics approximating Spherical Cows?
assume spherical cow
Isn't this the opposite way ? Vectors are functions whose input space are discrete dimensions. Let's not pretend going from natural numbers to real is "simple", reals numbers are a fascinating non-obvious math discovery. And also the passage from a few numbers to all natural numbers (aleph0) is non obvious. So basically we have two alephs passages to transforms N-D vectors as functions over reals.
Vectors are not necessarily discrete-domained. Anything that satisfies the vector space properties is a vector.
I agree but I'm operating under the assumption of the article
That's not really the assumption of the article, that's the assumption the article makes of the reader's knowledge of vectors as they begin reading. The article is trying to make the point that functions are also vectors, strictly speaking, and give some intuition as to what that means.
Basically it's an introduction to "functional analysis", the field of math that looks at functions like vectors. The term is in there sneakily, but this is really what the article is all about.
I had classes on this subject and we were encouraged to build upon linear algebra intuition (much like the author does), with some caveats (such as a bounded closed set is not necessarily compact in infinite dimension). Things break down, as the author hints at when they mention they didn't prove the Laplacian to be self-adjoint but only symmetric (this has to do with the operator's domain, a concern that doesn't exist for linear operators on finite dimensional spaces)... but it's still a very convenient way to think and talk about things, conceptually.
Linear algebra isn’t limited to discrete-dimensional vector spaces. Or what do you mean?
See my other comment sibling.
And he's starting from the assumption vectors are finite (cf. the article)
He does not assume anything! Any assumption is in your head only. Of course he starts from the specific type of vector spaces that's the most familiar to readers. But then he shows that there's nothing that requires a vector space to have a finite, or even countably infinite, dimension. What matters are the axioms.
The article :
So the goal is to impress the reader by letting him believe we will easily apply our easy linear algebra to real -> real functions. But we can't.> So the goal is to impress the reader by letting him believe we will easily apply our easy linear algebra to real -> real functions. But we can't.
When you only have a finite number of functions (and the space they span), then you can apply your finite dimensional linear algebra, since n dimensional vector spaces over a field are isomorphic. You absolutely can gain intuition for real functions from arrows on a paper type vectors.
E.g. the article mentions the Cauchy-Schwartz inequalities for functions, that's something you can intuit when you imagine our functions being little arrows on a paper.
What intuition can you gain ? This applies to any field, not just functions. You're gaining intuition for a more general concept and not what makes real->real functions different from others maths object
Vectors are an abstract notion. If you have two sets and two operations that satisfy the definition of a vector space, then you have a vector space; and we refer to elements of the vector set as "vectors" within that vector space.
The observation here is that set of real value functions, combined with the set of real numbers, and the natural notion of function addition and multiplication by a real number satisfies the definition of a vector space. As a result all the results of linear algebra can be applied to real valued functions.
It is true that any vector space is isomorphic to a vector space whose vectors are functions. Linear algebra does make a lot of usage of that result, but it is different from what the article is discussing.
I agree but we're using functions for different things here. Yes some specific families of functions can be treated as vector spaces. In this article it seems like the author is pretending to take all real->real functions and treating them as if they are a vector space, whatever the content of the functions, quote :
and later he admits it is impossible They are uncountable because they are aleph1It is not required for vector spaces to have a basis. As it turns out, the claim that every vector space has a basis is equivalent to the axiom of choice, which seems well beyond the scope of the article.
However, the particular vector space in question (functions from R to R) does have a basis, which the author describes. That basis is not as useful as a basis typically is for finite dimensional (or even countably unfitine dimensional) vector spaces, but it still exists.
But it's the article talking about vectors as a sequence of reals and having a basis, then extending that to infinite sequences of reals. The author is playing on multiple definitions of vector to produce a "woaw that's cool" effect, and that's bad maths
There is only one definition of "vector space" (up to isomorphism anyway), and that's what the author uses. You'll note that he doesn't talk about bases at all, the assumption of a basis is entirely in your mind. The entire point of the article is that the ℝ→ℝ function space is a vector space. A vector space is not required to have a basis, but assuming the axiom of choice, every vector space does have (at least) one, including that of ℝ→ℝ functions.
The choice of basis is very important for the applications. It doesnt matter that function could be vector spaces in theory without constructing a base, the article is about "hey look at functions they're as easy to operate in as this thing you already know with 3D vectors"
You are misreading the article. At a guess, you have come into it "knowing what a vector is" from physics or computer science. Specifically, a vector is not "a sequence of reals", it is any object that obeys a certain set of properties. He is not "extending" your definition, he's trying to teach you the actual definition, which is more general than the examples you are used to.
(Imagine/remember what it feels like when children first learn that the integers aren't the only number, there are also fractions, then irrationals, then complex numbers...this is a very similar situation).
With that in mind, you may want to reread the text and pay attention to the definitions he is using, and not assume that your definitions are the whole story.
I learned vector spaces in maths and I'm aware of your definition, that's how I learned it. I'm also aware that the same word can be used with different meanings depending on the context.
Imagine now an article saying "complex numbers are just natural numbers, you can apply euclidean division to them the exact same way"The set of all real->real functions is still a vector space.
This vector space also has a basis (even if it is not as useful): there is a (uncountably infinite) subset of real->real functions such that every function can be expressed as a linear combination of a finite number of these basis functions, in exactly one way.
There isn't a clean way to write down this basis, though, as you need to use Zorn's lemma or equivalent to construct it.
If you restrict yourself to Lebesque-integrable functions, can’t you take the complex Fourier transform of the function and call the terms of the Fourier series a basis, with the coefficients being the components of the vector of the function? This is a bit above my current mathematical paygrade, so forgive me if I’m not expressing the idea accurately but I’m learning a lot both from the article and the ensuing discussion - hopefully you understand what I’m getting at.
I think what I may be asking is “Does the complex Fourier transform make a Hilbert space?” but I might be wrong both about that and about that being the right question.
Sorry, I couldn't find the page in English, but what you're talking about is a Hilbert basis: https://fr.wikipedia.org/wiki/Base_de_Hilbert . There is a paragraph on this in the orthonormal basis English page: https://en.wikipedia.org/wiki/Orthonormal_basis
Another example is the eigenvectors of linear operators like the Laplacian. Recall how, in finite dimension, the eigenvectors of a full rank operator (matrix) form an orthonormal basis of the vector space. There is a similar notion in infinite dimension. I can't find an English page that covers this very well, but there's a couple of paragraphs in the Spectral Theorem page (https://en.wikipedia.org/wiki/Spectral_theorem#Unbounded_sel... ). The article linked here also touches on this.
Regarding your last sentence, one thing to note is that having a basis is not what makes you a Hilbert space, but rather having an inner product! In fact, to get the Fourier coefficients, you need to use that inner product.
That's awesome info thank you so much. Reading it, a Hilbert basis is exactly what I am talking about. It's always exciting when my intuition guides me on the right path. I'll check out the Spectral theorem page also.
No, and this is where this formal notion of basis I mentioned unfortunately diverges from what is perhaps more useful in practice.
You can represent any function f: [-pi, pi] -> R as an infinite sum
for some coefficients a_k and b_k as long as f is sufficiently nice (I don't remember the exact condition, sorry).This is very useful, but the functions sin(x), sin(2x), ... , cos(x), cos(2x), ... don't constitute a basis in the formal sense I mentioned above as you need an infinite sum to represent most functions. It is still often called a basis though.
Thanks for that. That’s the trigonometric Fourier series. A complex Fourier series (which is what I mentioned) is equivalent and works similarly except that the terms are all uniform, so
You can derive one from the other by using the identities I specifically mentioned the complex series because I didn’t like the fact that the alternating terms use a different trig function and it seemed weird to me to have every second dimension in a space be different in that way but they are equivalent.The convergence criteria for Fourier series vary depending on how strongly you need convergence but I think basically if a function is differentiable on the interval you care about then the Fourier series provably converges on that interval and otherwise if it has jump discontinuities and that sort of thing, then depending on whether it is square-integrable or a bunch of other properties) you can prove weaker forms of convergence (absolute, pointwise etc).
To address your comment I don’t see why an infinite sum prevents something being a basis. In fact I would specifically say that can’t be true because then there would never be a basis for any infinite-dimensional space- any time you want to take an inner product in such a space you need an infinite sum, and you need such an inner product to construct the basis. A sibling comment pointed me in the direction of a Hilbert basis, which seems to be what I was thinking of.
You can’t, because a Fourier series is not a linear combination.
Linear combination of what ?
I'd love to read more about that, he's not talking about that at all in this article though
If you're familiar with Zorn's Lemma, the construction is just to order bases by inclusion and to consider chains created by noting that there must be an independent dimension and adding it inductively. You can upper bound each of these chains by unioning the members of the chain (which preserves linear independence). By Zorn's Lemma that means there is a maximal linearly independent system and if an element existed outside of that system's span it would contradict that maximality.
He doesn't need to talk about it (though you might like to look up the notorious Hilbert's Basis Theorem); it happens to be the case that any vector space has a basis, but even if you don't know that, a vector space is still a vector space and its elements are still vectors.
The choice of basis is very important for the applications, which is what the article is about
> Vectors are functions whose input space are discrete dimensions.
This is _also_ true. But the fact that finite dimensional real vectors can be viewed as a special case of set-theoretic functions or as a special case of real functions on a discrete finite space is probably less useful than the opposite: the set of real functions have a vector-space structure, and you can use all the neat theorems about (finite or infinite dimensional) vector spaces.
Well, at least the article is about the latter direction.
A few questions occur to me while reading this, which I am far from qualified to answer:
- How much of this structure survives if you work on "fuzzy" real numbers? Can you make it work? Where I don't necessarily mean "fuzzy" in the specific technical sense, but in any sense in which a number is defined only up to a margin of error/length scale, which in my mind is similar to "finitism", or "automatic differentiation" in ML, or a "UV cutoff" in physics. I imagine the exact definition will determine how much vectorial structure survives. The obvious answer is that it works like a regular Fourier transform but with a low-pass filter applied, but I imagine this might not be the only answer.
- Then if this is possible, can you carry it across the analogy in the other direction? What would be the equivalent of "fuzzy vectors"?
- If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
- The argument arrives at fourier analysis by considering an orthonormal diagonalization of the Laplacian. In linear algebra, SVD applies more generally than diagonalizations—is there an "SVD" for functions?
Convolution with dirac delta will give you an exact sample of f(0), and in principle a whole signal could be constructed as a combination of delayed delta signals - but we can't realize an exact delta signal in most spaces, only approximations.
As a result we get finite resolution and truncation of the spectrum. So "Fourier analysis with pre-applied lowpass filter" would be analysis of sampled signals, the filter determined by the sampling kernel (delta approximator) and properties of the DFT.
But so long as the sampling kernel is good (that is the actual terminology), we can form f exactly as the limit of these fuzzy interpolations.
The term "resolution of the identity" is associated with the fact that delta doesn't exist in most function spaces and instead has to be approximated. A good sampling kernel "resolves" the missing (convolutional) identity. I like thinking of the term also in the sense that these operators behave like the identity if it were only good up to some resolution.
1. Numerical methods for solving differential and integral equations are algorithms for solving algebraic equations (vector solutions) that arise from discretizing infinite-dimensional operator equations (function solutions). When we talk about whether these methods work, we usually do so in terms of their consistency and stability. There is a multistage things that happens here: we start by talking about the well-posedness of the original equation (e.g. the PDE), then the convergence of the mathematical discretization, and then examine what happens when we try to program this thing on a computer. Usually what happens is these algorithms will get implemented "on top" of numerical linear algebra, where algorithms like Gaussian elimination, and different iterative solvers, have been studied very carefully from the perspective of floating point rounding errors etc. This kind of subsumes your concern about "fuzzy" real numbers. Remember that in double precision, if the number "1.0" represents "1 meter", then machin epsilon is atomic scale. So, frequently, you can kind of assume the whole process "just works"...
2/3. I'm not really sure what you mean by these questions... But if you want to do "fourier analysis with a filter preapplied", you'd probably just work with within some space of bandlimited functions. If you only care around N Fourier modes, any time you do an operation which exceeds that number of modes, you need to chop the result back to down to size.
4. In this context, it's really the SVD of an operator you're interested in. In that regard, you can consider trying to extend the various definitions of the SVD to your operator, provided that you carefully think about all spaces involved. I assume at least one "operator SVD" exists and has been studied extensively... For instance, I can imagine trying to extend the variational definition of the SVD... and the algorithms for computing the SVD probably make good sense in a function space, too...
Your can replace the real numbers with the rational numbers and maintain all of the vector structure.
If you wanted something more quantized, you can pick some length unit, d, and replace the real numbers with {... -2d, -d, 0, d, 2d,... }. This forms a structure known as a "ring" with the standard notion of addition, subtraction, and multiplication (but no notion of division. Using this instead of R does lose the vector structure, but is still an example of a slightly more general notion of a "module". Many of the linear algebra results for vector spaces apply to modules as well.
> If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
If that is where you want to end up, you could pretty much start there. If you take all real value functions and apply a courier analysis with a low pass filter to each of them, the resulting set still forms a vector space. Although I don't see any particular way of arriving at this vector space by manipulating functions pre Fourier transform.
I'm not sure I quite understand what you're aiming at with these questions, but there are certainly techniques in ML based on thinking of functions as vectors. The first one that comes to mind is Anyboost [1], which views boosting as doing "gradient descent" in function space - where each "gradient step" is not a typical vector (as you'd see e.g in neural nets) but a function, corresponding in practice to a base classifier. Another that is very popular are gaussian processes - one way to think about them is as modeling functions as samples from an infinite-dimensional gaussian (at least some of them).
[1] https://proceedings.neurips.cc/paper_files/paper/1999/file/9...
I’d guess that it would be factored as “nonlinearity”, which might be characterized as some form of harmonic distortion, analogous to clipping nonlinearity of finite-ranged systems?
Perhaps some conjugate relation could be established between finite-range in one domain and finite-resolution in another, in terms of the effect such nonlinearities have on the spectral response.
A fuzzy vector is a Gaussian? Thinking of what it would be in 1, 2, 3 and n dimensions.
Discussion at the time https://news.ycombinator.com/item?id=36921446
The one place that I think the previous discussion lost something important, at least to me with functions.
The popular lens is the porcupine concept when infinite dimensions for functions is often more effective when thought of as around 8:00 in this video.
https://youtu.be/q8gng_2gn70
While that video obviously is not fancy, it will help with building an intuition about fixed points.
Explaining how the dimensions are points needed to describe a functions in a plane and not as much about orthogonal dimensions.
Specifically with fixed points and non-expansive mappings.
Hopefully this helps someone build intuitions.
I see this a lot with math concepts as they begin to get more abstract: strange visualizations to try to build intuition. I think this is ultimately a dead-end approach which misleads rather than enlightens.
To me, the proper way of continuing to develop intuition is to abandon visualization entirely and start thinking about the math in a linguistic mode. Thus, continuous functions (perhaps on the closed interval [0,1] for example) are vectors precisely because this space of functions meet the criteria for a vector space:
* (+) vector addition where adding two continuous functions on a domain yields another continuous function on that domain
* (.) scalar multiplication where multiplying a continuous function by a real number yields another continuous function with the same domain
* (0) the existence of the zero vector which is simply the function that maps its entire domain of [0,1] to 0 (and we can easily verify that this function is continuous)
We can further verify the other properties of this vector space which are:
* associativity of vector addition
* commutativity of vector addition
* identity element for vector addition (just the zero vector)
* additive inverse elements (just multiply f by -1 to get -f)
* compatibility of scalar multiplication with field multiplication (i.e a(bf) = (ab)f, where a and b are real numbers and f is a function)
* identity element for scalar multiplication (just the number 1)
* distributivity of scalar multiplication over vector addition (so a(f + g) = af + ag)
* distributivity of scalar multiplication over scalar addition (so (a + b)f = af + bf)
So in other words, instead of trying to visualize an infinite-dimensional space, we’re just doing high school algebra with which we should already be familiar. We’re just manipulating symbols on paper and seeing how far the rules take us. This approach can take us much further when we continue on to the ideas of normed vector spaces (abstracting the idea of length), sequences of vectors (a sequence of functions), and Banach spaces (giving us convergence and the existence of limits of sequences of functions).
> I see this a lot with math concepts as they begin to get more abstract: strange visualizations to try to build intuition. I think this is ultimately a dead-end approach which misleads rather than enlightens.
Isn't this how people arrived at most of these concepts historically, how the intuition arose that these are meaningful concepts at all?
For example, the notion of a continuous function arose from a desire to explicitly classify functions whose graph "looks smooth and unbroken". People started with the visual representation, and then started to build a formalism that explains it. Once they found a formalism that was satisfying for regular cases, they could now apply it to cases where the visual intuition fails, such as functions on infinite-dimensional spaces. But the concept of a continuous function remains tied to the visual idea, fundamentally that's where it comes from.
Similalrly with vectors, you have to first develop an intuition of the visual representation of what vector operations mean in a simple to understand vector space like Newtonian two-dimensional or three-dimensional space. Only after you build this clean and visual intuition can you really start understanding the formalization of vectors, and then start extending the same concepts to spaces that are much harder or impossible to visualize. But that doesn't mean that vector addition is an arbitrary operation labeled + - vector addition is a meaningful concept for spatial vectors, one that you can formally extend to other operations if they follow certain rules while retaining many properties of the two-dimensional case.
the notion of a continuous function arose from a desire to explicitly classify functions whose graph "looks smooth and unbroken"
This notion falls down when you get to topology where you have continuous functions on topological spaces (which need not have any concept of distance nor even "smoothness"), since a topology can be defined on a finite (or infinite) set (of objects which may not even be numbers).
But that doesn't mean that vector addition is an arbitrary operation labeled + - vector addition is a meaningful concept for spatial vectors, one that you can formally extend to other operations if they follow certain rules while retaining many properties of the two-dimensional case
Vector addition need not even look like addition. For example, the positive real numbers can be defined as a vector space over the real numbers, with:
* (+) vector addition: u + v = uv (adding two vectors by multiplying two positive real numbers)
* (.) scalar multiplication: av = v^a
* (0) zero vector: 1 (the identity for multiplication)
Now we can verify some of the vector space axioms:
* let a be a scalar and u, v be vectors, then: a(u + v) = a(uv) = (uv)^a = (u^a)(v^a) = au + av, thus distributivity of scalar multiplication over vector addition holds
* let a, b be scalars and v be a vector, then: (a + b)v = v^(a + b) = (v^a)(v^b) = av + bv, thus distributivity of scalar multiplication over scalar addition holds
The rest can also be similarly verified but you get the picture.
Another weird vector space is the set of spanning subgraphs of a finite, simple, undirected graph over the finite field F[2] (which yields only 0 and 1 as scalars). In this one the idea of vector addition between subgraphs G + H is about determining whether two vertices are adjacent in one or the other of G or H, or adjacent in neither or both. This isn't really like addition at all, so none of the intuitions you might develop for vector addition in a Euclidean two-dimensional space would apply at all.
None of these examples contradict what I've said. The historical fact is that continuous functions and vector addition were discovered as interesting mathematical properties based on certain visual intuitions. Only later were did they come to be extended to more general notions applicable to wildly different conditions.
What I claim in addition is that it's still useful for your intuition to understand this history and the leaps that were made by extremely talented mathematicians of the past who came up with these formalizations of intuitive properties.
I'd also claim that they couldn't have ever arrived at the current formal systems if they hadn't started with certain intuitions for simple systems.
You're arguing for including historical motivation in order to make a definition seem more immediately useful and intuitive. I agree with that.
What the person you replied to is suggesting, however, is that this can only get you so far. At some you have to sit down and study the formal definition and understand how it relates to the examples you've seen so far. Otherwise it's very easy to have your intuition lead you astray.
I'd that's what they're arguing, then I fully agree. My fear was that they're arguing that the formal definition is the only thing that you should spend time on, that the original motivation is irrelevant - this is what I was trying to argue against.
Funny, I agree that visualizations aren't that useful after a point, but when you said "start thinking about the math in a linguistic mode" I thought you were going to describe what I do, but then you described an entirely different thing! I can't learn math the way you described at all: when things are described by definitions, my eyes glaze over, and nothing is retained. I think the way you are describing filters out a large percentage of people who would enjoy knowing the concepts, leaving only the people whose minds work in that certain way, a fairly small subset of the interested population.
My third way is that I learn math by learning to "talk" in the concepts, which is I think much more common in physics than pure mathematics (and I gravitated to physics because I loved math but can't stand learning it the way math classes wanted me to). For example, thinking of functions as vectors went kinda like this:
* first I learned about vectors in physics and multivariable calculus, where they were arrows in space
* at some point in a differential equations class (while calculating inner products of orthogonal hermite polynomials, iirc) I realized that integrals were like giant dot products of infinite-dimensional vectors, and I was annoyed that nobody had just told me that because I would have gotten it instantly.
* then I had to repair my understanding of the word "vector" (and grumble about the people who had overloaded it). I began to think of vectors as the N=3 case and functions as the N=infinity case of the same concept. Around this time I also learned quantum mechanics where thinking about a list of binary values as a vector ( |000> + |001> + |010> + etc, for example) was common, which made this easier. It also helped that in mechanics we created larger vectors out of tuples of smaller ones: spatial vector always has N=3 dimensions, a pair of spatial vectors is a single 2N = 6-dimensional vector (albeit with different properties under transformations), and that is much easier to think about than a single vector in R^6. It was also easy to compare it to programming, where there was little difference between an array with 3 elements, an array with 100 elements, and a function that computed a value on every positive integer on request.
* once this is the case, the Fourier transform, Laplace transform, etc are trivial consequences of the model. Give me a basis of orthogonal functions and of course I'll write a function in that basis, no problem, no proofs necessary. I'm vaguely aware there are analytic limitations on when it works but they seem like failures of the formalism, not failures of the technique (as evidenced by how most of them fall away when you switch to doing everything on distributions).
* eventually I learned some differential geometry and Lie theory and learned that addition is actually a pretty weird concept; in most geometries you can't "add" vectors that are far apart; only things that are locally linear can be added. So I had to repair my intuition again: a vector is a local linearization of something that might be macroscopically, and the linearity is what makes it possible to add and scalar-multiply it. And also that there is functionally no difference between composing vectors with addition or multiplication, they're just notations.
At no point in this were the axioms of vector spaces (or normed vector spaces, Banach spaces, etc) useful at all for understanding. I still find them completely unhelpful and would love to read books on higher mathematics that omit all of the axiomatizations in favor of intuition. Unfortunately the more advanced the mathematics, the more formalized the texts on it get, which makes me very sad. It seems very clear that there are two (or more) distinct ways of thinking that are at odds here; the mathematical tradition heavily favors one (especially since Bourbaki, in my impression) and physics is where everyone who can't stand it ends up.
I can't learn math the way you described at all: when things are described by definitions, my eyes glaze over, and nothing is retained. I think the way you are describing filters out a large percentage of people who would enjoy knowing the concepts, leaving only the people whose minds work in that certain way, a fairly small subset of the interested population.
If you told me this in the first year of my math degree I would have included myself in that group. I think you’re right that a lot of people are filtered out by higher math’s focus on definitions and theorems, although I think there’s an argument to be made that many people filter themselves out before really giving themselves the chance to learn it. It took me another year or two to begin to get comfortable working that way. Then at some point it started to click.
I think it’s similar to learning to program. When I’m trying to write a proof, I think of the definitions and theorems as my standard library. I look at the conclusion of the theorem to prove as the result I need to obtain and then think about how to build it using my library.
So for me it’s a linguistic approach but not a natural language one. It’s like a programming language and the proofs are programs. Believe it or not, this isn’t a hand-wavey concept either, it’s a rigorous one [1].
[1] https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
> When I’m trying to write a proof, I think of the definitions and theorems as my standard library. I look at the conclusion of the theorem to prove as the result I need to obtain and then think about how to build it using my library.
fwiw, this is exactly the thing that you when you're trying to formally prove some theorem in a language like Lean.
I do want to learn theorem proving in Lean just for a hobby at some point. I haven't found a great resource for it though.
Have you seen: https://leanprover-community.github.io/mathematics_in_lean/
I hadn’t seen that. Thanks!
> So I had to repair my intuition again: a vector is a local linearization of something that might be macroscopically, and the linearity is what makes it possible to add and scalar-multiply it. And also that there is functionally no difference between composing vectors with addition or multiplication, they're just notations.
Except none of this is true of vectors in general, although it might be true of very specific vector spaces in physics that you may have looked at. Matrices or continuous functions form vector spaces where you can add any vectors, no matter how far apart. Maybe what you're referring to is that differentiability allows us to locally approximate nonlinear problems with linear methods but that doesn't mean that other things aren't globally linear.
I also don't understand what you mean by "no difference between composing vectors with addition or multiplication", there's obviously a difference between adding and multiplying functions, for example (and vector spaces in which you can also multiply are another interesting structure called an algebra).
That's the problem if you just go from intuition to intuition without caring about the formalism. You may end up with the wrong understanding.
Intuition is good when guided by rigour. Terence Tao has written about this: https://terrytao.wordpress.com/career-advice/theres-more-to-...
The vector space axioms in the end are nothing more than saying: here's a set of objects that you can add and scale and here's a set of rules that makes sure these operations behave like they're supposed to.
You have unfortunately written off everything I said by assuming the most naive understanding of it.
The general theme is that I am interested in the metaphysical concept of vectors, not the thing that human mathematicians have labeled vectors. The universe doesn't care if you write ax+by or x^a y^b, hence addition vs multiplication is just a choice of coordinate system. And matrices and functions are vector spaces sure, but out in the world, when they show up in modeling things, they are local linearizations of curved things. Every linear algebra is (inevitably) a local point in a nonlinear one, as far as I can tell. Not in a formal sense, but in the sense that when you go out into the world and find them, it turns out to be the case.
The general theme is: I don't want to spend my life mastering the rigor of these simplistic models so that I can do it intuitively (in Tao's sense); I want to use them to learn intuition of the things that they are simplistic models of, and then master that.
> The general theme is that I am interested in the metaphysical concept of vectors, not the thing that human mathematicians have labeled vectors.
That's fine but then you shouldn't be surprised that you can't read higher level mathematics textbooks, because those are not about metaphysics.
The rest of what you wrote is... just not true. Matrices are used in plenty of areas where they are not mere approximations (e.g. cryptography), and spaces of functions remain vector spaces even when the functions themselves are not linear (because there's a difference between requiring that f(x+y)=f(x)+f(y) and that (f+g)(x) = f(x)+g(x)).
> and I was annoyed that nobody had just told me that because I would have gotten it instantly.
Right?! In my path through the physics curriculum, this whole area was presented in one of two ways. It went straight from "You don't need to worry about the details of this yet, so we'll just present a few conclusions that you will take on faith for now" to "You've already deeply and thoroughly learned the details of this, so we trust that you can trivially extend it to new problems." More time in the math department would have been awfully useful, but somehow that was never suggested by the prerequisites or advisors.
oh, my point was the opposite of that. The math department was totally useless for learning how anything made sense. I only understood linear algebra when I took quantum mechanics for instance. The math department couldn't be bothered to explain anything in any sort of useful way; you were supposed to prove pointless theorems about things you didn't understand.
I did get a lot of that in the lower level math courses, where it kinda felt like the math faculty were grudgingly letting in the unwashed masses to learn some primitive skills to apply [spit] to their various fields, and didn't really give a shit if anybody understood anything as long as the morons could repeat some rituals for moving x around on the page. I didn't really understand integrals until the intermediate classical mechanics prof took an hour or two to explain what the hell we had been doing for three semesters of calculus.
But when I did go past the required courses and into math for math majors, things got a lot better. I just didn't find that out until I was about to graduate.
> infinite dimensions for functions is often more effective when thought of as around 8:00
I guess it works if you look at it sideways.
Nice: the variable l and m values can allow you to get orbitals from chemistry.
(This is where I learned at least half of the math on this page: theoretical chemistry.)
also known as Applied Quantum Mechanics.
"Functions Are Vectors"?
Let's see: Let A be a non-empty set, N the set of positive whole numbers, and X the set of all functions f
f: A --> N
with usual notation.
Assume as is common, the scalars are the set of real numbers, but the set of complex numbers will also do.
So, is X a vector space and, thus, each f in X a vector?
No, since -f is not in X. Neither is (1/2)f.
Some references (with TeX markup):
Paul R.\ Halmos, {\it Finite-Dimensional Vector Spaces, Second Edition\/}
linear algebra treated as functional analysis.
Walter Rudin, {\it Real and Complex Analysis\/}
with Lebesgue integration and, then, Banach and Hilbert vector spaces.
Walter Rudin, {\it Functional Analysis\/}
with Fourier theory.
Jacques Neveu, {\it Mathematical Foundations of the Calculus of Probability\/}
with random variables, that is, functions from a probability space to, usually, the set of real numbers with convergence results, building on the work A. Kolmogorov building on the work of H. Lebesgue.
Functions with arbitrary codomains are not vectors, but when your codomain is some field, they are.
This is what the article is very explicitly about. I guess you can quibble that the title is imprecise, but it's just a title and the article makes it clear.
In all those books I listed, and more, e.g., on axiomatic set theory and other foundations, never saw definition or mention of codomain. So, the term is obscure. Readers are supposed to guess at the meaning of obscure terms?
What I did was follow, as in the references, long established convention, that for a function to be a vector at least it had to be in a vector space where (1) can multiply a function by a number (e.g., reals or complex) and (2) add two functions and still get a function in the vector space. To be general, I omitted metrics, inner products, topologies, convergence, probability spaces, and more.
Or, as in the references I gave, math talks about vector spaces and vectors, and each vector is in a vector space. The references are awash in definitions of vector spaces with (1) and (2) and much more.
Computing is awash in indexes for data, e.g., B-trees, SQL (structured query language) operations on relational data bases, addressing in central processors, collection classes in Microsoft's .NET, REDIS, and calling all such also functions confuses established material, conventions, and understanding.
Codomain is not an obscure term. One second of Googling would have helped.
Codomain is different than just range; just think for a bit. In the older literature you are used to their usage is probably synonymous though.
Not just when the codomain is a field, but more generally when the codomain is itself a vector space. The former is a special case of the latter where you construct a 1D vector space from a field.
> Functions with arbitrary codomains are not vectors,
Well, not with the operations pulled from the codomain at least.
An engineering, signal processing extension/perspective:
An infinite sequence approximates a general function, as described in the article (see the slider bar example). In signal processing applications, functions can be considered (or forced) to be bandlimited so a much lower-order representation (i.e. vector) suffices:
- The subspace of bandlimited functions is much smaller than the full L^2 space - It has a countable orthonormal basis (e.g., shifted sinc functions) - The function can be written as (with sinc functions):
x(t) = \sum_{n=-\infty}^{\infty} f(nT) \cdot \text{sinc}\left( \frac{t - nT}{T} \right)
- This is analogous to expressing a vector in a finite-dimensional subspace using a basis (e.g. sinc)
Discrete-time signal processing is useful for comp-sci applications like audio, SDR, trading data, etc.
This idea is also at the core of the Finite Element Method (and other Galerkin methods, such as DG) which are probably the most widespread methods to approximate the solutions of partial differential equations (PDEs) used everywhere in physics and engineering.
Basically, define a finite dimensional function space V_N of dimension N, in such a way that you could grow N to be arbitrarily large. Solve not the PDE (originally defined over an infinite dimensional function space V, such as H^1), but its discretization as if it dealt only with functions in V_N rather than all functions. The PDE is then simply a linear system, easy to solve. And you can prove, for instance in the case of elliptic PDEs, that the solution to the discrete problem is the orthogonal projection of the true solution of the PDE (in V) onto V_N (Céa's Lemma). Finally, you can produce estimations of the error this projection incurs as a function of N, and thus give theoretical guarantees that the algorithm converges to the true solution as N goes to infinity.
(N in this case is the number of vertices in a mesh that is used to define the basis functions of V_N)
Full $L_2$ also has a countable orthonormal basis. Hermite functions are one example.
So cool! This is the first time I’ve ever read about a math idea and felt a deep pull to know more.
Well, yeah, function spaces are an example of vector spaces: https://en.wikipedia.org/wiki/Vector_space#Function_spaces
Yeah, and it's such a boring thing to write about.
Given an vector space V with (+, ), you can define the vector space over functions F whose codomain is V and where F.+ and F. both take two functions as argument and return another function applying V.+ or V.* on the result. All the linear algebra properties come from the original vector space. Hence it is boring.
* is used on HN for italic delimiters, you need to escape it with a backslash to avoid that.
I will need to read through the rest of the article later, but the initial intuition building is a bit sloppy. None of those vectors drawn in the initial examples belong to the same vector space. Vectors need to emanate from the same origin to be considered as part of the same vector space.
> Vectors need to emanate from the same origin to be considered as part of the same vector space.
That's absolutely not the case.
> That's absolutely not the case.
Can you elaborate? Which vector space(s) do these vectors belong to, as drawn?
If anything, this just highlights the sloppiness of these types of illustrations. Things aren't precise enough, in my opinion, for the illustrations to do anything except confuse.
In the mathematical sense, a vector is nothing but an object that can be added to other objects and scaled by some field (with some reasonable properties attached).
In physics, a vector is often more specifically something with magnitude and direction. This still doesn't mean that it needs to be anchored at the origin. Vectors that are anchored at the origin are IIRC called position vectors, but mathematically, if you translate them away from the origin they're still the same vector.
The question is one of identification between the drawing and the vector space. See my comment here: https://news.ycombinator.com/item?id=44491474.
The graph of a function f: X -> Y is the set {(x, f(x)) | x in X}. It is much more clear and precise to associate elements of the graph with vectors such that the 0 vector is identified with the R^2 origin, and then points in R^2 are identified with vectors. Then there is a mapping between vectors in this vector space to the graph, i.e., to points (x, f(x)).
> In physics, a vector is often more specifically something with magnitude and direction.
Physics is sloppy. :) This is not a general description of a vector, where vector is an element of a general vector space. Not all vector spaces have a norm, which is required for magnitude to make any sense.
> but mathematically, if you translate them away from the origin they're still the same vector.
Right, and you cannot always translate vectors without more machinery, such as parallel transport.
> Physics is sloppy. :) This is not a general description of a vector, where vector is an element of a general vector space. Not all vector spaces have a norm, which is required for magnitude to make any sense.
Fine, then let's be precise. An element of the vector space R^n is nothing more than a function from {1,...,n} to R. And every n-dimensional R-vector space is isomorphic to it.
How you wanna draw this in a coordinate system is up to you. It is customary to identify vectors with points on that coordinate system and then equivalently with an arrow pointing from the origin to that point. In that case the zero vector is the origin, or an arrow from the origin to itself.
It is equally customary to use vectors as displacement, i.e. a directed difference between two points. In that case, the vector that you can't actually draw because it has zero length is your zero element. Now your arrows don't have to be anchored at the origin anymore.
Both of these spaces are of course isomorphic.
If your vector space is infinite-dimensional it's of course not gonna be really possible to draw anymore.
What do you call the origin of a vector? I don't think that's a notion in vector spaces.
Vector spaces have an addition identity normally referred to as the 0 vector.
When drawing vectors, such as on a 2D space, the origin is identified with this 0 vector, and then the points in the 2D space can be associated with the vectors in the vector space.
The author seems to be a great educator and computer scientist, much respect to his work. But from what I can gather, although I'd love to study more infinite sized matrices, he proved / showed nothing in this article. What he wrote is not true at all, they are only analogies and not rigorous maths. Functions are not vectors. But finite polynomials are vectors yes, this is trivial.
https://thenumb.at/Functions-are-Vectors/#proofs
It's not a particularly interesting proof, but the author does prove that real valued functions are vectors. The bulk of the article is less about proofs, and more about showing how the above result is useful.
Vectors in the way he talks about in the beginning. With indices (and then extending to "In higher dimensions, vectors start to look more like functions!"). Of course if you use the general meaning of every word, vectors are functions and functions are vectors, and this article shouldn't then have anything interesting to talk about.
It doesn't seem useful at all to me, the examples in the article are not that interesting. On the contrary it is more confusing than anything to apply linear algebra to real valued functions.I think you are confused about the analogies at the beginning at the article (although there is nothing technically wrong about them).
Here's the definition of a vector space which agrees with the one everyone in mathematics use: https://thenumb.at/Functions-are-Vectors/#vector-spaces
From this it's fairly easy to prove (and done in the article) that the set of all functions R->R is a vector space.
I understand that. The crux of the article is supposedly being able to apply what we learned with easy small vector space to infinite function vector space : we can't. But he's taking the example of polynomials and analytical functions which are MUCH smaller spaces than functions in general.
Yes we can. Several people have tried explaining this to you. Maybe you should sit down and work through the material.
Maybe you could envision that I learned maths a similar way than you and that's why I have problem with this material. It's trying to portray complicated space as simple, they're not
Any basic liniear algebra course should talk about this, at least in the finite dimensional case.
Polynomials come to mind.
Finite degree polynomials are vectors yes. Polynomials is a typical example you study when learning about linear algebra. Doesn't say anything about real functions in general though, I don't think any linear algebra course should make the analogies made in this article, that'd be confusing
I think they should.
You don't need to go into intricacies like metrics and approximations because that's more like analysis.
You can however talk about infinite dimensional vectors spaces and talk about projections onto finite subspaces.
Really well explained. I managed to follow it for quite a while.
I love the prerequisites section. Every technical blog post should start with this.
Only functions on a finite domain are vectors.
Functions on a countable domain are sequences.
Why is this being downvoted ? Could a downvoter elaborate ?
Because it makes little sense.
Vector spaces can have infinite dimension, so the "only" in the first sentence does not belong there.
The second sentence is also odd. How do you define "sequence"? Are there no finite sequences?
I think it is "vector" taken in the way the author wrote about it / showed illustrations in the article.
For the second sentence, he's right, we could also write (wrongly) an article titled "Functions are Sequences" and (try to) apply what we know about dealing with countable sequences to functions
sequence is typically defined as a function from N -> A, thus countable domain
Oh, my. Alice, meet rabbit hole.
Same thing in video form explained by a different person - https://youtu.be/mhEFJr5qvLo
It seems like mentioning some of the applications at the beginning would motivate learning all these definitions.
The genre of this article is not pedagogical, really. One usually learns these techniques in the course of a particular field like physics, electrical engineering, or theoretical chemistry. This article is best thought of as "a story you've seen before, but told from the beginning / ground up, with a lot of the connections to other topics and examples laid out for you". For that purpose, it's excellent, perhaps the best I've ever seen. It might also whet the appetite of a novice, but it's not really for that.
This perspective is crucial for understanding signal processing, machine learning, and quantum mechanics. Viewing functions as vectors enables practical techniques like Fourier transforms and kernel methods that underlie many modern technologies.
> "The material is not motivated." Not motivated? Judas just stick a dagger in my heart. This material needs no motivation. Just do it. Faith will come. He's teaching you analysis. Not selling you a used car. By the time you are ready to read this book you should not need motivation from the author as to why you need to know analysis. You should just feel a burning in you chest that can only be quenched by arguments involving an arbitrary sequence {x_n} that converges to x in X.
https://www.amazon.com/review/R23MC2PCAJYHCB
Not sure what I'm supposed to get from that. I guess some people care a little too much about math and have trouble relating to others?
If you need "practical applications" for some part of math to have value to you, then large parts of math will not be for you. That's fine, but that's also something you should accept and internalize: math is already its own application, we dig through it in order to better understand it, and that understanding will (with rather advanced higher education) be applicable to other fields, which in turn may have practical uses.
Those practical uses are someone else's problem to solve (even if they rely on math to solve them), and they can write their own web pages on how functions as vectors help solve specific problems in a way that's more insightful than using "traditional" calculus, and get those upvoted on HN.
But this link has a "you must be this math to ride" gate, it's not for everyone, and that's fine. It's a world wide web, there's room for all levels of information. You need to already appreciate the problems that you encountered in non-trivial calculus to appreciate this interpretation of what a function even is and how to exploit the new power that gives you.
I don't see any such "math gate" on this link. Also, this math does have practical applications, but they're not mentioned until very late in the article.
My suggestion is that briefly mentioning them up front might be nice. I didn't mean to start a big argument about it.
i'll never fathom why people on hn treat a post as an auto-invite for unsolicited feedback.
Yet some parts of math are 'preferred' over others, in that most 'serious' mathematicians would rather read 100 pages about functional analysis than 100 pages of meandering definitions from some rando trying to solve the Collatz conjecture.
Some people would like to have a filter for what to spend their time on, better than "your elders before you have deemed these ideas deeply important". One such filter is "Can these ideas tell us nontrivial things about other areas of math?" That is, "Do they have applications?"
Short of the strawman of immediate economic value, I don't think it's wrong to view a subject with light skepticism if it seemingly ventures off into its own ivory tower without relating back to anything else. A few well-designed examples can defuse this skepticism.
you're supposed to get that the cynical lens you're applying here doesn't fit - if you aren't intrinsically motivated to read this stuff then it's not for you. which is fine btw because (functional) analysis isn't a required class.
The first paragraph and table of context both mention applications.
Yes, so it does. Perhaps I read too quickly.
The author asserts vectors are functions, specifically a function that takes an index and returns a value. He notes that as you increase the number of indices, a vector can contain an arbitary function (he focuses on continuous, real-valued functions).
It's fun to simulate one thing with another, but there is a deeper and more profound sense in which vectors are functions in Clifford Algebra, or Geometric Algebra. In that system, vectors (and bi-vectors...k-vectors) are themselves meaningful operators on other k-vectors. Even better, the entire system generalizes to n-dimensions, and decribes complex numbers, 2-d vectors, quaternions, and more, essentially for free. (Interestingly, the primary operation in GA is "reflection", the same operation you get in quantum computing with the Hadamard gate)
Thanks for mentioning this. I think I encountered Geometric Algebra in a video related to making video games. It's this one (Why you can't multiply vectors): https://www.youtube.com/watch?v=htYh-Tq7ZBI