Unfortunately that's somewhat expected - regardless of implementation, the sole fact that the GC needs to somehow walk the entire tree to trace still referenced sections makes swap highly impractical -- even if you had not had 40ms stop-the-world pauses the LRU cache used by swap would get thrown away at every GC.
I believe that's actually the same reason why Apple stopped using GC in their frameworks in favour of automatic reference counting.
I remember a research paper about swap and GC, where the GC cooperated with the OS to avoid this kind of issue. AFAIK it went nowhere, too bad.
[–]andreiross[S] 15 points il y a 21 jours
Are you talking about this one? https://cse.buffalo.edu/\~mhertz/bc-pldi-2005.pdf. If so, yes. Too bad. I don't know the repercusions this paper had in the past, though, in the sense of pros and cons of the bookmark collector. Don't know if anyone tried to actually implement it or design it at some point.
Modern OSes need a facility to signal to a thread that it hit a paged out chunk of memory. It's not like it's not possible to create a swap-aware GC (or any other kind of code that touches lots of memory).
I would go further - I think the whole swap lifecycle needs to be communicated.
Before the OS swaps out a page, if it could invoke the GC which would clean up that piece of memory so that we dont end up writing garbage to swap.
The OS should also allow marking pages as piority to stop them from being swapped out.
> The OS should also allow marking pages as piority to stop them from being swapped out.
This is IIUC possible using the mlock(2) family of syscalls: https://man7.org/linux/man-pages/man2/mlock.2.html. (On Linux, though I'm guessing other UNIXen and operating systems have it or something equivalent.)
Whether reference counting is a GC algorithm depends on how you define what GC is.
I prefer to consider GC only the methods of memory management where reclaiming the no longer used memory is done either asynchronously with the main program or as late as possible, i.e. when new allocation requests cannot be satisfied.
In the normal implementation of reference counting, memory is freed as soon as possible, i.e. exactly like stack memory, when blocks are exited, so I do not consider reference counting as GC.
The problem with GC in the strict sense is that you cannot predict when it will happen. With both stack memory and reference counted heap memory you know that whenever you exit a block, some time will be spent with running destructors and for freeing memory, but such interruptions will not happen in other points of the program.
> Whether reference counting is a GC algorithm depends on how you define what GC is.
Pretty much all the high-performance GC/refcounting algorithms are hybrids in one form or the other; it's a spectrum of choices. https://dl.acm.org/doi/10.1145/1028976.1028982 explores this in some detail.
That is a classic paper and obviously I am aware of it.
However, if you have distinct names it is efficient to use them with distinct meanings.
Making "garbage collection" synonymous with "freeing memory" is bad, because it eliminates a means to distinguish various methods for freeing memory.
Like I have said, I consider useful to define "garbage collection" as any method of freeing memory where the memory is not freed as soon as possible (i.e. when a block is exited), but freeing is deferred to be performed at a later time, even as late as possible (i.e. when new memory allocation requests cannot be satisfied).
Indeed, many garbage collection algorithms use reference counts, where memory deallocation is deferred, but when I use the term "reference counting" without any other qualifier, I mean it in the sense in which it was originally defined in 1960, where the time when memory deallocation is run is predictable, exactly like for stack-allocated memory.
I prefer to write programs with well-defined worst-case behavior, so I normally prefer deterministic algorithms. Thus I always prefer to use reference counts instead of GC. I have never encountered a case when avoiding reference cycles was difficult.
I define the academic view of GC algorithms in Computer Science, not what random developers decide to call GC.
Which in an industry where some folks call themselves Software Engineers after a bootcamp, without any kind of accreditation, I rather stay with the definition from those that do language design and compiler algorithms research.
I prefer to use the terms as they were originally defined by the authors who introduced them, and not with the modified meanings that become fashionable after the authors of some paper written some decades later decide randomly to change the definitions of the old terms. This is also true for the terms "garbage collector" and "reference counts", which were both introduced in 1960 to denote 2 clearly distinct methods of memory management, and the main difference between them does not consist in whether some kind of reference counts exist somewhere, or not.
The paper "A unified theory of garbage collection", which has started the fashion of considering reference counting as a kind of garbage collection, only shows correctly that both tracing and reference counting are complementary implementation techniques for a garbage collector.
It does not mention anywhere the essential practical difference between the traditional standalone memory management with reference counts and a garbage collector, which stays the same regardless whether the garbage collector also happens to use reference counts for some purposes, which is the difference between predictable and unpredictable times when memory reclamation is done.
Many of the modern authors of academic papers are a poor model of using computer terminology (or for the terminology in other domains), because very frequently it is obvious that they have not read the old works where such terms were introduced for the first time, even when such works are cited in the bibliography.
Unlike them, I have done an extensive research to find when and where various computing terms have been used for the first time, and I strive to use most terms with their original meanings, not with corrupted meanings, even in the cases when the latter have become more popular lately.
So as fellow digital archaeologist I would find interesting to find Lisp, BASIC, CLU, SIMULA, or Cedar papers, where the authors mention reference counting not being a form of garbage collection algorithms.
Cedar was already combining reference counting with a cycle collector, as one of the very first systems programming languages with automatic resource management.
In the normal implementation of reference counts, counters can be decremented only at block exits and not at any other program point.
At a block exit some of the local variables that are freed may contain references, so freeing them will decrement some reference counts. Then some counters will reach zero, triggering other deallocations and the decrementing of other counters. This will repeat until no other counters reach zero.
All the memory deallocation happens predictably, only at block exits.
If a variable is not freed immediately when a counter reaches zero, but the deallocation is deferred for a later time, which is not predictable, that is no longer classic memory management with reference counts, but it is a garbage collector, which happens to also use reference counts, probably in combination with some tracing algorithm.
When reference counts are implemented, manual memory deallocation, like with C free() or C++ delete, should be forbidden, but even if it were used that would just introduce other program points besides the block exits, where it is known that memory deallocation will happen.
You've described where deallocation can happen, not when it will. Every block exit is a candidate, but which one drops the last reference depends on runtime state: how many other owners the object has and which of them goes away first. With shared ownership across threads it's a race by construction. The free runs on whichever thread happens to release last.
The cost isn't bounded either. Dropping the last reference to the head of a list or the root of a tree frees the whole structure at that block exit, and its size is a runtime property.
And it isn't only block exits. Every assignment to a variable or field holding a reference decrements the old target, and so does removing an element from a container. Swift's ARC doesn't even promise the scope boundary: the optimizer may release right after the last use, which is why withExtendedLifetime exists.
By the same "where" criterion a non-concurrent tracing GC is predictable too, because it can only run at allocation points. That doesn't tell you which allocation will trigger it, just as knowing the block exits doesn't tell you which one will free.
Useful to them usually, or it goes to show how little our profession cares about learning the proper terms, or its historical background for that matter.
Anyway it doesn't matter any longer, AI is going to replace most of us, and it comes with automatic everything.
A region-based collector like Hotspot's G1 combined with madvise could probably operate in a manner that's more swap-friendly by focusing on a smaller working-set at any given moment and announcing its intent to switch the working set to the OS ahead of time.
Apple should just stop using a silly mark & sweep collector. All the good collectors are copying, if you have enough memory. Only on tiny devices you need mark & sweep.
I don't understand why they felt necessary to rewrite a core component in another language. If you have a garbage collection problem my first intuition would be to produce less garbage!
For example better using the stack, or pulling out the big gun of manual memory management.
I'm sure they had reasons to choose Go when they first designed this project but they don't go into them at all.
Feels like they just wanted to play with a new toy.
> If you have a garbage collection problem my first intuition would be to produce less garbage!
FTA:
“These latency spikes definitely smelled like garbage collection performance impact, but we had written the Go code very efficiently and had very few allocations. We were not creating a lot of garbage.
[…]
the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references”
They explained in the post why this wasn't an issue: they were producing very little garbage, but there was a very large object graph.
> manual memory management
If you need to do manual memory management in a GC language with no builtin support for it, like Go, that's probably a sign that you should switch to a different language.
And importantly in a tracing GC the work load is a function of the non-garbage data not of the garbage.
Generational GC can avoid some of it by ignoring the old code entirely, but Go does not implement generations because its relatively strong ability to stack-allocate generally replaces the nursery, and for most work loads you don’t recoup the costs.
Problem with languages like Go is, because they have GC, their API is less tuned to the ability to prevent new allocations.
In C you see lots of functions whose first parameter is a buffer that will be re-used.
And even in Rust you don't see this often, because reusing a buffer means having to do &mut, which means having to re-structure your code. And it's really easy to do Vec::new().
One of big issue with GC is it need to scan every reachable objects to mark it is reachable. That mean even if your code produce no garbage but have a large amount of long lived object the GC still need to traverse all of those objects every cycle.
This is not entirely correct. If you care about latency, then it doesn't matter on which major fault your application gets paused. Disabling swap protects you from data pages being evicted, but code pages can still be paged out.
If you care about latency, mlock() your memory, do not disable swap. Swap is good and gives the kernel an equal opportunity to evict data and code pages.
For GC enabled languages swap is universally bad. Some gc-pauses are indistinguishable from a system crash. It's a side effect on not having tightly specified memory limits.
I'd rather have applications be oom_killed than having them swap out, the former is rather obvious and demands action.
If you want you application to stay in memory, then make it explicitly with mlock()/mlockall().
Disabling swap will just moves pressere elsewhere: to code pages. And evicted code page is no better: full stall while kernel loads that page from disk.
FYI: This `delamon` account seems to be meat proxying us. Just relaying garbage AI answers without understanding his own words. Doubt he's ever used mlock himself for anything.
I see that you've rewritten your AI generated reply there. So when have you used mlock in production for Go services, the way you're advising others to do? What were the circumstances?
Yes, I was and still running go and zig binaries with mlockall. Apps have large in-application caches and also perform large streaming reads of datasets bigger than available RAM. If it happens that several streaming reads running at same time, they push memory pressure hard enough, so kernel swaps out cold parts of in-app caches. That later leads to elevated latencies => mlockall to avoid that.
While I've seen systems grinding to a halt due to evicting code pages, the idea that disabling swap distributes load between code pages and data pages makes no sense to me. Data pages are also disk backed, so they could also be evicted at any time. Why would data pages be written to swap if they came fron the disk in the first place?
So that only leaves anonymous data pages, i.e., regular data in memory, that could be swapped. Ok but you're not easing the load on code pages, those still get evicted during memory pressure whether you have swap or not.
The assumption that data pages can always be easily discarded is incorrect. Dirty data pages can't be discarded until they are written back. Recently accessed data pages are also not discarded until all inactive data and code pages are discarded.
I think discussing it in a vacuum is pointless, it should depend on how much memory you have. You can have a swap on, but with a low `vm.swappiness` number.
Surely as long as it is all in memory these pauses are not going to be that significant, details like that should be left for the language devs to optimize.
You can tell an average Golang user to use mutexes, you shouldn't be telling them to lock memory manually.
Maybe, but it still there and probably has a different value between distros as well. Default value is 60, what is that good for exactly?
If you're in a position where you still want swap to be there, but only used in extreme situations, then you should set it yourself. Otherwise disable swap or switch a language.
What do you think this value of 60 means? Also, as I said, in extreme situations, when the system is right before invoking the OOM killer, swap is used regardless of `vm.swappiness` setting.
Value 60 is not an arbitrary number. It makes the kernel reclaim 140 file pages for each 60 swapped-out anon pages. This proportion is supposed to reflect the relative performance cost between sequential file access and random swap I/O on HDD. For SSDs, the value 100 is more appropriate.
> I don't see a reason why the default shouldn't be 0 then on modern systems.
The zero value makes reclaim very unbalanced and is not appropriate for generic systems. This will cause the kernel to discard all the page cache, slowing down the system with uncached I/O, while keeping all unused anonymous pages in memory.
I don't get the point that you're even arguing. Are you saying that 60 is fine as a default? Or are you saying that people should even think about vm.swappiness because it doesn't matter? The alternative here is to disable swap completely, that's what started this whole thread.
There are two types of pages: anonymous and mapped from files. The code pages are mapped from a binary; they are not very special.
Under memory pressure, the kernel evicts less popular pages from memory. If a page has been mapped from a file, it is dropped (if dirty, then it is written out first). If it is needed later, the kernel can read it back from the file. If a page is anonymous (read: heap page), then there is no backing file and the kernel copies it to swap before dropping it. This is swapping.
So, what happens if you disable swap and the kernel is low on memory? What can it evict? Anonymous pages cannot be evicted: there is no swap to put a copy in. The only choice the kernel has is to evict pages that are mapped from files. Those include pages mapped from the executable. You don't eliminate stalls by disabling swap, you just move them elsewhere: the kernel will page out code and your app gets paused whenever the execution flow hits such a page.
I haven't had to analyze the performance of no-swap processes before. My assumption is that code is hot enough to avoid eviction and that evicted code pages are rather the exception. To strong-man the argument, I can imagine long running complex (bloated) services could have parts that are not touched unless a specific request comes in.
>So, what happens if you disable swap and the kernel is low on memory? What can it evict?
Your userspace early OOM killer triggers and lets you know you're trying to run more than will fit in memory so you don't do it again. (In my experience, the kernel OOM killer can't be trusted to kill processes soon enough.)
You are missing that the file LRU is not only the code pages or memory mapped files but all file data. The active code pages are specifically excluded from being discarded during the reclaim process.
I wish this is true but evidence is against you. Just try testing memory pressure on a no swap system. You will see that the system grinds to a half for a few minutes before you even get to an OOM event.
Could you clarify what exactly is "not true"? Also, could you explain what exactly your evidence proves? What do you think happens when the system becomes unresponsive under memory pressure, and why?
> The active code pages are specifically excluded from being discarded during the reclaim process.
This is not true, or the system wouldn't grind to a halt due to having to reload code pages (even those that are recently in use) from disk over and over.
For example, key presses and output streaming in SSH become unresponsive. The code pages for handling key presses and output streaming are obviously in use — I just used them!
I'll try again. Why do you think that unresponsiveness is caused by code pages being discarded? Do you have evidence for this? Have you considered that there are other mechanisms that cause programs to become unresponsive under memory pressure?
It's the explanation that I've read when scouring forums for reasons why it happens. I'd love to hear alternative explanations and how to measure/verify them. But the measurement tools also become unresponsive so I can't check anything.
But frankly, as a user I don't really care what the real technical reason is. I just don't think the system should become unresponsive for minutes on end during memory pressure. Just OOM-kill earlier, when it's supposed to.
The assumption that a system gets unresponsive due to code pages being reclaimed doesn't make any sense. It is a common myth, cargo-cult knowledge. The kernel doesn't discard any active referenced pages at all, and the active code pages are the last that are getting reclaimed.
The proper explanation is relatively straightforward. Normally, any memory request for a process is served by getting a page from the free memory pool. But when the system free memory goes below the low watermark (calculated from vm.min_free_kbytes and vm.watermark_scale_factor), the kernel stops giving the pages from the free memory pool and instead goes into direct reclaim, trying to find unused pages. While the kernel does reclaim, the allocating process remains stalled. The kernel can't return control to the process - there is no page mapped that the process tried to access. This applies to any types of pages - anon, file-backed, code pages. So, under the condition of free memory below the low watermark, any process trying to allocate memory gets stalled. This is measured as memory pressure stalls (PSI) in /proc/pressure/memory.
The problem with the system being stuck in an unresponsive state is that the design choice for the kernel OOM handling is to be optimistic and keep the workload running as long as the kernel is making any progress reclaiming pages. The direct reclaim process needs to fail to make any progress 16 times before the OOM killer is invoked.
The solution is to use cgroup memory limits to avoid getting into the situation of memory exhaustion and use user-space oomd to recover from this situation earlier.
Perhaps I wasn't clear, but I did say that all file pages are treated equally.
I would assume that page with PC is excluded from reclamation process, but other code pages are valid targets.
All recently referenced code pages in the active LRU are excluded from reclaim.
Code pages are paged in on demand: for any sufficiently large binary just some code pages will be mapped right after start. Given enough memory pressure, the active LRU will be shrunk often enough to push code pages out of the active LRU and then they are eventually evicted.
I'm not aware how it works in modern MGLRU, but I would assume similar ovservable behevior holds.
Active code pages with the "Accessed" bit set are specifically excluded from being moved to inactive LRU when active LRU is being shrunk. Instead, they are moved to the head of the active LRU. All other pages are moved from active LRU to the inactive LRU unconditionally, even if they have the "Accessed" bit set. So, no, the active code pages are not going to be eventually evicted.
I think we are saying the same thing. Active code pages are not evicted; non-active code pages (that is, pages that had not been accessed since last time of active LRU shrinking) may be evicted.
Under memory pressure the system will free the ram used for code pages because it can always load them back from the executable on disk. It’s the same virtual memory mechanism as swap but without needing dedicated swap space. The op is saying disabling swap doesn’t prevent long pauses during memory pressure, because the system just swaps code out instead of dynamically allocated memory.
There are good sibling replies answering your question, but to give a specific example: if your application writes something but then doesn't need it anymore. Since we're talking about go, perhaps a data structure you need also holds a reference to data you don't need.
When the kernel needs memory, it goes hunting for a page it can discard. But since that's transparent, the kernel can only discard a page if it knows it can get it back (after all, it's still got valid data on it, and maybe you'll try access it again later).
If there's swap, a page full of stale/unneeded data can be written out to swap. But if there's no swap, your page of "dangling data that you'll never use, but is still valid & referenced" can't be discarded; the kernel doesn't know you won't want it later, and it can't recreate the page if it throws it away.
So like sibling said, at that point it has to find other pages it can evict from memory, ones that _do_ have somewhere persistent they can be written out to. Pages loaded from binaries on disk satisfy that, so those will get dropped instead.
For JIT languages most of the code is dynamically generated, not available to be loaded back from the disk. Morealso, the allocations do happen at large chucks where the oom_killer also happens, so it's significantly less of an issue.
That doesn't necessarily improve matters. Now any code that gets JIT'd is also going to be permanently pinned in memory too, no matter whether it'll get used again, if you don't have swap.
Run your process in a cgroup that's set to disallow swap, then. The rest of your system benefits from having swap. This was an easy and safe thing that required minimal convincing to roll out to production at my company.
No, do not try to use mlock to avoid swapping, except in very specific scenarios. That way lies madness.
To be more explicit about my advice: disable swap and ensure you have sufficient memory for your workload.
Your 14 MB Go binary is not the source of your memory pressure. Your memory pressure is coming from the allocated anonymous pages supporting your programs data structures on the heap.
Have sufficient memory for your workload, monitor it, and let the Go GC and kernel manage memory normally. This will work exactly as you'd hope.
Can you elaborate, why mlock is bad? And how disabling swap is better?
You are correct, that 90MB Go binary is not major memory consumer. But it doesn't matter what is the source of memory pressure. What matters is what kernel do under pressure. If you disable swap, then under memory pressure kernel can only pageout memory mapped files, no matter how small they are. And your code will be paged out.
We're trying to solve a problem: we don't want huge latency spikes caused by swapping memory to disk. The only good solution to that is to have sufficient RAM. It's that simple.
Introducing mlock is operationally annoying and trying to solve the wrong problem. It requires elevated container permissions, it's not something your SRE team is going to expect, few programs do it, and it has all kinds of technical implications and interactions with kernel OOM system, forking, the Go GC, etc.
And nothing about mlock solves the problem of having insufficient RAM for your workload.
There are very specific scenarios where mlock is exactly the right solution but the typical Go HTTP service is not one of them. The KISS principle applies.
Have you actually done this yourself for Go programs that you've deployed into production? If so, what were the circumstances? Was it your solution to latency spikes?
It was the default for every production binary. I read the c++ implementation, not really that crazy. It was for latency, and in some cases (with other measures) to keep the binary running if the disk failed.
The alternative (which I’ve also done), is to put the binary in tmpfs, after making sure it’s not bloated.
Your Go binary was being swapped out due to memory pressure, causing latency, and using mlock was your fix for this problem?
Or did someone else do this work and you assumed it was effective? Or did you personally implement this change and then validate that the before/after metrics improved as a result?
As I mentioned, it had long been the standard, the core library gave you a simple flag to use. I would occasionally find cases where it was not set correctly and you could see it causing short spikes (the simplest way to measure is something like a pacemaker task on an event loop, or task scheduling delay since it strips out other variables).
Alright, so you didn't implement it. And you're describing a "pacemaker task on an event loop"? That doesn't sound like a Go program at all, where we don't have a single event loop to measure and we have built-in runtime metrics.
Just seems kinda weird to give advice on forums when you're not speaking from actual relevant experience. I'm getting meat proxy answers and confident replies from people who've never done what they're suggesting others do.
As I said, I was more familiar with the C++ implementation, but we had it for every language in use.
I ultimately change the runtime environment to no longer allow any local durable storage for job (with a handful of exceptions), including swap and binary staging. The fact that almost everything was already using mlock made the transition simpler.
I am struggling to understand which specific problems that pertain to the use of mlock(2) you are alluding to.
Java is an example of the most widely deployed garbage-collected language in production around the world, deployment number and scale-wise. Most of the world's missing critical systems today run on Java. Java can run as a process or in a container. Containerised Java deployments do not require unique operational SRE arrangements, nor do they require any unique/dedicated permissions to run in a container.
Upon start-up, the JVM checks whether the system has sufficient memory to satisfy the max heap size constraint, and it will duly fail to start up if not. Afterwards, the JVM will allocate the required amount of memory, and it will mlock it. Multi-terabyte[0] RAM JVM deployments[1] do exist and exhibit exactly the same behaviour. Java has had such behaviour since its inception.
The prevailing number of Java deployments run on systems with swap enabled, and, from the operational standpoint, it is pure insanity to run mission critical (banking, payments, government etc) apps without a swap. Swapless JVM deployments are virtually unknown or are operational negligence.
The principal exceptions for running a JVM on a system without a swap are hard real-time scenarios and embedded systems. That is it.
What makes Go different or unique that it can't use mlock and that it would require special permissions or unique/distinct SRE/operational arrangements? It is eluding me.
[0] The Azul JVM supports up to 20 Tb of RAM which it will mlock.
[1] See the published Azul/Forrester evidence for reference. It is an interesting read regardless of the mlock debate.
I'm pretty sure almost all of our (very large) java systems were running without swap (it took effort to enable swap for your jobs, and eventually was almost impossible).
Java is generally simpler to run without swap since you know in advance how large the heap can be, so it's just a small amount of variability in the jvm internals to consider.
You don't want swap of heap/stack, or paged out code pages. The latency hit is pretty bad even for non-"real"-time systems. And really, you should know how much memory you are going to use (to within some fair bounds).
Adding only a small amount of swap for very cold pages does not save very much ram. And using it for anything else is very bad for interactive systems. In the same vein paging out code pages you won't ever run just does not save very much.
It requires some discipline in monitoring, load testing, etc, but you should be doing that anyway.
You are correct, of course: if your application needs 2x RAM and you only have 1x, then there is no way mlock can magically save you from swapping. What it gives is protection from a noisy neighbour.
The good thing is that mlock/mlockall does not require elevated permissions. But it is bounded by the memlock rlimit, which you need to configure for the container. If you don't want paging/swapping, set it equal to the total memory you give to the container and call it a day.
I'm curious about the technical implications. Which ones do you have in mind?
To be honest, I don't quite see how mlock makes it complex. It's quite the opposite: it has very clear semantics and makes reasoning about system behavior simple. Disabling swap is the opposite: you're making a bet and hoping it works.
That would work reasonably well. With one important caveat: memory controller tracks all memory, including page cache. Setting memory.min will prevent reclaim of page cache charged to that cgroup, as long as total memory usage stays below the limit.
Another important caveat: people keep assuming that page cache can be easily discarded and it doesn't count towards the working memory set. Then they get surprised by unexpected OOMs. There is a decade-old issue on Kubernetes https://github.com/kubernetes/kubernetes/issues/43916
Choosing a language that forgives bad memory management practices and promotes sloppy programming practices is your decision. Don't be surprised that it has to do its job. You could have chosen modern C++ or Rust and use RAII and proper automatic deallocation.
I started wondering if there could be swap-aware GC, like first make the required page swapped in (not that there's any obvious API for that...) and only then pause the world?
You could do lots of interesting things with sufficiently deep inter-layer integration.
For example, why not swap out not by LRU page but by dense node clusters on the heap graph, maintaining in-memory summaries of inbound and outbound edges for liveness? If you do this, you don't have to swap the cluster in to do a GC involving it.
If the whole cluster becomes unreachable, you wouldn't even have to swap it back in to get rid of it: you'd just drop the swap reference and deem the swap space free.
I don't see anything this deeply integrated happening near-term, but it's fun to think about.
Probably more mlock (keep memory area in RAM), mincore just tells you what is or is not in RAM, and madvise is about access patterns.
You could MADV_WILLNEED the GC metadata when you start the GC process hoping they’ll have been paged in by the time you STW, but assuming that area is not massive it’s probably a better idea to just prevent it being paged out.
I was assuming they meant for the heap. Not even executable memory is locked though, so if you mlock too much memory the kernel will page out your executable instead. Something has to give, so it's better to schedule it.
On a lower level, the OCaml and cpython GCs use a prefetch buffer during marking to schedule around the cache.
One can just add APIs to the Linux kernel, and this would be a pretty straightforward one. (though it might be a little more difficult to avoid a syscall here)
I'm sure an agent can work on this and get some numbers with a day's worth of tokens.
I was biten by not understanding madvise(MADV_WILLNEED) semantics fully. It will prefetch, but it will not pre-fault. You still gonna pay minor page fault on every new page access. There is MADV_POPULATE_READ, but it is synchronous.
I think we are not realizing the paradigm shift here. A coding agent can implement this with a modest little day’s worth of tokens. This means that no one needs to read or know the APIs any more. We can just add them to the kernel as we need them. Then when we forget that we needed them we can just rediscover the API idea later and spend a day’s worth of tokens. (But let’s be real here. By then it will probably be just 1/3 worth of tokens with all the model improvements as well as the ample training material.)
A GC latency SLO should include operating-system memory pressure. Otherwise, a page-fault problem will look like a collector problem and lead to the wrong fix.
Swap latency explodes when you get into a swap storm where tasks continuously touch different pages and fight to get things paged in and out, forming a queue waiting for swap IO.
The other thing causing multi-second pauses is approaching OOM and the system attempting some last-ditch efforts before unleashing the OOM reaper, but that's not strictly swapping, it'd also happen on a swapless system.
If it's only a single fault I'd expect it to be to be dominated by IO latency, which is pretty good on NVMe. As the article shows those 40ms were accumulated over hundreds of pagefaults. The problem there was that there was a stop-the-world pause stalled by all those pagefaults together.
A fully-concurrent and swap-friendly GC you could maybe define that it only increases the active working-set by x GB on top of what the application itself (and the rest of the system) is actively using and will only page-in y GB per second. And it'd do so on GC threads, not the application threads.
Whether that would be sufficient to not cause a swap storm would still depend on all the other stuff happening besides the GC.
Do not write an unconditional latency SLO for a swapped runtime.
Set the latency limit to apply only when memory pressure equals zero. Monitor /proc/pressure/memory. When swap stalls occur, the fault belongs to capacity planning, not runtime GC performance.
This matches what we saw in production. Our Go service had great GC pause numbers in staging, but on the host the VM was overcommitted, so major page faults during the mark phase showed up as multi-second pauses that the runtime couldn't explain. We started tracking major faults right next to the gc pause metrics, and pinning the process memory in the cgroup cleared it up. The runtime's numbers are only as honest as the OS underneath.
When my system starts using swap space, I've already given up all hopes of a decently running system, so IMHO this particular "bug" doesn't matter much.
I'm being downvoted, but honestly, is anyone always testing their software while swap is active?
> I'm being downvoted, but honestly, is anyone always testing their software while swap is active?
Yes, not only is my system using swap 100% of the time, but I am working on and using an application (Rust) which often is used during a high demand of swap and expected to run reliably.
In general when you tune knobs for GC, you pay for benefits in one area with sacrifices in another. Two big knobs to turn are pause latency and throughput. You probably wouldn’t want to go full “optimize for latency” because you’d end up with poor throughput. Also vice versa. Java’s reputation for poor GC performance is partly due to historical defaults that tune it for throughput.
Go’s GC is already a “concurrent mark-sweep garbage collector” and already has “extremely low mutator pause times, on the order of tens of microseconds”. It sounds like on-the-fly is just a different flavor of what Go already has.
It's a well known algorithm. Folks who do GCs for a living know about it. The folks who work on Go are surely aware of it. I'm assuming that they do not use it for a good reason, hence my question!
Fil-C's GC (Fil's Unbelievable Garbage Collector) uses an alternative on-the-fly algorithm, which I call Phil's Concurrent Marking.
Phil's Concurrent Marking differs from DLG in that it only requires a Djikstra barrier and uses a permagrey stack (something that Go used to do).
However, FUGC does clever things for coroutines (as in ucontexts, which Fil-C supports) - they are not permagrey; they only become grey if they execute. That's relevant to Go because Go moved away from permagrey stacks because of coroutine scan overheads, which the FUGC coroutine strategy might avoid.
But even if Go could not go back to permagrey, then the answer would be to use DLG, which would involve using the combined Yuasa+Dijstra barrier, which Go uses today anyway
Wow, thanks for that. I have been hacking creating a novel memory reclamation scheme for Go and somehow I missed that. It is similar to something I have implemented but DLG has yhe model for a TLA proof and it might indicate a possible improvement that I wasn't making.
I'm not aware of any way to find that out as an application. Even if it exists it will probably be a full system call and therefore prohibitively expensive to defensively do it everywhere.
Databases can do this via the page cache since that's basically an implementation of swapping where the DBMS has full control.
Edit: anyway, the problem in TA was that GC metadata had to be accessed. The GC cannot delay this. It might decide to abort the stop-the-world phase so the application can continue while GC metadata is swapped in, but then one runs into liveness issues since next time the GC runs it might already be swapped out again.
Reference counting has high runtime overhead, especially atomic reference counting with multiple threads. If you overwrite a pointer, you'd need to also look up and update two counts and do all that in a manner that is effectively atomic -- and that is complex on current hardware. I've heard that Swift programs could have as much as 40% runtime overhead from ARC.
I still think that reference counting is promising though. First because it meshes well with static analysis memory-management techniques such as inference of uniqueness and borrowing -- that can optimise away RC altogether. (and Swift's compiler already does some of that). I have not seen any work that could optimise away tracing GC in a similar way.
Second, because I believe that it would be possible to design hardware with object-memory addressing that would performs atomic reference counting with no additional runtime cost.
ref counting is expensive in multi-threaded applications. Overall it would have worse performance. When it comes to predictability: deallocating a linked list (for instance) would have to deallocate all of the elements. Dealing with reference cycles is also not simple, either.
Linked-list freeing need not all happen in one go. There can be a queue of free-but-not-zeroed, push RC=0 stuff there instead of recurring on referents, anyone doing memory allocation work can pop a few items off that queue and decrement the counts of their referents, etc.
That loses timeliness, adds problem of queue management.
A weird trick I stumbled across years ago is to not strictly pop from the head of the stack/queue -- instead choose randomly from the last N elements of push end of the stack. This seemed to blunt the sort of growth that you get from "visiting wrong". I never did figure out the theory of why this worked.
Because the industry has plenty of experience with referece counting as the very first GC algorithm, already in the early 1960's, in early Lisp implementations, BASIC, Cedar, and several other languages.
The predictable runtime performance is also a myth, because they never take into account the use of NUMA memory, lock contention, possible stack overflow and stop the world in the case of cascaded deletions in naive implementations.
Can you elaborate on how cascaded deletions "stop the world" with reference counting? I understand how GC could lead to arbitrarily large latencies for whatever task triggers that kind of cascaded deletions, but as I understand it, "stop the world" usually means that no threads are allowed to execute application code.
Imagine a graph or tree data structure where the deletion of a node causes a cascade deletion of all child nodes, which also causes a deletion of their children and so on.
This is proportional to the data structure being deleted.
Unless you use techniques to move the deletion into background threads, e.g. C++/WinRT with COM AddRef/Release, the thread will be "blocked" doing busy work cleaning all those nodes, running the cleanup code (destructors, deinit, whatever), node after node.
Yes, that's exactly what I said I understood. When Go (and Java) people say "stop the world" for GC, they mean all goroutines/threads stop, not just one. I don't think you get that with even naive reference counting.
Places that were doing C++, but found out that the right GC, and JIT compiler, can achieve good enough performance for their business case.
There is nothing "special GC" about it, the failure is to assume there is only one way to implement GC algorithms, as if there is only one way to implement hash tables, tree re-balancing algorithms, .... and then place all languages into the same bucket.
Then we have the modern times with AI driven code generation, where no one cares how their agents are actually doing the work, with what kinds of resource management approaches.
Immediate reclamation, yes. There do exist a number of reference-allocation algorithms that performs deferred reclamation similar to how tracing GC does.
There do exist algorithms for resolving them, and that process can too be deferred, but also restricted to subgraphs and only when cycles are possible.
Not that much more. Drop the last reference to e.g. a tree and you're doing an arbitrary amount of work to free it, right there. People resort to hacks like sending messages to "freeing threads" dedicated to that.
Either the size of the data structures created is limited and known, or it isn't. Whether the work is done in a refcount-decrementing recursive descent or a tracing garbage collector doesn't change that part.
And a refcounted tree freeing won't necessarily run at the end of the current scope, that depends on what other work is being done currently, and the size of the tree is a subset of all allocations until last user drops it.
and yet whether or not other work is being done currently is in your control, thus predictable (unless you chose to give control out of your hands e.g. by introducing two threads racing against each other); same regarding the subset
This. Memory bloat bugs are relatively easy to fix in Go, but sometimes it’s an adventure to remove swap bloat. And I think it’s worth removing all swap bloat!
The nasty bit is that swap doesn't just make the allocation slower; if GC metadata gets paged out, you have turned memory pressure into a stop-the-world latency spike.
Unfortunately that's somewhat expected - regardless of implementation, the sole fact that the GC needs to somehow walk the entire tree to trace still referenced sections makes swap highly impractical -- even if you had not had 40ms stop-the-world pauses the LRU cache used by swap would get thrown away at every GC.
I believe that's actually the same reason why Apple stopped using GC in their frameworks in favour of automatic reference counting.
> Unfortunately that's somewhat expected - regardless of implementation
Yes, that's expected and no not "regardless of implementation": the GC implementation CAN be improved.
See this discussion on reddit: https://old.reddit.com/r/programming/comments/1wf2fei/40ms_g...
Copy/pasted here: >>
I remember a research paper about swap and GC, where the GC cooperated with the OS to avoid this kind of issue. AFAIK it went nowhere, too bad.
[–]andreiross[S] 15 points il y a 21 jours
Are you talking about this one? https://cse.buffalo.edu/\~mhertz/bc-pldi-2005.pdf. If so, yes. Too bad. I don't know the repercusions this paper had in the past, though, in the sense of pros and cons of the bookmark collector. Don't know if anyone tried to actually implement it or design it at some point.
[–]renozyx 9 points il y a 21 jours
Yes, congratulations for finding it. And I don't know either,. Except that they did implement it on Linux (of course) https://plasma.cs.umass.edu/emery/cooperative-memory-managem...
<<
Modern OSes need a facility to signal to a thread that it hit a paged out chunk of memory. It's not like it's not possible to create a swap-aware GC (or any other kind of code that touches lots of memory).
I would go further - I think the whole swap lifecycle needs to be communicated. Before the OS swaps out a page, if it could invoke the GC which would clean up that piece of memory so that we dont end up writing garbage to swap.
The OS should also allow marking pages as piority to stop them from being swapped out.
> The OS should also allow marking pages as piority to stop them from being swapped out.
This is IIUC possible using the mlock(2) family of syscalls: https://man7.org/linux/man-pages/man2/mlock.2.html. (On Linux, though I'm guessing other UNIXen and operating systems have it or something equivalent.)
Cool. So this should be a bug ticket, not irrefutable proof that GCs suck :D
Nor to falsely indicate that swap is bad.
https://www.kernel.org/doc/html/latest/admin-guide/mm/userfa...
I'd guess it ain't gonna perform great either, though.
Reference counting is a GC algorithm, and no this isn't expected, it depends pretty much on the implementation.
Many make the mistake to think there is only one way to do a GC.
One of the authoritative books on the subject, https://gchandbook.org/contents.html
And a quite well known paper on the matter as well, https://dl.acm.org/doi/10.1145/1035292.1028982
Whether reference counting is a GC algorithm depends on how you define what GC is.
I prefer to consider GC only the methods of memory management where reclaiming the no longer used memory is done either asynchronously with the main program or as late as possible, i.e. when new allocation requests cannot be satisfied.
In the normal implementation of reference counting, memory is freed as soon as possible, i.e. exactly like stack memory, when blocks are exited, so I do not consider reference counting as GC.
The problem with GC in the strict sense is that you cannot predict when it will happen. With both stack memory and reference counted heap memory you know that whenever you exit a block, some time will be spent with running destructors and for freeing memory, but such interruptions will not happen in other points of the program.
> Whether reference counting is a GC algorithm depends on how you define what GC is.
Pretty much all the high-performance GC/refcounting algorithms are hybrids in one form or the other; it's a spectrum of choices. https://dl.acm.org/doi/10.1145/1028976.1028982 explores this in some detail.
That is a classic paper and obviously I am aware of it.
However, if you have distinct names it is efficient to use them with distinct meanings.
Making "garbage collection" synonymous with "freeing memory" is bad, because it eliminates a means to distinguish various methods for freeing memory.
Like I have said, I consider useful to define "garbage collection" as any method of freeing memory where the memory is not freed as soon as possible (i.e. when a block is exited), but freeing is deferred to be performed at a later time, even as late as possible (i.e. when new memory allocation requests cannot be satisfied).
Indeed, many garbage collection algorithms use reference counts, where memory deallocation is deferred, but when I use the term "reference counting" without any other qualifier, I mean it in the sense in which it was originally defined in 1960, where the time when memory deallocation is run is predictable, exactly like for stack-allocated memory.
I prefer to write programs with well-defined worst-case behavior, so I normally prefer deterministic algorithms. Thus I always prefer to use reference counts instead of GC. I have never encountered a case when avoiding reference cycles was difficult.
I define the academic view of GC algorithms in Computer Science, not what random developers decide to call GC.
Which in an industry where some folks call themselves Software Engineers after a bootcamp, without any kind of accreditation, I rather stay with the definition from those that do language design and compiler algorithms research.
I prefer to use the terms as they were originally defined by the authors who introduced them, and not with the modified meanings that become fashionable after the authors of some paper written some decades later decide randomly to change the definitions of the old terms. This is also true for the terms "garbage collector" and "reference counts", which were both introduced in 1960 to denote 2 clearly distinct methods of memory management, and the main difference between them does not consist in whether some kind of reference counts exist somewhere, or not.
The paper "A unified theory of garbage collection", which has started the fashion of considering reference counting as a kind of garbage collection, only shows correctly that both tracing and reference counting are complementary implementation techniques for a garbage collector.
It does not mention anywhere the essential practical difference between the traditional standalone memory management with reference counts and a garbage collector, which stays the same regardless whether the garbage collector also happens to use reference counts for some purposes, which is the difference between predictable and unpredictable times when memory reclamation is done.
Many of the modern authors of academic papers are a poor model of using computer terminology (or for the terminology in other domains), because very frequently it is obvious that they have not read the old works where such terms were introduced for the first time, even when such works are cited in the bibliography.
Unlike them, I have done an extensive research to find when and where various computing terms have been used for the first time, and I strive to use most terms with their original meanings, not with corrupted meanings, even in the cases when the latter have become more popular lately.
So as fellow digital archaeologist I would find interesting to find Lisp, BASIC, CLU, SIMULA, or Cedar papers, where the authors mention reference counting not being a form of garbage collection algorithms.
Cedar was already combining reference counting with a cycle collector, as one of the very first systems programming languages with automatic resource management.
I think it's saner to be literal here. Reference counting is a way to automatically collect garbage for you. Why shouldn't it be called a GC then?
Because some people are allergic to the word GC and pretend that a duck isn't a bird.
> The problem with GC in the strict sense is that you cannot predict when it will happen.
It’s the same with ARC. You also don’t know when the counter will reach zero.
No, that is not true.
In the normal implementation of reference counts, counters can be decremented only at block exits and not at any other program point.
At a block exit some of the local variables that are freed may contain references, so freeing them will decrement some reference counts. Then some counters will reach zero, triggering other deallocations and the decrementing of other counters. This will repeat until no other counters reach zero.
All the memory deallocation happens predictably, only at block exits.
If a variable is not freed immediately when a counter reaches zero, but the deallocation is deferred for a later time, which is not predictable, that is no longer classic memory management with reference counts, but it is a garbage collector, which happens to also use reference counts, probably in combination with some tracing algorithm.
When reference counts are implemented, manual memory deallocation, like with C free() or C++ delete, should be forbidden, but even if it were used that would just introduce other program points besides the block exits, where it is known that memory deallocation will happen.
You've described where deallocation can happen, not when it will. Every block exit is a candidate, but which one drops the last reference depends on runtime state: how many other owners the object has and which of them goes away first. With shared ownership across threads it's a race by construction. The free runs on whichever thread happens to release last.
The cost isn't bounded either. Dropping the last reference to the head of a list or the root of a tree frees the whole structure at that block exit, and its size is a runtime property.
And it isn't only block exits. Every assignment to a variable or field holding a reference decrements the old target, and so does removing an element from a container. Swift's ARC doesn't even promise the scope boundary: the optimizer may release right after the last use, which is why withExtendedLifetime exists.
By the same "where" criterion a non-concurrent tracing GC is predictable too, because it can only run at allocation points. That doesn't tell you which allocation will trigger it, just as knowing the block exits doesn't tell you which one will free.
In common use, GC refers to mark and sweep and similar systems where allocations are not freed at an easy to predict time.
In common use, by those that never learn the theory behind their tools.
That is a tracing GC by the way.
There are also tracing GC implementations with deterministic resource management APIs, .NET and D have them for example.
People use words in a way that is useful. Really this just shows that the pedantic academic definition of GC is not the most useful one.
Useful to them usually, or it goes to show how little our profession cares about learning the proper terms, or its historical background for that matter.
Anyway it doesn't matter any longer, AI is going to replace most of us, and it comes with automatic everything.
You just need to remember that garbage (unreachable reference cycles) isn't collected.
The issue described in the article isn't really GC specific. The Go GC has a critical section that reads metadata.
Any program that has a rarely accessed, but vital chunk of memory is vulnerable to this sort of issue.
A region-based collector like Hotspot's G1 combined with madvise could probably operate in a manner that's more swap-friendly by focusing on a smaller working-set at any given moment and announcing its intent to switch the working set to the OS ahead of time.
Apple should just stop using a silly mark & sweep collector. All the good collectors are copying, if you have enough memory. Only on tiny devices you need mark & sweep.
That "good collector" has higher memory usage, has to pause mutator threads, and makes application code run slower. Thanks, but no thanks.
Copying GC will make swap even less efficient, no?
A Copying GC will not interfere with swap, as it's reserving 2 fixed memory regions, not more. And no stop the world.
All the better systems use that for decades.
Discord learned this back in 2020 and published a great blog post on it: https://discord.com/blog/why-discord-is-switching-from-go-to...
I don't understand why they felt necessary to rewrite a core component in another language. If you have a garbage collection problem my first intuition would be to produce less garbage!
For example better using the stack, or pulling out the big gun of manual memory management.
I'm sure they had reasons to choose Go when they first designed this project but they don't go into them at all.
Feels like they just wanted to play with a new toy.
> If you have a garbage collection problem my first intuition would be to produce less garbage!
FTA:
“These latency spikes definitely smelled like garbage collection performance impact, but we had written the Go code very efficiently and had very few allocations. We were not creating a lot of garbage.
[…]
the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references”
> produce less garbage!
They explained in the post why this wasn't an issue: they were producing very little garbage, but there was a very large object graph.
> manual memory management
If you need to do manual memory management in a GC language with no builtin support for it, like Go, that's probably a sign that you should switch to a different language.
Like D, a proper systems language with GC, and builtin support for manual memory management, and there are others as well.
If you read the link you'll see that they found out go forces a GC every two minutes no matter the amount of garbage.
And importantly in a tracing GC the work load is a function of the non-garbage data not of the garbage.
Generational GC can avoid some of it by ignoring the old code entirely, but Go does not implement generations because its relatively strong ability to stack-allocate generally replaces the nursery, and for most work loads you don’t recoup the costs.
Problem with languages like Go is, because they have GC, their API is less tuned to the ability to prevent new allocations.
In C you see lots of functions whose first parameter is a buffer that will be re-used.
And even in Rust you don't see this often, because reusing a buffer means having to do &mut, which means having to re-structure your code. And it's really easy to do Vec::new().
One of big issue with GC is it need to scan every reachable objects to mark it is reachable. That mean even if your code produce no garbage but have a large amount of long lived object the GC still need to traverse all of those objects every cycle.
Only in very bad implementations of GC.
By that definition, Go has very bad GC. Many (myself included) would strongly disagree with this.
Every memory management scheme makes trade-offs (even manual ones), and golang's is no different. It seems to work well in many scenarios.
It is better to scan everything than to copy it.
The op, most likely, was referring to generational GCs. Golang notably doesn't have one.
That’s a great read. And an incredibly annoying “floating footer”
If you're on desktop, you can use uBlock origin's element zapper to remove annoying things like that.
The zapper works on android Firefox too
"It hurts when I do this"
"Stop doing that"
If you care about latency, disable swap. System wide or for the specific the cgroup.
This is not entirely correct. If you care about latency, then it doesn't matter on which major fault your application gets paused. Disabling swap protects you from data pages being evicted, but code pages can still be paged out.
If you care about latency, mlock() your memory, do not disable swap. Swap is good and gives the kernel an equal opportunity to evict data and code pages.
For GC enabled languages swap is universally bad. Some gc-pauses are indistinguishable from a system crash. It's a side effect on not having tightly specified memory limits.
I'd rather have applications be oom_killed than having them swap out, the former is rather obvious and demands action.
If you want you application to stay in memory, then make it explicitly with mlock()/mlockall().
Disabling swap will just moves pressere elsewhere: to code pages. And evicted code page is no better: full stall while kernel loads that page from disk.
FYI: This `delamon` account seems to be meat proxying us. Just relaying garbage AI answers without understanding his own words. Doubt he's ever used mlock himself for anything.
His reply below was presumably killed by mods for doing this: https://news.ycombinator.com/item?id=49970841
I'm not. If you think that I said something not correct, do correct me. We're all learning.
I see that you've rewritten your AI generated reply there. So when have you used mlock in production for Go services, the way you're advising others to do? What were the circumstances?
I did not rewrite anything.
Yes, I was and still running go and zig binaries with mlockall. Apps have large in-application caches and also perform large streaming reads of datasets bigger than available RAM. If it happens that several streaming reads running at same time, they push memory pressure hard enough, so kernel swaps out cold parts of in-app caches. That later leads to elevated latencies => mlockall to avoid that.
While I've seen systems grinding to a halt due to evicting code pages, the idea that disabling swap distributes load between code pages and data pages makes no sense to me. Data pages are also disk backed, so they could also be evicted at any time. Why would data pages be written to swap if they came fron the disk in the first place?
So that only leaves anonymous data pages, i.e., regular data in memory, that could be swapped. Ok but you're not easing the load on code pages, those still get evicted during memory pressure whether you have swap or not.
The assumption that data pages can always be easily discarded is incorrect. Dirty data pages can't be discarded until they are written back. Recently accessed data pages are also not discarded until all inactive data and code pages are discarded.
I think discussing it in a vacuum is pointless, it should depend on how much memory you have. You can have a swap on, but with a low `vm.swappiness` number.
The point is that disabling swap does not prevent page eviction. Instead of swap, this discussion should be about the page cache.
Surely as long as it is all in memory these pauses are not going to be that significant, details like that should be left for the language devs to optimize.
You can tell an average Golang user to use mutexes, you shouldn't be telling them to lock memory manually.
vm.swappiness doesn’t work the way most people assume it does. and it is ignored under heavy memory pressure.
Maybe, but it still there and probably has a different value between distros as well. Default value is 60, what is that good for exactly?
If you're in a position where you still want swap to be there, but only used in extreme situations, then you should set it yourself. Otherwise disable swap or switch a language.
What do you think this value of 60 means? Also, as I said, in extreme situations, when the system is right before invoking the OOM killer, swap is used regardless of `vm.swappiness` setting.
> What do you think this value of 60 means?
Some arbitrary number.
https://docs.kernel.org/admin-guide/sysctl/vm.html#swappines...
> At 0, the kernel will not initiate swap until the amount of free and file-backed pages is less than the high watermark in a zone.
I don't see a reason why the default shouldn't be 0 then on modern systems.
Value 60 is not an arbitrary number. It makes the kernel reclaim 140 file pages for each 60 swapped-out anon pages. This proportion is supposed to reflect the relative performance cost between sequential file access and random swap I/O on HDD. For SSDs, the value 100 is more appropriate.
> I don't see a reason why the default shouldn't be 0 then on modern systems.
The zero value makes reclaim very unbalanced and is not appropriate for generic systems. This will cause the kernel to discard all the page cache, slowing down the system with uncached I/O, while keeping all unused anonymous pages in memory.
I don't get the point that you're even arguing. Are you saying that 60 is fine as a default? Or are you saying that people should even think about vm.swappiness because it doesn't matter? The alternative here is to disable swap completely, that's what started this whole thread.
> If you care about latency, disable swap.
Disabling swap disables loading a process' binary to memory? Or do you mean that evicted code pages will need to get loaded from disk every time.
There are two types of pages: anonymous and mapped from files. The code pages are mapped from a binary; they are not very special.
Under memory pressure, the kernel evicts less popular pages from memory. If a page has been mapped from a file, it is dropped (if dirty, then it is written out first). If it is needed later, the kernel can read it back from the file. If a page is anonymous (read: heap page), then there is no backing file and the kernel copies it to swap before dropping it. This is swapping.
So, what happens if you disable swap and the kernel is low on memory? What can it evict? Anonymous pages cannot be evicted: there is no swap to put a copy in. The only choice the kernel has is to evict pages that are mapped from files. Those include pages mapped from the executable. You don't eliminate stalls by disabling swap, you just move them elsewhere: the kernel will page out code and your app gets paused whenever the execution flow hits such a page.
Thanks for the clarification, learning everyday.
I haven't had to analyze the performance of no-swap processes before. My assumption is that code is hot enough to avoid eviction and that evicted code pages are rather the exception. To strong-man the argument, I can imagine long running complex (bloated) services could have parts that are not touched unless a specific request comes in.
>So, what happens if you disable swap and the kernel is low on memory? What can it evict?
Your userspace early OOM killer triggers and lets you know you're trying to run more than will fit in memory so you don't do it again. (In my experience, the kernel OOM killer can't be trusted to kill processes soon enough.)
You are missing that the file LRU is not only the code pages or memory mapped files but all file data. The active code pages are specifically excluded from being discarded during the reclaim process.
I wish this is true but evidence is against you. Just try testing memory pressure on a no swap system. You will see that the system grinds to a half for a few minutes before you even get to an OOM event.
Could you clarify what exactly is "not true"? Also, could you explain what exactly your evidence proves? What do you think happens when the system becomes unresponsive under memory pressure, and why?
> The active code pages are specifically excluded from being discarded during the reclaim process.
This is not true, or the system wouldn't grind to a halt due to having to reload code pages (even those that are recently in use) from disk over and over.
For example, key presses and output streaming in SSH become unresponsive. The code pages for handling key presses and output streaming are obviously in use — I just used them!
I'll try again. Why do you think that unresponsiveness is caused by code pages being discarded? Do you have evidence for this? Have you considered that there are other mechanisms that cause programs to become unresponsive under memory pressure?
It's the explanation that I've read when scouring forums for reasons why it happens. I'd love to hear alternative explanations and how to measure/verify them. But the measurement tools also become unresponsive so I can't check anything.
But frankly, as a user I don't really care what the real technical reason is. I just don't think the system should become unresponsive for minutes on end during memory pressure. Just OOM-kill earlier, when it's supposed to.
The assumption that a system gets unresponsive due to code pages being reclaimed doesn't make any sense. It is a common myth, cargo-cult knowledge. The kernel doesn't discard any active referenced pages at all, and the active code pages are the last that are getting reclaimed.
The proper explanation is relatively straightforward. Normally, any memory request for a process is served by getting a page from the free memory pool. But when the system free memory goes below the low watermark (calculated from vm.min_free_kbytes and vm.watermark_scale_factor), the kernel stops giving the pages from the free memory pool and instead goes into direct reclaim, trying to find unused pages. While the kernel does reclaim, the allocating process remains stalled. The kernel can't return control to the process - there is no page mapped that the process tried to access. This applies to any types of pages - anon, file-backed, code pages. So, under the condition of free memory below the low watermark, any process trying to allocate memory gets stalled. This is measured as memory pressure stalls (PSI) in /proc/pressure/memory.
The problem with the system being stuck in an unresponsive state is that the design choice for the kernel OOM handling is to be optimistic and keep the workload running as long as the kernel is making any progress reclaiming pages. The direct reclaim process needs to fail to make any progress 16 times before the OOM killer is invoked.
The solution is to use cgroup memory limits to avoid getting into the situation of memory exhaustion and use user-space oomd to recover from this situation earlier.
Perhaps I wasn't clear, but I did say that all file pages are treated equally. I would assume that page with PC is excluded from reclamation process, but other code pages are valid targets.
All code pages in the active file LRU are excluded from the reclaim.
All recently referenced code pages in the active LRU are excluded from reclaim.
Code pages are paged in on demand: for any sufficiently large binary just some code pages will be mapped right after start. Given enough memory pressure, the active LRU will be shrunk often enough to push code pages out of the active LRU and then they are eventually evicted.
I'm not aware how it works in modern MGLRU, but I would assume similar ovservable behevior holds.
Active code pages with the "Accessed" bit set are specifically excluded from being moved to inactive LRU when active LRU is being shrunk. Instead, they are moved to the head of the active LRU. All other pages are moved from active LRU to the inactive LRU unconditionally, even if they have the "Accessed" bit set. So, no, the active code pages are not going to be eventually evicted.
I think we are saying the same thing. Active code pages are not evicted; non-active code pages (that is, pages that had not been accessed since last time of active LRU shrinking) may be evicted.
Under memory pressure the system will free the ram used for code pages because it can always load them back from the executable on disk. It’s the same virtual memory mechanism as swap but without needing dedicated swap space. The op is saying disabling swap doesn’t prevent long pauses during memory pressure, because the system just swaps code out instead of dynamically allocated memory.
There are good sibling replies answering your question, but to give a specific example: if your application writes something but then doesn't need it anymore. Since we're talking about go, perhaps a data structure you need also holds a reference to data you don't need.
When the kernel needs memory, it goes hunting for a page it can discard. But since that's transparent, the kernel can only discard a page if it knows it can get it back (after all, it's still got valid data on it, and maybe you'll try access it again later).
If there's swap, a page full of stale/unneeded data can be written out to swap. But if there's no swap, your page of "dangling data that you'll never use, but is still valid & referenced" can't be discarded; the kernel doesn't know you won't want it later, and it can't recreate the page if it throws it away.
So like sibling said, at that point it has to find other pages it can evict from memory, ones that _do_ have somewhere persistent they can be written out to. Pages loaded from binaries on disk satisfy that, so those will get dropped instead.
For JIT languages most of the code is dynamically generated, not available to be loaded back from the disk. Morealso, the allocations do happen at large chucks where the oom_killer also happens, so it's significantly less of an issue.
That doesn't necessarily improve matters. Now any code that gets JIT'd is also going to be permanently pinned in memory too, no matter whether it'll get used again, if you don't have swap.
Under normal conditions the code gets jitted after some use... and the VM is free to deoptimize it when it reaches its max allowed generated cache.
The only code in danger is the one from libraries (e.g. zlib)
Please read the article. The issue was caused by swapped out metadata being read in a critical section, not typical heap traversal shenanigans.
This is not exclusive to GC, any program that reads a rarely accessed piece of memory is vulnerable to this.
Run your process in a cgroup that's set to disallow swap, then. The rest of your system benefits from having swap. This was an easy and safe thing that required minimal convincing to roll out to production at my company.
Plus, it wouldn't necessarily be that hard to selectively mlock the pages accessed in STW critical regions.
It would probably make sense to disable swapping "code" also for non-GC languages (or at least do it as a last resort).
No, do not try to use mlock to avoid swapping, except in very specific scenarios. That way lies madness.
To be more explicit about my advice: disable swap and ensure you have sufficient memory for your workload.
Your 14 MB Go binary is not the source of your memory pressure. Your memory pressure is coming from the allocated anonymous pages supporting your programs data structures on the heap.
Have sufficient memory for your workload, monitor it, and let the Go GC and kernel manage memory normally. This will work exactly as you'd hope.
Can you elaborate, why mlock is bad? And how disabling swap is better?
You are correct, that 90MB Go binary is not major memory consumer. But it doesn't matter what is the source of memory pressure. What matters is what kernel do under pressure. If you disable swap, then under memory pressure kernel can only pageout memory mapped files, no matter how small they are. And your code will be paged out.
We're trying to solve a problem: we don't want huge latency spikes caused by swapping memory to disk. The only good solution to that is to have sufficient RAM. It's that simple.
Introducing mlock is operationally annoying and trying to solve the wrong problem. It requires elevated container permissions, it's not something your SRE team is going to expect, few programs do it, and it has all kinds of technical implications and interactions with kernel OOM system, forking, the Go GC, etc.
And nothing about mlock solves the problem of having insufficient RAM for your workload.
There are very specific scenarios where mlock is exactly the right solution but the typical Go HTTP service is not one of them. The KISS principle applies.
Using mlock on all the code and static pages in a binary at startup is not hard or risky.
I didn't claim it was hard or risky.
Have you actually done this yourself for Go programs that you've deployed into production? If so, what were the circumstances? Was it your solution to latency spikes?
It was the default for every production binary. I read the c++ implementation, not really that crazy. It was for latency, and in some cases (with other measures) to keep the binary running if the disk failed.
The alternative (which I’ve also done), is to put the binary in tmpfs, after making sure it’s not bloated.
Your Go binary was being swapped out due to memory pressure, causing latency, and using mlock was your fix for this problem?
Or did someone else do this work and you assumed it was effective? Or did you personally implement this change and then validate that the before/after metrics improved as a result?
As I mentioned, it had long been the standard, the core library gave you a simple flag to use. I would occasionally find cases where it was not set correctly and you could see it causing short spikes (the simplest way to measure is something like a pacemaker task on an event loop, or task scheduling delay since it strips out other variables).
The impact is pretty obvious.
Alright, so you didn't implement it. And you're describing a "pacemaker task on an event loop"? That doesn't sound like a Go program at all, where we don't have a single event loop to measure and we have built-in runtime metrics.
Just seems kinda weird to give advice on forums when you're not speaking from actual relevant experience. I'm getting meat proxy answers and confident replies from people who've never done what they're suggesting others do.
As I said, I was more familiar with the C++ implementation, but we had it for every language in use.
I ultimately change the runtime environment to no longer allow any local durable storage for job (with a handful of exceptions), including swap and binary staging. The fact that almost everything was already using mlock made the transition simpler.
I am struggling to understand which specific problems that pertain to the use of mlock(2) you are alluding to.
Java is an example of the most widely deployed garbage-collected language in production around the world, deployment number and scale-wise. Most of the world's missing critical systems today run on Java. Java can run as a process or in a container. Containerised Java deployments do not require unique operational SRE arrangements, nor do they require any unique/dedicated permissions to run in a container.
Upon start-up, the JVM checks whether the system has sufficient memory to satisfy the max heap size constraint, and it will duly fail to start up if not. Afterwards, the JVM will allocate the required amount of memory, and it will mlock it. Multi-terabyte[0] RAM JVM deployments[1] do exist and exhibit exactly the same behaviour. Java has had such behaviour since its inception.
The prevailing number of Java deployments run on systems with swap enabled, and, from the operational standpoint, it is pure insanity to run mission critical (banking, payments, government etc) apps without a swap. Swapless JVM deployments are virtually unknown or are operational negligence.
The principal exceptions for running a JVM on a system without a swap are hard real-time scenarios and embedded systems. That is it.
What makes Go different or unique that it can't use mlock and that it would require special permissions or unique/distinct SRE/operational arrangements? It is eluding me.
[0] The Azul JVM supports up to 20 Tb of RAM which it will mlock.
[1] See the published Azul/Forrester evidence for reference. It is an interesting read regardless of the mlock debate.
I'm pretty sure almost all of our (very large) java systems were running without swap (it took effort to enable swap for your jobs, and eventually was almost impossible).
Java is generally simpler to run without swap since you know in advance how large the heap can be, so it's just a small amount of variability in the jvm internals to consider.
You don't want swap of heap/stack, or paged out code pages. The latency hit is pretty bad even for non-"real"-time systems. And really, you should know how much memory you are going to use (to within some fair bounds).
Adding only a small amount of swap for very cold pages does not save very much ram. And using it for anything else is very bad for interactive systems. In the same vein paging out code pages you won't ever run just does not save very much.
It requires some discipline in monitoring, load testing, etc, but you should be doing that anyway.
You are correct, of course: if your application needs 2x RAM and you only have 1x, then there is no way mlock can magically save you from swapping. What it gives is protection from a noisy neighbour.
The good thing is that mlock/mlockall does not require elevated permissions. But it is bounded by the memlock rlimit, which you need to configure for the container. If you don't want paging/swapping, set it equal to the total memory you give to the container and call it a day.
I'm curious about the technical implications. Which ones do you have in mind?
To be honest, I don't quite see how mlock makes it complex. It's quite the opposite: it has very clear semantics and makes reasoning about system behavior simple. Disabling swap is the opposite: you're making a bet and hoping it works.
In simple cases, instead of using `mlock` just set cgroup memory.min
That would work reasonably well. With one important caveat: memory controller tracks all memory, including page cache. Setting memory.min will prevent reclaim of page cache charged to that cgroup, as long as total memory usage stays below the limit.
Another important caveat: people keep assuming that page cache can be easily discarded and it doesn't count towards the working memory set. Then they get surprised by unexpected OOMs. There is a decade-old issue on Kubernetes https://github.com/kubernetes/kubernetes/issues/43916
Fun fact: being able to run garbage-collected languages (i. e. Lisp) was one of original motivations for adopting demand paging.
I use Go where i'd use Node, Ruby or Python.
I use Rust where i need low latency.
Choosing a language that forgives bad memory management practices and promotes sloppy programming practices is your decision. Don't be surprised that it has to do its job. You could have chosen modern C++ or Rust and use RAII and proper automatic deallocation.
I started wondering if there could be swap-aware GC, like first make the required page swapped in (not that there's any obvious API for that...) and only then pause the world?
You could do lots of interesting things with sufficiently deep inter-layer integration.
For example, why not swap out not by LRU page but by dense node clusters on the heap graph, maintaining in-memory summaries of inbound and outbound edges for liveness? If you do this, you don't have to swap the cluster in to do a GC involving it.
If the whole cluster becomes unreachable, you wouldn't even have to swap it back in to get rid of it: you'd just drop the swap reference and deem the swap space free.
I don't see anything this deeply integrated happening near-term, but it's fun to think about.
Aren't madvise and mincore the APIs you want?
Probably more mlock (keep memory area in RAM), mincore just tells you what is or is not in RAM, and madvise is about access patterns.
You could MADV_WILLNEED the GC metadata when you start the GC process hoping they’ll have been paged in by the time you STW, but assuming that area is not massive it’s probably a better idea to just prevent it being paged out.
I was assuming they meant for the heap. Not even executable memory is locked though, so if you mlock too much memory the kernel will page out your executable instead. Something has to give, so it's better to schedule it.
On a lower level, the OCaml and cpython GCs use a prefetch buffer during marking to schedule around the cache.
One can just add APIs to the Linux kernel, and this would be a pretty straightforward one. (though it might be a little more difficult to avoid a syscall here)
I'm sure an agent can work on this and get some numbers with a day's worth of tokens.
The API already exists: mlock.
Sure, but I meant more in the sense of asking about the current status of the page.
It’s the mincore which debugnik mentioned. And there’s madvise(MADV_WILLNEED) which triggers an asynchronous prefetch.
I was biten by not understanding madvise(MADV_WILLNEED) semantics fully. It will prefetch, but it will not pre-fault. You still gonna pay minor page fault on every new page access. There is MADV_POPULATE_READ, but it is synchronous.
You can ask about the status (mincore), but it's an inherently racy operation.
I think we are not realizing the paradigm shift here. A coding agent can implement this with a modest little day’s worth of tokens. This means that no one needs to read or know the APIs any more. We can just add them to the kernel as we need them. Then when we forget that we needed them we can just rediscover the API idea later and spend a day’s worth of tokens. (But let’s be real here. By then it will probably be just 1/3 worth of tokens with all the model improvements as well as the ample training material.)
We don't even need a kernel anymore. An agent can write the relevant parts of the OS on the fly, as we need them.
You don’t need an api. Just touch the page.
A GC latency SLO should include operating-system memory pressure. Otherwise, a page-fault problem will look like a collector problem and lead to the wrong fix.
Can’t page faults be arbitrarily long though? How could you provide a meaningful SLO in that case?
Swap latency explodes when you get into a swap storm where tasks continuously touch different pages and fight to get things paged in and out, forming a queue waiting for swap IO. The other thing causing multi-second pauses is approaching OOM and the system attempting some last-ditch efforts before unleashing the OOM reaper, but that's not strictly swapping, it'd also happen on a swapless system.
If it's only a single fault I'd expect it to be to be dominated by IO latency, which is pretty good on NVMe. As the article shows those 40ms were accumulated over hundreds of pagefaults. The problem there was that there was a stop-the-world pause stalled by all those pagefaults together.
A fully-concurrent and swap-friendly GC you could maybe define that it only increases the active working-set by x GB on top of what the application itself (and the rest of the system) is actively using and will only page-in y GB per second. And it'd do so on GC threads, not the application threads. Whether that would be sufficient to not cause a swap storm would still depend on all the other stuff happening besides the GC.
Do not write an unconditional latency SLO for a swapped runtime.
Set the latency limit to apply only when memory pressure equals zero. Monitor /proc/pressure/memory. When swap stalls occur, the fault belongs to capacity planning, not runtime GC performance.
Oh I see. only specify an SLO under conditions you can reasonably control. That makes sense.
This matches what we saw in production. Our Go service had great GC pause numbers in staging, but on the host the VM was overcommitted, so major page faults during the mark phase showed up as multi-second pauses that the runtime couldn't explain. We started tracking major faults right next to the gc pause metrics, and pinning the process memory in the cgroup cleared it up. The runtime's numbers are only as honest as the OS underneath.
When my system starts using swap space, I've already given up all hopes of a decently running system, so IMHO this particular "bug" doesn't matter much.
I'm being downvoted, but honestly, is anyone always testing their software while swap is active?
> I'm being downvoted, but honestly, is anyone always testing their software while swap is active?
Yes, not only is my system using swap 100% of the time, but I am working on and using an application (Rust) which often is used during a high demand of swap and expected to run reliably.
Is there a reason why Go isn’t using on the fly GC, where there’s no STW at all?
What I found when I searched for "on the fly" is this: https://github.com/mthom/on-the-fly-gc — it looks like it has pauses, from the docs.
In general when you tune knobs for GC, you pay for benefits in one area with sacrifices in another. Two big knobs to turn are pause latency and throughput. You probably wouldn’t want to go full “optimize for latency” because you’d end up with poor throughput. Also vice versa. Java’s reputation for poor GC performance is partly due to historical defaults that tune it for throughput.
Go’s GC is already a “concurrent mark-sweep garbage collector” and already has “extremely low mutator pause times, on the order of tens of microseconds”. It sounds like on-the-fly is just a different flavor of what Go already has.
The classic on the fly GC algorithm is DLG, hilariously published in two papers, because the first one had a bug. Here's the second paper: https://caml.inria.fr/pub/papers/doligez_gonthier-gc-popl94....
It's a well known algorithm. Folks who do GCs for a living know about it. The folks who work on Go are surely aware of it. I'm assuming that they do not use it for a good reason, hence my question!
Fil-C's GC (Fil's Unbelievable Garbage Collector) uses an alternative on-the-fly algorithm, which I call Phil's Concurrent Marking.
I've documented it here: https://fil-c.org/fugc
Here's the source: https://github.com/pizlonator/fil-c/blob/deluge/libpas/src/l...
Phil's Concurrent Marking differs from DLG in that it only requires a Djikstra barrier and uses a permagrey stack (something that Go used to do).
However, FUGC does clever things for coroutines (as in ucontexts, which Fil-C supports) - they are not permagrey; they only become grey if they execute. That's relevant to Go because Go moved away from permagrey stacks because of coroutine scan overheads, which the FUGC coroutine strategy might avoid.
But even if Go could not go back to permagrey, then the answer would be to use DLG, which would involve using the combined Yuasa+Dijstra barrier, which Go uses today anyway
Wow, thanks for that. I have been hacking creating a novel memory reclamation scheme for Go and somehow I missed that. It is similar to something I have implemented but DLG has yhe model for a TLA proof and it might indicate a possible improvement that I wasn't making.
If memory is paged out we could defer cleaning out garbage from it.
I'm not aware of any way to find that out as an application. Even if it exists it will probably be a full system call and therefore prohibitively expensive to defensively do it everywhere.
Databases can do this via the page cache since that's basically an implementation of swapping where the DBMS has full control.
Edit: anyway, the problem in TA was that GC metadata had to be accessed. The GC cannot delay this. It might decide to abort the stop-the-world phase so the application can continue while GC metadata is swapped in, but then one runs into liveness issues since next time the GC runs it might already be swapped out again.
I don't get why people do not prefer reference counting, it has more predictable runtime performance
(though of course a swap is a swap - but you can "trigger" it depending on your memory or file access pattern)
Reference counting has high runtime overhead, especially atomic reference counting with multiple threads. If you overwrite a pointer, you'd need to also look up and update two counts and do all that in a manner that is effectively atomic -- and that is complex on current hardware. I've heard that Swift programs could have as much as 40% runtime overhead from ARC.
I still think that reference counting is promising though. First because it meshes well with static analysis memory-management techniques such as inference of uniqueness and borrowing -- that can optimise away RC altogether. (and Swift's compiler already does some of that). I have not seen any work that could optimise away tracing GC in a similar way. Second, because I believe that it would be possible to design hardware with object-memory addressing that would performs atomic reference counting with no additional runtime cost.
That is exactly what escape analysis does, granted the way it is done across implementations varies.
Such hardware has been designed in the past, Lisp machines, Ada machines, the famous iAPX 432 Intel's failure.
ref counting is expensive in multi-threaded applications. Overall it would have worse performance. When it comes to predictability: deallocating a linked list (for instance) would have to deallocate all of the elements. Dealing with reference cycles is also not simple, either.
Linked-list freeing need not all happen in one go. There can be a queue of free-but-not-zeroed, push RC=0 stuff there instead of recurring on referents, anyone doing memory allocation work can pop a few items off that queue and decrement the counts of their referents, etc.
That loses timeliness, adds problem of queue management.
A weird trick I stumbled across years ago is to not strictly pop from the head of the stack/queue -- instead choose randomly from the last N elements of push end of the stack. This seemed to blunt the sort of growth that you get from "visiting wrong". I never did figure out the theory of why this worked.
Honestly some (container) objects makes me think that they should be "parent obj management only"
Overhead per allocation, if you care about that, and reference circles.
Because the industry has plenty of experience with referece counting as the very first GC algorithm, already in the early 1960's, in early Lisp implementations, BASIC, Cedar, and several other languages.
The predictable runtime performance is also a myth, because they never take into account the use of NUMA memory, lock contention, possible stack overflow and stop the world in the case of cascaded deletions in naive implementations.
Can you elaborate on how cascaded deletions "stop the world" with reference counting? I understand how GC could lead to arbitrarily large latencies for whatever task triggers that kind of cascaded deletions, but as I understand it, "stop the world" usually means that no threads are allowed to execute application code.
Imagine a graph or tree data structure where the deletion of a node causes a cascade deletion of all child nodes, which also causes a deletion of their children and so on.
This is proportional to the data structure being deleted.
Unless you use techniques to move the deletion into background threads, e.g. C++/WinRT with COM AddRef/Release, the thread will be "blocked" doing busy work cleaning all those nodes, running the cleanup code (destructors, deinit, whatever), node after node.
Yes, that's exactly what I said I understood. When Go (and Java) people say "stop the world" for GC, they mean all goroutines/threads stop, not just one. I don't think you get that with even naive reference counting.
Lock contention yes, it's fundamentally a RC issue, but makes me think per-thread objects make more sense
I'm not sure most GC implementations worry about the rest neither (as by the several complaints we see going around)
The ones across several JVM implementations (there is a world beyond OpenJDK), and the CLR certainly do.
Yes, and the fact you have to have a "special GC" for your case makes me wonder that those constraints are not so obvious.
(makes me wonder who's buying those - things like Azul, etc)
Places that were doing C++, but found out that the right GC, and JIT compiler, can achieve good enough performance for their business case.
There is nothing "special GC" about it, the failure is to assume there is only one way to implement GC algorithms, as if there is only one way to implement hash tables, tree re-balancing algorithms, .... and then place all languages into the same bucket.
Then we have the modern times with AI driven code generation, where no one cares how their agents are actually doing the work, with what kinds of resource management approaches.
> more predictable runtime performance
Not really, reference counting can cause a single object deallocation to trigger an arbitrarily long chain of deallocations.
Yes but then you can move your deallocation out of the critical path
Variable time yes but you know when you're going to pay it
(I mean yes you can put your gc.run() there as well, but it might not give you the results you want)
Immediate reclamation, yes. There do exist a number of reference-allocation algorithms that performs deferred reclamation similar to how tracing GC does.
Yes, but that still leaves the circular dependency problem.
There do exist algorithms for resolving them, and that process can too be deferred, but also restricted to subgraphs and only when cycles are possible.
And tbh I don't think I ever saw a circular dependency in a system
I'm not saying they don't exist of course (or that GC/RC shouldn't cater for them) but it's a very specific use case
A doubly linked list already creates a circular dependency.
True (especially if it's a circular one) but those can be weakrefs/softrefs
> it has more predictable runtime performance
Not that much more. Drop the last reference to e.g. a tree and you're doing an arbitrary amount of work to free it, right there. People resort to hacks like sending messages to "freeing threads" dedicated to that.
the work is predictable and the amount not arbitrary
By that standard, all gc work is predictable and the amount not arbitrary.
no, it is not. A gc won't run at the end of the scope when the reference is droped. And if it runs, the amount of work is unknown.
Either the size of the data structures created is limited and known, or it isn't. Whether the work is done in a refcount-decrementing recursive descent or a tracing garbage collector doesn't change that part.
a GC still won't run at the end of the scope - it will run at an unknown time and do an unknown* amount of work at the times it is running.
* subset of all allocations until it runs
And a refcounted tree freeing won't necessarily run at the end of the current scope, that depends on what other work is being done currently, and the size of the tree is a subset of all allocations until last user drops it.
and yet whether or not other work is being done currently is in your control, thus predictable (unless you chose to give control out of your hands e.g. by introducing two threads racing against each other); same regarding the subset
It is also unknown how much work will be performed if ARC goes out of scope.
if the arc is the last: the data structure is destructed and deallocated + atomic check
if the arch is not the last: atomic check
You might lose 100 ns or 300 ms. When? Nobody knows. You lose nothing when going out of scope with GC.
as a programmer you know and you can even control it by designing the appropriate scope and moving between scopes
> You lose nothing when going out of scope with GC.
actually, you don't know
Yes, I can, e.g. by implementing deferred deallocation, which is a primitive GC form.
> actually, you don't know
I know because I’ve verified it.
> Yes, I can, e.g. by implementing deferred deallocation, which is a primitive GC form.
sounds more like a form of manual memory management
Manual management involves an explicit function call. Functions called by a destructor are considered automatic.
How does this relate?
What exactly don't you understand?
how this relates to garbage collectors
[dead]
This. Memory bloat bugs are relatively easy to fix in Go, but sometimes it’s an adventure to remove swap bloat. And I think it’s worth removing all swap bloat!
So, uh, mlock those pages?
[flagged]
[flagged]
Yet another reason why "no runtime" is such a huge advantage.
You're probably coding for a 'runtime' in the broad sense: libc and your kernel.
Microcontroller! The platform is the peripherals.
What is stopping the author to use latest version of Go and Linux Kernel ?
cheers my retarded friends !!!
The nasty bit is that swap doesn't just make the allocation slower; if GC metadata gets paged out, you have turned memory pressure into a stop-the-world latency spike.