We have these at work and there was a big outcry when IT tried to get rid of them. I use them from time to time to do things like: Keep git and other backups. Convenient place to scp files to/from. Upload files that coworkers can grab. I don't use it, but others used them as permanent IRC endpoints (using screen or tmux presumably).
Notice a cloud VM or container probably doesn't work here. You need something with a permanent presence, and shared between users (with separate Unix accounts).
Using IRC for the company IM is true old-school. I've worked at places which do the same, and also use SIP for A/V calls. Not being dependent on Big Tech, or the Internet for that matter, for services which exist entirely inside the corporate LAN is a great way to work.
We moved off IRC to Slack ages ago. Then they decided Slack was too expensive so we were forced to Teams which is bundled with the inescapable O365 license. We now use IRC again (UnrealIRCd) which runs on a debian VM on someone's workstation in the office.
irssi is surprisingly decent, and would even makes you think IRC the protocol was designed around irssi (and TUI), although the protocol is actually much older
Uh, how?.. It’s a conversation window with a contact list to the left, with GNOME-standard (libadwaita) look and feel. If you want separate conversation windows (like ICQ or Pidgin), then I understand the desire, but it’s pretty much orthogonal to web-view-ness (also very rare these days regardless of protocol).
Maybe Adwaita looks like that, I don't know. But there is a ton of whitespace, lots of clickable UI elements that look just like regular text, that kind of thing.
Mostly annoying network configs and token expirations etc. Not saying it can't be set up well, but in my experience some security guru gets a hardon for making my life miserable.
That depends... If your AWS account is already wired up the internal network then the ec2 is basically just another VM just like the onprem VMware servers or physical machines.
Now if all your AWS accounts are only public facing then yes it can get a bit more complicated .
Perhaps it is worth noting that all super computers I know (like the Dutch Snellius and the Finnish Lumi) are Slurm clusters with login nodes.
Bioinformaticians (among others) in (for example) University Medical Centers won’t get much more bang for the buck than on a well managed Slurm cluster (ie with GPU and Fat nodes etc to distinguish between compute loads). You buy the machines, they are utilized close to 100% over their life time.
Yes, I spend a majority of my professional life on similar systems writing code in vim and running massive jobs via slurm. Required for processing TBs of data on secured environments with seamless command line access. I hate web-based connections or vscode type system. Although open to any improvements, this works best to me. It’s like a world inside one’s head with a text-based interface.
Graphical data exploration and stats with R, python, etc is a beautiful challenge at that scale.
Aside from how slow and user hostile it is compared to a text editor, my biggest complaint about vs code is the load it puts on the login node. You get 40 people each running multiple vs code servers and it brings the poor computer to its knees.
Every job on an HPC cluster should have a memory and CPU limit. Nearly every job should have a time limit as well. I/O throttling is a much trickier problem.
I wound up having a script for users on a jump host that would submit an sbatch job that ran sshd as the user on a random high level port and stored the port in the output. The output was available over NFS so the script parsed the port number and displayed the connection info to the user.
The user could then run a vscode server over ssh within the bounds of CPU/memory/time limits.
One of the more prominent uses of Slurm to hit the headlines recently is by the data access centres for the LSST data from the Vera Rubin Observatory. Such as the U.S.A. facility in Stanford and the U.K. facility at the University of Edinburgh's Somerville.
I had 70% in mind but it certainly sounds reasonable (do you have a source? That would be great for our management). In our university medical hospital we have a very hard time to get rid of "shadow IT" because a single 50K machine can just process so much data (ie Next Generation Sequencing data) and can be amortized over 5 (probably 10!) years.
And then we aren't even talking about the EPD servers that are amortized in 4 years and can easily become a compute node in the cluster for another 6 (only problem are the bookkeepers who just can't live with post-amortized hardware! What a world!!)
I don't have any hard data, my role is somewhat HPC-adjacent, rather than directly in it, so this is mostly what I've heard. One way to look at it is that for most HPC operators, they are not charged any of the following for their gear, it's just provided by the organization as part of the larger pool: Power, Cooling, Real Estate, Networking, Security, Silicon-Valley SRE salaries, 38% Cloud Vendor Profit Margin.
Of course, the organization will pay for some of those eventually, so it's not fully fair to not roll them into the IT costs, but there are also lots of ways that non-profits also don't pay those costs at the same levels that the cloud providers do either due to differences in overall costs, or in providing a lesser level of capability. (As a quick example, cloud providers need extensive physical security for their datacenters. A hospital server needs a locked door, and can leverage the existing hospital security team for free.)
Cloud is great if your need is elastic, or if you have time-sensitive revenue dependent on your calculations. In non-profit research environments, that is often not the case. Users have compute they want done "eventually", but they don't really care if it's done in 1 hour or 4 hours; they have lots of other good work to do while waiting for the compute to run in their background.
Snellius was the Latin name of the Dutch mathematician Willebrord Snel van Royen [0] (" ... best known for Snell's law, named after him, which indicates how light rays are broken when light passes through different materials").
In this context, it is a Unix machine configured for general purpose use (so no minimal web-server-centric OS, for example) where everybody in the organisation gets a login (probably automatically and tied to whatever SSO they use). There may also be some niceties installed and/or preconfigured - like software used frequently by this and that research group or auto-mounting the user's file share from the central storage cluster.
I started with UNIX in 1993 with Xenix, have used most well known commercial flavours since then.
When I started working, our PC were thin clients for all practical purposes, we had development servers where everyone logged in to work, via telnet or remote X session.
Never ever did I heard about login server term as it is being described here.
I’ve seen this setup in many academic institutions - sometimes also called a jump server too. It handles auth to the rest of the intranet and HPCs, as well as storing the user’s common data like docs, code and dotfiles - shared across all subsequent ssh hosts.
Running a remote VScode backend on those while it is technically using the traditional shared server feels like that is not for the "shared" part but just because it is free and available?
Free and available is always good (especially in a research environment). ;) The author also says this in the P.S.:
"I believe the reason people run IDE backends on our login servers is because they have their code on our fileservers, in their (NFS-mounted) home directories. And in turn I suspect people put the code there partly because they're going to run the code on either or both of our SLURM cluster or the general compute servers."
I use VSCode's remote/ssh functionality all the time, particularly when I need to develop code on an environment that's more capable than my local machine (or when my internet is weak). Still use Git, no reason why you'd change that when working on a remote machine.
I mean rather than work off a remote machine for the convenience of having files you can deploy to a compute cluster use git (or maybe scp) to work local and then deploy them when needed. For a lower latency editing experience.
Open up other machines to the internet and recommend people upload their code to github instead of the school's file server. People who want to read email can use their web browser to load gmail or outlook depending on what the school goes with. For the cron jobs I would want to know what is being scheduled before providing a recommendation on how you can get rid of the login server for it.
It’s a University: students are there to learn. Mulituser UNIX machines are great place to learn; I probably wouldn’t be where I am now if I hadn’t cut my teeth on AIX, with ircii and pine for mail and X windows and a public_html directory making exploration fun and easy. Sure, you can give a student Outlook365 access and Wordpress, but will he learn anything from them?
The servers are for professors and graduate students, so this IT guy wants to block PhD students from doing AI research, demanding that they explain their cron jobs to him, because he thinks he can optimize them and save a few bucks.
And when he does, the blast radius is a single unix box/VM or even a single account. You don't put institutionally-critical stuff on things like this. If it gets hacked, you shut it down, have everyone who logged in during the window change their password, and restore from backups or rebuild from scratch.
Going around a CS research department and asking the population there to justify their cron jobs and other stuff they run sounds like an excellent use of a (rare-ish) university admin / IT staff's time... /s
And deal with the ire of Professor Foo (and grad students Bar and Baz) who want/need to use obscure software XYZ and have done so for years (or decades) without any fuss. And build some interface between your HPC clusters and Github. And keeping with regulations and agreements on privacy and security. And so on and so forth. All that for the low, low cost of... Well, certainly not less than keeping and maintaining like one to three unix machines that don't need no fancy hardware or other special attention in the data center you are maintaining anyway?! Why?
edit: By the way, from their documentation, the department mentioned by the author runs their own E-Mail-Servers (as many universities do - fortunately, in this world, there often still is a bit more choice than 'use Gmail/Outlook in the browser').
One of the things about Research Universities in particular is that there's always weird custom stuff everywhere. Thus "But why isn't it standard?" is a very stupid question. Doing the same things over and over but expecting different results isn't research it's insanity. So they're doing non-standard things because that's the whole point. There are systems that are "just" ordinary corporate stuff. Finance are not researching whether to pay suppliers, HR are not researching whether work contracts exist. But a lot of the organisation at any time is engaged in research, otherwise you're just a teaching university and to some extent that can actually be standardized and is the worse for it.
Because this is something to do and solving real problems is way more annoying.
If you work on a smaller cluster in a research institution where in silico work represents only a small portion of the research output the management of the cluster will sometimes be subcontracted out to the general IT support shop. There an administrator - usually with not nearly enough experience will start receiving support requests from users with decades of unix experience which take hours of research to solve. Unable or unwilling (and because inactivity will look bad in the next meeting of department heads) the technician will start working on some "security" matter (so it sounds urgent and important). And this is how elimination of login nodes, cutting internet access to compute nodes, elimination of switches in the offices because they pose security risks (one might be able to connect an additional devices to the network) and implementation of 2FA on pubkey only login servers come in to existence.
Most of the cluster operators are wonderful. But a bad one can make a cluster significantly less useful.
We have these at work and there was a big outcry when IT tried to get rid of them. I use them from time to time to do things like: Keep git and other backups. Convenient place to scp files to/from. Upload files that coworkers can grab. I don't use it, but others used them as permanent IRC endpoints (using screen or tmux presumably).
Notice a cloud VM or container probably doesn't work here. You need something with a permanent presence, and shared between users (with separate Unix accounts).
Using IRC for the company IM is true old-school. I've worked at places which do the same, and also use SIP for A/V calls. Not being dependent on Big Tech, or the Internet for that matter, for services which exist entirely inside the corporate LAN is a great way to work.
It was replaced by Slack last year. Sigh ...
We still use IRC in some upstream communities although it has been replaced by Matrix in some (which is also terrible).
We moved off IRC to Slack ages ago. Then they decided Slack was too expensive so we were forced to Teams which is bundled with the inescapable O365 license. We now use IRC again (UnrealIRCd) which runs on a debian VM on someone's workstation in the office.
Clients are all irssi on WSL2 or Macs.
irssi is surprisingly decent, and would even makes you think IRC the protocol was designed around irssi (and TUI), although the protocol is actually much older
Matrix the protocol seems fine, but client software seems to be all web views or (weirdly) imitating the look of web views.
Fractal for GNOME is neither, and it’s decent.
I want a text only client, running in a terminal. I don't care that I won't be able to see the pictures & animations - in fact that's an advantage.
https://weechat.org/ -- supports matrix (as well as IRC).
Yes this is fair. I'm even using Weechat for IRC & Slack, so I just need to get around to setting it up.
The emacs client is nice, ement.el, is nice. And there's probably a weechat plugin somewhere.
It does look and work very much like a web view. I just tried it.
Uh, how?.. It’s a conversation window with a contact list to the left, with GNOME-standard (libadwaita) look and feel. If you want separate conversation windows (like ICQ or Pidgin), then I understand the desire, but it’s pretty much orthogonal to web-view-ness (also very rare these days regardless of protocol).
Maybe Adwaita looks like that, I don't know. But there is a ton of whitespace, lots of clickable UI elements that look just like regular text, that kind of thing.
Hmm, a VPS with Copyparty might work, although security would be in question compared to an internal unix login server..
What's the difference between this and a Ubuntu running on EC2 with various users and SSH?
Mostly annoying network configs and token expirations etc. Not saying it can't be set up well, but in my experience some security guru gets a hardon for making my life miserable.
That depends... If your AWS account is already wired up the internal network then the ec2 is basically just another VM just like the onprem VMware servers or physical machines.
Now if all your AWS accounts are only public facing then yes it can get a bit more complicated .
Perhaps it is worth noting that all super computers I know (like the Dutch Snellius and the Finnish Lumi) are Slurm clusters with login nodes.
Bioinformaticians (among others) in (for example) University Medical Centers won’t get much more bang for the buck than on a well managed Slurm cluster (ie with GPU and Fat nodes etc to distinguish between compute loads). You buy the machines, they are utilized close to 100% over their life time.
Yes, I spend a majority of my professional life on similar systems writing code in vim and running massive jobs via slurm. Required for processing TBs of data on secured environments with seamless command line access. I hate web-based connections or vscode type system. Although open to any improvements, this works best to me. It’s like a world inside one’s head with a text-based interface.
Graphical data exploration and stats with R, python, etc is a beautiful challenge at that scale.
Aside from how slow and user hostile it is compared to a text editor, my biggest complaint about vs code is the load it puts on the login node. You get 40 people each running multiple vs code servers and it brings the poor computer to its knees.
Every job on an HPC cluster should have a memory and CPU limit. Nearly every job should have a time limit as well. I/O throttling is a much trickier problem.
I wound up having a script for users on a jump host that would submit an sbatch job that ran sshd as the user on a random high level port and stored the port in the output. The output was available over NFS so the script parsed the port number and displayed the connection info to the user.
The user could then run a vscode server over ssh within the bounds of CPU/memory/time limits.
That’s a really cool idea!
I know indeed that our sys-admins also don't like it.
> It’s like a world inside one’s head with a text-based interface.
I had a co-worker describe it as a giant Linux playground.
Another as ETL nirvana.
One of the more prominent uses of Slurm to hit the headlines recently is by the data access centres for the LSST data from the Vera Rubin Observatory. Such as the U.S.A. facility in Stanford and the U.K. facility at the University of Edinburgh's Somerville.
* https://developer.lsst.io/usdf/batch.html
* https://epcc.ed.ac.uk/hpc-services/somerville
But they're all over the place, from the James Hutton Institute to London Imperial.
* https://www.cropdiversity.ac.uk
* https://www.imperial.ac.uk/computing/people/csg/guides/hpcom...
In HPC, the general rule of thumb is that if you can keep your machine busy more than 40% of the time, it will be cheaper to run on-prem than cloud.
I had 70% in mind but it certainly sounds reasonable (do you have a source? That would be great for our management). In our university medical hospital we have a very hard time to get rid of "shadow IT" because a single 50K machine can just process so much data (ie Next Generation Sequencing data) and can be amortized over 5 (probably 10!) years.
And then we aren't even talking about the EPD servers that are amortized in 4 years and can easily become a compute node in the cluster for another 6 (only problem are the bookkeepers who just can't live with post-amortized hardware! What a world!!)
I don't have any hard data, my role is somewhat HPC-adjacent, rather than directly in it, so this is mostly what I've heard. One way to look at it is that for most HPC operators, they are not charged any of the following for their gear, it's just provided by the organization as part of the larger pool: Power, Cooling, Real Estate, Networking, Security, Silicon-Valley SRE salaries, 38% Cloud Vendor Profit Margin.
Of course, the organization will pay for some of those eventually, so it's not fully fair to not roll them into the IT costs, but there are also lots of ways that non-profits also don't pay those costs at the same levels that the cloud providers do either due to differences in overall costs, or in providing a lesser level of capability. (As a quick example, cloud providers need extensive physical security for their datacenters. A hospital server needs a locked door, and can leverage the existing hospital security team for free.)
Cloud is great if your need is elastic, or if you have time-sensitive revenue dependent on your calculations. In non-profit research environments, that is often not the case. Users have compute they want done "eventually", but they don't really care if it's done in 1 hour or 4 hours; they have lots of other good work to do while waiting for the compute to run in their background.
Dutch Snellius sounds like an obscure baseball player from the 1940s
Snellius was the Latin name of the Dutch mathematician Willebrord Snel van Royen [0] (" ... best known for Snell's law, named after him, which indicates how light rays are broken when light passes through different materials").
[0] https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/30660184/...
So it seems like a "login server" is simply any server you can log in to?
In this context, it is a Unix machine configured for general purpose use (so no minimal web-server-centric OS, for example) where everybody in the organisation gets a login (probably automatically and tied to whatever SSO they use). There may also be some niceties installed and/or preconfigured - like software used frequently by this and that research group or auto-mounting the user's file share from the central storage cluster.
I started with UNIX in 1993 with Xenix, have used most well known commercial flavours since then.
When I started working, our PC were thin clients for all practical purposes, we had development servers where everyone logged in to work, via telnet or remote X session.
Never ever did I heard about login server term as it is being described here.
I’ve seen this setup in many academic institutions - sometimes also called a jump server too. It handles auth to the rest of the intranet and HPCs, as well as storing the user’s common data like docs, code and dotfiles - shared across all subsequent ssh hosts.
This is why I’m setting up a modern shell host community for like minded individuals. Think of it as a high trust social network niche.
Back in the day, Stallman ran a bunch of public servers without any passwords. High trust indeed. https://opensource.stackexchange.com/questions/2295/why-did-... https://news.ycombinator.com/item?id=2519009
Competing with sdf.org, eh? (-:
In some sense perhaps, although I doubt there will ever be more than a couple of thousand members.
Feel free to help further modernize ours running 20+ years!
https://hashbang.sh
Running a remote VScode backend on those while it is technically using the traditional shared server feels like that is not for the "shared" part but just because it is free and available?
Free and available is always good (especially in a research environment). ;) The author also says this in the P.S.:
"I believe the reason people run IDE backends on our login servers is because they have their code on our fileservers, in their (NFS-mounted) home directories. And in turn I suspect people put the code there partly because they're going to run the code on either or both of our SLURM cluster or the general compute servers."
Wonder why they dont use git. Maybe a lot of that isn't committed but is large data files?
that’s orthogonal, right? they may or may not be using Git regardless of where the files are
I use VSCode's remote/ssh functionality all the time, particularly when I need to develop code on an environment that's more capable than my local machine (or when my internet is weak). Still use Git, no reason why you'd change that when working on a remote machine.
I mean rather than work off a remote machine for the convenience of having files you can deploy to a compute cluster use git (or maybe scp) to work local and then deploy them when needed. For a lower latency editing experience.
Honest question. Do you experience subjectively observable latencty differences when editing local vs over the lan?
I ran irssi for a long time on uni servers.
One time X forwarding Matlab saved me a hefty sum of money (for a student) as I could complete an assignment remotely.
Our admins urged people to nice their processes, but my overthewire password cracking sessions were always killed no matter how nice they were.
In this spirit. I had one of those mad inspirations and put together my ideal social network.
https://www.public.outband.net
Probably one of the stupider things I have thrown together, But I had fun making it.
Open up other machines to the internet and recommend people upload their code to github instead of the school's file server. People who want to read email can use their web browser to load gmail or outlook depending on what the school goes with. For the cron jobs I would want to know what is being scheduled before providing a recommendation on how you can get rid of the login server for it.
It’s a University: students are there to learn. Mulituser UNIX machines are great place to learn; I probably wouldn’t be where I am now if I hadn’t cut my teeth on AIX, with ircii and pine for mail and X windows and a public_html directory making exploration fun and easy. Sure, you can give a student Outlook365 access and Wordpress, but will he learn anything from them?
The servers are for professors and graduate students, so this IT guy wants to block PhD students from doing AI research, demanding that they explain their cron jobs to him, because he thinks he can optimize them and save a few bucks.
He will certainly learn what an RCE is.
And when he does, the blast radius is a single unix box/VM or even a single account. You don't put institutionally-critical stuff on things like this. If it gets hacked, you shut it down, have everyone who logged in during the window change their password, and restore from backups or rebuild from scratch.
Going around a CS research department and asking the population there to justify their cron jobs and other stuff they run sounds like an excellent use of a (rare-ish) university admin / IT staff's time... /s
And deal with the ire of Professor Foo (and grad students Bar and Baz) who want/need to use obscure software XYZ and have done so for years (or decades) without any fuss. And build some interface between your HPC clusters and Github. And keeping with regulations and agreements on privacy and security. And so on and so forth. All that for the low, low cost of... Well, certainly not less than keeping and maintaining like one to three unix machines that don't need no fancy hardware or other special attention in the data center you are maintaining anyway?! Why?
edit: By the way, from their documentation, the department mentioned by the author runs their own E-Mail-Servers (as many universities do - fortunately, in this world, there often still is a bit more choice than 'use Gmail/Outlook in the browser').
One of the things about Research Universities in particular is that there's always weird custom stuff everywhere. Thus "But why isn't it standard?" is a very stupid question. Doing the same things over and over but expecting different results isn't research it's insanity. So they're doing non-standard things because that's the whole point. There are systems that are "just" ordinary corporate stuff. Finance are not researching whether to pay suppliers, HR are not researching whether work contracts exist. But a lot of the organisation at any time is engaged in research, otherwise you're just a teaching university and to some extent that can actually be standardized and is the worse for it.
Because this is something to do and solving real problems is way more annoying.
If you work on a smaller cluster in a research institution where in silico work represents only a small portion of the research output the management of the cluster will sometimes be subcontracted out to the general IT support shop. There an administrator - usually with not nearly enough experience will start receiving support requests from users with decades of unix experience which take hours of research to solve. Unable or unwilling (and because inactivity will look bad in the next meeting of department heads) the technician will start working on some "security" matter (so it sounds urgent and important). And this is how elimination of login nodes, cutting internet access to compute nodes, elimination of switches in the offices because they pose security risks (one might be able to connect an additional devices to the network) and implementation of 2FA on pubkey only login servers come in to existence.
Most of the cluster operators are wonderful. But a bad one can make a cluster significantly less useful.
>Why
The author proposed the thought experiment. Ask him, not me.