You can assume the API price is roughly proportional to electricity usage.
If you buy $10 in tokens, that probably folds into ~$3 to $5 dollars in electricity.
Which would be around 30 to 90 kWhr in electricity.
Depending on the source, it could be anywhere from ~500g/kWhr (for natural gas) and ~24g/kWhr for hydroelectric.
It's a really wide spread, but I'd say for $10 in tokens, you'd probably be in the neighbourhood of 1 kg to 40 kg of emissions.
What's a good thing is that a lot of the spread comes from the electricity source. So if we can get all of these datacenters on clean energy sources it could change emissions by over an order of magnitude compared to gas turbines (like XAi uses).
if you bought a nvidia h100 at wholesale prices (around $25k) and ran it 24/7 at commercial electric rates (lets say $0.1 per kwh), then it would take you over 40 years to spend the purchase price of the gpu in electricity. Maybe bump it down to 20 for data center cooling.
I don't think the cost of the ai is close to converging to the price of power yet. Right now its mostly the price of hardware and data center space minus subsidies.
Spending drinking water for toilet flushes is indeed a problem. Perhaps not CO2 measurements directly, but informing people in general of how much high quality water is wasted on flushes alone will hopefully bring more momentum into more efficient flushing mechanism and introducing grey water systems to new and old buildings alike. Good idea!
people downvote your sarcasm, but if you do the calculations you're kinda right.
1Kg of Beef costs:
- The energy equivalent of 60.000 ChatGPT queries.
- The water equivalent of 50.000.000 ChatGPT queries.
Applied to their metric Mistral Large 2 used:
- The water equivalent of 18.8 Tons of Beef.
- The CO2 equivalent of 204 Tons of Beef.
France produces 3836 Tons of Beef per day,
and one large LLM per 6 months.
So yeah, maybe use ChatGPT to ask for vegan recipes.
People will try to blame everything else they can get a hold on before changing the stuff that really has an impact, if it means touching their lifestyle.
Wow, thanks. I’m even coming up with 500K chatGPT queries for the amount of energy consumed as a KG of beef, though I might have moved a decimal place somewhere - feel free to check my math :)
“average query uses about 0.34 watt-hours of energy” - or 0.00034MWH
Using this calculator: https://www.epa.gov/energy/greenhouse-gas-equivalencies-calc... - in my zip, 0.0002KG of CO2 per MWH. (Though, I suppose it depends more on the zip where they’re doing inference, however this translation didn’t seem to vary much when I tried other zips)
Then, 99.48KG/0.0002KG= 497,400 chatGPT queries worth of CO2 per KG of beef?
This is spot on because there can’t be two issues that exist simultaneously. There can only be one thing that wastes enormous amounts of energy and that thing is beef
You can try to misconstrue and ridicule the argument,
but that won't change the math that if you have one thing that causes 1 unit of damage, and another thing that causes 100.000 units of damage, then for all intents and purposes the thing that produces 1 unit of damage is irrelevant.
And any discussion that tries to frame them as somewhat equally important issues is dishonest and either malicious or delusional.
My guess, as I've expressed earlier in the comment chain, is that it's emotionally easier for people to bike-shed about the 0.01% of their environmental impact, than to actually tackle things that make up 20%.
And no it's not only beef (which is a stand-in for meat and diary), another low hanging fruit is also transport, like switching your car for a bike.
But switching from meat and diary to a vegan diet would reduce up to 20% of your personal environmental impact, in terms of CO2.
And about 80-90% of rainforest deforestation is driven directly or indirectly by livestock production.
So it's simply the easiest most impactful thing everyone can do. (Switching your car for a bike isn't possible for people in rural areas for example.)
>1 unit of damage, and another thing that causes 100.000 units of damage, then for all intents and purposes the thing that produces 1 unit of damage is irrelevant
You make a good point. A problem is only a real problem if you can’t find a bigger thing that makes it look small by comparison. For example, the worldwide concrete industry creates more co2 than beef does so there is no reason to stop eating beef if you enjoy it.
Now I know that some might say that “all of this is cumulative” or “the material problems that stem from entrenched industries is actually a reason not to invent completely novel wasteful things rather than a justification for them” but in reality only two things are true: only the biggest problem is real, and the only problem is definitely some other guy’s doing. If I waste x energy and my neighbor wastes y amount, a goal of reducing (x+y) is oppressive whereas a goal where I just need to try to keep x lower than y feels a lot nicer.
They just suggested a different bike shed — one for the purpose of their argument won’t ever get fixed. J-pb’s point is that running a bunch of generators 24/7 in Memphis is fine because people eat meat. Inefficient LLMs in the real world are okay because people could theoretically become vegan but have not. It’s just a thought experiment
If something costs too much, and you find a way to completely pay for it, that's not bikeshedding.
And it's not a thought experiment. It's a very real suggestion. If you're worried about the resource cost from your personal use, doing something to 100% offset it lets you stop worrying.
> become vegan
For one day per year. Replacing a day you would have otherwise eaten meat. That is an extremely attainable action for anyone that cares enough about LLM resource use enough to strongly consider avoiding them. It's not something that "will not change".
By the way, your goal of running efficiently on consumer hardware isn't as great as it sounds. One of the best ways to improve efficiency is batching multiple requests, and datacenter hardware generally uses more efficient nodes and runs at more efficient clock speeds. There's an efficiency sweet spot where models are moderately too big to run at home.
And it really undermines your argument when you throw in this stupid strawman about elon's toxic generators. You know j-pb was talking about typical datacenter resource use and not that. Get that insulting claim out of here.
It is only a “very real suggestion” if you believe that your argument might be effective.
Do you believe that “skip meat for a day use LLMs for a year” will have a climate impact?
Because if not then you agree with me that in this case theoretical vegans are just being used to justify more real consumption, not less
>stupid strawman about elon's toxic generators
They exist in the real world, right now. It is a real phenomenon and no matter how many vegans I imagine it’s still there. I’m not really clear on why the real thing that’s really happening is a strawman unless you think that the existence of that system is so bad that it undermines your position. Even then it wouldn’t be a strawman though, just a thing that doesn’t support your position that using LLMs is categorically fine because you can picture a vegan in your head
> Do you believe that “skip meat for a day use LLMs for a year” will have a climate impact?
If "use LLMs for a year" is enough to count as having a climate impact (negatively), then yes I believe "skip meat for a day use LLMs for a year" is enough to count (positively).
I'd be tempted to write off both of those, but the whole point of your argument is to consider LLM resource use important, so I'm completely accepting that for the sake of the above argument.
There are no theoretical vegans involved.
And the suggestion doesn't even involve vegans, unless there's a massive contingent of americans that only eat meat one day per year that I wasn't aware of.
And to get at what I think is your core objection: The fact that people can do this isn't being used to let companies off the hook. If only 2% of LLM users set up a meat skipping day, then LLM companies are only 2% let off the hook.
But at the same time let's keep a proportional sense of how big the hook is.
> They exist in the real world, right now. It is a real phenomenon
The strawman is you accusing people of supporting those generators.
> your position that using LLMs is categorically fine
>If "use LLMs for a year" is enough to count as having a climate impact (negatively), then yes I believe "skip meat for a day use LLMs for a year" is enough to count (positively).
Sorry, I should have clarified. In this case I meant “argument” as a thing that leads real people to either understand or agree with your position, not the construction of an idea in your mind.
With that in mind, do you think that “skip meat for a day use LLMs for a year” will convince enough real people, in real life, to not eat meat, that it offsets the emissions from LLM use?
Like imagine the future.
Since LLM use is a new category of energy use, you would have to convince people that haven’t already been convinced to skip meat by animal cruelty, health, philosophy, or existing climate concerns. People that were vegan before LLMs became popular obviously don’t count. The group of people that resisted decades of all that messaging will now make a meaningful adjustment to their consumption to cancel that out — and there will be enough of these new part time/full time vegans that it offsets the entire chat bot industry’s energy usage.
Do you imagine that being what happens?
If not it’s just somebody advocating for increased consumption in real life by invoking imaginary vegans.
As somebody that’s spent years as a vegan I am incredibly wary of “vegans can recruit” as a pitch. I’ve only ever heard that from people that have never tried to recruit in earnest or charlatans. Like I’ve mostly heard that from people that are not, never have been, and have no interest in being vegan.
Edit:
>The strawman is you accusing people of supporting those generators.
That’s not what a strawman is and it’s not an accusation, it’s an observation. If you say “I want subscription based online batched mega-high-compute language models” you are advocating for that industry, and those generators are part of it. Saying you feel that they’re somehow special and different because they’re icky does not make them any different from the thing that you say is necessarily the future. That you want!
I think anyone that does get convinced and skip meat should be able to use LLMs without shame or guilt, while we continue to pressure everyone else to save resources and we continue to pressure LLM companies to save resources.
LLM companies only get let off the hook if a very large fraction of their users do the meat skip thing, which is not very likely but could theoretically happen.
LLMs being a new category of energy use should get them some extra scrutiny, but only some. Maybe 3x scrutiny per wasted kilowatt hour compared to entrenched uses? If our real motivation is resource use, and not overreacting to change, LLMs should get some pressure but most of the pressure should go toward preexisting wasteful uses.
Nobody is advocating to ignore LLMs. But we shouldn't overstate them too much either.
And the giving up meat defense is not a defense for the companies, it's a defense for individual users that actually do it.
Like not an if or maybe thing, what do you see when you picture the future?
Do you think “Skip meat for a day use LLMs for a year” will produce enough new vegans to offset the energy usage and co2 produced by the LLM architecture of your choice?
Not asking if you want it to happen or if it’s something you can imagine could happen, I’m asking if you think it will
[_] yes
[_] no
Because if no, then the idea is just advocating for increased real consumption by invoking imaginary vegans!
Edit:
>LLM companies only get let off the hook if a very large fraction of their users do the meat skip thing, which is not very likely but could theoretically happen.
The person I was initially talking to took the position that LLM companies have negligible impact because people can be vegan. J-bp was saying that LLM companies shouldn’t be on anybody’s radars because uh, meat is 100,000 times worse.
The person you hopped in to defend was saying that LLM companies do not and should not have a “hook” because meat eaters exist
> It was a yes or no question [...] I’m asking if you think it will
[x] no
> Because if no, then the idea is just advocating for increased real consumption by invoking imaginary vegans!
Wrong.
> The person I was initially talking to took the position that LLM companies have negligible impact because people can be vegan.
He said "LLMs are not the problem here", which is true.
And he was arguing for individual use being offset when he said "maybe use ChatGPT to ask for vegan recipes".
The top level comment was also about individual use. "I would really like it if an LLM tool would show me the power consumption and environmental impact of each request I’ve submitted."
The comments right before you replied were also about individual use. "lifestyle choice".
> J-bp was saying that LLM companies shouldn’t be on anybody’s radars because uh, meat is 100,000 times worse.
The 100,000 number was a throwaway hypothetical to make a point. Not a number he was applying to LLMs in particular. Two lines later he threw in a 2,000x too.
And what he said is that LLM companies are not "somewhat equally important". Which is true. He didn't say you should ignore them entirely, just to have a sense of proportion.
-
Edit: Here is an important distinction that I think isn't getting through. There are multiple separate points being made by j-bp:
Point A, about not eating meat for a day, is only excusing anyone that actually does it. It's not a hypothetical that excuses the entire company.
Point B, about the size of the impact, suggests caring less about LLMs based on raw resource use. Point B does not care about the relatively small group of people that take up the offer in Point A. Point B is just looking at the big picture.
Then it is not a “very serious suggestion”. It is a thought experiment which should be taken with commensurate weight.
>Wrong
Explain what “skip a day of meat do a year of LLMs” is then. If it’s not just an ad for feeling good about using LLMs, what is it?
>The 100,000 number was a throwaway hypothetical to make a point
>Two lines later he threw in a 2,000x too.
Alright he said that meat is 2,000 times worse than language models as well as 100,000 times worse than language models. He might have meant 100k but could also mean 2k.
Do you have a real problem in real life where if somebody called you and said “it’s gotten two thousand times worse” versus “it’s gotten a hundred thousand times worse?” the former would be fine and the latter alarming?
If yes, what is the problem? Why was it a problem at 1x? 2000x? 100,000x? Why was it a problem at at 1x and 100,000x but not 2000x?
> Explain what “skip a day of meat do a year of LLMs” is then. If it’s not just an ad for feeling good about using LLMs, what is it?
You can stop being part of the problem if you do it. The problem still exists, but you are no longer part of it. You reduced it by more than your fair share. While the problem would stop existing if everyone made the same choice, there's no pretense that that's actually going to happen. LLM companies are not being excused by such an unlikely hypothetical.
j-lb also made an argument to not care much about LLMs at all, but it was separate from the "skip a day of meat" argument. That's where the big multiplier comes in. But again, separate argument.
I don't want to argue about the example ratio he used. The real ratio is very big if the numbers cited earlier are correct. So if you're going to sit here and say 2000x might as well be arbitrarily large then I think you just joined the "LLM resource use doesn't matter" team, because going by the above citation 2000x is in the ballpark of the correct number, so LLM use is 1 divided by arbitrarily large, making it negligible. Congrats.
Just wanted to chime in and say you represented my case perfectly and got all my points (and their separation) 100%!
You're right, I never said we should not care about LLMs because we also "rightfully don't care about meat".
To me the whole AI resource discussion is just a distraction for people who want to rally against a new scary thing, but not look at the real scary thing that they just gotten used to over the years.
In a sense it's the `banality of evil`, or maybe `banality of self destruction`:
The “banality of evil” is the idea that evil does not have the Satan-like, villainous appearance we might typically associate it with. Rather, evil is perpetuated when immoral principles become normalized over time by people who do not think about things from the standpoint of others.
We've gotten so used to using huge amounts of resources in our day to day lives, that we are completely unwilling to stop and reflect about what we could readily change. Instead we fight against the new and shiny, because it tells a better story, distracting us from what really matters.
In a sense we are procrastinating on changing.
It's not the Skynet like AI that is going to be the doom of humankind, but the hot-dogs, taking your car for the commute, and shitty insulation.
> Whatever you need to tell yourself to keep eating meat buddy.
I’m not the one that brought up moralizing or food. I can’t really comment on your relationship with your diet but it kind of seems like you saw somebody mention power usage and unprompted shared “well I don’t eat meat or cheese or yogurt” so I guess keep that up while you use enough energy to power your home to write some code slower than you would without it?
I agree. Humans have been eating meat and doing construction for the entire history of civilization, they are not the sort of things that could be affected by posting online. LLMs on the other hand are new, largely in the hands of a small handful of companies, and a couple of those companies are bleeding cash in such a way that they might actually respond to consumer pressure. It is cynical to compare them to things that we know will not change as a justification for a blanket excuse for them.
Seeing as these models being wasteful is integral to the revenue of companies like OpenAI and Anthropic, the more people that tell them that the right business strategy is to start perpetually building data centers and power plants, the less incentive they have to build models that run efficiently on consumer hardware.
Where does a Youtube LetsPlay video fall into that calculation? My understanding is that a single watch of a video is orders of magnitude more than a day's active use of ChatGPT.
That would be ... thousands of time less useful than giving you the same information at the motor fuel pump. Unfortunately this isn't one of those situations where every little bit counts. There are 2 or 3 things you can do to reduce your environmental impact and not using chatbots isn't one of the things.
So, using the smallest model for the task would help, as expected.
A very small model could run on device to automatically switch and choose the right model based on the request. It would help navigate the difficult naming of each model of each vendor for sure.
> ” A very small model could run on device to automatically switch and choose the right model based on the request.”
This is harder than it looks. A “router” model often has to be quite large to maintain routing accuracy, especially if you’re trying to understand regular user requests.
Small on-device models gating more powerful models most likely just leads to mis-routes.
What is the levelised cost per token? As in how we calculate levelised cost of energy.
If we take the total training footprint and divide that by the number of tokens the model is expected to produce over its lifetime, how does that compare to the marginal operational footprint?
My napkin math says per token water and material footprints are up 6-600% and 4-400% higher respectively for tokens on the order of 40B to 400M.
I don't have a good baseline on how many tokens Mistral Large 2 will infer over the course of its lifetime, however. Any ideas?
Even if the company is "green" they make money, they pay employees/stockholders, those people use the money to buy more things and go on vacations in airplanes. Worse, they invest the money to make more money and consume more goods.
Even your gains and vegetables are shipped in to feed you, if you walk to the grocery store. You pay rent/mortgage for a house built with concrete and steel. The highest priced items you pay for are also likely the most energy and environmentally costly. They create GDP.
It's a little weird with LLMs right now, because everything is subsidized by VC, Ads, BigCo investment so you can't see real costs. They're probably higher than the $30-200/mo you pay, but they're not 10x the price like your rent, car payment, food, vacation, investment/pension are.
This is a fantastic report. As someone tasked to get the most of AI at our company, in conversations I'm frequently getting questions about it's environmental impact. Great to have a reference.
France has a pretty low-carbon grid (relatively) already, this doesn't fix the water consumption issue though, which is getting more problematic as climate change cause droughts all over the country.
These conclusions are broadly compatible with "The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink" or, as I prefer, the PDF metadata title that they left in there, "Revamped Happy CO2e Paper".
Despite the incredible focus by the press on this topic, Mistral's lifecycle emissions in 18 months were less than the typical annual emissions of a single A320neo in commercial service.
The press focus is a mix of the usual "new thing BAD", and the much more insidious PR work by fossil fuel megacorps.
Fossil fuel companies are damn good at PR, and they know well that they simply can't make themselves look good. The next best thing? Make someone else look worse.
If an Average Joe hears "a company that hurts the environment" and thinks OpenAI and not British Petroleum, that's a PR win.
I suspect the press is also aligned against machine learning because they are still Big Mad® that the internet destroyed their revenue model (charging individuals $50 to advertise used cars, for example).
This is interesting but I'd love it if they'd split training and inference. Training might be highly expensive and conducted once, while inference might be less expensive but conducted many, many times.
I would really like it if an LLM tool would show me the power consumption and environmental impact of each request I’ve submitted.
You can assume the API price is roughly proportional to electricity usage.
If you buy $10 in tokens, that probably folds into ~$3 to $5 dollars in electricity.
Which would be around 30 to 90 kWhr in electricity.
Depending on the source, it could be anywhere from ~500g/kWhr (for natural gas) and ~24g/kWhr for hydroelectric.
It's a really wide spread, but I'd say for $10 in tokens, you'd probably be in the neighbourhood of 1 kg to 40 kg of emissions.
What's a good thing is that a lot of the spread comes from the electricity source. So if we can get all of these datacenters on clean energy sources it could change emissions by over an order of magnitude compared to gas turbines (like XAi uses).
if you bought a nvidia h100 at wholesale prices (around $25k) and ran it 24/7 at commercial electric rates (lets say $0.1 per kwh), then it would take you over 40 years to spend the purchase price of the gpu in electricity. Maybe bump it down to 20 for data center cooling.
I don't think the cost of the ai is close to converging to the price of power yet. Right now its mostly the price of hardware and data center space minus subsidies.
I don’t think that you can make this assumption.
People are selling AI at a loss right now.
And each toilet flush you make should also have a Co2 calculation which should go against your daily carbon allowance.
Spending drinking water for toilet flushes is indeed a problem. Perhaps not CO2 measurements directly, but informing people in general of how much high quality water is wasted on flushes alone will hopefully bring more momentum into more efficient flushing mechanism and introducing grey water systems to new and old buildings alike. Good idea!
people downvote your sarcasm, but if you do the calculations you're kinda right.
1Kg of Beef costs:
Applied to their metric Mistral Large 2 used: France produces 3836 Tons of Beef per day,and one large LLM per 6 months.
So yeah, maybe use ChatGPT to ask for vegan recipes.
People will try to blame everything else they can get a hold on before changing the stuff that really has an impact, if it means touching their lifestyle.
The LLMs are not the problem here.
Those are incredible stats. As a vegan who uses LLMs at work frequently, I would love to have the source as well :)
Wow, thanks. I’m even coming up with 500K chatGPT queries for the amount of energy consumed as a KG of beef, though I might have moved a decimal place somewhere - feel free to check my math :)
“average query uses about 0.34 watt-hours of energy” - or 0.00034MWH
Using this calculator: https://www.epa.gov/energy/greenhouse-gas-equivalencies-calc... - in my zip, 0.0002KG of CO2 per MWH. (Though, I suppose it depends more on the zip where they’re doing inference, however this translation didn’t seem to vary much when I tried other zips)
Then, 99.48KG/0.0002KG= 497,400 chatGPT queries worth of CO2 per KG of beef?
Thanks for sharing!
`0.00034MWH`should probably be kilo not mega, but I think you still did it correctly :D
I think I used Btu (thermal units) in my calculation, so I only calculated actual energy expenditure for my 60.000 queries result.
But you are right, a better metric might be to use CO2-equivalent, because cows emit a lot of methane, whereas chatbots don't.
99.48KG / 0.228kg/kq = 436,315 queries per kg of beef.
Yup checks out!
This is a good article on the subject:
> Using ChatGPT is not bad for the environment
— https://andymasley.substack.com/p/individual-ai-use-is-not-b...
He’s done some good followup articles as well:
https://andymasley.substack.com/s/ai-and-the-environment
The difference is that food is important and live-giving and LLMs are a very fancy magic 8-ball
You don't need beef, beef is a lifestyle choice.
I use LLMs to do all of my coding these days, it's certainly more essential for feeding me than beef.
Also a lifestyle choice
Yes but one with a much much much smaller impact as we just demonstrated.
This is exactly the kind of cognitive dissonance in people that I meant.
You literally see the math and go "but I like my meat, why should I give that up if you got your AI".
Because, as I just demonstrated, my AI takes a infinitesimal fraction of your meat.
It literally takes you only going vegan for a day to offset your entire AI usage of a year.
This is spot on because there can’t be two issues that exist simultaneously. There can only be one thing that wastes enormous amounts of energy and that thing is beef
You can try to misconstrue and ridicule the argument, but that won't change the math that if you have one thing that causes 1 unit of damage, and another thing that causes 100.000 units of damage, then for all intents and purposes the thing that produces 1 unit of damage is irrelevant.
And any discussion that tries to frame them as somewhat equally important issues is dishonest and either malicious or delusional.
My guess, as I've expressed earlier in the comment chain, is that it's emotionally easier for people to bike-shed about the 0.01% of their environmental impact, than to actually tackle things that make up 20%.
And no it's not only beef (which is a stand-in for meat and diary), another low hanging fruit is also transport, like switching your car for a bike.
But switching from meat and diary to a vegan diet would reduce up to 20% of your personal environmental impact, in terms of CO2.
And about 80-90% of rainforest deforestation is driven directly or indirectly by livestock production.
So it's simply the easiest most impactful thing everyone can do. (Switching your car for a bike isn't possible for people in rural areas for example.)
>1 unit of damage, and another thing that causes 100.000 units of damage, then for all intents and purposes the thing that produces 1 unit of damage is irrelevant
You make a good point. A problem is only a real problem if you can’t find a bigger thing that makes it look small by comparison. For example, the worldwide concrete industry creates more co2 than beef does so there is no reason to stop eating beef if you enjoy it.
Now I know that some might say that “all of this is cumulative” or “the material problems that stem from entrenched industries is actually a reason not to invent completely novel wasteful things rather than a justification for them” but in reality only two things are true: only the biggest problem is real, and the only problem is definitely some other guy’s doing. If I waste x energy and my neighbor wastes y amount, a goal of reducing (x+y) is oppressive whereas a goal where I just need to try to keep x lower than y feels a lot nicer.
https://www.theguardian.com/cities/2019/feb/25/concrete-the-...
https://www.chathamhouse.org/sites/default/files/publication...
You're writing a whole lot of words that completely ignore they already gave a solution to things being cumulative.
Skip meat for one day, use AI for a year, come out ahead.
They just suggested a different bike shed — one for the purpose of their argument won’t ever get fixed. J-pb’s point is that running a bunch of generators 24/7 in Memphis is fine because people eat meat. Inefficient LLMs in the real world are okay because people could theoretically become vegan but have not. It’s just a thought experiment
https://futurism.com/elon-musk-memphis-illegal-generators
If something costs too much, and you find a way to completely pay for it, that's not bikeshedding.
And it's not a thought experiment. It's a very real suggestion. If you're worried about the resource cost from your personal use, doing something to 100% offset it lets you stop worrying.
> become vegan
For one day per year. Replacing a day you would have otherwise eaten meat. That is an extremely attainable action for anyone that cares enough about LLM resource use enough to strongly consider avoiding them. It's not something that "will not change".
By the way, your goal of running efficiently on consumer hardware isn't as great as it sounds. One of the best ways to improve efficiency is batching multiple requests, and datacenter hardware generally uses more efficient nodes and runs at more efficient clock speeds. There's an efficiency sweet spot where models are moderately too big to run at home.
And it really undermines your argument when you throw in this stupid strawman about elon's toxic generators. You know j-pb was talking about typical datacenter resource use and not that. Get that insulting claim out of here.
>It's a very real suggestion
It is only a “very real suggestion” if you believe that your argument might be effective.
Do you believe that “skip meat for a day use LLMs for a year” will have a climate impact?
Because if not then you agree with me that in this case theoretical vegans are just being used to justify more real consumption, not less
>stupid strawman about elon's toxic generators
They exist in the real world, right now. It is a real phenomenon and no matter how many vegans I imagine it’s still there. I’m not really clear on why the real thing that’s really happening is a strawman unless you think that the existence of that system is so bad that it undermines your position. Even then it wouldn’t be a strawman though, just a thing that doesn’t support your position that using LLMs is categorically fine because you can picture a vegan in your head
> Do you believe that “skip meat for a day use LLMs for a year” will have a climate impact?
If "use LLMs for a year" is enough to count as having a climate impact (negatively), then yes I believe "skip meat for a day use LLMs for a year" is enough to count (positively).
I'd be tempted to write off both of those, but the whole point of your argument is to consider LLM resource use important, so I'm completely accepting that for the sake of the above argument.
There are no theoretical vegans involved.
And the suggestion doesn't even involve vegans, unless there's a massive contingent of americans that only eat meat one day per year that I wasn't aware of.
And to get at what I think is your core objection: The fact that people can do this isn't being used to let companies off the hook. If only 2% of LLM users set up a meat skipping day, then LLM companies are only 2% let off the hook.
But at the same time let's keep a proportional sense of how big the hook is.
> They exist in the real world, right now. It is a real phenomenon
The strawman is you accusing people of supporting those generators.
> your position that using LLMs is categorically fine
I didn't say that.
>If "use LLMs for a year" is enough to count as having a climate impact (negatively), then yes I believe "skip meat for a day use LLMs for a year" is enough to count (positively).
Sorry, I should have clarified. In this case I meant “argument” as a thing that leads real people to either understand or agree with your position, not the construction of an idea in your mind.
With that in mind, do you think that “skip meat for a day use LLMs for a year” will convince enough real people, in real life, to not eat meat, that it offsets the emissions from LLM use?
Like imagine the future.
Since LLM use is a new category of energy use, you would have to convince people that haven’t already been convinced to skip meat by animal cruelty, health, philosophy, or existing climate concerns. People that were vegan before LLMs became popular obviously don’t count. The group of people that resisted decades of all that messaging will now make a meaningful adjustment to their consumption to cancel that out — and there will be enough of these new part time/full time vegans that it offsets the entire chat bot industry’s energy usage.
Do you imagine that being what happens?
If not it’s just somebody advocating for increased consumption in real life by invoking imaginary vegans.
As somebody that’s spent years as a vegan I am incredibly wary of “vegans can recruit” as a pitch. I’ve only ever heard that from people that have never tried to recruit in earnest or charlatans. Like I’ve mostly heard that from people that are not, never have been, and have no interest in being vegan.
Edit:
>The strawman is you accusing people of supporting those generators.
That’s not what a strawman is and it’s not an accusation, it’s an observation. If you say “I want subscription based online batched mega-high-compute language models” you are advocating for that industry, and those generators are part of it. Saying you feel that they’re somehow special and different because they’re icky does not make them any different from the thing that you say is necessarily the future. That you want!
I think anyone that does get convinced and skip meat should be able to use LLMs without shame or guilt, while we continue to pressure everyone else to save resources and we continue to pressure LLM companies to save resources.
LLM companies only get let off the hook if a very large fraction of their users do the meat skip thing, which is not very likely but could theoretically happen.
LLMs being a new category of energy use should get them some extra scrutiny, but only some. Maybe 3x scrutiny per wasted kilowatt hour compared to entrenched uses? If our real motivation is resource use, and not overreacting to change, LLMs should get some pressure but most of the pressure should go toward preexisting wasteful uses.
Nobody is advocating to ignore LLMs. But we shouldn't overstate them too much either.
And the giving up meat defense is not a defense for the companies, it's a defense for individual users that actually do it.
It was a yes or no question
Like not an if or maybe thing, what do you see when you picture the future?
Do you think “Skip meat for a day use LLMs for a year” will produce enough new vegans to offset the energy usage and co2 produced by the LLM architecture of your choice?
Not asking if you want it to happen or if it’s something you can imagine could happen, I’m asking if you think it will
[_] yes
[_] no
Because if no, then the idea is just advocating for increased real consumption by invoking imaginary vegans!
Edit:
>LLM companies only get let off the hook if a very large fraction of their users do the meat skip thing, which is not very likely but could theoretically happen.
The person I was initially talking to took the position that LLM companies have negligible impact because people can be vegan. J-bp was saying that LLM companies shouldn’t be on anybody’s radars because uh, meat is 100,000 times worse.
The person you hopped in to defend was saying that LLM companies do not and should not have a “hook” because meat eaters exist
> It was a yes or no question [...] I’m asking if you think it will
[x] no
> Because if no, then the idea is just advocating for increased real consumption by invoking imaginary vegans!
Wrong.
> The person I was initially talking to took the position that LLM companies have negligible impact because people can be vegan.
He said "LLMs are not the problem here", which is true.
And he was arguing for individual use being offset when he said "maybe use ChatGPT to ask for vegan recipes".
The top level comment was also about individual use. "I would really like it if an LLM tool would show me the power consumption and environmental impact of each request I’ve submitted."
The comments right before you replied were also about individual use. "lifestyle choice".
> J-bp was saying that LLM companies shouldn’t be on anybody’s radars because uh, meat is 100,000 times worse.
The 100,000 number was a throwaway hypothetical to make a point. Not a number he was applying to LLMs in particular. Two lines later he threw in a 2,000x too.
And what he said is that LLM companies are not "somewhat equally important". Which is true. He didn't say you should ignore them entirely, just to have a sense of proportion.
-
Edit: Here is an important distinction that I think isn't getting through. There are multiple separate points being made by j-bp:
Point A, about not eating meat for a day, is only excusing anyone that actually does it. It's not a hypothetical that excuses the entire company.
Point B, about the size of the impact, suggests caring less about LLMs based on raw resource use. Point B does not care about the relatively small group of people that take up the offer in Point A. Point B is just looking at the big picture.
>[x] no
Then it is not a “very serious suggestion”. It is a thought experiment which should be taken with commensurate weight.
>Wrong
Explain what “skip a day of meat do a year of LLMs” is then. If it’s not just an ad for feeling good about using LLMs, what is it?
>The 100,000 number was a throwaway hypothetical to make a point
>Two lines later he threw in a 2,000x too.
Alright he said that meat is 2,000 times worse than language models as well as 100,000 times worse than language models. He might have meant 100k but could also mean 2k.
Do you have a real problem in real life where if somebody called you and said “it’s gotten two thousand times worse” versus “it’s gotten a hundred thousand times worse?” the former would be fine and the latter alarming?
If yes, what is the problem? Why was it a problem at 1x? 2000x? 100,000x? Why was it a problem at at 1x and 100,000x but not 2000x?
> Explain what “skip a day of meat do a year of LLMs” is then. If it’s not just an ad for feeling good about using LLMs, what is it?
You can stop being part of the problem if you do it. The problem still exists, but you are no longer part of it. You reduced it by more than your fair share. While the problem would stop existing if everyone made the same choice, there's no pretense that that's actually going to happen. LLM companies are not being excused by such an unlikely hypothetical.
j-lb also made an argument to not care much about LLMs at all, but it was separate from the "skip a day of meat" argument. That's where the big multiplier comes in. But again, separate argument.
I don't want to argue about the example ratio he used. The real ratio is very big if the numbers cited earlier are correct. So if you're going to sit here and say 2000x might as well be arbitrarily large then I think you just joined the "LLM resource use doesn't matter" team, because going by the above citation 2000x is in the ballpark of the correct number, so LLM use is 1 divided by arbitrarily large, making it negligible. Congrats.
Just wanted to chime in and say you represented my case perfectly and got all my points (and their separation) 100%!
You're right, I never said we should not care about LLMs because we also "rightfully don't care about meat".
To me the whole AI resource discussion is just a distraction for people who want to rally against a new scary thing, but not look at the real scary thing that they just gotten used to over the years.
In a sense it's the `banality of evil`, or maybe `banality of self destruction`:
We've gotten so used to using huge amounts of resources in our day to day lives, that we are completely unwilling to stop and reflect about what we could readily change. Instead we fight against the new and shiny, because it tells a better story, distracting us from what really matters.In a sense we are procrastinating on changing.
It's not the Skynet like AI that is going to be the doom of humankind, but the hot-dogs, taking your car for the commute, and shitty insulation.
Whatever you need to tell yourself to keep eating meat buddy.
> Whatever you need to tell yourself to keep eating meat buddy.
I’m not the one that brought up moralizing or food. I can’t really comment on your relationship with your diet but it kind of seems like you saw somebody mention power usage and unprompted shared “well I don’t eat meat or cheese or yogurt” so I guess keep that up while you use enough energy to power your home to write some code slower than you would without it?
https://www.infoworld.com/article/4020931/ai-coding-tools-ca...
I gave a little more detail to my point about climate impact here
https://news.ycombinator.com/item?id=44654552
Human energy, attention, and effort are finite. Put the effort where it will have the biggest effect.
I agree. Humans have been eating meat and doing construction for the entire history of civilization, they are not the sort of things that could be affected by posting online. LLMs on the other hand are new, largely in the hands of a small handful of companies, and a couple of those companies are bleeding cash in such a way that they might actually respond to consumer pressure. It is cynical to compare them to things that we know will not change as a justification for a blanket excuse for them.
Seeing as these models being wasteful is integral to the revenue of companies like OpenAI and Anthropic, the more people that tell them that the right business strategy is to start perpetually building data centers and power plants, the less incentive they have to build models that run efficiently on consumer hardware.
Nothing stopping us to live the hermit life in the mountains. But here we are trying to get bits and bytes to write that JIRA ticket instead of us.
You could eat a few bites less of beef in one meal and that would be your equivalent AI use for a lifetime.
Where does a Youtube LetsPlay video fall into that calculation? My understanding is that a single watch of a video is orders of magnitude more than a day's active use of ChatGPT.
I don’t flush my toilet, I Kildwick [0] but J-pd has a more interesting comparaison
[0] https://www.kildwick.com/
[1] https://news.ycombinator.com/threads?id=j-pb
Toilets are already labeled with their usage rate.
This is a good point because being curious about energy usage is the same thing as advocating for an imaginary rule about energy usage
Let’s call it GreenOps
That would be ... thousands of time less useful than giving you the same information at the motor fuel pump. Unfortunately this isn't one of those situations where every little bit counts. There are 2 or 3 things you can do to reduce your environmental impact and not using chatbots isn't one of the things.
So, using the smallest model for the task would help, as expected.
A very small model could run on device to automatically switch and choose the right model based on the request. It would help navigate the difficult naming of each model of each vendor for sure.
> ” A very small model could run on device to automatically switch and choose the right model based on the request.”
This is harder than it looks. A “router” model often has to be quite large to maintain routing accuracy, especially if you’re trying to understand regular user requests.
Small on-device models gating more powerful models most likely just leads to mis-routes.
What is the levelised cost per token? As in how we calculate levelised cost of energy.
If we take the total training footprint and divide that by the number of tokens the model is expected to produce over its lifetime, how does that compare to the marginal operational footprint?
My napkin math says per token water and material footprints are up 6-600% and 4-400% higher respectively for tokens on the order of 40B to 400M.
I don't have a good baseline on how many tokens Mistral Large 2 will infer over the course of its lifetime, however. Any ideas?
Within marginal error, dollars=destruction.
Even if the company is "green" they make money, they pay employees/stockholders, those people use the money to buy more things and go on vacations in airplanes. Worse, they invest the money to make more money and consume more goods.
Even your gains and vegetables are shipped in to feed you, if you walk to the grocery store. You pay rent/mortgage for a house built with concrete and steel. The highest priced items you pay for are also likely the most energy and environmentally costly. They create GDP.
It's a little weird with LLMs right now, because everything is subsidized by VC, Ads, BigCo investment so you can't see real costs. They're probably higher than the $30-200/mo you pay, but they're not 10x the price like your rent, car payment, food, vacation, investment/pension are.
They report that the emissions of 400 output tokens, "one page of text," equates to 10 seconds of online video streaming in the USA.
So I guess one saves a lot of emissions if one stops tiktok-ing, hulu-ing, instagram reel-ing, etc.
This is a fantastic report. As someone tasked to get the most of AI at our company, in conversations I'm frequently getting questions about it's environmental impact. Great to have a reference.
It's sad to see the French of all people fall for guilt-trip austerity thinking. Just decarbonize the grid and move on. Energy is good.
France has a pretty low-carbon grid (relatively) already, this doesn't fix the water consumption issue though, which is getting more problematic as climate change cause droughts all over the country.
These conclusions are broadly compatible with "The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink" or, as I prefer, the PDF metadata title that they left in there, "Revamped Happy CO2e Paper".
Despite the incredible focus by the press on this topic, Mistral's lifecycle emissions in 18 months were less than the typical annual emissions of a single A320neo in commercial service.
https://arxiv.org/pdf/2204.05149
The press focus is a mix of the usual "new thing BAD", and the much more insidious PR work by fossil fuel megacorps.
Fossil fuel companies are damn good at PR, and they know well that they simply can't make themselves look good. The next best thing? Make someone else look worse.
If an Average Joe hears "a company that hurts the environment" and thinks OpenAI and not British Petroleum, that's a PR win.
I suspect the press is also aligned against machine learning because they are still Big Mad® that the internet destroyed their revenue model (charging individuals $50 to advertise used cars, for example).
This is interesting but I'd love it if they'd split training and inference. Training might be highly expensive and conducted once, while inference might be less expensive but conducted many, many times.