> They can’t get a visa. They live in another city. The contract terms don’t work for them.
I have to share my stories. For a position I’m a really good fit for at a company I’m familiar with, I asked a guy to ask a guy, and they said they would only hire in the US because the team is in the US.
For another position, they reached out to me. They can’t hire in Japan but offered relocation.
In both cases, I’m super grateful that people are willing to help, but I could sense that times have changed since 2023. Globalization is no longer that strong; many, many jobs are restricted to certain locations, such as certain cities in the US or India. I don’t know about the EU, though.
This gives me some hope that if I’m willing to leave Japan, I can get a good job.
Yes, but I see the same thing being in Latin America, similar time zone of the US.
Go check the “Who is hiring?” thread. About 90% of jobs at US companies that are remote, are “Remote (US)”.
This is much different from a few years ago.
I don’t know what happened, but my guess is that there are many more American software engineers looking for a job, so there is plenty to choose from within the US (which seems an easier hire in terms of bureaucracy and cultural similarity)
> Globalization is no longer that strong; many, many jobs are restricted to certain locations
It's not entirely clearly what time periods you're referring to, but when I entered the software development industry (Europe), almost 0% were remote, and it was virtually unheard of. Skype was just beginning to become mainstream around this time, and slowly more remote positions appeared, which culminated with COVID, and then slowly disappeared over time.
But today there is still a lot more remote jobs than existed even a decade ago, and a huge amount of remote jobs available today compared to two decades ago. Globalization seems to be going up and down but nowhere near as isolated as it used to be. But again, I think the perspective changes greatly depending on your reference here.
-If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
-I believe some of this is temporary. Just about everyone is now constantly using AI, so I believe we'll soon care more about someone actually being able to explain how they'd go about something and why rather than just "can you produce code that does this"
-The idea of "applications are a numbers game" and spewing hundreds of resumes everywhere is dying. I know extremely highly qualified people whose resumes never get looked at and recruiters who never look at applications. Everyone is making application materials with AI and HR teams review them with AI. I believe we'll enter an era that's much more manual and high touch.
> If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
The post said it's a risk and liability. I assume they mean the applicant may use the AI credits for malicious purposes like hacking or other illegal things and get the company in hot water, their accounts and subscriptions banned or sued or similar.
Why you can just setup an account, or give interviewee a credit to spend on OperRouter.
Or you can have internal infrastructure to host some models and langfuse there to inspect what they were doing actually.
But you could also just pay them cash. There are hiring processes where the take-home assignment is of such a size that the applicant gets paid for it. If you can't figure this out for the sake of maximizing the potential applicant pool then you likely don't think it's that important and critical for the company.
> why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter?
Because 90% of IT departments' eyes just glazed over. Assuming you're interviewing at a company big enough to have one.
> Just about everyone is now constantly using AI,
As far as I can tell, this is only true in California. There are plenty of managers at every company who have to do the dance and pretend everyone uses it, I'll give you that.
I appreciate that I'm out of touch here, since I have been several years at the same job, but I can't really see how the LLM should have any role in the interview process at all - indeed, other places are going to increasing lengths to keep them out! You're not interviewing the LLM, after all.
I suspect LLMs make the "take home coding challenge" even more useless than it already was. If you want to make it part of the candidate interview, then I guess you've got to sit them down with a laptop.
It's their business to sell AI for ecommerce so you can basically ignore this post as this company/startup is just another proxy business and likely irrelevant soon.
You can wonder why the font gets all wonky or why the Help center link returns an error.
But all that won't matter because STARTUP and AI ;)
>, but I can't really see how the LLM should have any role in the interview process at all
Some companies, and especially new startups that are "AI native" or "AI first", view skilled usage of LLMs, Codex, Claude Code, Cursor, orchestrating multiple agents, etc as the new baseline of basic skills.
If some coders prefer "writing all code by hand" or their skill level with LLMs is unsophisticated one-shot vibe prompts such an "Make me an Adobe Photoshop and don't make any mistakes." ... it means they're not going to be a good fit for those AI-native companies.
It seems like the author of this thread prefers candidates that already have productive skills with LLMs, and thus, want to see the applicants demonstrate competency those tools.
> I can't really see how the LLM should have any role in the interview process at all.
Presumably, the job involves the use of LLMs, so they want to check whether they can use LLMs.
Of course, job interviews don't have to be so narrowly similar to job tasks, and they could test for whatever proxy correlates with the qualities they are looking for.
> How you behave, how you relate to others, how you approach problems and the energy you bring to a team matter as much as your technical ability, and often more.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
20€ for a claude code subscription will get you through every interview. I spent way more on o'reily books in the late 90s and early 2000s on programming languages and frameworks. As a tech nerdy kid, we spent every last penny on all things hardware while our peers told us to be crazy to spend X amount of money on a new 15GB harddrive. If you don't have 20€ for a claude subscription as a senior engineer it is not because you are poor, it is because you have different priorities.
Do you think it will be always 20 dollars? I think at some point everybody will ramp up the prices to the point when half of your salary will be tokens. And I am not sure I want to be in IT then.
I am not sure if this is the same as talking about you buying a hardware when you were 14. You did not had a mortgage and pay for insurance and car oil change then. Everything is super costly now. You have family you need to take care of. Your kids money for their hobbies. Paying for AI in order to be able to afford not a job, but job interview... This is very bad.
I even think it will get cheaper (not short-term, but long-term). But that doesn't matter because:
As long as it is 20 dollars, I as a recruiter expect you to have access to one and will create a job interview task, that asumes you have access to one. If it is 2000$/month, I as a recruiter will not asume you have access to one. Easy as that.
Some things are just obviously implied. You clearly would asume a software engineer has access to a computer, don't you? Would be funny if you apply for a job and ask for a computer so you can prepare for the test, as you don't own any...
>From my admittedly anecdotal and narrow point of view, once you clear a baseline of competence, your willingness to do the work takes you much further than knowing a bit more.
From my own, 15+ years of experience focusing entirely on this problem, point of view, I can't emphasise enough how much I agree with this principle. Finding people who are technically capable of doing the job isn't hard. Finding people who can do the job well, that actively make doing the job more rewarding is much more difficult and no technical assessment on earth can screen for that.
>Nobody should burn tokens for a company that hasn’t had a proper conversation with them yet.
PREACH! This same sentiment applies to all tasks that you expect candidates to do during an interview process. If you want them to put in the work, get buy in first. You get that buy in by having conversations with them and helping them understand why they should care about your company and your opportunity in the first place.
I honestly have no notes on this post other than to encourage you to hold on to your current way of approaching the problem because I promise you, you are absolutely on the right track!
What's the risk in giving someone a deepseek api key that is loaded with $5? Five dollars in deepseek tokens is enough for like a week of constant use.
Yeah I don’t see why the idea was thrown out so flimsily. Surely you can set up a sandbox possibly a wholly separate account from the corporate one and give that to the candidates
- I'd hire someone who keeps building even when jobless.
What you build is your CV/Cover, and your learning opportunity. These days, you have to know LLMs inside and out, which means using them heavily to learn.
- The subscription is probably below or around the daily rate, so it pays off quickly.
I'd argue that it's an investment in your education - not only learning to work with LLMs, but also using LLMs to teach you almost anything (even beyond tech).
I don’t understand why giving someone an api key for their limited use while doing an interview or assignment was thrown out. This is an obvious answer and other companies are doing this. For example, C1 talked about doing this on a recent episode of the Cup o’ Go podcast.
I'm surprised companies are doing take homes in an era of LLMs.
I suppose it can save valuable paid time by forcing the candidate to waste theirs instead, but surely you get a lot more signal from actually talking to the candidate and seeing how they think and understand what they're doing.
You can actually talk to the candidate about their solution: why they picked one approach over another, why X wouldn't work, how well they use LLMs (e.g. 10-line comments for a single line of simple code? do they reuse functions or create four different clones?), etc. With LLMs, assignments can be larger/more complex so you can see more.
I'm not saying it's better or worse, but I don't think assignments are inherently bad. It works the same way in education from what I've heard - written assignments may not be very valuable on their own anymore, but you can ask students questions about specific things irl to see if they actually know their stuff.
Very interesting post about hiring. I am no hiring professional, but after reading to the end this stuck out the most: Do it together.
If you expect LLM usage proficiency you can easily check that on the spot, no need to bother the applicant for private token usage.
Why not put a proxy in front of the model, give each candidate a key with a hard budget, and revoke it after? If you require AI for the interview, you should pay for it
I think you know how to do this. You are head of engineering at an AI startup. You know how to create and revoke API keys and how to share them with your interview candidates.
I thought about this but it's also a liability and puts too much strain on me.
1. Whatever limit I put needs to be completely arbitrary. 5$? 10$?
2. I cannot guarantee they will use it correctly. They might use it for the challenge. They might use it to generate harmful content.
3. It requires me to manually create and revoke keys. Candidates might want more credits. They might get leaked.
There are solutions. I just can't think of good ones.
virtual card. some of the startup finance/banking tools can issue one for a fixed dollar amount. candidate uses it to bump their own sub for a month or top up credits.
It’s not all candidates, only candidates who progress to a certain gate in the process.
I think “we offer a $50 (or $20 or whatever) Visa/Amex gift card as part of this interview step” is easier to admin, might skate by finance more easily, and avoids all the tech support angles of “here’s how to use our specific tooling” and concerns of them misusing your key for something nefarious.
In some hiring processes there was already an element of paid work, like they pay a day's worth of salary and ask you to work on a day's worth of assignment. Rare, but exists.
Setup a tcp/http proxy behind auth, ask candidates to use that endpoint, set that proxy to add whatever you need for properly authed LLM inference. You get both instrospection + reuse whatever subscriptions you have. Mention to the candidate that the requests go through your infra though, so you set the right privacy expectations.
I like the idea to consider the token cost for the candidate, good.
But why not keep it "on paper", asking how the candidate would use an LLM or agent and why?
Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly.
Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
I honestly don't get the point of asking the candidate to use a LLM. If you think that they can replace technical skills (which is fair), why testing these skills anyway ?
I have worked with some who believe everything it says, copy paste code without thought no matter how bad a solution it is, spend hours in pathological loops iterating in circles and getting nowhere, and various other ways to generate a lot of code that kind of works so long as you don't go beyond limits and constraints that the software engineer committing it has no understanding of.
I would definitely like some way to know if they can be trusted with an LLM or if someone is going to be hard reviewing everything they submit forever. Don't know quite how to test that but asking them to use an LLM must be part of it.
> Giving candidates a company account for the challenge is off the table: it’s a liability, and I can’t guarantee how it will be used. Reimbursing what they spend doesn’t work either. It’s hard to justify as a business expense, and it quickly drifts into a financial gray area.
If you can't give qualified candidates who request it an OpenRouter key with $50 on it how are you affording to pay salaries? How little do you value the interviewing and CV review time of your current engineers? What's the liability you're worried about incurring?
I agree. OP is oddly risk averse to what i would regard as normal conduct of business.
Don't do an on-site either, they might connect to the wifi in the office and guess the password, we can't guarantee they won't guess it and download bad things.
Don't hire in the first place, there is a liability they turn up to work in the wrong mood, we can't guarantee it.
If you're looking for a job without a stable source of income every expenditure matters. You need to cover essentials -- rent/mortgage, clothing, food, utilities, etc. Those give you a baseline burnthough rate: how long you can survive on just your savings. Any additional expense decreases that and makes it harder to build back up if doing things like saving for a house.
I've been a broke student with loans where I had to decide between gas to go to an interview or eating that day. It's not about 20$, is about how morally ethic it is to ask people to use their own personal money to just have a chance to get hired and how to try and minimise that.
The relevant question is “how incidental/baseline to life is this expense?”
We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews. We don’t pay gas/mileage for on-site interviews for local candidates. People think of those as incidental/de minimus expenses and we ignore them.
I think LLM subs are going to be just as baseline as mobile phone service very soon (far, far faster than mobile phone adoption was in the US or Europe).
> We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews.
Yes but none of them are solely for the purpose of the interview.
> We don’t pay gas/mileage for on-site interviews for local candidates.
Gas is one of those gray areas because technically it's not a required expense. You could live in front of the office and just walk there, you could live in another state and have to spend 100$ on gas to get there. Since it technically is on the candidate, you applied for this job, on site, knowing how far away it was from your home. Therefore the cost is on you.
But LLM's are different. Is a cost on the candidate, that we require them JUST for the interview. Maybe that person would have spent the money elsewhere, or have spent the tokens for another job. That's where I think the main difference lays.
Ok how about this. Employers create an automated interview that requires you to burn through a $20/month subscription for every job application. When you run out of tokens, you can't apply to another job, so now you need $200/month subscriptions during the job hunting phase.
If you think $200/month is still too cheap for a job, then let's say each individual job application requires a $200/month subscription.
> They can’t get a visa. They live in another city. The contract terms don’t work for them.
I have to share my stories. For a position I’m a really good fit for at a company I’m familiar with, I asked a guy to ask a guy, and they said they would only hire in the US because the team is in the US.
For another position, they reached out to me. They can’t hire in Japan but offered relocation.
In both cases, I’m super grateful that people are willing to help, but I could sense that times have changed since 2023. Globalization is no longer that strong; many, many jobs are restricted to certain locations, such as certain cities in the US or India. I don’t know about the EU, though.
This gives me some hope that if I’m willing to leave Japan, I can get a good job.
Even for remote jobs being in a similar time zone is important to avoid slowing down communication.
Yes, but I see the same thing being in Latin America, similar time zone of the US.
Go check the “Who is hiring?” thread. About 90% of jobs at US companies that are remote, are “Remote (US)”.
This is much different from a few years ago.
I don’t know what happened, but my guess is that there are many more American software engineers looking for a job, so there is plenty to choose from within the US (which seems an easier hire in terms of bureaucracy and cultural similarity)
> Globalization is no longer that strong; many, many jobs are restricted to certain locations
It's not entirely clearly what time periods you're referring to, but when I entered the software development industry (Europe), almost 0% were remote, and it was virtually unheard of. Skype was just beginning to become mainstream around this time, and slowly more remote positions appeared, which culminated with COVID, and then slowly disappeared over time.
But today there is still a lot more remote jobs than existed even a decade ago, and a huge amount of remote jobs available today compared to two decades ago. Globalization seems to be going up and down but nowhere near as isolated as it used to be. But again, I think the perspective changes greatly depending on your reference here.
I suppose instead of remote, you instead have globally distributed but in-office, these days.
I think there are a few interesting things here:
-If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
-I believe some of this is temporary. Just about everyone is now constantly using AI, so I believe we'll soon care more about someone actually being able to explain how they'd go about something and why rather than just "can you produce code that does this"
-The idea of "applications are a numbers game" and spewing hundreds of resumes everywhere is dying. I know extremely highly qualified people whose resumes never get looked at and recruiters who never look at applications. Everyone is making application materials with AI and HR teams review them with AI. I believe we'll enter an era that's much more manual and high touch.
> If the AI part is so important, why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter? That way you could even level the playing field by giving everyone the same access to the same models.
The post said it's a risk and liability. I assume they mean the applicant may use the AI credits for malicious purposes like hacking or other illegal things and get the company in hot water, their accounts and subscriptions banned or sued or similar.
Why you can just setup an account, or give interviewee a credit to spend on OperRouter. Or you can have internal infrastructure to host some models and langfuse there to inspect what they were doing actually.
Completely on point
But you could also just pay them cash. There are hiring processes where the take-home assignment is of such a size that the applicant gets paid for it. If you can't figure this out for the sake of maximizing the potential applicant pool then you likely don't think it's that important and critical for the company.
> why doesn't the company use OpenCode/Kilo etc. and fund it via OpenRouter?
Because 90% of IT departments' eyes just glazed over. Assuming you're interviewing at a company big enough to have one.
> Just about everyone is now constantly using AI,
As far as I can tell, this is only true in California. There are plenty of managers at every company who have to do the dance and pretend everyone uses it, I'll give you that.
I appreciate that I'm out of touch here, since I have been several years at the same job, but I can't really see how the LLM should have any role in the interview process at all - indeed, other places are going to increasing lengths to keep them out! You're not interviewing the LLM, after all.
I suspect LLMs make the "take home coding challenge" even more useless than it already was. If you want to make it part of the candidate interview, then I guess you've got to sit them down with a laptop.
It's their business to sell AI for ecommerce so you can basically ignore this post as this company/startup is just another proxy business and likely irrelevant soon.
You can wonder why the font gets all wonky or why the Help center link returns an error.
But all that won't matter because STARTUP and AI ;)
>, but I can't really see how the LLM should have any role in the interview process at all
Some companies, and especially new startups that are "AI native" or "AI first", view skilled usage of LLMs, Codex, Claude Code, Cursor, orchestrating multiple agents, etc as the new baseline of basic skills.
If some coders prefer "writing all code by hand" or their skill level with LLMs is unsophisticated one-shot vibe prompts such an "Make me an Adobe Photoshop and don't make any mistakes." ... it means they're not going to be a good fit for those AI-native companies.
It seems like the author of this thread prefers candidates that already have productive skills with LLMs, and thus, want to see the applicants demonstrate competency those tools.
> I can't really see how the LLM should have any role in the interview process at all.
Presumably, the job involves the use of LLMs, so they want to check whether they can use LLMs.
Of course, job interviews don't have to be so narrowly similar to job tasks, and they could test for whatever proxy correlates with the qualities they are looking for.
> How you behave, how you relate to others, how you approach problems and the energy you bring to a team matter as much as your technical ability, and often more.
As much as I agree with this, I don't think that the whole rest of the post (about how best to examine LLM usage in an interview loop) coheres with this.
You cannot conclusively test for common decency in either interviews or recommendations. People are on their best behavior in interviews, and it's rare for someone to pass along a recommendation that will criticize them (or their behavior). Length of tenure at prior organizations is not proof, since too many will tolerate poor behavior. Accept that this is a risk involved with any hire, have clear, written business conduct policy, and be willing to show the door to people who cannot control their behavior after written warnings.
Asking how people approach problems with LLMs need not be more than a conversation. Architectural interviews need little more than a whiteboard. If you wouldn't hover over the shoulder of an employee while they work, why do you feel the need to do so in an interview?
I get it - bad hires are expensive in more ways than one. But screen-to-offer should not take longer than two weeks in small companies. Interviews help you derisk, but do not ever fool yourself into thinking that an interview process will ever reduce your hiring risk to zero. Getting a handful of dates before a shotgun wedding will always be risky. Accept it and learn to be more comfortable with risk.
> the energy you bring to a team matter
This is all the more true for hiring managers. Bad management will take strong, high-energy, high-ownership performers and handcuff them. Good management can, sometimes, take people with weaker skills, train them, set them up for success, and inspire them into becoming great performers. Someone who is interviewing, who might accept your job offer, is also taking a risk on whether you are the bad kind or the good kind of manager. The only difference is that job seekers often have more appetite for that risk than job offerers.
20€ for a claude code subscription will get you through every interview. I spent way more on o'reily books in the late 90s and early 2000s on programming languages and frameworks. As a tech nerdy kid, we spent every last penny on all things hardware while our peers told us to be crazy to spend X amount of money on a new 15GB harddrive. If you don't have 20€ for a claude subscription as a senior engineer it is not because you are poor, it is because you have different priorities.
Do you think it will be always 20 dollars? I think at some point everybody will ramp up the prices to the point when half of your salary will be tokens. And I am not sure I want to be in IT then.
I am not sure if this is the same as talking about you buying a hardware when you were 14. You did not had a mortgage and pay for insurance and car oil change then. Everything is super costly now. You have family you need to take care of. Your kids money for their hobbies. Paying for AI in order to be able to afford not a job, but job interview... This is very bad.
> Do you think it will be always 20 dollars?
I even think it will get cheaper (not short-term, but long-term). But that doesn't matter because:
As long as it is 20 dollars, I as a recruiter expect you to have access to one and will create a job interview task, that asumes you have access to one. If it is 2000$/month, I as a recruiter will not asume you have access to one. Easy as that.
Some things are just obviously implied. You clearly would asume a software engineer has access to a computer, don't you? Would be funny if you apply for a job and ask for a computer so you can prepare for the test, as you don't own any...
This is a fantastic post.
>From my admittedly anecdotal and narrow point of view, once you clear a baseline of competence, your willingness to do the work takes you much further than knowing a bit more.
From my own, 15+ years of experience focusing entirely on this problem, point of view, I can't emphasise enough how much I agree with this principle. Finding people who are technically capable of doing the job isn't hard. Finding people who can do the job well, that actively make doing the job more rewarding is much more difficult and no technical assessment on earth can screen for that.
>Nobody should burn tokens for a company that hasn’t had a proper conversation with them yet.
PREACH! This same sentiment applies to all tasks that you expect candidates to do during an interview process. If you want them to put in the work, get buy in first. You get that buy in by having conversations with them and helping them understand why they should care about your company and your opportunity in the first place.
I honestly have no notes on this post other than to encourage you to hold on to your current way of approaching the problem because I promise you, you are absolutely on the right track!
What's the risk in giving someone a deepseek api key that is loaded with $5? Five dollars in deepseek tokens is enough for like a week of constant use.
Yeah I don’t see why the idea was thrown out so flimsily. Surely you can set up a sandbox possibly a wholly separate account from the corporate one and give that to the candidates
Very good point. But I'm predicting that it would just be integrated into Hackerrank or some other platform.
Hmm I'm thinking:
- I'd hire someone who keeps building even when jobless.
What you build is your CV/Cover, and your learning opportunity. These days, you have to know LLMs inside and out, which means using them heavily to learn.
- The subscription is probably below or around the daily rate, so it pays off quickly.
I'd argue that it's an investment in your education - not only learning to work with LLMs, but also using LLMs to teach you almost anything (even beyond tech).
I don’t understand why giving someone an api key for their limited use while doing an interview or assignment was thrown out. This is an obvious answer and other companies are doing this. For example, C1 talked about doing this on a recent episode of the Cup o’ Go podcast.
I'm surprised companies are doing take homes in an era of LLMs. I suppose it can save valuable paid time by forcing the candidate to waste theirs instead, but surely you get a lot more signal from actually talking to the candidate and seeing how they think and understand what they're doing.
You can actually talk to the candidate about their solution: why they picked one approach over another, why X wouldn't work, how well they use LLMs (e.g. 10-line comments for a single line of simple code? do they reuse functions or create four different clones?), etc. With LLMs, assignments can be larger/more complex so you can see more.
I'm not saying it's better or worse, but I don't think assignments are inherently bad. It works the same way in education from what I've heard - written assignments may not be very valuable on their own anymore, but you can ask students questions about specific things irl to see if they actually know their stuff.
Very interesting post about hiring. I am no hiring professional, but after reading to the end this stuck out the most: Do it together. If you expect LLM usage proficiency you can easily check that on the spot, no need to bother the applicant for private token usage.
Hmmm what if I do the assigned task without a chatbot? Will they even know…?
Why not put a proxy in front of the model, give each candidate a key with a hard budget, and revoke it after? If you require AI for the interview, you should pay for it
With the added benefit that you can monitor it and review the cost and usage patterns of the candidates vs the result, maybe even get some cool evals.
Is there not a way for candidates to use tokens via the company's own subscription during the test?
interviewed with one company that gave a virtual card with $500 on it for tokens. that seemed super generous and reasonable.
Not an easy way afaik :/ but if someone knows a good way please let me know
I think you know how to do this. You are head of engineering at an AI startup. You know how to create and revoke API keys and how to share them with your interview candidates.
I thought about this but it's also a liability and puts too much strain on me.
1. Whatever limit I put needs to be completely arbitrary. 5$? 10$? 2. I cannot guarantee they will use it correctly. They might use it for the challenge. They might use it to generate harmful content. 3. It requires me to manually create and revoke keys. Candidates might want more credits. They might get leaked.
There are solutions. I just can't think of good ones.
Create your own endpoint with some token limit the candidate would use.
virtual card. some of the startup finance/banking tools can issue one for a fixed dollar amount. candidate uses it to bump their own sub for a month or top up credits.
I think finance will hang me upside down if I even mention that we need to provision virtual cards for every candidate
It’s not all candidates, only candidates who progress to a certain gate in the process.
I think “we offer a $50 (or $20 or whatever) Visa/Amex gift card as part of this interview step” is easier to admin, might skate by finance more easily, and avoids all the tech support angles of “here’s how to use our specific tooling” and concerns of them misusing your key for something nefarious.
In some hiring processes there was already an element of paid work, like they pay a day's worth of salary and ask you to work on a day's worth of assignment. Rare, but exists.
Can imagine. This was a maybe 50 person startup? So probably easier to admin something like this.
Setup a tcp/http proxy behind auth, ask candidates to use that endpoint, set that proxy to add whatever you need for properly authed LLM inference. You get both instrospection + reuse whatever subscriptions you have. Mention to the candidate that the requests go through your infra though, so you set the right privacy expectations.
I like the idea to consider the token cost for the candidate, good. But why not keep it "on paper", asking how the candidate would use an LLM or agent and why? Actually using an agent means asking the candidate to deliver something that you must validate. And you then need to evaluate the candidate by looking at the output of the LLM indirectly. Isn't it better to directly ask the candidate to explain how an LLM could be used (or if/when it should be used) for your use cases?
At the end of the day, it is the same old 'interview coding game' problem, and I suspect it is working even worse now, using LLMs and agents.
I honestly don't get the point of asking the candidate to use a LLM. If you think that they can replace technical skills (which is fair), why testing these skills anyway ?
I have worked with some who believe everything it says, copy paste code without thought no matter how bad a solution it is, spend hours in pathological loops iterating in circles and getting nowhere, and various other ways to generate a lot of code that kind of works so long as you don't go beyond limits and constraints that the software engineer committing it has no understanding of.
I would definitely like some way to know if they can be trusted with an LLM or if someone is going to be hard reviewing everything they submit forever. Don't know quite how to test that but asking them to use an LLM must be part of it.
> Giving candidates a company account for the challenge is off the table: it’s a liability, and I can’t guarantee how it will be used. Reimbursing what they spend doesn’t work either. It’s hard to justify as a business expense, and it quickly drifts into a financial gray area.
If you can't give qualified candidates who request it an OpenRouter key with $50 on it how are you affording to pay salaries? How little do you value the interviewing and CV review time of your current engineers? What's the liability you're worried about incurring?
I agree. OP is oddly risk averse to what i would regard as normal conduct of business.
Don't do an on-site either, they might connect to the wifi in the office and guess the password, we can't guarantee they won't guess it and download bad things.
Don't hire in the first place, there is a liability they turn up to work in the wrong mood, we can't guarantee it.
[flagged]
"Software engineers are poor and can't afford $20/month".
linkedin-drivel.
If you're looking for a job without a stable source of income every expenditure matters. You need to cover essentials -- rent/mortgage, clothing, food, utilities, etc. Those give you a baseline burnthough rate: how long you can survive on just your savings. Any additional expense decreases that and makes it harder to build back up if doing things like saving for a house.
Exactly, I canceled my $20/mo Emby subscription the morning after I lost my job.
I've been a broke student with loans where I had to decide between gas to go to an interview or eating that day. It's not about 20$, is about how morally ethic it is to ask people to use their own personal money to just have a chance to get hired and how to try and minimise that.
The relevant question is “how incidental/baseline to life is this expense?”
We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews. We don’t pay gas/mileage for on-site interviews for local candidates. People think of those as incidental/de minimus expenses and we ignore them.
I think LLM subs are going to be just as baseline as mobile phone service very soon (far, far faster than mobile phone adoption was in the US or Europe).
> We require clothes for on-site interviews. We require phone service for phone screens and electricity and internet service for video interviews.
Yes but none of them are solely for the purpose of the interview.
> We don’t pay gas/mileage for on-site interviews for local candidates.
Gas is one of those gray areas because technically it's not a required expense. You could live in front of the office and just walk there, you could live in another state and have to spend 100$ on gas to get there. Since it technically is on the candidate, you applied for this job, on site, knowing how far away it was from your home. Therefore the cost is on you.
But LLM's are different. Is a cost on the candidate, that we require them JUST for the interview. Maybe that person would have spent the money elsewhere, or have spent the tokens for another job. That's where I think the main difference lays.
My point is that LLMs will soon be seen as not "solely for the purpose of the interview".
I think that's probably already the case for the median software engineer and maybe even for the median software engineering candidate.
Beyond the price itself, it's insane that they make you pay for the tests they give you. What's next ? you pay them during the training period ?
Ok how about this. Employers create an automated interview that requires you to burn through a $20/month subscription for every job application. When you run out of tokens, you can't apply to another job, so now you need $200/month subscriptions during the job hunting phase.
If you think $200/month is still too cheap for a job, then let's say each individual job application requires a $200/month subscription.