If I am understanding this correctly, this is pretty damn cool. I got 15 minutes of research on it, but no better way to get corrected than be wrong on the internet.
Essentially it seems that they can statistical magic "fuzz" the training set in such a way that it becomes very difficult for the model to leak information from the training set, while still providing the same output whether or not that exact info was in the training set. So I suppose the goal would be something like the ability to train on medical data, while making it so the model won't be able to complete the prompt "Workaccount 2 has a serious medical condition called ______" and would give the same response regardless of whether or not I was present in the database.
prob(training_process(data)(Work account 2 has a serious medical condition called) = anaemia) <= e^epsilon * prob(training_process(data without that piece of information)(Work account 2 has a serious medical condition called) = anaemia)) + delta
Here epsilon = 2, and delta is small. Basically, there is a theoretical guarantee that if it had trained on that sentence, it would be no more than 7x as likely to output that in response to any prompt, compared to when it hadn't trained on that sentence at all. Sentence here is defined to be 1024 tokens long[1].
You might think 7x is not that big of a deal, but note that this is a theoretical guarantee( and with some mathematics it's possible to get an even tighter bound(see: Renyi DP)). In practice, actually getting private data out of a DP-trained model is difficult even for epsilon=8 (corresponds to 2000x likely!).
Edit: [1] this can be problematic, if a piece of information greater than 1024 tokens long gets split into two sentences, then there is no theoretical guarantee across sequences. However this is an implementation detail of this model, I've yet to see the effect of increasing this number to a more reasonable value.
Thanks, that's quite exciting, because personally the thing I'm most excited about AI is the medical and scientific research capabilities. Exciting times!
> VaultGemma is a variant of the Gemma family of lightweight, state-of-the-art open models from Google. It is pre-trained from the ground up using Differential Privacy (DP). This provides strong, mathematically-backed privacy guarantees for its training data, limiting the extent to which the model's outputs can reveal information about any single training example.
> VaultGemma was trained using Tensor Processing Unit (TPU) hardware TPUv6e. Training large language models with the significant computational overhead of differential privacy requires specialized hardware. TPUs are designed to handle the massive computations involved, offering the performance, memory, and scalability necessary to train models like VaultGemma efficiently and sustainably.
Seems like it requires TPUs to run, as DP has a huge performance impact, so we're unlikely to see this in homelabs and similar environments, as far as I understand.
Edit: On second read, the TPUs were only used for training, but no description if anything specific for the hardware is needed, so assuming it's fine with a regular GPU?
Ignoring what this model architecture could do and just considering what this model does do, why would I (or anyone) want to run this model (locally) to do <insert use-case>? Is it entirely a proof-of-concept for future training on medical data? Are they looking to use this to attempt to ethically justify training on (free-tier) user's personal data via the application of noise to the training data?
This could be very good for scaling data while avoiding copyright claims since the copyright argument is a lot weaker (at least to the layman) if no memorization is happening. It even may open the door to Snow Crash like distributed training where people feed the model continuous streams of data of their computer use or even daily lives without worrying about PII leakage
training_algorithm(training data with a row that has "ForHackernews blood test report...") hard to distinguish from training_algorithm(training data without that) upto a factor of epsilon. They have explained further in the article itself with concrete values for epsilon.
It is very difficult for someone to coax the model into regurgitating a sequence from the training data. So as you can imagine, the first usecase is going to be google training on your gmail inbox without me being able to prompt your emails out of it.
User-level DP on the other hand, which the article alludes to near the end, would mean that it's very difficult to make the model regurgitate a particular user's data.
Since this is a theoretical guarantee, you can do whatever prompt engineering you like, it will be really difficult all the same.
How difficult it is depends on a bunch of quantitative factors. Mostly, the value of epsilon.
You might think this would be useful for copyright protection as well, but there is a subtle difference. It's been a while and I'm hazy on the details, so I'll refer you to the Near Access Freeness paper which discusses it in detail and proposes another framework for that.
If I am understanding this correctly, this is pretty damn cool. I got 15 minutes of research on it, but no better way to get corrected than be wrong on the internet.
Essentially it seems that they can statistical magic "fuzz" the training set in such a way that it becomes very difficult for the model to leak information from the training set, while still providing the same output whether or not that exact info was in the training set. So I suppose the goal would be something like the ability to train on medical data, while making it so the model won't be able to complete the prompt "Workaccount 2 has a serious medical condition called ______" and would give the same response regardless of whether or not I was present in the database.
Yes.
prob(training_process(data)(Work account 2 has a serious medical condition called) = anaemia) <= e^epsilon * prob(training_process(data without that piece of information)(Work account 2 has a serious medical condition called) = anaemia)) + delta
Here epsilon = 2, and delta is small. Basically, there is a theoretical guarantee that if it had trained on that sentence, it would be no more than 7x as likely to output that in response to any prompt, compared to when it hadn't trained on that sentence at all. Sentence here is defined to be 1024 tokens long[1].
You might think 7x is not that big of a deal, but note that this is a theoretical guarantee( and with some mathematics it's possible to get an even tighter bound(see: Renyi DP)). In practice, actually getting private data out of a DP-trained model is difficult even for epsilon=8 (corresponds to 2000x likely!).
Edit: [1] this can be problematic, if a piece of information greater than 1024 tokens long gets split into two sentences, then there is no theoretical guarantee across sequences. However this is an implementation detail of this model, I've yet to see the effect of increasing this number to a more reasonable value.
Thanks, that's quite exciting, because personally the thing I'm most excited about AI is the medical and scientific research capabilities. Exciting times!
The actual weights: https://huggingface.co/google/vaultgemma-1b
> VaultGemma is a variant of the Gemma family of lightweight, state-of-the-art open models from Google. It is pre-trained from the ground up using Differential Privacy (DP). This provides strong, mathematically-backed privacy guarantees for its training data, limiting the extent to which the model's outputs can reveal information about any single training example.
> VaultGemma was trained using Tensor Processing Unit (TPU) hardware TPUv6e. Training large language models with the significant computational overhead of differential privacy requires specialized hardware. TPUs are designed to handle the massive computations involved, offering the performance, memory, and scalability necessary to train models like VaultGemma efficiently and sustainably.
Seems like it requires TPUs to run, as DP has a huge performance impact, so we're unlikely to see this in homelabs and similar environments, as far as I understand.
Edit: On second read, the TPUs were only used for training, but no description if anything specific for the hardware is needed, so assuming it's fine with a regular GPU?
So far Gemma models were capable of running on ordinary GPUs or CPUs, and I think it's safe to assume that this trend is continuing here.
Ignoring what this model architecture could do and just considering what this model does do, why would I (or anyone) want to run this model (locally) to do <insert use-case>? Is it entirely a proof-of-concept for future training on medical data? Are they looking to use this to attempt to ethically justify training on (free-tier) user's personal data via the application of noise to the training data?
You can hide that you pirated content for training
You can't hide that. You can't use technical measures to hide from discovery.
I think an entire book is a little too large to mask with this method and still end up learning anything.
U can avoid book publisher lawsuit which Anthropic is dealing with using this approach
It's the last option.
The whole framing of DP is:
Probability that you reveal private info is same whether or not you train on a particular users data.
It is useful in many cases, but google the product company specifically is going to use it for ads.
The purpose is research
This could be very good for scaling data while avoiding copyright claims since the copyright argument is a lot weaker (at least to the layman) if no memorization is happening. It even may open the door to Snow Crash like distributed training where people feed the model continuous streams of data of their computer use or even daily lives without worrying about PII leakage
https://lifearchitect.ai/models-table/
Can someone explain what this actually means? I assume this still runs on Google's cloud so it's not 'private' in any meaningful sense.
Differentially private means that:
training_algorithm(training data with a row that has "ForHackernews blood test report...") hard to distinguish from training_algorithm(training data without that) upto a factor of epsilon. They have explained further in the article itself with concrete values for epsilon.
I got that from the article, but I'm not getting what does it means in practice? What's the use case?
It is very difficult for someone to coax the model into regurgitating a sequence from the training data. So as you can imagine, the first usecase is going to be google training on your gmail inbox without me being able to prompt your emails out of it.
User-level DP on the other hand, which the article alludes to near the end, would mean that it's very difficult to make the model regurgitate a particular user's data.
Since this is a theoretical guarantee, you can do whatever prompt engineering you like, it will be really difficult all the same.
How difficult it is depends on a bunch of quantitative factors. Mostly, the value of epsilon.
You might think this would be useful for copyright protection as well, but there is a subtle difference. It's been a while and I'm hazy on the details, so I'll refer you to the Near Access Freeness paper which discusses it in detail and proposes another framework for that.
It does not run on Google’s cloud. You can download the model and host it yourself, locally or using a provider you trust.
That's actually great. I didn't realize Google had any models that could be self-hosted.
The Gemma models are available for self hosting. I've used these one on the ollama website myself.
https://ollama.com/library/gemma3