1. Problem Background

In risk-control scenarios, structured data (e.g. behavior sequences, social graphs, transaction tables) plays a core role in threat detection. While large language models (LLMs) excel at understanding text and images, they still face limitations in making sense of and reasoning over structured data:

• Difficulty aligning structure with semantics: Graph or temporal structures are hard to map into the language-based semantic space
• Input length and efficiency bottlenecks: Linearizing large-scale structured data easily exceeds the input limits of LLMs
• Weak multi-step reasoning ability: Complex relationships in structured data are difficult for LLMs to handle using chain-of-thought
• Immature fusion of heterogeneous sources: Combining graph, sequence, and tabular data into a unified representation remains an open challenge

Models like GraphGPT, GraphRAG, and StructGPT have begun to explore how to integrate structural information with language, but most still focus on a single data type (either graphs or sequences), leaving multi-source fusion and reasoning mechanisms underdeveloped.

2. Limitations of LLMs for This Problem

• Training data lacks structure-driven tasks: LLMs haven’t been pretrained on graph-specific or sequence-inductive tasks
• Prompt + tool integration is limited: While RAG or tools like StructGPT can assist, they still depend on external structural models
• No native structured reasoning: Chain-of-thought prompting is not well suited for deeply structured contexts; specialized structural reasoning strategies are needed

This suggests exploring a biologically inspired architecture that mimics the brain’s ways of processing structured inputs to enhance LLM capabilities for complex reasoning.

3. How the Human Brain Tackles It: Four Structural Mechanisms

3.1 Brain Rich‑Club Network (Long-range hub organization)

The brain features a network of “rich-club” nodes—highly interconnected hub regions that integrate information across functional zones and support high-level cognition.

3.2 Structured Slots (Sequence Memory + Cognitive Map Integration)

Through prefrontal–hippocampal mechanisms, the brain unifies sequence memory (state-action transitions) and cognitive maps (graph structures). Whittington et al. (2025) propose the structured slots model to explain this integration.

3.3 Episodic Buffer (Working Memory Integration Mechanism)

According to Baddeley’s working memory model, the brain uses an episodic buffer to fuse visual, language, and structural information into a coherent contextual representation.

3.4 Predictive Coding (Error-driven Learning Mechanism)

The brain employs top-down predictions and bottom-up error signals in a loop, iteratively updating its internal model, enabling stable structure perception and semantic fusion.

4. Biologically Inspired Modular Design (Python / PyTorch Implementation)

System Architecture Overview

Graph Module (rich‑club graph)
↓
Slots Module (structured slots)
↓
Episodic Buffer (structure + semantic fusion)
↓
Predictive Coding Layer (prediction + SGD learning)

Each module corresponds to one of the brain-inspired mechanisms and works together to facilitate structured data understanding and semantic integration.

4.1 Graph Module: rich-club structure + GNN implementation

• Build a central dense subgraph (rich-club) plus two sparse subgraphs (each for a subtasks) and interconnect them
• Use NetworkX to generate the graph, and PyTorch Geometric’s GCNConv to extract node embeddings
• Aggregate embeddings of the central nodes to produce a rich-club representation

import networkx as nx
import random
import torch
from torch_geometric.utils import from_networkx
from torch_geometric.nn import GCNConv
...

4.2 Slots Module: Structured Slots Implementation

• Use a set of trainable slots to simulate prefrontal activity slots
• Apply attention to read relevant slot states and write updates to them
• Train the module so it encodes both sequence memory and cognitive map representations

class SlotsModule(nn.Module):
    def __init__(...):
        ...
    def forward(self, key):
        weights = softmax(self.read(slots) @ key)
        slot_read = weighted sum over slots
        new_slot = slot_read + self.write(key)
        return slot_read, new_slot

4.3 Episodic Buffer: Multimodal Structure + Semantic Fusion

• Concatenate the rich-club representation from GraphModule with the slot_read representation
• Use a linear projection layer to produce a fused contextual embedding

class EpisodicBuffer(nn.Module):
    def __init__(...):
        ...
    def forward(self, graph_repr, slot_repr):
        fused = torch.cat([graph_repr, slot_repr], dim=-1)
        return torch.relu(self.combine(fused))

4.4 Predictive Coding Layer: Error-driven learning

• Build a layer predict(state) to predict the next state
• Compute error state - pred as the training loss
• Use SGD to update parameters and simulate predictive coding dynamics

class PredCodeLayer(nn.Module):
    def __init__(...):
        ...
    def forward(self, state):
        pred = self.predict(state)
        err = state.detach() - pred
        loss = err.pow(2).mean()
        return pred, loss, err

5. Integrated Example Pseudocode Structure

# Inputs include: graph data, behavior sequence key, expected next slot state, etc.

h, center_repr = graph_module(data)
slot_read, new_slot, att = slots_module(key)
buffer_state = episodic_buffer(center_repr, slot_read)
pred, loss_pc, err = predcode_layer(buffer_state)
loss_slot = ...
loss = loss_pc + loss_slot
loss.backward()
optimizer.step()

This design is intended to simulate rich-club architecture, structured slots, episodic buffer-based integration, and predictive-coding-based learning, all trained with SGD.

6. Future Outlook

• Starting from LLM limitations in structured data understanding, we draw inspiration from four brain mechanisms
• Each module is mapped to a neurally plausible function and implemented in PyTorch
• The integrated system uses predictive coding + SGD to simulate bio-inspired learning and semantic-structure fusion
• It can be extended to downstream risk-control tasks: graph-based account network reasoning, slot-based behavior sequence storage, buffer-level multimodal fusion, and predictive coding for structure-aware reasoning
• If this paradigm shows promising results on specific tasks, it may further motivate bio-inspired research directions

1. 问题背景

风控场景中，结构化数据（如行为序列、社交图谱、交易表格）是风险检测的核心。当前大语言模型（LLM）对文本和图像理解能力强，但对结构化数据的理解与推理仍存在难点：

• 结构语义对齐困难：图结构或时序结构难以映射到语言语义空间
• 长输入与效率瓶颈：大规模结构数据直接线性化极易超出模型输入限制
• 推理能力弱：缺乏对多步链式推理中复杂结构关系的处理能力
• 异构数据融合不成熟：图 + 序列 + 表格等结构数据的联合理解仍是开放问题

相关模型如 GraphGPT、GraphRAG、StructGPT 都开始探索结构与语言融合的方法，但仍主要聚焦单一数据类型（图或序列），对多源融合与推理机制仍不足。

2. LLM 面对该问题时的局限

• 训练数据不包含结构化任务，LLM 本体未暴露图结构或序列归纳学习任务
• prompt + 工具调用有限：虽然可用 RAG 或 struct‑GPT 结构接口辅助，但仍依赖外部结构模型
• 缺乏逐步推理机制：思维链设计不适合结构复杂场景，需要更多针对结构的推理策略

因此，或许可以尝试引入一种仿脑式结构，模仿人脑处理结构化输入的机制，来尝试从仿生的角度提升模型对复杂结构推理的理解能力。

3. 人脑如何解决：四种结构处理机制

3.1 Brain Rich‑Club 网络（大规模脑结构组织）

大脑中有“rich‑club”节点——即高度互联的枢纽子网，协调不同功能区间的信息整合与高级认知处理。

3.2 Structured Slots（序列记忆与认知地图融合）

通过前额叶-海马体机制，人脑可将序列记忆（状态‑动作转换）与认知地图（graph）统一表示。Whittington 等2025 年提出的structured slots 模型将这两类结构整合解释。

3.3 Episodic Buffer（工作记忆结构融合机制）

Baddeley 的工作记忆模型指出，大脑通过 episodic buffer 将视觉、语言、结构信息融合在一起，构建统一的情境表示。

3.4 Predictive Coding（预测‑误差驱动学习机制）

大脑通过顶层预测、底层误差信号迭代更新内部模型，形成稳定的结构感知与语义融合机制。

4. 仿脑式模块设计思路（Python / PyTorch 实现）

模块总体架构

Graph Module（rich‑club 图）
↓
Slots Module（structured slots）
↓
Episodic Buffer（结构+语义融合）
↓
Predictive Coding Layer（预测编码 + SGD 学习）

每个模块分别对应人脑的四种机制，通过融合实现结构理解与语义集成。

4.1 Graph Module：rich‑club 架构 + GNN 实现

• 构建一个中心 dense 子图（rich-club）+ 两个 sparse 子图（处理不同子任务）+ 跨子图连接
• 使用 NetworkX 构造图，并通过 PyTorch Geometric 的 GCNConv 提取节点 embedding
• 聚合中心节点生成 rich‑club 表征

import networkx as nx
import random
import torch
from torch_geometric.utils import from_networkx
from torch_geometric.nn import GCNConv
...

4.2 Slots Module：Structured Slots 实现

• 使用一个可读写的 slot 集合模拟前额叶活动槽
• 利用 attention 从 slots 中读取相关槽状态，并更新特定槽表示
• 通过训练使 slot 模块能编码序列记忆与认知图状态

class SlotsModule(nn.Module):
    def __init__(...):
        ...
    def forward(self, key):
        weights = softmax(self.read(slots) @ key)
        slot_read = weighted sum of slots
        new_slot = slot_read + self.write(key)
        return slot_read, new_slot

4.3 Episodic Buffer：多模态结构与语义融合

• 将 GraphModule 中 rich-club 表征与 SlotsModule 返回的 slot 读取表示拼接
• 用一个线性投影层构造统一上下文 embedding

class EpisodicBuffer(nn.Module):
    def __init__(...):
        ...
    def forward(self, graph_repr, slot_repr):
        fused = concat(...)
        return activation(combine(fused))

4.4 Predictive Coding Layer：预测-误差驱动学习

• 构建一个预测层 predict(state)，预测下一时刻 state
• 计算误差 state - pred，作为损失进行 SGD 更新
• 结合 PyHGF 或使用原生 SGD 实现预测编码机制的结构化学习

class PredCodeLayer(nn.Module):
    def __init__(...):
        ...
    def forward(self, state):
        pred = self.predict(state)
        err = state.detach() - pred
        loss = err.pow(2).mean()
        return pred, loss, err

5. 模拟整合代码示例结构（伪码）

# 输入包括：graph data、行为序列 key、预期下一个 slot state 等

h, center_repr = graph_module(data)
slot_read, new_slot, att = slots_module(key)
buffer_state = episodic_buffer(center_repr, slot_read)
pred, loss_pc, err = predcode_layer(buffer_state)
loss_slot = (...)
loss = loss_pc + loss_slot
loss.backward()
optimizer.step()

这种设计尝试模拟人脑 rich‑club 架构、structured slots、episodic buffer 多模态融合、predictive coding-based 更新机制，并通过 SGD 更新整体系统。

6. 展望

• 从LLM 的结构理解弱点出发，我们借鉴人脑四大机制（rich‑club、slots、buffer、predictive coding）构建模块化系统；
• 每个模块对应脑机制，并可替换成 PyTorch 代码；
• 整体系统通过预测编码机制加 SGD 更新，模拟生物启发式学习与结构语义融合；
• 可逐步扩展到风控任务：graph 表征推理账号网络、slots 存储行为序列、buffer 融合多源信息、predcode 提高结构推理能力。
• 如果此模式在特定任务上具备一定的效果，或许可以尝试进一步开展仿生方面的研究。

Content

• Basic definition of differential privacy
• Hypothesis interpretation of differential privacy
• Membership inference attack (MIA) and hypothesis testing
• Auditing differential privacy with MIA
• Conclusion

Basic definition of differential privacy

Let’s start with the basic definition of differential privacy. Specifically, we only focus on the widely adopted relaxed \((\epsilon,\delta)-\text{differential privacy}\).

Definition 1: \(M\) is \((\epsilon,\delta)-\text{differential privacy}\) if, for all neighboring data \(D_0\) and \(D_1\) that differ in at most one sample, and for all measurable sets \(\mathcal{O}\),

and symmetrically,

These symmetric conditions highlight that differential privacy mathematically defines the “indistinguishability” of the output of \(M(D_0)\) and \(M(D_1)\) on the measurable sets \(\mathcal{O}\) (output space).

This “indistinguishability”, a binary classification problem, can be naturally described by hypothesis testing.

Hypothesis interpretation of differential privacy

Based on the “indistinguishablity” on the \(M(D_0)\) and \(M(D_1)\), we set up two hypothesis,

Now let’s interpret what benefits we can get from it. We start with two errors in the basic hypothesis testing.

Type I Error (significants level

This occurs when the null hypothesis (\(H_0\)) is rejected even though it is true, i.e., M is predicted did not use \(D_0\) when it actually did. The probability of this error is defined as:

where \(S\) is the rejection region for \(H_0\).

Type I error is also called “significance level.” It represents the maximum probability of making a Type I error that we are willing to tolerate in a test. When we say a test is conducted at the \(\alpha\) significance level, we mean:

That is, if we repeated the same test many times under conditions where \(H_0\) is actually true, we’d expect to falsely reject \(H_0\) about \(\alpha \times 100\%\) of time. Thus, \(\alpha\) control how “strict” our test is in claiming a significant result

Type II Error

This occurs when the null hypothesis (\(H_0\)) is not rejected even though it is false, i.e., M is predicted to use \(D_0\) when it actually did not. The probability of this error is defined as:

where \(\bar{S}\) is the complement of the rejection region \(S\).

Since \(\beta\) is the probability of mistakenly accepting the null hypothesis, \(1-\beta\) represents the correctly accepting the null hypothesis. In other words, if there really is a difference or effect to be found, a test with high power (\(1-\beta\)) is more likely to detect it. Thus, we also term \(1-\beta\) as the “power” of hypothesis testing.

Differential Privacy in Terms of Hypothesis Testing

Here we turn to why we want to use a hypothesis to represent differential privacy. Actually, the hypothesis interpretation of differential privacy condition imposes constraints on the probabilities of false alarms (Type I errors) and missed detections (Type II errors). Let’s see how:

For a mechanism \(M\) to satisfy \((\epsilon, \delta)\)-differential privacy, the following conditions must hold for all neighboring databases \(D_0\) and \(D_1\), and for all rejection regions \(S\):

1. \[\alpha + e^\epsilon \beta \geq 1 - \delta \tag{3}\]
2. \[e^\epsilon \alpha + \beta \geq 1 - \delta \tag{4}\]
3. \[\alpha \leq e^\epsilon(1-\beta) + \delta \tag{5}\]
4. \[1 -\beta \leq e^\epsilon \alpha + \delta \tag{6}\]

Proof:

Assuming \(M\) is \((\epsilon,\delta)-\text{differentially private}\) on neigboring datasets \(D_0\) and \(D_1\), and the complement sets \(\bar{S}\) of rejection region \(\), we have:

and

Combining equations 1, 2, 7, and 8, we can derive equations 3, and 4.

Since \(\bar{S}\) is the complement set of rejection region \(S\), and \(\bar{S}\) is a measurable sets. \(S\) is also a measurable set. Thus equations 7, and 8 are also valid on \(S\):

and

Combining equations 1, 2, 9, and 10, we can derive equations 5, and 6.

Privacy Region

Now we can define a privacy region for \((\epsilon,\delta)-\text{differential privacy}\) as :

We can also define a privacy region for \((\epsilon,\delta)-\text{differentially private}\) mechanism \(M\) on neigboring datasets \(D_0\) and \(D_1\), and any rejection region \(S\subseteq \mathcal{O}\) as:

Conv is the convex hull of a point set.

Membership inference attack (MIA) and hypothesis testing

What are membership inference attacks (MIA)

Membership inference attacks aim to distinguish if a given data sample(s) is in the training set of a target model. It’s a typical binary distinguishing attack. MIA usually classifies a sample as “member” or “non-member” which represents whether the sample is training data or not. Typically, “Member” is a positive prediction, and “non-member” is a negative prediction.

Differential privacy and MIA

Since MIAs are distinguishing attacks, we can also use the hypothesis to perform the attacking target:

Assuming mechanism \(M\) is a machine-learning model, victim sample is \(x\),

Assuming \(x\in D_1\) and \(x\notin D_0\), that is, \(D_0\) and \(D_1\) are neighboring datasets that differ only in \(x\). Then we compare the current hypothesizes in MIA with the hypothesizes in differential privacy (equation DP: Null hypothesis and DP: Alternative hypothesis), and we found that both the null hypothesizes represent the same thing - \(M\) is not trained on \(x\), and so do both alternative hypothesizes.

Type I and Type II errors in MIA

Now we revisit the definitions of Type I and II errors and corresponding equations. We know that for the hypothesis-based membership inference attack:

For simplicity we show the relationship between the confusion matrix and type I and II errors in the following table:

MIA audits implementation of differential privacy

According to this table:

1. \[\alpha + e^\epsilon \beta \geq 1 - \delta \iff \text{FPR}+e^\epsilon \text{FNR} \geq 1 - \delta \iff \frac{1-\text{FPR}-\delta}{1-\text{TPR}}\leq e^\epsilon \tag{13}\]
2. \[e^\epsilon \alpha + \beta \geq 1 - \delta \iff e^\epsilon \text{FPR} + \text{FNR} \geq 1 - \delta \iff \frac{\text{TPR}-\delta}{\text{FPR}} \leq e^\epsilon \tag{14}\]
3. \[\alpha \leq e^\epsilon(1-\beta) + \delta \iff \text{FPR} \leq e^\epsilon(1-\text{FNR}) + \delta \iff \frac{\text{FPR}-\delta}{\text{TPR}} \leq e^\epsilon \tag{15}\]
4. \[1 -\beta \leq e^\epsilon \alpha + \delta \iff 1- \text{FNR} \leq e^\epsilon \text{FPR} +\delta \iff \frac{\text{TPR}-\delta}{\text{FPR}} \leq e^\epsilon \tag{16}\]

We can find that equation 14 and 16 are equivalent. When we draw the privacy region, we can use equations 13, 15, and one from 14 and 16.

With these three inequality, or namely the lower bound of $\epsilon$, we can audit the implementation of differential privacy. But why do we need to do auditing? In the recent couple of years, with the flourishing of machine learning, differential privacy has been incorporated into machine learning models to make a trained model differentially private. However, machine learning models are typically black-box, we don’t really know what happens inside when we train a model, let go of how epsilon changes through the entire training. To debug the potential implementation mistakes when training a differentially private model, we need to use some tool to audit the implementation.

In the next section, I’ll introduce the basic concept of auditing differential privacy.

Auditing differential privacy with MIA

Basically, auditing differential privacy aims to use membership inference attacks to attack the target model and get the TPR, and FPR on each sample. Then we compute the lower bounds in equations 13, 15, and one of 14 and 16. If one of the lower bounds is not valid, then the implementation of differential privacy may have some bugs.

As a simple example, you can try this app.This app plots the privacy region of differential privacy on Type I & Type II error rates (FPR and 1-TPR). You can try to understand the differential privacy by adjusting the \(\epsilon\) and \(\delta\) and inspecting the area of privacy region. Typically, smaller \(\epsilon\) will make a smaller privacy region and allow less combination of \((TPR,FPR)\). You can also try different points of \((TPR,FPR)\) and check if they lie in the privacy area. If a green dot shows, it indicates the \(TPR\) and \(FPR\) is valid for all equations 13 - 16. If it’s a red point, the \(TPR\) and \(FPR\) make at least one of equation 13-16 invalid.

Problem in simple auditing

A problem in the simplest method upon is that due to the empirical estimation of TPR, FPR are on a finite number of models, the uncertainty is inevitable if we don’t have statistical results or estimation of TPR and FPR.

Imagine you’re conducting a privacy audit on a differentially private algorithm \(M\). The goal is to empirically estimate the privacy parameters \(\varepsilon\) and \(\delta\) by assessing an adversary’s ability to distinguish whether \(M\) was trained on dataset \(D_0\) or a neighboring dataset \(D_1\). Specifically, you aim to estimate:

Type I Error (\(\alpha\))

Type II Error (\(\beta\))

To accurately bound \(\alpha\) and \(\beta\), ensuring the privacy guarantees (\(\varepsilon\), \(\delta\)) hold with high confidence. This requires not just single estimates of TPR and FPR but statistical bounds that account for uncertainty due to finite sample sizes.

Let’s take a simple example:

You got these two estimated TPR and FPR after running MIA on multiple DP models where \(\epsilon = 2.5\), \(\delta = 0.0001\):

\(\text{Estimated } FPR (\alpha) = 10\%\) \(\text{Estimated } TPR (1-\beta) = 90\%\)

Based on these single estimates, you might conclude that:

well, the implementation of differential privacy has no bugs.

But we all know that the model training has uncertainty and finite hypothesis testing also has uncertainty. How confident can we say that the next estimation will not show the implementation has bugs?

What we need is confidence. For example, with 95\% confidence, the estimated results will stay in the privacy region.

In the next blog, I’ll introduce how to bound TPR and FPR with Clopper-pearson interval. This mathematical tool will give us a range of TPR and FPR with a specific confidence level. Through this bound, we can not only debug the implementation of differential privacy, we can also see how well the differential privacy is implemented.

This document provides a detailed overview of the strategy employed to fine-tune a Vision Transformer (ViT) on the Tiny ImageNet dataset, achieving a validation accuracy of 90.5% within 10 epochs.

Dataset Description

• Dataset: Tiny ImageNet
• Number of Classes: 200
• Image Size: 64x64 resized to 384x384 for ViT

Model Configuration

• Model: ViT-Base with patch size 16 (vit_base_patch16_384)
• Pretrained Weights: Used pretrained weights from ImageNet
• Optimizer: SGD with momentum (0.9)
• Learning Rate: 1e-4
• Weight Decay: 0.01
• Scheduler: Cosine Annealing Learning Rate
• Loss Function: Soft Target Cross-Entropy (for Mixup/CutMix)
• Augmentation: RandAugment, Random Erasing, Mixup, and CutMix

Strategy

Data Preprocessing

1. Image Resizing:
   • Images were resized to 384x384 to match the input dimensions required by the Vision Transformer (ViT) model. This ensures that the patching mechanism of the ViT (16x16 patches in this case) works seamlessly, dividing the images into the correct number of patches for transformer-based processing.
2. Enhanced Data Augmentations:
   • RandAugment:
     • Method: This augmentation policy applies a random combination of transformations such as rotation, brightness adjustment, and flipping, chosen from a predefined pool of operations.
     • Implementation: Integrated using the RandAugment class from torchvision.transforms.
     • Intuition: Augmentations simulate diverse scenarios in the dataset, enhancing model robustness to unseen variations in real-world applications.
   • Random Erasing:
     • Method: Randomly erases parts of an image during training by replacing selected regions with random pixel values.
     • Probability: Set to 0.25, meaning 25% of training images had a random region erased.
     • Intuition: Prevents the model from over-relying on specific regions of an image, encouraging it to learn more generalized features.

Training Enhancements

1. Mixup and CutMix:
   • Mixup:
     • Method: Mixup blends two training examples and their labels, creating a synthetic training sample:
       [ \tilde{x} = \lambda x_i + (1 - \lambda) x_j, \quad \tilde{y} = \lambda y_i + (1 - \lambda) y_j ]
       where ( \lambda ) is sampled from a Beta distribution.
     • Implementation: Integrated using the Mixup utility from the timm library.
     • Intuition: Mixup smoothens decision boundaries and reduces overfitting, as the model cannot rely on “hard” training labels.
   • CutMix:
     • Method: Similar to Mixup, but instead of blending the entire images, rectangular patches of one image replace patches in another. Labels are proportionally adjusted.
     • Implementation: Configured with probabilities for blending and patch placement using timm.data.Mixup.
     • Intuition: Encourages spatially aware feature learning, improving robustness to occlusions or corruptions.
2. Stochastic Depth:
   • Method: During training, randomly drops a subset of transformer blocks in each forward pass, controlled by a drop probability.
   • Implementation: Applied a drop probability of 0.1 to regularize deeper layers using timm.layers.DropPath.
   • Intuition: Mimics an ensemble effect by allowing the model to explore multiple sub-networks, reducing overfitting and improving generalization.
3. AMP (Automatic Mixed Precision):
   • Method: Combines half-precision and full-precision computations dynamically during training.
   • Implementation: Enabled with torch.amp.GradScaler and torch.cuda.amp.autocast.
   • Intuition: Reduces GPU memory usage and accelerates training while maintaining model performance, especially useful for computationally intensive ViT models.

Training Loop

• Epochs: Trained for up to 50 epochs but utilized early stopping after achieving peak validation accuracy (90.5%) at 10 epochs.
• Batch Size: Set to 128, optimized for GPU memory utilization.
• Logging: Metrics, including training and validation loss and accuracy, were logged using TensorBoard. Logging frequency was every 100 batches to balance granularity and performance overhead.

Validation

• Standard Cross-Entropy loss was used during validation for hard-label accuracy computation. Unlike the training phase, which used soft-label losses (Mixup and CutMix), validation focused purely on the model’s ability to classify with confidence in real-world scenarios.

Layer Fine-Tuning Strategy

The experiment tested two configurations for fine-tuning:

1. Fine-Tuning All Layers:
   • In this setting, all layers of the ViT model were unfrozen, allowing gradient updates to modify the pretrained weights.
   • Result: Achieved a validation accuracy of 90.5%, demonstrating the ability of the model to adapt its internal representations to the Tiny ImageNet dataset.
2. Fine-Tuning the Last Fully Connected Layer Only:
   • In this setting, only the final classification head (Fully Connected Layer) was updated, while all transformer layers were frozen.
   • Result: Achieved a validation accuracy of 72.3%, indicating limited capacity to adapt the learned features to the new dataset.

Analysis:

• Why Fine-Tuning All Layers Performed Better:
  • The pretrained ViT model was trained on ImageNet, which shares some similarities with Tiny ImageNet but differs in scale and distribution.
  • Fine-tuning all layers allowed the model to adjust its intermediate representations to the specific features and patterns of the Tiny ImageNet dataset, leading to significantly better performance.
• When to Fine-Tune Specific Layers:
  • Fine-tuning specific layers, such as only the classification head, may suffice for tasks with highly similar datasets (e.g., same domain). However, for diverse datasets, fine-tuning more or all layers is generally necessary.

Key Takeaway: Fine-tuning the entire network maximized the model’s adaptability to Tiny ImageNet, yielding superior performance. However, this comes at a higher computational cost compared to only tuning the last layer.

Results

• Validation Accuracy: 90.5% after 10 epochs
• Training Time: Approximately 30 minutes per epoch on a single GPU
• Best Model Saved: Model checkpoint saved at ./models/best_vit_tiny_imagenet.pth

Key Insights

1. Enhanced Augmentations: The combination of RandAugment, Mixup, and CutMix improved generalization.
2. Cosine Annealing: Helped achieve smooth convergence with the learning rate.
3. Pretrained Weights: Accelerated convergence and boosted performance significantly.

Repository Setup: The code for this implementation, including the preprocessing and training pipeline, is structured for easy reproducibility. Ensure you have the following dependencies installed:

• PyTorch
• torchvision
• timm
• tqdm

Code

```python

import os
import shutil
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from timm import create_model
from torch.optim.lr_scheduler import CosineAnnealingLR
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from tqdm import tqdm  # For progress bar
os.environ['HF_HOME'] = '/tmp/ygu2/hf_cache_custom'
os.environ['HUGGINGFACE_HUB_CACHE'] = '/tmp/ygu2/hf_cache_custom/hub'

# Import Mixup and CutMix utilities from timm
from timm.data import Mixup
from timm.loss import SoftTargetCrossEntropy

# Optional: Import RandAugment for enhanced data augmentation
from torchvision.transforms import RandAugment

# Set CuDNN Benchmark for optimized performance
torch.backends.cudnn.benchmark = True

# Paths and Constants
data_dir = "./datasets/tiny-imagenet-200"
num_classes = 200
batch_size = 128  # Adjust based on GPU memory
num_epochs = 50  # Increased number of epochs for better convergence
learning_rate = 1e-4  # Lowered learning rate for fine-tuning
weight_decay = 0.01  # Adjusted weight decay
image_size = 384
log_interval = 100  # Log metrics every 100 batches

# Reorganize validation data
val_dir = os.path.join(data_dir, 'val')
val_images_dir = os.path.join(val_dir, 'images')
val_annotations_file = os.path.join(val_dir, 'val_annotations.txt')

# Create a mapping from image filenames to their labels
val_img_dict = {}
with open(val_annotations_file, 'r') as f:
    for line in f.readlines():
        words = line.strip().split('\t')
        val_img_dict[words[0]] = words[1]

# Create directories for each class if they don't exist
for label in set(val_img_dict.values()):
    label_dir = os.path.join(val_images_dir, label)
    if not os.path.exists(label_dir):
        os.makedirs(label_dir)

# Move images into the corresponding label directories
for img_filename, label in val_img_dict.items():
    src = os.path.join(val_images_dir, img_filename)
    dst = os.path.join(val_images_dir, label, img_filename)
    if os.path.exists(src):
        shutil.move(src, dst)

# Data Augmentation and Transformations
transform_train = transforms.Compose([
    transforms.Resize((image_size, image_size), interpolation=transforms.InterpolationMode.BICUBIC),
    transforms.RandomHorizontalFlip(),
    RandAugment(),  # Enhanced augmentation
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
    transforms.RandomErasing(p=0.25),
])

transform_test = transforms.Compose([
    transforms.Resize((image_size, image_size), interpolation=transforms.InterpolationMode.BICUBIC),
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])

# Load Datasets
train_dataset = datasets.ImageFolder(os.path.join(data_dir, 'train'), transform=transform_train)
train_loader = DataLoader(
    train_dataset,
    batch_size=batch_size,
    shuffle=True,
    num_workers=2,  # Reduced from 8 to 2
    pin_memory=True,
    prefetch_factor=4,
    persistent_workers=True
)

val_dataset = datasets.ImageFolder(val_images_dir, transform=transform_test)
val_loader = DataLoader(
    val_dataset,
    batch_size=batch_size,
    shuffle=False,
    num_workers=2,  # Reduced from 8 to 2
    pin_memory=True,
    prefetch_factor=4,
    persistent_workers=True
)

# Create Vision Transformer (ViT) Model
model = create_model('vit_base_patch16_384', pretrained=True, num_classes=num_classes)

# Apply Stochastic Depth
from timm.layers import DropPath  # Updated import path

def apply_stochastic_depth(model, drop_prob):
    for module in model.modules():
        if isinstance(module, DropPath):
            module.drop_prob = drop_prob

apply_stochastic_depth(model, drop_prob=0.1)

# Unfreeze the entire model for fine-tuning
for param in model.parameters():
    param.requires_grad = True

# Use DataParallel for multiple GPUs if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

if torch.cuda.device_count() > 1:
    print(f"Using {torch.cuda.device_count()} GPUs")
    model = nn.DataParallel(model)  # This will use all available GPUs

model = model.to(device)

# Mixup and CutMix
mixup_fn = Mixup(
    mixup_alpha=0.8,
    cutmix_alpha=1.0,
    cutmix_minmax=None,
    prob=0.5,  # Reduced probability to allow some original images
    switch_prob=0.5,
    mode='batch',
    label_smoothing=0.1,
    num_classes=num_classes
)

# Loss, Optimizer, and Scheduler
criterion = SoftTargetCrossEntropy()  # For Mixup and CutMix

# Using SGD with momentum for better fine-tuning
optimizer = optim.SGD(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=learning_rate,
    momentum=0.9,
    weight_decay=weight_decay
)

# Scheduler adjusted to steps per epoch
scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs)

# Initialize AMP scaler for mixed precision
scaler = torch.amp.GradScaler(device='cuda')  # Updated instantiation

# Training and Validation Loop
writer = SummaryWriter()  # For TensorBoard logging

def train_one_epoch(epoch):
    model.train()
    running_loss, correct, total = 0.0, 0, 0

    # Progress bar for training loop
    train_loader_tqdm = tqdm(train_loader, desc=f"Epoch {epoch+1}/{num_epochs} [Training]", leave=False)
    for batch_idx, (images, labels) in enumerate(train_loader_tqdm):
        images, labels = images.to(device, non_blocking=True), labels.to(device, non_blocking=True)

        # Apply Mixup/CutMix
        images, labels = mixup_fn(images, labels)

        optimizer.zero_grad()

        with torch.cuda.amp.autocast():
            outputs = model(images)
            loss = criterion(outputs, labels)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        running_loss += loss.item() * images.size(0)
        total += labels.size(0)

        # Since labels are soft, calculate accuracy based on predicted class vs hard labels
        _, predicted = outputs.max(1)
        _, targets = labels.max(1)
        correct += predicted.eq(targets).sum().item()

        # Update progress bar (accuracy in percentage)
        if (batch_idx + 1) % log_interval == 0 or (batch_idx + 1) == len(train_loader):
            current_loss = loss.item()
            current_acc = 100. * correct / total
            train_loader_tqdm.set_postfix(loss=f"{current_loss:.4f}", accuracy=f"{current_acc:.2f}%")

    epoch_loss = running_loss / total
    epoch_acc = 100. * correct / total  # Multiply by 100 to get percentage
    writer.add_scalar('Loss/train', epoch_loss, epoch)
    writer.add_scalar('Accuracy/train', epoch_acc, epoch)
    print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {epoch_loss:.4f}, Acc: {epoch_acc:.2f}%")  # Acc in %

def validate(epoch):
    model.eval()
    val_loss, correct, total = 0.0, 0, 0

    # Progress bar for validation loop
    val_loader_tqdm = tqdm(val_loader, desc=f"Epoch {epoch+1}/{num_epochs} [Validation]", leave=False)

    criterion_val = nn.CrossEntropyLoss()  # Standard loss for validation

    with torch.no_grad():
        for batch_idx, (images, labels) in enumerate(val_loader_tqdm):
            images, labels = images.to(device, non_blocking=True), labels.to(device, non_blocking=True)

            with torch.cuda.amp.autocast():
                outputs = model(images)
                loss = criterion_val(outputs, labels)

            val_loss += loss.item() * images.size(0)
            total += labels.size(0)
            _, predicted = outputs.max(1)
            correct += predicted.eq(labels).sum().item()

            # Update progress bar (accuracy in percentage)
            if (batch_idx + 1) % log_interval == 0 or (batch_idx + 1) == len(val_loader):
                current_loss = loss.item()
                current_acc = 100. * correct / total
                val_loader_tqdm.set_postfix(loss=f"{current_loss:.4f}", accuracy=f"{current_acc:.2f}%")

    epoch_loss = val_loss / total
    epoch_acc = 100. * correct / total  # Multiply by 100 to get percentage
    writer.add_scalar('Loss/val', epoch_loss, epoch)
    writer.add_scalar('Accuracy/val', epoch_acc, epoch)
    print(f"Validation Loss: {epoch_loss:.4f}, Acc: {epoch_acc:.2f}%")  # Acc in %

    return epoch_acc

# Main Training Loop
best_acc = 0
for epoch in range(num_epochs):
    train_one_epoch(epoch)
    val_acc = validate(epoch)

    # Scheduler step
    scheduler.step()

    # Save best model
    if val_acc > best_acc:
        best_acc = val_acc
        os.makedirs('./models', exist_ok=True)
        # If using DataParallel, save the underlying model
        if isinstance(model, nn.DataParallel):
            torch.save(model.module.state_dict(), './models/best_vit_tiny_imagenet.pth')
        else:
            torch.save(model.state_dict(), './models/best_vit_tiny_imagenet.pth')
        print(f"New best model saved with accuracy: {best_acc:.2f}%")

print("Training complete. Best validation accuracy:", best_acc)

writer.close()

以下是机器学习模型常用评估指标的详细说明，包括准确率、精确率、召回率、F1 值、AUC 和 G-mean 的定义、优缺点。

1. 准确率 (Accuracy)

• 定义：准确率是模型预测正确的样本占所有样本的比例。 \(\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}\) 其中，TP 为真正例，TN 为真负例，FP 为假正例，FN 为假负例。

• 优点：简单直观，适用于类别分布均衡的数据集。
• 缺点：对于类别不平衡的数据集效果较差，因为少数类的错误可能被多数类的正确预测掩盖。

2. 精确率 (Precision)

• 定义：精确率表示在预测为正类的样本中，真正为正类的比例。 \(\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}\)

• 优点：在对误报容忍度较低的应用中（如癌症检测或欺诈检测）尤为重要。
• 缺点：当负类样本非常多而正类样本很少时，精确率可能较高，导致对召回率的忽视。

3. 召回率 (Recall)

• 定义：召回率表示所有实际为正类的样本中被正确预测为正类的比例。 \(\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}\)

• 优点：在对漏报敏感的应用（如疾病检测）中效果显著。
• 缺点：当模型倾向于将样本预测为正类时，召回率可能较高，但精确率会下降。

4. F1 值 (F1 Score)

• 定义：F1 值是精确率和召回率的调和平均值，用于综合评估模型的精确度和召回度。 \(\text{F1 Score} = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\)

• 优点：适用于类别不平衡的数据集，因为它权衡了精确率和召回率。
• 缺点：不适用于只关注精确率或召回率的应用场景，因为它无法提供各个指标的详细信息。

5. AUC (Area Under Curve)

• 定义：AUC 表示 ROC 曲线下的面积，ROC 曲线是通过绘制不同阈值下的 TPR（真正率）和 FPR（假正率）得到的曲线。 \(\text{AUC} = \int \text{ROC Curve}\)

• 优点：不受类别分布影响，能够直观衡量模型的分类能力。AUC 越接近 1，模型效果越好。
• 缺点：对于类别不平衡严重的情况，AUC 可能掩盖少数类的错误分类问题。

6. G-Mean (几何平均)

• 定义：G-mean 是模型在正类和负类上的分类效果的几何平均值，通常用于处理类别不平衡的数据。 \(\text{G-Mean} = \sqrt{\text{Recall}_{\text{positive}} \times \text{Recall}_{\text{negative}}}\)

• 优点：能够平衡模型在不同类别上的表现，适用于类别不平衡的情况。
• 缺点：可能在极端不平衡数据上表现不佳，因为少数类的表现过于依赖召回率。

总结

TPR、FPR、TNR、FNR、L2 Error 和 L1 Error

1. TPR (True Positive Rate)

• 定义：TPR（真正例率）是所有实际为正类的样本中被正确预测为正类的比例，等同于召回率。 \(\text{TPR} = \frac{\text{TP}}{\text{TP} + \text{FN}}\)

2. FPR (False Positive Rate)

• 定义：FPR（假正例率）是所有实际为负类的样本中被错误预测为正类的比例。 \(\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}\)

3. TNR (True Negative Rate)

• 定义：TNR（真负例率）是所有实际为负类的样本中被正确预测为负类的比例，也称特异性。 \(\text{TNR} = \frac{\text{TN}}{\text{TN} + \text{FP}}\)

4. FNR (False Negative Rate)

• 定义：FNR（假负例率）是所有实际为正类的样本中被错误预测为负类的比例。 \(\text{FNR} = \frac{\text{FN}}{\text{FN} + \text{TP}}\)

5. L2 Error (Mean Squared Error, MSE)

• 定义：L2误差是预测值与实际值之间差值的平方和的均值。 \(\text{L2 Error} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2\)

6. L1 Error (Mean Absolute Error, MAE)

• 定义：L1误差是预测值与实际值之间差值的绝对值的均值。 \(\text{L1 Error} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{y}_i|\)

为什么在 MIA 中更关注低 FPR 下的 TPR？

在成员推断攻击（Membership Inference Attack, MIA）中，更关注在低 FPR（False Positive Rate）下的 TPR（True Positive Rate）。原因包括：

1. 隐私风险：低 FPR 能降低误判非成员为成员的可能性。
2. 误报影响：高 FPR 会导致大量误报，掩盖攻击模型的真实效果。
3. 攻击强度：在极少误报情况下，高 TPR 表示攻击模型在严格条件下的攻击强度。
4. 防御敏感性：低 FPR 下的 TPR 能更好地评估防御机制的有效性。
5. 实际需求：在金融或医疗领域，低 FPR 的高 TPR 更符合实际隐私需求。

低 FPR 下的 TPR 能帮助揭示模型在严格隐私保护条件下的风险。

Membership Inference Attacks (MIAs) are designed to determine if a specific data record was used in a model’s training set. Introduced by Shokri et al. in 2017[1], MIAs have since been recognized for their relevance to privacy concerns within machine learning. This introduction will cover the fundamental concept of MIAs and their initial application. Following that, we’ll explore the adaptation of MIAs to large-scale Large Language Models (LLMs) and identify the challenges that diminish MIAs’ effectiveness in this context. To conclude, the discussion will shift to prospective developments and the role MIAs may play in enhancing the privacy of LLMs.

Membership inference attacks

The intuition behind MIAs lies in overfitting of the model to the training data. Overfitting is a common challenge in machine learning, where a model learns too much from its training data, capturing excessive detail that hinders its ability to generalize to unseen data. This occurs when a model, after extensive training, performs exceptionally well on the training data (for instance, achieving 100% accuracy) but poorly on a testing dataset, effectively making random guesses. The root of this issue lies in the model memorizing specific details of the training images—such as the exact pixel positions—rather than learning the broader, general features necessary to distinguish between different individuals’ faces. Although overfitting cannot be completely eliminated due to the model’s inherent limitation of learning solely from its training dataset, various strategies can mitigate its effects.

This concept of overfitting also underpins the differential performance of models on training versus non-training samples, evident through their loss function outputs. For example, a model might show lower loss (e.g., smaller cross-entropy) for training samples than for testing samples. This discrepancy is exploited in membership inference attacks, which aim to determine whether a specific sample was part of the model’s training set based on the model’s response to that sample.

In early implementations of membership inference attacks, the approach involves splitting an original dataset into two parts: one for members (data included in the training set) and one for non-members (data excluded from the training set). The “member” data is used to train a target model, resulting in a model trained specifically on that subset. Subsequently, samples from both the member and non-member datasets are input into the trained model, which outputs a probability vector for each sample if the model is of the classification type. These vectors, along with labels indicating “member” or “non-member” status based on the sample’s origin, are combined with the sample’s true classification label to form a new dataset. This new dataset, referred to as the attack dataset, includes each sample’s probability vector, its true class label, and its membership label. An attack model—a binary classification model—is then trained on this attack dataset to discern whether a given sample was part of the model’s training set.

MIA on LLM

Although MIA has been deeply investigated in small-scale networks, in the era of large languge model (LLM), it is not studied on the LLM enough. Most of the works directly adopt the intuition of MIA-classify the members and non-members based on some indicators (The implementations of the attacks can be found at [6]):

1. LOSS [2]: Considers the model’s computed loss over the target sample. \begin{equation}f(x;M) = L(x;M)\end{equation}

2. Reference-based [3]: Attempts to improve the precision of the LOSS attack and reduce the false negative rate by accounting for the intrinsic complexity of the target point \(x\) by calibrating \(L(x;M)\), with respect to another reference model \((M_{ref})\), which is trained on data from the same distribution as \(D\), but not necessarily the same data. \begin{equation}f(x;M) = L(x;M) - L(x;M_{ref})\end{equation}

3. Zlib Entropy [3]: Calibrates the sample’s loss under the target model using the sample’s zlib compression size. \begin{equation}f(x;M) = \frac{L(x;M)}{zlib(x)}\end{equation}

4. Min-k% Prob [4]: Uses the \(k\%\) of tokens with the lowest likelihoods to compute a score instead of averaging over all token probabilities as in loss. \begin{equation}f(x;M) = \frac{1}{|min-k(x)|} \sum_{x_i \in min-k(x)} -\log(p(x_i | x_1, …, x_{i-1}))\end{equation}

5. Neighborhood Attack [5]: Uses an estimate of the curvature of the loss function at a given sample, computed by perturbing the target sequence to create \(n\) ‘neighboring’ points, and comparing the loss of the target \(x\), with its neighbors \(\tilde{x}\). \begin{equation}f(x;M) = L(x;M) - \frac{1}{n} \sum_{i=1}^{n} L(\tilde{x}_i;M)\end{equation}

Why MIA cares

Membership Inference Attacks (MIA) are significant for privacy because they can expose whether an individual’s data was used in training a machine learning model. This might seem technical but has real-world privacy implications, especially in sensitive contexts. For a simple example, consider a machine learning model trained on health records to predict disease outcomes. If an attacker can use MIA to determine that a particular individual’s data was used in the training set, they might indirectly learn something about that individual’s health status, even if the data was supposed to be anonymous.

Imagine a hospital that develops a model to predict diabetes risk based on patient records. If an attacker can apply MIA to this model and discover that John Doe’s data was used in training, they could infer that John might have been at risk of or diagnosed with diabetes, information that’s supposed to be private. This scenario illustrates why MIA is a concern for privacy: it can lead to unintended disclosures of personal information, undermining the anonymity and confidentiality of sensitive data.

Problems in MIA

As research into Membership Inference Attacks (MIA) progresses, several complexities and challenges have emerged. A key question is the interpretation of the probabilities produced by MIA models. For instance, if an MIA model assigns a 60% probability to a sample being a “member” (i.e., part of the training data), the implications for privacy remain unclear. This uncertainty extends to minor modifications of the sample, such as altering a single pixel—does this still warrant a “member” classification, and what does that mean for privacy?

Carlini et al. have highlighted that MIA tends to be more effective at identifying “non-member” records rather than “members.”[7] This observation suggests that the privacy risks associated with membership inference might not be as significant for member samples as previously thought.

Furthermore, the nuances of Large Language Models (LLMs) and their susceptibility to MIAs warrant further exploration. Specific characteristics of LLMs, such as their capacity for data memorization, play a crucial role in how they might compromise the anonymity of training data. However, the full extent of these features’ impact on training data membership remains partially understood.

Given these issues, the future and practical relevance of MIA are subjects of ongoing debate. As we continue to unravel the complexities of MIAs and their effects on privacy, it is crucial to refine our understanding of these attacks and their implications for the security of machine learning models.

Potential of MIA

Recent research highlights the potential of Membership Inference Attacks (MIA) for auditing the privacy of algorithms designed for differential privacy. Differential privacy provides a solid guarantee of privacy, leading to the proposal of differentially private machine learning algorithms, such as DP-SGD and DP-Adam. Users set a theoretical privacy budget, \(\epsilon_{theory}\), and a relaxation parameter, \(\delta\), aiming for the trained model to achieve \((\epsilon_{theory},\delta)\)-differential privacy. However, assessing a model’s actual privacy level has been challenging, as differential privacy offers a worst-case guarantee, suggesting that the practical privacy level might be more stringent (practical \(\epsilon\) is smaller than \(\epsilon_{theory}\)).

Studies now indicate that MIA can estimate the lower bound of the practical \(\epsilon\), denoted as \(\epsilon_{LB}\). This suggests the actual \(\epsilon\) for a model’s privacy lies within the range \([\epsilon_{LB},\epsilon_{theory}]\), providing a clearer picture of its privacy assurances. This insight underscores MIA’s role in evaluating and ensuring the privacy of differentially private machine learning algorithms[8][9][10].

Several differentially private fine-tuning techniques have been developed for Large Language Models (LLMs) to secure their differential privacy [11][12]. Despite these advancements, there remains a lack of research into the auditing of these models’ actual privacy levels. Therefore, a valuable direction for future work could be to employ Membership Inference Attacks (MIAs) to assess and audit the privacy of LLMs that have undergone differential privacy fine-tuning. This approach could provide a clearer understanding of how private the LLMs really are, beyond theoretical guarantees.

References

[1] Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017, May). Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) (pp. 3-18). IEEE.

[2] Yeom, S., Giacomelli, I., Fredrikson, M., & Jha, S. (2018, July). Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) (pp. 268-282). IEEE.

[3] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., … & Raffel, C. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) (pp. 2633-2650).

[4] Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., … & Zettlemoyer, L. (2023). Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789.

[5] Mattern, J., Mireshghallah, F., Jin, Z., Schölkopf, B., Sachan, M., & Berg-Kirkpatrick, T. (2023). Membership inference attacks against language models via neighbourhood comparison. arXiv preprint arXiv:2305.18462.

[6] https://github.com/iamgroot42/mimir

[7] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis and F. Tramèr, “Membership Inference Attacks From First Principles,” 2022 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, 2022, pp. 1897-1914, doi: 10.1109/SP46214.2022.9833649.

[8] Tramer, F., Terzis, A., Steinke, T., Song, S., Jagielski, M., & Carlini, N. (2022). Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202.12219.

[9] Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagielski, M., … & Terzis, A. (2023). Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium (USENIX Security 23) (pp. 1631-1648).

[10] Steinke, T., Nasr, M., & Jagielski, M. (2024). Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36.

[11] Behnia, R., Ebrahimi, M. R., Pacheco, J., & Padmanabhan, B. (2022, November). Ew-tune: A framework for privately fine-tuning large language models with differential privacy. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW) (pp. 560-566). IEEE.

[12] Singh, T., Aditya, H., Madisetti, V. K., & Bahga, A. (2024). Whispered Tuning: Data Privacy Preservation in Fine-Tuning LLMs through Differential Privacy. Journal of Software Engineering and Applications, 17(1), 1-22.

Differential privacy (DP) is a privacy-preserving concept in data analysis and statistics. The goal of differential privacy is to allow the inclusion of an individual’s data in a dataset for analysis while protecting the privacy of that individual.

Example

So, how does DP achieve this? Let’s take a simple example:

Suppose you are part of a company that wants to calculate the average salary of its employees without revealing the salary of any specific employee. The company collects the following data:

Employee A: $60,000 Employee B: $70,000 Employee C: $80,000 Employee D: $90,000

Non-Differentially Private Calculation:

In a traditional average calculation, you might sum all salaries and divide by the number of employees: \begin{equation} \text{Average} = \frac{60,000 + 70,000 + 80,000 + 90,000}{4} = 75,000 \end{equation}

Differentially Private Calculation:

In a differentially private approach, you add random noise to the calculation to protect individual privacy. Let’s say you add random noise between -1,000 and +1,000 to each employee’s salary:

\begin{equation} \text{Average} = \frac{60,000 + 70,000 + 80,000 + 90,000 + \text{noise}}{4} \end{equation}

Here, the noise is a random value chosen from the range -1,000 to +1,000. This ensures that even if an individual’s salary changes slightly, it is challenging to determine which specific employee’s salary contributed to the final result.

In essence, differential privacy aims to introduce uncertainty in the output distribution. Considering two datasets that differ only in one data record, a differentially private algorithm will generate output distributions that are slightly different for these two datasets. In other words, observers cannot discern which dataset the algorithm used from the output, and thus, they cannot determine whether a specific data record is in the dataset or not.