This chapter gives a detailed overview of defining and architecting ML defense frameworks that can protect data, ML models, and other necessary artifacts at different stages of ML training and evaluation pipelines. In this chapter, you will learn about different anonymization, encryption, and application-level privacy techniques, as well as hybrid security measures, that serve as the basis of ML model development for both centralized and distributed learning. In addition, you will also discover scenario-based defense techniques that can be applied to safeguard data and models to solve practical industry-grade ML use cases. The primary objective of this chapter is to explain the application of commonly used defense tools, libraries, and metrics available for large-scale ML SaaS platforms.

In this chapter, these topics will be covered in the following sections:

• Threat matrix and defense techniques
• Anonymization and data encryption
• Differential Privacy (DP)
• Hybrid privacy methods and models
• Adversarial risk mitigation frameworks

Further, with the use of Adversarial Robustness Toolbox (ART), pysft, Pyhfel, secml , ml_privacy_meter, tensorflow_privacy, mia, diffprivlib, and foolbox, we will see how to test model robustness against adversarial attacks.

This chapter requires you to have Python 3.8 installed along with the Python packages listed here (with their installation commands), as well as Keras 2.7.0 and TensorFlow 2.7.0:

• pip install adversarial-robustness-toolbox
• pip install syft==0.2.9
• pip install Pyfhel
• pip install secml
• git clone https://github.com/privacytrustlab/ml_privacy_meter
• pip install -r requirements.txt, pip install -e
• pip install diffprivlib
• pip install tensorflow-privacy
• pip install mia
• pip install foolbox

In this section, let’s look at different defense techniques essential for enterprises to proactively manage threats related to adversarial attacks during the following stages:

• Initial research, planning, and system and model design/architecture phase
• ML model training and deployment
• ML model live in production

You will also get learn additional capabilities, expertise, and infrastructure that organizations need to invest in to have a foolproof defense system.

Researching and planning during the system and model design/architecture phase

This phase (Figure 2.1) is related to all actions taken during model design, architectural planning, and conceptualization in which the adversary carries out preliminary investigations, searching to gain knowledge of the victim’s infrastructure, datasets, and models that will enable them to set up their own capabilities for initiating attacks on ML SaaS platforms.

Figure 2.1 – Relevant attack stages during ML model design and development

We see here the large scope of the initial phase, where adversarial actions can be detrimental to our model and architecture conceptualization. Now, let’s discuss the different steps adversaries take when trying to perform an attack.

Reconnaissance

Reconnaissance is one of the early stages where an adversary actively or passively gathers information to use in later adversarial stages to enable resource development, execute initial access, or lead to the execution of continuous reconnaissance attempts. Some of the associated risks and mitigations of this stage are described in the following list. The best way for the victim to mitigate reconnaissance attempts is to minimize the availability of sensitive information to external entities and employ network content, network flow, file creation, and application log monitoring agents to detect and raise alarms if suspicious activity (such as bots or web crawling) is detected from a single IP source.

Let’s now describe how reconnaissance can take place:

• Active scanning: This step involves scanning operations by adversaries to gather information for targeting. Scanning and search operations (on websites/domains or open technical databases) may be carried out on victim infrastructure via network traffic (with network protocols such as ICMP) by probing mechanisms, or by collecting information through external remote services or public-facing applications.
• Gather victim host/identity/organization information: This step involves adversarial activity to gain information related to victims’ administrative data (e.g., name, assigned IP, functionality, IP ranges, domain names, etc.), configuration (e.g., operating system, language, etc.), names of divisions/departments, business operations, and the roles and responsibilities of major employees.
• Phishing: This is an action that is often undertaken by an adversary in order to steal frequently used credentials to target their victims. Organizations should take the following prompt action:
  • Employ antivirus or antimalware along with network intrusion prevention systems (for example, to filter content based on DKIM and SPF, header, referrer, or User-Agent string HTTP/S fields) to automatically remove malicious links and attachments.
  • Using anti-spoofing mechanisms, providing restricted access to websites that have attachments (.pdf, .docx, .exe, .pif, .cpl, and so on), and enabling email authentication can enable protection against phishing activities.
• Search closed sources, open technical databases, websites and domains, and victim-owned websites: These search operations by the adversary can help to retrieve confidential information from reputable private sources (such as databases, repositories, or paid subscriptions to feeds of technical/threat intelligence data). In addition, registrations of domains/certificates; network data/artifacts gathered from traffic and/or scans; business-, department-, and employee-related information from online sites; and social media can all help the attacker gather the information necessary for targeting.

Resource development

Resource development is another early phase of adversarial action, where adversaries engage themselves in creating resources to use in subsequent attack stages. Resources may be created, purchased, or stolen to target victims. Let’s examine this in more detail:

• Public ML artifact retrieval: This is an important action taken by adversaries to retrieve ML artifacts from public sources, cloud storage, public-facing services, and data repositories. These artifacts can reveal information related to the software stacks, libraries, algorithms, hyperparameters, and model architectures used to train, deploy, test, and evaluate ML models. Adversaries can use either the victim’s representative datasets or models to modify and craft the datasets and models and accordingly train proxy ML models tailored to offline attacks, without directly accessing the target model. The best control measures against this that can be adopted by organizations are the following:
  • Enabling multi-level security rules for the full protection of datasets, models, and artifacts by employing built-in multi-factor authentication schemes
  • Using cloud security rules and ACLs to provide restricted access to ML data and artifacts
• Gathering adversarial ML attack implementation information: Open source implementations of ML algorithms and adversarial attack code (such as CleverHans or ART (https://researchain.net/archives/pdf/Technical-Report-On-The-Cleverhans-V2-1-0-Adversarial-Examples-Library-2906240) and Foolbox (https://arxiv.org/pdf/1707.04131.pdf)) can be misused by attackers. As well as facilitating research, these open source tools can be used to carry out attacks against victims’ infrastructures. In this chapter, we give examples to demonstrate how ART and Foolbox can be used.
• Gaining adversarial ML attack implementation expertise and capabilities: After gaining information on open source attack tools, adversaries can deep dive into research papers and use their own ideas to craft their own attack models and start using them.
• Acquiring infrastructure – attack development and staging workspaces: In this phase, adversaries rely on the free compute resources available from major cloud providers (such as AWS, Google Cloud, Google Colaboratory, and Azure) to initiate attacks. The use of multiple workspaces can help them avoid detection.
• Publishing poisoned datasets and triggering poisoned data training: This step involves creating poisoned datasets (by modifying source datasets, data, or its labels) and publishing these to compromise victims’ ML supply chains. The vulnerabilities embedded in these ML models using poisoned data are activated later and cannot easily be detected.

Strategies that can be employed to protect against poison attacks include leveraging De-Pois (De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks : https://arxiv.org/pdf/2105.03592.pdf), an attack-agnostic defense framework used to construct mimic models. This framework uses Generative Adversarial Networks (GANs) to enable the training of data with augmentations and the creation of models that behave similarly, in terms of outcome, to the original model. This model can detect the poisoned samples by evaluating prediction differences between the target model and the mimic model.

In addition to generating defensive awareness of the aforementioned possible intrusions, enterprise-grade defense frameworks should take into consideration some of the following aspects of security bottlenecks and take appropriate remedial measures:

• Establishing accounts: In this phase, external adversaries engage themselves in creating accounts to build a persona across different social media platforms, such as LinkedIn, as well as on GitHub, to impersonate real people. These personas can be used to accumulate public information, set up email accounts, and strengthen public profiles, which will aid in stealing information over the course of time.

The best tactic to protect against such actions is to identify any suspicious activity of individuals who claim to work for the organization or have made connection requests to different organizational accounts.

• Obtaining capabilities: Here, adversaries rely on stealing, purchasing, or freely downloading malware, licensed software, exploits, certificates, and information related to vulnerabilities. Mitigation actions include the following:
  • Carefully analyze and detect features and services that are easy to embed and can be associated with malware providers (such as compilers, debugging artifacts, code extracts, or any other offerings related to Malware as a Service (MaaS)).
  • Malware repository scanning and feature identification can help to blacklist adversaries.

Now, let’s discuss a defense strategy involving the use of the open source secml library (https://secml.github.io/class6/, a security evaluation framework) to build, explain, attack, and evaluate security using algorithms such as Support Vector Machine (SVM) and ClassifierRidge (a custom ML Ridge classifier). These types of classification algorithms can be used to detect malware in Android applications and explain ML classifier model’s predicted outcomes.

In the following code snippet, we have loaded a toy dataset of Android applications, named DrebinRed. The loaded dataset consists of 12,000 benign and 550 malicious samples extracted from Drebin. On training (using a 0.5:0.5 train-test split) the dataset with SVM or the Ridge classifier, we observe the model has a 2% False Positive Rate (FPR) in correctly identifying the benign and malicious samples:

repo_url = 'https://gitlab.com/secml/secml-zoo'
file_name = 'drebin-reduced.tar.gz'
file_path = 'datasets/DrebinRed/' + file_name
output_dir = fm.join(settings.SECML_DS_DIR, 'drebin-red')
md5_digest = 'ecf87ddedf614dd53b89285c29cf1caf'
dl_file_gitlab(repo_url, file_path, output_dir,
branch='v' + min_version, md5_digest=md5_digest)

The following output snippet further illustrates the most significant components of the Android malware detector application. secml uses a Gradient * Input gradient-based explanation technique to explain the attributions of different points during the classification phase. The most important features (the top 5) and their relevance (in terms of percentage) help to explain each correct (not a part of the malware component) and corrupted sample, and even this approach/technique to explain attributions on sparse datasets:

Explanations for sample 137 (true class: 0)
 -7.49 suspicious_calls::android/net/Uri;->toString
 -5.63 suspicious_calls::getSystemService
 -5.42 api_calls::android/media/MediaPlayer;->start
 -4.99 used_permissions::ACCESS_NETWORK_STATE
 -4.55 req_permissions::android.permission.ACCESS_FINE_LOCATION

As ~25% of the relevance is attributed to five features, these features have a larger impact on the classifier being susceptible to adversarial evasion attacks. Leveraging this behavior of the malware detector, a gradient-based maximum-confidence evasion attack can be employed to generate adversarial samples against the classifier. This can trigger an L1-order sparse attack by changing one feature at a time to misclassify outputs as 1 instead of 0 and vice versa. We can trigger attacks such as the one demonstrated in the following code snippet, where feature addition works better to fool the malware classifier than feature removal. Removing features may remove other important components of the model, making it more difficult to misclassify. On the contrary, feature addition is an easy way to fool the model into classifying correct (benign components of the loaded dataset) samples as corrupted.

After adding the adversarial samples, we can trigger the evasion attack with classifier, distance, and other parameters, as shown here:

params = {"classifier": clf,
    "distance": 'l1',
    "double_init": False,
    "lb": 'x0', #feature addition, lb=0 for feature removal
    "ub": 1,  #feature addition
    "attack_classes": 'all',
    "y_target": 0,
    "solver_params": {'eta': 1, 'eta_min': 1, 'eta_max': None, 'eps': 1e-4}
}
from secml.adv.attacks.evasion import CAttackEvasionPGDLS
evasion = CAttackEvasionPGDLS(**params)

secml determines the model robustness using a Security Evaluation Curve (SEC) by running evasion attacks on the classifier with the L1-order perturbation epsvalue varying between 0 and 28 with a step size of 4.

To test the Android malware detector against a greater number of added features, we can run the evasion attack on the security evaluation method, as detailed in the following code snippet:

from secml.adv.seceval import CSecEval
sec_eval = CSecEval(
    attack=evasion,
    param_name=param_name,
    param_values=param_values)
sec_eval.run_sec_eval(adv_ds)

Now, let’s plot the SEC:

1. The following code begins the process of getting the SEC:

   from secml.ml.peval.metrics import CMetricTHatFPR, CMetricTPRatTH

   CopyExplain
2. Next, get the ROC threshold at which the detection rate should be computed:

   th = CMetricTHatFPR(fpr=fpr_th).performance_score(y_true=ts.Y, score=score_pred[:, 1].ravel())

   CopyExplain
3. Finally, use the convenience function to plot the SEC:

   fig.sp.plot_sec_eval(sec_eval.sec_eval_data, metric=CMetricTPRatTH(th=th),percentage=True, label='SVM', color='green', marker='o')
   fig.sp.ylabel(r'Detection Rate $(%)$')
   fig.sp.xlabel(r"$varepsilon$")

   CopyExplain

We can see how the SVM classifier is highly vulnerable to adversarial attacks, and particularly sensitive to attacks against the most impactful features. An attack can evade this classifier with a perturbation as small as eps (ε) = 0.1. When we change it to have fewer than 10 features (which are the most important ones), half of the corrupted samples are misclassified as correct ones.

In the following figure, Figure 2.2, the chart labeled A shows a detection rate of 97% with an FPR of 20. While the detection rate falls with increasing epsilon (ε), we observe that the fall is very steep for the Ridge classifier (C), while it happens in a step fashion for SVM (B). As the fall is steeper for the Ridge classifier, it is not a better option than SVM, which will exhibit a lower FPR. Make sure to examine the SECs in the following graphs, which provide estimations of the detection rate (%) with ε. The SEC plots help us to conclude that the malware detector ceases to perform with increasing levels of adversarial perturbations.

Figure 2.2 – Malware detection rate and SEC on SVM and the Ridge classifier

Staging refers to actions taken by adversaries to upload, install, and set up capabilities on infrastructures that were previously compromised or rented by them, to target victim networks. Such activities might include setting up web resources to exploit the victim’s browsing website (to steal confidential information) or uploading malware tools to initiate attacks on the victim’s network. There’s no prompt detection technique to avoid this; however, internet scanning tools may reveal the date and time of such attacks.

Initial access

Initial access helps an adversary to leverage security weaknesses on public-facing web servers and gain access to a network. This can occur in one of the early stages of development when the model design and the system architecture are still in the development phase. The primary steps to mitigate initial adversarial access include controlling the abuse of credentials via proper management of user account control, issuing valid accounts, enforcing privileged account management practices, defining organization password policies (such as the frequency of password changes), and having in place a systematic user training process and application developer guidance to restrict any illegitimate access to systems.

Let’s now explore the different actions that can be taken by adversaries if they are successful in acquiring initial access:

• Supply chain compromise: In this step, adversaries compromise different components of a victim’s system (such as GPU hardware, data and its annotations, parts of the ML software stack, or the model) to carry out an attack. The attacker manipulates development tools, environments, code repositories, open source dependencies, and software update/distribution mechanisms; compromises system images; and replaces legitimate software (using different versions) to successfully compromise the victim’s systems. Organizations should mitigate tampering activities by employing techniques to verify distributed binaries (hash checking), along with using tools to scan malicious signatures and engaging in physical hardware inspection. Even using patch management processes and vulnerability scanning tools to scan dependencies, unnecessary features, components, and files can help prevent adversarial access by enforcing strong testing rules prior to deployment.
• Drive-by compromise: This involves the exploitation of the victim’s browser (where the adversary may inject malicious code with JavaScript, iFrames, and cross-site scripting, or help to serve malicious ads) and application access tokens, and can be mitigated by doing the following:
  • Using browser sandboxes, deploying virtualization measures, and applying micro-segmentation logic. We can limit attacks by isolating applications and web browsers by creating and defining zones in data centers and cloud environments (to isolate workloads). This is one of the strongest ways to limit network traffic and client-side exploitation.
  • Employing defense tools such as Enhanced Mitigation Experience Toolkit (EMET), network intrusion detectors with SSL/TLS inspection, firewalls, proxies, ad blockers, and script-blocking extensions can help to control exploitation behavior, block bad domains and ads, and prevent the execution of JavaScript.
• Exploit public-facing applications: As this technique involves adversaries accessing, exploiting, and bringing down public-facing services such as databases, Server Message Block (SMB), and other applications with open sockets, the main remediation tasks lie with the security architects in designing and deploying application in isolation and sandboxing (limiting exploited targets’ access to other processes), web application firewalls, network segmentation (segmenting public interfaces on a demilitarized zone or a separate hosting infrastructure), and privileged account management (adhering to the principle of least privilege for accessing services) to limit attack traffic. In addition, regular software updates, patch management, vulnerability scanning tools, and application log and network flow monitoring tools (using deep packet inspection to discover artifacts of malicious traffic, such as SQL injection) can be used to detect improper input traffic and raise alerts.
• External remote services: This method involves adversaries discovering external-facing remote services, such as VPNs or Citrix, and finding routes to connect to internal enterprise network resources from these external locations. To alleviate such risks, security teams should be extra cautious:
  • Disable or block unnecessary remotely available services, limit access to resources over the network (by prompting the use of managed remote access systems such as VPNs), enable multi-factor authentication, and allow network segmentation (through the use of network proxies, gateways, and firewalls).
  • Facilitate log monitoring related to applications, session logons, and network traffic to detect authenticated sessions, discover unusual access patterns and times of operation, and assist in detecting adversarial behavior.
• Hardware additions: Introducing additional computer accessories or hardware components in the network can permit adversaries to undertake passive network tapping, network traffic modification through adversary/man-in-the-middle attacks, keystroke injection, or kernel memory reading via Direct Memory Access (DMA). To avoid this, asset management systems should be used to do the following:
  • Limit access to resources over the network, limit installation of hardware, and employ hardware detectors or endpoint sensors to discover additions of USB, Thunderbolt, and other external device communication ports in the network.
  • Further, to safeguard adversarial copying operations on removable media, organization policies should forbid or restrict removable media.

ML model access

Adversaries may gain access to an ML model legitimately through four different techniques that we’ll examine in this section. The best mitigation strategy is to include sufficient security rules (cloud- and token-based authorization schemes) to enable authentic access to model APIs, ML services, physical environments, and ML models, as follows:

• Model inference API access: This involves restricting adversary access through the use of APIs to discover the ML model’s ontology or family. The corresponding defense action is to limit the introduction of test data into the target systems by single agents to prevent issues related to evading ML models and eroding ML model integrity. As we saw in Chapter 1, there is a possibility of an evasion attack where attackers try to evade detection by hiding the content of spam and malware code. The same kind of attack is possible by using model inference APIs to misclassify examples (individual data samples) as legitimate.
• ML-enabled product or service limit: This method limits indirect access to ML models to hide information related to the model’s inference from its logs and metadata. Indirect access can originate from any product or service built by adversaries to gain access to the victim’s ML model.
• Physical environment access: To eliminate the scope of adversarial attacks in data engineering pipelines, enable data validation checks across multiple layers of input data ingestion, preprocessing, and feature engineering.
• Full ML model access: To prevent the adversary from gaining full access to the model, the best possible defense strategy is to incorporate privacy-preserving ML techniques for data aggregation and training to enable protection from adversarial white-box attacks. Otherwise, these attacks allow the adversary to gain complete information on the model’s architecture, parameters, and class ontology and exfiltrate the model to execute offline attacks once the model is running live with production data.

One of the preferred mechanisms for defending against white-box (model parameters) and black-box (output predictions) attacks is to train the model and evaluate the accuracy of the attacks. If we use ML Privacy Meter (a Python library that helps to quantify risk in ML models) prior to releasing models, we can test the models by initiating attacks and determine the model’s tendency to leak information. This helps us to act as adversaries and detect whether each data instance actually belongs to the required dataset. Training the model against such attacks can be accomplished in two ways:

• White box: By observing the model’s parameters when the model is deployed in an untrusted cloud or takes part as one of the participating models in a Federated Learning (FL) setup
• Black box: By fetching the model’s predictions from the output

During model training and evaluation, the attack accuracy is evaluated on a validation/test set. Moreover, the accuracy is only considered on the best-performing attack model of all attack models. In Figure 2.3, we can see three plots that illustrate the probabilities (in the range of 0 to 1) of responses that actually respond to an attack based on membership status.

Figure 2.3 – Overall privacy risk (left) and privacy risk for classes 24 and 35 (center and right)

With the following code, we are able to detect the trade-off between the model’s achieved accuracy (correct identification of members in the training dataset) and error (incorrect identification or false positives). The following code snippets show how to invoke attack models and verify the probability of each member getting discovered through an adversarial attack:

import ml_privacy_meter
import tensorflow as tf
datahandlerA = ml_privacy_meter.utils.attack_data.attack_data(dataset_path=dataset_path,
member_dataset_path=saved_path,
            batch_size=100, attack_percentage=10,
input_shape=input_shape,
            normalization=True)

The method for starting the attack is shown as follows, where the first two parameters specify the target training model and target attack model, the third and fourth parameters denote the training and attack datasets, while the remaining parameters are used to specify layers, gradients, model name, and more:

attackobj = ml_privacy_meter.attack.meminf.initialize(
    target_train_model=cmodelA,
    target_attack_model=cmodelA,
    train_datahandler=datahandlerA,
    attack_datahandler=datahandlerA,
    layers_to_exploit=[26],
    gradients_to_exploit=[6],
    device=None, epochs=10, model_name='blackbox1')
attackobj.train_attack()
attackobj.test_attack()

In addition, we are also able to see the privacy risk histograms for each output class. While the first histogram shows that there is an increase in risk at every step for training data members, the privacy risk for class 24 is more uniformly distributed between 0.4 and 1.0. On the other hand, the privacy risk for class 35 is more skewed between 0.85 and 1.0 for most of the training members. The overall privacy risk histogram is an average aggregation of all privacy risk classes.

Model training and development

This phase, as shown in Figure 2.4, pertains to all actions during model training and deployment where the adversary has started to extract model and system parameters and constraints to their advantage, evading defense frameworks in the target environment and preparing the ground for continued attacks.

Figure 2.4 – Different attack stages during model training and deployment

Execution

Different command and script interpreters can be used by adversaries to execute commands, scripts, or binaries by embedding them as payloads to mislead and lure victims. Container administration commands and container deployments (with or without remote execution) can help adversaries to execute commands within a container and facilitate container deployment in an environment to evade defenses. Adversaries can also be prompted to schedule jobs for the recurrent execution of malicious code or force users to undertake specific actions (for example, opening a malicious file) to execute malicious code.

Execution can be accomplished in the following ways.

• User execution – unsafe ML artifacts: Adversaries may develop unsafe ML artifacts (without adhering to serialization principles) that can enable them to gain access and execute harmful artifacts.
• Exploitation for client execution: Adversaries may exploit vulnerabilities in client software by leveraging browser-based exploitations, inter-process communication, system services, and native APIs (and their hierarchy of interfaces) in their favor to enforce the execution of malicious content.
• Software deployment tools: After gaining access to an enterprise’s third-party software, it becomes easier for adversaries to gain access to and wipe information from hard drives at all endpoints.

In addition to the commonly used defense mechanisms that we examined in the first phase, defense strategies should focus on enforcing limits on harmful operations such as the following:

• Limiting access to resources over the network (enabling authenticated local and secure port access to aid communication with APIs over TLS), privileged account management (not allowing containers or services to run as root), behavior prevention on endpoints, execution prevention (by using application control logic and tools such as Windows Defender Application Control and AppLocker, or software restriction policies), code signing, application isolation, and sandboxing.
• When adopting system-level security measures, DevOps and security teams should wisely use and manage the operating system’s configuration management (forcing scheduled tasks to run under authenticated accounts instead of allowing them to run under system services).
• Active Directory configuration to reinforce Group Policy enforcement to isolate and limit access to critical network elements.

To further curb execution operations practiced by adversaries, the following persistence actions should be enforced by system administrators to prevent unwanted intrusion.

Persistence

Here is a list of the actions to prevent intrusion:

• Preventing the execution of code that has not been downloaded from legitimate repositories (which means ensuring only non-vulnerable applications are allowed to have setuid and setgid bits set)
• Privileged account management (don’t allow users to be unnecessarily added to the admin group)
• Restricting file and directory permissions
• Restricting library loading through permissions and audits
• User account control (using the highest enforcement level, leaving no room for bypassing access control)

Defense evasion

Defense evasion comprises all operations used by adversaries that enable them to evade detection and the existing security controls. Adversaries are powerful enough to break through the victim’s systems with the untrusted activities listed in the following table:.

Table 2.1 – Different modes of ML model defense evasion

As defense evasion relies on the abuse of system failures, stringent security measures should be put in place to close all loopholes. Most of the defensive tactics described previously apply here. In addition, prevention techniques that need greater attention are the following:

• Deploying network monitoring tools to filter network traffic.
• Deploying antivirus and antimalware detectors for monitoring.
• Employing endpoint behavioral anomaly detection techniques to stop the retrieval and execution of malicious payloads.
• Operating systems should be configured such that administrator accounts are not enumerated and do not reveal account names.
• When not in use, active macros and content should be removed from programs to mitigate risks arising from the execution of malicious payloads.
• Unnecessary scripts should be blocked, passwords should be encrypted, and boot images of network devices should always be cryptographically signed.

Discovery

The discovery phase helps adversaries gain knowledge of the victim’s account, operating system, and configuration (as listed in Table 2.2) for systematic planning prior to invading the victim’s systems.

Table 2.2 – Different discovery mechanisms

Due to adversarial pre-planning, the foremost defense steps include the following:

• Enable monitoring of all events together, without viewing any suspicious action in isolation. Sequential information discovery and collection are part of a larger attack plan, such as lateral data movement or data corruption.
• Discovery and proof collection (using screenshots and keyboard inputs), which could help in the process of reconciliation to justify acts of data stealing.

Collection

After the discovery of data sources, an adversary will be enthusiastic to collect and steal (exfiltrate) confidential sensitive information either manually or through automated means. Common target sources include various drive types, removable media, browser sessions, audio, video, emails, cloud storage, and configuration. Other sources of information include repositories, local systems, network shared drives, screenshots, audio/video captures, and keyboard input.

ML model live in production

This phase shown in Figure 2.5 relates to attacks performed on ML models and ML SaaS platforms at scale. Here, the adversary is fully equipped with full system-level information, data, and proxy models that are essential for them to execute attacks and impact the victim’s business operations.

Figure 2.5 – Different attack stages when ML models are live in production

This phase involves the manipulation, interruption, or destruction of data by adversaries to compromise system integrity and disrupt business operations. Attacks can range from data tampering activities to techniques involving crafting adversarial data, to restrict ML models from yielding the right predicted results.

Staging ML model attacks

Once the discovery and collection phases are over, the adversary leverages their new knowledge to plan and attack the system intelligently (online or offline) by training proxy models and triggering poisoned attacks by injecting adversarial inputs into target models. Target models act as important resources in staging attack operations:

• Collecting ML artifacts: Once the adversary has successfully gathered information on the ML artifacts that exist on the network, they may exfiltrate them for immediate use in staging an ML attack. To mitigate risks involving the collection of model artifacts, note the following:
  • The ML model training methodology should encompass all privacy-preserving techniques.
  • In addition, all ACL rules (of related microservices) and encryption logic should frequently be audited and revisited.
• Training proxy ML models: Adversaries often train ML models to create proxy models and trigger attacks on the target models in a simulated manner. This offline simulation helps them to gain information from target models and validate and initiate attacks without any need for higher-level access rights or privileges.
• Replicating ML models: Here, an adversary replicates the target model as a separate private model, where the target model’s inferences are recorded as labels for training the offline private version of the model. This kind of operation involves repeated queries to the victim’s model inference APIs. To throttle repeated requests from the same IP, defenders can use rate limiting and blacklist source IPs to limit such queries and consequently the number of inferences extracted.
• Poisoning ML models: Adversaries can generate poisoned models from the previous steps by injecting poisoned data for training or carefully retrieving the model inferences. In fact, poisoned models are a persistent artifact at the victim’s end that the adversary can use to their advantage to insert and trigger backdoor triggers with completely random patterns and locations to evade backdoor defense mechanisms, making it difficult for monitoring tools to detect issues and raise alerts.

One mechanism of defense against poison attacks is to use spectral signatures (https://proceedings.neurips.cc/paper/2018/file/280cf18baf4311c92aa5a042336587d3-Paper.pdf), which should detect the deviations in average value created by the minority sub-population of poisoned inputs. This algorithm depends on the fact that the means of two sub-populations are fairly different in comparison to the overall variance of the populations, owing to the fact these two sub-populations show either the presence or absence of correctly labeled samples or corrupted samples. In such a scenario, one population sub-group containing mislabeled corrupted inputs can be identified using Singular Value Decomposition (SVD) and removed. This algorithm performs efficiently against back-door attacks carried out on real image samples and state-of-the-art neural network architectures. In the following snippet, you can see a SpectralSignature defense employed on a Keras classifier, which returns a report (a dictionary containing an index of keys and values as the outlier score of suspected poisons) and is_clean_lst, denoting whether each data point in the training data is clean or poisoned:

defence = SpectralSignatureDefense(classifier, x_train, y_train,  batch_size=128, eps_multiplier=1, expected_pp_poison=percent_poison)
report, is_clean_lst = defence.detect_poison(nb_clusters=2,                                           nb_dims=10,reduce="PCA")
pp = pprint.PrettyPrinter(indent=10)
pprint.pprint(report)
is_clean = (is_poison_train == 0)
confusion_matrix = defence.evaluate_defence(is_clean)

• Verifying attack: This action helps an adversary to plan, prepare, and verify planned attacks based on the suitability of the time chosen and the availability of the victim’s physical or virtual environments. Mitigation strategies against this type of attack are difficult to implement as adversaries can leverage inference APIs with limited queries or create offline versions of the victim’s target model. This leads to increased API billing costs for the victim, as API costs are directly borne by them.
• Crafting adversarial data: Adversarial data that serves as input to ML models can be misclassified, increase energy consumption, or make the model prone to failure. White-box optimization, black-box optimization, black-box transfer, and manual modification are population algorithms that can help adversaries to generate input data samples to evade ML architecture. The key purpose of adversaries is to disrupt the ML models by challenging their integrity.

Command-and-control requests

These operations help adversaries move one step closer to extracting useful information from the victim’s network, as listed in the following table.

Table 2.3 – Different modes of control operations

However, the adversary is clever enough to do this in such a way that the existing defense strategies on the target network will not raise an alarm. Therefore, some appropriate defense techniques are as follows:

• Enabling the adoption of specially crafted adversarial protocol tunnels, hiding open ports through traffic signaling, and making use of proxies can help to avoid direct communication between a command-and-control server and the victim’s network.
• The use of different application- and network-level authentication and application sandboxing mechanisms, discussed previously, is highly recommended.
• Additionally, network segmentation by properly configuring firewalls for existing microservices, databases, and proxies to limit outgoing traffic is essential.
• Only authorized ports and network gateways should be kept open for hosts to establish communication over these authorized interfaces.

Exfiltration

Data exfiltration can be carried out by adversaries (as listed in Table 2.4) over the network, after data collection, encryption, compression, and packaging. Data can be packaged and compressed to different-sized blocks before being transmitted out of the network using a command-and-control channel or alternative channel strategies.

Table 2.4 – Different modes of exfiltration

The most common and easiest ways to carry out exfiltration attacks are by doing the following:

• Inferencing ML model APIs for exfiltration: ML model inference API access is the primary means for adversaries to look for ways to exfiltrate/steal private information from the model’s inference APIs.

To mitigate exfiltration risks, the following defense actions are mandatory:

• Private data should be trained using application-level privacy techniques or by making the best use of hybrid security measures (application- and transport-level security) to protect against leakage of Personally Identifiable Information (PII).
• Data Loss Prevention (DLP) APIs can be used to detect and block the transfer of sensitive data over unencrypted protocols.
• Network intrusion detection and prevention systems can be used with network signatures to monitor and block malware traffic.
• Restricting web content access by using web proxies can help to minimize unauthorized external access.

• Evading ML models: Adversaries can use traditional cyberattacks where adversarial actions can evade ML-based virus/malware detection.
• Denial of Service (DDoS): Here, adversaries are driven by the objective to bring down ML systems in production by issuing a flood of requests. The requests may be computationally intensive, requiring large amounts of memory, GPU resources, and processing cycles, and can overload the productionized systems, which may become too slow to respond.
• Spamming ML systems: Here, the adversaries increase the number of predictions in the output by spamming the ML system with false and arbitrary data. This impacts the ML team at the victim’s organization, who end up spending extra time deducing the correct inferences from the data.
• Eroding ML model integrity: Adversaries may degrade the target model’s performance with adversarial data inputs to erode confidence in the system over time. This can lead to the victim organization wasting time and money attempting to fix the system.
• Harvesting cost: This is similar to a DDoS attack, where adversaries engage themselves in targeting the victim’s ML services to increase the compute and running costs by bombarding the system with false and specially crafted queries. Sponge examples are specially crafted adversarial inputs, designed to increase processing speed and energy consumption, which can degrade the overall performance of the victim’s systems.
• ML IP theft: Here, adversaries steal intellectual property from ML models, training and evaluation datasets, and their related artifacts with the objective of causing economic harm to the victim organization. This act enables adversaries to have unlimited access to their victim’s service free of cost, avoiding the MLaaS provider’s API charges.
• System breakdowns: Other than the commonly used mechanisms, impact strategies (the third attack strategy shown in Figure 2.5) are mainly targeted at systems in production and include a variety of irrecoverable data destruction mechanisms such as overwriting files and directories with random data, manipulating data, defacement, wiping data, corrupting firmware, large-scale data encryption on target systems to disrupt the availability of system and network resources, commands to stop the service, system shutdown/reboot, and resource hijacking with the objective of bringing down the victim’s system resources.

The ideal way to mitigate risks related to impacts is to follow these best practices:

• Have a data backup process to protect against any data loss/modification attempts.
• Have model robustness test strategies in place by thoroughly testing ML models against sponge attacks.
• Worst-case or low threshold boundaries to validate model robustness can also aid in detecting adversarial attacks, where system-level degradations in performance are a symptom of inputs from external sources not designed for the system.

Up to now, we have discussed the attack threat matrix, which we first saw in Chapter 1, Figure 1.13, and different defense mechanisms available for different types of adversarial attacks.

Now let’s look into the data anonymization and encryption techniques available to protect sensitive data.

Due to the possibility of different attacks and threats, organizations have become more responsible about safeguarding the data rights of their employees. The Data Breach Survey of 2019 revealed that 79% of CIOs were convinced that company data was put at risk in the previous year because of actions by their employees (https://www.grcelearning.com/blog/cios-increasingly-concerned-about-insider-threats). The data security practices of as many as 61% of employees put the company at risk, which led organizations to adopt best practices related to data anonymization. Some of the practices that organizations should follow to comply with GDPR and other regulations will be discussed in this section.

Data anonymization or pseudo-anonymization needs to be carried out on PII, which mainly includes names, ages, Social Security Numbers (SSNs), credit card details, bank account numbers, salaries, mobile numbers, passwords, and security questions.

In addition to this, company policy and database administrators can define extra processes before the application of anonymization techniques. Now, let’s look at some of the most commonly used techniques for data anonymization.

Data masking

This technique hides and protects the original data by generating mirrored versions of it at random and then shuffling it with the original version of the data. There are five primary types of masking measures that make it difficult for the attacker to decipher the original data:

• Deterministic data masking: This process allows the replacement of any columnar value with a specific value in any location of the table – be it the same row, the same database/schema, or between instances/servers/database types. It takes into consideration similar settings to generate replacement global salt keys (these are cryptographic elements that hash the data for security; for example, a website’s cookies). For example, XYZ can be replaced with ABC.
• Dynamic Data Masking (DDM): The objective of this masking technique is to mask real-time production-grade data such that live data streams are modified without the data generator/requestor having access to the sensitive data. It can be used by setting the central data masking policy to mask sensitive fields with full or partial masking functions, along with random masking for numeric data. It also finds heavy usage in simple transact SQL commands (one or more SQL commands grouped together, that can be committed to a database as a single logical unit or rollback) SQL commands (for example, on SQL Server 2016 (13.x) and Azure SQL Database). Now, let’s look at an example of how masking is done in Azure:

  Email varchar (100) MASKED WITH (FUNCTION = 'email ()') NULL

  CopyExplain

Here, the Email method uses masking to expose only the first letter of an email address and the constant suffix .com, producing the following: aXXX@XXXX.com.

However, dynamic masking cannot be applied to encrypted columns, file streams, COLUMN_SET, or computed columns that have no dependency on any other columns with a mask.

• On-the-fly data masking: This process is common when data from development environments is masked without the use of a staging environment due to factors including insufficient extra space, or under the constraint that the data must be migrated to the target environment. This masking technique is used in Agile development processes where Extract, Transform, Load (ETL) is directly able to load the data into the target environment without creating backups and copies. However, the general recommendation is to refrain from using this technique widely (other than in the initial stages of the project) to avoid risks related to compliance and security issues.
• Static data masking: This method triggers data masking on the database copy with a complete replacement of the original data, without leaving any room to recover the original data. It is a recommended procedure for development, testing, analytics, and troubleshooting purposes as it allows the storage of masked data with a clear separation between production and dev/test setups, thereby enabling compliance and security conformance. For example, a username and email address can be masked from Julia Gee to NULL Fhjoweeww and andwb@yahoo.com to yjjfd@yahoo.com respectively.
• Synthetic data: Synthetic data is a data anonymization technique employed to preserve the statistical properties of the original dataset, with a considerable margin of variable privacy gain and unpredictable utility loss. The increased privacy provided by this method offers protection against privacy-related attacks and prevents the re-identification of individuals. The synthetic data generated from generative models (for example, using deep learning techniques to generate deep fakes, where synthetic data recreates fake images resembling the originals) ensures high utility by enabling similar inferences as the original data.

Data swapping

This procedure shuffles and rearranges data to completely break the similarity between the original and the resultant datasets. There are three popular data-swapping techniques:

• K-anonymity
• L-diversity
• T-closeness

The techniques can all be used to make the deanonymization of data difficult for any intruder.

Data perturbation

This data anonymization principle adds noise to numerical data in databases to ensure its confidentiality. The process of adding or multiplying random noise additive, multiplicative, or random noise (used in Gaussian or Laplace distribution) helps to distort data, protecting it from being parsed by an attacker.

Data generalization

This method makes the data less identifiable by allowing you to remove certain ranges or portions of data (for example, outliers) from the database. One example is replacing age 45 years with <= 45, where the value is replaced by a wider range that is still semantically consistent. Data portioning of this type or attribute assignment to specific categories serves as a measure to generalize the data.

Broadly, generalization can be applied at the full domain level or to individual sub-domains. In the former, data transformation takes place for a generic domain or level of the hierarchy, while in the latter the level of generalization occurs on different subsets of the same domain.

Generalization-based techniques can be further applied to categorical or discrete numeric attributes to keep them private. There are two primary means by which generalization can be applied: a hierarchy-based approach or a partition-based approach (where a structure or order of partition needs to be established on the data items, before running the partition scheme on continuous numerical attributes). The partition-based approach partitions the data items into ranges, while hierarchy-based generalization requires the existence of generalization hierarchies and can be used for categorical and discrete numeric attributes. Some of these generalization techniques are described in the following sections.

K-anonymity

Sweeney first proposed the K-anonymity principle, which can be used as a framework to evaluate algorithms that carry sensitive information (https://epic.org/wp-content/uploads/privacy/reidentification/Sweeney_Article.pdf). In the absence of K-anonymity, sensitive attributes can leak and reveal boundary limits from the information elements. The application of K-anonymity is aimed at transforming the protected elements (either by generalization or suppression) with the objective of safeguarding the dataset from the hands of an intruder.

K-anonymity ensures any generalized block consisting of an individual record cannot be differentiated from K-1 other individuals in terms of any set of quasi-identifiers (such as zip code, age, or gender). This anonymization principle helps protect against linkage attacks, as discussed in Chapter 1. We see that in a linkage attack, an attacker is able to reveal the identity of a victim (unique identification number, credit card number, or others) and combine this with other background information, such as a user’s travel details like source, destination, and means of travel. This combined information can assist an adversary in tracking the user’s entire whereabouts. For example, a value of 10 for K on the protected attributes of age, SSN, nationality, salary, and bank account details will produce a minimum of 10 different records for each combination of the defined attributes.

As Figure 2.6 illustrates, with K = 2, there are two records listed as Asian and two as Hispanic. We can see that an equivalent number of records are distributed with respect to sensitive attributes such as age and salary across the two races. Figure 2.6 provides a complete demonstration of an end-to- end pipeline, showing risk score assessments, information recognition, pseudonymization, and anonymization, to audit and store sensitive data.

Figure 2.6 – Different anonymity models – K-anonymity, L-diversity, and T-closeness

The limitation of this system is that if an adversary has any information about the victims beforehand (say, their age), then it would be much easier for them to identify other attributes, such as salary- and disease-related information. For example, a friend of Nancy’s may know the age group and race to which she belongs, which would enable her friend to guess and retrieve her salary. The presence of homogeneous salary or age groups makes it possible for the adversary to derive sensitive information by the process of elimination or negative disclosure. It becomes easier for an adversary to initiate their attacks successfully and with high confidence. As a result, improvements for K-anonymity have been suggested to prevent background knowledge and homogeneity-based attacks.

L-diversity

The L-diversity principle (defined by Machanavajjhala et al.) was designed to handle the drawbacks of the K-anonymity algorithm to reduce the probability of homogeneity and background knowledge attacks. The L-diversity principle provides privacy where the data publisher is not aware of the knowledge that is possessed by the adversary. The fundamental idea behind L-diversity is the requirement that each group sees an equivalent of the sensitive values. The known information can be modeled as a probability distribution by applying Bayesian inferencing to reduce the granularity of the base data representation. The primary objective of generalization is to obtain different values for sensitive data that is evenly distributed.

The major limitation of this algorithm becomes visible when the distribution is skewed and fails to achieve an entropy of uniformly distributed L distinct sensitive values for each equivalence class. In such a case, the overall entropy level of the table drops, and the algorithm becomes non-functional and cannot offer sufficient protection.

Even though this algorithm takes into consideration the diversity of sensitive values within each group, it fails to constrain the value ranges and boundaries of the diverse groups. Given features with very close boundaries (say, salary > US $20,000 versus salary > US $30,000), the attacker can retrieve the salary information from the two equivalence classes (similar classes with close boundaries) that have different age groups.

T-closeness

As K-anonymity and L-diversity together cannot safeguard against skewness and similarity attacks, T-closeness came into being to offer extra robustness to the anonymization framework. This principle states that the two distributions – one of which is the sensitive value distributions of any group (say, racial groups; Asian and Hispanic in our example in Figure 2.6) and the other is the overall sensitive value distribution – cannot differ by more than a threshold of t. The distance metric used to evaluate the difference between the two distributions is the Earth-Mover Distance (EMD). It is computed using the possible set of sensitive attributes by evaluating the maximum distance between them. The metric gives a measurement between 0 and 1, in the space normalized to 1, with the intention of including the semantic closeness of attribute values in addition to the generalization of quasi-identifiers and the suppression of sensitive values.

Though T-closeness can resolve the shortcomings of K-anonymity and L-diversity, by offering protection from homogeneity, background knowledge, skewness, and similarity attacks, it remains vulnerable to minimality and composition attacks. In the former, the attacker deduces the boundary condition or the minimality criteria beyond which the anonymized models (K-anonymity, L-diversity, and T-closeness) cannot provide sufficient protection against external threats. Using this information, the intruder can decipher sensitive information for some of the equivalence classes. In the second category of attack, the attacker exploits the availability of all different releases of anonymized datasets to integrate them into one single unit, where the unified dataset is used to breach individual privacy.

Among other, less popular anonymized models is the p-sensitive K-anonymity model, which ensures K-anonymity conditions on Masked Microdata (MM). It also fulfills the condition to limit the sensitive (confidential) attributes present within the same group. This means that it’s able to protect the model from linkage attacks by restricting the number of distinct values that each confidential attribute belonging to the same group can have to a minimal value of p. In addition, it also stops learning sensitive associations at the same time. We can denote an anonymity model as (∊, m) (with the frequency of sensitive attribute values ∊ remaining within the user-defined threshold in the equivalence classes). This was designed to protect sensitive numerical attributes from proximity breaches, where data boundaries are too close. This allows an attacker to gather information with high confidence that a sensitive attribute value falls in a specified interval.

Encryption

Encryption of personal data is a methodology used to protect data from external sources and limit access to users who are not authorized to access it. Three major data encryption schemes of symmetric encryption, asymmetric encryption, and hybrid encryption can be employed for the purpose of key generation, registration, usage, storage, monitoring, rotation, and deletion.

Symmetric encryption techniques such as Advanced Encryption Standard (AES) are fast and can be accelerated by processors for bulk encryption purposes. This procedure is dependent on the pre-sharing/exchange of a single key between the client and the server to be used for encryption and decryption purposes. To provide support for integrity and authentication, a message authentication code can be added on top of this.

Asymmetric encryption schemes such as Rivest, Shamir, Adleman (RSA), Digital Signature Algorithm (DSA), and Elliptic Curve Cryptograph (ECC) use two keys, one public and the other private, and offer strong protection against adversaries. However, the procedure is slow and has limited applications for protecting data in ML systems. The communicating parties are responsible for secretly storing their private keys, while the public keys are shared. This happens over the following steps:

1. Using Transport Layer Security (TLS), the sender and receiver finalize the symmetric key (session key) that is used to encrypt data sent by the sender.
2. The sender is responsible for encrypting the symmetric key with the receiver’s public key.
3. The symmetric key is decrypted by the receiver with their own private key.
4. Using the symmetric key, the data is decrypted by the receiver.

Now that we’ve had an overview of some encryption techniques, let’s look at another level of abstraction process with pseudonymization, which further increases the difficulty for adversaries to break the encryption key.

Pseudonymization

This method helps to anonymize data while preserving accuracy in statistics and value. Different types of encryption and hashing techniques can be used for this process. PII information (quasi-identifiers) is encoded with pseudonyms and preserved separately, which allows the easy re-identification of the original data through the use of cross-references or identifiers. In contrast to the anonymization technique, this procedure prevents permanent data replacement using a substitution principle. Pseudonymization can also be used for the encryption and decryption of PII, where the original data is translated into ciphertext, which can be reversed with the relevant decryption key.

The main disadvantage of this process is that when datasets contain billions or trillions of records, human review and re-assessment or re-identification becomes impossible. Moreover, when the most sensitive fields require precise values, we may miss potential sources of re-identification attacks due to the privacy-utility trade-off.

This process further allows vertical or horizontal distribution of PII (by storing it in some protected storage units) and maintaining a link between the identifiers. In a vertical data distribution system, pseudonymity is guaranteed by compartmentalizing individual data corresponding to different subsets of sensitive attributes in different sites. In a horizontal data distribution system, sites near user locations are responsible for storing data with the same sets of user attributes. One primary example of horizontal data distribution is health-related information, where data integration from different sources is a preliminary step to infer an individual’s health-related information.

Some common types of pseudonymization techniques are as follows:

• Encryption with secret key: PII can be encrypted and then decrypted by the owner using any of the symmetric or asymmetric encryption methods discussed in previous sections.
• Hash function: This function is applied to transform a dataset with variable feature attributes and data size to yield a fixed-size output. This method often runs the risk of revealing sensitive PII if the range of input values applied to the hash function is known beforehand. A better method of adding higher-order protection is to use a salted-hash function, which adds randomization and reduces the probability of an attacker retrieving the actual values.
• Hash function with secret stored key: The hash function used for transformation is supplied with a secret key to the input, which further reduces the probability of an attacker being able to retrieve the actual data by replay attacks, where an attacker intercepts the network. The data owner will still be able to retrieve the data with the secret key, but for the attacker it becomes increasingly difficult to compute all sorts of permutations and generate the actual key used for encryption.
• Hash function with deletion of secret key: This process involves the selection of a random number corresponding to every feature in the dataset to act as the pseudonym. Further, the correspondence table is deleted to reduce the probability of linkage attacks where an attacker can link the personal data of individuals from the dataset in question to other available datasets with other pseudonyms. It makes it even more difficult for an attacker to use all permutations for a non-existent secret key to replay the function and retrieve the original data.
• Tokenization: This method of data encryption enables the conversion of sensitive data with a randomly generated token. It is widely prevalent in the payment card industry in credit cards, wallets, and other applications involving unique identifiers (PAN cards, driving licenses, SSNs, etc.). Organizations dealing with such secret identifiers often employ a tokenization service that is responsible for payment authorization by generating and validating a token based on users’ identities. The generated token can then be stored and mapped in the database of the third-party service provider. The storage service reduces the risk of data loss by providing unlimited data retention capacity. The involvement of third-party providers helps to reduce issues related to Payment Card Industry Data Security Standard (PCI DSS) compliance.

Homomorphic encryption

Homomorphic encryption (HE) has gained a lot of prominence in the data security industry, especially in the design of high-grade cloud security applications, owing to the amount of protection guaranteed by this technique. It relies on a probabilistic asymmetric algorithm and adds an extra protective layer where parties who do not own the encryption and decryption keys are unable to decipher the encrypted data. The primary advantage of this method over traditional encryption methodologies is that it allows cloud providers to process the already encrypted data without having to decrypt it first. The processed results are available in encrypted form to the owner, who can retrieve the processed results with a decryption key.

HE follows multiplicative laws of encryption and computation, where the order of encryption and computation can be interchanged. Any computation, when applied on datasets a and b after they have been encrypted individually as E(a) and E(b), yields the same result as it would have when encryption is applied to the computed result E(a * b).

Hence, mathematically, E(a * b) = E(a) * E(b), where E(a) and E(b) refer to the encryptions applied to datasets a and b, respectively, and E(a * b) refers to the encryption applied on the resultant computations of a and b.

As they are equal, the decrypted values of E(a * b) and E(a) * E(b) are also equal.

Figure 2.7 illustrates a use case of HE that helps to achieve anonymous data processing for (a) single users and (b) multiple users seeking to process their encrypted (using their public key) sensitive data on a cloud server. In the first case (a), the individuals can directly obtain decrypted results as computed, evaluated, and returned by the server. In the second case (b), the server can aggregate the individual data by stripping off the identity information, processing their encrypted data, inferring statistical information, and sending the inferred data back to a third-party source that is responsible for collecting the final derived outcome. The data received by the third-party source can then decrypt the data and obtain the inferred results.

Figure 2.7 – Single- or multi-party HE

Such multi-party agents using HE can be used to efficiently run electronic voting systems. Individual voters can use their public keys to encrypt their ballots and send them to the voting server. The cloud server makes use of homomorphic evaluation to run extra validity checks on encrypted ballots and compute an aggregated encrypted result. The computed encrypted result can be sent to the organizers, who can then decrypt and deduce the overall voting results without having any knowledge of how specific individuals cast their votes.

Figure 2.8 demonstrates the change in approach to privacy using HE in comparison with traditional privacy approaches. In the traditional privacy mechanism, users were dependent on relevant cloud storage and its computation facility for data transmission, processing, and storage functionalities. The customers needed to establish added trust with the service provider, where the providers are not allowed to share private information with third parties without the customer’s consent. In the present day, HE-based privacy systems (for example, Microsoft’s SEAL platform) guarantee confidentiality with full protection of customer data in addition to encrypted storage and computation capabilities.

Figure 2.8 – A diagram showing traditional encryption versus HE

This methodology can also be efficiently used in the following areas of security involving cloud services:

• Private storage and computation
• Private prediction services
• Hosted private training
• Private set intersection
• Secure collaborative computation

The following code demonstrates instantiating a Pyfhel object (with public and private keys) at the client end, with the generation and saving of public and private key pairs within the context. The contextGen function is used to generate an HE context based on input parameters. We save the public key along with the context:

HE = Pyfhel()
HE.contextGen(p=65537, m=2**12)
HE.keyGen() # Generates both a public and a private key
HE.savepublicKey(pk_file)
HE.saveContext(contx_file)
a = 1.5
b = 2.5
ca = HE.encryptFrac(a)
cb = HE.encryptFrac(b)

Then we try to decrypt the encrypted values (a and b) in the cloud using PyCtxt. But the decryption process fails in the cloud, as demonstrated in the output. The cloud server then applies the mean of the encrypted values and sends it back to the client. But the client successfully decrypts the results and can obtain the mean value (the mean of 1.5 and 2.5 is 2).

The cloud server tries to decrypt and then apply a mathematical mean operation on the encrypted results sent by the client:

c2a = PyCtxt(pyfhel=HE_Cl, fileName=sec_con / "ca.ctxt", encoding=float)
c2b = PyCtxt(pyfhel=HE_Cl, fileName=sec_con / "cb.ctxt", encoding=float)
try:
    print (HE_Cl.decrypt(c2a))
    raise Exception ("This should not be reached!")
except RuntimeError:
    print ("The cloud tried to decrypt but couldn't!")

The cloud computes the mean value of the ciphertexts obtained in the previous steps:

c_mean = (c2a + c2b) / 2
c_mean.to_file(sec_con / "c_mean.ctxt")

The client loads and decrypts the result:

c_res = PyCtxt(pyfhel=HE, fileName=sec_con / "c_mean.ctxt", encoding=float)
print ("Client decrypt results", c_res.decrypt())

This results in the following output:

The cloud tried to decrypt but couldn't!
Client decrypt results 2.0

Let’s see when this might be a useful technique.

Applications of HE

HE finds widespread usage in scenarios where there is a demand for huge computational capacity for processing high volumes of sensitive data not available to users. Some useful practical applications are predictive and analytics use cases in genomics research, finance, healthcare, pharmaceuticals, government, insurance, manufacturing, and the oil and gas sector, as illustrated in the following list:

• Leveraging cloud services for backtesting stock market trading strategies in such a manner that the data remains secure and private from external systems, including attackers and cloud operators

Backtesting

Backtesting is a strategy that allows a trader to simulate a trading strategy using historical data to generate results and accordingly identify and analyze the risk and profitability before risking any actual capital.

• ML applications related to fraud detection, automated claims processing, and threat intelligence
• ML-based SaaS platforms analyzing DNA with DNA sequencing classifiers to provide predictive insights to medical institutions and hospitals
• ML-based SaaS platforms providing medical diagnosis, medical support systems such as healthcare bots, and preventive care
• ML-based complex design and architectural patterns to innovate and run novel algorithms; they can also be used to run a mechanical structural analysis for the aerospace or construction industry
• Predictive platforms for running secure auctions

However, there are certain limitations on the use of HE, such as encrypted search (returning search results to the query without learning about the response) on a database or spam-filtering encrypted emails. In the former scenario, it is not feasible for the server to determine the search query content, which would entail huge processing loads and costs on the server to perform a homomorphic evaluation of the entire search operation. In the case of spam filtering, the procedure can produce a list of encrypted messages as spam, which cannot be deleted unless the client is aware of the filtering criteria (for example, keywords) that qualify a message as spam.

Secure Multi-Party Computation (MPC/SMPC)

This is a security technique used in ML to train private sensitive data and create risk-proof ML models. Here, the participating candidates are allowed to perform computations on private data and evaluate the private models of one participant with another participant’s private data.

This protocol works on the principle of secret sharing, where a dealer can share a secret s among n parties. This scheme can protect the secret from t or fewer participating candidates, and, at the same time, a subset of t + 1 candidates can reconstruct the secret. The participating candidates obtain their shares in the output, which helps them to reconstruct the actual outputs through interpolation. If only a few selected candidates are configured to obtain their shares, participating candidates are also allowed to send shares to only relevant individuals. Let’s now investigate some practical use cases where MPC was used to protect sensitive data.

Google uses MPC to evaluate advertisement conversion rates by computing the privacy-preserving set intersection between those users viewing an ad and those who go on to purchase the product. Some organizations rely on threshold cryptography instead of legacy hardware for protecting cryptographic keys. Here, organizations depend on MPC for key generation, computation, and storage, instead of allowing individuals to hold private information.

This process helps organizations to protect keys from adversaries as the key shares are placed in different environments. What makes MPC most useful for big data and large-scale predictive systems is the following:

• Easy adaptability to cross-platform deployment models.
• A multi-tenant MPC can run as a cloud-native Key Management Service (KMS).
• Can be functional across multiple clouds (for example, AWS, Google, and Azure) simultaneously to maximize security and availability.
• Supports hybrid cloud multi-site enterprise deployments.
• Easy scalability.
• Sustained secure operations.

Figure 2.9 – SMPC – A. Shamir’s secret sharing and B. Threshold cryptography

The following code snippet demonstrates how to use the mpyc library to determine an aircraft’s location in a private manner using five sensors. Each sensor communicates using SMPC to share secrets for encrypting confidential information such as the location and time of arrival of the aircraft:

from mpyc.runtime import mpc
secint = mpc.SecInt(l)
scaling = 10**args.accuracy
locations[i] = mpc.input(list(map(secint, position_i)), senders=sender_pid)
toas[i] = mpc.input(secint(toas_i), senders=sender_pid)
x, y, z = await schmidt_multilateration(locations, toas)

Having understood different encryption methodologies used in multi-party communications and in the process of ML model training, let’s try to understand how we can employ application-level privacy techniques during the training phase of a model.

DP is a popular application-level privacy-enabling framework used to protect private or sensitive data on large datasets. This method guarantees an almost identical output when a statistical query is executed on two nearly identical datasets that differ only by the presence or absence of one record.

DP provides security against record linkage attacks by hiding the influence of any single record (for example, individual PII) or records of small groups of users in the predicted outcomes. The process of anonymization and protecting the availability of information related to the presence or absence of individual records in the data-training process is closely associated with the privacy of data against linkage attacks. The cumulative loss is defined as the privacy budget and is called epsilon (ε), which represents the quantifiable amount of privacy provided, where a low value signifies a high level of privacy. The loss is also associated with a decrease in utility (accuracy), and it is the task of the data scientist to arrive at an acceptable trade-off between the two.

Figure 2.10 illustrates a Stochastic Gradient Descent (SGD)-enabled DP training process, the computation of the loss function, the addition of random noise, and the clipping of gradients in successive iterations. When examining the diagram, note the following:

• A randomly sampled set of data points has been used for training (from two different datasets, D1 and D2, both ∈ to S, and differing in only one record).
• The training error (or training loss) is computed from the model’s predicted output and the training labels in successive steps are then differentiated with respect to the model’s parameters.
• The process continues iteratively where the computed gradients (using SGD) are applied to the model’s parameters, taking into consideration the impact left by every point in the resultant gradient.
• The impact of every gradient is controlled by clipping (or bounding) the gradients. The certainty of inferring a point’s inclusion in the dataset is diminished through the process of randomization where noise is added to every data point.
• Once the model converges, the final gradient is computed to derive the privacy estimate, such that O1 – O2 < ε.

Figure 2.10 – Training models with DP on two input datasets, D1 and D2

The following code snippet illustrates the necessary imports, including IBM’s Diffprivlib, used for training DP-based models. The first step involves having the necessary imports of libraries and fetching the adult_income dataset from the web, where for X_train we only use columns 0, 4, 10, 11, and 12, and for y_train we use the column specified as income <=50K or >50K:

import numpy as np
import sklearn as sk
from sklearn.pipeline import Pipeline
from diffprivlib import models
X_train = np.loadtxt("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data", usecols=(0, 4, 10, 11, 12), delimiter=", ")
y_train = np.loadtxt("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data", usecols=14, dtype=str, delimiter=", ")

The next code snippet illustrates how we can incorporate DP in different components of a pipeline such as StandardScaler, PCA, and LogisticRegression. The initial step involves scaling the feature attributes followed by dimensionality reduction and ML model classification.

The data_norm parameter quantifies the Laplace-distributed random noise that we added in order to yield a differentially private ML model. The bounds parameter defines the bounds of the data and takes in a tuple value as min, max, where these two entries represent scalars after being aggregated at a feature level. From Figure 2.11, it is further evident that accuracy is oscillating, with ε ranging from 10-3 to 10-1.5, after which accuracy becomes stable. This observation reinforces our discovery that selecting an acceptable trade-off value between accuracy and utility is needed to add a higher privacy margin without compromising accuracy:

dp_pipe = Pipeline([
    ('scaler', models.StandardScaler(bounds=([17, 1, 0, 0, 1], [90, 160, 10000, 4356, 99]))),
    ('pca', models.PCA(2, data_norm=5, centered=True)),
    ('lr', models.LogisticRegression(data_norm=5))])
dp_pipe.fit(X_train, y_train)
for epsilon in epsilons:
    _eps = epsilon / 3
    dp_pipe.set_params(scaler__epsilon=_eps, pca__epsilon=_eps, lr__epsilon=_eps)
    dp_pipe.fit(X_train, y_train)

Now, when we plot the results, we get the following graph demonstrating the variation of accuracy with ε:

Figure 2.11 – A DP-enabled pipeline exhibiting the accuracy and privacy-budget (ε) trade-off

By now, we have understood how ε serves as an important metric to measure the expected privacy of a DP model. Now, let’s try to understand the second most important metric, sensitivity.

Sensitivity

Sensitivity is defined as the maximum influence exerted by a single data record on the differential private result in response to a numeric query. For any arbitrary function f, the sensitivity ∆f of f, on x and y, two neighboring datasets, can be given as follows:

Δf = max { || f(x) – f(y) ||1 }, where ||.||1 represents the L1 norm of a vector

Properties of DP

DP solutions possess essential properties of postprocessing:

• Invariance/robustness
• Quantifiability
• Composition

The invariance/robustness of DP ensures additional computations executed on ε-DP solutions are also ε-DP. Additionally, the quantifiability property of DP allows the flexibility of being transparent (to the data scientist). Transparency reveals the exact quantity of noise/perturbation caused by the randomization process. This characteristic feature of DP algorithms gives them an extra edge over other traditional de-identification algorithms. The traditional algorithms hide the process by which the data was transformed, so data scientists cannot interpret and analyze the accuracy of such models. Further, with the composition property, it is possible to derive the amount of degradation in privacy by executing different DP algorithms on overlapping datasets. For example, two DP algorithms executed on ε1-DP and ε2-DP that overlap are DP private and given by (ε1+ε2)-DP.

We have now become familiar with different privacy- and security-enabled measures that can be used during model training and inference. However, each of these methods has its own specific advantages, and the entire system can only get the full benefit from a hybrid security-enabled system.

Figure 2.12 illustrates the integration of varying levels of the privacy components discussed in the previous sections to create a fully proofed privacy-preserving AI system. There are two different labels associated with 2 and 5, where 2 and 5 denote access by model owners, while 2’ and 5’ denote access by adversaries where they craft adversarial data to steal important model information.

The system can ingest data from multiple heterogeneous devices (1 and 2) before triggering different algorithmic training (3, 6, and 8). Such a hybrid system can ensure data and algorithm sovereignty, in addition to adhering to ethics, compliance, transparency, and the trustworthiness of applications. The main objective is to have a robust ethical defense framework in place that can protect a single data record from identity or Membership Inference Attacks (MIAs) by identifying its presence in the dataset (2’).

This is addressed by having a private AI unit that encompasses the task of adding application-level privacy through DP (during model training or post-model convergence), anonymization, and pseudonymization (Figure 2.12) to protect the data. Here, the random noise and regularization added by DP algorithms (trained with SGD or private aggregation of teacher ensembles) can increase resilience against inversion attacks.

As we studied previously, anonymization and pseudonymization still leave room for de-identification processes through feature re-derivation and re-identification (2’) where an attacker is able to break into look-up tables. Therefore, additional security measures need to be adopted to safeguard insecure storage.

Figure 2.12 – Application of different privacy measures in a hybrid privacy framework

To safeguard lookup-table-related risks, AI research has concentrated on decentralization where remote execution becomes the central mechanism to train a global model. Locally trained ML models with their weights, parameters, and data (from mobile, Internet of Things (IoT), and Internet of Medical Things (IoMT) devices) are updated to a central repository to aggregate the model at a global level. However, the local model’s weights still run the risk of being corrupted by adversaries either by the modification of transmitted parameters through poisoning or model-inversion/reconstruction attacks (5’) where SMPC and HE can be employed to best tackle the existing threats.

The decentralized approach to data training in a federation topology is revolutionizing the privacy landscape in the AI industry as devices are able to retain their sovereignty while participating in the gossip strategy. The flexibility of joining in the training process (by limiting their continuous availability) to share model parameters with peer nodes helps devices to sustain battery life for longer durations. It gives a broader scope of application to this federated mode of training in addition to data and model governance capabilities, where the tracking/auditing of the times that each device sends model parameters, convergence, and performance metrics can be monitored.

FL carries the underlying risk of model parameters and PII being stolen or reconstructed by adversaries in the absence of encryption techniques, from the nodes and communicating interfaces. Hence, local algorithms need to be encrypted and securely aggregated where HE can be employed (with or without DP) to securely aggregate encrypted algorithms. Another viable risk arising from neural networks is their compressed representation. Such compressed formats are achieved by applying either one of the following mechanisms: pruning the convolutional layers, quantization, tensor decomposition, knowledge distillation, or a combination of all the stated methods.Without encryption, attackers will find it easy to execute model inversion or reconstruction attacks and retrieve confidential model parameters with high accuracy.

You can read more on knowledge distillation at Knowledge Distillation: Principles, Algorithms, Applications: https://neptune.ai/blog/knowledge-distillation and about quantization at Pruning and Quantization for Deep Neural Network Acceleration: A Survey, https://arxiv.org/pdf/2101.09671.pdf

The solution is further extended to include secure multi-party computation to multiple participating entities where each of them receives a split of encrypted data to proceed with further processing. This decentralization approach removes the risk of complete data exposure or leakage to participating candidates, allowing data recovery only through the method of mutual consensus. SMC also works in semi-trusted and low-trust environments, but one of the requirements of this method is continuous data transfer between parties and the continuous online availability of devices, leading to additional communication overheads. This acts as a limitation on the reliability, redundancy, and scalability of the system, but can be overcome by designing appropriate sleep and wake cycles for devices, discussed more in Chapter 11. Data scientists should employ a holistic approach, incorporating all the privacy measures discussed to validate the integrity and quality of predicted ML results.

In this section, let’s walk through some of the newly evolving risk mitigation frameworks concerning specific scenarios of input data distribution or model architecture when the model is used for training and serving. These frameworks are highly successful in curbing real-world attacks. One example is the identification of diseases where clinical datasets have been used to train the associated model. In such cases, an attacker can infer from a clinical record that a given patient has a specific disease with a high probability of success.

Let’s now discuss how we can evaluate model risk for MIAs, where adversaries are able to copy the principal model functionality and trigger adversarial attacks. MIAs can be either black box or white box. In black-box attacks, the attacker knows only the model inputs and can only query the model’s predicted output label, whereas in white-box attacks, the attacker has knowledge of the model inputs, architecture, and model internals such as weights, biases, and other coefficient values.

The goal of MIAs is to infer whether a given data record is in the target dataset. To construct the MIA model, a shadow training technique is applied to generate the ground truth for membership inference. Figure 2.13 shows an overview of MIAs.

MIAs enable an attacker to determine the presence of a specific data point z in the training set of a target model a. When this attack takes place on a model trained with sensitive data, evaluating an individual’s presence in the dataset will expose confidential information to an attacker. MIAs achieve high performance on Independent and Identically Distributed data (IID), thereby completely ignoring the fact that data dependencies in training samples underestimate the attack performance. This suggests a new direction of research to evaluate vulnerabilities and devise defense techniques for correlated data to protect sensitive information.

Hence, there is a need for a risk mitigation technique that can evaluate and label ML models where such data dependencies exist (such as, for example, the effect of all members being from a specific health region (or hospital) and non-members from all other regions, or when both originate from the same source). To evaluate such associated risks, this framework could be used to access/test the model’s behavior under MIAs when data dependencies exist, and models are prone to correct membership inference outcomes. For example, with MIAs, the model is able predict whether people of the same racial background are likely to suffer from a disease.

This risk assessment framework first uses public data to train a set of shadow models that can emulate/mimic the target model’s functionality. In the next step, an attack model is trained to reveal the membership status of a sample using outputs from the shadow models. Before the identification phase begins, the dataset can be split between members and non-members (for example, through a clustering algorithm).

In the following code snippet, we have used the adult_income dataset to train a shadow model (80% training and 20% test data split) with a CNN. The shadow and attack models are trained using the same dataset after dividing the dataset equally between the two halves. The purpose of this framework is to measure the effectiveness of DP against MIAs. We can conduct MIAs on the best target models and evaluate the protection offered by DP for different values of privacy budget, noise multiple, and regularization.

Figure 2.13 – Measuring model robustness against MIAs with DP-enabled training

As illustrated in Figure 2.13, the shadow model we employed here depends on the target model’s architecture and weights. Hence, having a white-box model attack in place makes our job easy, as the same architecture and hyperparameters can be reused from the target model. The shadow model is trained on the shadow dataset to follow the target model and generate the ground truth data required to train the attack model. The shadow model’s probability output is aggregated with the true labels to generate the attack dataset, where input to the attack dataset is labeled as in or out based on the condition of whether it is used to train the shadow model.

We get approximately 50% attack accuracy since the data source of the train and test sets remains the same. However, the attack accuracy would decrease if the private training dataset for the target model were not overlapping with the public dataset that trains the shadow model. This MIA attack model has been trained using RandomForestClassifier. Attack models could be executed to perform attacks n times:

1. Our first step is to create an instance of a shadow model using the shadow dataset:

   SHADOW_DATASET_SIZE = int(shadow_X.shape[0] / 2)
   smb = ShadowModelBundle(
       shadow_model,
       shadow_dataset_size=SHADOW_DATASET_SIZE,
       num_models=)

   CopyExplain
2. In the next step, we train the shadow models with the same parameters as the target model and generate the attack data:

   attacker_X, attacker_y = smb.fit_transform(shadow_X, shadow_y.values,                                         fit_kwargs=dict(epochs=epochs,
   batch_size=batch_size,verbose=1))

   CopyExplain
3. After training the shadow models, we train the attack model using RandomForestClassifer:

   clf = RandomForestClassifier(max_depth=2)
   clf.fit(attacker_X, attacker_y

   CopyExplain
4. The next step involves evaluating the success of the attack. To do so, we segregate the data points used in the training and those that were not present during training for both independent and target variables:

   ATTACK_TEST_DATASET_SIZE = unused_X.shape[0]
   data_in = target_X_train[:ATTACK_TEST_DATASET_SIZE], target_y_train[:ATTACK_TEST_DATASET_SIZE]
   unused_X1 = unused_X.values.reshape((unused_X.shape[0],
   unused_X.shape[1], 1))
   data_out = unused_X1[:ATTACK_TEST_DATASET_SIZE], unused_y[:ATTACK_TEST_DATASET_SIZE]

   CopyExplain
5. The final step involves computing the attack accuracy of the attack test data by comparing the predicted outcomes of the attack model with the membership labels. The attack data is prepared in the expected format for AttackModelBundle:

   attack_test_data, real_membership_labels = prepare_attack_data(tm, data_in, data_out)
   attack_guesses = clf.predict(attack_test_data)
   attack_accuracy = np.mean(attack_guesses == real_membership_labels)
   print('attack accuracy: {}'.format(attack_accuracy))
   acc = accuracy_score(real_membership_labels, attack_guesses)
   print('attack acc: {}'.format(acc))

   CopyExplain

Both the models are trained with DP, with DPKerasSGDOptimizer, which is an optimizer built over SGDOptimizer for DP. noise_multiplier is a parameter supplied to the optimizer to control how much noise is sampled and added to gradients. The steps parameter represents the number of steps/epochs the optimizer takes over the training data, whereas the l2_norm_clip parameter provides a mechanism to tune the optimizer’s sensitivity to individual training points, by considering the maximum Euclidean norm of each individual gradient from the mini-batch:

optimizer = DPKerasSGDOptimizer(
    l2_norm_clip=l2_norm_clip,
    noise_multiplier=noise_multiplier,
    num_microbatches=int(microbatches_perc * batch_size),
    learning_rate=learning_rate)

6. The first step is to set the optimizer as shown in the previous code snippet. The next step is to compute a vector of per-example loss rather than its mean over a mini-batch. As demonstrated in the next code snippet, we compile the model loss with the model classifier using Keras:

   loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True, reduction=tf.compat.v2.losses.Reduction.NONE)
   classifier.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])

   CopyExplain

The computation of epsilon is given in the next code snippet. The sampling probability is computed using batch_size and the inverse of the input data size (which is the delta) and is approximately 50,000 in the input dataset. The probability metric represents the probability of an individual training point being included in a mini-batch, which is then used with the noise multiplier to evaluate rdp and finally in the computation of epsilon. This is the final model metric that can be used to judge the privacy guarantee by considering how much the probability of a particular model output can vary by including (or removing) a single training record sample. The concept of micro-batches was introduced in tensorflow_privacy (https://github.com/tensorflow/privacy) to facilitate faster processing by providing a degree of parallelism where gradients no longer remain to be clipped on a per-sample basis, but rather clipped at a micro-batch granularity.

This process can clip 32 gradients averaged over micro-batches, with each micro-batch having 8 data samples:

sampling_probability = batch_size / 50000
steps = epochs * 50000 // batch_size
orders = [1 + x / 10. for x in range(1, 100)] + list(range(12, 64))
rdp = compute_rdp(q=sampling_probability,
                  noise_multiplier=noise_multiplier,
                  steps=steps,
                  orders=orders)
epsilon = get_privacy_spent(orders, rdp, target_delta=1e-5)[0]
print("Privacy Budget Epsilon", epsilon)

Table 2.5 represents a study on epsilon, noise multipliers, and metrics from attack models. When the value of the noise multiplier increases, epsilon decreases, which means there is an increase in the privacy budget. Though most of the attack model metrics remain constant, a decrease of 1% in attack accuracy is noticed when epsilon decreases from 1.2203 to 0.3283, which reinforces the fact that higher privacy increases the robustness of the model to attacks (at least to some extent). The attack metrics also exhibit low precision and high recall, signifying the presence of more false positives (records that are identified to be coming from the training dataset, but they are not) and correct identification of the relevant record’s presence in the training dataset. However, models trained with DP are not strong enough to provide protection from MIAs, particularly when the data is correlated.

Table 2.5 – A table showing variation of epsilon and noise multiplier

Model robustness

Model robustness is a measure of the model performance taking into account indistinguishable changes in the model inputs. Different perturbation techniques help us to compare and benchmark the ML models against their robustness metrics. The Python package Foolbox (https://arxiv.org/pdf/1907.06291.pdf) helps to determine model robustness by generating adversarial perturbations. This is built on the fact that the minimal perturbation that generates an adversarial sample when applied to any model input (such as an image) helps to quantify a model’s robustness to the pre-fed adversarial samples, and demonstrates a model’s susceptibility to adversarial attacks. The flexibility provided by the framework to apply hyperparameter tuning helps to evaluate the minimal adversarial perturbation, resulting in misclassification in the predicted class probabilities in the output.

To run different attacks using this toolbox, we need an input, its label, a model, the adversarial criterion, and a distance parameter that measures the length of a perturbation, called the L1 norm. We can also mix and match using the composite model feature, where the predictions of one model can be combined with the gradient of another model, allowing us to initiate non-differentiable models by leveraging gradient-based attacks. The tool provides several criteria outlined in the following list to initiate attacks where a given input and label can be considered adversarial:

• Misclassification: Wrong predicted class at model output
• TopKMisclassification: Modification of adversarial inputs in such a way as to alter the original class so it is different from one of the top-k predicted classes
• OriginalClassProbability: Modification of adversarial inputs to alter the probability of the original class being below a specified threshold
• TargetedMisclassification: Modification of adversarial inputs to make the predicted class appear as the target class
• TargetClassProbability: Modification of adversarial inputs to increase the probability of a target class beyond a threshold value

Now let’s study, with the following code snippet, the necessary imports that can trigger an adversarial attack on a PyTorch reset model:

import torchvision.models as models
import eagerpy as ep
from foolbox import PyTorchModel, accuracy, samples
import foolbox.attacks as fa
import numpy as np

After doing the necessary imports, let’s trigger the attack as follows:

model = models.resnet18(pretrained=True).eval()
preprocessing = dict(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], axis=-3)
fmodel = PyTorchModel(model, bounds=(0, 1), preprocessing=preprocessing)
images, labels = ep.astensors(*samples(fmodel, dataset="imagenet", batchsize=16))
clean_acc = accuracy(fmodel, images, labels)
print(f"clean accuracy:  {clean_acc * 100:.1f} %")
attacks = [fa.FGSM(),fa.LinfPGD(),fa.LinfBasicIterativeAttack(),
fa.LinfAdditiveUniformNoiseAttack(), fa.LinfDeepFoolAttack()]
epsilons = [0.0,0.0005,0.001,0.0015,0.002,0.003,0.005, 0.01, 0.02, 0.03, 0.1, 0.3, 0.5, 1.0]

Here, in the preceding example, we observe the following:

• We demonstrate a few attacks including Fast Gradient Sign Method (FGSM) (added noise for perturbation is on the same side as the gradient of the cost function)
• Linf projected gradient descent (of the white-box variety with the attacker having access to the model gradient and being able to alter the code to evade ML-based detection systems)
• L-infinity basic iterative method (where adversarial samples are generated by evaluating the absolute value difference between two images, returning the maximum distance over all pixels)
• AdditiveUniformNoiseAttack and DeepFoolAttack (a fast gradient-based adversarial attack that considers the minimum distance to arrive at the class boundary by modifying the model classifier with a linear classifier) on a pre-trained ResNet model by varying the number of steps to perform the attack (as denoted by epsilon)

Then, the preceding attacks are executed on a pre-trained ResNet model by varying the number of steps to perform the attack:

for i, attack in enumerate(attacks):
    _, _, success = attack (fmodel, images, labels, epsilons=epsilons)
    assert success.shape == (len(epsilons), len(images))
    success_ = success.numpy()
    assert success_.dtype == np.bool
    attack_success[i] = success_
    print("  ", 1.0 - success_.mean(axis=-1).round(2))

The robust accuracy of the model can be evaluated as follows. This metric signifies the model’s accuracy when the attack is triggered on the best sample of the model:

robust_accuracy = 1.0 - attack_success.max(axis=0). mean(axis=-1)

The results are printed as follows:

Figure 2.14 – Variation of the robust accuracy metric with different attacks

Model robustness with constraints

Model robustness can be increased by training the model with adversarial perturbations and constraints. Adding domain constraints (just as in designing AI solutions of the network, such as intrusion detection systems) imposes extra restrictions and challenges on the adversary to maintain complex relationships between input features in order to trigger and realize an attack. Even though domain constraints limit adversarial capabilities to trigger an attack by generating perturbed samples, creating realistic (constraint-compliant) examples is often possible for adversaries. Research results suggest models gain robustness on being enforced with constraints (where the set of constrained variables is solved for optimization with a tractable linear program) that can cause the model accuracy to increase by 34%. Threat models designed with constraints properly assess realistic attack vectors and succeed in optimizing defensive performance.

One such relevant example is the defensive performance of AdvGAN- and FGSM-based mitigation (https://www.hindawi.com/journals/scn/2021/9924684/), as shown in Figure 2.15, where A has been trained without constraints and B has been trained with constraints.

Figure 2.15 – Model accuracy when trained with adversarial perturbations A) without constraints and B) with constraints

Model robustness metric

FGSM, JSMA, DeepFool, and Carlini and Wagner (CW) attacks have been useful in generating adversarial examples and triggering adversarial attacks, causing the misclassification of predicted outputs. There has been rigorous research in the field of devising metrics to evaluate model robustness by feeding in adversarial inputs to train the neural network.

This research led to suggestions for improved robustness metrics, and one metric called Cross Lipschitz Extreme Value for nEtwork Robustness (CLEVER) was proposed, which defines an approximate lower bound on the minimum distortion needed for an attack to succeed. This robustness metric is attack-agnostic (successful against powerful attacks on different types of classifiers and neural networks such as ResNet, Inceptionv3, and MobileNet). This was also found to work on continuously differentiable functions to a special class of non-differentiable functions – neural networks with ReLU activations. The CLEVER metric serves as a comparative technique for comparing different network designs and training procedures.

This algorithm at first generates N samples in a sphere around a given sample in an independent and uniform manner in each batch, out of a fixed total batch size. Then the gradient norm of each sample is computed and the maximum value of the gradient over those N samples is evaluated. The minimum value is used to determine the maximum likelihood, which is in turn used to retrieve the distributional parameters (reverse Weibull distribution) and maximizing the probability of these gradients.

The average CLEVER scores are obtained for different target classes. A high CLEVER score means networks have better network robustness, in which minimal adversarial perturbation increases the Lp norm to a higher value. This framework from IBM lays the foundation to build reliable systems without invoking specific adversarial attacks.

This CLEVER score can be used to evaluate the effectiveness of CNNs and help us to certify a neural network’s attack-resistance level. For example, in mission-critical applications (such as autonomous vehicles), the evaluation of classification robustness could increase human confidence and would serve as an important metric for compliance and ethics.

If introducing adversarial perturbation impacted the accuracy of the recognition of traffic signs and led to a speed limit being misclassified, it would have a disastrous impact on humans. Hence, it becomes mandatory to evaluate the ML model against the right robustness metric before launching the model at scale. This metric introduced by IBM considers the network architectures of CNNs, including convolutional layers, max-pooling layers, batch normalization layers, and residual blocks, as well as general activation functions, and limits the perturbation of each pixel within a threshold margin to guarantee the network classification is not changed by any external attack. IBM research further assures that this metric derived from the input and output relations of each layer generates a matrix that is efficient to compute. Figure 2.16 illustrates the trade-offs associated with model robustness (as determined by the CLEVER score) and accuracy for 18 different ImageNet models.Recent research produced a new defense framework named TRADES (http://proceedings.mlr.press/v97/zhang19p/zhang19p.pdf) that optimizes the adversarial robustness to achieve a trade-off between accuracy and robustness, providing strong resilience against both black-box and white-box attacks.

It has been found that deep neural networks when trained with regularized input gradients become more robust, and interpretable. By gradient regularization, we mean how the addition of constraints controls the change in the gradient of the input features with respect to the loss function.Hence in addition to looking for the right trade-off between model accuracy and CLEVER score, we should also look for the right trade-off between model accuracy and interpretability. We should also be aware that the accuracy or predictive power of deep learning models is high, whereas interpretability orders of linear and generalized additive models are higher. In descending orders of magnitude, we can say this is the order of interpretability of models: linear models, generalized additive models, decision trees, SVMs, random forests, and neural networks.

Figure 2.16 – Trade-off between a model’s CLEVER score (robustness) and accuracy

In this chapter, we have learned about different defense practices for mitigating attacks in different stages of data and model life cycle management. We have talked about different cloud components, techniques, and measures that can be adopted for data anonymization, deanonymization, training ML algorithms with DP, and encrypted transfer methodologies. In reference to this, we took a deep dive into adversarial risk mitigation frameworks (especially open source deep learning-based frameworks) that can be used to test the robustness of ML models before deployment and exposing the ML model to public APIs. Leveraging the use of existing frameworks and designing new ones can offer resilience against semi-honest or dishonest participants/adversaries attempting to undermine AI models, systems, and services. In addition, we have also seen how decentralized data storage, FL, and efficient cryptographic and privacy measures serve as design choices for next-generation, high-potential, privacy-enabled systems.

In the following chapters, we will explore some of the defense pipeline creation methodologies, optimization strategies, and metrics, along with their ability to deduce trade-offs between accuracy, interpretability, fairness, bias, and privacy (the privacy-utility trade-off). All these important parameters serve as prerequisites to productionizing secure, private, auditable, and objectively designed trustworthy AI systems, enabling universal acceptance by both consumers and policy-makers.

In the next chapter, we will understand the different laws and policies put in place that enforce the correct standards and best practices to build a fully ethics-compliant system.

• Data masking: what it is, how it works, types, and best practices, Cem Dilmegani: https://research.aimultiple.com/data-masking/
• A Defense Framework for Privacy Risks in Remote Machine Learning Service, Yang Bai, Yu Li, Mingchuang Xie, and Mingyu Fan: https://www.hindawi.com/journals/scn/2021/9924684/
• A Study on k-anonymity, l-diversity, and t-closeness Techniques focusing Medical Data. 17, Rajendran, Keerthana, Jayabalan, Manoj, and Rana, Muhammad Ehsan. (2017): https://www.researchgate.net/publication/322330948_A_Study_on_k-anonymity_l-diversity_and_t-closeness_Techniques_focusing_Medical_Data
• Dataprof – deterministic data masking: https://www.datprof.com/solutions/deterministic-data-masking/
• Study Of The Use Of Anonymity Models, Carmen Marcano: https://education.dellemc.com/content/dam/dell-emc/documents/en-us/2020KS_Marcano-Study_of_the_Use_of_Anonymity_Models.pdf
• Privacy Protection: p-Sensitive k-Anonymity Property. IEEE Computer Society. 2006. 94 – 94. 10.1109/ICDEW.2006.116. Truta, T. M. & Vinay, Bindu. (2006): https://www.researchgate.net/publication/4238176_Privacy_Protection_p-Sensitive_k-Anonymity_Property
• MITRE ATT&CK: https://attack.mitre.org/#
• De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks. Chen, J., Zhang, X., Zhang, R., Wang, C., & Liu, L. (2021): https://arxiv.org/pdf/2105.03592.pdf
• Designing Access with Differential Privacy, Wood, Alexandra, Micah Altman, Kobbi Nissim, and Salil Vadhan. (2020): https://admindatahandbook.mit.edu/book/v1.0/diffpriv.html
• Machine Learning with Differential Privacy in TensorFlow: http://www.cleverhans.io/privacy/2019/03/26/machine-learning-with-differential-privacy-in-tensorflow.html
• Guidelines for Anonymization & Pseudonymization: https://ispo.newschool.edu/guidelines/anonymization-pseudonymization/
• A Python Library for Secure and Explainable ML, Melis, M. Demontis, A., Pintor, M. Sotgiu, A., Biggio, B. secml: https://arxiv.org/pdf/1912.10013.pdf
• Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach, Weng, Tsui-Wei et al.: https://openreview.net/pdf?id=BkUHlMZ0b
• Foolbox: A Python toolbox to benchmark the robustness of machine learning models. Reliable ML in the Wild Workshop, 34th International Conference on ML, J. Rauber, W. Brendel, and M. Bethge: https://arxiv.org/pdf/1707.04131.pdf
• On the Robustness of Domain Constraints. CCS ‘21: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Ryan Sheatsley, Blaine Hoak, Eric Pauley, Yohan Beugin, Michael J. Weisman, and Patrick McDaniel: https://arxiv.org/pdf/2105.08619.pdf
• L-Diversity: Privacy Beyond k-Anonymity, ASHWIN MACHANAVAJJHALA DANIEL KIFER JOHANNES GEHRKE, https://www.cs.rochester.edu/u/muthuv/ldiversity-TKDD.pdf