In this talk, we will describe TrinityAI - a neuro-symbolic artificial intelligence (AI) architecture being developed for detecting complex events across varied environments. Complex events are scenarios composed of multiple atomic events that occur over different times and locations, involving various actors. TrinityAI aims to enhance monitoring and situational awareness in critical applications such as healthcare, surveillance for public security, and traffic monitoring. The desiderata of AI needed to detect such complex events include the following: first, we need multimodal perception in very high dimension; second, we need to reason to compose atomic events and infer complex event; and finally, we need the capability to ingest background knowledge from human experts, and provide symbolic justification of AI decisions back to the human users. TrinityAI satisfies these requirements and its architecture is motivated by predictive processing - a theory of mind which posits that human cognition is not bottom-up wherein we attach meanings to our perceptions in isolation from each other but instead we build models or narratives of the world and use that to interpret our sensed perceptions. This model-based perception is key to the robustness and context-sensitivity of our sensing and decision-making. TrinityAI has a three-layered architecture composing deep neural networks, foundation models, and logic programming with each layer designated to handle specific tasks ranging from raw data perception to high-level reasoning and inference.

As machine learning (ML) models are integrated more and more into critical systems, questions regarding their ethical use intensify. This paper advocates for a loss-driven engineering approach, incorporating concepts from systems-theoretic process analysis (STPA), to identify external systems, technologies, and processes essential for ethical ML deployment and therefore crticial to assessing AI ethics. STPA facilitates a deep analysis of potential hazards and system-level vulnerabilities, generating actionable insights for designing support systems and safeguards. Resilience engineering principles can be utilized to convert these insights into testable requirements for assessing AI ethics. This innovative, multi-disciplinary approach addresses a critical gap in current ML practices by extending ethical evaluation beyond the model, offering a robust framework for the responsible development and deployment of AI technologies.

There has been great attention paid to the risks with the proliferation of Artificial Intelligence across both commercial and defense sectors. While the National Security Commission on Artificial Intelligence report highlighted the need to “establish justified confidence in AI systems,” it did acknowledge that “the United States will field safe and reliable AI…” through the execution of existing procedures. This acknowledgment reinforces the prevailing thought from Combat Capability Development Command Armaments Center that there are robust processes in place, such as the Materiel Release (MR) process. The authors spent years engaged with stakeholders, board members, coordinator, and chair to collect the prevailing concerns, distill them into risks, and identify mitigations that would enable the board to properly and thoroughly review an AI enabled system. This paper will outline the findings of this analysis, where the MR Process may be altered or tailored to account for AI, and suggestions for stakeholders to reduce risk. The culmination of this work is a roadmap to ensure not just responsible, but safe, suitable, and supportable systems continue to be fielded by the US Army.

Deep Neural Networks (DNNs) do not inherently compute or exhibit empirically-justified task confidence. In mission critical applications, it is important to both understand associated DNN reasoning and its supporting evidence. In this paper, we propose a novel Bayesian approach to extract explanations, justifications, and uncertainty estimates from DNNs. Our approach is efficient both in terms of memory and computation, and can be applied to any black box DNN without any retraining, including applications to anomaly detection and out-of-distribution detection tasks. We validate our approach on the CIFAR-10 dataset, and show that it can significantly improve the interpretability and reliability of DNNs.

As artificial intelligence (AI) becomes a prevalent requirement in military systems, assuring the security and reliability of AI-enabled technologies becomes paramount. This paper explores assurance and security mechanisms imperative for AI-enabled military systems, laying specific emphasis on what guidelines and documentation have been established to date as well as understanding and characterizing bias, explainability, transparency, and trust to help remove the mystery behind AI systems. AI introduces transformative capabilities to military operations; however, its deployment is often hindered by challenges regarding security, assurance, and ethical concerns. Secure and reliable AI systems are crucial in a military context, where decision-making needs to be rapid, accurate, and trustworthy. Addressing bias, ensuring explainability, fostering transparency, and understanding trust are four important factors crucial for the successful integration and acceptance of AI within military applications. Bias in AI systems can inadvertently lead to unfair or unethical outcomes, potentially endangering lives, and compromising missions. Understanding and characterizing bias in AI-enabled military systems is crucial for developing fair and impartial algorithms. The paper delves into methodologies and frameworks that assist in identifying, measuring, and mitigating bias, thereby promoting the development and deployment of ethical and fair AI in military operations. Explainability in AI refers to the ability to describe the internal mechanisms of a system or the relationships between input features and predictions in an understandable manner to humans. In the context of military applications, explainability is not merely desirable but often legally and ethically required. The paper explores techniques and approaches to enhance the explainability of AI systems, thereby instilling confidence in their use and facilitating their acceptance among military personnel and policymakers alike. Transparency, the third pillar, involves making the AI's decision-making processes and data handling practices clear to the end-users and stakeholders. In military settings, where accountability and trust are foundational. Transparency in AI systems is non-negotiable. The paper sheds light on mechanisms that foster transparency and trust, providing insights into how these can be integrated into the design and deployment phases of AI-enabled military systems, ensuring that they align with legal standards and ethical norms. Finally, the paper highlights the measures that governments have put in place to maintain ethical use of AI both in the commercial and military realm. The importance of creating an ecosystem where assurance and security are not afterthoughts but are integrated into the entire life cycle of AI-enabled military systems is emphasized. From the design and development phases through deployment and operational stages, every step needs to be imbued with practices and checks that ensure that the systems are unbiased, explainable, transparent, and trustworthy.

Artificial Intelligence (AI) has shown a tremendous impact on civilian and military applications. However, traditional AI will remain inadequate because of issues such as explicit and implicit biases and explainability for operating independently in dynamic and complex environments for the foreseeable future. For human-autonomy teaming (HAT), trustworthy AI is crucial since machines/autonomy and humans work in collaboration for shared learning and joint reasoning for a given mission with combat speed and high accuracy, trust, and assurance. In this paper, we present a brief survey of recent advances, some key challenges, and future research directions for neuro-symbolic-reinforcement-learning-enabled trustworthy HAT.

With resource constraints being a reality for many organizations, effective security for Artificial Intelligence (AI) systems has become a critical focus. To address this challenge, we introduce the AI Security Pyramid of Pain, a framework that adapts the cybersecurity Pyramid of Pain to categorize and prioritize AI-specific threats. This framework provides a structured approach to understanding and addressing various levels of AI threats. Starting at the base, the pyramid emphasizes Data Integrity, which is essential for the accuracy and reliability of datasets and AI models, including their weights and parameters. Ensuring data integrity is crucial, as it underpins the effectiveness of all AI-driven decisions and operations. The next level, AI System Performance, focuses on MLOps-driven metrics such as model drift, accuracy, and false positive rates. These metrics are crucial for detecting potential security breaches, allowing for early intervention and maintenance of AI system integrity. Advancing further, the pyramid addresses the threat posed by Adversarial Tools, identifying and neutralizing tools used by adversaries to target AI systems. This layer is key to staying ahead of evolving attack methodologies. At the Adversarial Input layer, the framework addresses the detection and mitigation of inputs designed to deceive or exploit AI models. This includes techniques like adversarial patterns and prompt injection attacks, which are increasingly used in sophisticated attacks on AI systems. Data Provenance is the next critical layer, ensuring the authenticity and lineage of data and models. This layer is pivotal in preventing the use of compromised or biased data in AI systems. At the apex is the tactics, techniques, and procedures (TTPs) layer, dealing with the most complex and challenging aspects of AI security. This involves a deep understanding and strategic approach to counter advanced AI-targeted attacks, requiring comprehensive knowledge and planning. This paper provides an overview of each layer, discussing their significance and interplay within AI system security. The AI Security Pyramid of Pain offers a strategic approach for organizations to navigate and mitigate the multifaceted threats in AI security effectively.

Representing context, reasoning within contexts, and providing quantitative assessments of machine learning (ML) model certainty are all tasks of fundamental importance for secure, interpretable, and reliable model development. Recent enthusiasm regarding generative ML models has highlighted the importance of representing context, which is contingent on relevant and contextual features of data and model predictions are unreliable on out-of-context inputs. Herein, we develop the theory of graph representation learning (GRL) to extend to Bayesian Graph Neural Networks and to incorporate various forms of uncertainty quantification to improve model development and application in the presence of adversarial attacks. Within this framework, we approach the challenge of adversarial patch detection using a synthesized dataset consisting of images from the APRICOT and COCO datasets to study various binary classification models for patch detection. We present GRL models with two layers of edge convolution that are capable of detecting patches with up to 93.5% accuracy. Further, we find evidence supporting the use of the certainty and competence framework for model predictions as a tool for detecting patches, particularly when the former is included as a model feature in graph neural networks.

As artificial intelligence (AI) continues to develop rapidly and influence numerous applications affecting billions of lives, it is crucial to form AI red teams whose objective is to identify AI-enabled system vulnerabilities before deployment to reduce likelihood or severity of real-world security risks. In response, we present a playbook to establish a formalized and repeatable process for AI red teaming. By describing the process as part of a larger framework known as Build-Attack-Defend (BAD), we define a collaborative process between the AI-enabled system development and security teams, as well as various stakeholders. Complementing An AI Blue Team Playbook, this paper contains the red teaming historical context, process, and lessons learned, serving as a starting point for proactively identifying weaknesses, enhancing the overall performance, security, and resilience of AI-enabled systems.

Recent progress in computer vision has been fueled by advances in artificial intelligence (AI) using techniques such as deep learning. However, computer vision models are often sensitive to distribution shifts, such as those naturally occurring in real operational data, which can degrade performance and cause unforeseen failure modes. As such, it is critical to perform comprehensive test and evaluation of these models across a wide range of natural effects in order to assure their downstream use and deployment. However, it is often difficult, impractical or simply impossible to collect and annotate datasets incorporating the expected range of conditions in operational domains. Here, we introduce the open source natural robustness toolkit (NRTK), a platform for evaluating the natural robustness of computer vision models through realistic augmentations of datasets. The NRTK package provides a set of validated scene- and sensor-specific perturbations using physics-based modeling of the entire image formation process, covering pre-sensor, at-sensor, and post-sensor perturbations. We demonstrate its modular and flexible nature through the example use cases of evaluating object detection models on satellite and unmanned aerial vehicle (UAV) imagery, where critical sensor parameters such as focal length and aperture diameter can be varied to assess model performance. We also show how the NRTK package can be paired with visualization tools to support the interactive exploration of datasets and their perturbations via NRTK. Our results suggest that natural robustness is a critical dimension for evaluation towards the verification and validation of computer vision models, ensuring their trusted use and deployment. Our code is publicly available at: https://github.com/Kitware/nrtk.

Warfighters will distrust AI-enabled systems that perform complex and high stakes work but do not effectively interact with humans. In many cases, this distrust in systems lacking effective Human-Machine Teaming (HMT) will be warranted, as poor teaming is often hazardous. Assurance is needed throughout development and fielding of AI-enabled systems to provide justified confidence that relational aspects of these systems are safe and effective. Particularly, developmental testing of HMT is critical because it enables HMT risks to be mitigated through AI-enabled system design. We present practical guidance for assessing HMT during developmental testing, including techniques, tools, and approaches that have the potential for immediate impact across the DoD Test and Evaluation (T&E) community. To further the adoption of HMT assessment in AI-enabled system assurance, we discuss DoD T&E domain-specific considerations which may support the development of guidance as well as future research.

Deep distributed neural networks (DDNNs) use partitioning and data compression to perform neural network inference under the tight resource constraints of edge computing systems. Existing DDNN applications focus on efficient execution without accounting for how these features impact data privacy. In this work, we develop a side-channel attack that exploits the use of compressed communication in DDNN systems. We demonstrate how the size of compressed messages provides information about the DDNN’s results, even when the system uses data encryption. We demonstrate this side-channel by developing a probabilistic attack that uses message sizes to infer the DDNN’s results with over 2.3× the accuracy of random guessing. In the worst case, our attack discovers over 90% of the DDNN’s outputs. We mitigate this side-channel through a novel defense called dropout stable compression (DRSC), which guarantees fixed-length messages for DDNNs. DRSC acts as a wrapper around lossless compression and delivers an overall compression ratio equal to that of the underlying lossless method. To achieve this behavior, DRSC controls the compressed size through Dropout.1 DRSC limits the impact of Dropout on the DDNN’s accuracy by dropping values with the smallest magnitude. This design allows DRSC to eliminate the discovered side-channel while displaying negligible overhead and an inference accuracy within 0.3 percentage points (i.e., 0.3%) of existing systems. With this behavior, DRSC enables resource constrained systems to gain the benefits of DDNNs without suffering from the privacy issues stemming from data compression.

Explainable Reinforcement Learning (XRL) techniques help to provide insight into the decision-making process of a Deep Reinforcement Learning (DRL) model and can be used to identify vulnerabilities and weaknesses within the policy prior to deployment, but these techniques also expose critical policy information and associated vulnerabilities that adversaries can exploit to develop more effective and efficient adversarial attacks against the trained policy. This paper introduces the ARLIN (Assured Reinforcement Learning Model Interrogation) Toolkit, an open-source Python library that identifies potential vulnerabilities and critical points within trained DRL models through detailed, human-interpretable explainability outputs. To illustrate ARLIN’s effectiveness, we provide explainability visualizations and vulnerability analysis for a publicly available DRL model and demonstrate how the outputs generated by ARLIN can be exploited to successfully reduce the overall performance of the model while limiting overall detectability. The open-source code repository is available for download at https://github.com/mitre/arlin.

This is far more than just prompt injection and LLM jailbreaks. Apply a systematic lifecycle perspective to frame adversarial testing requirements and tactics for artificial intelligence (AI) systems. In this talk, we’ll explore why red teaming AI-enabled systems has some nuance that might not be covered during traditional testing and evaluation. We’ll discuss how to build AI red teaming capabilities, measure their performance and effectiveness, and explore the future of adversarial testing.

Significant advancements have been achieved in radio frequency (RF) machine learning (ML), yet much of the existing research remains confined to proof-of-concept demonstrations or narrowly defined scenarios that do not fully encompass real-world operational conditions. As machine learning algorithms increasingly assume decision-making roles in this domain, ensuring the reliability and validity of their outcomes becomes imperative. This paper seeks to expand the discourse on ML assurance within the RF ML domain. Building upon the foundation laid by the RiftNet deep learning classifier, this study investigates the robustness of this solution in the context of instilling confidence in RFML applications. Specifically, the paper delves into the challenges posed by inaccurately labeled data during artificial intelligence agent development, the quantification of outcome uncertainties, and the implementation of calibration techniques for RF modulation recognition. In essence, this research marks a pivotal shift from the initial successes of RF ML towards the development of techniques that inspire confidence in real-world, edge-system applications.

It is a major challenge to maintain Differential Privacy (DP) when fine-tuning a Large Language Model (LLM) while also preserving the increased functionality that make fine-tuned LLMs appealing. In this paper, we explore the utilization of an LLM that has been modified for the purpose of encoded message transmission using text output as a medium, a task which falls under the classification of Linguistic Steganography. By examining the impact that DP preserving fine-tuning has on an LLM intended for such a specific and technical functionality, we evaluate what performance cost is imparted. Our experimentation focuses on using a modified implementation of Differentially Private Stochastic Gradient Descent, while fine-tuning a LLM on curated data taken from the ConvoKit Reddit dataset. We were able to securely fine-tune the LLM while maintaining a relatively strict DP privacy budget, and still benefit from the domain specific increased performance that LLM fine-tuning provides.

CDAO will share their lessons learned from testing critical AI systems, such as the measurement of data drift, testing of AI integration with legacy systems, and continuous testing. They will also provide with their insights on AI test and evaluation moving forward, including avenues for industry and DoD collaboration.

Artificial Intelligence (AI) and Machine Learning (ML) based systems have seen tremendous progress in the past years. This unprecedent growth has also opened new challenges and vulnerabilities for keeping AI/ML based systems safe and secure. With a multitude of studies investigating adversarial machine learning (AML) and cyber security for AI/ML systems, there is a need for novel techniques and methodologies for securing these systems. Cyber security is often used as a blanket term meaning all defenses used in the context of cyber. This leaves out methodologies and techniques, being used more offensively, such as cyber deception. This study provides a comprehensive overview of cyber-deception for securing AI/ML systems including its relevance, effectiveness, and its potential for AI/ML assurance. The study provides an overview of behavioral sciences for cyber-deception, the benefits of using cyber deception, and the ethical concerns associated with cyber deception. Additionally, we present a use-case for the utilization of cyber deception with zero-trust architecture (ZTA) for assurance and security for AI/ML based systems.

We present a cognitively-inspired salience technique that utilizes our knowledge of human information processing mechanisms to determine the extent to which each feature is responsible for making a given decision in a given context. While measures of model-based feature importance are prevalent in ML packages (e.g., the fitted attribute feature importances and permutation importance function in the sklearn package), there are fewer off-the-shelf techniques for class-based and decision-based importance. Our measures of cognitive salience result from deriving the output of the cognitive mechanisms with respect to specific features or instances in the current situation. Applying this technique to intrusion detection datasets (e.g., UNSW-NB15, CICIDS2017) we can compute the salience of each feature for any given classification, including different feature salience for correct vs incorrect decisions. By tracing (i.e., clamping) our model’s decisions to that of a specific classification technique or individual decision maker, we are able to provide measures of model-agnostic salience. While extant feature importance techniques (e.g., selectKbest, random forest) focus on which features improve overall model performance, cognitive salience instead considers how much a given feature influences the current decision. This allows for more fine-grained analyses of model performance and has been used previously to introspect over the decisions of deep reinforcement learning algorithms and explain why RL agents failed at a StarCraft 2 task or which strategies they discovered in grid world multi-agent adversarial tasks, as well as introspecting over features in an insider attack scenario to identify how (and when) humans learn the relative value of each feature. We report a comparison between cognitive salience and extant ML-based techniques.

Outsourcing computational tasks to the cloud offers numerous advantages, such as availability, scalability, and elasticity. These advantages are evident when outsourcing resource-demanding Machine Learning (ML) applications. However, cloud computing presents security challenges. For instance, allocating Virtual Machines (VMs) with varying security levels onto commonly shared servers creates cybersecurity and privacy risks. Researchers proposed several cryptographic methods to protect privacy, such as Multi-party Computation (MPC). Attackers unfortunately can still gain unauthorized access to users’ data if they successfully compromise a specific number of the participating MPC nodes. Cloud Service Providers (CSPs) can mitigate the risk of such attacks by distributing the MPC protocol over VMs allocated to separate physical servers (i.e., hypervisors). On the other hand, underutilizing cloud servers increases operational and resource costs, and worsens the overhead of MPC protocols. In this ongoing work, we address the security, communication and computation overheads, and performance limitations of MPC. We model this multi-objective optimization problem using several approaches, including but not limited to, zero-sum and non-zero-sum games. For example, we investigate Nash Equilibrium (NE) allocation strategies that reduce potential security risks, while minimizing response time and performance overhead, and/or maximizing resource usage.

As application workloads transition to artificial intelligence (AI) based workloads, there is a greater need for more secured neural network runtimes and developer tools to increase operational cybersecurity effectiveness of the deployed software. Additionally, as more systems deploy, they are more likely to be captured and reversed engineered by external bad actors, making AI model security a critical aspect in the Machine Learning Operations (MLOps) pipeline. In this paper, we describe our approach to simultaneously optimize and secure AI runtimes, in order to address both agility and security of the AI system simultaneously. A more capable neural network runtime enables a greater ability to detect, assess, and mitigate malicious attacks on the deployed AI-based system. We propose an open-standard application programming interface (API), with security features such as encryption and watermarking intimately integrated into the model runtime.

The Cross-Industry Standard Process for the development of Machine Learning applications with Quality assurance (CRISP-ML(Q)) framework describes the full AI model lifecycle from data sourcing to deployment, along with risk mitigation procedures for each phase of development. In this work we implement an example model pipeline that utilizes CRISP-ML(Q), along with mitigations for risks documented in MITRE ATLAS and the National Institute of Standards and Technology Artificial Intelligence Risk Management Framework (NIST AI RMF) and its associated playbook, with the goal of creating a generalizable reference framework for model developers and decisionmakers. This work builds on our previous effort in cross-referencing CRISP-ML(Q) with published ML risk frameworks by developing a working and reproducible pipeline for image classification that validates our risk mitigation approach. In this pipeline, we implement multiple risk mitigations and demonstrate their effectiveness in the areas of drift detection, spam mitigation, and defense against adversarial attacks, along with demonstrations of model explainability to improve model trustworthiness. We present results from a variety of A/B mitigation tests that show the effectiveness of our mitigations, and we demonstrate full pipeline runs for Convolutional Neural Network (CNN) and Resnet18 model architectures that utilize our risk mitigation process.

In a fiercely competitive landscape, we are deploying AI systems faster than they can be security tested and defended. With developers under pressure to deliver on functionality and performance as quickly as possible, security is too often left as an afterthought. In response to emerging security challenges, we present a playbook to establish an AI blue teaming process for mitigating vulnerabilities before they can be exploited in the wild. By describing the process as part of a larger framework known as Build-Attack-Defend (BAD), we define an iterative and collaborative process between the AI system development and security teams, as well as various stakeholders. Our playbook contains the blue teaming historical context, process, lessons learned and hypothetical examples, serving as a starting point for embedding security at the heart of AI-enabled systems.

Susceptibility to adversarial attacks is an issue that plagues many deep neural networks. One method to protect against these attacks is adversarial training (AT), which injects adversarially modified examples into training data, in order to achieve adversarial robustness. By exposing the model to malignant data during training, the model learns to not get fooled by them during inference time. Although, AT is accepted to be the de facto defense against adversarial attacks, questions still remain when using it for practical applications. In this work, we address some of these questions: What ratio of original-to-adversarial in the training set is needed to make them effective?, Does model robustness from one type of AT generalize to another attack?, and Does the AT data ratio and generalization vary depending on model complexity? We attempt to answer these questions using carefully crafted experiments using CIFAR10 dataset and ResNet models with varying complexity.

Federated Learning (FL) has emerged as a prominent branch of Machine Learning due to the increasing prevalence of mobile computing and IoT technologies. Unlike centralized systems, in FL the devices often operate beyond the confines of centralized protection mechanisms. Consequently, the adoption of this methodology gives rise to various security concerns, including data leakage, communication vulnerabilities, and poisoning. In this paper we propose new distance-statistical aggregation algorithms that provide robustness against Byzantine failures, and we compare them with the well-known FedAvg on a set of simulations that recreate realistic scenarios. Achieved results demonstrate the functionality of the solutions in terms of efficiency and accuracy.