UNIFAI – UNIfied Framework for Ai Impact assessment

Introduction

Software engineering has long been a critical field of study, focusing on optimization problems and techniques for producing high-quality code. However, the advent of Artificial Intelligence (AI) has significantly altered the landscape of software development. The emphasis has shifted from traditional code-centric paradigms toward usability, user experience, and the emergence of knowledge-driven, code-agnostic development environments.

The paradox in modern AI development lies in the critical need to identify and control code quality and characteristics, despite the inherent complexity and opacity of many AI paradigms. AI systems, particularly those employing deep learning, often operate through multiple layers of abstraction, making the underlying processes and decision-making mechanisms non-transparent. This factor can obscure the quality and reliability of the code, leading to potential issues in performance, ethics, and compliance. Given this complexity, there is an “moral” obligation for designers to revisit fundamental questions about code evaluation and modeling. This involves not just understanding the technical aspects but also ensuring/evaluating that the AI systems align with ethical standards and societal values.

Limitations of Conventional AI Evaluation

The work on AI evaluation is not new; it has been a topic of research for over four decades. For instance, provides a foundational analysis on how to evaluate AI research by extracting insights and setting goals at each stage of development. While such analysis may appear trivial at first glance, it offers critical insights into the foundational drivers of a concept’s necessity and reveals the underlying value systems it implicitly adheres to. This type of analysis is often underreported or entirely absent in the documentation and discussion of AI systems. Metrics such as task-specific accuracy, precision, and false negative rate are frequently highlighted as benchmarks for model performance . However, the methodological foundations and contextual relevance of these metrics are occasionally communicated in detail. On top of that, in certain instances , additional tuning is applied not to improve the model’s inherent performance, but rather to enhance the perceived accuracy as experienced by end users—introducing an additional layer of abstraction that may obscure the system's actual behavior. In other cases, additional tuning is employed to enhance the perception of accuracy. This might involve adjusting models to better align with human expectations or to mitigate biases, thereby improving the system's perceived reliability. In summary, the development of AI systems needs a return to the moral code analysis, not only to ensure technical robustness but also to align and conform with ethical and societal expectations that are expected.

While these benchmarks serve as valuable indicators of success in controlled environments, they are insufficient for assessing the complexities and uncertainties of real-world deployments, particularly when they are not tested with real data or concepts relevant to the intended use case of the system. Similarly with the previous comparison, data represents another dimension of evaluation, especially through the leveraging of benchmark datasets. Although this approach can effectively demonstrate the accuracy of a model in comparison with known discrete concepts, it predominantly fails to reveal or adequately explore the "correctness" in real-world scenarios for decision making. Although such datasets enable the assessment of model accuracy against well-defined and discrete tasks , they frequently fall short in demonstrating—or even investigating—the model’s correctness and reliability in real-world decision-making contexts.

Societal Impact, Ethical Frameworks, and Enforcement Gaps

While performance-based evaluation methods remain prevalent, they offer a limited perspective on AI system assessment. This approach's dominance can be attributed to two primary factors: its effectiveness as a marketing strategy that emphasizes performance metrics, and its close alignment with economic considerations that typically drive enterprise decision-making. However, this narrow focus on performance may overlook other crucial aspects of AI system evaluation, such as ethical considerations, societal impact, and long-term sustainability. A growing body of literature has raised concerns regarding the widespread and accelerated deployment of AI algorithms and platforms in diverse aspects of everyday life . Numerous research efforts have proposed strategies to mitigate the associated social and economic impacts , while standardization bodies and international organizations have issued frameworks and ethical guidelines aimed at supporting responsible AI deployment. However, these initiatives are largely advisory in nature, lacking enforceability, and often fall short of establishing binding rules or regulatory mechanisms for the development and use of AI systems.

Toward a Dynamic and Holistic Evaluation Framework

To address the growing need for comprehensive algorithmic oversight, an Algorithmic Impact Assessment (AIA) tool has been developed to offer a structured and holistic framework for evaluating AI systems. The tool is organized into six main categories, each designed to be completed sequentially—beginning with the initial planning stage and concluding with the withdrawal or retirement of the AI system or platform.

This tool extends and enriches the approach proposed by CapAI , as outlined in the CapAI Internal Review Protocol (IRP). It has been designed to align with the principles and regulatory requirements of the European Union Artificial Intelligence Act (AIA), ensuring that ethical, legal, and technical robustness is embedded throughout the AI lifecycle. While CapAI provides a strong foundation for documenting and supporting goals such as risk mitigation, trustworthiness, and accountability, this new assessment tool introduces key advancements. It places greater emphasis on user-friendliness and is built to be dynamically extensible, enabling contributions and updates from a wide range of stakeholders, including conformity assessment authorities, policymakers, researchers, AI experts, and the broader community. Contemporary AI paradigms exhibit significant variation in their foundational design principles, objectives, operational workflows, and modeling techniques. Approaches such as active learning, swarm intelligence, neuro-symbolic and other emerging paradigms are not developed according to a uniform framework, nor do they pursue identical outcomes or follow standardized developmental processes. This diversity of design logic underscores the dynamic and heterogeneous nature of AI system development. As a result, any static or rigid assessment framework risks becoming obsolete over time. The emergence of novel paradigms may necessitate the introduction of new evaluation criteria or even entirely new assessment categories, in order to adequately capture the nuances and specificities of these innovations.

Alignment with a Multimodal Regulatory and Ethical Environment

Acknowledging the growing complex and multimodal regulatory context of AI, UNIFAI is meant as not only a flexible tool, but also a compliance and adaptable framework. UNIFAI seeks to align with the main regulatory instruments of the European Union: The Artificial Intelligence Act, the Data Act, and the Data Governance Act. These laws address core requirements about data use, transparency, accountability, and risk mitigation and data use across the artificial intelligence ecosystem particularly for high-risk systems. UNIFAI embeds these obligations into its framework by embedding lifecycle-based documentation, risk tiering and governance by design (for the algorithm, the model and the data). In parallel, is inspired from HUDERIA methodology (Human Dignity, Democracy, and the Rule of Law) as well as the recommendations of it, to ensure that human dignity, democracy and the rule of law are addressed throughout the AI development process. This is further supplemented by assessments such as the Human Rights AI Impact Assessment of Ontario and the BSR (Business for Social Responsibility) Framework for a Human Rights-Based Approach to Impact Assessment - both of which offer practical guidance on identifying, mitigating, and remedying harms associated with non-discrimination and equal treatment.

In addition to regional approaches, UNIFAI connects with globally recognized ethical standards and sustainability frameworks: the UNESCO Recommendation on the Ethics of Artificial Intelligence, the UN Guiding Principles on Business and Human Rights (UNGPs), and the OECD Guidelines for Multinational Enterprises on Responsible Business Conduct. Moreover adopts the IEEE Recommended Practice for Assessing the Impacts of Autonomous and Intelligent Systems on Human Well-Being (IEEE Std 7010-2020) as the recommendations are useful insights to develop ethical and socially responsible frameworks. Notably, the IEEE 7010 standard has a multidimensional series of guiding questions that rigorously interrogate the potentiality of an AI intervention unsought or intended, to undermine or otherwise remove human autonomy, agency, and well-being. Moreover, to enhance the applied value, the IEEE 7010 framework includes illustrative examples across major AI use cases including facial recognition technology, automated hiring systems, and autonomous vehicles. These examples systematically map outcomes to stakeholders impacted by the AI intervention, providing an example for constructibility of the model for evaluating against stakeholder-specific outcomes. All of which serves dual value for UNIFAI as an additional information resource for its ethical oversight functions and for evaluating across the comprehensive, stakeholder-centered evaluation of the AI system and its impacts.

Positioning UNIFAI: A Comprehensive Benchmark Against 10 FRIA Frameworks

Following the in-depth analysis carried out by Algorithm Audit - which assessed 10 current Fundamental Rights Impact Assessment (FRIA) frameworks against 12 significant requirements related to legal, organizational, technical and social dimensions - an assessment of rigor was performed in parallel with UNIFAI. As shown in the figure, UNIFAI meets all 12 requirements fully, this makes it one of the most comprehensive and adaptable frameworks currently available.

Comparative review *^{The original figure is available in the Algorithm Audit study}

With respect to the social dimension, UNIFAI is architecturally capable of embedding comprehensive social considerations via its social consideration framework. The framework offers processes of stakeholder engagement, public deliberation, and the identification of normative conflicts that may emerge throughout the lifecycle of an AI system. While UNIFAI is architecturally designed to accommodate these aspects, they are not prescriptive or required. Rather, the framework offers a modular and contextual framework for implementation, permitting those implementing UNIFAI the opportunity to adjust the level of social considerations to the use case, regulatory environment, and stakeholder environment. This architecture provides with both scalability and flexibility, to use heavy social accountability processes where appropriate and lighten the burden on systems which may not require these processes as a matter of practicality or appropriateness.

---

References :