Algorithmic Impact Assessment Structure - Revision history

Vrettos: /* Operation Phase */

2025-04-14T07:44:24Z

Operation Phase

← Older revision		Revision as of 07:44, 14 April 2025
Line 106:		Line 106:

	== Operation Phase ==		== Operation Phase ==
			The Operation Phase focuses on the real-world deployment and continuous oversight of AI systems. At this phase, the evaluation concentrates on mechanisms for monitoring, feedback, cybersecurity, and resilience—ensuring that systems remain ethically aligned, secure, and responsive to emerging risks throughout their lifecycle.

	Consists of two primary subthemes:		Consists of two primary subthemes:

Vrettos: /* Development Phase: Model */

2025-04-14T05:31:26Z

Development Phase: Model

← Older revision		Revision as of 05:31, 14 April 2025
Line 70:		Line 70:

	== Development Phase: Model ==		== Development Phase: Model ==
	~~Divided~~ into six ~~critical subthemes~~:		At this time in the assessment process, questions are organized into six main thematic subcategories which involve their own relatedness but each involve a unique aspect of algorithmic assessment. The structure provided for these items allows for a thorough and multidisciplinary assessment of the system being evaluated. The subcategories are:

	=== Algorithm Description ===		=== Algorithm Description ===
Line 91:		Line 91:

	== Evaluation Phase ==		== Evaluation Phase ==
	~~Questions~~ are ~~divided~~ into ~~four key subthemes:~~		The Evaluation Phase is an essential moment in the lifecycle of AI systems, frameworks, and platforms. This is where the design assumptions of the system, its functional capacity, and overall implications are scrutinized. It is more than a technical checkpoint; The Evaluation Phase is recognizably a multi-sided evaluation process that seeks to uncover the reliability, fairness, transparency, and legality of the AI system in live or contextualized conditions. This phase is methodologically designed to promote accountability and trust through the identification of risks, the testing of robustness, and the discovery of unintended consequences, before the system is fully launched into service. The method draws on interdisciplinary evaluation principles and best practices from around the world, promoting a balanced approach to quantitative testing and qualitative judgment. It should be noted that evaluating risk, performance across a diverse group of users, and evaluation of the process of traceability are emphasized as part of this phase.

	=== Testing ===		=== Testing ===
Line 115:		Line 115:

	== Retirement Phase ==		== Retirement Phase ==
	~~Focuses~~ on ~~the~~ decommissioning of the AI system~~, including~~ risk assessment, stakeholder ~~participation~~, data ~~handling~~, and environmental ~~evaluation~~ <ref>Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022). Putting AI ethics into practice: The hourglass model of organizational AI governance. ''arXiv preprint arXiv:2206.00335''.</ref><ref>Cath, C., & Jansen, F. (2021). Dutch Comfort: The limits of AI governance through municipal registers. ''arXiv preprint arXiv:2109.02944''.</ref>.		The Retiring Phase is the final stage of the AI lifecycle. It is the formal final stage for a responsible ceasing process of an AI system and any residual risks and consequences. The Retiring Phase is important to protect the engaged stakeholders and ensure transparency and accountability even if an AI system is inactive. It also focuses on decommissioning the AI system and includes risk assessment, stakeholder engagement, data usage, and environmental assessment <ref>Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022). Putting AI ethics into practice: The hourglass model of organizational AI governance. ''arXiv preprint arXiv:2206.00335''.</ref><ref>Cath, C., & Jansen, F. (2021). Dutch Comfort: The limits of AI governance through municipal registers. ''arXiv preprint arXiv:2109.02944''.</ref>.

	=== Risk Assessment ===		=== Risk Assessment ===

Vrettos at 02:15, 11 April 2025

2025-04-11T02:15:36Z

Show changes

Vrettos at 02:13, 11 April 2025

2025-04-11T02:13:08Z

← Older revision		Revision as of 02:13, 11 April 2025
Line 1:		Line 1:
	= Algorithmic Impact Assessment =		Algorithmic Impact Assessment
	This document provides a comprehensive framework for assessing algorithmic systems across their lifecycle. Its primary aim is to document and communicate key information about the AI system. The structure of this assessment supports the systematic collection of detailed, standardized information that benefits both internal stakeholders and external evaluators. The resulting document can accompany the AI system’s source code or technical documentation, serving as a transparent reference that reflects the system’s design rationale, implementation scope, and ethical considerations. In addition, the framework incorporates readiness metrics, offering visual and analytical cues to assess maturity levels across different operational categories.		This document provides a comprehensive framework for assessing algorithmic systems across their lifecycle. Its primary aim is to document and communicate key information about the AI system. The structure of this assessment supports the systematic collection of detailed, standardized information that benefits both internal stakeholders and external evaluators. The resulting document can accompany the AI system’s source code or technical documentation, serving as a transparent reference that reflects the system’s design rationale, implementation scope, and ethical considerations. In addition, the framework incorporates readiness metrics, offering visual and analytical cues to assess maturity levels across different operational categories.

Vrettos at 02:11, 11 April 2025

2025-04-11T02:11:00Z

← Older revision		Revision as of 02:11, 11 April 2025
Line 76:		Line 76:

	=== 5.2 Fairness ===		=== 5.2 Fairness ===
	Evaluates applied fairness principles, tools used, and accessibility for users with disabilities ~~[40][41][42][43]~~.		Evaluates applied fairness principles, tools used, and accessibility for users with disabilities <ref>Fairness Metrics in AI: Your Step-by-Step Guide to Equitable Systems https://shelf.io/blog/fairness-metrics-in-ai/</ref><ref>Binns, R. (2020, January). On the apparent conflict between individual and group fairness. In ''Proceedings of the 2020 conference on fairness, accountability, and transparency'' (pp. 514-524).</ref><ref>Luo, L., Nakao, Y., Chollet, M., Inakoshi, H., & Stumpf, S. (2024). EARN Fairness: Explaining, Asking, Reviewing and Negotiating Artificial Intelligence Fairness Metrics Among Stakeholders. ''arXiv preprint arXiv:2407.11442''.</ref><ref>Foulds, J. R., Islam, R., Keya, K. N., & Pan, S. (2020, April). An intersectional definition of fairness. In ''2020 IEEE 36th international conference on data engineering (ICDE)'' (pp. 1918-1921). IEEE.</ref>.

	=== 5.3 Explainability and Transparency ===		=== 5.3 Explainability and Transparency ===
	Assesses openness of code, explanation mechanisms, and user-level adaptability ~~[44]~~.		Assesses openness of code, explanation mechanisms, and user-level adaptability <ref>Foulds, J. R., Islam, R., Keya, K. N., & Pan, S. (2020, April). An intersectional definition of fairness. In ''2020 IEEE 36th international conference on data engineering (ICDE)'' (pp. 1918-1921). IEEE.</ref>.

	=== 5.4 Traceability ===		=== 5.4 Traceability ===
Line 85:		Line 85:

	=== 5.5 Robustness and Testing ===		=== 5.5 Robustness and Testing ===
	Evaluates robustness tests performed and their methodologies ~~[45][46][47]~~.		Evaluates robustness tests performed and their methodologies <ref>Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M., & Yang, J. (2025). Ai robustness: a human-centered perspective on technological challenges and opportunities. ''ACM Computing Surveys'', ''57''(6), 1-38.</ref><ref>Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1–36.</ref><ref>Chander, B., John, C., Warrier, L., & Gopalakrishnan, K. (2025). Toward trustworthy artificial intelligence (TAI) in the context of explainability and robustness. ''ACM Computing Surveys'', ''57''(6), 1-49.</ref><ref>Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., ... & Gao, W. (2023). Ai alignment: A comprehensive survey. ''arXiv preprint arXiv:2310.19852''.</ref>.

	=== 5.6 Ethical Considerations ===		=== 5.6 Ethical Considerations ===
	Assesses best/worst case scenarios, societal and environmental impacts, and risks such as autonomy infringement, physical/psychological harm, reputational damage, economic harm, human rights violations, sociocultural and political disruption. Risk dimensions are adapted from the Ada Lovelace Institute and Interpol [~~38][28][32~~].		Assesses best/worst case scenarios, societal and environmental impacts, and risks such as autonomy infringement, physical/psychological harm, reputational damage, economic harm, human rights violations, sociocultural and political disruption. Risk dimensions are adapted from the Ada Lovelace Institute and Interpol <ref>Algorithmic impact assessment: a case study in healthcare https://www.adalovelaceinstitute.org/report/algorithmic-impact-assessment-case-study-healthcare/</ref> <ref>INTERPOL and UNICRI, “Risk Assessment Questionnaire.” [https://www.interpol.int/es/content/download/20929/file/Risk%20Assesment%20Questionnaire.pdf (Revised version February 2024) https://www.interpol.int/es/content/download/20929/file/Risk%20Assesment%20Questionnaire.pdf]</ref><ref>Abercrombie, G., Benbouzid, D., Giudici, P., Golpayegani, D., Hernandez, J., Noro, P., ... & Waltersdorfer, L. (2024). A collaborative, human-centred taxonomy of ai, algorithmic, and automation harms. ''arXiv preprint arXiv:2407.01294''.</ref>.

	== 6. Evaluation Phase ==		== 6. Evaluation Phase ==
Line 100:		Line 100:

	=== 6.3 Compliance Controls ===		=== 6.3 Compliance Controls ===
	Evaluates alignment with IEEE and ISO standards, data protection impact assessments, and adherence to ethical codes and best practices ~~[48]~~.		Evaluates alignment with IEEE and ISO standards, data protection impact assessments, and adherence to ethical codes and best practices<ref>G. Mentzas, M. Fikardos, K. Lepenioti, and D. Apostolou, “Exploring the landscape of trustworthy artificial intelligence: Status and challenges,” Intelligent Decision Technologies, vol. 18, no. 2, pp. 837–854, Jun. 2024, doi: 10.3233/idt-240366.</ref>.

	=== 6.4 Certification and Audits ===		=== 6.4 Certification and Audits ===
Line 112:		Line 112:

	=== 7.2 Security and Adversarial Threats ===		=== 7.2 Security and Adversarial Threats ===
	Evaluates the complexity of the development environment, cybersecurity measures, backup systems, adversarial threat detection, and data poisoning prevention ~~[49]~~.		Evaluates the complexity of the development environment, cybersecurity measures, backup systems, adversarial threat detection, and data poisoning prevention <ref>Ministry of Digital Governance, “Cybersecurity Self-Assessment Tool for Organizations.” https://mindigital.gr/wp-content/uploads/2022/03/cybersecurity-self-assessment.xlsm</ref><ref>Cyber Security Evaluation Tool (CSET) https://www.cisa.gov/resources-tools/services/cyber-security-evaluation-tool-cset</ref><ref>Rahman, M. M., Kshetri, N., Sayeed, S. A., & Rana, M. M. (2024). AssessITS: Integrating procedural guidelines and practical evaluation metrics for organizational IT and Cybersecurity risk assessment. ''arXiv preprint arXiv:2410.01750''.</ref><ref>Xiong, W., Legrand, E., Åberg, O., & Lagerström, R. (2022). Cyber security threat modeling based on the MITRE Enterprise ATT&CK Matrix. ''Software and Systems Modeling'', ''21''(1), 157-177.</ref><ref>Ibitoye, O., Abou-Khamis, R., Shehaby, M. E., Matrawy, A., & Shafiq, M. O. (2019). The Threat of Adversarial Attacks on Machine Learning in Network Security--A Survey. ''arXiv preprint arXiv:1911.02621''.</ref><ref>Hu, Y., Kuang, W., Qin, Z., Li, K., Zhang, J., Gao, Y., ... & Li, K. (2021). Artificial intelligence security: Threats and countermeasures. ''ACM Computing Surveys (CSUR)'', ''55''(1), 1-36.</ref>.

	== 8. Retirement Phase ==		== 8. Retirement Phase ==
	Focuses on the decommissioning of the AI system, including risk assessment, stakeholder participation, data handling, and environmental evaluation ~~[50]~~.		Focuses on the decommissioning of the AI system, including risk assessment, stakeholder participation, data handling, and environmental evaluation <ref>Mäntymäki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022). Putting AI ethics into practice: The hourglass model of organizational AI governance. ''arXiv preprint arXiv:2206.00335''.</ref><ref>Cath, C., & Jansen, F. (2021). Dutch Comfort: The limits of AI governance through municipal registers. ''arXiv preprint arXiv:2109.02944''.</ref>.

	=== 8.1 Risk Assessment ===		=== 8.1 Risk Assessment ===

Vrettos at 01:33, 11 April 2025

2025-04-11T01:33:11Z

← Older revision		Revision as of 01:33, 11 April 2025
Line 3:		Line 3:

	== 1. General Information ==		== 1. General Information ==
	This initial stage focuses on gathering essential background information regarding the organization responsible for the AI system, the ~~respondent~~(s) ~~completing~~ the ~~questionnaire~~, and the AI system ~~itself~~.		This initial stage focuses on gathering essential background information regarding:

			a) the organization or individuals responsible for the design and development of the AI system,

			b) the team or person completing the assessment questionnaire,

			c) the AI system itself, including abstract information about its role, design intent, and operational context.

			The scope of this section is to clearly state the identity of the creator(s) of the system, provide essential contact details for follow-up or accountability purposes, and specify who is responsible for the information disclosed in this report. Furthermore, it aims to briefly introduce the broader AI paradigm under which the system operates and clarify the extent to which this paradigm influences the system's design and functionality.

	=== 1.1 Organization Details ===		=== 1.1 Organization Details ===
	Captures the type of organization (e.g., provider, developer, distributor, importer), its name, description, website, mission statement, and contact details.		Captures the type of organization (e.g., provider, developer, distributor, importer), its name, description, website, mission statement, and contact details.

	=== 1.2 ~~Questionnaire Respondents~~ ===		=== 1.2 Assessment Contributors ===
	Clarifies whether the ~~questionnaire~~ was completed by an individual or a team, their relationship to the system, and the sources of their information.		Clarifies whether the assessment was completed by an individual or a team, their relationship to the system, and the sources of their information.

	=== 1.3 System Details ===		=== 1.3 System Details ===
Line 15:		Line 23:

	== 2. Design Phase: Organizational Governance ==		== 2. Design Phase: Organizational Governance ==
	This phase ~~draws on~~ the ~~Cisco AI Readiness Model, assessing~~ AI strategy, infrastructure, data governance, workforce, and organizational culture.		This phase assesses the AI strategy, infrastructure, security, data governance, workforce, and organizational culture. It is intended to evaluate the overarching organizational governance framework, specifically focusing on how ethical principles are implemented in practice across the company Reference models of AI readiness—such as those developed by Cisco<ref>Cisco 2024 AI Readiness Index https://www.cisco.com/c/m/en_us/solutions/ai/readiness-index.html</ref> and comparable institutional frameworks—have been employed to structure this evaluation, ensuring alignment with industry standards and best practices.—regardless of the specific AI project in question.These elements are closely tied to the overall operational philosophy and procedural norms of the organization responsible for producing AI systems. Importantly, this assessment is not limited to the current system under review but reflects the organization's general approach to software and AI system development.

	=== 2.1 Strategy ===		=== 2.1 Strategy ===
Line 36:		Line 44:

	== 3. Design Phase: Use Case Definition ==		== 3. Design Phase: Use Case Definition ==
	~~Questions are~~ divided into three key subcategories:		The Design Phase constitutes a foundational component of the AI system lifecycle, emphasizing the systematic analysis of ethical, legal, and technical dimensions during the initial modeling process. This phase is essential for ensuring that the AI system is aligned with the organization’s core values, complies with relevant regulatory frameworks, and proactively addresses risks related to problem misdefinition or ethical oversights. Its scope encompasses the compilation of the system’s purpose, high-level objectives, and performance metrics, while also integrating governance structures and ethical principles into its conceptual design. Additionally, relevant software design patterns and development best practices may be documented to inform the system’s architectural choices and overall implementation strategy.

			The structure can be divided into three key subcategories:

	=== 3.1 Application Objectives ===		=== 3.1 Application Objectives ===
Line 45:		Line 55:

	=== 3.3 Ethical Risk Evaluation ===		=== 3.3 Ethical Risk Evaluation ===
	Using the taxonomy proposed in "A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms" ~~[38]~~, this section evaluates societal and environmental risks.		Using the taxonomy proposed in "A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms" <ref>Abercrombie, G., Benbouzid, D., Giudici, P., Golpayegani, D., Hernandez, J., Noro, P., ... & Waltersdorfer, L. (2024). A collaborative, human-centred taxonomy of ai, algorithmic, and automation harms. ''arXiv preprint arXiv:2407.01294''.</ref>, this section evaluates societal and environmental risks.

	== 4. Development Phase: Data ==		== 4. Development Phase: Data ==
	~~Questions are grouped into three subthemes:~~		The Development Phase is a critical stage in the lifecycle of AI, during which there is a focus on data acquisition, preparation, and validation such that it is kept intact and suitable for AI model training. Throughout this stage, particular emphasis is placed on ensuring compliance with ethical and legal frameworks—such as the GDPR—while also addressing critical technical challenges, including data bias, incompleteness, provenance, and representativeness. Organizations must capture the source of data sources, assess their quality with respect to attributes like accuracy, completeness, and timeliness, and present how preprocessing techniques to address missing values or class imbalance problems were applied. Organizations need to state how protected attributes were identified in order to prevent discrimination and guarantee that data does not rely on proxy variables, which can introduce biases.

	=== 4.1 Source and Documentation ===		=== 4.1 Source and Documentation ===
Line 54:		Line 64:

	=== 4.2 Privacy and Legality ===		=== 4.2 Privacy and Legality ===
	Focuses on personal data use, anonymization/pseudonymization, GDPR compliance, and Data Protection Impact Assessments ~~[39]~~.		Focuses on personal data use, anonymization/pseudonymization, GDPR compliance, and Data Protection Impact Assessments <ref>Art. 9 GDPR Processing of special categories of personal data https://gdpr-info.eu/art-9-gdpr/</ref>.

	=== 4.3 Data Quality and Bias ===		=== 4.3 Data Quality and Bias ===

Vrettos: /* Algorithmic Impact Assessment */

2025-04-11T00:16:47Z

Algorithmic Impact Assessment

← Older revision		Revision as of 00:16, 11 April 2025
Line 1:		Line 1:
	= Algorithmic Impact Assessment =		= Algorithmic Impact Assessment =
	This document provides a comprehensive framework for assessing algorithmic systems across their lifecycle. Its primary aim is to document and communicate key information about the AI system~~, including its architecture, development process, and intended purpose~~. The structure of this assessment supports the systematic collection of detailed, standardized information that benefits both internal stakeholders and external evaluators. The resulting document can accompany the AI system’s source code or technical documentation, serving as a transparent reference that reflects the system’s design rationale, implementation scope, and ethical considerations. In addition, the framework incorporates readiness metrics, offering visual and analytical cues to assess maturity levels across different operational categories.		This document provides a comprehensive framework for assessing algorithmic systems across their lifecycle. Its primary aim is to document and communicate key information about the AI system. The structure of this assessment supports the systematic collection of detailed, standardized information that benefits both internal stakeholders and external evaluators. The resulting document can accompany the AI system’s source code or technical documentation, serving as a transparent reference that reflects the system’s design rationale, implementation scope, and ethical considerations. In addition, the framework incorporates readiness metrics, offering visual and analytical cues to assess maturity levels across different operational categories.

	== 1. General Information ==		== 1. General Information ==

Vrettos: /* Algorithmic Impact Assessment */

2025-04-11T00:05:41Z

Algorithmic Impact Assessment

← Older revision		Revision as of 00:05, 11 April 2025
Line 1:		Line 1:
	= Algorithmic Impact Assessment =		= Algorithmic Impact Assessment =
			This document provides a comprehensive framework for assessing algorithmic systems across their lifecycle. Its primary aim is to document and communicate key information about the AI system, including its architecture, development process, and intended purpose. The structure of this assessment supports the systematic collection of detailed, standardized information that benefits both internal stakeholders and external evaluators. The resulting document can accompany the AI system’s source code or technical documentation, serving as a transparent reference that reflects the system’s design rationale, implementation scope, and ethical considerations. In addition, the framework incorporates readiness metrics, offering visual and analytical cues to assess maturity levels across different operational categories.

	== 1. General Information ==		== 1. General Information ==

Vrettos: /* AI Lifecycle Impact Assessment Framework */

2025-04-10T23:57:19Z

AI Lifecycle Impact Assessment Framework

← Older revision		Revision as of 23:57, 10 April 2025
Line 1:		Line 1:
	= ~~AI Lifecycle~~ Impact Assessment ~~Framework~~ =		= Algorithmic Impact Assessment =

	== 1. General Information ==		== 1. General Information ==

Vrettos at 23:54, 10 April 2025

2025-04-10T23:54:29Z

Show changes