Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/LibraryOfCongress/labs-ai-framework

Planning Framework used by LC Labs for planning AI experiments towards responsible implementation
https://github.com/LibraryOfCongress/labs-ai-framework

Last synced: 3 months ago
JSON representation

Planning Framework used by LC Labs for planning AI experiments towards responsible implementation

Host: GitHub
URL: https://github.com/LibraryOfCongress/labs-ai-framework
Owner: LibraryOfCongress
License: cc0-1.0
Created: 2023-11-03T18:23:35.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2024-04-26T16:52:45.000Z (7 months ago)
Last Synced: 2024-04-27T16:50:56.620Z (6 months ago)
Language: CSS
Homepage: https://libraryofcongress.github.io/labs-ai-framework/
Size: 1.7 MB
Stars: 20
Watchers: 11
Forks: 0
Open Issues: 1
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

awesome-machine-learning-interpretability - Library of Congress, LC Labs AI Planning Framework - ai-framework?style=social) (Community and Official Guidance Resources / Community Frameworks and Guidance)

README

---
id: ai-framework
layout: loc
title: LC Labs AI Planning Framework
permalink: /
---

# LC Labs AI Planning Framework

## About
LC Labs has been exploring how to use emerging technologies to expand the use of digital materials since our launch in 2016. We quickly saw machine learning (ML), one branch of artificial intelligence (AI), as a potential way to [provide more metadata](https://labs.loc.gov/work/experiments/experimental-access/) and [connections](https://citizen-dj.labs.loc.gov/) between collection items and users. [Experiments and research](https://blogs.loc.gov/thesignal/2023/11/why-experiment-machine-learning-at-the-library-of-congress/) have shown the risks and benefits of using AI in libraries, archives and museums (LAMs).

To account for these challenges and realities, LC Labs has been developing a planning framework to support the responsible exploration and potential adoption of AI at the Library. Read the full overview of the Labs framework in our [related post on the Signal Blog](https://blogs.loc.gov/thesignal/2023/11/introducing-the-lc-labs-artificial-intelligence-planning-framework).

At a high level, the framework includes three planning phases: 1) Understand 2) Experiment and 3) Implement, which supports the evaluation of three elements of ML: 1) Data; 2) Models; and 3) People. We rely on a set of worksheets, questionnaires, and workshops to engage stakeholders and staff and identify priorities for future exploration. The mechanisms, tools, collaborations, and artifacts together form the AI Planning Framework. Our hope in sharing the framework and associated tools in this initial version is to encourage others to try it out and to solicit additional feedback.

In planning for and conducting AI and ML experiments at the Library of Congress, we've simplified ML processes into three main elements: Data, Models, and People. The details of all three elements and how they are put together helps us understand whether an application of this technology is useful, ethical and effective.

Elements of Machine Learning / AI

Data

Library content

Data readiness

Training data

Tuning data

Validation data

Target data

Output data

Models

End-to-end workflow or pipeline

Architectures

Type of training

Libraries utilized

Frameworks or platforms

People

Develop use cases

Represented in the data

Design and sell AI systems

Impacted by AI systems

Evaluate and implement

_[Download this table as an image](/img/AIframework-1.jpg)_

Considering the data, models and people involved in an AI system is baked into our AI Planning Framework. The Understand, Experiment and Implement steps include collaborative activities and result in documentation that inform the development of practices and policies for responsible AI. We have yet to move all the way through to the implement stage, but this planning process will build the foundation for a solid and responsible AI strategy based on evidence and Library strengths.

LC Labs AI Planning Phases

Understand

Experiment

Implement

→ Governance and Policy →

_[Download this table as an image](img/AIframework-2.jpg)_

## Understand
Collaboratively articulate principles; assess risks and benefits, map needs, priorities and expertise; learn about data readiness.

Tools for use in this phase:

| Title | Description | Last Revised | Download |
| ----------- | ----------- | ----------- | ----------- |
| Use Case Risk Worksheet | This questionnaire is meant to assist staff in assessing the risk profile of an AI use case. The risk level will inform planning for the level of the risk mitigation efforts, estimated timeline for safety, quality and performance verification, and resources required. | 2023-11-15 | Link to [worksheet](https://github.com/LibraryOfCongress/labs-ai-framework/raw/main/Understand/Use_Case_Assessment_Worksheet_2023-11-15-draft.docx). |
| Phase II Risk Analysis | Fill out this worksheet to articulate success criteria, measures, risks, and benefits for an AI Use Case. | 2023-10-30 | Link to [worksheet](https://github.com/LibraryOfCongress/labs-ai-framework/raw/main/Understand/PhaseII_Risk_Assessement_-_2023-10-30-draft.docx) |
| Data Readiness Assessment | Questionnaire to assess readiness and availability of data for the proposed use case. | 2023-11-14 | Link to [questionnaire](https://github.com/LibraryOfCongress/labs-ai-framework/raw/main/Understand/Data_Assessment_Worksheet_2023-11-14.docx). |

## Experiment
We use the following tools and mechanisms for experiments:

The Digital Innovation Indefinite Delivery Indefinite Quantity (IDIQ) contract
The Data Processing Plan documents data transformations and the predicted and actual AI model performance for specific tasks. It combines elements from a model card, data cover sheet and documents curatorial provenance. Vendors are required to fill it out as part of the Digital Innovation IDIQ.
In Development: NLP vendor evaluation guide and quality review recommendations.
Under Recommendation: Balanced datasets for benchmarking newly available AI models and tools.

Test specific use cases, models and data with staff and users to document performance and build quality baselines and benchmarks

Tools for use in this phase:

| Title | Description | Last Revised | Download |
| ----------- | ----------- | ----------- | ----------- |
| Data Processing Plan | This template documents data transformations and the predicted and actual AI model performance for specific tasks. It combines elements from a model card, data cover sheet and documents curatorial provenance. Vendors are required to fill it out as part of the Digital Innovation IDIQ. | 2021-12-01 | Attachment J2 on the Library of Congress [Digital Innovation IDIQ solicitation](https://github.com/LibraryOfCongress/labs-ai-framework/raw/main/Experiment/Data-Processing-Plan-template-2021-12-01-draft.docx) |
| Digital Innovation Indefinite Delivery Indefinite Quantity (IDIQ) | The Library of Congress [Digital Innovation IDIQ](https://sam.gov/opp/1e6e3f521dd443809e6d0107b5101580/view#general) contract is a multi-year contracting mechanism that we can use to fulfill individual AI experiment at the Library of Congress, and includes requirements that may be valuable to the broader community. | 2022-07-28 | Library of Congress [Digital Innovation IDIQ solicitation](https://sam.gov/opp/1e6e3f521dd443809e6d0107b5101580/view#general) |

## Implement
AI or ML services are operational and supported by strategy, policies, integrations, shared quality standards and a skilled workforce.

This phase will include tools to assist with:
- Strategy and Roadmap
- Skills and Capacities
- Monitoring and Measuring
- Shared Quality Standards

Framework Activities

Understand

Use tools to collaborate and assess:

Risks and benefits

Principles and values

Data readiness

Local and domain expertise

Experiment

Create practices and documentation to:

Test data and models with use case

Review output with staff and users

Build baselines

Implement

Create policies and standards, including:

Strategy and roadmap

Skills and capacities

Monitoring and measuring

Shared quality standards

_[Download this table as an image](img/AIframework-3.jpg)_