Documenting AI Models

I don’t (currently) work on any AI related courses, but this strikes me as something that could be easily co-opted to support: a) best practice; b) assessment — the use of model cards. Here is where I first spotted a mention of them:

My first thought was whether this could be a “new thing” for including in bibliographic/reference data (eg when citing a module model, you’d ideally cite its model card).

Looking at the model card for the Whisper model, I then started to wonder whether this would also be a Good Thing to teach students to create to describe their model, particularly as the format also looks like the sort of thing you could easily assess: the Whisper model card, for example, includes the following headings:

• Model Details
• Release Date
• Model Type
• Paper / Samples
• Model Use
• Training Data
• Performance and Limitations
• Broader Implications

The broader implications is an interesting one..

It also struck me that the model card might also provide a useful cover sheet for a data investigation.

Via Jo Walsh/@ultrazool, I was also tipped off to the “documentation” describing the model card approach: https://huggingface.co/docs/hub/models-cards. The blurb sugges:

The model card should describe:

• the model
• its intended uses & potential limitations, including biases and ethical considerations
• the training params and experimental info (you can embed or link to an experiment tracking platform for reference)
• which datasets were used to train your model
• your evaluation results

Hugging Face: model cards, https://huggingface.co/docs/hub/models-cards

The model card format is more completely described in Model Cards for Model Reporting, Margaret Mitchell et al., https://arxiv.org/abs/1810.03993 .

A largely similarly structure card might also be something that could usefully act as a cover sheet / “executive report metadata” for a data investigation?

PS also via Jo, City of Helsinki AI Register, ” a window into the artificial intelligence systems used by the City of Helsinki. Through the register, you can get acquainted with the quick overviews of the city’s artificial intelligence systems or examine their more detailed information”. For more info on that idea, see their Public AI Registers white paper (pdf). And if that sort of thing interests you, you should probably also read Dan McQuillan’s Resisting AI, whose comment on the Helsinki register was “transparency theatre”…

PPS for an example of using Whisper to transcribe, and translate, an audio file, see @electricarchaeo’s Whisper, from OpenAI, for Transcribing Audio.