Fantastic Futures 2024 - Day 2 - Session 9

This transcript was generated by NFSA Bowerbird and may contain errors.

I would like to begin by acknowledging that we are on the lands of the Ngunnawal and Ngambri people, so that this land was never ceded. And also to acknowledge that the centre we work on is on the lands of the Gadigal people of the Eora Nation. And we also acknowledge our Indigenous colleagues and research partners whose knowledge, expertise and leadership have contributed to different parts of this work and whose research we hope to continue to support through these projects. We're going to share the stage today, and I'm going to start off by very briefly framing some of the particular challenges of working in art history and material culture research using digital tools, including AI and machine vision. And then Marnie's going to follow up with some of our work on the solutions. So yeah, suggestions. So the Power Institute, our foundation for art and visual culture where we both work, recently embarked on a multi-year program of research events and project development that we have called the Visual Understanding Initiative. Our work is focused are grouped around three key areas, critical looking, indigenous ways of seeing and visualizing ideas. And one of the goals of this work, there's quite a few goals, but one of them is to develop a suite of platforms and digital applications, research infrastructure, that can be used across the research and publication process. And our focus is on providing something that will be used by people working in art history and practice-led research, as well as, we hope, people working in cultural glam institutions and, hopefully, with visual research, visual culture research more broadly. So in my limited time, really limited time, because we're sharing the stage, I want to quickly frame two challenges. So I've just picked these to try and illustrate sort of where I guess where we're coming from in tackling issues. So as this audience is only too aware, working with AI in the cultural sector is not just a technological challenge. It's also a sort of cultural challenge to work well with collections. You need a nuanced understanding of the longer histories of collection, histories of description, reproduction, categorization. And as we've already seen over the last day and a half, all of these have an effect on how we use AI or machine vision or computational vision technologies. And they also particularly have an impact on whether these technologies will assist us to do the work we want to do effectively, whether they help us to investigate the questions important to our disciplines or our communities or our research. So data sets for the study of art are rapidly increasing. It's been a bit slower maybe than text, but it's very fast now. and becoming more accessible, though not in a consistent fashion. If you're after a data set of images from a major US or European institution, great. If you're after something from a smaller regional museum or a community, then you'll have more work to do. And that's just one type of gap. I think there's also a hesitancy from some researchers in this field around the use of AI born from the fact that we're People are very conscious of the way that the data set aligns or flattens a lot of the differences, so things like size, physical material, contextual relationships disappear when art is turned into pixels, and they're reattached often through a text description, so watercolour on paper. marble sculpture, or three meters by two meters. And this is brief, it's incomplete. Reproduced colors are reliable. Sometimes they're not reliable. Textures get flattened. Frames are emitted. So, you know, you get the picture. So I know this is sort of obvious, especially after the last day and a half, but I just wanted to articulate this as a way of framing part of the challenge we're addressing. I think the other thing particularly we've been thinking about in regards to AI and machine vision is the way that a lot of research, particularly in art history, is a bit disconnected from the data sets. We still have a tendency to publish in books and monographs, traditional forms of publishing. Outcomes are still often only in hard copy. They're behind paywalls. They can be hard to access for people. And some of them are being acquired by companies compiling machine vision models, but the process is opaque. So we're not really sure what's getting attached and what isn't. And I wanted to just show quickly this example from a paper. This is actually a paper that's a few years old. It predates the rise of the Kraken, as we've been calling it. But I think what it does show is this was a study which they were enriching the machine vision model by looking for other places where a work of art appeared on the web. But you can see that the top 10, or top 15 rather, relatively generic websites, mostly things like Pinterest and Wikipedia, whereas something like the bibliography for this particular object is much more detailed. Perhaps it's in archive.org, but perhaps not. So there's this real sense that the rich research is not always getting picked up by these models. And it's something I think, this is a survey we did in the middle of this year, that Marnie led, that sort of shows that the discipline is really interested in working with this, but they do feel that it's not very well supported. And when asked about priorities, hopefully you can read that, it was really small on my laptop and it's a bit bigger here. But people are still quite concerned about the need for access through digitization and digital collections and things like generative AI and machine learning are coming in as sort of third priorities. So there's a sense that there's still work to be done to get ready. So over to you. Thanks. Thanks for setting up the problems. It's very useful. I'd also like to acknowledge that I'm on Ngunnawal and Ngambri lands where I live and work, and also that I live and work on Gadigal land. And this is Judy Watson's barra monument for the Eora. And it's based on a shell hook, which was used by Gadigal fisherwomen for thousands of years. And it was the result of really extensive research in the Australian Museum. And mentioning that, I'm a settler of non-Aboriginal heritage. And I wanted just to put Watson's work up here in particular because, in a way, just to acknowledge the longstanding and continuing work of First Nations artists. And particularly, and Judy Watson's a really good example of this, The importance of First Nations artists with sharing knowledge of history and country through object or belongings based storytelling and also through practice based research which is what a lot of our work will deal with. So the other thing is my position is as an editor and two decades deep into print publishing. I've been part of a problem for quite some time, and really played a consistent part in those dominant structures of knowledge, which really are quite text-centric and English language-centric, particularly in our context of the Power Institute being in an academic institution. So we do partner with glam organisations quite a lot on our publications, but it's been for a long time that I've felt we needed to open our systems up, and hopefully what we're trying to do here can invite some other people in to help us and co-create that sort of structural change. So for all the cultural and epistemic diversity of our contents, this is just an example of what we publish, it's all limited by structure in the same way. It's linear, codex-centric, English language, and really written mostly from individual academic perspectives. and also structured thematically by those same academic perspectives. There's nothing wrong with academic publishing, it's got its place and it's important as well, but this structure doesn't support the diversity of knowledge holders or knowledge systems that particularly Power Institute and art and visual culture cover. So I've had this gnawing awareness of this for quite a long time, but I didn't really know how to address it. We're really small, we're a publishing house of You see it. So it was difficult to know where to start. But with digital humanities, I could see that there were ways to begin to de-center text and start to look through visualization. And also, Mitchell Whitelaw's generous interfaces was quite useful to me in terms of thinking about how he was gaining visual ways into cultural collections. So that did emerge, but at the same time, digital publishing infrastructure that was being built was kind of stubbornly codex-centric as well. So there was multimodality, but it was also still linear, text-centric, and all of those things, and ordered by academics. So I began to think about research communication as data, and to actually really lean into the relations that perhaps we could see from different viewpoints. That started to look a bit like this, which is from quite a while ago now. So we've been collaborating with Ian McRab from Systemic Solutions, which is a digital humanities lab. Ian was doing his PhD just before me at Sydney Uni, so it worked out really well. And the brief was to, could we come up with an open nodal relational kind of an infrastructure? We've since kind of jumped on from that point to have two prototypes, and we're working on a distribution. eventually for hopefully other people to play with as well. So I'll come back to the node bit, but I wanted to just touch on this image Gandara project, which is something that actually Ian and his research is central to. It's a prototype that we put together and we supported for image annotation. Some of you who came to the workshop will have seen this already. Apologies. But it's called the Glycerin Image Annotation Workbench. And it was developed by Ian and Young Lee and with the support of the ARDC. So this is a year five Buddha. And it's from the Met Museum. And this project has five different Buddhas from Peshawar Museum in Pakistan, Art Gallery of New South Wales, National Gallery of Australia, which has since been returned. and Chow Chak Wing at the University of Sydney. So we've been building on this and Katrina and I are working on with, again, Ian and Young and the team at Systemic, the idea of producing what we're calling power viewers. We'd like to have a more diverse palette of annotation tools so we can think about how The practice of annotation might be useful as a way of critical looking in education and curatorial use, but also in public presentation. And also thinking about further development, we've got a project coming up called Digital Keeping Places, which will be First Nations led with this as part of it as well. So, I mean, we're not in the space of building models and really working in the thick of AI. You can tell we're right at this end of the spectrum thinking how do we create something that's been sealed off as publishing and try to create data out of it that might be of use to lots of other people. Also, we come without a collection, so we're from a bit of a side angle here, but hopefully it gives you a way of thinking about where data maybe isn't yet, and that's where we've been coming from. Hopefully this gives a voice to some cultural belongings as well that have been removed from their communities of origin. This one not so much, but the previous. So it's really about also thinking about giving the ownership of those digital files to communities and thinking about control over access. We have, with the annotation tool, the ability to actually not see the image and only see the annotation. So we're sort of trying to work through what different communities might find useful and they might need. And secondly, we also have the, it's wonderful that there's the AAAF standard to work to as well. So it gives us the chance, so I'm moving from a position of commenting on, at the end of a research project and publishing at the very end, to sort of shifting to thinking about, okay, how do we layer in diverse vocabularies and multilingual and multimodal interpretations of things, And it means that research communication becomes part of the research process. So it's a big change for us, but I think it's a way that I think if we don't do that, we don't get the nuanced data that we need that comes from the types of researchers who work on basically individual researcher-led projects and not collection-focused necessarily. So one of the problems that Katrina brought up was also about how materiality gets removed from images and I think one of the things with the image annotations is really just about layering information but the ability to perhaps impose a preparatory sketch or to have a really high-res image of the canvas or a material in a dress or something like that. So we're thinking about layering those references back in to the images. And that's a really different way of thinking about, you know, going from illustrating in a book to something that can kind of accrue a lot of extra data and build cultural data over time. So there's a lot of diverse and changing interpretations that could be laid over the top of each other with this kind of work as well. It seems that like the visualization of data visualization, we can sort of see relational data, I hope, eventually in the conversations that happen around these annotations. And also, as they get aggregated into big lists, we can hopefully see patterns or divergent iconography. So this is really about trying to find ways to go from publishing something at the very end of something that you can't change to thinking about how the research communication process can also be about trying to model pluriversality and have different ways of thinking, seeing, doing in built within the process of coming up with content. So the other thing is, you know, it can be an internal conversation. It doesn't always have to be open. And I've focused quite a bit there on the image annotation. I'll move back to the notes now. But one thing that I find interesting and we're kind of excited about with the image annotation workbench is the ability to embed the viewer, the interactive viewer for the annotation in any web page. And we hope that institutions like many of you represent might be interested in having annotations from various community members speaking back to items in your collection. So, and it also gives us a micro and a mobile kind of form of publishing that's really different to the kind of slow, monographic kind of publishing we've been doing. So, I just wanted to sort of touch on this, this is not news to anyone, these sorts of biases that play out in collections, I'm sure, but this study of 18 major US museums highlights, I think, the difficulty involved in knowing who isn't represented in collections, and I don't think that's necessarily for us to be answering, but I think if we are, and we are from this slightly different position, going to prioritise infrastructure that makes visible, hopefully, to humans and machines, knowledge holders and knowledge systems that aren't always represented in institutions, then I think we need to start to generate cultural data as part of that process. And that needs to be about perhaps the artists who aren't exhibited, the objects that haven't been collected, archives that haven't really been studied, and the voices that maybe aren't heard quite as much. And I say that not just to, you know, have a nice sounding list, but a lot of the things we do publish sort of don't have another home somewhere else, and it's sort of a lot of practices like performance practices, things that are hard to collect. So I won't go through all of these examples in the interest of time, but I will just say that I think that we've based, so back to this is what we're calling power view, the nodal approach. It's based on the idea that by facilitating the production of knowledge in multiple modes and connecting them up via their relations, researchers and knowledge holders might be able to visualise their own diversity and complexity through the research process or perhaps co-creative processes. And if we can set them up that they might be able to generate data that's structured as they go. then we're in a better position to make that interoperable. The other thing is we're really trying to, these are the two prototypes that we've put together with the nodal system, and they're very provisional design, so don't judge necessarily as final product, but the idea here is the top project is site and space in Southeast Asia. It was a collaborative site-specific project that went over three years, funded by Getty's Connecting Art Histories, And the relations that formed within that research project were between collaborative groups of researchers from different disciplines and the spaces and the sites. So the nodes there are connected to space. This one is Wool Manifesto. They're a women's art collective. And they're a group that's really been around since the 90s, grown in all sorts of ways across many, many countries now. And they've adapted, they've never had a kind of institutional home. They make up their outputs as they go. Sometimes it's exhibitions, sometimes it's digital projects or creating books. And the idea there was that we were really trying to use the nodal approach to represent their own network. So it's people, events, it's also a digitised archive in there. So hopefully, and we're hoping to build out different interface types. The idea being that once we've got the nodes in the back end, they get prepared. you can push them out into different interface types and see your research in different ways. So it's also about removing hopefully some hierarchies in terms of the mode of delivery from the beginning between languages and knowledge holders. We hope the system might also help collaborators to influence each other's ways of thinking and knowing and being. And that's really It's funny that we're at this particular venue, because I've been calling this a generative model of publishing, but with no reference to AI whatsoever. So it's given me a lot of ways to sort of rethink that metaphor. But the idea being that if we're doing this in a way that we want to, it should be changing the communication structures as we go along. So I think I'll get towards the end now. Really, for both projects, the aim is to keep all the data in the back end, not sealed off project to project, like you would with an ISBN. This is that book, and that's this one. But hopefully, where appropriate, we might be able to build up gradually a data set or various data sets over time, and also to be able to visualize what we're doing. So just to conclude, I will... There were sort of four points we were trying to cover, so I'll just cover off on them. The visual and material complexities of this diverse cultural data that we deal with poses many challenges to AI and machine vision, we know all of that, but they're really based on also these dominant forms of imagery and distance views, which I find reflected in the book form as well. There's a lot of individuals we work with who are working really up close with objects and histories. And we think we can have something to contribute perhaps in situated small, slow data and data with nuance. And as our sort of organisation, we're really keen to see what we can do to partner with bigger organisations and with collections to try to bring that approach. And we also hope that this nodal and multimodal CMS that hopefully will be useful to researchers who don't have a lot of research infrastructure or funding for themselves, will help them to build a sense of their own relations and see the complexity and the value of the complexity in humanities research. So finally, I think we anticipate that the outputs will hopefully expand out from writing for particular traditional audiences, and maybe move a bit into data production and intersecting with knowledge networks like the ones that you guys are creating. So thank you, and love to hear what you all think. And thanks for having us.