Have you heard the neural aesthetic?
Neural networks and machine learning
Neural networks consist of different layers of independent ‘neurons’ where each layer is a mathematical representation of patterns in the previous layer’s data. In shallow networks, there is a single layer between the input and output layer. Deep networks have many (hidden) layers so that more complex and abstract features can be represented in these layers.
Neural networks are fed with a training dataset and an algorithm is applied to make sense of these data. When fed input data, this model is used to predict new output data, and a function is used to evaluate and improve these predictions. At the start of learning, connections between neurons are random but these become fine tuned and weighted during learning.
Machine learning solves optimisation problems
Given a large set of examples (dataset), a training process (algorithm) optimises a computational machine (neural network) to iteratively improve its performance (output) by assessing its performance against an evaluative function.
Computational creativity
In Art in the Age of Machine Learning, artist Sofian Audry distinguishes between three types of computational creativity:
Exploratory. Explore a conceptual space to create new objects in it
Combinatorial. Combine objects from different conceptual spaces
Transformational. Disrupt a conceptual space
Exploratory creativity
This approach reproduces human creativity perhaps similar to a ‘Turing Test’ for art. For example, Deep Bach uses machine learning to create music in the style of Bach. But how does this relate to artistic creativity? Does a Turing Test for art that simply reproduces human creativity miss the point?
Combinatorial creativity
A Variational AutoEncoder supports image prediction. The encoder represents key properties of the original image at the input into a ‘latent space’ (z) and the decoder converts the representations in the latent space into new images at the output
Normally, mixing two instruments creates a sound of them playing simultaneously. N-synth mixes two instruments by mixing their latent space representations to create a single hybrid. N-synth has a huge dataset of pre-made sounds that are loaded onto the X and Y axes so that we can move through the grid to explore the latent space of different hybrid possibilities.
Transformational creativity
Sofian Audry stresses that art is not an optimisation problem but rather seeks to subvert optimisation. For him, neural aesthetics relates to ‘a desire to move beyond existing aesthetic norms through an engagement with the unique materialities of machine learning … as an assemblage of three components: training process, model, data’
Algorithmic collage
By navigating through the latent space, Sofian Audry notes the possibility of ‘collages of images never meant to be seen and sounds never to be heard’. Raphik Anadol’s Machine Hallucination collected data from digital archives and publicly available resources, and the movement of his artwork captures how the network moves through the latent space.
Neural glitch
The inner structure of the model can also be manipulated; for example, by injecting noise into the neural weights, disconnecting neurons or adding new connections.
Mario Lingermann
Deep remixes
Either the content or process can be remixed by changing training data to train the model differently or apply a pre-trained model to new kinds of input data (‘style transfer’). For example, Google’s Deep Dream was trained on animal images so when shown images of clouds it interpreted them as animals.
Change the evaluation function
The parameters of the evaluation function could be changed to create new desired outcomes but this requires technical skill.
Alternatively, artist William Lantham would programme a computer to make shapes and objects so that he could then step in as ‘gardener’ to choose which forms to select and continue to breed.
Aesthetics of the learning process
Not just the final artwork but the training process could also have aesthetic potential. Artworks could allow audiences to observe the learning process to see how they might interpret it and the unfolding narrative.
The potential of audio AI artworks
Artists have been exploring machine learning mainly for visual art. Less attention seems to have been paid to sound applications beyond certain musical applications.
This is in part because the temporal nature of sounds means modelling audio is more complex than static images. High sampling rates generate large amounts of data so existing approaches have involved a trade off between fast or high quality audio synthesis.
However, recent advances are bing made in deep learning for fast and high quality real time audio synthesis, such as IRCAM’s RAVE that uses a variational autoencoder to map sounds o a trajectory through the latent space rather than to single coordinates. MAX MSP provides an interface for artists to interact with RAVE; the Max nn~ object translates between pre-trained RAVE models and MAX/MSP.