Stable Diffusion Release, August 2022: Open AI Art

---

The Technology

The technology behind Stable Diffusion was developed, primarily, by researchers at LMU Munich and Runway. The foundational paper, “High-Resolution Image Synthesis with Latent Diffusion Models,” was written by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, and it was presented as an oral paper at CVPR 2022 (the year’s premier computer-vision conference). The paper introduced the “latent diffusion” architecture — a refinement of the diffusion model approach that made it dramatically more efficient and capable.

Diffusion models, which had been developed in the early 2020s, worked by a process of gradual denoising. A diffusion model was trained to reverse a noising process — given a noisy image, the model learned to predict the noise that had been added, and to remove it, step by step, until a clean image emerged. The approach was powerful — diffusion models could generate high-quality images, and they were more stable and easier to train than the GANs (see B27) that had dominated image generation in the late 2010s. But diffusion models were also computationally expensive. The denoising process operated in “pixel space” — it worked with the full-resolution image, pixel by pixel — and this made both training and inference slow and memory-intensive.

The result was a model that could generate high-quality images with a fraction of the compute that pixel-space diffusion required. The latent diffusion paper, presented at CVPR 2022, demonstrated the approach, and it attracted the attention of Stability AI, which offered to fund the compute for a larger, more capable version of the model.

---

Stability AI and Emad Mostaque

Stability AI was incorporated in late 2020 by Emad Mostaque. Mostaque, who was Oxford-educated (he held a bachelor’s degree in mathematics and computer science from Oxford — the Oxford “MA” he sometimes cited is the automatic post-BA status the university confers, not a separately earned degree), had worked as a hedge fund manager before turning to AI. He founded Stability AI with the stated mission of building “open” AI — AI whose models, data, and tools were freely available to the public, in contrast to the “closed” models of companies like OpenAI and Google.

Mostaque’s vision was, in some ways, ideological. He argued that the dominant AI companies — OpenAI, Google, Anthropic — were building “closed” systems that concentrated power in the hands of a few large corporations. The alternative, he argued, was “open” AI — AI whose models were freely available, that could be run, modified, and built on by anyone. The open approach, Mostaque argued, was both more democratic and more innovative — it would allow a wider range of people and organisations to participate in the AI revolution, and it would produce better, more diverse, and more accountable systems.

The vision attracted both supporters and sceptics. Supporters saw in Stability AI a counterweight to the concentrated power of the large AI labs, and they welcomed the company’s commitment to open models. Sceptics questioned whether the open approach was sustainable — whether a company could fund the enormous compute costs of training frontier models while giving the models away for free — and they noted that Mostaque’s claims about the company’s partnerships, revenue, and capabilities were, at times, exaggerated. A June 4, 2023 Forbes investigation reported that Mostaque had made misleading claims about his academic credentials, about the company’s partnerships, and about its financial position. The controversies would, in time, contribute to Mostaque’s departure from the company.

But in August 2022, the controversies were in the future. Stability AI, partnering with the CompVis group at LMU Munich and with Runway, funded the compute for the training of a large latent diffusion model. The model was trained on the LAION-5B dataset — a massive collection of 5.85 billion image-text pairs, assembled by LAION (the Large-scale Artificial Intelligence Open Network), a non-profit organisation. The dataset, scraped from the internet, contained images and text from a vast range of sources — photographs, illustrations, artworks, screenshots, and more. The training, which required substantial compute, was completed in the summer of 2022, and the model was released on August 22.

---

The Release

The Stable Diffusion release, on August 22, 2022, was, by most accounts, a watershed moment. The model — version 1.4, the first public release — was made available on Hugging Face and on GitHub, under the CreativeML Open RAIL-M license.

The license was, by the standards of AI model releases, permissive — it allowed both commercial and non-commercial use, with certain use-based restrictions (the model could not be used to generate certain kinds of harmful content, for example). The permissive license was a deliberate choice — Stability AI wanted the model to be as widely used and as widely built on as possible.

The key to the release’s impact was the model’s efficiency. Stable Diffusion, thanks to the latent diffusion architecture, could run on a consumer graphics card with as little as four gigabytes of video memory — a card that cost a few hundred dollars and that was owned by millions of gamers and creators. This was a dramatic departure from the previous state of the art. DALL-E 2, OpenAI’s image generation model, was available only through OpenAI’s API — users could not run it on their own hardware. Midjourney, the other leading image generation service, was available only through a Discord bot. Stable Diffusion was the first state-of-the-art image generation model that users could download, run on their own hardware, modify, and build on.

The release was, by most measures, a viral success. Within days, the model had been downloaded hundreds of thousands of times. Within weeks, it had been integrated into a range of applications — web interfaces, plugins for popular design tools, mobile apps. Within months, it had spawned a substantial ecosystem of derivative models, tools, and services. The open-weights approach, which Stability AI had championed, demonstrated its power: by making the model freely available, Stability AI had enabled a far wider range of innovation than a closed, API-based approach would have allowed.

The release also, however, triggered the controversies that would define the next phase of the generative AI conversation. The most immediate controversy was the artist backlash. Stable Diffusion, like all image generation models, was trained on images created by human artists — images that had been scraped from the internet, without the artists’ consent, and that had been used, without credit or compensation, to train a system that could now produce images in the artists’ styles. The artists, who had been raising these concerns since the release of DALL-E 2 and Midjourney, now had a new and more egregious target: a model that was freely available, that anyone could use, and that was explicitly built on the unauthorised use of their work. The lawsuits followed (see B80, Getty Images v. Stability AI).

---

The LAION Controversies

The LAION-5B dataset, on which Stable Diffusion was trained, also became a source of controversy. The dataset, as noted, contained 5.85 billion image-text pairs scraped from the internet. The scraping was indiscriminate — it included photographs, illustrations, artworks, screenshots, and, inevitably, content that should not have been in a training dataset.

On December 20–21, 2023, researchers at the Stanford Internet Observatory published a report identifying 1,008 externally validated instances of child sexual abuse material (CSAM) in the LAION-5B dataset (the broader suspected total was over 3,200 instances, of which 1,008 were independently confirmed). The Stanford report, covered by Bloomberg on December 20, prompted LAION to pull the dataset offline the same day. (A cleaned version, “Re-LAION-5B,” would be re-released in August 2024.) The discovery was, for the field, a significant blow — it demonstrated that the large-scale, indiscriminate scraping that had produced the training datasets for the generative AI revolution had also, inadvertently, included deeply harmful content.

The LAION controversies, taken together with the artist lawsuits, raised serious questions about the data practices of the generative AI industry. The datasets on which the models were trained were enormous, they were scraped without consent, they contained harmful content, and they were used to train models that were then deployed, at scale, to the public. The questions — about consent, about compensation, about safety, about accountability — were, in many ways, the central questions of the generative AI era, and Stable Diffusion, as the most widely used open-weights model, was at the centre of them.

---

The Later Troubles

Stability AI’s later history was, in some ways, a cautionary tale. The company, despite the success of Stable Diffusion, struggled to convert that success into a sustainable business. The open-weights approach, while it had demonstrated its power for innovation, did not, by itself, generate revenue. The company attempted to build a business around premium services — hosted APIs, custom models, enterprise offerings — but the revenue was, by most accounts, insufficient to cover the company’s substantial compute costs.

According to an October 2023 internal financial forecast reported by Forbes, the company had less than $4 million in cash remaining, with an annual compute spend forecast of approximately $99 million. The financial difficulties were compounded by executive and researcher departures. Through 2023 and 2024, a series of key researchers and executives left the company, including several of the researchers who had been central to the Stable Diffusion project. The departures were, by most accounts, a consequence of the financial pressures, of disagreements about the company’s direction, and of the broader turbulence that surrounded Stability AI.

On March 23, 2024, Emad Mostaque stepped down from his position as CEO and from the board of Stability AI, citing a desire to pursue “decentralised AI” — a loosely defined alternative to the centralised AI lab model. The resignation was, by most accounts, a consequence of the financial and organisational difficulties that the company had been experiencing. Mostaque was replaced, initially on an interim basis, and the company began a period of restructuring.

The troubles of Stability AI did not, however, diminish the impact of Stable Diffusion. The model, once released, took on a life of its own. It was forked, modified, extended, and built on by a global community of developers, researchers, and artists. The open-weights ecosystem that Stable Diffusion had seeded continued to grow, producing new models, new tools, and new applications. The company that had created the model might have struggled, but the model itself had become a permanent part of the AI landscape.

---

The Legacy

The legacy of the Stable Diffusion release is, by any measure, the legacy of the open-weights movement in generative AI. The release demonstrated, in concrete and dramatic form, that open-weights models could compete with closed models, and that the open approach could produce innovation at a scale and pace that the closed approach could not match. The ecosystem that Stable Diffusion spawned — the derivative models, the tools, the applications, the community — was, by most accounts, the most vibrant and productive part of the generative AI landscape in the years following the release.

The legacy is also, in important ways, the legacy of the controversies that the release triggered. The artist lawsuits, the LAION dataset debates, the regulatory responses — all of these were, in some ways, consequences of the Stable Diffusion release. The release made generative AI accessible to everyone, but it also made the problems of generative AI — the copyright issues, the consent issues, the safety issues — visible to everyone. The conversations that the release triggered, about the ethics of training on scraped data, about the rights of artists, about the regulation of generative AI, have shaped the field ever since.

The legacy is also, finally, a reminder of the trade-offs of openness. The open-weights approach, championed by Stability AI, had clear benefits — it democratised the technology, it enabled innovation, and it provided a counterweight to the concentrated power of the large AI labs. But it also had clear risks — it made the technology available to bad actors, it made the problems of the technology (bias, harmful content, misinformation) more difficult to contain, and it raised the stakes of the regulatory debate. The trade-off — between the benefits of openness and the risks of openness — is one that the field continues to grapple with, and the Stable Diffusion release was the moment when the trade-off became impossible to ignore.

---

---

Was stable diffusion inevitable — the product of forces too large to redirect — or was it a series of choices, each of which could have gone differently? The answer matters, because it determines whether the future is something that happens to us or something we make.