At the height of Henry Ford’s influence, the American industrialist used the media, created exhibitions, and became the world’s largest film distributor to mold the United States into a car-loving country. Influential figures shaped societies long before Ford, but as the latest generation of elites moves to shape today’s cutting-edge technologies, a new movement seeks to put everyday people back in the driving seat.
A former researcher at Google's DeepMind AI team, Saffron Huang co-founded the Collective Intelligence Project in 2022 to direct technological development towards the collective good. With support from luminaries like former CEO of the Wikimedia Foundation, Katherine Maher, plus researchers from the world’s leading universities, AI firms, civic groups, and governments, the organisation has big plans to give society the ability to shape transformative technologies for the better.
In conversation with Leo Nasskau, she explains why collective control of transformative technologies is so critical, the types of changes society needs to make, and how our cultural narratives and incentives could help us make those changes, but currently lead us in the opposite direction.
Read below for the transcript:
Leo: Hi Saffron, thank you so much for coming on to the Culture3 podcast. I'm really excited to be diving into your vision for how we should be shaping technology and what we should be shaping it to. We live in a tumultuous and increasingly uncertain time when it comes to emerging tech – I'm curious, what sort of future do you hope we end up in?
Saffron: Thank you for having me on the show. So, I run an organisation called the Collective Intelligence Project and, before this, I was working full time in artificial intelligence research, where I was thinking a lot about the questions: what are we doing this for? What ends are we trying to enable with our increasingly capable means?
The idea of the Collective Intelligence Project, and a lot of this is in our whitepaper, is essentially asking how we can have more collective control over the direction of our frontier technologies, so that we can steer it towards collective benefit. So, there's a collective input and ownership question but also a collective benefit question.
I think the vision is: let's make sure that technologies like AI are being harnessed to benefit people broadly, rather than creating an underclass of data labellers, which would obviously be very unequal and pretty bad, but is currently the direction we seem to be heading in with AI.
I think another aspect of this is that we want there to be more avenues for people to have input into these technologies. For me and my co-founder, Divya Siddarth, the question is: how much should we be setting the vision? In some respects, we obviously need to set the vision, but we also want to build a pipeline so that more people can have a say on what they want their technology to do for them, and we can aggregate and understand those visions, and make real decisions from that basis.
In certain respects we think that real democracy has never been tried. When voters are just voting once every four or five years, and choosing a party or a specific person, you’re actually getting so little information from voters to understand and enact what they want – but technologies like AI can help us with this too. They can process information at a higher fidelity and get more nuance. So, we’re really thinking about how we can harness intelligence to better govern our technologies and benefit more people, and part of the answer is in using the technology. There's a bit of a loop there. Really, it’s about collective benefit and collective control.
Leo: It seems that orienting your work and the thinking around collective intelligence feels almost like, and I don't know if this is overstating it, but like a political statement at its core. The idea that it should be the collective making their different perspectives heard, and that that should be right at the root of how we make decisions and how we shape our technology.
Saffron: Yes, I think so. Think about how many people actually get to have input on these frontier technologies. Right now we’re talking a lot about AI, a while ago it was crypto, but there's also lots of things such as biotech, or the way that Elon Musk’s Starlink has so much control over low orbit satellites. How many people really had input on that?
In the same way, OpenAI is a reasonably small company (with 375 employees) but is becoming such a bottleneck in our technological and cognitive infrastructure. So, thinking from the ground up: is there a way for the people who build technology to better incorporate social input as they're doing this? I think this is an unsolved problem, but we’ve been throwing around some ideas. For example, what if companies who want to make transformative technologies like AI or brain-computer interfaces, have to have a few democratically-elected board seats?
How can we think about venture capital funding, which shapes what kinds of companies get built, the incentives of those companies, and the directions they're likely to go? We’re asking these kinds of questions, thinking about the entire development cycle of a technology, and how governance could be different.
Leo: I think you mentioned a really critical distinction there. We were just talking about politics and how governments can create and shape technology, but there are companies like Starlink and OpenAI that are assuming equal importance, on par with states. I think the vision you espouse of how millions and billions of people in different countries and around the world can shape technology, not only through governments, but through every aspect of society, is a really powerful one.
I'm curious as to where that vision comes from, what that world might look like, and your ideas on how that change can happen.
Saffron: We think about this in two parts. One part is asking how we better surface and aggregate information, which involves R&D on specific tools or mechanisms; ideas like quadratic funding and quadratic voting.
They come from the book, Radical Markets, which essentially asks: if people really care about a particular idea, is there a way where you can weigh that more when it comes to voting on specific things in your community? You could have a certain amount of credits and then you can allocate it across different options or different projects, and you have the option to save up all your credits because you really care about a certain thing that is happening.
This is a way of protecting minorities against tyranny from the majority, in a sense. If I really care about a particular thing I can allocate more credits to it. But, the more I add the less each additional credit is actually worth; they have a diminishing value, following a quadratic curve, in terms of the final vote count.
There are mechanisms that people are working on asking: how do we surface information that bridges different perspectives? If we're trying to make collective governance decisions on AI, and there are conflicting values, how do we figure out which ones to follow? How do we hold up the views that bridge entrenched political stances and create consensus across differences?
I think real change happens slowly and with lots of people doing their bit, but we're thinking about the research side and about piloting new ways of doing things. There’s some aspect of building the communities that want to try these things out and take them to their local policymakers, or to the various institutions and communities that they're a part of.
There's also an aspect that's a little less ‘bottom up’: not just trying out new methods within the institutions that you have, but also building new kinds of institutions that have different incentives. Ideas of redistribution, data cooperatives, or legally binding institutions that actually enact a different flow of incentives and flow of benefits. Obviously these are a little less easy to experiment with, but are also the kind of thing where, if enabled at scale, can be really powerful.
Leo: I want to talk a little bit more about quadratic voting and the principles that underlie that. What frameworks do you have for thinking about the values those futures should have? What sort of things do we need to be thinking about when considering inclusive societies and collective intelligence?
Saffron: I think this goes back to the question: what should technology be developed for? And I don't want to tell people what that should be, there should be more people with a voice in this.
There's this trilemma concept that there is a trade-off between having (technological) progress, safety, and participation. Lots of people take two out of those three things. The progress plus participation camp is more free-market and focused on accelerated innovation at the sacrifice of safety. There’s a more authoritarian camp which views these technologies as dangerous and restricts access to a few people, which sacrifices participation to gain progress and safety. Then you have a democracy-first, grassroots, community-led approach which sacrifices progress, but gains safety and participation
I think those are some potential values trade-offs, with people assuming that one is more important than another. However, I would love to create a world in which people can actually acknowledge those trade-offs, gather information in light of them, and make decisions based on that.
Leo: Is it fair to say that the bedrock for these societies are transformative technologies which are safe, create progress, and are also inclusive and participatory?
Saffron: Yes.
Leo: Why were those the three things that emerged from your research? Why did they emerge as the three that felt most important and most related to each other?
Saffron: My co-founder and I are both pretty close to a lot of AI governance discourse. I've worked in AI governance, in AI research, and she's also done a lot of tech governance while at Microsoft.
We felt like we wanted to position ourselves relative to what is actually happening and what other people are working on, to show that we need to zoom out, take a more macro perspective, and imagine solutions that can give us all three of these things.
This is a really hard problem and it's hard to not trade anything off. There's people who are working on citizen assemblies for AI, which are very large national samples of the population. For example, if you're trying to do global climate regulation, you might choose a representative sample of the entire globe, get those people together, have them deliberate for weeks about what consensus can be reached on climate, and then present those recommendations to policymakers. That is really pure in a way, and really wonderful, but it's not as scalable, and it doesn't move at the speed of technological progress.
So, what are people potentially overlooking and how can we shine a light on that and create a way for people to think differently about getting progress, participation, and safety.
Leo: Were there other (values) that you considered putting into that trilemma?
Saffron: Those have always been the main things, but we talked a lot about how to word them, because ‘progress’ means a lot of things to a lot of people, and we really just meant technological advancement. Or the term ‘participation’: are we really getting at participation or is it more about agency? People can have agency over their futures, even if they don't participate in every nitty-gritty decision, but then we thought that might get confusing, so we stuck with participation.
Leo: So, we've got this trilemma and I think it's easy to imagine examples of organisations that have fallen to one of the different corners. I remember from your whitepaper: you've got ‘capitalist acceleration’, and it's easy to imagine how that can sacrifice safety; there’s authoritarianism, and, once again, it's easy to imagine how that can go wrong (in terms of participation); and stagnation, where you sacrifice progress to make sure that everything goes how everyone wants it to (maximising safety and participation).
Are there organisations that have navigated those three aspects successfully? Are there organisations or institutions that we can take lessons from?
Saffron: That's a great question. We have successfully navigated making technology that works for people: we regulated the car industry and put seat belts in and made that mandatory; we successfully regulated planes and now giant flying metal objects can land safely, thousands of times per hour around the world. It’s quite incredible that we have been able to do that, and so I don’t think we're coming up with anything that's entirely novel here.
There's definitely lessons to be learned from how the US government regulated the car industry in the 20th century. Public-private partnerships have been useful and, with the internet, there's been a lot of voluntary standards-setting committees that have managed to make it work kind of independently of the usual nation-state infrastructure. I think it would do us well to start putting up more of these case studies publicly and putting up resources for folks. I think there have been a lot of successful cases.
Right now in today's world, I'm not sure there's an institution that I would currently point at and say they're doing an amazing job. Partly this is because the technologies that I'm concerned with, like AI and crypto, are so new; it’s always going to take time to figure things out. I do think that the UK is doing a reasonably good job of trying to be a leader in AI regulation.
Leo: Similarly on blockchain: I think the UK government has done a relatively good job, so far at least.
Saffron: They just put £100 million into a Foundation Model Taskforce, they've got ARIA; I think it could be a bit of a renaissance for UK public interest technology work, which is super cool.
Leo: I think what the transformative technology trilemma gives us is a vision or a target point for the things that we should be aiming for. What are the steps to actually get there? In the whitepaper, you talk about values and incentives, and you've spoken a little bit about this in terms of how technology gets developed and how values and incentives lead our technological progress. I wondered if you could talk a bit more about what you mean by that, and why that ordering is significant.
Saffron: From our vantage point, when we talk about digital tech as the thing that all the bright young people want to do, as the place where innovation seems to move really fast, I think there really is like an interesting culture here that I want to see more dialogue on. There's been a lot of discussion about the ‘move fast and break things’ culture which has had a bunch of downsides. I think it's tough, because there are a lot of people who are super well intentioned in tech and want to do the right things. So, harnessing that energy and steering it towards broadly beneficial things is really critical.
A lot of people in tech do really value having a good impact on the world, but maybe incentives lead them in a different direction. A lot of founders have problems with raising from venture capital because the dynamics of that funding structure asks them to return their investment in a reasonably short amount of time. Funds are usually around 10 years long, and the pressure to return (money), often well within that time frame, is pretty great
The way that folks usually get that return on investment is you either go public or you get sold to a larger company. The larger companies get larger and larger as they buy up all the smaller companies. If it goes public, then it's already going to be a big company – so, there's not a lot of space for all of the things that we need in society.
There's just a lot of weird incentives around this, and it probably means that we need a lot of different funding structures rather than just a single primary one (in venture capital). You might have particular values, but then incentives pull you in a direction. Is there a way that the incentives can be aligned with the values that you have?
We've been running public input processes into AI and when we talk to the public about how they see technology, they're worried, but they don't feel like they know enough to have an opinion. They say things like ‘I'm pretty worried, I want my grandkids to have a good future, but I don't really know what should be done’. I think there's a way in which we sell ‘tech’ as this magical, fearsome thing and that's just a cultural choice that creates certain power dynamics for the tech industry.
Leo: I think this is really interesting because there are lots of different levels in which we can see culture as having a role. We have cultures at the level of ‘move fast and break things’, which is a philosophy of how people develop technology, but then there's also the role of culture in setting a vision for how that technology can be used – i.e., we should be using AI for XYZ instead of ABC.
How important do you think both of those two levels of culture are, and where do they fit in, do you think?
Saffron: I think they're both very important. I think the culture that sets how we make technology should be serving the part of culture which sets the vision for how the tech should be used. This is a frustration that I have: we don’t talk a lot about how specific technologies could be used, we just talk about how powerful and amazing they are.
Leo: I think that often lends to those feelings of worry – where we see this powerful thing but there's no vision for how it can actually be used. It's a natural human tendency to be fretful about whether something can be used for bad outcomes.
Saffron: It's like technology for technology's sake, and I understand that impulse. It's cool to build a thing, but at the end of the day, what is the reason we're doing this? Especially if we're going to scale it and sell it to every kid – at that point, ‘this is a cool achievement’ is no longer a good enough reason.
I don't know if I fully stand behind (where I’m going with) this, but just to think about it: why are we building things? We have historically built things for a very specific need that we had at the time and we innovated towards filling that need. Now we have more of an industry that makes the thing for its own sake. Is that the case? Are we a historical anomaly in that we're building solutions without problems?
Leo: That idea of someone building something because it's cool is relevant because the concept of ‘cool’ is a subjective thing that comes from the broader culture they're involved in. We live in societies that are so different, with so many different types of people, and so many subcultures. It's very easy to get completely different perspectives on what is cool and what gets built.
It leads me to that second thing you talked about: incentives. You mentioned that the main way in which companies get built through venture funding only allows for big companies and start-ups and doesn't really sustain those middle-sized companies. I thought that was a really interesting idea, but I want to talk about incentives writ large.
What incentives should we be working towards – governments, where financing comes from, just give us an overview of the landscape, and what it should look like. Where should we be going?
Saffron: The two pillars that Collective Intelligence Project works on is an information problem – how do we process, understand, and combine people's input and values, and then there's the incentive problem, which is the one that you pointed out.
On the incentives front, there are new movements now that are trying to change things. There's Exit to Community and Zebras Unite, there's ways people are using crypto to try and crowdfund their work and give people a stake in that via tokens, and that's pretty cool. But by and large if you're not doing crypto experiments or maybe Kickstarter, then you have VC funding and normal small business loans, which are really difficult to get and so–
Leo: Of course, you might be bootstrapping your start-up, but generally it does tend to be that venture capital funding.
Saffron: People have described VC funding like jet fuel: it's not good for everything. I think culturally it’s seen as the ‘thing to do’, but you don't want jet fuel on your bicycle and you don't want jet fuel to be valorised as the only valid and important outcome. So, I think alternatives like Exit to Community and Zebras Unite, all of that work is super interesting, and I think different legal vehicles, like Perpetual Purpose Trusts and B Corps, are also necessary in terms of corporate governance.
Leo: What types of organisations do you want these new and different incentives to be pushing companies towards? What's the outcome of these different incentives?
Saffron: Being more aligned towards the collective good, essentially. A lot of people want things like this, we've talked to people building start-ups who want to be much surer that they’re building something which is actually good for people.
This is a really difficult problem, but is there's a way that people can say, ‘okay, I'm building this thing, I'm incorporating public input at the ground level, and maybe I'm using AI to reach a lot more people than I would otherwise, in terms of understanding their impact or understanding what people want.’
Leo: I'm interested in the next logical step. So, maybe with different models of funding you're able to build a company in a different way, how does that lead us towards technology which delivers for the collective? What are the steps in that logic process?
Saffron: Sometimes it's just designing bespoke institutions that, for that specific technology, get at the collective good you want. Democracy tries to do this, obviously, so it’s quite a general thing, it's not technology specific. The government is an agent of the collective and they regulate technologies to try and keep them collectively aligned.
In the case of AI, democracy is doing something, but it's kind of slow, so maybe we need folks innovating on bespoke structures that will do something specific to keep AI well aligned. For example, can we have data cooperatives where particular communities put data into a model, maybe they also train that model, and it pays them all for their work and their data? Can you get data trusts to work on a larger scale? I think folks in the UK are working on that.
We are also thinking and working on research that is specifically about what values are in a large language model and how you can have a community collectively train a large language model based on their values and preferences. It's less about incentives and much more of a pure information question that we're tackling there: how do you even enable this to happen at all? We haven't even got to the part of ‘how do you incentivise people to use this technique.’
Leo: Yes, but these are the sorts of things that you want the incentives to be working towards, right? You want to incentivise people to create these collective data sets rather than ones that are just in OpenAI's servers, for example.
You also hinted at the other things that you're working on. I want to just get the lowdown on what you're going to be working on next – what is the rest of this year looking like for you and for the Creative Intelligence Project.
Saffron: One of the biggest projects that we're currently working on is our Alignment Assemblies — it’s about how we can have public input into AI: running different Alignment Assemblies, with different communities, and feeding it into different aspects of training in AI or governing in AI. We've been doing a few pilots on that front, one in Taiwan, a few in the US, and one in the UK later this year. We are going to report on this concept and try to create a toolkit for more people to run these kinds of processes wherever they are.
One related piece of work is looking at how to get people's values: how to ask a group of people how they want AI to behave, and then actually guide the AI in that direction. We've been working on the research part of that, and we'll hopefully have something out by November.
Then there's some work with the UK government on AI looking at how we evaluate AI and how we can even know what they are doing.
Leo: When you say ‘they’, do you mean the AI models or the companies?
Saffron: It's the models, but also the companies! With the models, if you download one and try to figure out how it behaves… that's actually really difficult because the input space is basically infinite and the output space is basically infinite. If you ask simple things like, ‘hey, is this thing politically biased?’ It's actually really hard to figure out whether it is.
We’re thinking about how we harness the collective intelligence of different experts and other people who can come up with ways of evaluating the models. That's a project that we're hoping to work on in concert with the UK government because a lot of this should be public infrastructure.
Then there’s much more work to be done on the incentives piece. I’ve been thinking about it, but I’ve been really focused on the information question recently. So, hiring people and spinning up more work on how we can reimagine the term sheet, asking what are the incentives, and potentially putting out things that appeal much more to builders than necessarily to policy makers, in terms of people experimenting with new governance structures for their start-ups.
So, it's going to be a busy rest of the year.
Leo: Saffron, thank you so much for coming on. I think it's so important to get into the gritty details of how we can shape a positive future, but also the vision for the infrastructure, and how that future actually works. Thank you so much for bringing it to life.
Saffron: Thank you so much for having me. I'm pretty excited to keep putting things out and seeing where this research goes.
