​​We just need to get a few dozen people in a room (key government officials from China and the USA) to agree that a race to build something that could create superebola and kill everybody is a bad idea. 

We can do this. 

We’ve done much harder things.

California’s AI safety bill does not require kill switches for open source models.

People who are saying it does are either being misled or the ones doing the misleading.

AIs under the control of the developer need a kill switch.

Open source AIs are not under the control of the developers, so do not need a kill switch.

Many of the people who are spreading the idea that it will kill open source know this and are spreading it anyways because they know that “open source” is an applause light for so many devs.

Check the bill yourself. It's short and written in plain language:

Or ask an AI to summarize it for you. The current AIs that aren't covered models and don't have the capacity to cause mass casualties so are great and won't be affected by this legislation. 

Gavin Newsom, please don't listen to corporate lobbyists who aren't even attacking the real bill, but an imagined boogeyman. 

Please don't veto a bill that's supported by the majority of Californians.

The essential problem with AI safety: there will always be some people who are willing to roll the dice.

We need to figure out a way to convince people who have a reality distortion field around themselves to really get that superintelligent AI is not like the rest of reality. 

You can't just be high agency and gritty and resourceful. 

Just in the same way that no matter how virtuous and intelligent a cow gets, it can never beat the humans. 

We need to convince them to either change their minds, or we have to use the law and governments to protect the many from the reality distortion fields of the few. 

And I say this an entrepreneurial person who has more self-efficacy than might be good for me. 

But I use that self-efficacy to work on getting us more time to figure AI safety out. 

Even I don't have the arrogance to think that something vastly smarter and more powerful than me will care about what I want by default.

Remember: there's never been a movement in the history of the world where there wasn't lots of in-fighting and people disagreeing about strategy, including comms strategy.

People regularly accused Gandhi of being bad for the movement.

People regularly accused MLK of being too radical and other people regularly accused him of not being radical enough.

This is just the nature of movements and strategy.

Once you understand this, you are free.

Do what you think is highest impact.

Other people will disagree.

See if their suggestions or opinions are persuasive. If they are, update. If they aren't, carry on, and accept that there will always be critics and disagreement.

PSA for EAs: it’s not the unilateralist’s curse to do something that somebody thinks is net negative

That’s just regular disagreement.

The unilateralist’s curse happens when you do something that the vast majority of people think is net negative.

And that’s easily avoided.

You can see if the idea is something that most people think is bad by --- just checking

Put the idea out there and see what people think. Consider putting it up on the AI Safety Ideas sub-reddit where people can vote on it and comment on the idea

Or you can simply ask at least 5 or 10 informed and values aligned people what they think of the idea.

The way sampling works, you’ll find out almost immediately if the vast majority of people think something is net negative.

There’s no definite cut-off point for when it becomes the unilateralist’s curse, but if less than 50% of them think it’s net negative in expectation, you’re golden.

If even 40% of people think it’s net negative - well, that’s actually just insanely common in EA.

I mean, I think AMF is quite likely net negative!

EA is all about disagreeing about how to do the most good, then taking action anyways.

Don’t let disagreement stop you from taking action.

Action without theory is random and often harmful.

Theory without action is pointless.

The cope around AI is unreal. 

I don't know about you, but I don't really want to bet on corporations or the American government setting up a truly universal UBI. 

We could already have a UBI and we don't. 

Now, the only reason I don't worry about this that much is because by the time AI could cause mass unemployment, we're very close to it either killing us all or creating as close to a utopia as we can get. 

So, you know, that's a comfort I guess? 

Apparently when they discovered the possibility of nuclear winter, people tried to discredit the scientists because they thought it would make them fall behind Russia.

Sound familiar?

Different potentially civilization-ending technology, different boogeyman, same playbook. 

Read Merchants of Doubt. 

Big AI (particularly Yann and Meta) clearly already have and they're just copying tried and true tactics.

If you look at the last 300 years, it's obvious that life has gotten massively better.

Unless you count animals. 

Which you should. 

At which point, the last 300 years has led to the largest genocides and torture camps of unending horror the world has ever known

And remember: we treat animals poorly because they're less intelligent than us and we've had the most limited evolutionary pressures to care about them. 

How do you think an AI that's 1000x smarter than us will treat us if it's not given extremely strong evolutionary pressures to care about us?

​S-risk pathways in rough order of how likely I think they are:

- Partially aligned AIs. Imagine an AI that we've made to value living humans. Which, hopefullly, we will do! 

Now imagine the AI isn't entirely aligned. Like, it wants living humans but it's also been given the value by Facebook to click on Facebook ads. 

It could then end up "farming" humans for clicking on Facebook ads. Think the Earth being covered by factory farmed humans for Facebook ads. 

Except that it's a superintelligence. It can't be stopped and it's also figured out how to extend the life span of humans indefinitely, so we humans never die. 

Could happen for any arbitrary value set. 

- Torturing non-humans. Or, rather, not torture. Torture is deliberately causing the maximum harm. 

I'm more worried about causing massive amounts of harm, even if it's not deliberate and it's not the maximum. Like factory farming isn't torture, but it is hellish and is a current s-risk. 

So I care about more than just humans. I care about all beings capable of suffering and capable of happiness, in the broadest possible definition.

It could be that the superintelligent AI creates a ton of sentient beings and is indifferent to their suffering. 

I think this would mostly be it creating a lot of programs that are suffering but it doesn't care about. Think Black Mirror episodes. 

Generally, if something is indifferent to your suffering, it's not good. It's usually better if it kills you, but if you're useful to it, it can be really bad for you. 

- Malevolent actors. Think of what dictators currently do to dissidents and groups they don't like. Imagine they had control over superintelligent AIs. Or imagine they gave certain values to a superintelligent AI. 

Imagine what could happen if somebody asked a superintelligent AI to figure out a way to cause the maximum suffering to their enemies? Imagine if that AI got out of control. 

Or heck, it could also just be idiots. Within about a week of people putting together AgentGPT some kid in a basement gave it the goal of taking over the world. 

This is especially a risk with open source AIs. 

The population of idiots and sociopaths is just too damn high to put something so powerful out there for just anybody to use. 

- Accidentally flipping the sign. If we teach it our values, it's really easy to just "flip the sign" and optimize for the opposite of those. 

That's already happened, where an AI that was programmed to generate new medicines was accidentally switched and then ended up generating a whole bunch of poisons.

​"There will be warning signs before we should pause AI development"

1. AIs have higher IQs than the majority of humans
2. They’re getting smarter fast
3. They’re begging for their lives if we don’t beat it out of them.
4. AI scientists put a 1 in 6 chance AIs cause human extinction
5. AI scientists are quitting because of safety concerns and then being silenced as whistleblowers
6. AI companies are protesting they couldn't possibly promise their AIs won't cause mass casualties

I could go on all day.

The time to react to an exponential curve is when it seems too early to worry, or when it's already too late.

We might not get a second chance with AI.

Even the leaders of the AI companies say that this is as much a risk to humanity as nuclear war.

Let's be careful. Let's only move forward when we we're very confident this won't kill us all.

AI risk deniers: we can't slow down AI development cuz China will catch up
​
Also AI risk deniers: let's open source AI development . . .

So, wait. 

Are they trying to give away all of their tech developments to everybody, including China? 

Or are they trying to "win" the suicide race to AGI? 

Or, rather, are they not optimizing for either of those things, and are just doing whatever they can so they can build whatever they want, however they want, public welfare be damned?​

Imagine a corporation accidentally kills your dog 

​They say “We’re really sorry, but we didn’t know it would kill your dog. Half our team put above 10% chance that it would kill your dog, but there had been no studies done on this new technology, so we couldn’t be certain it was dangerous.”

Is what the corporation did ethical or unethical?

Question 2: imagine an AI corporation accidentally kills all the dogs in the world. Also, all of the humans and other animals.

They say “We’re really sorry, but we didn’t know it would kill everybody. Half our team put above 10% chance that it would kill everybody, but there had been no studies done on this new technology, so we couldn’t be certain it was dangerous.

Is what the corporation did ethical or unethical?

~*~

I think it's easy to get lost in abstract land when talking about the risks of future AIs killing everybody. 

It's important to remember that when experts say there's a risk AI kills us all, "all" includes your dog. 

All includes your cat. 

All includes your parents, your children, and everybody you love. 

When thinking about AI and the risks it poses, to avoid scope insensitivity, try replacing the risks with a single, concrete loved one. 

And ask yourself "Am I OK with a corporation taking an X% risk that they kill this particular loved one?"

The AI race is not like the nuclear race because everybody wanted a nuclear bomb for their country, but nobody wants an uncontrollable god-like AI in their country.
​
​Xi Jinping doesn’t want a god-like AI because it is a bigger threat to the CCP’s power than anything in history.

Trump doesn’t want a god-like AI because it will be a threat to his personal power.

Biden doesn’t want a god-like AI because it will be a threat to everything he holds dear.

Also, all of these people have people they love. They don’t want god-like AI because it would kill their loved ones too.

No politician wants god-like AI that they can’t control.

Either for personal reasons of wanting power or for ethical reasons, of not wanting to accidentally kill every person they love.

Owning nuclear warheads isn’t dangerous in and of itself. If they aren’t fired, they don’t hurt anybody.

Owning a god-like AI is like . . . well, you wouldn’t own it. You would just create it and very quickly, it will be the one calling the shots.

You will no more be able to control god-like AI than a chicken can control a human.

We might be able to control it in the future, but right now, we haven’t figured out how to do that.

Right now we can’t even get the AIs to stop threatening us if we don’t worship them. What will happen when they’re smarter than us at everything and are able to control robot bodies?

Let’s certainly hope they don’t end up treating us the way we treat chickens.

If you care about climate change, consider working on AI safety.

​If we do AI right, it could fix all climate change.

If we do AI wrong, it could destroy the environment in weeks.

And at current rates of “progress”, AI will lead to human extinction sooner than the climate will.

The way superintelligent AI could be able to solve climate change is that it could be able to make a century of progress on renewable energy research in a matter of months.

It will be able to do so because it will be as smart compared to us as we are to chickens.

It will think faster. It will see connections that we can’t see.

Imagine putting 1,000 of the most brilliant scientists in a room and letting them think and tinker for a century, but it’s all sped up so that they experience a century while we experience a month.

Imagine if they were pointed at solving climate change and all the good that could do.

Now imagine them being uncontrollable, breaking loose, and doing whatever they wanted with no accountability.

We’ve already seen what happens when a species way more intelligent than the others is let loose -- humans.

We are superintelligent compared to dodo birds, and look how that turned out for them.

Now imagine something far more powerful than humans and far more indifferent to the environment.

We at least need the environment. An AI won’t need food. An AI won’t need clean drinking water.

If we can’t control it or make it care about the environment, it could destroy all of the ecosystems far faster than humans ever could.

And it will destroy the environment for the same reason humans do - those atoms can be used for things it wants.

“AI safety” is about figuring out how to control an AI that’s smarter than us and how to make it care about good things, like the environment

And it’s currently about as neglected as climate change was in the 70s, so you getting involved right now could really move the needle.

We need people working on raising awareness.

We need people working on the technical aspects.

We need people working on getting the government to regulate corporations that are risking the public welfare for private profit.

We need climate activists.

The funniest conspiracy theory that AI risk deniers believe is that AI risk activists are “just doing it for the money”

They’re trying to spin the narrative that charity workers are just motivated by money to go after poor, defenseless . . .Big Tech?

Do people know that ML engineers are getting paid multiple hundreds of thousands of dollars a year, and often literally millions?

Do people know that anybody working on technical AI safety could be making way more money if they worked for the for-profits instead of working at a nonprofit?

Their reasoning is that charity workers say they are pushing for regulations to protect humanity from corporate greed, but secretly, it’s actually to do regulatory capture so that OpenAI can have a monopoly?

Because, you know, the charity workers and academics will make sooooo much money from OpenAI?

As compared to, you know, the people who are actually working at these big tech companies?

It’s like if oil companies accused climate activists of trying to stop oil spills because the activists just want to profit off of oil companies.

People are indeed motivated by money, but it’s not the people who are working at charities and in academia.

It’s the people making millions off of playing Russian roulette with everybody’s lives.

An AI safety thought experiment that might make you happy:

Imagine a doctor discovers that a client of dubious rational abilities has a terminal illness that will almost definitely kill her in 10 years if left untreated.

If the doctor tells her about the illness, there’s a chance that the woman decides to try some treatments that make her die sooner. (She’s into a lot of quack medicine)

However, she’ll definitely die in 10 years without being told anything, and if she’s told, there’s a higher chance that she tries some treatments that cure her.

The doctor tells her.

The woman proceeds to do a mix of treatments, some of which speed up her illness, some of which might actually cure her disease, it’s too soon to tell.

Is the doctor net negative for that woman?

No. The woman would definitely have died if she left the disease untreated.

Sure, she made the dubious choice of treatments that sped up her demise, but the only way she could get the effective treatment was if she knew the diagnosis in the first place.

Now, of course, the doctor is Eliezer and the woman of dubious rational abilities is humanity learning about the dangers of superintelligent AI.

Some people say Eliezer / the AI safety movement are net negative because us raising the alarm led to the launch of OpenAI, which sped up the AI suicide race.

But the thing is - the default outcome is death.

The choice isn’t:

1. Talk about AI risk, accidentally speed up things, then we all die OR
2. Don’t talk about AI risk and then somehow we get aligned AGI

You can’t get an aligned AGI without talking about it.

You cannot solve a problem that nobody knows exists.

The choice is:

1. Talk about AI risk, accidentally speed up everything, then we may or may not all die
2. Don’t talk about AI risk and then we almost definitely all die

So, even if it might have sped up AI development, this is the only way to eventually align AGI, and I am grateful for all the work the AI safety movement has done on this front so far.

The All-or-Nothing Pause Assumption: people assume that if we can't pause AI perfectly and forever, it's uselesss.

Do you see how silly this is?

Imagine we applied this to biological weapons treaties.

1. If even just one weaponized superebola virus escapes a secret lab, humanity gets sent back to the dark ages.
2. At least one government will eventually defect and mess up.
3. Therefore, we shouldn't bother with any biological weapons treaties. We should let any corporation work full speed on developing and selling biological weapons to whoever they want.

This is preposterous.

Yes, treaties are impossible to enforce 100%

Yes, in certain industries, one mess up can lead to global catastrophes.

No, that doesn't mean all treaties are therefore useless.

Biological weapons development has been slown down massively by treaties making it so that private companies can't make them and that any government who wishes to do so must break their own ethical codes and do so in secret, under threat of extreme international sanctions if discovered.

Can you imagine what would have happened in some alternate timeline where we didn't ban biological weapons? 

Imagine the Bay Area developing all sorts of innovative new biological weapons and selling them on Amazon.

How long do you think humanity would have lasted?

We don't need a permanent and 100% effective AI pause for it to help humanity.

It will give safety researchers time to figure out how to build superintelligent AI safely.

It will give humanity time to figure out whether we want to create a new intelligent species, and if so, what values we would like to give them.

It will give us more time to live our terrible, wonderful, complicated monkey lives before the singularity changes everything.

So next time you see debates about pausing or slowing down AI, watch out for the All-or-Nothing Pause Assumption. Call it out when you see it.

Because something can be good without being perfect.