Pandora's Box Is Already Open
Updated: 8 minutes ago
Technology, Human Nature, and the Problem of Control
Find Part One <here>
In the first part of this Thought Project, I considered a fairly simple question: just because humanity can develop a technology, does that mean we should?
That question led me to something I've begun thinking of as a Technological Reformation — a possible future cultural shift in which technological progress is measured not only by what humanity becomes capable of doing, but by whether we possess the wisdom to do it responsibly.
But there's an obvious problem with that idea. In many respects, the decision has already been made.
We already split the atom. We already learned how to manipulate DNA. We already developed the ability to alter viruses and other organisms. We already created artificial intelligence. We already developed autonomous weapons, global computer networks, sophisticated surveillance technologies, and machines capable of operating with decreasing levels of direct human involvement.
And we cannot simply decide to forget how any of this works.
Pandora's Box is already open.
So perhaps the more important question is no longer whether humanity should open the box. It's this: how do we control what comes out — and how does humanity control itself now that the box is open?
Knowledge Is Different From Almost Everything Else
A machine can be destroyed. A factory can be closed. A weapon can be dismantled. But knowledge is different.
Once a fundamental scientific discovery becomes widely understood, eliminating it becomes extraordinarily difficult, if not impossible. Humanity can't collectively decide tomorrow that nuclear fission was a mistake and return to 1938. We can't undiscover genetics. We can't uninvent computation. We can't erase the accumulated understanding of molecular biology. And increasingly, scientific knowledge is stored, copied, and transmitted throughout a globally connected civilization.
That makes technological knowledge fundamentally different from many earlier forms of power. A medieval king might control a fortress. An empire might control a fleet. An industrial nation might control factories. Those things required enormous concentrations of physical resources. Knowledge escapes those boundaries. Once it spreads, someone else can use it.
And that creates a dilemma sitting at the heart of almost every attempt to control consequential technology.
If We Don't Do It, Someone Else Will
Suppose a country concludes that a particular technology presents an unacceptable risk. It chooses not to pursue it. But another country does.
Now restraint may create vulnerability. The first country faces a choice: continue exercising restraint and potentially allow a competitor or adversary to acquire a decisive advantage — or resume the research to ensure it isn't left behind.
The logic becomes circular. We probably shouldn't develop it. But someone else might. Therefore we need to understand it. Understanding it requires researching it. Researching it gives us the capability to develop it. And once we have the capability, someone will argue we should deploy it before someone else does.
Nuclear weapons demonstrated this problem dramatically. But the same logic can apply to artificial intelligence, biotechnology, autonomous weapons, surveillance systems, cyber capabilities, and technologies that don't yet exist.
This creates what may be one of the most dangerous forces in technological development: technological competition can make individually rational decisions collectively irrational. No individual nation may actually want a particular arms race. Yet every participant may conclude it can't afford to be the one that stops.
The same problem can exist between corporations. Or universities. Or research institutions. Or eventually, individuals.
And that last possibility may be particularly important.
Power Is Becoming Smaller
Historically, the ability to produce enormous consequences required enormous resources. To conquer a country required an army. To build a battleship required an industrial economy. To develop the first nuclear weapon required thousands of people, massive infrastructure, extraordinary scientific expertise, and the resources of a nation-state.
But technological progress often makes yesterday's extraordinary capability tomorrow's ordinary tool.
Computers once occupied rooms and were available only to governments, universities, and large corporations. Today billions of people carry vastly more computing power in their pockets. Genetic analysis that once required major laboratories can now be performed with comparatively modest equipment. Sophisticated software can be distributed globally almost instantaneously. Artificial intelligence is making specialized knowledge increasingly accessible to people who may not have years of formal training in the fields they're working in.
That democratization has tremendous benefits. But it also raises an uncomfortable possibility: what happens when technologies capable of producing enormous consequences become available to progressively smaller groups of people?
Perhaps someday a capability that once required a nation will require a corporation. Then a laboratory. Then a small group. Perhaps eventually an individual.
If that happens, technological governance becomes much harder. A government can monitor uranium enrichment facilities. It's considerably harder to monitor millions of computers. It may be harder still to monitor biological or computational capabilities that can fit inside an ordinary laboratory — or eventually, on an ordinary computer.
At some point, controlling technology by controlling physical access to it may no longer be sufficient. Which means the problem increasingly becomes one of controlling human behavior.
Human Nature Hasn't Accelerated
This may be the part of the problem that concerns me most.
Technology changes rapidly. Human beings do not. We remain ambitious, curious, competitive, creative, generous, fearful, greedy, compassionate, jealous, idealistic, selfish, cooperative, tribal, brilliant, and occasionally irrational. Those characteristics aren't necessarily defects — together they constitute much of what it means to be human.
Curiosity produces discovery. Ambition produces achievement. Competition can produce innovation. But the same characteristics can also produce terrible outcomes.
The significant change isn't necessarily human nature. It's the amount of power technology places behind it. A reckless individual once had a limited capacity to cause harm. Technology can amplify that capacity enormously. A malicious person with a sword presents one level of danger. A malicious person controlling an industrial weapon presents another. A malicious person controlling a nuclear weapon presents something altogether different.
Now extend that trajectory another century. What capabilities might eventually be available to one person — or a handful of people?
That question leads to an uncomfortable possibility: human nature may remain relatively constant while the consequences of individual human decisions become progressively greater. And that changes the nature of technological risk.
The Problem of the Good Person
But focusing only on malicious actors misses something important. Some of the greatest technological risks may come from people who aren't malicious at all. They may be intelligent, ethical, educated, well-intentioned — and completely convinced that what they're doing is beneficial.
That may actually be the more difficult problem.
A scientist pursuing dangerous research may sincerely believe the knowledge could save lives. An engineer building an autonomous system may genuinely believe it will make people safer. A technology company may honestly believe its product will improve human communication. A government may develop a weapon because it believes doing so will prevent war. A researcher may manipulate an ecosystem because the intended outcome is environmental restoration.
None of these people necessarily intends harm. The problem is that good intentions don't guarantee good outcomes. Complex systems behave in unexpected ways. People make mistakes. Models contain assumptions. Safeguards fail. Unanticipated interactions occur. And sometimes we simply don't know what we don't know.
That's one of the reasons should we? becomes so difficult. It isn't a contest between good people and bad people. Often it's a contest between possible benefit and uncertain consequence.
Some Mistakes Can Be Reversed
This leads to another distinction I think deserves greater attention in technological decision-making: reversibility.
Many technological mistakes can be corrected. A defective automobile can be recalled. Bad software can be patched. A poorly designed machine can be replaced. A failed product can disappear from the market. These failures may be expensive or even tragic, but they remain bounded.
Other mistakes may not be. A nuclear weapon can't be recalled after detonation. An engineered organism released into an ecosystem may reproduce. A genetic alteration introduced into a population may pass to future generations. An intervention in a sufficiently complex ecological system might initiate changes that can't simply be undone. Information released globally may be impossible to retrieve. The consequences of some future technologies may extend across generations.
That suggests a fairly simple principle: the more consequential and irreversible a technological action becomes, the greater the responsibility should be before taking it.
That doesn't mean demanding absolute certainty — absolute certainty rarely exists, and if humanity required proof of zero risk before doing anything, progress would stop. But perhaps the burden of justification should increase with the magnitude of the possible consequence. A new consumer appliance and a technology capable of altering the human germline shouldn't necessarily be treated as equivalent categories of innovation. One is relatively reversible. The other may not be.
The Mathematics of "Almost Never"
There's another problem.
Suppose an experiment has only a one-in-a-million chance of producing a catastrophic outcome. That sounds extraordinarily safe. Perhaps it is. But what happens if the experiment is conducted ten million times?
This isn't intended as a literal risk calculation — real technological risk is far more complicated — but the principle matters. Civilization doesn't perform one experiment. It performs millions of them. Thousands of companies develop technologies. Universities conduct research. Governments operate laboratories. Individuals experiment. And this continues year after year.
So perhaps technological risk can't always be evaluated only at the level of an individual experiment or product. It sometimes has to be considered cumulatively. An acceptable individual risk, repeated often enough and at sufficient scale, may become an unacceptable civilizational risk. And as technological development accelerates, the number of opportunities for low-probability, high-consequence failures may increase with it.
Again, the problem may ultimately be speed.
Can We Control the Machine?
When people discuss technological risk, the conversation often focuses on control. Can an artificial intelligence system be controlled? Can an engineered organism be contained? Can autonomous weapons remain under meaningful human command? Can nuclear weapons remain secure? Can a geoengineering project be stopped if something goes wrong?
These are essential questions. But I increasingly wonder whether they're the secondary ones.
The primary question may be: can humanity control itself? Can nations resist developing a technology merely because an adversary might develop it? Can corporations delay releasing a profitable product because its societal consequences are uncertain? Can researchers walk away from an experiment because the potential knowledge doesn't justify the potential risk? Can individuals possessing extraordinary technological capability voluntarily choose not to use it irresponsibly? Can humanity accept that there may be things we're capable of doing that we nevertheless choose not to do?
No engineering safeguard can fully answer those questions. They're questions about human behavior.
Law Cannot Stand Beside Everyone
Civilization has always faced this problem in other forms. We have laws against murder, theft, fraud, and countless other harmful behaviors. But law isn't what prevents most people from committing those acts most of the time. There isn't a police officer standing beside every person.
Healthy societies depend on people internalizing certain expectations about acceptable behavior. Family, culture, faith, philosophy, education, professional ethics, reputation, and community have historically helped establish those expectations. People learn that some actions are simply things one doesn't do. Law reinforces those norms. It can't replace them.
If increasingly consequential technological capabilities become widely distributed, the same may eventually have to become true of technology. We may reach a point where technological safety can't depend entirely on governments preventing people from doing dangerous things. Some people will have to choose not to do them.
That moves the Technological Reformation beyond regulation. It becomes cultural. Perhaps even moral.
The Real Meaning of Pandora's Box
The story of Pandora is usually invoked as a warning about curiosity. Don't open the box. Some knowledge is better left alone.
But that metaphor no longer quite describes our technological predicament. The box has been opened repeatedly, and humanity has benefited enormously from what came out — medicine, electricity, transportation, agriculture, computing, communication, energy, spaceflight. Modern civilization itself is inseparable from technological progress. There's no reasonable path backward, and I wouldn't advocate one.
The challenge is learning how to move forward while recognizing that the contents of the box now include capabilities powerful enough to harm the civilization that created them.
That's a fundamentally different problem.
Perhaps the question of the twenty-first century and beyond won't be how do we close Pandora's Box? We can't. The better question may be: how does a civilization learn to live responsibly with an open box?
And perhaps that's where the idea of a Technological Reformation becomes less about technology and more about humanity. Technology will continue advancing. Knowledge will continue expanding. Human curiosity will continue pushing boundaries. That's part of who we are, and it has produced extraordinary things.
But increasingly powerful technology places increasingly consequential choices into human hands. Eventually, technological maturity may depend not only on the sophistication of our machines, but on the sophistication of the people using them.
The greatest technological control problem humanity may ultimately face may therefore not be controlling technology at all.
It may be learning to control ourselves while possessing it.
And that raises the question I want to explore next: if law and technological safeguards can never be enough, what kind of culture — and what kind of people — would a technologically mature civilization require?
Perhaps the answer begins with an old idea that technological civilization may someday need to rediscover: virtue.
~ Eric
Find Part Three <here>
Comments