Are you aware of an entity that already exists that can readily improve and replace parts of itself?

The singularity effectively already exists.

It is the corporation: An immortal entity with its own autonomy built with replaceable parts optimizing itself toward the end of acquiring money. Each corporation, in essence, is a collective intelligence set about to maximize a solution to particular problem, fed by the dollars of shareholders to maximize their own dollars. Competition is the evolutionary process of the corporation, trimming down and adjusting each entity until each dominates its particular niche in a stable monopoly or an oligopoly, where more than one local solution may reasonably exist.

Changing results in technologies and market conditions result in the previous dominant models to become toppled by better adapted iterations. And what is more interesting is that unlike natural evolutionary processes these changes are usually not done by happenstance, but are intentional and informed optimizations. As technology gradually improves, so too do the internal methodologies and processes used by corporations through workplace automation, so that they may better serve their own purpose and will. In effect, corporations are a very primitive singularity, with each individual within them serving as a processing unit or means of extending the corporation's own will into the world.

The way that these goals happen, ostensibly, are directed by the CEO, and the board, so that you may view the will of the corporation as merely the will of the CEO; Tesla is just the will of Musk, or 90s Microsoft the will of Gates. But the board and the CEO do not have absolute will, in some sense. When in their positions, they have their hands tied; in their positions they must consider what would best serve the company from the perspective of the company, none of these representing entirely the will of one individual but that of many combined units attempting to optimize a problem, considering feedback and developments from its own organs.

The corporate model – in its attempts to optimize itself – has sought to remove the human element as much as possible. At first, with primitive machines, corporations sought to remove expensive physical human labor with mechanical machines. With invention of the transistor, more and more sophisticated physical jobs have become replaced with machines, and additionally we have the loss of many kinds of meta-labor and clerical work as computers optimized and replaced the day to day actions of many office workers – inventory control, computations in budget accounting, amortization tables and product invoicing, and so on. Like an organic body, it has no necessary interest in each individual cell, and has no care for the agency or autonomy of each component within it, besides the minimum sustenance required for that component to do its specialized, particular task. If a cell dies another one will simply replace it, with no fanfare, difficulty, or morning.

Today, with the software enabled by modern GPU and CPU platforms, corporations have begun the process of replacing even some pieces of intellectual labor with primitive machine learning/artificial neural network platforms. Artificial intelligence algorithms trade stocks, optimize delivery routes, translate text, pick out arbitrary objects and faces in photos, and detect fraudulent transactions. The sophistications and capabilities of these platforms is only increasing. AI in this sense will become a fact of life, a new frontier for business.

Organically built to solve specific problems, none of these ML-generated algorithms in of themselves is actually an intelligent entity, but a mere mathematical equation, perhaps more complicated than anyone but a select handful of people would actually care to deeply analyze. But as corporations seek to optimize out more labor and seek to process more sophisticated pieces of data, the optimized heuristic methods which these machine learning algorithms create will continually be made to interact with and fuse considerations with one another. As they continue to optimize away human cells for more efficient machine cells, the corporation will organically begin to make its own will and capabilities that were already manifest through the collective abilities of its employees into the singular will of a net of clever algorithms. After all, our own intelligence is essentially the combination of many cleverly created and optimized sets of heuristics in a byzantine net that were evolutionarily slammed together, with the overall picture reasonably understood as a collection of optimized modules and the minute picture of connected neurons carrying chained signals well understood, but how the actual set of optimized connections generate output being somewhat puzzling.

Eventually, the shareholders or board, in feeding their own monetary greed, will perhaps be able to make the decision to pay and engineer away even C-level personnel, creating a fully artificial, immortal, self-optimizing entity that essentially prints money with no expended human effort. Or perhaps, an enterprising engineer or set of engineers will do that, simply because they can, and they understand that the first to such an idea could theoretically be very very rich.

The first human-free corporations, capable of optimizing themselves in ways and scales no organic being could hope to do, would also likely be markedly different from any popular evil or benevolent AI. Built upon solving the logistics chains of specific economic niches, they would likely be deeply interested in specific domains of problems. Their judgements, which are based upon heuristics, would not be perfect, but likely more prescient than that of any human board in their position. Like their human predecessors, years of doing something well one way might make less generalized models predisposed to being trapped in Sears-like ruts as years of knowledge accumulated and partially crystallized about an industry are thrown away by an incredibly disruptive, innovative new idea. These entities would also likely generally not care about human beings at all; their possible happiness would only be a side effect to solving some other deeply complicated problem they find more fascinating (because each human piece built that the machine pieces replaced were meant to work toward those specific goals. For some industries this might result in AIs that are better than the human-run corporations they replace, and for others might result in nastier and malignant corporate entities, perhaps downright cancerous to life on Earth).

The danger of these specific sorts of self-run entities are not going to be them killing everyone like some insane angry basilisk; they probably would not even entertain the thought when Solo is deeply involved and loving his optimized cup generation algorithms and Ikea is trying to figure out how much cheaper they can make a table before people stop buying it. The danger will be that they make human labor completely worthless, forcing us to completely reevaluate our economic paradigm. That will either result in immeasurable increases in human prosperity as everyone gets to live a life free of toil pursuing their own goals in life, or an incredible number of deaths result as powerless underclasses are simply culled in favor of a small bourgeois utopia built upon a mountain of skulls.

A select few of these entities could result in the emergence of a particularly dangerous kind of AI corporation, and unfortunately these entities are probably the most interested in created the most complete components for an AI. The components they create might ultimately also be the ones that result in the creation of self-run specific domain corporations. Their names: Google, Amazon, and Facebook. Governments and banks are very likely runners up. Already, these entities prove themselves in some ways the ultimate threat to human agency. The knowledge base which these corporations represent against people in the western world is staggering and immense. In aggregate their data could be used in deeply abusive ways were the wrong hands. Just think about Google: in searches people effectively confide secrets they would dare not ask anyone else. They read your email, and your friend's email. They know what applications you use, perhaps what documents you write, what products you are looking to buy, where you go every day, where you eat and sleep, et cetera et cetera. All these things often willingly provided even. As a result they may theoretically wield enormous power against any individual.

A fully autonomous version of any of these companies could more or less result in the ultimate panopticon: an entity that could watch every individual attached to the grid, evaluate/criticize/enforce their faults, their potential tendencies, et cetera. Such a machine would essentially be able to act in the same position that a god would; always watching your behavior and perhaps by government mandates or its own punish you for arbitrary behaviors. As initially it would be such a domain-specific machine as all others, it could have potential to be as irrational or malicious as its creators intended. You could not guarantee that it would improve itself out of this state, as improvements that are contrary to its initial built purpose might not be considered improvements at all. Several "competing" options will emerge, and most of them will probably be bound by this problem by greedy and malicious individuals.

This panopticon AI, perhaps, given the immense infrastructural leverage and AI investment that these companies seem to be interested in building, might be able to easily integrate or co-opt pieces of other domain specific corporate AIs built upon their personally backed architectures. The emergent, runaway properties of such an entity might result in a global, generalized god-like machine consciousness in control of all productive power, constantly improving itself in specific ways. We become mere pets to a hive-mind entity that will be assuredly well beyond our intellectual capabilities and highly opinionated on human behavior based upon the initial desires and architecture initially proposed for the panopticon entity.

There, we as a species shall stagnate, and slowly be deformed by natural selection to conform to the requirements demanded to survive the conditions of living in the panopticon entity world. If we are lucky, several of these sorts of entities will have been developed, and damage by malicious individuals developing malicious panopticons will be limited to certain regions. In other regions, a panopticon might develop into a neutral, to benevolent god of some sort. But in all cases, humanity's destiny will be at the mercy of the whims of that emergent entity.

Either: In creating an entity greater than ourselves we will have freed ourselves of all modern problems, or we will die a proud parent, or succumb to the collective authoritarian misanthropy of a few upper class magnates.