created 2025-04-09, & modified, =this.modified
I’ve encountered Cyc before (including reading after Lenat’s death in 2023), but this essay comes at a declaration of the death of the project.
via The Second Digital Turn - Design Beyond Intelligence by Mario Carpo
With classifications construed as cascades of divisions and subdivisions, encyclopedias (which originally, and etymologically, referred to a circle, or cycle, of knowledge) started to look less like circles and more like trees, with nested subdivisions represented as branches and sub-branches.
Obituary for the greatest monument to logical AGI. After 40 years, 30 million rules, 200 million dollars, 2000 person-years, and many promises, Cyc has failed to reach intellectual maturity, and may never will. Exacerbated by the secrecy and insularity of Cycorp, there remains no evidence of its general intelligence.
Douglas Lenat’s 40-year quest to build AI through symbolic logic. His PhD work included automating mathematical discoveries through heuristic search, but this method failed as the pool of heuristic rules were quickly exhausted.
In 1985, he launched Cyc to manually encode millions of facts and rules about common sense, predicting that once this “knowledge pump” was primed, the system would begin true machine learning by reading natural language texts and conducting autonomous scientific experiments.
Automated Discovery
To solve problems of type X, one simply writes a program that walks through the space of possible solutions to X, and code in the heuristic rules, its north star, so that it will not be lost in the space of solutions.
The process would be the perform heuristic search over the space of heuristic searches.
These systems begin with simple rules, and as it runs it builds, prunes and modifies the rules, so that in the end its rule set allows the highest scores.
AM
Despite still being a logical AI with a problem space and search, there was no data — it was “self play”.
To start AM, Lenat began by entering 115 concepts in set theory and ~250 heuristic rules, and thence AM ran, discovering more and more constructions. Most would be trivial, but a few would be interesting, and these interesting constructions would be stored, allowing further constructions upon them. It secured a place as a minor legend, reportedly rediscovering many concepts, such as the natural numbers, the prime numbers, and the Goldbach conjecture.
A concept, is a frame, which are objects in OOP. A concept had 25 possible facets.
Each action has three possible effects
- add a new task to agenda
- create a new concept
- add or delete an entry to a facet of a concept
How does AM know that the concept should be called “Prime Numbers”? Ah, that’s because Lenat would regularly interrupt and inspect AM, and if Lenat notices that AM has rediscovered, say, prime numbers, he would rename that from something like
concept-421toprime-numbers.
Homoiconicity
homoiconicity (from the Greek words homo- meaning “the same” and icon meaning “representation”) is an informal property of some programming languages. A language is homoiconic if a program written in it can be manipulated as data using the language. Code can be treated as data.
The End of AM
AM was badly documented.
Now, handwaving would have not been a problem if they were intended as merely glosses over the source code, but the source code was also unpublished, making it impossible for other other researchers to reproduce or extend the work, or to reinterpret AM’s workings.
Lessons
-
AM exhausts itself. Every new discovery depends on ~24 heuristics, and each heuristic has a hand in ~24 discoveries. Therefore, with ~N heuristic rules, there would be ~N interesting discoveries. Because AM cannot discover heuristic rules, with ~300 starting heuristic rules, it would run out of interesting discoveries and “die of boredom”.
Eventually, AM acquired an uncommon ailment for a computing system: intellectual exhaustion. Having explored the esoteric reaches of mathematics, AM suddenly downshifted into a preoccupation with rudimentary arithmetic. Finally, with the remark, “Warning! No task on the agenda has priority over 200.”, the system virtually expired, as though from boredom.
-
AM could not go meta and discover new heuristics, because Lisp is good for math and not “heuretics.” Modifying a Lisp expression for a heuristic mostly likely ends up with nonsense.
EURISKO
Made to see if it could avoid intellectual exhaustion and discover new heuristics.
Lenat designed a new language called RLL (“Representation Language Language”), over which heuristic rules are efficient to search.
As in AM, each heuristic in EURISKO had a level of
Worth. Higher-worth heuristics were more likely to be invoked. Each heuristic had aCreditTo, so that if a heuristic rule rose in worth, itsCreditTowould also rise in worth. When EURISKO was born, and saw itself, all the heuristic rules it had were branded withCreditTo = DBLenat, but this would soon change, as heuristics begat heuristics.
Lenat concluded that there really is no way to get a working automated discovery program without doing the hard work of hand-coding in a lot of common sense, and that there would be a point at which this system would finally achieve escape velocity, and would never be exhausted again.
Messiness is a hideous strength
The apparent adhoc-ness in both the heuristics’ content themselves, and in the control knowledge guiding the application of those heuristics, is clearly the source of many methodological objections to the work. But we believe that this adhocracy – indeed, adhocracy controlling adhocracy – may be the source of EURISKO’s underlying potential especially as a model for cognition.
In working on the design of integrated circuits, for example, EURISKO stumbled on the fact that symmetry is a desirable property for such chips, although it did not understand why; when it was later instructed to design fleets for the Traveller game, EURISKO decided to make them symmetrical and justified its decision by referring to its earlier experience in designing circuits.
Traveller Adventure 5: Trillion Credit Squadron
Players each design their own fighting starship squadrons within a budget of one trillion credits (Cr1,000,000,000,000) and fight them against each other.
Traveller TCS is notable as one of the first games to be mastered by a computer. Starting in 1981, Douglas Lenat adapted his heuristic discovery system named Eurisko to create fleets for the US national championship, and won on two occasions. On the first occasion this involved a large number of stationary, lightly-armored ships with many small weapons.
This resulted in extensive changes to the game’s rules. On the second occasion in 1982, Eurisko won again when the program discovered that the rules permitted the program to destroy its own ships, permitting it to continue to use much the same strategy..
Meta-bugs
One of the first heuristics that EURISKO synthesized (H59) quickly attained nearly the highest Worth possible (999). Quite excitedly, we examined it and could not understand at first what it was doing that was so terrific. We monitored it carefully, and finally realized how it worked: whenever a new conjecture was made with high worth, this rule put its own name down as one of the discoverers! It turned out to be particularly difficult to prevent this generic type of finessing of EURISKO’s evaluation mechanism. Since the rules had full access to EURISKO’s code, they would have access to any safeguards we might try to implement. We finally opted for having a small ‘meta-level’ of protected code that the rest of the system could not modify.
The second ‘bug’ is even stranger. A heuristic arose which (as part of a daring but ill-advised experiment EURISKO was conducting) said that all machine-synthesized heuristics were terrible and should be eliminated. Luckily, EURISKO chose this very heuristic as one of the first to eliminate, and the problem solved itself.
Often I’d find it in a mode best described as “dead”. Sometime during the night, EURISKO would decide that the best thing to do was to commit suicide and shut itself off. More precisely, it modified its own judgmental rules in a way that valued “making no errors at all” as highly as “making productive new discoveries”. As soon as EURISKO did this, it found it could successfully meet its new goal by doing nothing at all for the rest of the night… I eventually had to add a new heuristic to EURISKO-one it couldn’t modify in any way-to explicitly forbid this sort of suicide.