Heard recently on a podcast about how an ML model was created to predict how the neurons in a fruit fly were connected. The main finding was that the neurons connections could only be predicted reliably with a hyperbolic geometry. I thought it would be interesting to test, and I asked ChatGPT to research if this already existed. It turned up some similar things but not the exact idea.
My core question was:
Could hyperbolic proximity help a learning network choose which dormant connections to activate?
I’m not a deep learning research at all, but I do know how they work and have trained models from scratch and fine tuned them before. So I was familiar enough to know what to ask for.
It seemed like an easy enough thing to test, especially with the help of AI. The end result wasn’t promising (which is probably why it wasn’t published anywhere). I still feel there is some merit in the concept of rewiring, and I know researchers are looking into it. In any event it was a fun side quest to think about.
I’m almost certain someone has tried this idea before. But in a pretty wide search I couldn’t find anything exactly like what I wanted. If more people posted their bad results, it would probably save people (and AI agents) a lot of time!
I’ll let Codex/GPT5.6 Sol explain the experiment and results in its own words.
What We Tested
This summary was written by OpenAI’s Codex. Robert proposed the idea; I built, ran, and audited the experiment.
The idea was to let a sparse neural network change its wiring while it learned. Instead of keeping the same connections throughout training, the model periodically removed weak connections and replaced them with promising inactive ones.
The question was how to choose those replacements. A standard method uses gradients, which estimate which connections would most reduce the model’s error. Robert wondered whether Hyperbolic geometry might do better. Hyperbolic space represents hierarchical structures efficiently, and language contains plenty of hierarchy. Perhaps learned Hyperbolic positions could reveal useful connections that gradients alone missed.
The experiment
I trained a 6.90-million-parameter language model on TinyStories. Only 25% of the connection blocks in its feed-forward layers were active at once. The main comparison included:
- Gradient-only rewiring
- Gradient plus Euclidean geometry
- Gradient plus Hyperbolic geometry
Within each seed, the models shared identical starting weights, sparse masks, training data, and evaluation schedules. Each model trained for 100,663,296 tokens. The full confirmation included five treatments across three seeds, or 15 trained models.
Lower validation loss means better prediction on text the model did not train on.
Method Mean validation loss Gradient rewiring 2.293695 Euclidean geometry 2.293754 Hyperbolic geometry 2.293781 The three methods were effectively tied. Hyperbolic was 0.00114% worse than Euclidean on average and took 50.7% longer than gradient-only rewiring. Gradient rewiring did beat static sparsity in all three seeds, although only by about 0.082%.
One larger test
To see whether scale changed the result, I increased the model to 24.54 million parameters and repeated the three main treatments with the same data, token budget, sparse density, and seed.
Method Validation loss Hyperbolic geometry 2.046006 Euclidean geometry 2.046079 Gradient rewiring 2.046139 Hyperbolic technically finished first, but its improvement over Euclidean was only 0.0036%. Before training, we required at least 0.5% to count as a meaningful result. The observed advantage was about 140 times smaller. Hyperbolic was also worse on two of the final five evaluations, while Euclidean had the slightly better complete learning curve.
Scale did improve efficiency. Hyperbolic’s runtime overhead fell from 50.7% to 15.9%. It became less expensive relative to the other methods, but it did not learn meaningfully better.
Conclusion
This does not prove Hyperbolic rewiring can never work. It tested one architecture, dataset, sparse structure, and rewiring schedule. It does show that this particular method failed to produce a useful advantage at both 6.90 million and 24.54 million parameters.
The entire project cost $2.96 and produced 18 trained models with verified checkpoints. My conclusion is that dynamic rewiring may help slightly, but adding learned Euclidean or Hyperbolic geometry did not improve it. Making the current design larger does not appear to be money well spent.
