Atlan's catalog is built on relationships, so we stored it in a graph database. Years later we checked what our reads actually asked for and found they were all lookups by key, never traversals. We removed the graph layer and store the catalog as objects and lists of connections on Cassandra instead. An agent then carried that change across 600+ live customer instances, with the risky work scripted and the judgment left to the agent.
- Every read in the product is 'given this one thing, what is directly attached to it'. That is a lookup by key, and a graph engine earns its cost on traversal.
- Infrastructure you pick early quietly turns into infrastructure you stop questioning. The gap was not the first choice, it was never rechecking it.
- Keeping the storage interface unchanged turned a rewrite into a config flag, and gave every catalog an independent one-setting revert.
- When you automate something risky, split the mutation from the judgment: keep the data movement deterministic and let the agent own the operational calls around it.
Atlan started out as a data catalog. When a company has tens of thousands of tables, dashboards, and pipelines spread across a dozen tools, Atlan is where someone goes to make sense of them: what is this, where did it come from, is it safe to use, who owns it.
Underneath the product, all of that is relationships. A table relates to its columns. A pipeline relates the tables it reads to the ones it writes. An asset relates to the person who owns it and the terms that describe it. Everything in the catalog is one thing pointing at another.
Relationships feel like a graph. So when we built Atlan, storing them in a database built for graphs felt like the obvious choice. We ran one in the middle, between our service and the two systems that actually held the data, translating our assets into its format and back.
What started going wrong
For a few years that setup was fine. Then, as we scaled, the nature of the problem changed. We had assumed we were solving one kind of problem. What we were actually solving had quietly evolved into another.
Two things caught up with us at once. The first was size: more customers, bigger catalogs, more relationships inside every one. The second was the cost of carrying all of it. The graph layer in the middle held several gigabytes of memory in every pod, on every catalog we ran. It needed more than twenty seconds to initialize on every restart. And under real concurrent load, it had a throughput ceiling we kept hitting.
None of that was a crisis on any given day. It was worse in one specific way: it scaled against us. The bigger we grew, the more that layer cost us, in money and in latency, for work we were increasingly unsure it needed to be doing.
So we finally asked the question we had been stepping around. Not “is our graph database fast enough,” which assumes we need one. The real question: what do people actually ask this catalog, and does answering it need a graph engine at all?
So we went and looked
Open any asset in Atlan. Say a table. The product loads that table and everything attached directly to it: its columns, its owner, the glossary terms that define it, its tags, its README, the assets related to it. That is what fills the page, and some version of it happens on nearly every click in the product.
Look at what that read actually is. It is not a walk across the graph. It is: given this one asset, hand me the things directly attached to it. One thing and its neighbours.
We checked whether the asset page was just a convenient example, and it was not. Search results are decorated with owners and tags, a glossary term and its linked assets, a data model and its entities, even the way our AI retrieval pulls an asset for an agent: every one of them is the same shape. Given this one thing, what is directly attached to it. That is a lookup by key, not a traversal across a graph.
Here is the distinction the whole project rests on. A graph engine earns its cost when you traverse, when you sweep the whole graph for paths of any length that match some filter. That is a real need for some products. It was never ours, and our customers never asked us for it. We were only ever doing lookups, and paying traversal prices to do them.
Which raises a fair question about the original decision. Reaching for a graph database when everything you model is a relationship is a reasonable default on day one. The gap was not that first choice. It was that we kept it for years without going back to check whether the shape of the problem had changed underneath us. It had, and the data had been saying so for a long time before we looked. Infrastructure you pick early quietly turns into infrastructure you stop questioning.
What the middle layer was actually costing
Once you see it as a lookup problem, that layer in the middle stops looking like infrastructure and starts looking like a tax. There were four costs, roughly in the order they hurt.
Memory. The layer’s caches and runtime held four to eight gigabytes in every pod, across every catalog. That is not a rounding error on an infrastructure bill.
Startup. More than twenty seconds of its own initialization on every pod restart. Fine on a normal day. Not fine at minute three of an incident, when restarting is the fix and every second is customer impact.
Opacity. Our data was written to storage in the layer’s own binary encoding. Answering a question as basic as “is this asset actually in storage” required the right version of the layer and someone who understood its serializer. That turned a five-minute check into a one-person dependency, and it meant we could never fully prove a backup restore was clean.
Coupling. The layer’s API had leaked into our business logic across more than forty files. Every change we made carried a little more risk than it should have.
None of those four bought us anything, because the one capability we were paying for was the one we never used.
What we built instead
If the question is always “given this one thing, what is directly connected to it,” then the answer should sit in one place, already assembled, waiting to be read.
That is an old, well-understood idea, not something we invented. You store the graph as objects and lists of connections: one record per thing, and one record per connection, filed under the thing it belongs to. No traversal engine anywhere in it. Large systems have stored graphs on ordinary databases this way for years.
Ours runs on Cassandra, which is very good at exactly the thing we need. Give it a key and it hands back everything filed under that key from one place on disk. So a table’s columns live together under the table. An asset’s upstream connections live together under the asset. Expanding a lineage node becomes one read of one key.
We did not get this for free, and it is worth being straight about the two costs. We store each connection twice, once under each end, because “what columns does this table have” and “what table owns this column” are different keys. Writes cost more so that reads cost less, which is the right trade for a catalog, where reads vastly outnumber writes. And we now maintain our own indexes instead of declaring them and forgetting about them. Nothing is hidden anymore, which also means nothing is free. That is a fair price for being able to read our own data.
Proving it before we bet on it
A design that reads well on a whiteboard is not a reason to touch a customer’s data. So before we committed, we spent the time trying to break it.
First, head to head against the old path: fifty concurrent workers for an hour on identical hardware, the same mix of writes, reads, and lineage on both. The new path cleared roughly thirteen times as many requests at essentially the same success rate, on about half the heap. Treat a multiple that large with suspicion, and we did. It is not one system being thirteen times quicker at the same work. It is a queue: under fifty concurrent writers, the old path’s locking turned into waiting, and waiting has no ceiling. The honest reading is that the old path had a throughput limit we hit regularly, and the new one did not reach its limit here. The number we actually trust came later, from real customer traffic.
Speed was the easy half. Correctness was the half that could quietly ruin someone’s catalog, so we tested it harder. Round-trip checks wrote every data type we store, from strings and numbers to arrays, maps, enums, nested structs, classifications, and relationship edges, then read them back to confirm nothing changed shape. We sampled thousands of existing records and checked that not one connection had gone missing. We ran the full REST and SDK suites against the new backend before it went anywhere near a customer.
And the part that actually earned our trust: the experiments argued back. A pre-production run surfaced a sync bug where connector workflows finished but asset counts did not line up between the two stores. We traced it, fixed it, and confirmed the counts matched. A test that only ever confirms your design is not a test. These ones found real defects, which is the only reason we trusted the checks that passed.
Scaling it to 600+ live customer catalogs, with an agent
Designing the thing and proving it in a lab took a few months. Doing it to 600+ live customer catalogs, each one somebody’s working Monday, was a different problem, and it is the one we built an agent for.
The rule we set first: the risky work is deterministic, the judgment is the agent’s. Script is the engine, agent is the pilot. A plain migrator does the dangerous part, reading the old store, decoding it, writing the new layout, reindexing, validating, and flipping one config setting, and it does exactly the same thing every time. We deliberately kept that script out of the model’s hands, because copying a customer’s data is not a place for a model to improvise.
Everything around the script is where the agent lives. Each catalog gets a ticket. An agent picks it up, connects to that one catalog and nothing else, and works a fixed sequence: confirm this is the right tenant and it is healthy, back it up, rehearse with a dry run, copy, switch, verify against the real product endpoints, and report. What makes it an agent and not a cron job is that each of those is a decision, not a step. Is this the tenant the ticket names? Is it on the right release channel, with disk headroom, inside its maintenance window? Did the script finish, stall, or fail? Retry, run a narrower repair, stop, or recommend a rollback?
Failure is where it earned its keep. When a run broke, the agent pulled diagnostics from across the stack and used a model to classify what went wrong and what to do next. In one real run it read a batch of missing permission documents as a validation mismatch, recommended a targeted re-index instead of a full redo, and flagged the search mappings for a human to check. It was not always right at first. Early on it blamed the search layer for what was really the database under pressure, and we had to feed it better diagnostics and sharpen the prompts before its calls were worth trusting. That is the honest center of an AI-native story: the agent got good because we treated its wrong answers as bugs to fix, not magic to admire.
And it wrote everything down. Every phase, every failure, the counts on both sides, its diagnosis and confidence, the action it took, and whether it wanted a human, all posted back to the same ticket. The audit trail was not something anyone remembered to keep. It was a side effect of the agent doing its job. Months later, “what happened to this customer that Saturday” is a link, not an investigation. Verification that failed reverted the switch on its own, without waiting for someone to wake up.
What we did not hand over: humans decided when to open each ring, starting with our own environments, then the smallest customer catalogs, then outward in increasing size, and deliberately held the biggest and hottest catalogs for last. The agent removed the repetitive labor of five hundred migrations. It did not remove the engineering judgment, and we never asked it to.
What changed, for customers and for us
Two different stories here, and they are worth keeping apart.
For customers, the catalog got faster and more reliable. On the first catalog we cut over, measured across seven days before and seven days after on that customer’s real traffic: bulk ingestion 3.1 times faster, internal writes 7.1 times, search p99 4.7 times, overall p99 2.3 times. Request volume actually rose 36% in the same window, so this is a faster system carrying more load, not a quieter one. Four dedicated read-replica pods became unnecessary. And the error rate did not move off zero, which is the only condition under which any of the rest of it counts.
For us, it got cheaper and safer to run. Average heap per pod fell by roughly half. Cold start, across the first batch of migrated catalogs, went from about 38 seconds to about 7, for the boring reason that the old layer’s startup work simply does not happen anymore. And we can finally read our own data without a specialist and a specific library version on the classpath.
Where it got to
The first catalog moved in early April. By early August, more than 600 customer catalogs had migrated, carrying around 1.7 billion records between them. Four out of five live catalogs now run the new backend, and not one has been rolled back.
That last part sounds like bragging, and mostly it is not. We were not lucky. Rollback was cheap by construction, which is exactly why we never had to use it: we could afford to abort early and often, on a failed preflight or a closing window, instead of pushing through and hoping. Cheap reverse gears make for careful driving.
The catalogs left are the largest we run, and they are the genuinely hard ones. Bigger data means longer windows, hotter partitions, and less room for a run to go sideways. That work is ahead of us, not behind us.
What generalizes
Four things, if you are looking at a piece of your own stack and wondering.
Audit the questions before you defend the answer. We did not need a benchmark to justify this. We needed an inventory of what our reads actually asked for, and the answer was “one thing and its neighbours, every time.” Everything else followed from that.
Question the defaults you chose early. The tool we replaced was a fine choice on day one. The cost was keeping it for years without checking whether the shape of the problem had changed underneath us. It had, and the usage was saying so long before we looked.
Do not break the interface. We wanted a cleaner layout than our existing abstraction allowed, and we gave that up on purpose. Keeping the interface unchanged turned a rewrite into a config flag, and gave every catalog an independent, one-setting revert. That revert is what let the migration agent abort early and often instead of pushing through and hoping.
When you automate something risky, split the mutation from the judgment. The data movement was deterministic and identical every time. The agent owned the operational calls around it: whether to start, whether the environment was safe, how to read a failure, whether to retry, whether verification was enough. Keep the dangerous part boring, and let the agent be the pilot, not the engine.
Frequently Asked Questions
Why remove the graph database instead of replacing it with a faster one?
A graph engine earns its cost when you traverse, sweeping the whole graph for paths of any length that match a filter. Our reads were never that shape. Opening an asset, decorating search results, loading a glossary term's linked assets, even AI retrieval, are all 'given this one thing, what is directly attached to it', which is a lookup by key. Swapping one graph engine for another would have kept a capability we never used and kept paying for it in memory and startup time.
How does lineage work without a graph engine?
The graph is stored as objects and lists of connections: one record per thing, and one record per connection filed under the thing it belongs to. On Cassandra, giving it a key returns everything filed under that key from one place on disk, so a table's columns live together under the table. Expanding a lineage node becomes one read of one key, and walking further is another read.
What did the new design cost you?
Two things, both deliberate. Each connection is stored twice, once under each end, because 'what columns does this table have' and 'what table owns this column' are different keys, so writes cost more in exchange for cheaper reads. And indexes are now maintained explicitly rather than declared and forgotten, so nothing is hidden and nothing is free.
How do you migrate hundreds of live customer catalogs safely?
Split the mutation from the judgment. A plain migrator does the dangerous part the same way every time: read the old store, decode it, write the new layout, reindex, validate, flip one config setting. An agent owns everything around it, working one ticket per catalog through a fixed sequence of preflight, backup, dry run, copy, switch, verify and report, deciding at each point whether to proceed, retry, repair or stop, and writing its reasoning back to the ticket.
How much faster is it in practice?
On the first catalog measured across seven days before and after on real customer traffic: bulk ingestion 3.1 times faster, internal writes 7.1 times, search p99 4.7 times, overall p99 2.3 times, with request volume up 36% in the same window and no change in error rate. Average heap per pod fell by roughly half and cold start went from about 38 seconds to about 7.
