The short of it… we’re building a home for data teams. And everyone’s invited! 🏠
And now, here’s the long of it… Like most good things in life, Atlan too started with a thought. An observation. A realization.
The way teams work with data is broken. It’s outdated and unfair.
We know this through countless hours of research, the time we’ve spent talking to data teams of all shapes and sizes, and our own experience of working with data. And while people may think that data jobs are among the most “sexiest jobs of the century”, the thing is…
Data teams today are overwhelmed and under enormous pressure
Just ask any data analyst, data scientist or chief data officer, and they’ll tell you about all the business expectations riding on them. All the sleepless nights they’ve spent trying to make disparate data sets talk to each other, solve for missing values in columns, solve for getting access to data, or answer the question that all business managers ask—how can your data add value to my business?
The truth is that while companies are investing significantly in big data and in setting up data teams, they are struggling to reap the returns.
Need the numbers? Here you go: companies were expected to invest $114 billion in big data in 2018, up from $31 billion in 2013. Yet only 27 percent of company executives described their big data projects as successful.
To win with data, diverse individuals need to come together
They say it takes a village to raise a child. This could not be more true than it is for data projects.
It takes an entire team of people with diverse skill sets to come together to make one data project successful—from engineers solving for scale to scientists building models, business users who best understand the problem, machine learning engineers, data analysts and so on.
Look at the most successful data teams around the world… from the 400,000 “Humans of Data” at NASA (engineers, technicians and scientists) who spent years studying the lunar surface 🛰️ before humankind’s first step on the moon, to scientists from 20 research centers across six countries coming together to map the complete genetic blueprint of a human being.
The world has so, so many great stories of what data teams can achieve when they come together. But the problem is that all these Humans of Data have their own DNA and tool preferences, leading to what we call…
Collaboration overhead—too many tools, workflows and variables!
And it’s not that companies aren’t investing in data lakes, or infrastructure or tools. Most of the teams we work with routinely use S3, Azure, Hadoop, GitHub, Jupyter and more as part of their stacks. Yet their teams find it extremely difficult to find, access or share their data assets. And you can just forget about context and metadata.
And what about the time data peeps spend on high-value work vs. repetitive tasks aka human grunt work? A news report once pegged the time that data scientists spend on simply getting data ready for analysis at 50-80 percent. How can we reduce this “data debt”?
We might need to…
Rethink every part of the human-data interaction.
Why do we say this? First, a little back story. We started out as a data team ourselves while building SocialCops.
As some of you may know (and were even a part of ❤️ ), in the beginning we were just a small team of data scientists, engineers and economists who came together to solve big problems facing humanity by using data. As we got our hands dirty project by project, things started to break and chaos reigned.
Now, we were using the best-in-class open-source tools available at the time. That’s when we realised, these were human problems… not technology ones.
An engineer pushed the wrong data file to production. An analyst used the wrong variable in the analysis. An API spec was changed by a client without telling us. A data scientist left and rerunning the model they built was crazy. Our ML engineers struggled to explain how a model worked to the business users. Business users complained that the data team was a black box. Analysts complained the data pipeline was a black box. Engineers complained that the client problem statement was not properly communicated to them.
That’s how it all started. We began to build small tools to help our own team collaborate effectively.
These started with small fixes—like opening up the data pipeline visually (we used Airflow) so that analysts could troubleshoot failures and to help build trust among the team when something failed. Automating basic alerts and notifications. Internal data discovery and access. Knowledge management. And so on.
By making these small, incremental changes in the human-data interactions for our own team, we were able to do big things. Our team was performing at 2x the quality, as we needed far fewer iterations to push a final product… in 50 percent of the time, thanks to the power of automation… and with 1/3rd the resources, given reduced engineering time.
And we went on to do some incredible work, including building the DISHA platform in record time for a platform of its size and scale, and being chosen as a global partner for the United Nations.
And that’s when we began to think. What if we opened up our tools to data teams around the world? Could we help them achieve their goals faster, just like we’d done with our own team… That’s how Atlan was born.
Introducing Atlan—a home for data teams.
Where you can bring together diverse data, tools and people to create a frictionless collaboration experience. Think of us as the glue—the collaboration layer where you can work with data and all your tools.
We’re trying to remove all the friction points between people and data by asking some fundamental questions.
For instance, code has a profile on GitHub—what would a “profile” of data look like? What if you could share data as easily as a Google Sheets link, without worrying about the size or format? Or what would a data versioning and approval workflow look like? What if data scientists could acquire external data within minutes, instead of the months it takes right now?
For the past 18 months, we’ve been working very closely with our beta customers to validate all these ideas and ensure that we’re adding value to these teams.
And we’re happy to share that…
Our customers are already winning with Atlan
It makes us so happy to share that today, more than 200 teams in 50+ countries rely on us to collaborate frictionlessly on data projects.
Here are some of their stories:
- “Your product saves 1.5 hours of my day, daily. Thank you!”
- “With Atlan, we have built a predictive sales model to increase leads by 30%.”
- “Atlan is becoming the data spine on which data products in our company are built.”
If you’d like to experience data in a whole new way, come join us at www.atlan.com. We can’t wait to see you!
With ❤️ from Team Atlan
P.S. In case you’re wondering, our name, Atlan, is inspired by one of the earliest big data projects—Matthew Fontaine Maury’s work in the 1880s to map winds, currents and more for every 5-degree latitude-longitude square of the Atlantic Ocean. With daily data from 17 nations and thousands of ships, Maury’s maps made ocean travel safer, reduced sailing time by up to 50 percent, and saved millions of dollars. Atlan is also derived from the name of Atlantis—the world’s most technologically advanced mythical civilization. The name reflects what we believe at Atlan; we believe in human achievement and progress, and in the power of data to get us there.
Oh, and Atlan is pronounced just like Atlas… only with an ‘N’ at the end! 😎