Over time, Alinta Energy has transitioned the majority of the Azure-based solutions that previously powered its data framework to Databricks, resulting in significant reductions in operational expenses.
Featuring Jake Roussis from Alinta.
Jake Roussis, the lead data engineer, shared during a Databricks data intelligence event in Melbourne that the modifications led to a “40-to-50 percent decrease in platform expenses over the past year, resulting in over $1 million in savings.”
The company, which operates as both an electricity producer and retailer, had been utilizing Azure Synapse, Microsoft’s comprehensive data platform, for data transformation, querying, processing, and data delivery to users.
Previously, the usage of Databricks was limited to specific data processing and analytics tasks, alongside Tableau and Power BI for analysis and reporting purposes.
Following a concentrated re-platforming effort, most of these components have now been substituted by Databricks, aligning with the engineering team’s strategy to enhance “cost, observability, reliability, and performance,” as stated by Roussis.
Roussis mentioned, “Only a small portion of our data framework remains outside of Databricks, and we are actively working to transition everything into Databricks.”
Adopting Databricks over Azure services has led to cost savings, according to Roussis.
He added, “While I don’t want to dwell on Synapse, since we have moved past it, it is worth noting the 40 percent cost decrease observed by shutting it down and migrating to Databricks.”
“Another significant achievement for us was the implementation of serverless SQL warehouses. Transitioning to Databricks SQL from traditional data warehouses saved us an additional 38 percent annually, translating to about $300,000 per year,” Roussis explained.
Roussis highlighted that the move to serverless SQL was accompanied by optimizing data workloads to facilitate cost reduction.
Explaining the past challenges, Roussis recalled instances of resource depletion occurring on a regular basis before the re-platforming.
“Picture a scenario at 3am on a Tuesday morning where your phone starts buzzing with notifications about a failed critical data pipeline,” he illustrated.
“Previously, I could observe running queries but was unable to track the aftermath. With Databricks, I can monitor the query’s duration, cost, and understand its plan,” Roussis added.
Referring to improved analytics capabilities, Roussis mentioned the efficiency in adjusting poorly written queries and how it has enabled the business to enhance its operations.
For instance, he shared that with the conversion of a pricing variation calculator to Databricks SQL using dbt, the runtime reduced from over an hour to less than 15 minutes.
Enhancing natural language query capabilities
Alinta Energy embarked on integrating AI/BI Genie, Databricks’ generative AI feature, to facilitate natural language interactions with data for business teams.
Roussis explained, “Through a proof-of-concept in late 2024, we demonstrated to the Alinta Board how call center agents could leverage AI insights to understand retail customers better and address their needs effectively.”
He emphasized the need for robust processes to ensure the accuracy and reliability of AI outputs to prevent informational discrepancies.
About Author
