Alinta Energy has progressively swapped out most of the Azure-powered services that previously fueled its data structure with Databricks, a move it claims has significantly reduced its operational expenses.
Alinta’s Jake Roussis.
Jake Roussis, the lead data engineer, disclosed at a Databricks data intelligence day in Melbourne that these modifications had led to a “40-to-50 percent decrease in production platform costs over the last year, [equivalent to] over $1 million in savings”.
The “gentailer” – operating as both an electricity producer and seller – had initially utilized Azure Synapse, Microsoft’s comprehensive data platform and analytics services suite, for data transformation, querying, processing, and data delivery to users.
While Databricks was previously present in a limited capacity for specific data processing and analytics tasks, alongside Tableau and Power BI for analysis and reporting, the majority of these components have now been supplanted by Databricks through a dedicated re-platforming initiative aligning with the engineering department’s pursuit of enhancements in “cost, observability, reliability, and performance,” as mentioned by Roussis.
“There’s only a small fraction [of our data structure] that’s not [Databricks], and we’re striving to transition every aspect into [Databricks],” he emphasized.
The gradual elimination of Azure services has resulted in reduced expenditure, Roussis highlighted.
“I prefer not to delve too deeply into Synapse – as we’ve moved away from it – but it’s worth mentioning the 40 percent reduction in costs we achieved by discontinuing its use and migrating to Databricks,” he expressed.
“Another significant achievement for us recently [was] the adoption of serverless SQL warehouses. Databricks SQL, by itself, is incredibly potent; however, the switch from a conventional warehouse to serverless SQL warehouses has yielded an additional 38 percent in annual savings. That translates to about $300,000 per year, which is a substantial sum,” he elaborated.
Roussis noted that the deployment of serverless SQL necessitated some “rightsizing” of data workloads to realize the cost efficiency.
“We needed to ensure that when users executed queries … we had the optimal cost efficiency within the serverless SQL infrastructure, a task that wasn’t overly challenging to accomplish,” he remarked.
Roussis highlighted the refined alerting configuration in Databricks compared to native Azure setups, noting that alerts could be seamlessly directed to Alinta’s IT operations platform, PagerDuty.
He also underlined the benefits of “real-time query monitoring” available in Databricks.
Prior to the re-platforming initiative, Roussis recalled instances of “resource depletion” being a common issue, occurring “every couple of weeks.”
“Imagine it’s 3am on a Tuesday morning. Your phone starts buzzing, you groggily reach for it, only to realize that a crucial [data] pipeline has failed,” he recounted.
“The day before everything was functioning fine, pipelines were running smoothly, the system was flawless.
“After 20 minutes of investigation, you finally pinpoint the issue: a user initiated a badly-written query at 10pm, consuming all available resources, halting operations since then.
“That was a recurrent scenario when I initially joined Alinta.”
Roussis mentioned that previously he could monitor a query in real-time but lacked post-execution visibility. “I couldn’t ascertain the aftermath of its execution,” he observed.
“Now, I can pinpoint the query’s duration, cost, and comprehend the query plan,” he stated.
“Databricks excels in this aspect as well, attempting query optimization autonomously. However, sometimes users formulate subpar queries.
“Consequently, we can scrutinize executed queries, deconstruct them, and aid users in enhancing their practices. This empowers our business to thrive,” he added.
For instance, Roussis mentioned that a calculator for electricity “pricing variation events” now completes its runtime in “less than 15 minutes,” down from over an hour previously.
“We simply revamped it from poorly-scripted Python code to exclusively Databricks SQL using dbt,” he described. Data build tool or dbt is employed to preprocess raw data for analytical purposes.
Natural language querying
Alinta Energy has initiated its first application of AI/BI Genie, Databricks’ innovative AI functionality designed to facilitate interaction with data using natural language for business teams.
“We left no stone unturned in comprehending and swiftly grasping its capabilities, limitations, and areas of excellence,” Roussis highlighted.
“In late 2024, we successfully executed a proof-of-concept presented to the Alinta Board, offering insights on retail customers, with the aim to equip call center agents with this tool,” he shared.
“If a customer calls, [a call center agent] can seek information from the AI to understand the customer’s needs before engaging.
“This served as a valuable learning curve for us. Subsequently, we are now focused on establishing processes to expedite AI deployment, innovate new AI solutions, and continuously introduce more models in production.
“However, it is imperative to have robust safeguards in place, ensuring that operations align with our expectations to prevent any misinformation from AI sources,” he asserted.
About Author
