Comment on page
🍃
Snowflake Eco Mode
Jul 26, 2023
Author: Jacqueline Cheong
Eco Mode - minimize time and maximize resource utilization
Eco Mode is an advanced setting that allows customers to define peak and off-peak hours and configure various sync time intervals.
We built Eco Mode with startups in mind, especially startups that have time-of-day traffic patterns, analytics dashboards that are only needed on certain days/hours, and workflows that are only required during business hours.
Take Company ABC that only offers customer service Monday-Friday from 8am to 6pm, and doesn’t respond to support tickets at any other times. For the ‘support_ticket’ table, Company ABC might set the following rules:
- Peak hours: Monday-Friday 8am-6pm, flush every 10 seconds
- Off-peak hours: All other times, flush every 6 hours
Eco Mode helps customers lower their data warehouse compute costs. Let’s use Snowflake’s virtual data warehouse (vDWH) as an example.
Snowflake’s vDWH plays a key role in data analysis, data ingestion, scheduling tasks and others. From our discussions with customers, about 70-90% of their Snowflake costs comes from compute, which includes data ingestion.
Snowflake’s vDWH is billed based on time utilization (down to the second), regardless of usage capacity. So running on 100% capacity for 1 minute costs the same as running on 1% capacity for 1 minute. As you can imagine, for startups that have peak hours where critical workloads are running and off-peak hours where there is very little data being processed, having the same sync method at all times is not the most cost efficient.
In our example above, Company ABC can minimize time utilization of their Snowflake vDWH when the ‘support_ticket’ table is not being used. In this example, ‘support_ticket’ table is set to off-peak ~70% of the time, which results in Snowflake vDWH cost savings of 65-70%!
Customers can define peak and off-peak hours along with flush preferences from their dashboard. This can help minimize time utilization of Snowflake’s vDWH during off-peak hours.
As a quick refresher – Artie flushes data to the DWH based off of three variables (you can read about how Artie Transfer works in our previous blog):
- 1.Time
- 2.Size of the in-memory database based on the # of de-duplicated rows
- 3.Size of the in-memory database based on the # of bytes
Whichever one of the three rules is met will trigger a flush cycle. With Eco Mode, we are providing granularity to control the Time rule for peak and off-peak hours.
Here, Company ABC will define their peak hours as:
- Day of week (M-F)
- Timezone (PST)
- Time of day start ( 9AM)
- Time of day end (5 PM)
You may be wondering if increasing the flush time interval for off-peak hours will negatively impact source database performance, such as causing replication slot overflow from the accumulation of CDC logs. Turning Eco Mode on will have zero impact to the database due to the way Artie is architected.
Even during off-peak periods, Debezium continues to stream CDC logs to Kafka. We rely on Kafka’s buffer and Artie Transfer’s in-memory database to temporarily hold the accumulated logs.
Snowflake Eco Mode will be available soon on Artie Cloud! Request early access here or reach us directly at [email protected].