I would be curious to know if they evaluated any cloud-based data stores or streaming services from AWS or GCP before deciding to building this from scratch. It seems like a common set of requirements for event analytics pipelines.
Hi, Jin here from Amplitude. The real-time data store is part of a bigger columnar store we built last year called Nova (https://amplitude.com/blog/2016/05/25/nova-architecture-unde...). In designing Nova, we’ve looked at many existent solutions including Amazon Redshift and Google BigQuery, but none of them sufficiently supports all our use cases. You can read more in the linked blog.
I read that and your motivations for building nova align very well with bigquery. E.g. immutabled (big query was append only), felaxability (break out of SQL with dataflow).
Comments
I would be curious to know if they evaluated any cloud-based data stores or streaming services from AWS or GCP before deciding to building this from scratch. It seems like a common set of requirements for event analytics pipelines.
Hi, Jin here from Amplitude. The real-time data store is part of a bigger columnar store we built last year called Nova (https://amplitude.com/blog/2016/05/25/nova-architecture-unde...). In designing Nova, we’ve looked at many existent solutions including Amazon Redshift and Google BigQuery, but none of them sufficiently supports all our use cases. You can read more in the linked blog.
I read that and your motivations for building nova align very well with bigquery. E.g. immutabled (big query was append only), felaxability (break out of SQL with dataflow).
It uses Amazon Redshift under the hood.
We don't use Redshift to run our queries. Nova, our customized columnar store, is designed to handle more specific use cases. You can read more here, https://amplitude.com/blog/2016/05/25/nova-architecture-unde....