I moved us away from Tableau towards a Python based stack for processing, in-memory structures for queries on data relevant in our typical time windows, and on demand retrieval from cloud into the same data structures for historic data.
We were trying to handle 200m data points per day into a SQL database when I joined, and then display via Tableau for time and location series slicing in what were particularly complicated workbooks. To say that it was a terrible approach is an understatement.
Adding detailed geospatial and meteological data to the mix, matching with IoT sensor data, other ETL data from.dispatch systems, and many more has basically required some custom thinking to make it all work in an efficient manner.
My personal bug bear is that I still want great looking easy configuration for front end display of analytics dashboards. It becomes far harder to achieve when you go sufficiently far down the rabbit hole. I'm currently leaning towards python driven dashboards since we can connect them to a REST api which will talk to our data structures and keep everything humming along.
Note: not a single database in use here - though the timeseries sensor data could use one.
Yeah, time-series on that scale has unique requirements that can really cripple most DBs.
I'm currently leaning towards python driven dashboards since we can connect them to a REST api which will talk to our data structures and keep everything humming along.
Give Bokeh a try if it's not already on your radar - I've found it to be really useful for composable & dynamic python dashboards: https://docs.bokeh.org/en/latest/index.html.
Comments
I moved us away from Tableau towards a Python based stack for processing, in-memory structures for queries on data relevant in our typical time windows, and on demand retrieval from cloud into the same data structures for historic data.
We were trying to handle 200m data points per day into a SQL database when I joined, and then display via Tableau for time and location series slicing in what were particularly complicated workbooks. To say that it was a terrible approach is an understatement.
Adding detailed geospatial and meteological data to the mix, matching with IoT sensor data, other ETL data from.dispatch systems, and many more has basically required some custom thinking to make it all work in an efficient manner.
My personal bug bear is that I still want great looking easy configuration for front end display of analytics dashboards. It becomes far harder to achieve when you go sufficiently far down the rabbit hole. I'm currently leaning towards python driven dashboards since we can connect them to a REST api which will talk to our data structures and keep everything humming along.
Note: not a single database in use here - though the timeseries sensor data could use one.
What do you work on? And where? That all sounds super interesting.
Yeah, time-series on that scale has unique requirements that can really cripple most DBs.
Give Bokeh a try if it's not already on your radar - I've found it to be really useful for composable & dynamic python dashboards: https://docs.bokeh.org/en/latest/index.html.