Here is how I think about this, there are three high level paradigms you see for processing:
1. Request/response (e.g. most UI actions, REST, etc)
2. Stream (e.g. subscribing to a Kafka topic)
3. Batch/periodic (e.g. Hadoop, DWH)
You actually need all of these at least somewhere in a company, and they each have their place.
The dividing line for request/response is that someone is waiting on the other end and that if the request fails you can just show them an error and continue on your way. So when a web service gets overwhelmed it usually just times out requests.
Consider an e-commerce site as an example:
1. Request/response
Displaying a product
Making a sale
2. Stream or batch
Restocking
Logistics and shipping
Price adjustments
Analytics
Product catalog import
Search index updates
Etc
The later category is asynchronous so it can be done in a batch fashion (once an hour or day) if latency is not a concern, or in a streaming fashion if it needs to be faster.
Comments
Here is how I think about this, there are three high level paradigms you see for processing: 1. Request/response (e.g. most UI actions, REST, etc) 2. Stream (e.g. subscribing to a Kafka topic) 3. Batch/periodic (e.g. Hadoop, DWH)
You actually need all of these at least somewhere in a company, and they each have their place.
The dividing line for request/response is that someone is waiting on the other end and that if the request fails you can just show them an error and continue on your way. So when a web service gets overwhelmed it usually just times out requests.
Consider an e-commerce site as an example: 1. Request/response Displaying a product Making a sale
2. Stream or batch Restocking Logistics and shipping Price adjustments Analytics Product catalog import Search index updates Etc
The later category is asynchronous so it can be done in a batch fashion (once an hour or day) if latency is not a concern, or in a streaming fashion if it needs to be faster.