The faq mentions there are no security features at the moment, but do you have any thoughts as to what the security goals are? In particular, will there be cell level security a la accumulo or support for data at rest?
We haven't scoped out the security features. Cell level security can be difficult to implement efficiently, but if we see enough demand for it, I could imagine it happening.
My guess is that the first pass will be table and column level authorization, plus of course strong authentication. Row/cell/predicate-based security could be added in a later release, but it's a feature that's less commonly required.
As for encryption at rest, I imagine that will also be fairly high priority as we move towards GA or the first few releases after GA. But again, we haven't done the scoping exercise yet, so I'm cautious to throw out dates :)
If you're interested in helping to contribute either feature, let us know! kudu-dev@googlegroups.com
The insert latency seems to be related to the random read latency (seems that the unique key constraint has that effect). Do you have some data on the insert latency distribution?
We're working on running some more thorough YCSB benchmarks, but here are some of the percentiles on the uniform "workload A" running on a 9 node cluster for 1 hour:
9 worker nodes plus one master node. Each with
48GB of RAM, 3 data disks, and dual 4-core Xeon L5630
processors at 2.13GHz.
So, pretty old stuff. But note that the 100G dataset fit comfortably in RAM. If you don't fit in RAM, then the latencies are obviously going to be much worse on spinning disk. Our bet is that for OLTP workloads, you're probably going to be on flash or mostly RAM-resident these days, given the dropping prices and increasing speeds of solid-state storage.
Comments
Todd from the Kudu team here. If anyone has any questions, feel free to ask them here, will try to check back throughout the day.
Should add that those looking for a technical deep dive might enjoy our draft paper: http://getkudu.io/kudu.pdf and/or browsing our source: http://github.com/cloudera/kudu
The faq mentions there are no security features at the moment, but do you have any thoughts as to what the security goals are? In particular, will there be cell level security a la accumulo or support for data at rest?
We haven't scoped out the security features. Cell level security can be difficult to implement efficiently, but if we see enough demand for it, I could imagine it happening.
My guess is that the first pass will be table and column level authorization, plus of course strong authentication. Row/cell/predicate-based security could be added in a later release, but it's a feature that's less commonly required.
As for encryption at rest, I imagine that will also be fairly high priority as we move towards GA or the first few releases after GA. But again, we haven't done the scoping exercise yet, so I'm cautious to throw out dates :)
If you're interested in helping to contribute either feature, let us know! kudu-dev@googlegroups.com
Hey Todd,
The insert latency seems to be related to the random read latency (seems that the unique key constraint has that effect). Do you have some data on the insert latency distribution?
Thanks, Cosmin
We're working on running some more thorough YCSB benchmarks, but here are some of the percentiles on the uniform "workload A" running on a 9 node cluster for 1 hour:
Throughput: 28280 ops/sec Read: 2821us avg, 467us min, 3519us 95p, 6843 99p Update: 1688us avg, 714us min, 1983us 95p, 8855us 99p
Workload D, which has some inserts (and reads recently written data):
Throughput: 36286 ops/sec Read: 1765us avg, 491us min, 2537us 95p, 4259us 99p Insert: 1614us avg, 838us min, 1595us 95p, 11575us 99p
Hope that helps. I'll try to push our latest YCSB bindings to github later this afternoon/evening if you'd like to reproduce on your own.
That looks excellent. What hardware?
9 worker nodes plus one master node. Each with 48GB of RAM, 3 data disks, and dual 4-core Xeon L5630 processors at 2.13GHz.
So, pretty old stuff. But note that the 100G dataset fit comfortably in RAM. If you don't fit in RAM, then the latencies are obviously going to be much worse on spinning disk. Our bet is that for OLTP workloads, you're probably going to be on flash or mostly RAM-resident these days, given the dropping prices and increasing speeds of solid-state storage.
Does Kudu colocate data sets with identical keys? If so, are there plans to have Impala take advantage of this?