• kstore-core : the core engine
  • kstore-hdfs : HDFS support additional module
  • kstore-s3 : S3 support additional module
  • kstore-sample : samples using KStore

Main abstractions

Abstraction of a table, a KStore owns a schema defined as a list of columns, and stores data in buckets. A Device is associated to the KStore to define the physical storage.
Defines an abstract storage of data rows.
Implementation of a Bucket.
Column & ColumnType
Abstraction of a column, mainly define by a type.
A vector allowing reading one line of data when reading buckets.
The physical abstraction of a storage, which may be file system, hadoop, s3, azure, whatever…
Property file configuration.

Default implementations

A default implementation of a KStore.
A default implementation of a Column.
A default implementation of a Line.
A device implementing regular file systems.
HDFSDevice (in kstore-hdfs)
A device implementing HADOOP file system.
S3Device (in kstore-s3)
A device implementing S3 file system.

A 5 minute start guide

// Create a new schema with columns and types
List<Column> schema = Lists.newArrayList(new DefaultColumn(ColumnType.STRING));

// Build a new kstore in a folder. Right now, we're using a temporary directory
Path temp = Files.createTempDirectory("kstore");
KStore kStore = new DefaultKStore("My KStore", schema, temp.toString());

// Create a new bucket to store / retrive our data, and
// add two lines
Bucket bucket = kStore.newBucket();
	.add(0, new Object[]{"France"})
	.add(1, new Object[]{"Spain"})

// Prepare to traverse our bucket. We want all lines from 0 to 2
Range range = new Range(0, 2);
// When traversing, we want to retrieve the column 0
Line line = new DefaultLine(0);
// Iterate through the bucket and print each line
bucket.readLines(line, range, (int rowId, Line l) -> showMeTheLine(rowId, l));
// Free resources

Other operations

Saving / Loading (); // save the store metadata
bucket.load (); // load the store metadata
Deleting lines
bucket.deleteRowId(2); // delete the line with identifier 2
bucket.rollback(); // cancel any operation since the last commit

Parquet vs K-store benchmark

for reading and writing operations : access here.