gravitylabs / hpaste Goto Github PK
View Code? Open in Web Editor NEWHBase DSL for Scala with MapReduce support
Home Page: http://www.gravity.com/technology/
License: Apache License 2.0
HBase DSL for Scala with MapReduce support
Home Page: http://www.gravity.com/technology/
License: Apache License 2.0
It would be nice to have an "official" Scala 2.10 build of HPaste checked into one of the standard repos.
Sometimes you want to do the following:
HPaste assumes rows returned from Hbase are immutable and there is therefore no way to do this without injecting another step, with is to re-fetch the data from Hbase. This is in most cases a preferred way to do things, because in-memory state increases the chance of subtle bugs creeping into the system.
That said, there are performance critical hotspots where this needs to be supported.
The tentative way to do this is to simply provide a copy constructor for an HRow object, along with direct setters for column and row families. In other words, don't manipulate the state of the original object, instead provide a scenario where you can copy the object and manipuate the state during the copy operation. This will reduce the number of subtle bugs that can arrive from doing things like caching ancillary results in HRow objects.
Say funcA and funcB create operations and you want to defer their execution until funcC.
PutOp should support + so this can be done.
Its signature should be such that only PutOps from the same table can be thus chained (otherwise it breaks hbase's concept of a batched put).
There's two ways of dealing with families in HPaste. First, you can define a family with strongly-typed columns inside of it. This is for a table-like scenario.
val myfam = family[String,String,Any]("myfam")
val mycolumn1 = column(myfam,"mycol1",classOf[Int])
val mycolumn2 = column(myfam,"mycol2",classOf[Int])
In that scenario, you typically access columns individually.
In the second scenario, you're dealing with the family as a map with dynamic keys and values.
val myfam = family[String,String,String]("myfam")
These two cases can walk over one another because the serializer is not polymorphic. In other words, you cannot treat a family with strongly typed columns as a Map, unless all of the columns and column keys have the same types.
We haven't had a use case where it's compelling to have polymorphic serialization, and there's plenty of scenarios where that is too complex, so we probably won't support it. So we should separate families into "tabular" families and "map" families. Or "dynamic" families.
Column families require type parameters, whereas columns do not. For example:
//Define a family whose name is a String, whose Key Type is a String, and whose Value is an Int
family[String,String,Int]("myfamilyname")
versus
//Define a column whose value type is a String
column(familyRef, "mycolumn", classOf[String])
The second syntax is cleaner because it makes everything explicit, esp if you use named parameters. So we'll upgrade families to use that syntax.
in the wiki under Data Retreival it's using the Query class, I'm assuming this should be updated to Query2 since Query is deprecated?
val dayViewsRes = ExampleSchema.ExampleTable.query.withKey(key).withColumnFamily(_.viewCountsByDay).single()
Based on the example provided, I carved out a sample executable and stumbling on the following error on execution
import com.gravity.hbase.schema.{ HRow, DeserializedResult, HbaseTable, Schema }
import com.gravity.hbase.schema._
import com.gravity.hbase.mapreduce._
import java.lang.String
import org.joda.time.{ DateMidnight, DateTime }
import com.gravity.hbase.mapreduce.{ HMapReduceTask, HJob }
import java.net.URL
import com.gravity.hbase.schema._
import WebCrawlingSchema.WebPageRow
import org.apache.hadoop.io.Text
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.HBaseConfiguration
import org.joda.time.{ DateMidnight, DateTime }
object HBaseConfig {
lazy val config: Configuration = {
val c = HBaseConfiguration.create
val localConfigPaths = List("resources/hbase-site.xml", "../resources/hbase-site.xml")
localConfigPaths.foreach((p) => {
val path = new org.apache.hadoop.fs.Path(p)
if ((new java.io.File(path.toString)).exists) {
c.addResource(path)
}
})
c
}
}
object WebCrawlingSchema extends Schema {
implicit val conf = HBaseConfig.config
class WebTable extends HbaseTable[WebTable, String, WebPageRow](tableName = "pages", rowKeyClass = classOf[String]) {
def rowBuilder(result: DeserializedResult) = new WebPageRow(this, result)
val meta = family[String, String, Any]("meta")
val title = column(meta, "title", classOf[String])
val lastCrawled = column(meta, "lastCrawled", classOf[DateTime])
val content = family[String, String, Any]("text", compressed = true)
val article = column(content, "article", classOf[String])
val attributes = column(content, "attrs", classOf[Map[String, String]])
val searchMetrics = family[String, DateMidnight, Long]("searchesByDay")
}
class WebPageRow(table: WebTable, result: DeserializedResult) extends HRow[WebTable, String](result, table) {
def domain = new URL(rowid).getAuthority
}
val WebTable = table(new WebTable)
class SiteMetricsTable extends HbaseTable[SiteMetricsTable, String, SiteMetricsRow](tableName = "site-metrics", rowKeyClass = classOf[String]) {
def rowBuilder(result: DeserializedResult) = new SiteMetricsRow(this, result)
val meta = family[String, String, Any]("meta")
val name = column(meta, "name", classOf[String])
val searchMetrics = family[String, DateMidnight, Long]("searchesByDay")
}
class SiteMetricsRow(table: SiteMetricsTable, result: DeserializedResult) extends HRow[SiteMetricsTable, String](result, table)
val Sites = table(new SiteMetricsTable)
}
object MainClz extends App {
val exTable = WebCrawlingSchema.WebTable.createScript()
println(exTable)
}
Error:
could not find implicit value for parameter kv: com.gravity.hbase.schema.ByteConverter[Map[java.lang.String,java.lang.String]]
[error] val attributes = column(content, "attrs", classOf[Map[String, String]])
[error] ^
[error] one error found
Right now maps are serializable and deserializable if they're imported as the generic interface: scala.collection.Map[X,Y].
This is to support both mutable and immutable Maps, which are useful in different circumstances. However, in Scala you typically import immutable.Map, so it's confusing to require the generic interface.
Because mutable and immutable Maps are stored in the same binary layout, it's simple for Hpaste to support all interfaces simultaneously. So we'll do that.
On a fresh SBT project including HPaste:
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.hadoop#hadoop-core;0.20-append-r1056497: not found
[warn] :: org.apache.thrift#thrift;0.2.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
Details as follows:
// HPaste version
val hpaste = "com.gravity" % "gravity-hpaste" % "0.1.11"
// SBT version
sbt.version=0.12.3
// Scala version
scalaVersion := "2.10.0",
Looks like this project hasn't had any updated since the end of 2013.
Is it still being used or maintained? The README.md says its being actively maintained...
a) because 0.92.0 is the version supported on Elastic MapReduce
b) because HBase 0.90.0 is built on top of:
<hadoop.version>0.20-append-r1056497</hadoop.version>
which isn't in public repos any more.
http://svn.apache.org/viewvc/hbase/tags/0.90.4/pom.xml?view=co&content-type=text%2Fplain
I'm confused by the schema design found in the HPaste test suite:
object ExampleSchema extends Schema {
//There should only be one HBaseConfiguration object per process. You'll probably want to manage that
//instance yourself, so this library expects a reference to that instance. It's implicitly injected into
//the code, so the most convenient place to put it is right after you declare your Schema.
implicit val conf = LocalCluster.getTestConfiguration
This approach is tightly-coupling a test Hadoop configuration into the schema object. Obviously this is fine for a project which will never be run on a real cluster, but what's the recommended approach for a schema which will be used "in anger", i.e. needs to support LocalCluster.getTestConfiguration
and the real Hadoop cluster's Configuration
? (Bearing in mind that implicit values in Scala can't cross object boundaries.)
The HTablePool takes resources (and creates a lot of thread pools). Make the size configurable.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.