Toru Maesaka

It’s been almost two weeks since I left mixi where I literally had a great time for the past 3 years and 8 months. Here’s what I’ve been up to lately.

• Having dinner every night with life-long friends that I made through working at mixi.
• BBQ in the Mountains of Chiba with friends.
• Riding my new road bike (bicycle).
• Day trip to Kamakura.
• Studying relevant technologies for my next gig.
• Searching for a new apartment to rent in Tokyo.
• Throwing things out to make moving easier.
• Cancelled my gym membership due to moving.
• Writing final evaluation for Djellel on Google Summer of Code.
• Retired my iPhone 3G that I’ve been using for over two years.

My free time comes from a surprise stack of unconsumed vacation time that I had left at mixi. You see, I hardly used my paid leave while I was at mixi due to working on exciting projects and having fun people around me. Time just flew.

As a replacement for my iPhone 3G, I bought a BlackBerry Bold 9700 which now makes me a BlackBerry + Android user. I don’t have anything against the iPhone as a product (although I have improvement suggestions) but I wanted a change after using it for over two years. I’m also hoping that the new iPod Touch will come with a camera which eliminates my desire for an iPhone 4. I still actively use my HTC Magic for mobile web surfing and tethering my b-mobile on the field.

My opinion on the BlackBerry so far is that it’s a powerful email machine combined with Google Sync and Google Contacts (now part of Gmail). I’m certain that I’ve been replying to more emails while I’m out than when I was using the iPhone. The web browsing experience on the 9700 is poor but I have an Android to fulfill that void.

Anyhow, I just wanted to let my readers know that I’m doing fine. My vacation ends next week so until then, please feel free to ping me for a hand on your project, drinking or whatever :)

Toru Maesaka random life, phone

Crash Safety is a big deal in the database league. Lack of durability can lead to all sorts of terrible things upon a catastrophic event. Many projects, especially in the so called NoSQL world compromises crash safety in return for higher QPS. The argument there is that the availability of the overall system should be accomplished by replication since a database server can’t be rescued if the physical disk breaks. I happen to agree with this philosophy but I am also aware that this isn’t a correct answer for everyone. So, what will I do with BlitzDB?

Several relational database hackers have pointed out that BlitzDB isn’t any safer than MyISAM since it doesn’t guarantee crash safety. This is currently true but I plan on making BlitzDB much safer than MyISAM by providing following features.

1. Auto Recovery Routine (startup option)
2. Tokyo Cabinet’s Transaction API (table-specific option)

The second feature above would actually guarantee BlitzDB to be crash safe (especially combined with auto recovery) but I won’t get into depth in this post since this topic deserves a blog post of it’s own. Let me just state that this feature will be provided in a form like this:

CREATE TABLE t1 (
  a int PRIMARY KEY,
  b varchar(256)
) ENGINE = BLITZDB, CRASH_SAFE;

From here on, I’ll cover how I plan on hacking auto recovery in BlitzDB.

Auto Recovery Challenges

As I blogged a while back, recovering Tokyo Cabinet is relatively simple. However, this is not a sufficient solution in BlitzDB since the data file (hash database that actually holds the rows) and the index file(s) are independent from each other. That is, the likelihood of the data file and the index file(s) to be inconsistent is very high after a crash. So, how can we hack on this? Pretty simple.

Indexes aren’t Important at Recovery Phase

Because BlitzDB logically separates the data file and it’s indexes, index files aren’t that important. If a server crash had occurred, BlitzDB could delete the index file(s) and recompute them from the data file. Needless to say, this process would involve a lot of random access and computation but it would not dominate the time space of the system since it’s a one-time cost. This approach however has one flaw in it such that the index files can’t be recomputed if the data file is broken or is unrecoverable.

Therefore to guarantee crash safety, BlitzDB must ensure that the data file is unbreakable. This is precisely where Tokyo Cabinet’s Transaction API comes in. I’m planning on using it to protect the data file from breaking. If the data file is protected, the table can be rescued. Simple!

So, that’s what I have in mind for making BlitzDB a safer engine. Unfortunately I can’t start hacking on it immediately since I have several bugs to fix first. Nevertheless I’m looking forward to start hacking on it. This challenge should be quite fun to tackle.

Toru Maesaka drizzle, oss blitzdb, hacking, recovery, storage

The flexibility to add table-specific options for things like compression, encryption and optimization can be useful to storage engine developers as this flexibility can open up new possibilities. Here’s what I’m talking about:

CREATE TABLE t1 (
  ...
) ENGINE = my_engine, MY_OPTION = your_arg;

Supporting this is relatively easy in Drizzle and this API feature (and a bit more) is available in MariaDB as well. Unfortunately Drizzle’s method to do this isn’t documented in the Wiki yet but it should be added when our Storage Engine API becomes stable (as in, no interface changes).

Implement StorageEngine::doValidateTableOptions()

Here’s the actual interface.

bool StorageEngine::doValidateTableOptions(const std::string &key,
                                           const std::string &state);

This function is called for each table options given at CREATE TABLE syntax execution. The first argument, key is a const reference to a string that represents the option name. The second argument, state represents the argument given for that option.

Therefore, given: COMPRESSION = YES_PLEASE, key would be “COMPRESSION” and state would be “YES_PLEASE”. The objective of this function is to check whether the key/state pair makes sense to your storage engine. If this function returns false, Drizzle will return an error for the CREATE TABLE query. Personally I think this interface can be improved to be a bit more Developer friendly, such as making life easier to validate numeric values without enforcing the developer to play around with the data. Saying that, given the pace that Drizzle is growing, this could be improved before we know it.

Access Options at StorageEngine::doCreateTable()

Here’s the actual interface for doCreateTable().

int doCreateTable(drizzled::Session &session,
                  drizzled::Table &table_arg,
                  const drizzled::TableIdentifier &identifier,
                  drizzled::message::Table &table_proto);

Given that the options were successfully validated, doCreateTable() is called next. In Drizzle, all information regarding a table (including options) is represented in a Google Protocol Buffer message. A reference to that message object is passed to doCreateTable() as the fourth argument so all you need to do is loop through the options list in the message object and extract what you need. Here’s a minimal example that only takes care of one option.

int n_options = table_proto.engine().options_size();
 
for (int i = 0; i < n_options; i++) {
  if (table_proto.engine().options(i).name() == "my_option_name") {
    // Do whatever you like with this stream.
    std::istringstream stream(table_proto.engine().options(i).state());
  }
}

The above example should be simple to extend to handle multiple options. What’s really important in the above example is that the option name can be accessed with the name() accessor and the state (value) of that option with the state() accessor.

So, that’s all I have to cover for now. I hope this feature will help storage engine developers create and provide useful table specific features for their engine.

Happy Hacking.

Toru Maesaka drizzle, oss api, drizzle

Happy to announce that BlitzDB has been merged with Drizzle’s Trunk.

As much as I’m excited, it’s time to come back to reality. This merge is merely a beginning. There is much more work that needs to be done to BlitzDB such as ensuring stability by adding more tests, find bugs, and eliminate them. I’m hoping that the likelihood of bugs being found will increase due to this merge. Admittedly, I want to hack on fancy (yet important) things like auto recovery but I’m going to resist doing this until I’m truly satisfied with the quality of BlitzDB. My plan is to have BlitzDB rock solid by Drizzle’s Beta release.

The review process to get BlitzDB into Drizzle was straight forward and smooth. This is mostly due to the fact that the community was very supportive about testing. Folks like Stewart Smith and Patrick Crews from Rackspace pointed out several bugs that I would not have found myself. I’m certainly lucky to have a supportive professional QA engineer (looking at you Patrick) to test out and give punishment to BlitzDB.

All I’ll be doing on BlitzDB for the next couple of weeks is debugging and refactoring to improve readability. What I need more of at the moment is test cases on JOINs that are likely to be used in practice. If you have a good test case, I would greatly appreciate it!

Toru Maesaka drizzle, oss blitzdb, drizzle

Having mobile internet connectivity was always something hot among tech geeks. What’s interesting though is that this luxury is gradually becoming a normal day to day lifestyle in urban Japan. Nowadays pretty much every major mobile carrier provides unlimited 3G Data SIM package for a competitive price. Pricing currently ranges between 2000 to 3000 JPY (Google Currency Conversion).

This movement in the mobile market is understandable since most modern lightweight laptops and netbooks sold in Japan has an internal 3G modem, which allows you to gain instant internet connectivity as long as there’s antenna coverage. Sony Vaio is a great example of this use-case. Needless to say, the iPad and Android is a contributing factor to the market shift as well.

The catch with these products is that you’re usually obliged to sign a contract for two years. This wouldn’t be a problem if you’re traveling to Japan and planning to stick around for that long but otherwise this can turn out to cost you unnecessary fees for cancellation.

Traveling to Japan? Read This

I’ve always been a fan of the prepaid model (as long as it’s not overpriced). So, last weekend I bought a 6 month package from b-mobile, which is a service that provides 3G internet connectivity (unlimited packets) for a finite period that you choose. The awesome thing about this service is that you don’t have to sign any contracts or register your personal information to the service provider (Japan Communications). All you need to do is prepay for a certain period and they’ll give you a SIM for it. No questions asked. Another good thing about b-mobile is that it runs on docomo’s FOMA network which is arguably the strongest mobile network in Japan.

For your interest I bought mine at Bic Camera in Shibuya and paid around 14,000 JPY for 6 months (works out to be around 2333 JPY per month). Their sales model is great for us consumers but the first impression I got was that this could be pretty dodgy if this product gets in the hands of folks with malicious intents.

Perfect with Android, Especially Nexus One

I decided to throw my new SIM into my HTC Magic (Dev Phone courtesy of Google) and setup tethering on it (both WiFi and USB Cable). Unfortunately the certain Android 2.1 kernel I was using wasn’t compatible with b-mobile so I had to go through several workarounds and ask my Android guru colleagues for help to get it working. The funny thing is that b-mobile will work out of the box with Nexus One running Froyo.

Despite the obstacles I’m happy with the outcome and I hope this blog entry would turn out to be helpful to those that are planning on traveling and staying in Japan for a while.

Toru Maesaka technology 3g, connectivity, internet, mobile

Google has a fantastic documentation on this topic but at the time I wrote this blog entry, the documentation covered how to download and upload data using appcfg.py but not with bulkloader.py (there is also bulkload_client.py). So, I decided to play around with the nifty bulkloader and keep a note on my findings.

Prepare the End Point for Loading Data

Loading data to the Data Store is accomplished by sending data to the application over HTTP. This means that your application needs a uniquely identifiable URI for you to send your data to. Creating a valid URI is just a matter of setting up a handler for it in the app.yaml config file. GAE takes care of the import logic with it’s own handler. There’s nothing special in this step and the documentation covers how to do this concisely.

Test Data for Demo Purpose

For this blog entry, I decided to prepare a CSV with four rows that represents users. In reality, there would be more information related to a user but I decided to keep things minimal for this blog entry. I saved this data as user.csv.

1, Daniel, Bernstein, xxxxxxx
2, Donald, Knuth, xxxxxxx
3, Bjarne, Stroustrup, xxxxxxx
4, Robert, Sedgewick, xxxxxxx

You can also represent your table in XML but I decided to use CSV for it’s simplicity.

Create a Bulk Loader Configuration File or Not

In addition to the CSV file, the bulk loader needs to know how each record in the CSV file should be represented as a Data Store entity. The modeling as far as I know can be done in two ways. One is to write a loader class in Python that the bulkloader can use. Another approach is to get bulkloader.py to generate a configuration file (in YAML).

I decided to write my own Python class to get through this step since according to the documentation at the time this blog post was written, this approach doesn’t work with the local development server.

With the above in mind, here is my loader class. You would usually keep the Data Model definition (the User class) in a separate file but for demo purposes, I decided to keep it in one file.

from google.appengine.ext import db
from google.appengine.tools import bulkloader
 
class User(db.Model):
  id = db.IntegerProperty()
  firstname = db.StringProperty()
  lastname = db.StringProperty()
  some_text = db.StringProperty()
 
class UserLoader(bulkloader.Loader):
  def __init__(self):
    bulkloader.Loader.__init__(self, 'User',
                               [('id', int),
                                ('firstname', str),
                                ('lastname', str),
                                ('some_text', str)])
loaders = [UserLoader]

The explanation on what this class does is described in the documentation. I saved this script as user_loader.py.

Load your Data to the Data Store

For demo purposes, I used my local development server on port 8083 to load the CSV file. Given that the application is running and that the API endpoint is active, it’s just a matter of providing bulkloader.py with essential information. For available options I recommend reading help by executing ‘bulkloader.py -h’.

The following command attempts to load our entity of ‘kind=User’ from user.csv using our loader class (user_loader.py) to the endpoint.

$ bulkloader.py --filename=user.csv --config_file=user_loader.py \
--kind=User --url=http://localhost:8083/import --app_id=your_app_id

Note that it’s essential to provide the --app_id option when uploading data to the local server. When asked for credentials, you can type anything you like. You only need to supply valid credentials when uploading to production.

Here’s the output from executing the above command.

[INFO    ] Logging to bulkloader-log-20100615.213842
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
[INFO    ] Opening database: bulkloader-progress-20100615.213842.sql3
Please enter login credentials for localhost
Email: foo
Password for foo: 
[INFO    ] Connecting to localhost:8083/import
[INFO    ] Starting import; maximum 10 entities per post
[INFO    ] 4 entites total, 0 previously transferred
[INFO    ] 4 entities (933 bytes) transferred in 4.0 seconds
[INFO    ] All entities successfully transferred

Success!

Toru Maesaka knowledge, technology datastore, gae, memo

Last month while being at the MySQL Conference, several people asked me about the status of BlitzDB. Specifically, they were interested in when I’ll release BlitzDB. Fair enough – I’ve been working on this project long enough for people to start questioning this.

The answer is, BlitzDB is done in terms of implementing the design. Right now it’s about finding bugs, fixing it and testing BlitzDB’s stability under concurrent load. Thanks to the motivation boost I gained at the conference, I’ve now fixed the bugs that were slowing me down and I’m gradually adding more tests into BlitzDB’s test suite. I consider BlitzDB’s initial release to be the day it gets merged into Drizzle’s trunk. This is almost ready as BlitzDB seems to be building fine on Drizzle’s Build Farm infrastructure. However, I won’t move to the next step until I’m satisfied with BlitzDB’s stability.

Yesterday I spent some time doing some concurrency testing on BlitzDB’s INSERT code with skyload. Needless to say, concurrency testing is also a convenient way to look at the performance of a particular component. So, I decided to publish my findings from this test. First, here is the background of the test.

Purpose of the Test

• Test BlitzDB’s slot-lock mechanism.
• Confirm that BlitzDB will not crash under concurrent INSERT workload.
• Confirm that key insertion to the index is working as expected.
• Confirm that writes to multiple indexes work as expected.
• Observe the write-performance impact of adding an index.

Two commodity boxes were used. One dedicated for the client and the other dedicated for the server (Drizzle + BlitzDB). Both boxes has the same spec: Intel Quad Xeon E5345 (2×4MB L2 cache), 8GB Memory, 500GB SATA II, gigabit NIC. Servers were connected by a gigabit switch. File system on the server was ext3.

By default, a BlitzDB table is optimized for up to 1 million rows. Therefore this test inserted 1 million rows to a table with different concurrency levels. A different concurrency level is used per run. The table used in this test only contains three integer columns. Tests are performed up to three indexes. The linux kenel’s dirty buffer is flushed before each test run. Tests were run until the performance curve flattened.

Result

As seen above, scalability from 1 thread to 4 thread showed an ideal curve. This is expected since the server is a 4 core box. From 4 threads, performance showed some improvements up to 12 threads. From there on, concurrency greatly exceeds the number of physical cores so we can’t observe decent performance growth. The highest insert QPS gained in this test was just over 86,000 QPS. With more cores on the server and more clients, I suspect BlitzDB can hit over 100k QPS.

Although this graph looks good at first sight, I’m not happy with it. The performance penalty for adding multiple indexes should be greater than what’s observed in this result. This is because TC’s B+Tree is internally protected by a single lock on writes. I suspect that the performance penalty is not observed in this graph because I didn’t give BlitzDB enough load to make TC work hard. This implies that a bottleneck could exist elsewhere (Network, Drizzle or BlitzDB’s handler level code).

However, I’m glad that BlitzDB stood stable on this concurrency test which was what I wanted to test in the first place. Admittedly I need to mix several types of queries to properly test BlitzDB’s stability. I plan on doing this next with sysbench and hopefully RQG.

Once this is done, I’ll submit a merge proposal to the Drizzle Project :)

Future Development Plans

• Find bugs, Fix bugs, Repeat.
• Write an inbuilt auto recovery routine.
• Eventually add a crash safe option to BlitzDB.

Toru Maesaka drizzle, oss blitzdb, drizzle, performance

One of many important things that the Drizzle project takes seriously is for the project sourcecode to successfully build in all our target platforms AND pass tests in them. This is not really specific to Drizzle as most open source projects would have the same policy. For example we do the same thing in memcached thanks to Dustin Sailing’s buildbot kungfu.

Yesterday, Monty Taylor gave me access to Drizzle’s Build Farm Infrastructure so that I could test BlitzDB on various Linux distributions and FreeBSD. Unfortunately most build machines didn’t have Tokyo Cabinet installed so I could only test builds on Ubuntu and Debian. Fortunately the build went fine on those platforms though this was predictable since Ubuntu is my primary development platform. What was disturbing was getting test errors on my index test suite. I guess it’s time to put my thinking cap on and see what the problem is there.

This is a big leap towards getting BlitzDB in Drizzle’s trunk which I’m steadily working towards. I also want to benchmark BlitzDB at it’s current state with sysbench‘s OLTP tests. This is still low in my priority queue but hopefully I’ll do it in the next couple of months.

Toru Maesaka drizzle, oss blitzdb, drizzle, hudson