Weave Minimal Server

Fazal Majid did a really interesting hack where, frustrated by trying to get the Weave Server running, he built his own minimal server. It’s not surprising that he found installation difficult; it’s a complex system, and the latest migration was a real challenge (ironically, we made it harder this time so that it would be easier in the future).

His effort wasn’t enough to be particularly usable outside his own needs – it lacks auth, security, data validation, storage, and a whole host of other things – but it got me to thinking about what our users want locally. We’ve been very careful to open up everything to the user so that they have the ability to do the sort of things we’re doing with the large production cluster. However, some of those decisions, driven by multi-box large-scale needs, will look really weird to someone who just wants to run a weave server for himself and half a dozen friends. It’s overkill.

Thus, I started hacking, and have come up with a super-easy to install minimal weave server that should work fine for a small installation. It’s a full implementation of the storage API, including auth, but setting it up requires just one line in your apache config. There are no options, or even a config file – it simply sets up a SQLite DB in the directory and uses it. Users are added and deleted by a script in the directory.

It’s still a prototype, and I’d love feedback on whether it really was as easy to install and get going with your client as it should be. For now, the tarball is here. Let me know how it works for you.

You’ll need a relatively recent php with sqlite, mbstring and json support, and apache, preferably running ssl. Give it a shot and let me know how it worked for you in the comments.

0.5 Server release

With the release of the 0.6 client today, we’re also publishing the 0.5 Weave server release. This release is required to support the 0.6 client if you are running your own server – it uses the brand new API.

If you have a 0.3 installation up, there are migration instructions, or full setup instructions.

The server is designed to run 0.3 and 0.5 side by side (the table is backwards compatible), so you don’t have to migrate all your accounts over at once. However, you should upgrade your clients as soon as possible, because the new client is much better.

Have fun, and hop onto the #labs channel on irc.mozilla.org if you need setup help.

A Preview of the 0.5 Weave Server API

As announced as part of the 0.6 Weave client roadmap, we’ll be moving over to the 0.5 server API. I wanted to make a post to talk about some of the changes, and what to expect in migrating over.

The documentation for the APIs can be found at:

https://wiki.mozilla.org/Labs/Weave/User/API
https://wiki.mozilla.org/Labs/Weave/0.5/API

Here are some of the changes and highlights from the 0.3 version:

• Better separation of the storage and user management pieces
  As Weave grows, there’s a fairly clear separation between the user-management portion and the storage portion (unfortunately, the break isn’t perfect, since creating a user in SQLite requires setting up their storage). Thus, each section has it’s own subfolder, with just the parts of the code it needs, and its own config files. This will allow you to deploy the user API separately from the storage API depending on your configuration.
• Default config
  
  The constants file for the storage API is now a little more complex. weave_constants.php is just a wrapper. It looks in it’s directory for {HTTP_HOST}_constants.php and uses that if it exists, falling back to default_constants.php.

  If you have a relatively simple setup, then not much changes – just rename your constants file to default_constants.php before pulling and you should be good to go (though check the .php.dist file to see if there’s any constants you’re missing!). This change will allow you to host multiple weave nodes on the same machine just by dropping in a separate constants file for each.

• New paths
  
  We’ve altered the paths in the API in recognition of the fact that there are API calls to get the data for a user and API calls to get metadata for that user. The old URL schema didn’t handle this elegantly (we’d have had to make use of params), so we’ve changed things around a bit. Everything should map pretty clearly to the old URLs, and the setup instructions will have new paths for the Apache aliases.
• Single query to get all modification times
  
  In the past, the client has issued a query to the server for each collection the user has to get the maximum timestamp. This is obviously somewhat inefficient, and we’ve combined all of them into a single API call that provides a has of all collections and last-modified timestamps.
• Query to get quotas (optional)
  
  This function will tell a user their current memory use and the quota for their account (currently defined as the optional constant WEAVE_QUOTA). We’re not using this at the moment, but may need to in the future, so it goes into the API.
• Additional column in the table – payload_size – not used yet
  
  There’s no code written to take advantage of this yet – we want to make sure that the tables have been modified before we start issuing invalid db calls! – but there will be an additional column on the wbo table: payload_size (an int) and a corresponding index on username and payload_size. This will speed up the quota call.
• X-Weave-Records response
  
  This header will return the total number of records being returned on a multiple-record GET request, and is useful for progress bars on slower clients. This will not work with all backends, but it should for MySQL installs.
• New output formats
  
  Additionally, multiple record GET requests have two more options for output. These are triggered by the contents of the “Accept” header, and are primarily designed for clients that may not be able to hold the whole response set for processing in memory. They are:
  • application/newlines – returns each record as a separate json object on a newline. Newlines in the body of the object will have been replaced by ‘\u000a’
  • application/whoisi – returns a 32-bit integer that contains the size of the next record, followed by the json object of the record itself.
• User API for all functions
  
  This replaces the current ad-hoc mishmash of scripts used to manage a user account, providing us with something a little more RESTful. Documentation for it can be found at https://wiki.mozilla.org/Labs/Weave/User/API

  It has it’s own separate space in the code, meaning that you’ll need a config file for it, but you can now run it on a separate machine. The spec isn’t fully locked-down yet, so if you see something you think would be useful, please let us know.

• Recaptcha generation query
  
  To support the User API, we also have a set of miscellaneous APIs, the first of which is the ability to generate a captcha through reCAPTCHA. If you want to take advantage of this, you’ll need your own key (available at http://recaptcha.net/), which goes in the constants file for the miscellaneous APIs as well as into the user constants file. This function is documented at the bottom of the User API documentation.

As you can see, we’ve been busy trying to improve the Weave experience as we move towards full releases, and we’d love to hear any feedback you might have on the server (or the client, though I can’t help you as much there). Feel free to comment below, or join us on the mozilla-labs-weave googlegroup.

A few changes to the Weave Server

As some of you may have seen in Anant’s post on username case-sensitivity, we’re updating Weave. As he mentioned, part of doing this involves a server change, to essentially turn usernames lowercase inside the Weave server.

If you are using the central Weave cluster, you shouldn’t notice any changes, aside from the downtime this Sunday. If you are running your own server, then the instructions below are for you.

Fixing this locally involves two changes – updating the code, then updating the db to retrofit your old data.

For Mysql

Before pulling down the code, if you are using mysql, go into the database and do ‘alter table users change column username username varchar(32)’ and ‘alter table wbo change column username username varchar(32)’. Then pull the latest hg (fc37b56bf414) and you should be all set.

For SQLite

If you’re only storing your browser data in weave, the fastest way may be to simply recreate the account (which will create in lowercase through the admin.php script). If you want to move the table, get the md5 of your current username, and move the data in STORAGE_PREFIX/[1]/[2]/[3]/[md5] (where [1] is the first character of the old username md5) to STORAGE_PREFIX/[1]/[2]/[3]/[md5] (the appropriate parts of the lowercase username md5). If you do this, I would shut down apache while you make the changes!

To fix the users table in SQLite, log into it and do ‘update users set username = lower(username)’

Then pull the hg code (fc37b56bf414) and everything should work.
—–

A second change that goes out with this release involves cleaner versioning. You’ll notice that there is now a 0.3 directory and a 0.5 directory. You don’t need to do anything with this at this point, as the top level files remain the same. However, if you point your apache server at the 0.3/index.php file instead (and copy your constants file into that directory), everything should work… exactly the same. Once we release the 0.5 server (and the numbering is just coincidence, we’re not trying to track it with the client), you’ll be able to point at that while supporting 0.3 server users during the transition. 0.3 will be deprecated upon the 0.5 release, but still supported for some time.

Weave Server Official First Cluster Launch

Today, we’ve rolled out the first official Weave cluster, built for scalability and handling a higher number of users, as well as a new client version (0.2.102) to make use of it. Thanks to all the people who made this effort possible – Chris, Dan, Anant, Jono and the rest of the labs team, Matthew, Justin and Jeremy over in operations, and all the people who provided suggestions, bugfixes and solutions in the #labs irc channel or in Bugzilla.

Ideally, if we’ve done our jobs well, people won’t notice the change at all. It should be transparent to the users (ignoring, of course, the big increase in speed and stability!)

What does this mean for Weave? Because of the open API model, the client and the server are moving independently, so we aren’t yet ready to call the whole thing “beta”. However, from a backend perspective, hg version b2613f8cdb48 f3f34e06cdc0 can be considered a milestone release, and it’s probably safe to say that the object API is locked down.

It also means that, for the first time in quite a while, we’ll be opening registrations again. In order to ramp up in a reasonable fashion, we’re going to use a GMail-style invitation code approach. You can put in a request for one here and hopefully you’ll be chosen to try it out soon.

One final caveat: the client requires Firefox 3.1 and won’t work with lower versions without disabling large chunks of the app. I’ve been running 3.1 forever and it’s rock solid, so you have no excuse not to enjoy Weave-y goodness.

(Edit: updated the milestone release to reflect a tip with a distracting file removed)

Notes on moving server from release 3 (12/19) to release 4 (1/26)

We continue to make tweaks to the Weave API to make it more efficient. The latest is the addition of the X-If-Modified-Since header. If you pass in an appropriately-formatted timestamp, the server will not make the requested changes unless the timestamp is later than the highest timestamp associated with the collection.

To make this work better, we’re changing the format of modified to be a floating point number. Unfortunately, this means that, yet again, we’re changing the format of the timestamp. If you’re using mysql, once you’ve updated the server from the hg tip, you’ll need to do the following:

update wbo set modified = floor(modified /1000);
alter table wbo change column modified modified decimal(12,2);

This should benefit a lot in the long run – the numbers in question can now be handled by the standard numeric size on a 32-bit system (mileage varied by language with the old numbers).

Notes on moving server from release 2 (12/01) to release 3 (12/19)

We added a few features to the server that should help people trying to work with trees and ordered lists (namely, some new sort options) and some optimizations to cut down on data transfers (the ability to overwrite just some portions of the metadata). As a result, there’s a tweak that needs to be made to make the server work beyond just updating the hg tip.

If you’re on mysql, you need to do

alter table wbo add column sortindex integer, depth tinyint after modified

If you’re on sqlite, you’ll need to do that inside each user’s storage file.

UPDATE: And we’ve changed the timestamp format, so you’ll also need

alter table wbo change column modified modified bigint

As always, all the new stuff you’re able to do is featured on the Weave API page.

Notes on moving server from release 1 (11/24) to release 2 (12/01)

Because we implemented the server to the 0.3 API and the only changes to the API have been in optional areas, the older version of the server will continue to work. However, at some point, you’ll want to upgrade to take advantage of some of the fixes. What then?

The bad news is that it’s not as simple as just updating from Mercurial. The good news is that it isn’t too hard.

1. Upgrade your tree
2. In the top level of the weaveserver directory is a file called weave_constants.php. Take the variables that you’d defined in your apache environment and move them into the appropriate places here. You can delete them from your environment after that. Note that you still need to leave in the Alias directives – you’re just moving the SetEnv ones.
3. Take weave_constants.php and put the file into your php include path. The easiest place (and least secure) is in the same directory as the rest of the weave code, but /usr/share/php and similar directories may be available to you depending on the level of control you have over your server.

That should do it. If you run into problems, hop over to the forums and we’ll see what we can do.

Weave 0.3 API and Server updates

Thanks to everyone who has provided input and suggestions. We’ve revised the 0.3 API to take them into account. Encoding and encryption were not really meta-information about the object, and are more properly handled as part of the payload, which is always expected to be a serialized json object (that may or may not contain some encrypted portion).

We’ve also been working on the server, pulling the constants out of the apache environment into their own constants file for more security. If you’ve installed the server before, you’ll want to check out the setup instructions to see what’s changed. Once you update to the latest code, then setup the config file, you should be all set.

Finally, thanks to some great work by Dan Mills, we should have a client out for this new server shortly. (If you downloaded the M1 client, please be aware that it isn’t compatible with the latest version of the server. Soon!)

Weave 0.3 Prototype Server

Feedback on the 0.3 API announcement went well, so it’s time to release a prototype of the server. The code is available from the weaveserver repository along with instructions for setting up a copy of the server and the local user creation API if you want to use that.

The server supports Mysql and SQLite backends, and the ability to mix and match between engines for handling the storage and the authentication. It is written entirely in PHP (the test program is in perl), and it requires json and PDO support for the storage backend of your choice.

There is not yet a corresponding Firefox client for this server, though one is obviously in the works. We debated internally whether it was useful to release just the server, and believe that it is of value. It allows the community to see (and comment on) the decisions we’ve been making, as well as spot potential pitfalls, bugs and potential areas for improvement. It also gives everyone the opportunity to play around with the server a bit and think about how they might use it once the full version goes live.

I’ve set up a thread on the Weave Forums for questions and discussion, and various members of the labs team are usually available on the irc.mozilla.org #labs channel.