engineering @ openlabs

RSS
Jan 9

Tryton Development Workflow at Openlabs

Last night as I was going through my git stash, when the mail icon on my Macbook started doing its tap dance suggesting the usual mundane result, there was an unread mail. The email was by my friend Jean from Copengo about something that I wanted to write since a long time. I quickly changed into my Star Wars t-shirt (sentimental reasons about the force) and started sabering my way through our internal workflow. For those of you, who are bored of technical jargon (I am one of them), a workflow is that which companies and individuals developing Tryton modules can adopt.

TL;DR; Here is the list of sabers that you will need to take down the Sith Lords (Stack):

The abridged version of our workflow (The workings of Rebel Alliance)

  1. Every module is a separate git repository owned by the Openlabs (A part of the Rebel Alliance) on Github.
  2. Employees (and community contributors) fork the repository and send back pull requests.
  3. The pull request is sent if the code passes the review, which is like winning against Master Yoda.
  4. The pull requests are automatically tested by Travis, our own R2-D2, which tells us if the tests are successful.
  5. If ‘all is well’ then the code gets merged.
  6. Customer projects are deployed to staging/production servers and open source projects are released to the python package index.

Take a step back and repeat the same action with your light saber.

The workflow in detail

Just as I finished writing this post I realised that it had become way bigger than I had expected. So I have split it into multiple parts for easier reading and posting.

Jan 9

Part 5: Tryton Development Workflow Series - Deploying & Monitoring

Deploying Code like destroying the Death Star

It is important to deploy code easily to production and staging servers. While there are a host of configuration management tools, we prefer to keep it simple and pythonic using fabric. Most customer projects can be deployed with just `fab deploy_staging` and `fab deploy_production`. Documentation (using sphinx) is also automatically built and deployed to documentation servers with fab commands.

While the deploy_staging command checks out the develop branch of the code, the deploy_production command requires the exact version that should be installed. The version is identified from the tag used to mark releases.

The deployment commands are capable of upgrading the Tryton database too.

PS: Automate the most redundant task in your workflow…. Deploying code

Monitoring tryton (and nereid)

Monitoring the application is important to ensure that your application works for everyone (and not just for you) and monitoring the server they are hosted on is even more important to avoid downtimes and to debug if the armageddon (in this case failure of app to run) happens.

image image

Our preferred tool for application monitoring is Newrelic (Our very own Princess Leia) which also monitors our servers (for free). All exceptions are automatically captured from both Tryton daemon and nereid webapps and collected in sentry

image

Before sentry, we spent most of our time trying to reproduce mysterious errors users claimed to have encountered. Sentry keeps records of the stack traces and a sanitized context (values of variables), and client information making debugging a zephyr.

image

Notifications about exceptions and incorrect logic reach all developers and also spam our chat room for the specific project (powered by hipchat).

Jan 9

Part 3: Tryton Development Workflow Series - Code Review

image

Github pull requests have support for lightweight reviews, but we still continue to use Rietveld (hosted on google app engine) for our reviews.

Here is a history lesson in technology, Rietveld is a web-based collaborative code review tool for Subversion written by Guido van Rossum to run on Google’s cloud service.

Rietveld is a django app and could be hosted anywhere, but deploying on app engine is cheap and it works better for us because the logins are integrated with google apps (which we use for emails and chat within Openlabs).

Read part 4 about Pull requests and Continous Integration

Jan 9

Part 4: Tryton Development Workflow Series - Pull Requests & Continuous Integration

image

Proudly powered by travis CI

Pull requests are an integral part of our workflow and the trigger for Travis, our continuous integration engine. Every repository has a .travis.yml file which defines the steps to install the module and run the tests. Travis makes every build on a fresh virtual machine and the installation of the module is done from scratch.

In addition to running the tests, Travis scripts are usually configured to run flake8 and look for quality issues in the python code. Some repositories are configured to build on both sqlite and postgresql as testing on sqlite yields a lot of false positives.

Pull requests for older versions are obviously built with older versions of Tryton and dependencies. This relieves the package maintainer from the headache of having a mandatory older test environment and installing everything locally to test.

image

Why this process ? The ability to know if a pull request can be merged without manually pulling the code from the contributor and testing it is a significant time saver. Travis fits beautifully into our test driven development workflow.

See the screenshot below from the travis blog:

image

Read part 5 about Deploying and Monitoring

Jan 9

Part 1: Tryton Development Workflow Series - Repository

The central ‘official’ repository

The notion of a central official repository is just a convention. From a technical standpoint Git does not make a difference between each developer’s repository and the official one.

Fork the ‘official’ repository

image

As a thumb rule and tribute to nerds around the globe, nobody pushes code to the main repository. Everyone has to fork the main repository (in our case owned by Openlabs) and send the patches to the develop branch.

Branching for each Tryton Series

image

Once a new major version of Tryton (Yeah, they liked Jupiter) is made available (like 2.8, 3.0) the current code in develop is checked out to a branch with the name of the Tryton major version (3.0) and the develop branch is updated to the latest development version. (Think of it as a tree, the code is the fruit on the branches)

image

Minor Releases

Every minor release is tagged with the git tag functionality. The description of the tag is used as a CHANGELOG which Github renders seamlessly.

image

Customer Projects

Source code for customer projects (unlike Tryton modules) undergo significant changes within the same major version. In addition, customer projects always have a separate staging server where we deploy code for the customer to test and give us feedback. This is where gitflow shows its might like Han Solo coming to rescue of Princess Leia:

  1. The staging server always has the code from the develop branch.
  2. Once a set of features are completed (on develop branch) and approved by the customer, a major version of the project is released.
  3. Sometimes, a critical feature or a bug might need to be fixed without waiting for the next version. In such cases, the (eye)patch is developed on develop branch,deployed to staging and on approval from customer it is cherry-picked into a hotfix release which includes this change alone and not other changes in the develop branch.

image

The Gatekeeper of our process (We call him C-3PO)

Though, we use a distributed version control system, the merging into the central repository is managed by a human-droid we call the C-3PO. C-3PO is also the package maintainer and is responsible for:

  • Merging pull requests
  • Making branches when new versions of tryton are out
  • Release management

You can read more about the forking workflow and how the gatekeeper works from the atlassian blog.

What next ? Read Part 2 about project management

Jan 9

Part 2: Tryton Development Workflow Series - Project Management

Project Management  (The Home of the Jedi Knights)

image

Project management at Openlabs is done using myOpenlabs which is powered by Tryton and Nereid (They really like Jupiter). myOpenlabs is a suite of nereid applications which includes project management (powered by nereid-project, which we have open sourced) and account management for our customers to see their account position, download invoices and make payments.

image

A project management system helps reduce the coordination cost and is crucial when you have developers and customers across multiple timezones. (It is the wife/girlfriend you never had) The system helps keep the information transparent across the customers, developers and consultants.

The system is also tightly integrated with other tools like github. Commit messages which mention task id in the format #1234 are automatically linked to the task and displayed as an update.

image

image

Time Tracking

It goes without saying how critical tracking time spent on tasks is. Nereid project makes it super easy to mark time on tasks. Openlabs customers love this transparency into how their project is progressing on a task by task basis.

image

What next ? Read part 3 about Code Review

Github mirror for Tryton repositories

We have set up a github mirror for the tryton repositories. 

The mirror is updated once every day between 01:00 UTC and 02:00 UTC. 

The mirror uses the branching capability of Git to keep the different versions of the code (tryton’s repository hosting has different repositories for different versions).

Read More

Migrating nereid modules to active record pattern

We just upgraded the core nereid module (the web framework) to Tryton 2.6 which uses the Active Record design pattern. The migration was easy since we had the wonderful guidelines written by Cedric Krier. Just as how the guidelines help in migrating tryton modules to 2.6, I hope this post helps in migrating nereid modules to 2.6.

1. Restful URLs

The 'methods' selection field was due for deprecation and has been removed. If you are migrating to 2.6.x from any version after 2.4.0.6 you should not face any issues with migration as the selection field is automatically migrated into the new format.

Remember that prior to this version, the GET HTTP method was allowed by default (implicit), but the new design disallows all HTTP methods by default. So to achieve the same behavior as 2.4, you will have to explicitly enable the GET method.

2. Active record pattern: Tryton 2.6 introduces the active record pattern and the model controllers which work on more than one record are class or static methods while the methods which act on a single record happen to be regular instance methods. The convention is extended to nereid too: If the handler works outside the context of a single active record, use static or class methods depending on whether your method delegates/uses other methods in the class. If not stick to an instance method. When you use an instance method, tryton expects the instance to be the active record itself and nereid allows you to do this again by convention. Just ensure that your url has an integer converter which generates a keyword argument ‘active_id’. Example:

Happy Hacking !!

Feel free to catch any of us on #tryton IRC if you need more help

Nereid going to be part of Tryton

The Tryton unconference at Liege was a lot of fun and one of the most exciting outcomes of it was the community’s request for Nereid - the web framework for Tryton built at Openlabs to be made a part of the core tryton modules.

As a first step into the process, Cedric Krier made a review of Nereid in the context of its inclusion and made his comments on the mailing list. We have translated his comments into issues on github, and we are getting them done one by one.

Keep following this blog for more updates on the changes being made to nereid.

Some screenshots of the upcoming nereid interface to the Tryton project management system.