[LRUG] Compiling native extensions during deployment?

[gareth rushgrove]

On 23 October 2011 14:24, Thom May <thom at may.lt> wrote:
> On Sun, Oct 23, 2011 at 10:06, gareth rushgrove
> <gareth.rushgrove at gmail.com> wrote:
>> On 20 October 2011 14:41, Thom May <thom at may.lt> wrote:
>>> We run bundle install --deployment as part of our CI process, and then
>>> push out the results of CI as a tarball to our app servers.
>>> I'm being hounded from various quarters to write a blog post about
>>> this, so I might do that shortly.
>>>
>> Building on that, have you thought about wrapping up the package as a
>> deb or rpm rather than just a tarball?
>>
> Yes, and consciously rejected it. I miss-spoke a little when i said
> "push the results to our app servers", actually, so let's go into a
> bit of detail.
>
> - jenkins is triggered to build by Github
>
> - the last stage of our standard builds produces a tarball and
> notifies our management webapp (octobutler) of the existence of the
> tarball. The tarball is placed in a standardised location and made
> available through http
>
> - Octobutler queries mcollective (using a plugin called 'trebuchet')
> to get the current deployed state of a given app, filtering on an ohai
> fact to get the list of nodes running the app. (This gives some side
> benefits like knife ssh 'trebuchet:app_name')
>
> - When the user wishes to deploy, octobutler does the moral equivalent
> of 'mc-rpc -F trebuchet:app_name trebuchet install app_name branch
> sha', which causes the plugin to download the tarball, extract it and
> then run chef-solo to do the final setup. We then set up a 'current'
> symlink. For management, we use foreman
> (http://ddollar.github.com/foreman) to generate upstart jobs.
>
> So, why not packages? The state of the machine is managed by chef, so
> pre/post scripts have little benefit (and we can use chef-solo for
> easy idempotent post-extraction configuration).

For what it's worth I agree with that, pre/post scrips can be used for
ill, especially in a config management setup.

> Using tarballs lets us
> have multiple versions of an app on the machine, so
> rollback/rollforward is just a symlink flip.

I actually do this with packages, (much to the horror of some pure
package folks I have to admit). Each deployment gets a new unique
package name, probably equivalent to whatever name you might be using
for your tarballs. I then have a meta package which depends on the
last x packages. I'm experimenting with the symlink being managed
either in the meta package or in config management - not sure which I
prefer yet but both work.

> Our approach also decouples deployment from configuration management,
> which i think is a good thing.
>

Ah, I think it's a bad thing :)

The main reason I'm of this mind is an application has system
dependencies (specific c libary or mysql version say). By modelling
deployments inside config management tools you can model this in code
in one place. Anything else either ends up saying it's not a problem
(admittedly it's often not) and managing it manually or basically
replicating the innards of chef/puppet in your deployment tool of
choice (say Cap).

Do you have machines that drop out of the cluster download the latest
tarball from the http endpoint when they come back up, or just destroy
them (yay for the cloud) or leave them for manual intervention?

> I completely agree that for system tools packages are the right
> approach - we have internal packages for scribe, reconnoiter, etc -
> but for deployments I think they're overly heavy weight.
>

I used to only do it at times due to tool chain woes, but FPM has made
creating packages much more light weight.

https://github.com/jordansissel/fpm

Worth checking out for anyone building packages or wanting to, whether
you think it's crazy to go down that route for your apps or not.

G

> I think we're probably in position to start opening some or all of
> these tools in the near future, but I can't commit to that right now.
> Hope this gives a little more insight into our approach.
>
> -Thom
> _______________________________________________
> Chat mailing list
> Chat at lists.lrug.org
> http://lists.lrug.org/listinfo.cgi/chat-lrug.org
>



-- 
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com