Self Hosted Gem Server with Jenkins and S3
On my team, we like to be able to keep our applications light. We use several internal libraries to manage things like distributed request tracing, authorization, configuration management, etc.
Isolating this reusable, generic code into a library keeps our application code concise and manageable. It also allows us to test changes to the library in isolation (not to mention keeping our tests fast).
The server #
Most of these gems are very specific, and it wouldn’t make much sense to make them public. So, we decided the best approach was to use our own, internally accessible gem server.
A gem server can really just be a set of static files – nothing fancy.
# http://guides.rubygems.org/command-reference/#gem_generate_index # Given a base directory of *.gem files, generate the index files for a gem server directory $ gem generate_index --directory=GEMS_DIR
And just like that, we’ve generated the static files we need for our gem server.
Using S3 #
We use S3 for plenty of things here at Climate. We’d rather not have another server to have to support, and S3 is a perfectly fine place to host static files. So, we just turn an S3 bucket into an internally accessible endpoint (i.e., through our internal DNS routing on our network).
Amazon has instructions for setting up a bucket as a static file server.
Tying it all together: Automated testing and build #
Now, we have our S3 bucket setup to behave like a static file server. This is the workflow we want:
- Make some changes. Commit. Push. Pull request.
- Pull request and code review
- Merge pull request
- Manually trigger a build of the new gem (which automatically runs tests) 4a. If tests pass, deploy the packaged gem to our gem server
- Run bundle update MY_NEW_GEM! to update our project
We use Jenkins to automate our builds here at Climate. Depending on the git server you use (we’re in the process of migrating to Atlassian Stash), Jenkins integration is a bit different. I won’t talk too much about the Jenkins configuration specifics, but this is basically what happens:
We have two Jenkins jobs: the first job is the
build and the second one is the
update server job. The reason behind this is to allow concurrent
builds. When the first job completes, it triggers the second.
We run each of these jobs in a Linux container. Docker is a nice project designed to make that process easy. In our containers we make sure we have the
ruby and s3cmd in the path.
build job is a parametrized Jenkins build. We pass in a parameter, which is the
PROJECT_DIR, or the directory relative to some root where Jenkins can find the gem we want to build. We keep all our gems in the same repo for simplicity.
Jenkins will check out the gems repo, and build the specified gem. This is the build script that Jenkins will execute, which is essentially equivalent to:
$ ./build.sh ~/gems_repo/my_new_gem
#!/usr/bin/env bash # Exit 1 if any command fails set -e if [ "$#" -ne 1 ]; then echo "PROJECT_DIR not specified" echo "Usage : `basename $0` <PROJECT_DIR>" exit 1 fi PROJECT_DIR=$1 GEMS_ROOT=<root directory of gems repo> echo "Building gems in PROJECT_DIR $PROJECT_DIR" # Check that PROJECT_DIR is in the path relative to GEMS_ROOT if ! [ -d "$GEMS_ROOT/$PROJECT_DIR" ]; then echo "Error: PROJECT_DIR does not exist" exit 1 fi ## Go to Gem project cd $GEMS_ROOT/$PROJECT_DIR # Create a build number # year.month.day.hour.minute.second.sha1 export GEM_BUILD=$(echo -n $(date +%Y.%m.%d.%H.%M.%S).;echo $(git rev-parse --short=7 HEAD)) echo "GEM_BUILD $GEM_BUILD" # Find the gemspec to build GEM_SPEC=$(find $GEMS_ROOT/$PROJECT_DIR -type f -name *.gemspec) echo "Building gem from gemspec $GEM_SPEC" # Bundle, run tests, and build gem bundle install bundle exec rake test --trace gem build $GEM_SPEC # Find target gem. Prune search to exclude vendor TARGET_GEM=$(find $WB_ROOT/$PROJECT_DIR -type f -not -path "*vendor/*" -name *.gem) echo "Uploading gem $TARGET_GEM to gem server" # Deploy (updating the Gem server index is left to another job) s3cmd put $TARGET_GEM s3://my-gems-s3-bucket
One thing to note about the build process is that it takes care of build versioning automatically for us. The way we handle this in each of our gems is:
# my_new_new/lib/my_new_gem/version.rb module MyNewGem MAJOR = "0" MINOR = "2" BUILD = ENV["GEM_BUILD"] || "DEV" VERSION = [MAJOR, MINOR, BUILD].join(".") end
Another thing to keep in mind is we always want to run our tests (you are writing tests, right?). This depends on our gem having a rake task named
test. This is fairly simple:
# Rakefile require 'bundler/gem_tasks' require 'rake/testtask' Rake::TestTask.new do |t| t.libs << 'spec' t.test_files = FileList['spec/**/*_spec.rb'] t.verbose = true end desc 'Run tests' task :default => :test
Update gem server #
The last step of our
build job is to push the new
*.gem file to the server. We now need to update the set of static files Bundler uses to retrieve the available gems from the server. This is fairly simple. Here’s the script for that job:
#!/usr/bin/env bash # Updates the index of available Gems on the S3 gem server # Exit 1 if any command fails set -e # CD to Gem project directory cd $GEM_ROOT/applications/gems # Create a placeholder directory export GEM_SERVER_DIR=./gemserver if [ ! -d $GEM_SERVER_DIR ] then mkdir -p $GEM_SERVER_DIR/gems mkdir $GEM_SERVER_DIR/quick fi # Install any dependencies in the Gems project # We require the builder gem # See: http://rubygems.org/gems/builder bundle install # Get existing files from Gem server s3cmd get --recursive s3://gem-server/ $GEM_SERVER_DIR # Generate new static files # See: http://guides.rubygems.org/command-reference/#gem_generate_index # "Update modern indexes with gems added since the last update" gem generate_index --directory $GEM_SERVER_DIR # Sync the index files to the server. No need to sync anything in gems/ cd $GEM_SERVER_DIR s3cmd sync . --exclude 'gems/*' s3://my-gem-server
And, that’s it! We have automated our build/deploy proces for our own, internal Rubygems. Feel free to reach out to me with any questions.
I highly recommend learning how Bundler and Rubygems work.