Showing posts with label elwood. Show all posts
Showing posts with label elwood. Show all posts

Friday, October 14, 2016

In Search For A Better CI/CD

This is going to be an exciting time!

Imagine me, writing me first Clojure app!

I'm halfway through writing my first Clojure program called "relman", short for "Release Manager".
It's a simple CLI application to help me deploy specific WAR files to a dockerised Tomcat. 

The recipe here is a simple pull deployment (unless I've hit unforeseen issues - fingers crossed).

Firstly, I've configured my Jenkins build to rename any successful elwood-parent builds to <feature>-<issue number>-<issue description>-<#build number>.war and copy this to a mounted volume shared with Tomcat.

Secondly, running relman on host to copy the war file as ROOT to the designated Tomcat directory mounted as "/webapp".

Thirdly, umm.. I think that's it. Trying to keep the solution simple.

This POC worked by manually copying the file across directories, so technically this idea should work!

Furthermore, I think this recipe is universal and applicable when I use different Git branches whenever Jenkins kicks in.

Elwood Continuous Integration and Delivery

As with relman in Clojure, this is a work in progress but I think I'm not far off from my end goal.

In summary, this is my first step to get CI/CD working and it's by no means an end to this exciting journey!

And the quest for CI/CD continues... (Clojure included)

Tuesday, July 12, 2016

♕ Keep Calm We're Moving To Gitflow

I'm evaluating Elwood development with Gitflow. One thing I like about Gitflow is that it gives the ability to better track changes, releases and bug fixes. It also allows me to stay focus in writing one feature at a time. You see, there is an adrenaline rush in me that I sometimes find myself shoving unrelated changes to my timeline. I could commit one large snowball of unrelated codes that may stretch the completion time to 1.5 times longer if I'm lucky or 3 times longer if I'm not-so-lucky.

This definitely contradicts the spirit of agile with the emphasis of delivering in small increments. (Ahhh, agile that starts with a lower case... How pragmatic do I sound?)

I did ask myself if this could be too much of an overhead for a lone developer like me?

Well, time will tell...

Monday, June 20, 2016

How Do I Laugh At Thee? Let Me Count The Ways...

I hate to admit that I'm resource poor, in the sense that I can't afford to keep running my hardwares for 24x7. Neither can I afford to keep installing a new Linux image (JDK/Maven/Jenkins) every time I find a temporary shelter.

Currently, I have Jenkins/Linux running on an old machine. There were a few attempts to have it "VirtualBoxed" on my old laptop. However, since there is a sad fact about me and my laziness, all attempts ended in failure.

How do I laugh at thee? Let me count the ways
I laugh at thee from morning till noon till night
My soul rants which, when Jenkins s'out of sight
For constant installs take hours to days.

By nature, VirtualBox allows me to move the image between hardwares, but I need something much more than this. I need something that I could bring up on demand, something that I can move the data around, something that is lightweight - and Docker seems to fit the bill!

I was fortunate to have a hands-on experience with Docker at work, coincidentally, after playing around a bit with it in my spare time. At that time, I was looking for possible solutions for my quest on CI/CD. 

After spending roughly 2 months, I finally ended up getting Jenkins 1.61 and Redis images running on its own subnet with data directories mounted to host. I had to build my own Redis image as the official Docker image had a few issues when persisting to files. It could be something as easy as disabling persistence which I don't need, but nonetheless, time is ticking and I have to get back to Elwood development!

Below is a diagram of POC:


Dockerised CI tools



The proof of concept is very basic, something that's good enough to keep me going and something to iron out when I come back.

Hopefully, when the dust settles, I'll push these images into Docker Hub.

Repository: https://github.com/lyeung/lyeung-jenkins-ci-tools


1Why am I using Jenkins as oppose to using Elwood for my build? Elwood is light years behind Jenkins and any mature products out there.



Thursday, March 3, 2016

AngularJS 1.5 And PhantomJS Compatibility Issue

Last week, I've committed the relevant changes1 to issue #5086802 but Codeship3 reported a broken build. I swear I ran the UI test suite before committing and Grunt said everything was good! I decided to take some time to investigate this issue tonight to get to the very bottom of this. 



Broken builds keep me awake at night.



It appears that I've been building from the recent release of AngularJS 1.5 (previously 1.4) and there seems to be some incompatibility with PhantomJS 1.x. 

Interestingly, Codeship reported:
Error: [$injector:modulerr] Failed to instantiate module ng due to: TypeError: '' is not an object (evaluating 'Function.prototype.bind.apply')

After some lucky searching, this link came up:
https://github.com/angular/angular.js/issues/13794

The suggested fix was to move up to PhantomJS2 and update the karma-phantomjs-launcher module.

As to why the test suite worked on my machine but not Codeship's is a big mystery to me...


Note: Last October, as a true believer of CI, I've got Codeship to automatically build Elwood every time I push my changes to BitBucket. Codeship has been doing a great job for the past couple of months happily doing the builds in the cloud. (Hoping CD to come soon but everything must come in baby steps :)) It's an irony that I run a local copy of Elwood and manually trigger the build on Elwood sources. Does this sound like eating my own dog food?



Friday, February 19, 2016

AngularJS Custom Directive: Display Elwood Build Result Stats

I've written my first successful custom AngularJS directive to display build result stats (see issue #508680).

The directive initially displays the build status indicating success, failure or in-progress. Adjacent to this is a clickable icon1 that goes back to the server and retrieve the build result statistics and displays them on the screen.

Display elwood build result stats

The journey to get here is challenging  (especially for someone who works at the back office) but very fulfilling except for one last thing - unit testing.


Below is the directive code:

angular.module('elwoodUiApp')
  .constant('BuildResultStatsUrl', 'http://localhost:8080/buildResultStats/:key/:count')
  .factory('BuildResultStatsResource', function($resource, BuildResultStatsUrl) {
    return $resource(BuildResultStatsUrl, {}, {
      'get': {method: 'GET'}
    });
  })
  .directive('elwBuildResultStats', function() {
    var controller = function($scope, BuildResultStatsResource, ToKeyCount) {
      $scope.toggle = false;
      $scope.getBuildResultStats = function(keyCount) {
        BuildResultStatsResource.get({'key': keyCount.key, 'count': keyCount.count},
          function (successResult) {
            console.log(successResult);
            if (!$scope.buildResultStats) {
              $scope.buildResultStats = [];
              $scope.buildResultStats[ToKeyCount(keyCount)] = {
                'successCount': successResult.successCount,
                'failedCount': successResult.failedCount,
                'ignoredCount': successResult.ignoredCount
              };
            }
            $scope.toggle = !$scope.toggle;
          }, function (errorResult) {
            console.log(errorResult);
          }
        );
      };

      $scope.isShowBuildResultStats = function(keyCount) {
        return $scope.toggle
          && ($scope.buildResultStats
            && ($scope.status == 'SUCCEEDED' || $scope.status === 'FAILED'));
      }
    };

    return {
      restrict: 'A',
      templateUrl: 'views/buildresultstats.html',
      controller: controller,
      scope: {
        status: '@',
        keyCount: '=',
      }
    };
  });


And below is the test code:

describe('Controller: BuildResultStatsCtrl', function () {

  // load the controller's module
  beforeEach(module('elwoodUiApp'));

  var
    scope,
    httpBackend;

  describe('BuildResultStatsCtrl', function() {
    var BuildResultStatsCtrl,
      buildResultStatsResource,
      toKeyCount;

    // Initialize the controller and a mock scope
    beforeEach(inject(function ($controller, $rootScope, $compile, $httpBackend, BuildResultStatsResource, ToKeyCount) {
      scope = $rootScope.$new();
      httpBackend = $httpBackend;
      buildResultStatsResource = BuildResultStatsResource;
      toKeyCount = ToKeyCount;

      var elem = '<div elw-build-result-stats="" key-count="keyCountTuple" status="{{buildStatus}}"></div>';
      scope.status = 'SUCCEEDED';
      scope.keyCount = {
        key: 'PRJ',
        count: '10'
      };

      httpBackend.whenGET('views/buildresultstats.html').respond(200, '');
      var template = $compile(elem)(scope, buildResultStatsResource, toKeyCount);
      console.log(template);

      scope.$digest();
    }));

    it('should display when status is SUCCEEDED', function() {
      var url = 'http://localhost:8080/buildResultStats/PRJ/10';

      expect(scope).toBeDefined();
      expect(scope.toggle).toBeFalsy();
      expect(scope.isShowBuildResultStats).toBeFalsy();
      expect(scope.buildResultStats).toBeUndefined();

      var mockData = {
        successCount: 10,
        failedCount: 3,
        ignoredCount: 2
      };

      httpBackend.expectGET(url).respond(mockData);

      expect(buildResultStatsResource).toBeDefined();
      scope.getBuildResultStats(scope.keyCount);
      httpBackend.flush();

      expect(scope.toggle).toBeTruthy();
    });
  });
});


After compiling the element at line number 30, I'm expecting the scope variables and functions will be fully defined and by the time I invoke "scope.getBuildResultStats(scope.keyCount)" the mocked GET should return the mock data.

Well, it looks like it didn't materialise for some unknown reason.


It's almost quarter past 2 in the morning.

Changes aren't committed yet.

I feel very tired but I think I'm almost there...

Update: (2016-02-23 12:10 AM) Extracting the directive's function into a full blown controller and referring the controller with a controller name might be an option. This allows me to test the controller as an standalone component outside of this directive.

.controller('BuildResultStatsCtrl', function($scope, BuildResultStatsResource, ToKeyCount) {
  // ...
}

return {
  restrict: 'A',
  templateUrl: 'views/buildresultstats.html',
  controller: 'BuildResultStatsCtrl',
  scope: {
    status: '@',
    keyCount: '=',
  }
};


1Generated from http://www.amp-what.com and I find this website very cool.

Thursday, January 14, 2016

Elwood Build Process High Level Design

Below is a diagram of how the build process works at a conceptual level.


Saturday, January 9, 2016

Elwood High Level Design


"A picture is worth a thousand words..."



Elwood HLD

Friday, January 8, 2016

Elwood Maven Surefire Integration Part 2

I find this solution slightly obtrusive. After the project is checked out, Elwood alters the original pom and adds a small dependency snippet to pull down the custom run listener (elwood-maven-runlistener) during the build. This is evidently found in the pom.xml sitting under the META-INF directory marked with provided scope to prevent this from becoming an official dependency of the artefact.

I could potentially use a custom Maven that has the run listener JAR as one of it's libraries. But then, this has to be another day.

I had to abandon the original idea of making the run listener POST the build statistics using a 3rd party REST library because I may inadvertently introduce a class loader issue when the official project dependency has this library defined. Same argument goes with Redis. I also wouldn't want roll up my sleeves and start writing network calls on my own as time is a scarce resource.

My other concern with using network calls is that the back-end REST service could be busily serving requests from the browser, which I suspect may slow down the build process. Note that the front-end code retrieves the build status every few seconds.

I don't want to stress the back-end service too much.

I've also tested the idea of keeping the statistics in memory as a JMX bean. However, as each test classes are started up and torn down, the numbers are lost. In the end, I had to come back to my old friend – the filesystem1.

Note: All changes2 related to capturing the Maven build statistics are found in “maven-integration” branch.


Tuesday, November 24, 2015

Elwood Maven Surefire Integration

I thought of adding a feature to Elwood to display failed Maven tests in real-time.

Below are my options:
1. Constantly watch and scrape the test results produced by Surefire plugin
2. Augment a test listener1 to Surefire plugin and expose the failed statistics back to Elwood

Since this is an experimental project, I decided to take the slightly complicated approach - option two. From my original thinking, I need to write a Maven model converter, hunt down the Surefire plugin and add the necessary test listener before I feed this pom back to Elwood (see Figure 1)

Figure 1 - Maven Model Converter to add a Surefire test listener


There are a few challenges here. Firstly, writing this converter is not a straightforward exercise as I originally anticipated. It requires deep a knowledge on Maven internals, plus an extra hurdle on the lack of API documentation. I've searched high and lows and luckily found what I was looking for from a test code found in Maven Model Builder project2. Secondly, I need to find a way to pass back the failed stats to Elwood through some mechanism. I can choose to do this by writing to a file, but then, how is this different from scraping the result as described in option one?

I've created a local branch containing the model converter and yet to push this to the remote repository as I still have to figure out how to hand the numbers back to Elwood. At this point, I'm not certain if this solution is achievable, so please stay tuned...

Update (24/11): The initial implementation is now available at:
https://bitbucket.org/lyeung/elwood-parent/commits/4b64304870bf58bf6041cdaf7bf6460575c2d04d?at=master


1https://maven.apache.org/surefire/surefire-api/apidocs/org/apache/maven/surefire/report/RunListener.html 2https://github.com/apache/maven/blob/master/maven-model-builder/src/test/java/org/apache/maven/model/building/DefaultModelBuilderFactoryTest.java

Wednesday, October 28, 2015

Elwood sources now available in Bitbucket

For the past few weeks, I've been squeezing every last drop of my own time to get Elwood working. From a simple process executor to a functional web application, where I could trigger a build by clicking a link. I have to say that this app is simple and crude, but is really fun to work on!

Think about this for a moment - we're writing a tool to build the tool itself!

Elwood is made up of a back-end REST service and a front-end AngularJS app. The REST service requires Redis 3 (never tested this on Redis 2+). The app keeps a list of projects and tracks the build results. Currently, it only supports Git with SSH (with or without passphrase).

"Elwood threw down his fork. He pushed his chair back and stood up. Liz stared up in astonishment at him, her cup half raised. Bob's mouth hung open, his sentence unfinished. Little Toddy said nothing."

The sources are available in Bitbucket:



 Looking ahead, this is going to be a very long journey...

Tuesday, August 18, 2015

Elwood - Building the build server (a prototype)

Things are coming together right now! It all started from an imaginary concept filled with dragons, knights and fairy god-mothers - well yeah, you know what I mean.

At the most basic level, a build server has to satisfy 3 requirements:
  • Get the source code
  • Build the code
  • Report if the build is successful

What I have right now is a work in progress. It's quite rudimentary and clunky. It clones your sources (currently Git only) and runs your build command (e.g. mvn clean package). The UI constantly polls for the log but goes endlessly. I have to manually kill the poller to stop this activity - how shameful of me!
He builds a boat in his back yard, to great annoyance to his wife and neighbors. When they ask him why he builds it, he tells them he doesn't know.

What's next?
I need to capture and store the build number, as well as the build result.