Subscribed unsubscribe Subscribe Subscribe

necojackarc’s blog

A software engineer's blog who works for a web service company

Make Images Watermarked with RMagick (+ CarrierWave)

The original post was published on Jul. 8, 2016, by me in Japanese.

Summary

This post explains how to cover images with an array of a watermark.
Plus, I show an example how to use it with CarrierWave.

What is Watermark?

Watermark is an image which often covers sample images.
I made a sample of fotowa*1 logo:

f:id:necojackarc:20161014223629p:plain

You almost can't see it as it's really transparent. Let's say combine it with this picture, the image would be like this:

f:id:necojackarc:20161205234711j:plain

This is also the final product of this post.

Make an array of a watermark by mosaic of RMagick

It takes a lot of time to combine images multiply when you make images covered with an array of a watermark. I've tried it once, but it didn't work in realistic time.

So, make an array of a watermark at first and combine it with target images. Then, it really speeds up the process.

mosaic is, as the name shows, a feature which makes mosaic by arraying a lot of images.

image = Magick::Image.read("target.png").first

mark = Magick::Image.read("watermark.png").first
mark.background_color = "none" # If background color is specified, alpha channel becomes that color after combining images

tile = Magick::ImageList.new
page = Magick::Rectangle.new(0, 0, 0, 0)

# Array watermarks to make it appropriate size
(image.columns / mark.columns.to_f).ceil.times do |x|
  (image.rows / mark.rows.to_f).ceil.times do |y|
    tile << mark.dup
    page.x = x * tile.columns
    page.y = y * tile.rows
    tile.page = page
  end
end

One point is the background color of a watermark. For example, the background color of the watermark I already showed is "white".

If you use a watermark whose background color is specified, images will be covered with the background color like this:

f:id:necojackarc:20161206001409j:plain

Use it with CarrierWave

You can easily manipulate images with CarrierWave and RMagick, like process resize_to_fit: [2000, 1000].

As it's okay to use any methods you define for process, so define a method and use it in version:

class ImageUploader < CarrierWave::Uploader::Base
  include CarrierWave::RMagick

  version :watermarked do
    process :watermark
  end

  def watermark
    mark = Magick::Image.read(Rails.root.join("lib", "assets", "watermark.png")).first
    mark.background_color = "none"

    manipulate! do |image|
      tile = Magick::ImageList.new
      page = Magick::Rectangle.new(0, 0, 0, 0)

      (image.columns / mark.columns.to_f).ceil.times do |x|
        (image.rows / mark.rows.to_f).ceil.times do |y|
          tile << mark.dup
          page.x = x * tile.columns
          page.y = y * tile.rows
          tile.page = page
        end
      end

      image.composite(tile.mosaic, 0, 0, Magick::OverCompositeOp)
    end
  end
end

Finally, this uploader makes and stores watermarked images on uploading images!

References

*1:Web service that I'm involved in as Software Architect / Lead Software Engineer

Remove all ads

Separate JavaScript Easily and Moderately from Rails with WebPack

Ruby Ruby on Rails JavaScript

The original post was published on Nov. 29, 2015, by me in Japanese.

Introduction

To dive into the world of front-end, I came up with an idea to separate JavaScript easily and moderately from a Rails project.

Some have already shared various ideas, but I found them too much or required more than couple of tools so they seemed like high hurdles. That's why I sought for easier ideas.

Advantages

  • To utilize good points of Sprockets
  • To introduce only WebPack as additional tool, practically
  • To set up a few items, which are easy to do
  • Almost maintenance-free, I believe

Summary

As the following flow, you will program, build and publish your application with JavaScript:

  1. Program JavaScript under frontend directory
  2. Build JavaScript you developed under frontend to app/assets/javascripts
  3. Publish files under app/assets/javascripts without combining files

I think a good point of Sprockets is to publish files with fingerprint in the 3rd part of the flow. It costs a lot to cover this feature using other build tools for front-end.

In other words, if you leave it to Sprockets, you can separate JavaScript from Rails without large scale settings.

Views are still in Rails but the separation of this extent is good enough because if you want to clear all front-end from Rails, it might be okay to use Rails API in the first place.

Concrete Flow

To concretize summary a little, the flow would be as below:

  1. Do modern JS development in frontend/src/javascripts/hoge.js
  2. Build frontend/src/javascripts/hoge.js with WebPack and output assets/javascripts/hoge.js
  3. Publish assets/javascripts/hoge.js with fingerprint using Sprockets

Tools and Versions

Basically, only WebPack is the additional tool. To transpile them on a build with WebPack, Babel is introduced.

  • Sprockets 3.4.1 (+ Rails 4.2.5)
    • Publication (with MD5 fingerprint)
  • WebPack 1.12.9 (+ Babel 6.1.18)
    • Build

Structure of Directories

Add frontend directory to the root of a Rails project.

$ ls
Gemfile       README.md     app/          config/       db/           lib/          spec/
Gemfile.lock  Rakefile      bin/          config.ru     frontend/     public/       vendor/

The structure of the frontend directory is as below:

$ tree frontend -I node_modules
.
├── config
│   ├── development
│   │   └── webpack.config.js
│   └── production
│       └── webpack.config.js
├── package.json
├── src
│   └── javascripts
│       └── application.js
└── test
    └── javascripts

Roughly, it consists of for elements:

  • package.json
  • Config of WebPack
  • Directory for source files
  • Directory for test files

Procedure of settings

I'll explain the procedure of settings.

  1. Set Sprockets to publish JS files separately
  2. Add settings of JS build to WebPack
  3. Hook build by WebPack to Precompile

1. Set Sprockets to publish JS files separately

As the default settings, Rails binds all JavaScript files into application.js to publish it, so we need to change this setting.

Clean up under app/assets/javascripts

Make the directory empty because app/assets/javascripts will be the directory where built JS files are placed.

Add the following lines to .gitignore to exclude this directory from git management.

!/app/assets/javascripts/.keep
/app/assets/javascripts

Plus, as I wanted to publish CSS files separately, I removed *= require_tree . from app/assets/stylesheets/application.scss.

Add targets of precompile by Sprockets

As only application.js and application.css are the targets as the default, add settings to config/initializers/assets.rb to change all JavaScript and CSS files to be compile targets.

Rails.application.config.assets.precompile += %w(*.js *.css)

Exclude files whose names start with underscore

Some gems related to views sometimes raise errors at precompile.

In this case, it's fine to use the following setting which excludes files starting from underscore from the targets instead of the above setting.

Rails.application.config.assets.precompile << /(^[^_\/]|\/[^_])[^\/]*(\.js|\.css)$/

Supplements: How to include JavaScripts and CSSs in Rails

Respectively, calling the following methods inside view templates enables you to use them:

  • javascript_include_tag helper
  • stylesheet_link_taghelper

2. Add settings of JS build to WebPack

Install WebPack and Bable which is used as transpiler with WebPack. If package.json isn't right under frontend, execute npm init.

$ npm init
$ npm install -D webpack babel babel-loader babel-core

Example: Use React.js, ES2015 and Stage2

As an example, do settings to use React.js (0.14.3), ES2015 and Stage2.

At first, install React.js:

$ npm install --save react react-dom

After that, install required presets of Babel:

$ npm install -D babel-preset-react babel-preset-es2015 babel-preset-stage-2

To enable WebPack to transpile JavaScript files, make setting files:

module.exports = {
  devtool: 'inline-source-map',
  entry: {
    application: './src/javascripts/application.js',
  },
  output: {
    path: '../app/assets/javascripts',
    filename: '[name].js'
  },
  module: {
    loaders: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        loader: 'babel?presets[]=react,presets[]=es2015,presets[]=stage-2'
      }
    ]
  }
}

If you exclude devlool options, you can use it as production settings, I believe:

module.exports = {
  entry: {
    application: './src/javascripts/application.js',
  },
  output: {
    path: '../app/assets/javascripts',
    filename: '[name].js'
  },
  module: {
    loaders: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        loader: 'babel?presets[]=react,presets[]=es2015,presets[]=stage-2'
      }
    ]
  }
}

Now, if you execute webpack --config config/{enviroment}/webpack.config.js, JavaScripts registered as entries will be built to app/assets/javascripts/*.js.

Using the above settings, frontend/src/javascripts/application.js will be built to app/assets/javascripts/application.js.

You just need to add new entries and include them in views using helper if you want to add another JavaScript files.

Using npm run

Preparing commands such as npm run build is handy, so register the following three commands:

  • release: webpack --config config/production/webpack.config.js"
  • build: webpack --config config/development/webpack.config.js"
  • watch: webpack --watch --config config/development/webpack.config.js"
{
  "scripts": {
    "release": "webpack --config config/production/webpack.config.js",
    "build": "webpack --config config/development/webpack.config.js",
    "watch": "webpack --watch --config config/development/webpack.config.js"
  }
}

Then, if you type npm run build right under frontend, build with WebPack will be exceuted.

3. Hook build by WebPack to Precompile

Finally, hook npm run release just before executing assets:precompile to do nothing at deployment:

task :build_frontend do
  cd "frontend" do
    sh "npm install"
    sh "npm run release"
  end
end

Rake::Task["assets:precompile"].enhance(%i(build_frontend))

Now, if you type rake assets:precompile, npm run release will be done just under frontend before assets:precompile.

Concrete example

I've made a repository with the above structure.

I've committed all setting that I mentioned here on initial commit.

Conclusion

To separate JavaScript easily and moderately from Rails, I adopt this approach:

  1. To build JavaScripts to app/assets/javascripts with WebPack
  2. Publish files under app/assets/javascripts as they are with Sprockets

The pros of this approach are that there are only a few settings and additional tools, and easy to maintain. It makes settings of building JavaScript simple and not required to replace Sprockets forcefully.

After I read Why we should stop using Grunt & Gulp, I've been wondering if I can set it succinctly without Gulp. I've realized that thanks to accepting Sprockets easier than I expected.

The isolation level seems to be moderate, I think.

Postscript: Complete separation

I said that the benefit of Sprockets is to publish them with fingerprints but I found a plugin which can do it for WebPack.

You can separate JavaSscript from Rails completely because WebPack with this plugin will also publish them with fingerprints.

After building JavaScripts, JSON as below will be outputted for resolving paths.

{
    "one": {
        "js": "/js/one_2bb80372ebe8047a68d4.bundle.js"
    },
    "two": {
        "js": "/js/two_2bb80372ebe8047a68d4.bundle.js"
    }
}

You can see an example with Rails at Using this with Rails in README and it looks really easy.

References

Remove all ads

Went to RubyKaigi 2016 as a Ruby Sponsor

Ruby

I went to RubyKaigi 2016 that was held at Kyoto International Conference Center in Kyoto, Japan from Sep. 8th to 10th.

rubykaigi.org

f:id:necojackarc:20160915003923j:plain

My company, PIXTA, was one of the Ruby sponsors*1 of this conference.

f:id:necojackarc:20160915004044j:plain

I had plans on the Saturday so I was able to attend only the first two days but I enjoyed the conference well.

Aside from the conference, the main shop of Tenkaippin Ramen (天下一品総本店) is located near the venue, so I had a ramen which was only available in the shop, Ramen with Beef Sinew (牛すじラーメン). That was great!

f:id:necojackarc:20160915005424j:plain

Many things in Kansai region brought back memories since I went Osaka University, and I met up with some friends in Osaka during this trip.

The trip was so awesome and I had a lot of fun!

*1:Ruby sponsors are the supreme sponsors!

Remove all ads

Cool Way to Control Retry with ActiveJob

Ruby Ruby on Rails

ActiveJob brings benefits but it has just simple features yet.

So, I've made a module with the APIs published officially to add some features of controlling retry.

An Issue about Retry

ActiveJob doesn't have enough features about retrying jobs as of today. It only provides theretry_job*1 which enqueues itself again with some options.

An naive sample is here:

class RetryJob < ApplicationJob
  queue_as :default

  rescue_from(StandardError) do
    retry_job(wait: 5.minutes)
  end

  def perform(*args)
    # Do something later
  end
end

Is seems okay. When a job fails for some reason, the job is enqueued and performed in 5 minutes.

However, what if you want to limit the number of retry times? On the above sample, if a job never succeeds, it retries forever. There is, unfortunately, no way to control it with ActiveJob on default settings.

Of course, you can find some gems like ActiveJob::Retry, but there is no dominant gems in this field yet. As far as I can make out, ActiveJob::Retry is the gem gathering stars the most in Github though, it is not enough sophisticated to use in production.

This is an alpha library in active development, so the API may change.

isaacseymour/activejob-retry

Plus, I feel the gem is kinda thick.

What We Really Want to Do

I think it's gonna be okay if we can set the limit number of retires and find out the number of attempts and whether retry count is exceeded or not.

So, what we want to do are:

  • Setting the number of retry limit
  • Finding out the attempt number
  • Checking whether the retry limit is exceeded or not

Like this:

class LimitedRetryJob < ApplicationJob
  queue_as :default
  retry_limit 5

  rescue_from(StandardError) do |exception|
    raise exception if retry_limit_exceeded?
    retry_job(wait: attempt_number**2)
  end

  def perform(*args)
    # Do something later
  end
end

Let's implement these methods.

How to Implement it

To tell you the truth, the official document tells us a great idea of that. Overriding the serialize and the deserialize enables us to carry over instance variables which contain serializable objects.

Now, we can implement the above idea like this:

class ApplicationJob < ActiveJob::Base
  DEFAULT_RETRY_LIMIT = 5

  attr_reader :attempt_number

  class << self
    def retry_limit(retry_limit)
      @retry_limit = retry_limit
    end

    def load_retry_limit
      @retry_limit || DEFAULT_RETRY_LIMIT
    end
  end

  def serialize
    super.merge("attempt_number" => (@attempt_number || 0) + 1)
  end

  def deserialize(job_data)
    super
    @attempt_number = job_data["attempt_number"]
  end

  private

  def retry_limit
    self.class.load_retry_limit
  end

  def retry_limit_exceeded?
    @attempt_number > retry_limit
  end
end

If you put this ApplicationJob, you will be able to set a limit on each jobs through ApplicationJob.retry_limit, get the number of attempts via ApplicationJob#attempt_number, and check if the retry count exceeds the limit or not calling ApplicationJob#retry_limit_exceeded?.

  • ApplicationJob.retry_limit
    • To set the number of retry limit
  • ApplicationJob#attempt_number
    • To Find out the attempt number
  • ApplicationJob#retry_limit_exceeded?
    • To check whether the retry limit is exceeded or not

Use in production

I’ve made a module based on this idea for use in production since it’s not a good idea to add methods, which not all subclasses require, to the superclass.

necojackarc/active_job_retry_controlable.rb - Gist

The module puts ApplicationJob.retry_limit, ApplicationJob#attempt_number and ApplicationJob#retry_limit_exceeded? on your jobs.

Pros

It only calls APIs declared officially, so it's not easy to brake. Just providing simple methods, you can easily make your own retry logic.

Cons

It's dead simple, so you need to implement your own retry features. It never enables jobs to retry themselves automatically.

*1:Apparently, you can use the enqueue instead of the retry_job. Both look identical. ref: ActiveJob::Enqueuing

Remove all ads

Have made a gem to retrieve holidays from Google Calendar

Ruby

I've made a gem to retrieve national holidays from Google Calendar with simple interface.

github.com

What motivates me is there is no suitable gem for my purpose as far as I can see.

What I want to do

What I want to do and some features the gem provides are below:

  • List holidays in a particular month
  • List holidays in a particular year
  • Check a date whether it is a holiday or not
  • Retrieve holidays of a lot of countries

Plus, I want to do these thing with LESS API ACCESS to Google Calendar.

Strengths

Notable features are CACHEING and PRELOAD. There are some tools which have similar features, but they need API access every time.

As for my gem, it needs a few API access when you use the preload feature properly.

For example, when you pass date_range: 1.year as a preload parameter, it is going to retrieve and cache holidays from "this day last year" to "this day next year" at initialization. As far as you access days only in this range, the gem never calls Google API.

In adittion, the caching feature adopts LRU algorithm to care about memory leak.

How to use it

Sample code is below:

require "holidays_from_google_calendar"

usa_holidays = HolidaysFromGoogleCalendar::Holidays.new do |config|
  config.calendar = {
    nation: "usa",
    language: "en"
  }

  config.credential = {
    api_key: "YOUR OWN GOOGLE API KEY"
  }
end

usa_holidays.in_year(Date.parse("2016-02-06")) # Retrieve 2016's holidays
=> [#<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42df748 @date=Fri, 01 Jan 2016, @name="New Year's Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42def28 @date=Mon, 18 Jan 2016, @name="Martin Luther King Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42de820 @date=Sun, 14 Feb 2016, @name="Valentine's Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42de0f0 @date=Mon, 15 Feb 2016, @name="Presidents' Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42dd808 @date=Sun, 13 Mar 2016, @name="Daylight Saving Time starts">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42dc8e0 @date=Sun, 27 Mar 2016, @name="Easter Sunday">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab3b40 @date=Wed, 13 Apr 2016, @name="Thomas Jefferson's Birthday">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab36b8 @date=Sun, 08 May 2016, @name="Mother's Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab3230 @date=Mon, 30 May 2016, @name="Memorial Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab2d08 @date=Sun, 19 Jun 2016, @name="Father's Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab26a0 @date=Mon, 04 Jul 2016, @name="Independence Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab21a0 @date=Mon, 05 Sep 2016, @name="Labor Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab1cc8 @date=Mon, 10 Oct 2016, @name="Columbus Day (regional holiday)">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab1610 @date=Mon, 31 Oct 2016, @name="Halloween">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab0f30 @date=Sun, 06 Nov 2016, @name="Daylight Saving Time ends">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab0a58 @date=Tue, 08 Nov 2016, @name="Election Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab0648 @date=Fri, 11 Nov 2016, @name="Veterans Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d4ab0238 @date=Thu, 24 Nov 2016, @name="Thanksgiving Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42d7f20 @date=Sat, 24 Dec 2016, @name="Christmas Eve">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42d7a70 @date=Sun, 25 Dec 2016, @name="Christmas Day">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42d7660 @date=Mon, 26 Dec 2016, @name="Christmas Day observed">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42d70e8 @date=Sat, 31 Dec 2016, @name="New Year's Eve">,

usa_holidays.in_month(Date.parse("3rd March 2016")) # Retrieve holidays of March, 2016
=> [#<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42dd808 @date=Sun, 13 Mar 2016, @name="Daylight Saving Time starts">,
 #<HolidaysFromGoogleCalendar::Holiday:0x007ff7d42dc8e0 @date=Sun, 27 Mar 2016, @name="Easter Sunday">]

usa_holidays.holiday?(Date.parse("Oct 31 2016")) # Halloween
=> true

usa_holidays.holiday?(Date.parse("April 16th 2016")) # Satruday
=> true

usa_holidays.holiday?(Date.parse("April 17th 2016")) # Sunday
=> true

usa_holidays.holiday?(Date.parse("Aug 2 2016")) # Weekday (Tuesday)
=> false

Impression

I need to learn data structures and algorithms more and more.

Remove all ads

How to read a text file in batches with Ruby

Ruby

When I was reviewing my colleague's source, I found awful code in his program.

Like this:

File.open("./production.log") do |file|
  results = {}
  file.readline # skip header
  file.each_line do |line|
    key, value = process(line)
    results[key] = value
    next if results.size < 2000
    output(results)
    results = {}
  end
  output(results)
end

Oops! How did this happen!?

He said "We need purge results of memory every 2,000 lines because that causes out of memory. And there is no method to get 'each batch of lines' from text files in Ruby."

I got it, we need some methods to read files in batches, like a "find_in_batches" of ActiveRecord. Then I searched such methods, and I found a same question as ours in Stack Overflow.

stackoverflow.com

Those answers will go well, but those are not cool! Finally, I couldn't find out great ways.

Ideal ways

I'd like to use like a "find_in_batches" method, like this:

File.open("./production.log", skip_header: true) do |file|
  file.batch_line(batch_size: 2000) do |lines|
    results = process(lines)
    output(results)
  end
end

Or, it seems nice to wrap text file by an object which has the "batch_line*1" method:

file = TextFilePager.new("./production.log", skip_header: true)
file.batch_line(batch_size: 2000) do |lines|
  results = process(lines)
  output(results) 
end

Both are better than previous code. We can easily understand what this code will do.

First Proposal Way

Using "Enumerator::Lazy" and "each_slice" seems one of the great ways:

File.open("./production.log") do |file|
  file.lazy.drop(1).each_slice(2000) do |lines|
    results = process(lines)
    output(results)
  end
end

It's really simple way, because we need no additional class and method.

All we need is understand "Enumerator::Lazy*2" and "each_slice", that's it.

Second Proposal Way

Making a new class, like "TextFilePager", seems a nice idea:

class TextFilePager
  DEFAULT_BATCH_SIZE = 1000

  def initialize(file_path, skip_header: false, delete_line_break: false)
    @file_path = file_path
    @skip_header = skip_header
    @delete_line_break = delete_line_break
  end

  def batch_line(batch_size: DEFAULT_BATCH_SIZE)
    File.open(@file_path) do |file|
      file.gets if skip_header?
      loop do
        line, lines = "", []
        batch_size.times do
          break if (line = file.gets).nil?
          lines << (delete_line_break? ? line.chomp : line)
        end
        yield lines
        break if line.nil?
      end
    end
  end

  def skip_header?
    @skip_header
  end

  def delete_line_break?
    @delete_line_break
  end
end

You can use this class like this:

file = TextFilePager.new("./production.log", skip_header: true)
file.batch_line(batch_size: 2000) do |lines|
  results = process(lines)
  output(results) 
end

This code is absolutely same what I wrote above section. The reason is clear, I've just created this class along with my ideal's interface, like TDD.

Conclusion

I prefer first proposal because it only requires knowledge of Ruby and we don't need to make a new class. Too many classes bother us, so we shouldn't add a class unless we really need that.

But second way gives us good interface, it seems intuitive. Plus, we can easily pass options to that object. If you need such text file processing time and time again, adding a new class may be a good way.

Perhaps, it's nice to add the "batch_line" method to "IO" or "File" object. But I don't like to modify core classes so much, because that affects our code widely.

Hence, I'll choose to use "Enumerator::Lazy" and "each_slice" at first when it comes to that matter.

Thanks for reading.

An appendix

Here is my original post about that matter, written in Japanese.

*1:Of course, this method doesn't exist yet.

*2:If you don't need to use `drop`, `lazy` is unnecessary.

Remove all ads