rails twitter search using search.atom

Recently, I had to create a twitter live feed for a work I’m doing…. so I found the gems for searching in twitter, but all of them were working with the search.json api, but since my client wanted the same results that he saw in twitter.search.com, I created a lib for that project (following the gem example code), in order to use the search.atom api and parse the xml results with hpricot.

If you deal with the same problem, here you have the code

require ‘rubygems’
require ‘net/http’
require ‘cgi’

class Twitter
TWITTER_SEARCH_API_URL = ‘http://search.twitter.com/search.atom’
DEFAULT_TIMEOUT = 5
HEADERS = { “Content-Type” => ‘application/rss+xml’,
“User-Agent” => ‘twitter-search’ }

def self.search(opts)
search_for = ‘#twitter’

url = URI.parse(TWITTER_SEARCH_API_URL)
url.query = sanitize_query(opts)

ensure_no_location_operators(url.query)

req = Net::HTTP::Get.new(url.path)
http = Net::HTTP.new(url.host, url.port)
http.read_timeout = DEFAULT_TIMEOUT

res = http.start { |h| h.get(”#{url.path}?#{url.query}”, HEADERS) }

if res.code == ‘404′
raise “Twitter responded with a 404 for your query”
end

self.parse_search(res.body)
end

def self.sanitize_query(opts)
if opts.is_a? String
“q=#{CGI.escape(opts)}”
elsif opts.is_a? Hash
“#{sanitize_query_hash(opts)}”
else
raise “sanitique_query expects a String or a Hash”
end
end

def self.sanitize_query_hash(query_hash)
query_hash.collect{|key, value| “#{CGI.escape(key.to_s)}=#{CGI.escape(value.to_s)}” }.join(’&’)
end

def self.ensure_no_location_operators(query_string)
if query_string.include?(”near%3A”) ||
query_string.include?(”within%3A”)
raise “near: and within: are available from the Twitter Search web interface, but not the API. The API requires the geocode parameter. See dancroak/twitter-search README.”
end
end

def self.parse_search(body)
doc = Hpricot.parse(body)

entries = []

items = (doc/:entry)
items.each do |raw_item|
entry = {}

entry

(Via Rorra’s blog.) Original Link: rails twitter search using search.atom

rake sh en otro directorio

Problema: Quiero ejecutar un comando desde otro directorio en rake (por ejemplo, ejecutar un makefile que está en un subdirectorio).
Solución: Agrego al rakefile la posibilidad de ejecutar comandos en otro directorio. Para eso, al principio de mi rakefile puse:

require 'fileutils'

def sh_in_dir( dirname, *args, &block )
  old_path = pwd
  FileUtils.chdir( dirname )
  sh( *args, &block )
  FileUtils.chdir( old_path )
end
Happy hacking,

Aureliano.

(Via aurelianito.) Original Link: rake sh en otro directorio

RubyInstaller: Updated packages, and other news.

A few days of silence in the middle, but definitely progress in Windows land… Readline, new packages and more news…

Pure-Ruby Readline for the win!

If was the case that under your environment every time you use IRB your CPU went crazy, you will be happy to hear that we found the problem.

More than just find it, we put our some money to it and paid Park Heesob (from win32utils project fame) to code a Pure-Ruby Readline library.

The crazy part is that works on 1.8, 1.9, on Windows and Linux!, even more crazy, it also worked under Rubinius!

The code is in GitHub here

Latest packages in RubyInstaller download page have replaced the GNU Readline binary with it, and IRB now works normally.

Ruby 1.9.1-p129 got out.

As announced in Ruby-Lang website (here) 1.9.1-p129 went out last week.

While test on Windows for either trunk and 1.9.1 branch still segfault, I decided to do another release of this and put the 7zip packages in the downloads page at RubyInstaller.

Yes, no official Windows Installer yet, still need to iron some quirks (mentioned in previous post here)

Again, you can download those from usual place

Moved to Windows 7

I definitely been one of the late adopters in OS changes. Been using Windows XP for work for the past 7 years (moved directly from Windows NT4) — yeah, love to skip versions.

So decided to install latest Release Candidate as main OS, and start using it. So far, the experience has been great.

Keep in mind that I rarely turned my laptop off or restarted it (even with XP). Sleep and hibernation are the two options I mostly use, and haven’t restarted the computer since installed Windows 7, so is good

The following is a list of things I noticed that will love to expand in other posts:

Mount Users in other partition, don’t keep with your system

Like on Linux, dedicate a partition to your personal files, so I did it for the user folder. Is tricky, but can be done. This ease the restoration process of the OS since you don’t need to worry about losing your personal files.

Set HOME

Ensure HOME get set to %HOMEDRIVE%@%HOMEPATH%@. This is to make RubyGems and Ruby expansion of @~@ will work properly (and avoid nasty issues with known scripts or gems).

Console 2 halts

For some reason, the combo of Console 2 and VIM when working with Git just halts, not accepting any input. This also affected IRB, which lead me to think that is readline related.

Anyhow, installed GVim and installed Fabio Akita’s vimfiles (instructions here)

Then, ensured Git uses it doing git config --global core.editor gvim

Hidden files behave as read only?

Found that Ruby and some programs don’t let you modify hidden files. They just return access denied.

This seems weird, since Notepad can edit those files without issues.

Will investigate further, seems a bad usage of Windows API.

Don’t pollute your system.

Put all your tools, DLLs and your scripts in your user folder. Mine is %HOME%\Tools\bin, add this to your User environment variables: %HOME%\Tools\bin, et voila!

Don’t copy files to system32 blindly, this includes SQLite3, MySQL or any of these tools, putting things there make things harder to find or update later.

Also, that requires Admin rights, which are annoying if you don’t have them (I have UAC to the highest value to avoid do any stuff that normal non-admin users will do).

See my Tools layout

Some gem issues and news

While doing the move to new OS, decided to build a few gems again, and maybe update them.

MySQL 5.1 and mysql gem are a nightmare, stick to 5.0 for now.

Yeah, it seems that mysql Ruby C Extension have several memory issues. Stick to 5.0 for know (or switch to DataMapper, I heard they got things working on Windows!)

Been working in my own fork to support MinGW properly. Check the code at GitHub:

http://github.com/luislavena/mysql-gem

Almost 100% cross-platform SQLite3 ruby gem!

Lot of progress on this, partially stolen from do_sqlite3 (and viceversa!)

My fork here is capable to build Windows binaries on Linux/OSX.

Ideas of making fat binaries are around, but I believe this can be worked out with some love to RubyGems (discussed last year).

Will make binaries of those gems this week.

So, what are you waiting for?

Start using it!, like Mike Hodgson that reported his success here

(Via DEV_MEM.dump_to(:blog) – Multimedia systems blog.) Original Link: RubyInstaller: Updated packages, and other news.

Rails 2.3, rake y reenable method

undefined method `reenable' for <Rake::Task db:schema:dump =>
[environment]>:Rake::Task

  s.add_dependency('rake', '>= 0.8.3')
  s.add_dependency('activesupport',    '= 3.0.pre' + PKG_BUILD)
  s.add_dependency('activerecord',     '= 3.0.pre' + PKG_BUILD)
  s.add_dependency('actionpack',       '= 3.0.pre' + PKG_BUILD)
  s.add_dependency('actionmailer',     '= 3.0.pre' + PKG_BUILD)
  s.add_dependency('activeresource',   '= 3.0.pre' + PKG_BUILD)

(Via Gastón Ramos – Ruby, Rails….) Original Link: Rails 2.3, rake y reenable method

RubyInstaller: Getting Started with Rails and SQLite3

A few folks asked this over the past weeks, and since I released preview1 version of RubyInstaller, wouldn’t be awesome I write a guide for it?

So, here we go

First, Getting Ruby

Please download from here one of the preview1 installers for Ruby.

For this demonstration, I’m going to use Ruby 1.9.1-p129. Downloaded, executed and installed to it’s default location C:\Ruby19

Now, start a Command prompt with Ruby (under start menu, inside Ruby 1.9.1-p129).

Getting SQLite3

As you may know, Windows do not came out of the box with libraries like SQLite3. For this, we are going to download it from Internet.

Going to SQLite downloads, under Precompiled binaries for Windows, download sqlitedll-3_6_16.zip and sqlite-3_6_16.zip. Those two packages contains the DLL (sqlite3.dll) and the command line executable for SQLite (sqlite3.exe).

Please extract those contents inside C:\Ruby19\bin

Now, let’s get the Ruby bindings for SQLite3

Getting SQLite3/Ruby bindings

As you may know, the preferred way to distribute Ruby libraries is using Gems. Users on other platforms usually compile themselves these components, using the mechanisms provided by the operating system distribution.

We are going to avoid the compilation process using pre-compiled binaries that has been cooked for us.

So, back to the Command Prompt with Ruby, let’s install the sqlite3 bindings:

gem install sqlite3-ruby --source http://gems.rubyinstaller.org

Adding --source help us indicate a non-standard location from where gems are going to be installed. RubyInstaller team has built and packaged special versions of these gems that we hope get published soon into RubyForge, the official place for gem distribution.

Once installed, you should see something like this at the screen:

Successfully installed sqlite3-ruby-1.2.4.1-x86-mingw32
1 gem installed

These special version of the gem are fat binaries, which means these can be safely installed on Ruby 1.8.6 or 1.9.1.

Getting Rails

Now it’s time to install and build a Rails application. First, let’s install the rails gem:

gem install rails

That command is going to take a bit, since is a 2MB or so download, and will install several of the Rails dependencies (ActiveRecord, ActionPack, ActiveSupport, etc). You should expect similar output to this:

Successfully installed activesupport-2.3.2
Successfully installed activerecord-2.3.2
Successfully installed actionpack-2.3.2
Successfully installed actionmailer-2.3.2
Successfully installed activeresource-2.3.2
Successfully installed rails-2.3.2
6 gems installed

Let’s build our application now.

Creating a Rails application.

Let’s call the application railsapp

rails railsapp

An excerpt of the output you should get:

      create
      create  app/controllers
      create  app/helpers
      create  app/models
      create  app/views/layouts
      create  config/environments
      create  config/initializers
      create  config/locales
      create  db
...

Rails 2.3.2 defaults it’s database format to SQLite3, so there is no need for us to tweak anything.

Checking if everything is ok, using script\about:

cd railsapp
ruby script\about

And the output should be something like this:

About your application's environment
Ruby version              1.9.1 (i386-mingw32)
RubyGems version          1.3.4
Rack version              1.0 bundled
Rails version             2.3.2
Active Record version     2.3.2
Action Pack version       2.3.2
Active Resource version   2.3.2
Action Mailer version     2.3.2
Active Support version    2.3.2
Application root          C:/Users/Luis/railsapp
Environment               development
Database adapter          sqlite3
Database schema version   0

Now, it’s up to you create your models, controllers and everything.

Some notes and considerations.

At the time of this writing, Mongrel has not been updated to build and install properly either on 1.9 or MinGW versions of Ruby.

As part of Mongrel development team, I’m going to work on a solution for this in the upcoming days.

If you find something wrong with the Ruby Installer, please report here, but issues with your code, Rails or other are not responsability of RubyInstaller.

The binary gems provided at gems.rubyinstaller.org are based on our forks of sqlite3-ruby, which can be cloned and explored here at GitHub.

Keep in mind that some gems would not work under Ruby 1.9, or you will need a compiler (DevKit) for it. See previous post with details how to get those from our download page.

(Via DEV_MEM.dump_to(:blog) – Multimedia systems blog.) Original Link: RubyInstaller: Getting Started with Rails and SQLite3

RubyInstaller: Getting Started with Rails and MySQL

This is a follow up instruction set from previous post, but this time, using MySQL

Getting Ruby

These steps are the same for Ruby 1.9 or Ruby 1.8, please feel free to download the installer from here

For this guide I’m going to use Ruby 1.9.1-p129, since it the coolest new version that all the guys are playing with

Now, start a Command prompt with Ruby (under start menu, inside Ruby 1.9.1-p129).

Getting the right MySQL version

While building the MySQL/Ruby bindings, we found that mixing versions of the bindings with different versions of MySQL installations ended on undesired results (abnormal program termination, weird errors, etc.)

For this guide, and because we are going to use binary gems, we are going to stick to MySQL version 5.0.83.

Now is time to download MySQL. For this guide, I’m going to install the essentials version, which contains only MySQL and command line tools, no Query Builder or any other administrative tool.

Please go to this page and download Windows Essentials (x86). Once downloaded you will end with mysql-essential-5.0.83-win32.msi file. Execute it and install with defaults.

Configure MySQL

If you’re an advanced and savvy MySQL user, you can skip the following steps. For the sake of this guide, I’m going to list the simple options you must follow when installing it.

Once you installed MySQL, the installer should have started the Configuration Wizard page.

Inside of it, please apply the following options:

Once done with all this, on the summary screen, please click Execute to complete the configuration of MySQL Server.

Under some system, starting of MySQL server will fail during this wizard, but don’t be afraid, most of the times, this can be safely ignored.

To check everything was properly installed, please go to the Start Menu, and inside MySQL Server 5.0, click on MySQL Command Line Client

If you changed the root password, or, like me, leaved it unchecked, you can simply press enter when asked for the password and see that the server is running!

If you decided to add MySQL to the PATH, you will require to restart your computer so the PATH change is available to the system.

If you decided not to add MySQL to the PATH, please go, with Explorer to the location where you installed MySQL Server and copy libmySQL.dll into C:\Ruby19\bin

In my case, I found this file in C:\Program Files\MySQL\MySQL Server 5.0\bin

Now, it is time to install the bindings.

Getting MySQL/Ruby

For this version of Ruby, there is no official binary gems for both Ruby 1.8 and 1.9. So we are going to install the specially built version from RubyInstaller gems repository.

At the Command Prompt with Ruby, please enter the following command:

gem install mysql --source http://gems.rubyinstaller.org
  

This is going to install the special version of MySQL bindings. This version works with Ruby 1.8 and 1.9, since bundles fat binaries. You should expect a similar output like this:

Successfully installed mysql-2.8.1.1-x86-mingw32
1 gem installed

Getting Rails

Now is time to install Rails and build our application. At the same command prompt, please enter the following command:

gem install rails

This is going to take a bit, since Rails and it’s dependencies takes around 2MB or so, and need to be downloaded and installed.

Once done, expect see at the screen something like this:

Successfully installed activesupport-2.3.2
Successfully installed activerecord-2.3.2
Successfully installed actionpack-2.3.2
Successfully installed actionmailer-2.3.2
Successfully installed activeresource-2.3.2
Successfully installed rails-2.3.2
6 gems installed

Creating a Rails application

Let’s name our application mysqlapp

rails mysqlapp --database=mysql

The --database option indicates to Rails that we want to use MySQL instead of the default database adapter (SQLite3).

Rails will output a lot of lines when creating your application structure, just an excerpt of what to see:

...
      create  config/database.yml
      create  config/routes.rb
      create  config/locales/en.yml
      create  config/initializers/backtrace_silencers.rb
      create  config/initializers/inflections.rb
      create  config/initializers/mime_types.rb
      create  config/initializers/new_rails_defaults.rb
      create  config/initializers/session_store.rb
      create  config/environment.rb
...

Configuring our Database

Now Rails have configured for us the name of the database we want to use, and you can verify it in config\database.yml

Rails will try to connect to mysqlapp_development, but that database do not exist in our fresh new MySQL server.

So, let’s create it:

cd mysqlapp
rake db:create

Just that, simple db:create is going to connect to our MySQL server, and create the database for us.

Keep in mind that if you changed root password or want to use other MySQL user to connect to the database, you need to edit database.yml to reflect those changes.

Let’s verify that everything is in place, using the following command:

About your application's environment
Ruby version              1.9.1 (i386-mingw32)
RubyGems version          1.3.4
Rack version              1.0 bundled
Rails version             2.3.2
Active Record version     2.3.2
Action Pack version       2.3.2
Active Resource version   2.3.2
Action Mailer version     2.3.2
Active Support version    2.3.2
Application root          C:/Users/Luis/mysqlapp
Environment               development
Database adapter          mysql
Database schema version   0

Now is up to you to create your models, controllers and views!

Some notes and considerations

On other post I’m going to guide you with steps on building the bindings against MySQL 5.1.36, since you will need to install the Ruby Development Kit and the development headers for MySQL.

If you find something wrong with the Ruby Installer, please report it here, but issues with your code, Rails or other are not responsability of RubyInstaller.

The binary gems provided at gems.rubyinstaller.org are based on our forks of mysql bindings, which can be cloned and explored here at GitHub.

Keep in mind that some gems would not work under Ruby 1.9, or you will need a compiler (DevKit) for it. See previous post with details how to get those from our download page.

(Via DEV_MEM.dump_to(:blog) – Multimedia systems blog.) Original Link: RubyInstaller: Getting Started with Rails and MySQL

I’m back with some Code …

Well, hello again, long time since the last post. I went on vacations, work a lot and did some programming. Let’s talk abount the programming part, since it is the most interesting one.

I created a small library called “Esearchy” capable of searching the internet for email addresses. Currently, we the supported search methods are engines such as Google, Bing, Yahoo, PGP servers, GoogleGroups, etc , but I intend to add many more.
Also, the library searches inside .pdf and .txt files for emails addresses and adds them to the list of found accounts.

For now, there are two main ways of performing a search, “the ruby way”

and the more classic way in which users can create an Esearchy objetc and work on it

For now , that’s it for now , but keep on tuned for more shitty code ajjajaa

(Via 自由編碼人.) Original Link: I’m back with some Code …

Patch en Regexp para poder usarlas como clave en un Hash

Siguiendo mi proyecto de hacer mi wiki en Ruby, encontré un comportamiento muy raro.
Generé un hash (que se llama @rules) que no tiene un elemento que tiene. O sea, @rules[@rules.keys[2]] da nil, pero @rules.values[2] devuelve el objeto asociado a la clave @rules.keys[2]. Como este hash tiene como claves un montón de expresiones regulares, me imaginé que había un problema con el hash y el eql? de Regexp, así que los implementé de nuevo y monkeypatchié.
Este es el código:

class Regexp
 alias_method   ld_rapidito_inspect, :inspect

 def inspect
   @inspect = old_rapidito_inspect if @inspect.nil?
   @inspect
 end

 def eql?( other )
   false if other.class != Regexp
   self.inspect == other.inspect
 end

 alias_method :"==", :eql?

 def hash
   self.inspect.hash
 end
end

Esta corrección me anduvo con la siguiente versión de ruby:

$ ruby --version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

Espero que les sirva.
Happy hacking,
Aureliano.

(Via aurelianito.) Original Link: Patch en Regexp para poder usarlas como clave en un Hash

Más rapidito

Como les estuve contando, sigo escribiendo mi wiki. Ya parsea un subconjunto interesante del lenguaje definido por trac.
Siguiendo la tradición, les cuento como está avanzando el tokenizer. Al tokenizer lo simplifiqué para que devuelva la expresión regular que matcheo junto con el match (en vez del “tipo”). Esto hizo que la interfase para definir las reglas para tokenizar sea más simple. Si no hay ninguna regla que matchee sigue devolviendo ["string", :text].
Sin más, acá el código:

module Rapidito
  class Tokenizer

    def initialize( *delimiters )
      @delimiter_list = delimiters +  [/\z/]
      @match_cache = nil
    end

    def source
      valid_cache? ? @match_cache[0].to_s + @source : @source
    end

    def source=(s)
      @match_cache = nil
      @source = s
    end

    def has_next?
      !@source.empty? || valid_cache?
    end

    def valid_cache?
      (!@match_cache.nil?) && (@match_cache[0].to_s.length > 0)
    end

    def next_match
      @delimiter_list.map {|regex| [regex.match(@source),regex]}.reject {|p| p[0].nil?}.inject do
        |better,new|
        better_pos = better[0].pre_match.length
        new_pos = new[0].pre_match.length

        if better_pos < new_pos
          better
        elsif new_pos < better_pos
          new
        elsif better[0].to_s.length > new[0].to_s.length
          better
        else
          new
        end
      end
    end

    def next_token
      if @match_cache #cached delimiter
        rv = @match_cache
        @match_cache = nil
        return rv
      end

      match = next_match
      p = match[0].pre_match.length
      @source = @source[p + match[0].to_s.length, @source.length]

      if p == 0 #delimiter
        match
      else #text
        @match_cache = match
        [match[0].pre_match, :text]
      end
    end

    def all_tokens
      tokens = []
      while has_next?
        tokens << next_token
      end
      tokens
    end
  end
end

Y si miran los tests de unidad, van a ver que también quedaron más lindos:

require 'test/unit'
require 'rapidito/tokenizer'

include Rapidito

class TokenizerTest < Test::Unit::TestCase

  def test_no_token
    tok = Tokenizer.new
    tok.source = "aaaa"
    assert_equal true, tok.has_next?
    assert_equal ["aaaa", :text], tok.next_token
    assert_equal false, tok.has_next?
  end

  def assert_all_tokens( expected, tokenizer )
    assert_equal expected,
      tokenizer.all_tokens.map { |token, kind| [token.to_s, kind] }
  end

  def test_two_delimiters
    tok = Tokenizer.new(
      /\|/, /;;/
    )

    tok.source = "aa|bbb;;;;cccc"
    assert_all_tokens \
      [ ["aa", :text], ["|", /\|/], ["bbb", :text],
        [";;", /;;/], [";;", /;;/], ["cccc", :text] ],
      tok

    tok.source = "aa;;bbb||cccc"
    assert_all_tokens \
      [ ["aa", :text], [";;", /;;/], ["bbb", :text],
        ["|", /\|/], ["|", /\|/], ["cccc", :text] ],
      tok
  end

  def test_choose_longest_match
    tok = Tokenizer.new(
      /aa/, /aaa/
    )
    tok.source = "aaaa"
    assert_all_tokens [ ["aaa", /aaa/], ["a", :text ] ], tok
  end

  def test_reset_precache
    tok = Tokenizer.new(
      /\|/, /,/
    )
    tok.source = "original start|original end"
    tok.next_token
    tok.source = "new start,new end"
    assert_equal ["new start", :text], tok.next_token
  end

  def test_almost_finished
    tok = Tokenizer.new( /!/ )
    tok.source = "bang!"
    tok.next_token
    assert_equal true, tok.has_next?
    tok.next_token
    assert_equal false, tok.has_next?
  end

  def test_carriage_return_ending
    tok = Tokenizer.new( /!/ )
    tok.source = "bang!\n"
    tok.next_token
    assert_equal true, tok.has_next?
    tok.next_token
    assert_equal true, tok.has_next?
    assert_equal "\n", tok.next_token[0].to_s
    assert_equal false, tok.has_next?
  end

  def test_transparent_caching
    tok = Tokenizer.new( /!/ )
    tok.source = "bang!pum"
    tok.next_token

    assert_equal "!pum", tok.source
  end

  def test_match_klass
    tok = Tokenizer.new( /!/ )
    tok.source = "!bang!pum"

    assert_equal \
      [MatchData, String, MatchData, String],
      tok.all_tokens.map { |tok, kind| tok.class }
  end
end

Happy hacking,
Aureliano.

(Via aurelianito.) Original Link: Más rapidito

Configurar passenger con nginx

Instalar Ruby Enterprise Edition

sudo apt-get install build-essential zlib1g-dev libssl-dev libreadline5-dev

wget http://rubyforge.org/frs/download.php/51100/ruby-enterprise-1.8.6-20090201.tar.gz
tar xvfz ruby-enterprise-1.8.6-20090201.tar.gz
rm ruby-enterprise-1.8.6-20090201.tar.gz
cd ruby-enterprise-1.8.6-20090201/
sudo ./installer

Agregar ruby entrerprise al path del sistema

echo "export PATH=/opt/ruby-enterprise-1.8.6-20090201/bin:$PATH" >> ~/.profile && . ~/.profile

Nginx

sudo /opt/ruby-enterprise-1.8.6-20090201/bin/passenger-install-nginx-module

Elegir la opcion 1. Yes: download, compile and install Nginx for me. (recommended)

Script de inicio Nginx

Agregar el siguiente codigo en

/etc/init.d/nginx
sudo chown root:root /etc/init.d/nginx

Probar una aplicacion rails en nginx
agregar un virtual host

server {
    listen 80;
    # server_name www.mycook.com;
    root /home/deploy/testapp/public;
    passenger_enabled on;
}

(Via Chebyte’s Blog.) Original Link: Configurar passenger con nginx