I want to be free

If you import unicode text into latin1 database column, the symbols would be screwed up — russian symbols become a shit like “залоговый депозит”.

To convert such quickly (as a test) you can use Lebedev’s convertor.

To convert the whole table do the following (thanks to dull.ru):

1. The most important step: dump such data in a file with mysqldump

   mysqldump -u user -p --default-character-set=latin1 --skip-set-charset --no-create-info --extended-insert --complete-insert dbname table > dbname.sql

   If not the whole table data is bad, create another table (CREATE TABLE t2 LIKE t1) and copy wrong rows to the new table.

2. Replace latin1 by utf8 in the file
   This can be done manually in your editor or by this command:

   sed -r 's/latin1/utf8/g' dbname.sql > dbname_utf.sql

3. Convert your latin1-table to utf8 (and maybe truncate it):

   ALTER TABLE `table` CONVERT TO CHARACTER SET 'utf8';

4. Import the utf8 data back to the table:

   mysql -u user -p --default-character-set=utf8 dbname < dbname_utf.sql

That’s it.

It’s handy to use View Helpers in Zend FW driven project. For example, you want to make an in-place tracker (e.g. Google Analytics) — you create a helper class My_View_Helper_Tracker inherited from Zend_View_Helper, Zend finds its automatically and then you are free to use your helper method:

echo $this->tracker( $trackerID );

The question is what you gonna do if you want a base class for a family of trackers?

It’s not so obvoius due to naming conventions.

Let’s say you want 2 kinds of trackers: Google Analytics and Euroads.

1. You create such files structure:

library
 - My
   - View
    - Helper
      - Tracker
        Abstract.php
        - Google
          Page.php
        - Euroads
          Owner.php
          Guest.php

2. You name your classes as:

• My_View_Helper_Tracker_Euroads_Guest (extends My_View_Helper_Tracker_Abstract)
• … so on …

3. Class methods are:

public function tracker_euroads_guest(...) ...

4. And the trickiest part: the helper call:

echo $this->tracker_Euroads_Guest

If you want to work with big files uploads (photos, videos, CSV data dumps) you might need to prepare your PHP server for this — increase the values of these variables in your php.ini:

upload_max_filesize = 512M
post_max_size = 512M
memory_limit = 512M

The same should be done if you try to upload SQL dumps by phpMyAdmin and get a correspondent error message.

During my work on SunnyRentals I’ve got a task to add 2 closest airports to every property an owner creates.

I implemented it quite fast since we have a database of airports with coordinates.

While playing with this feature, we found out that it is not good enough — the real task must be to add 2 closest and biggest airports. The problem is that we don’t have any data in airports DB to guess how big or famous a particular airport is.

So we need to rate every airport somehow…

The solution we found was simple — we need to google for the airport name and get the search results count. The count can be considered as rating value — London Heathrow airport has 2.33 million results while Kiev Zhulyany airport has only 0.77 mln which looks fair.

Several things to pay attention to:

• the query we formed was [city name] + [airport name] + ‘ airport’
• if this query gives zero result, I omit the city name — at times it hepls
• we put the query into quotes to google for the exact phrase, otherwise the London City airport gets the highest rating due to the fact that “city” is a general term
• if the airport name includes the city name (Melbourne Intl), we omit the city name — ”Melbourne Intl airport” is better then “Melbourne Melbourne Intl airport“
• in addition to previous idea — if the airport name sounds like the city name, we omit the city name as well. Example: Narsarsuaq airport in Narssarssuaq city. I used soundex function for this comparison — it’s present in PHP and MySQL.

To get the google results you can use the Google Search API:

$queryTemplate = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=%s';
$airportQuery = '"London Heathrow airport"';
$query = sprintf( $queryTemplate, urlencode( $airportQuery ) );
$json = json_decode( file_get_contents( $query ), 1 );
$rating = (int)$json['responseData']['cursor']['estimatedResultCount'];

Are you aware of Federated engine in MySQL (apart from MyISAM and InnoDB)?

This engine allows you to define a table that sucks data from another table, even from a remore server. The tables definition must be the same.

I use it for the following:

1. Every time I rebuild the project, I have wait for 15 minutes while two big tables are created and filled with data — these are geo data tables (world cities, regions, etc), 4 mln records, and POI table, 2 mln records. I use Federated tables to create two separate databases and just link these tables in my project.
2. These tables are shared between several environments (dev, test and live) on the same server.

To check if your MySQL server has the Federated engine supported, you can use just a phpMyAdmin — go to home page of you phpMyAdmin installation (click Home picture), then choose Engines tab and check there.

If it’s not enabled (gray), open your my.ini file, find the “[mysqld]” part and make it to look like this:

[mysqld]
federated

P.S. If you have an error in the table definition, phpMyAdmin shows your database as empty. To fix this, log in via mysql console and try to make a SELECT from this poorly defined table and you get the error message to work with.

That’s easy, you need 2 things:

1. Fresh currencies exchange rates
2. Some way to excange amount from one currency to another.

This how I did it: get values from European Central Bank (ECB) for step #1 and wrote MySQL user defined function for step #2.

Here is how to export currencies rates from ECB (EUR is a base currency, and I add self rate as 1:1). First I create such database table:

CREATE TABLE IF NOT EXISTS `currency` (
  `code` char(3) NOT NULL DEFAULT '',
  `rate` decimal(10,5) NOT NULL COMMENT 'Rate to EUR got from www.ecb.int',
  PRIMARY KEY (`code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Currency rates (regularly updated)';

Now let’s fill it with rates:

<?php

class CurrencyController extends Controller_Ajax_Action {

  public function importAction() {

    $db = Zend_Registry::get('db');
    $db->beginTransaction();

    $url = 'http://www.ecb.int/stats/eurofxref/eurofxref.zip?1c7a343768baab4322620e3498553b5a';
    try {
      $contents = file_get_contents($url);
      $contents = archive::unzip($contents);
      $contents = explode("\n", $contents);

      $names = explode(',', $contents[0]);
      $rates = explode(',', $contents[1]);

      $names[] = 'EUR';
      $rates[] = 1;

      for ($i = 1; $i < sizeof($names); $i++) {
        if (!(float) $rates[$i]) continue;
        $db->query( sprintf('INSERT INTO `currency`(`code`, `rate`)
              VALUES ("%s", %10.5f)
              ON DUPLICATE KEY UPDATE `rate`=VALUES(`rate`)',
             trim( $names[$i] ),
             trim( $rates[$i] )
        ) );
      }

      $db->commit();
    } catch ( Exception $O_o ) {
      error_log( $O_o->getMessage() );
      $db->rollback();
    }

  }
}

Now let’s create a SQL function for handy converts. Create a udf.sql file and add this in it:

DELIMITER //

DROP  FUNCTION IF EXISTS EXCHANGE;
CREATE FUNCTION EXCHANGE( amount DOUBLE, cFrom CHAR(3), cTo CHAR(3) ) RETURNS DOUBLE READS SQL DATA DETERMINISTIC
    COMMENT 'converts money amount from one currency to another'
BEGIN
    DECLARE rateFrom DOUBLE DEFAULT 0;
    DECLARE rateTo DOUBLE DEFAULT 0;

    SELECT `rate` INTO rateFrom FROM `currency` WHERE `code` = cFrom;
    SELECT `rate` INTO rateTo   FROM `currency` WHERE `code` = cTo;

    IF ISNULL( rateFrom ) OR ISNULL( rateTo ) THEN
        RETURN NULL;
    END IF;

    RETURN amount * rateTo / rateFrom;
END; //

DELIMITER ;

and run this command in your shell:

mysql --user=USER --password=PASS DATABASE < udf.sql

This is how you can use this function — how to convert 10 US dollars to Canadian dollars:

SELECT EXCHANGE( 10, 'USD', 'CAD')

which results in $10 = 10.93 Canadian dollars.

P.S. Consider adding the currency export action call to your cron scripts.

P.P.S. A function to unzip the data file can be got at php.net

I want to share a couple of features I use to handle AJAX requests in projects based on Zend Framework.

1. AJAX request handling

What: some parts of your application can be not loaded if currect request is AJAX.

Why: you don’t need views, templates, some routes — so you can add an AJAX check in your Initializer or bootstrap file and avoid loading not necessary things.

How: Zend Request object has a isXmlHttpRequest method to find out whether it’s AJAX request or not. It’s based on ‘X-Requested-With‘ header, which is sent by jQuery, Prototype, Scriptaculous, YUI and MochiKit frameworks.

2. AJAX Controller

Most AJAX controller’s methods I saw had an exit() inside to not to output Zend’s template — it is a work-around. The proper way to do so is to tell to Zend not to load anything. One step forward is to create an abstract Controller class and inherit all you AJAX classes from it:

/library/Koodix/Controller/Ajax/Action.php:

<?php

require_once 'Zend/Controller/Action.php';

abstract class Koodix_Controller_Ajax_Action
  extends Zend_Controller_Action
{
    public function init() {
        //disable the standard layout output
        $this->_helper->layout()->disableLayout();
        $this->_helper->viewRenderer->setNoRender();
    }

    public function postDispatch() {
        //envelope and output json field
            if( !empty( $this->json ) ) {
                echo json_encode( $this->json );
            }
        }
}

/application/modules/default/controllers/AjaxController.php:

<?php

class AjaxController extends Koodix_Controller_Ajax_Action
{
 // bla-bla-bla

Take a look at postDispatch method — idea behind it is to convert to JSON and output anything that is set to json field of your controller. If you want to send JSON data in special header (and not in body, like it’s done in my example), you can do it in this method.

You are working on a project and you want to protect the beta version with password so that only allowed people (beta testers) could access it.

You decide not to invent the wheel and to use the standard HTTP authentication.

First idea is to use your Apache web server to do this, so you write something like that in .htaccess file:

AuthName "Private zone"
AuthType Basic
AuthUserFile /path/to/.htpasswd
require valid-use

This solution is simple and that’s why good.

A problem comes on stage when a Flash file uploader is added to your project – usually it cannot “login” to your site, i.e. users are not able to use the Flash file uploader behind beta login.

That’s how I solved it.

It’s not the web server who must solve this (Apache), it’s the application server (PHP). So remove the lines above from .htaccess and use Zend_Auth_Adapter_Http for this purpose — it’s Zend’s HTTP Authentication Adapter.

What concerns the Flash uploader: it sends ‘Shockwave Flash’ as value of ‘User-Agent’ request header. So in your Initializer or Bootstrap file (where you load Zend_Auth_Adapter_Http) check this header value, and if it’s not Flash’s, go for HTTP authentication.

P.S. Hackers can assume this and fake the header to access your site. To cope with that, use an additional secret request variable (Flash uploaders allow this) and check it at server side.

Reverse Geocoding is process opposite to Geocoding (when you get map coordinates by city/country given). So the idea is to get city name by coordinates on the map.

Why you may need it? For example, user is supposed to add a marker on the map, and you want to check that the marker is within a particular country or region.

Solutions:

1. Google GeoCoder (JavaScript) can work with coordinates to retrieve location name. Advantage is that return information is translated to the preferred language of your browser. Here is an example you can play with.
2. GeoNames.org offers several services, among which you can find Reverse Geocoding web service (REST or JSON) which can work with coordinates and postal codes. Advantage — you can download city/region/country names and POI with coordinates.

By the way, here are results of benchmarking Google vs. GeoNames.

Data feeds often come in XML format, so your application must be able to deal with that format.

As I already wrote, data in CSV format (comma separated values) can be loaded to database extremely fast. So my idea was to convert XML data files to CSV and then use bulk load to database. My tests shown that this is faster in 10-100 times than one by one inserts.

Yesterday I decided to write a generalized solution for this, and it turned out that there is no need: it’s just coming — MySQL 6 will have such feature!

How it works: you create a table, name its columns exactly as XML nodes/attributes names or — and MySQL server will load it correspondently.

Example — you downloaded a POI list file (Points of Interest) called poi.xml that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<gpx>
    <wpt lat="58.691931900" lon="11.253125962">
        <name>Parking</name>
    </wpt>
    <wpt lat="58.315525000" lon="12.305828000">
          <name>Fast Food Restaurant:Max i trollhattan</name>
    </wpt>
    <wpt lat="57.717958100" lon="11.880860600">
          <name>Picnic spot</name>
    </wpt>
</gpx>

You create a MySQL table:

CREATE TABLE IF NOT EXISTS `poi` (
  `lat` varchar(255) NOT NULL,
  `lon` varchar(255) NOT NULL,
  `name` varchar(255) NOT NULL
) DEFAULT CHARSET=utf8;

OK, now you load the XML data to your table:

LOAD XML INFILE '\\path\\to\\poi.txt'
INTO TABLE `poi`
ROWS IDENTIFIED BY '<wpt>'

Voila!

The good thing is that MySQL 6 is already available in alpha version — good enough for development purposes; I gave it a try — it takes 5 seconds to load 4.8 Mb of data in 19 files.