To avoid bugs of wrong presentation of text on your page, make sure that all database entities and the application server (PHP) use the same encoding. That helps to forget about issues connected with text presentation.

On database side, make sure you set the correct encoding to:

• database,
• tables,
• columns,
• import-export tools parameters (a big source of wrong encoding bugs),
• corresponding SQL server variables

On application server it’s usually just one query –

SET NAMES utf8

Pay attention, that utf8 might be not the best choise for your project: every non-English character needs 2-6 bytes of memory, so if you built a one-language (local) project with lots of database data, consider using a 1 byte encoding like windows-1251 and save about half of the space on server file system.

To start with, it’s a watch dog of your database integrity — for example, it would be impossible to add user images records, if the user does not exist. No additional code, more control.

Another benefit is that all depending data is deleted cascadedly when the parent row gets killed. You remove a user — and voila: all images, comments, ratings and staff fly away too.

And third, it helps to autogenerate admin tools; for instance, in Symfony framework, the generator of admin interface relies on this info.

To be able to use foreign keys in your MySQL database, do 2 following steps:

• make sure the two tables you are connecting, are driven by InnoDB
• ensure that the columns you connect (user_id in our example) are of exactly the same type, including NULL and sign allowance.

The last task I was solving was inclusion of all cities in a touristic regions bounds automatically.

Yes, there are MySQL spatial data types, so region border and cities coordinates can be stored in native data types, but the processing functions are not implemented right: they only operate on MBRs (minimum bounding rectangles) to make things simpler…

So, let’s create our own MySQL function for that.

There are different approaches to distinguish if a point is within a given polygon. I prefered the way to iterate the edges of the polygon and check where the point relatively to the edge is – at right or at left; if the point is always at the same side, it lies within the border.

So we came to a fact that we need 2 functions:

1. Distinguish the side at which the point lies from the line
2. Check that for every edge in the polygon.

Here is the first function:

DELIMITER //

DROP FUNCTION `GetPointPositionOfLine`//
CREATE FUNCTION `GetPointPositionOfLine`( a POINT, b POINT, p POINT) RETURNS tinyint NO SQL
  COMMENT 'Distinguish the side at which the point lies from the line'
BEGIN
  DECLARE s DOUBLE;
  SET s = (X(b) - X(a))*(Y(p) - Y(a)) - (Y(b) - Y(a))*(X(p) - X(a));
  IF s > 0 THEN
    RETURN 1;
  END IF;
  IF s < 0 THEN
    RETURN -1;
  END IF;
  RETURN 0;
END //

All three params are of type POINT. Points a and b are the beginning and end of the line. Point p is the point we check.

Here is the main function:

DELIMITER //

DROP FUNCTION `IsPointWithin`//
CREATE FUNCTION `IsPointWithin`( ls LINESTRING, p POINT) RETURNS tinyint(1) NO SQL
BEGIN
  DECLARE i INT;
  DECLARE a, b POINT;
  DECLARE LineStringNumPoints INT;
  DECLARE Position, PrevPosition TINYINT;

  SET i = 1;
  SET PrevPosition = 0;
  SET LineStringNumPoints = NumPoints(ls);

  WHILE i <= LineStringNumPoints DO
    SET a = PointN( ls, i);
    IF i = LineStringNumPoints THEN
      SET b = StartPoint( ls);
    ELSE
      SET b = PointN( ls, i+1);
    END IF;

    SET Position = GetPointPositionOfLine( a, b, p);
    IF Position <> 0 THEN
      IF Position <> PrevPosition AND PrevPosition <> 0 THEN
        RETURN FALSE;
      END IF;
      SET PrevPosition = Position;
    END IF;
    SET i = i + 1;
  END WHILE;
  RETURN TRUE;
END //

It checks if a point p lies within a polygon of type LINESTRING.

Usage example. Given you have coordinates of cities in the separate float columns (to keep them readable) lat and lng of city table, and the polygon is given as a string too — but of course you can use a column of type LINESTRING:

SELECT *
FROM `city` c
WHERE IsPointWithin(GeomFromText('LINESTRING(4 8, 15 16, 23 42)'), POINT(c.`lat`, c.`lng`));

The solution works pretty fast, especially if you first limit the cities by the rectangle that the polygon lies in.

Of course, you can try this statement:

SHOW PROCESSLIST

but if you need a more poweful tool, try MySQL Proxy.

Although it’s still an alpha version, it’s a handy tool to monitor, filter and manipulate your queries since there is Lua scripting language interpreter supported which is quite obvious in usage.

The MySQL Proxy (as can be understood from its title) can help you to see communication between a few MySQL servers and a few clients.

My usage pattern is that the Proxy listens to 4040 port on my local machine, and if I change the port setting in connection string of any application (my job project or even PhpMyAdmin), I can see the queries logged in a console window.

You can track lots of characteristics of queries — just take a look at example Lua scripts in the package that you’ve downloaded.

@echo off
SET DATABASE=my_database
SET USER=root
SET PASS=1
SET FORMAT=*.sql

FOR /F "usebackq" %%i IN (`dir /on /b %FORMAT%`) DO echo %%i && mysql --user=%USER% --password=%PASS% --default-character-set=utf8 %DATABASE% < %%i

Make sure, mysql command can be run. If no, either set the full path to mysql command tool, or add the path to PATH environment variable.

Although, I found a case when using a pre-fetch query is not obvious. The pre-fetch query is a query that is run before the data is got from database, so thus you can set character encoding, tune database caching, set variables, etc.

So, if we have 2 sources, one is parent for all other sources since it has database connection params, and second one inherits from the parent.

source www {
    type                    = mysql

    sql_user                = root
    sql_pass                = 

    sql_query_pre           = SET NAMES utf8
}

source www_country : www {
    sql_query = SELECT * FROM country
}

All works fine – when www_country souce is used, the same sql_query_pre will be used because www_country is a child of parent source.

Although, if one day you decide to add a custom pre-query to the child source, it seems you loose the parent’s ones:

...
source www_country : www {
    sql_query_pre = SET @usefulVariable = 1

    sql_query = SELECT * FROM country
}

In this case it seems you have to duplicate the parental pre-query in every child =(
So the right child is gonna be:

...
source www_country : www {
    sql_query_pre = SET NAMES utf8
    sql_query_pre = SET @usefulVariable = 1

    sql_query = SELECT * FROM country
}

To make this, you can use a remote service like I have already described, or implement it yourself.

To do so you need two things:

1. A database of cities with coordinates (having a population value at least for major cities is a big plus)
2. A method to find a closest city

#1 is left for you, let’s talk about #2.

In the previous post I introduced the DISTANCE function (SQL) to count a distance between 2 points by coordinates. Let’s use it to find the biggest city closest to a given point:

DELIMITER //
DROP FUNCTION IF EXISTS REVERSE_GEOCODE;//
CREATE FUNCTION REVERSE_GEOCODE( latitude DOUBLE, longitude DOUBLE, radius DOUBLE )
    RETURNS DOUBLE READS SQL DATA DETERMINISTIC
    COMMENT 'Finds a city by coordinates given. Returns City ID.'
BEGIN
    DECLARE cityID INT(10) DEFAULT 0;
    DECLARE d DOUBLE DEFAULT 0; /* useless var */

    SELECT id, DISTANCE(`lat`, `lng`, latitude, longitude) AS dist
    INTO cityID, d
    FROM `geo_cities`
    WHERE lat BETWEEN FLOOR(latitude - 0.5)
                  AND CEIL(latitude  + 0.5)
      AND lng BETWEEN FLOOR(longitude- 0.5)
                  AND CEIL(longitude + 0.5)
    HAVING dist < radius
    ORDER BY IF(population > 0, population, 1) * (1/IF(dist > 0, dist, 1)) DESC
    LIMIT 1;

    RETURN cityID;
END; //
DELIMITER ;

Well, this function orders cities by (population and distance) function — so that the biggest closest city is returned.

WHERE clause is supposed to shorten the number of cities which are looked through; otherwise we would have to search through whole world’s cities. In my solution the search is limited by a reasonably small rectangle.

In the next post I’ll tell you how to find a biggest city depending on current Google Maps zoom, so that it won’t give you a small city near Paris if you click on France at a world-level map view.

DELIMITER //

DROP FUNCTION IF EXISTS DISTANCE; //

CREATE FUNCTION DISTANCE( lat1 DOUBLE, lon1 DOUBLE, lat2 DOUBLE, lon2 DOUBLE )
    RETURNS DOUBLE NO SQL DETERMINISTIC
    COMMENT 'counts distance (km) between 2 points on Earth surface'
BEGIN
    DECLARE dtor DOUBLE DEFAULT 57.295800;

    RETURN (6371 * acos(sin( lat1/dtor) * sin(lat2/dtor) +
        cos(lat1/dtor) * cos(lat2/dtor) *
        cos(lon2/dtor - lon1/dtor)));
END; //

DELIMITER ;

In next post I will explain how I used this function to implement a reverse geocoding — it’s useful when you need to find a closest city by given coordinates.

If you want to make it by one move, you can use a feature of preferrable sorting — you can define what value and in which field goes first if such value is present.

For instance, you want to show an item with item_id=15 first:

SELECT *
FROM items
ORDER BY item_id IN (15) DESC

Which returns something like (featured item goes first):

+----------+------------+
| item_id  | name       |
+----------+------------+
| 15       | Product 15 |
| 32       | Product 32 |
| 31       | Product 31 |
| 30       | Product 30 |
| 29       | Product 29 |
| ...      | ...        |
+----------+------------+

To convert such quickly (as a test) you can use Lebedev’s convertor.

To convert the whole table do the following (thanks to dull.ru):

1. The most important step: dump such data in a file with mysqldump

   mysqldump -u user -p --default-character-set=latin1 --skip-set-charset --no-create-info --extended-insert --complete-insert dbname table > dbname.sql

   If not the whole table data is bad, create another table (CREATE TABLE t2 LIKE t1) and copy wrong rows to the new table.

2. Replace latin1 by utf8 in the file
   This can be done manually in your editor or by this command:

   sed -r 's/latin1/utf8/g' dbname.sql > dbname_utf.sql

3. Convert your latin1-table to utf8 (and maybe truncate it):

   ALTER TABLE `table` CONVERT TO CHARACTER SET 'utf8';

4. Import the utf8 data back to the table:

   mysql -u user -p --default-character-set=utf8 dbname < dbname_utf.sql

That’s it.