XML to CSV conversion

db, development, ideas Add comments

MySQLData feeds often come in XML format, so your application must be able to deal with that format.

As I already wrote, data in CSV format (comma separated values) can be loaded to database extremely fast. So my idea was to convert XML data files to CSV and then use bulk load to database. My tests shown that this is faster in 10-100 times than one by one inserts.

Yesterday I decided to write a generalized solution for this, and it turned out that there is no need: it’s just coming — MySQL 6 will have such feature!

How it works: you create a table, name its columns exactly as XML nodes/attributes names or — and MySQL server will load it correspondently.

Example — you downloaded a POI list file (Points of Interest) called poi.xml that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<gpx>
    <wpt lat="58.691931900" lon="11.253125962">
        <name>Parking</name>
    </wpt>
    <wpt lat="58.315525000" lon="12.305828000">
          <name>Fast Food Restaurant:Max i trollhattan</name>
    </wpt>
    <wpt lat="57.717958100" lon="11.880860600">
          <name>Picnic spot</name>
    </wpt>
</gpx>

You create a MySQL table:

CREATE TABLE IF NOT EXISTS `poi` (
  `lat` varchar(255) NOT NULL,
  `lon` varchar(255) NOT NULL,
  `name` varchar(255) NOT NULL
) DEFAULT CHARSET=utf8;

OK, now you load the XML data to your table:

LOAD XML INFILE '\\path\\to\\poi.txt'
INTO TABLE `poi`
ROWS IDENTIFIED BY '<wpt>'

Voila!

The good thing is that MySQL 6 is already available in alpha version — good enough for development purposes; I gave it a try — it takes 5 seconds to load 4.8 Mb of data in 19 files.

Comments are closed.

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in