In this series of posts we’ll go over an example of an import plugin for magento that refines messy CSV data and automatically imports it, creating configurable products where necessary. Here’s the link to part 1, if you missed it.

Recap

In the last post we:

1. Listed the categories in our Magento website
2. Determined that we’d have faceted search and configurable products
3. Defined every product attribute we need to extract from the data, and which ones were configurable.

Now we need to get cracking on creating the actual import plugin.

Magento import plugin features

Before development we need to summarise the objectives we need to achieve. Most of the time your import plugin will need the following:

1. An admin panel listing all past imports. This panel should contain a grid showing:
1. The starting time and date of the import
2. The ending time and date of the import
3. The number of products disabled, enabled, added and updated
4. The number of products skipped due to errors in the data
2. A form to load a new CSV and manually start an import
3. A cron job to scan a directory for a fully-uploaded CSV file and automatically start an import
4. The ability to extract product attributes from our messy CSV for importing

The first two points are fairly standard for a Magento plugin and we’ll walk over them briefly. Then we’ll talk over the cron job, extracting product attributes and the actual import process in more detail.

Magento import plugin architecture

Now we know what the plugin needs to achieve we need to know the overall import process. It’s incredibly important to plan this out in depth – by writing each stage in a diagram and by documenting exactly what happens in each stage you can be completely sure that you’ll start developing the right thing. As a side note I didn’t do this and I ended up writing this plugin 3 times; that’s a tonne of wasted time and I don’t want you to make that mistake!

That in mind, here’s our architecture:

You can see that we’ve got six different import stages:

1. Importing data from the point-of-sale CSV export
2. Refining the messy data the POS system produces
3. Disabling indexing (doing this shortens the total import time by over 2 orders of magnitude)
4. Disable products that no longer exist in the POS system (we don’t delete because of past orders)
5. Add, update and enable all refined products
6. Enable automatic re-indexing and re-index everything

This, along with the admin panel and cron checks makes up our import plugin.

In the next blog post we’ll go over the creation of the admin panel, form for manual imports and cron scheduling (the easy bits). It won’t be that in depth and will assume you’ve already got a knowledge of the XML that makes up a plugin, and if you don’t you should see here.

Until next time!

In this series of posts we’ll go over an example of a plugin that refines messy CSV data and automatically imports it, creating configurable products where necessary.

Intro

Refining messy data. It’s a tough job, but it needs to be done. Maybe your client’s running an ancient point of sale system which they won’t leave because it will cost them too much money (and ironically you’re being paid to find a temporary workaround). Maybe they just can’t be bothered to trawl through thousands upon thousands of products, manually sorting data into a pretty awesome looking CSV.

Nightmare. It can be near on impossible. Nevertheless, it can be done. I won’t lie: it’s a complete pain in the ass, but you will get there.

The Situation

Lets say you’re running a website which has 3 completely different product types: carpets, sofas, and accessories to go with all this stuff. You’ve got to take one huge spreadsheet with data mixed all over, refine it so it actually makes sense, and get that in Magento as products.

The Data

Here’s a snippet of how the data might look:

Continue reading →