Banks, hydroplants, factories, retailers, manufacturers. So many organisations deal with masses of essential new data daily. Making use of that data often requires moving it to a central location as it’s collected. It sounds simple, but if you’re dealing with anything close to a high volume of data, the process of moving new additions becomes complex.

You essentially have two options. You could just move the entire database, replacing the old one, with the new day included. For this to work, you need to be certain that no one has touched the data since the last day was added, otherwise any changes will be lost. You’ll also need to allow for the time it’s going to take to move your entire database.

A much better option is to migrate just the new data accumulated each day. This allows you to work with data you’ve already moved, without worrying about lost changes, and will save time. This solution still has its drawbacks. You have to commit to some serious coding to make it work, and that means time. You also still have to contend with data errors, potential system crashes, and the cost of a long drawn out process. It’s also extremely difficult to make it work retrospectively, missing a day could potentially leave a gap in your data for good.

Conductor is about making data migration ‘just work.’ We wanted to improve on this outdated method, making it easy to migrate new data between databases as it is produced. As a result Conductor now performs incremental loads. Based off date, datetime or number, Conductor can search out and move all newly added data on schedule. Looking at only the data it needs, the application makes the process, faster, more efficient, and removes the need from any manual coding.

Conductor looks at your destination and determines what data you currently have by finding the most recent value of a nominated column (date, datetime, number). It then goes back to the source and pulls data from that point onwards. This feature is currently available for any relational databases, and will soon be implemented for NoSQL as well.

Of course, the biggest hurdle we’ve faced while developing incremental loading has been allowing for a rollback feature. Because we like making everything as easy as possible, we’re working on a seamless method of restarting the incremental load to maintain data quality, without adding additional work to the user. In the next iteration of this feature, if anything goes wrong during a migration you’ll be able to tell Conductor to rollback automatically, deleting the data from the most recent point to begin the load again.

If you want to learn more about incremental loading, or any other Conductor feature, get in touch with the Eight Wire team today.


About the Author: