We built Conductor as an agile data integration platform to save businesses time and money. So, how fast is SaaS data management when compared to a popular data integration tool?
To see how Conductor performed side-by-side with SQL Server Integration Services (SSIS) I used them both to migrate data between two SQL Server databases. Each database contained 10 tables and each table varied between 5 to 20 columns and 10,000 rows each. There were some differences between the databases, so the tables and columns were named slightly differently to make the test reflect real world scenarios.
Here’s the 30 second summary:
|SQL Server Integration Services||Conductor Data Integration|
|Built and ran the packages in 87 minutes.||Built and ran the integration process in 6 minutes.|
|Custom scripting needed for exception handling.||Exception handling supplied as a default.|
|Logs are scattered across three locations.||All logs are accessible via web browser.|
|Extra work needed to email results to users.||Job success or failure email sent.|
|Crashes if inconsistent data received.||Able to handle inconsistent source data.|
|May take 2-3 weeks to make production-ready.||Production-ready when the data feed is completed.|
The biggest learning I got from this exercise is that I couldn’t accurately say how long it would take to get the SSIS packages to a standard for deploying into a production environment. A week? Two weeks? There are too many possible issues when adding the exception handling, email notifications, and transformation tasks to produce an accurate estimate. Also, these databases were on the same machine, what about doing this in an enterprise environment where I would be dealing with data on different servers or different database platforms?
With Conductor, as soon as I completed the data feed, it provided all of the enterprise-level logging and exception handling needed, so it could be tested and deployed within hours. Also, Conductor is hardware agnostic, so as long as an Agent is installed on a server, I can push and pull data onto the server without dealing with permissions, logins or database drivers.
I’m still writing up the step-by-step process that I followed, so that should be up in a few days for the SSIS developers reading this that may have more questions. In the meantime, feel free to comment with any questions or comments you may have.