February 7, 2019
Here at XI, we are very proud of the quality of our data and the applications that our clients value so much. It’s easy to focus mostly on our software products because that’s what our clients interact with. As such, we don’t often talk about our data and what it takes to deliver it. For this blog, we thought you might enjoy hearing about the high-level data processes we have implemented over 15 years of operations.
The phrase “Data Driven. Solution Focused” cuts to the core of who we are as a company, and it’s the driving force behind our products. We provide software tools that help you to make smarter decisions, and those software tools are backed by the best quality data in Western Canada. We take pride in both the quality of our data and our solutions.
XI consumes vast quantities of raw data from our primary data provider; RS Energy (secondary sources include: Provincial governments, other 3rd party sellers and a couple secret sources). The data sets we acquire are made available to us at various times of the month, so XI’s data loading process must be nimble and as fast as possible. Our data team performs nightly data loads to incorporate new data, revised data, and newly approved code updates.
This is not to say there are never any issues with the data we are consuming. It is for this reason that XI has numerous levels of data verification and testing procedures to ensure that the data we release to our customers is of the highest quality.
To begin, while the data is running through the loading process there are many low-level checks to ensure that the data does not have any glaring deficiencies. These tests would include: ensuring that there are no large gaps in the data (missing production files for example) and flagging any obvious data format errors (date formats),
Our current load time (barring significant issues) is just under 20 hours to get the data into a consumable database so we can start digging deeper in the numbers. But, in our never-ending quest for improvement, we have undertaken some exciting new projects that could potentially drop our loading time to below 4 hours. That means our products would be able to have new and/or updated data much faster! Stay tuned for some exciting news on these improvements.
Once our internal test environments have the new data, everything is handed over to our in-house Quality Assurance Tester extraordinaire who tackles further data verification and tries to identify any smaller issues that our initial screening attempts failed to catch.
Once the go-ahead has been given from QA, we try to release our data to clients as soon as possible. And of course, updates are done with no downtime or service interruptions for our clients!
If you have any questions, feel free to contact XI Support and we will strive to answer any questions you may have.