Last week, the Center for Data Science welcomed Kx Systems—a software company based in New York—into our office, where they gave two consecutive day-long workshops on their q programming language, and their kdb+ database.
Kx Systems’ kdb+ database was designed for time-series analysis, so it’s a tool that is most useful for either collecting real-time data, or working with huge amounts of data where a time variable is a significant factor. This set of tools has been widely adopted by the finance industry, and is starting to become prevalent in the fields of health, communications, and other fields that require time-series analytics.
This was the second year that Kx Systems led a series of workshops in our office space, and both workshops were open to CDS graduate students and other members of the data science community.
We spoke with Simon Garland, a Senior Engineer at Kx Systems.
Scalability is frequently talked about on your website. Why is scalability such a challenge, and why is it important?
Scalability is important for working with huge amounts of data, and generating complex analytics. You do not want your system to run out of steam just when your results start to get interesting.
Large systems are often an uncomfortable mix of different technologies, all of which need to be maintained, upgraded, and kept in sync with each other. And the amount of data being collected today has brought a lot systems to their limits.
Kdb+ was engineered with extreme data flows, data stores, and parallel hardware in mind, so the program can easily be expanded to accommodate large amounts of data.
Can you talk about the q programming language, and why it’s better suited for this type of data, as opposed to a language like python?
Today, a lot of developers are choosing to learn multiple programming languages because of legacy computing resource requirements. However, we think Kx Systems’ technology covers most computing needs, while recognizing that seamless interfaces to more popular programming languages are beneficial to developers.
Most of Kx Systems’ clients are in the finance industry. Why is it that finance has adopted this software?
The finance industry is much less risk averse than other domains, and historically, has been open to new technologies. For example, IEX—the latest trading venue gaining approval on the stock exchange—uses our technology for real-time analytics on a system that sequences orders from broker subscribers, and accumulates quotes and trades from other regulated venues.
Data gathered from the “Internet of Things” (Iot) seems to be a big part of Kx Systems’ offering. Can you talk about how kdb+ interacts with Iot?
Kx Systems’ technology solves a different problem for IoT analytics than it does for financial services. In finance, there is a constant stream of market data; in IoT applications, there are large bursts of data at lightning fast speeds. Kdb+ is able to keep up with this data flow and analyze it.
Outside of finance, what are some of the other industries that have adopted kdb+?
Pharmaceutical companies are using kdb+ for epidemiological studies; sales support dashboards and are exploring IoT applications based on sensor data from manufacturing equipment.
Other IoT use cases are in the oil and gas industry and telecoms. Power utilities are using our technology for combining historical analysis with the real-time monitoring of smart meter sensor data.