Will use this thread for updates and progress from now on.
Also, CDA = Cryptographic Digital Asset (I prefer this term over token or coin)
Firstly, a little introduction to System Programming.
Every System can be broken down into a set of tasks and these tasks can either be functions inside a program, or completely separate programs. On a Linux system, all programs run as their own virtual machine. You may gain several advantages of running tasks in separate processes, which takes advantage of machines which can run things in parallel, such as having multiple CPU cores and each core able to run multiple hardware threads. Alternatively, you could make programs more complex and have the multi-threading built into the application process itself, but then you need to program your code to be thread safe. There is pros and cons for each approach.
For the trading bot in question, I broke it down to the following processes and tasks.
Optimus Prime (Truck) is the main data collector
This process simply connects to the Exchange and collects data for every CDA ticker (Trading Pair). It then sorts the data into individual data collections by ticker and writes it into a MongoDB database. Each CDA ticker is it’s own data collection within MongoDB. This way no indexes are needed as MongoDB doesn’t handle indexing very well. If you need data to have indexing for searches, you are better off using SQL (like Postgres or MySQL). The second task of this process is to expire old data as I need datasets to have the minimum required data in order to perform trading while maintaining maximum speed.
Data is stored very simply, as the collection is unique for each ticker, the data is simply the Unix Time, the Price and the Volume for that specific CDA Trading Pair.
This is functional up until the year 2038:
Hound (Jeep) is the main trade entry finder from the collected data
This is a process which reads data from MongoDB and processes the information to find potential entry points for a trade or potential cross-trades. Basically it runs a bunch of rules such as:
That’s the basic crux of AI programming, its a bunch of rules around asking questions to the datasets.
Cross-trades is where you buy a CDA of one type to trade for a different type and then another type and eventually back to the original type. Finding these would be near impossible for a human as the number of “What If” questions would take a human hours to work out, by which time the opportunity has passed.
Jazz (Porsche) spawns and runs a trade process to exit - at high precision
When Hound finds an entry point, it starts a Jazz process to run the trade and gives it the information on what ticker to trade and where to start. Hound will also flag the ticker as a trade in progress so as not to spawn more Jazz processes on the same ticker at the same time. Hound can spawn Jazz processes for each ticker and thus there can be many Jazz processes running at the same time, but one entry ticker each.
Jazz makes a websocket connection to the exchange for real time data and queries the bids and asks, then places an appropriate bid based on rules of how the price and volume is trending and what orders are currently in place. It’s job is to place the limit order and make sure it completes within a desired time and desired price (and of course checks I have enough credit to make a trade). If the buy order completes, it uses the price data and volume being received in real time to determine the exit.
If everything goes well and when the velocity of buys flattens which signals the sentiment on the bull run is dying out, it’s job is to set an appropriate sell limit order and ensure it completes. This may require it to adjust the price depending on what is happening. (which means it cancels limit orders and sets new ones).
If everything doesn’t go well, it also enacts a stop loss and market sells to ensure losses are minimised.
If Jazz is running a cross-trade, it will simply trade from one to the other using bids until it completes and checking the price to ensure it is all going to plan. (nothing ever goes to plan).
Ratchet (Ambulance) monitors the processes and restarts them when they fail
The job of this process is to monitor all the other processes to make sure they run.
If a process like Optimus Prime stops, it will check data integrity, do a cleanup and then start Optimus Prime again. Fairly much the same with Hound and Bumblebee.
However, if a Jazz process stops, it enacts an emergency trade takeover. It knows which ticker each Jazz process is tasked with and if the process dies for any reason, it checks order statuses on the ticker and does the following:
- If a buy order was placed, but not completed, it cancels the order.
- If a buy order was placed and completed, but no corresponding sell order was done, it checks the current price of the ticker and will attempt to perform a normal exit (either profit or loss).
- If a sell order was placed, but not completed, it will cancel the order and attempt to perform a normal exit (either profit or loss).
- It will then do a clean up, generate a report for the failed Jazz process for me to read later and then reset the triggers to allow Hound to say the CDA Trade Pair ticker is good to go again.
Bumblebee (Chevy) does the data analysis and cubes
This process collects the data from MongoDB and loads it into Postgres.
Once in Postgres, the process will perform fairly standard data warehousing routines and generate reports.
Its job is to provide me with an overall market view.
The system is not yet complete and there is still much work to be done.
I did a good code sprint today for 8 hours, so taking the rest of the day off \o/
Stuff that goes wrong
Stuff goes wrong all over the place and does require a lot of maintenance. Computers are not reliable, always have backups and have more than one backup.
Network timeouts can generally be worked around with retries.
Data corruption is horribad, there is sanity checks done in each program.
If things don’t look good, the whole system will stop by completing orders where they are and exits.