Mike Fishy and Automation System Programming (Bots)


#1

Hi All,

Will use this thread for updates and progress from now on.
Also, CDA = Cryptographic Digital Asset (I prefer this term over token or coin)

Firstly, a little introduction to System Programming.

Every System can be broken down into a set of tasks and these tasks can either be functions inside a program, or completely separate programs. On a Linux system, all programs run as their own virtual machine. You may gain several advantages of running tasks in separate processes, which takes advantage of machines which can run things in parallel, such as having multiple CPU cores and each core able to run multiple hardware threads. Alternatively, you could make programs more complex and have the multi-threading built into the application process itself, but then you need to program your code to be thread safe. There is pros and cons for each approach.

For the trading bot in question, I broke it down to the following processes and tasks.

Optimus Prime (Truck) is the main data collector

This process simply connects to the Exchange and collects data for every CDA ticker (Trading Pair). It then sorts the data into individual data collections by ticker and writes it into a MongoDB database. Each CDA ticker is it’s own data collection within MongoDB. This way no indexes are needed as MongoDB doesn’t handle indexing very well. If you need data to have indexing for searches, you are better off using SQL (like Postgres or MySQL). The second task of this process is to expire old data as I need datasets to have the minimum required data in order to perform trading while maintaining maximum speed.

Data is stored very simply, as the collection is unique for each ticker, the data is simply the Unix Time, the Price and the Volume for that specific CDA Trading Pair.

This is functional up until the year 2038:

Hound (Jeep) is the main trade entry finder from the collected data

This is a process which reads data from MongoDB and processes the information to find potential entry points for a trade or potential cross-trades. Basically it runs a bunch of rules such as:

When was…
How is…
What if…

That’s the basic crux of AI programming, its a bunch of rules around asking questions to the datasets.

Cross-trades is where you buy a CDA of one type to trade for a different type and then another type and eventually back to the original type. Finding these would be near impossible for a human as the number of “What If” questions would take a human hours to work out, by which time the opportunity has passed.

Jazz (Porsche) spawns and runs a trade process to exit - at high precision

When Hound finds an entry point, it starts a Jazz process to run the trade and gives it the information on what ticker to trade and where to start. Hound will also flag the ticker as a trade in progress so as not to spawn more Jazz processes on the same ticker at the same time. Hound can spawn Jazz processes for each ticker and thus there can be many Jazz processes running at the same time, but one entry ticker each.

Jazz makes a websocket connection to the exchange for real time data and queries the bids and asks, then places an appropriate bid based on rules of how the price and volume is trending and what orders are currently in place. It’s job is to place the limit order and make sure it completes within a desired time and desired price (and of course checks I have enough credit to make a trade). If the buy order completes, it uses the price data and volume being received in real time to determine the exit.

If everything goes well and when the velocity of buys flattens which signals the sentiment on the bull run is dying out, it’s job is to set an appropriate sell limit order and ensure it completes. This may require it to adjust the price depending on what is happening. (which means it cancels limit orders and sets new ones).

If everything doesn’t go well, it also enacts a stop loss and market sells to ensure losses are minimised.

If Jazz is running a cross-trade, it will simply trade from one to the other using bids until it completes and checking the price to ensure it is all going to plan. (nothing ever goes to plan).

Ratchet (Ambulance) monitors the processes and restarts them when they fail

The job of this process is to monitor all the other processes to make sure they run.

If a process like Optimus Prime stops, it will check data integrity, do a cleanup and then start Optimus Prime again. Fairly much the same with Hound and Bumblebee.

However, if a Jazz process stops, it enacts an emergency trade takeover. It knows which ticker each Jazz process is tasked with and if the process dies for any reason, it checks order statuses on the ticker and does the following:

  • If a buy order was placed, but not completed, it cancels the order.
  • If a buy order was placed and completed, but no corresponding sell order was done, it checks the current price of the ticker and will attempt to perform a normal exit (either profit or loss).
  • If a sell order was placed, but not completed, it will cancel the order and attempt to perform a normal exit (either profit or loss).
  • It will then do a clean up, generate a report for the failed Jazz process for me to read later and then reset the triggers to allow Hound to say the CDA Trade Pair ticker is good to go again.

Bumblebee (Chevy) does the data analysis and cubes

This process collects the data from MongoDB and loads it into Postgres.
Once in Postgres, the process will perform fairly standard data warehousing routines and generate reports.
Its job is to provide me with an overall market view.

The system is not yet complete and there is still much work to be done.
I did a good code sprint today for 8 hours, so taking the rest of the day off \o/

Stuff that goes wrong

Oh My!

Stuff goes wrong all over the place and does require a lot of maintenance. Computers are not reliable, always have backups and have more than one backup.

Network timeouts can generally be worked around with retries.
Data corruption is horribad, there is sanity checks done in each program.
If things don’t look good, the whole system will stop by completing orders where they are and exits.

Stay Fishy


Do Crypto Trading Bots Really Work?
1 BTC to 1.5 BTC in 1 month! Crypto Trading
Profit taking formula or percentge
#2

So, here is the setup.

A few weeks back, bought a factory refurbished ASUS laptop for AUD $129 (USD $99.50)

Laptop

Dropped it on the bench and installed Ubuntu Linux, so went all out on expenses here LOL

That machine has been running the programs above.

Got up this morning and I am 4x on ONT
Sold ONT and bought 6 BTC as it was in a dip at $6,611.10
Completed all the other trades and stopped all the processes, entering HODL mode for a while.

Now it’s time for data analytics, then might go out for Yumcha :smiley:

Stay Fishy


#3

You may have noticed in the above picture, there is some strange glassware.

I should also say, I am an Industrial Chemist and I process minerals from Salt that I collect from an inland Salt lake that is a 6 hour drive away from me near Broken Hill.

The process is quite simple, dissolve salt in water, add sodium hydroxide to precipitate out the minerals, separate minerals out and then use a process called “Electro Winning” to electroplate the Platinum Group Elements from the dissolved precipitate. I can make about 2Kg’s of Platinum per ton of Salt.

Platinum Price per Ounce: USD $935.85
There is 28 Ounces per Kilo.

There is more than one way to mine :smiley:

Stay Fishy


#4

Morning all,

Just wanted to say thanks to whomever it was that talked about getting salty and reminded me to buy SALT.
I set a nice low round number buy order at 0.005 ETHer
Sold it this morning for 0.0054 ETHer, nice 7.2% gain after transaction fees.

I do think it will perform quite well on Monday as well.

Finished running all the analytics last night which has shown some interesting data for me, one of which:

I now believe that most traders only work 4 days a week LOL

I have an algorithm which processes a “Sentiment” value based on rate of orders, volume and rate of price increase or decrease.

I’m in Australia, so GMT+10

Sentiment value on Monday is 6.9 (at 8am my time which appears to be trade start - this is 7am in Seoul)
Sentiment value on Tuesday is 7.5
Sentiment value on Wednesday is 6.3
Sentiment value on Thursday is 4.1 (looks to be end of trading period around 3pm my time, 2pm in Seoul)
Sentiment value on Friday is 2.2
Sentiment value on Saturday is 1.3
Sentiment value on Sunday is 0.4

Interesting data, thought I would share.

Best Wishes,

Mike Fishy


#5

Hi all,

I’m going to say, I look at a lot of blockchain projects and how they would monetize their CDA. A lot of them seem to fall short on how they really plan to profit from the project or the profits are so small, that it is hard to understand how they think they are worth so much.

However, occasionally I come across a diamond in the rough.

Today’s one is: https://www.nebula-ai.com/

Artificial Intelligence on the Blockchain.

The “mining” process of the decentralised blockchain is the GPU power allocated to running the AI.

The Whitepaper is here: (updated 8 hours ago)

The NBAI ecosystem consists of two major components, the NBAI foundation
and the NBAI system. The NBAI foundation supports the development, operation,
and management of blockchain development platforms, AI joint labs, and engineer
training centers. NBAI system integrates the top-level applications such as DAI
App, scientific research and application, university education, and the bottom-level
of NBAI blockchain, Artificial Intelligence Mining Machine and Artificial Intelligence
Data Center.

The Advisory team is impressive and includes someone I know.

Yan LIU - Cloud Computing and Distributed Professor
As Concordia University cloud computing and distributed system expert, Dr. Liu
published hundreds of articles, has more than nine years of defense system development
experience. She worked in America at Department of Energy Pacific Northwest
National Laboratory (PNNL) and National ICT Australia (NICTA) as senior engineer.

This might be one to put on your radars as AI systems is going to be huge in the next few years.

Best Wishes,

Mike Fishy


#6

Interesting stuff in that first post. Are Bumblebee’s analytics simply giving you reports to help refine the algorithms Hound uses to look for entry points? Or are you doing machine learning modeling and Hound is implementing a predictive model to get it’s entries?


#7

I’m kinda old and traditional, the role of Bumblebee is the traditional data warehousing and reporting.

Hound is simple AI logic, ask questions to the datasets to find appropriate entry points based on traditional Forex guidelines.

Some people would say algorithms like K-nn is Machine Learning, but it is more of a predictive algorithm that works on a principle of “drift”. You give it 10% of the dataset and calculate what the next number for that dataset should be, then give it what the actual number was and it calculates the drift from it’s logic and used the drift value for its next sequence. Then continue calculating what the next sequence should be, give it the actual and so on… With enough data, the drift becomes smaller in principal, but with datasets that are not really predictable sequences, I use only the “drift” value to see if the entry point is “a good bet”.

IE: with a smaller drift value, the dataset is behaving in a semi-predictable manner.

Believe it or not, Forex bots do a similar thing.

K-nn is one of the simplest algorithms to implement and is very fast. Given the amount of data and very short period of time allowed to react, it made sense to use it.

Enjoy.

Mike Fishy


#8

and i was simply turned on by my partner terminating cat5 cables (i have a thing about cat5) ha


#9

catday


#10

Thanks for the followup. I have used some other classification modeling techniques, but not k-NN. I’m currently in a big data class, and I had hopes of using crypto datasets for my projects, but thus far all the data has been supplied. Maybe when I have some free time, which is a non-existent thing these days.

I’m nowhere close to being able to do the rest of the stuff you’re doing. I was a programmer many moons ago (AS/400 iSeries stuff) but stepped away from the keyboard for awhile, so I have some work ahead to get back up to speed.


#11

Do you have a formal education in this or are you self taught? Have any resources you’d care to share I’ve recently become inclined to create a trading bot however my coding skills are fairly weak all I know is fairly basic web design.


#12

Both. I have been a programmer for 36+ years and over 26 different programming languages.

Everything from Fortran 77 and Cobol, C, C++ and all the way through to more modern languages like Go Lang, Stackless Python and Q#.

If I were to “begin again”, I would start with learning C++ as it is the basis of all other languages and what those languages are written in.

Secondly, you should consider doing a bachelor of mathematics.

If that is too much, perhaps rather than trying to re-invent the wheel, you could participate on projects which are already building the wheel.

(from the nebula-ai post above)

QUANT AI
Analyzing time series and training deep learning models based on AI algorithms to forecast the real-time trend and implement automatic trading of cryptocurrencies.

Best Wishes,

Mike Fishy


#13

Hi all,

HODL mode over, bots are running again. Yayy!

I think I did quite well buying low on Sunday, always seems to have reasonable gains by Tuesday afternoon.

Currently working on separating the cross-trade finder from Hound into a separate process.
When things are busy on a Tuesday, there always seems to be many more cross-trade opportunities of up to 3% every few minutes.

I am also buying another laptop to run a node. Was thinking of running a node for ICX, but seems the C-Node is not actually being made by a “community” and they are keeping the code under wraps. Maybe not a bad thing, when your companies value is the value of your software, you probably don’t want to be giving it away to potential competition, which so many others seem to do. However, on that same note, it’s not exactly fostering community contribution. Tough decision I guess they had to make, I hope they have good programmers, I have seen many in-house projects fail because the programmers just didn’t give a rodents rectum about the code they wrote, since they got paid anyway.

So, when laptop arrives, going to run a Litecoin node:

Mostly want to try this out on Ubuntu 18.04 which gets released in 2 weeks time.

Anyway, went and visited the Treasury today, they somehow think I’m old and wise and I assisted with the Federal Budget wording around blockchain. If any other Australians wish see pre-budget submissions:
https://consult.treasury.gov.au/budget-policy-division/2018-19-pre-budget-submissions/

The Australia Treasury Budget PDF will be available for download around May 4th or 6th.
I have high hopes they stick with Cryptographic Digital Assets as the classification.

As always - Stay Fishy!


#14

Thansks Mike for the info, for the amount of time i spent looking charts, i should make a bot as well. Interesting with the choice of a non sql database. With all the data you are collecting, do you do backtesting to improve your algos? Which programming language did you use?

I thought about python in order to protoype quickly, but maybe erlang in order to make it fault tolerant

Did you test long on data sets before going live? Also how many exchanges are yop running on, and do you find some exchanges better than others?


#15

Hi Lee,

I do a lot of testing on live data all the time. I use PHP to prototype with as it is a very “loose” language, so structure can be more ad-hoc. Once I have ideas coded, I then re-code them to C++, which often also leads to improvements. While I can write similar prototypes in Stackless Python, I just find it takes longer which impacts my ability to focus on the idea at the time.

I do testing all the time, I don’t think testing ever ceases, including inserting bad data into collections to see how well (or badly) the code can cater for it. The “What If” can be quite fun and when I see my programs crash and core dump, it actually makes me laugh, then I go fix them.

I am only using Binance and KuCoin at this time. I would like to add Cobinhood once it is more established.

I am visiting the ATO today to discuss appropriate taxation on Digital Assets. I am lucky, I have many good contacts in various Government Departments, plus I have appropriate security clearances.

On the subject of security, there has been another issue recently and I advise everyone to go change your passwords on popular platforms such as Google, Apple, Facebook and so on.

Personally I change my passwords weekly using an Australian DSD approved password manager that auto-generates secure passwords and stores them in an encrypted vault for which I have 3 backups. I would encourage people to do the same and use an appropriate password vault that does not synchronise with any cloud service.

As it is deemed - Stay Fishy


#16

Morning all,

Visited the Australian Tax Office yesterday and had an excellent conversation with them.

There were many aspects of CDA’s they were completely unaware of.

For example, there are CDA’s which exist that are not on any Exchange and thus have no current ability to convert to a dollar value. The dollar value people say it is worth, is completely speculative. The fact is, it has zero value until it can be converted to a dollar value. They were also completely unaware of how ERC20 tokens worked. They were also completely baffled on how to define a Cryptographic Digital Asset.

I cannot say more until after the Budget is delivered, but there will be things in there that Australian people may not like, which includes new rules around “Australian Trusts”.

On a more positive note, there is several Australian Officials which are attending the Singapore De/Centralize 2018 Conference today and tomorrow.

The Speakers at the conference includes Henry Lee, Head of Global Business for ICON (ICX).

There is officials from all around Asia attending this conference, there is some bigger things afoot.
Stay tuned for news around this event that mainstream media will most probably be publishing, due to some high profile people whom are attending.

Best Wishes,

Mike Fishy


#17

Interesting stuff

Are you using some pre built libraries for the trading logic or have you written all that yourself and pushing orders into the binance api?

In terms of results, how are you testing against time series data? Are you back testing/optimizing on in sample data and testing on out of sample data? Just curious to see how you are avoiding curve fitting since its pretty easy to do with data mining techniques.


#18

Hi all,

Bots put to sleep, bought 10 BTC at 6520.13 to HODL for the weekend.
(Yup, 6 BTC to 10 BTC in 4 days)

To answer the questions.

No I do not use pre-built libraries, I write all my own. The API is a lot easier than you think.
I do testing on live data stored in MongoDB and load the same data into PostgreSQL for analysis.
I don’t do a lot of back testing, that’s not very agile :stuck_out_tongue:

Curve fitting is avoided by several questions asked to the dataset, so the algorithms self adjust as they go along. My code writes code which in itself is a form of AI, but get unruley pretty fast and the bots get really quite aggressive, which is something I am still working on (590+ trades in the last 16 hours before I shut them down about 40 mins ago). One of the biggest issues with self writing code mutation, is it gets aggressive quite quickly due to the datasets providing too much information too quickly. It starts off in a reasonably sane manner and just gets faster as it finds more opportunities in the datasets as time progresses, this is mostly due to how the k-nn algorithm works and my code looking at drift values to identify entries. The drift value becomes lower the more data there is to learn from.

If you notice a lot of small trades done rapidly, yes, they are bots. They are generally after the trades which result in anything from a 0.5% gain to 3%. Knowing this can be used to your advantage.

There is certainly lots of pitfalls and they are far from perfect, but on average, they do quite well.

I built a bot years ago that bet on horse races, it did quite well until it was banned.
So just like betting on horses, you can hedge bets by running multiple trades at once.

Stay Fishy


#19

Afternoon all,

Finished separating the cross-trade finder out of Hound and compiled and tested.

Seems to be functioning as expected so far, but may need more tuning as it can take 3-4 seconds to go through all the pairs.

Thus, Bluestreak (Nissan 280ZX) is now the cross-trade launcher.
Going to work on fitting that turbo and getting it down to 1.5 seconds by caching several pairs in RAM.

Hound is performing better now it doesn’t look for cross-trades and I tweaked the k-nn algorithm by adding a randomised 3% buffer to the feedback drift calculation. This should help it remain a bit more sane for longer and be less aggressive on trades by artificially creating random outliers. It’s really not the best solution, so looking at another algorithm to replace it soon. In the mean time, I added a function to slowly reduce the volume in each trade, so as it gets more aggressive (more trades), it does so with less risk (less CDAs per trade).

I am looking forward to the usual Monday/Tuesday runs, but looking at the data right now, I think the market needs to consolidate a fair bit in order to see a good breakout, hopefully it will come around later tonight. I found the bots work best from Monday 7am to Thursday 3pm my time (GMT +10), so set up a cron job to auto-start and auto-terminate the bots.

Stay Fishy


#20

Hi all, quick update.

Been a good week so far, bots have traded up my 10 BTC to 20 BTC and it’s not Thursday yet.

I should note a feature built into Jazz, which runs the trade.

Once it buys a CDA, it immediate sets up a sell order at 2x
Then it tracks the trade and cancels that order if it needs to so it can exit where it should.

If you are wondering why I do that, it is because it catches the pump and dumps.

Instant 2x, thank you pumper dumpers.

Stay Fishy