{"id":13460786,"url":"https://github.com/miroblog/tf_deep_rl_trader","last_synced_at":"2026-01-17T10:23:39.614Z","repository":{"id":37623069,"uuid":"146087001","full_name":"miroblog/tf_deep_rl_trader","owner":"miroblog","description":"Trading Environment(OpenAI Gym) + PPO(TensorForce)","archived":false,"fork":false,"pushed_at":"2022-12-08T02:47:31.000Z","size":520,"stargazers_count":233,"open_issues_count":36,"forks_count":65,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-08-01T10:21:47.245Z","etag":null,"topics":["ppo","proximal-policy-optimization","stock-market","tensorflow","tensorforce","trading"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miroblog.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-25T10:38:56.000Z","updated_at":"2024-07-22T04:22:10.000Z","dependencies_parsed_at":"2023-01-24T21:15:10.061Z","dependency_job_id":null,"html_url":"https://github.com/miroblog/tf_deep_rl_trader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroblog%2Ftf_deep_rl_trader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroblog%2Ftf_deep_rl_trader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroblog%2Ftf_deep_rl_trader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miroblog%2Ftf_deep_rl_trader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miroblog","download_url":"https://codeload.github.com/miroblog/tf_deep_rl_trader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222004288,"owners_count":16914876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ppo","proximal-policy-optimization","stock-market","tensorflow","tensorforce","trading"],"created_at":"2024-07-31T10:00:48.847Z","updated_at":"2026-01-17T10:23:39.574Z","avatar_url":"https://github.com/miroblog.png","language":"Python","funding_links":[],"categories":["Strategies \u0026 Research"],"sub_categories":["Time Series Data"],"readme":"# Deep RL Trader + PPO Agent Implemented using Tensorforce\n\nThis repo contains \n1. Trading environment(OpenAI Gym) + Wrapper for Tensorforce Env \n2. PPO(Proximal Policy Optimization) Agent (https://arxiv.org/abs/1707.06347)\nAgent is implemented using `tensorforce`(https://github.com/reinforceio/tensorforce)     \n  \nAgent is expected to learn useful action sequences to maximize profit in a given environment.  \nEnvironment limits agent to either buy, sell, hold stock(coin) at each step.  \nIf an agent decides to take a   \n* LONG position it will initiate sequence of action such as `buy- hold- hold- sell`    \n* for a SHORT position vice versa (e.g.) `sell - hold -hold -buy`.    \n\nOnly a single position can be opened per trade. \n* Thus invalid action sequence like `buy - buy` will be considered `buy- hold`.   \n* Default transaction fee is : 0.0005  \n\nReward is given\n* when the position is closed or\n* an episode is finished.   \n  \nThis type of sparse reward granting scheme takes longer to train but is most successful at learning long term dependencies.  \n\nAgent decides optimal action by observing its environment.  \n* Trading environment will emit features derived from ohlcv-candles(the window size can be configured). \n* Thus, input given to the agent is of the shape `(window_size, n_features)`.  \n\nWith some modification it can easily be applied to stocks, futures or foregin exchange as well.\n\n[Visualization](https://github.com/miroblog/tf_deep_rl_trader/blob/master/visualize_info.ipynb) / [Main](https://github.com/miroblog/tf_deep_rl_trader/blob/master/ppo_trader.py) / [Environment](https://github.com/miroblog/tf_deep_rl_trader/blob/master/env/TFTraderEnv.py)\n\nSample data provided is 5min ohlcv candle fetched from bitmex.\n* train : `'./data/train/` 70000\n* test : `'./data/train/` 16000\n\n### Prerequisites\n\nkeras-rl, numpy, tensorflow ... etc\n\n```python\npip install -r requirements.txt\n\n```\n\n## Getting Started \n\n### Create Environment \u0026 Agent\n```python\n# create environment\n# OPTIONS\n# create environment for train and test\nPATH_TRAIN = \"./data/train/\"\nPATH_TEST = \"./data/test/\"\nTIMESTEP = 30  # window size\nenvironment = create_btc_env(window_size=TIMESTEP, path=PATH_TRAIN, train=True)\ntest_environment = create_btc_env(window_size=TIMESTEP, path=PATH_TEST, train=False)\n\n# create spec for network and baseline\nnetwork_spec = create_network_spec() # json format\nbaseline_spec = create_baseline_spec()\n\n# create agent\nagent = PPOAgent(\n    discount=0.9999,\n    states=environment.states,\n    actions=environment.actions,\n    network=network_spec,\n    # Agent\n    states_preprocessing=None,\n    actions_exploration=None,\n    reward_preprocessing=None,\n    # MemoryModel\n    update_mode=dict(\n        unit='timesteps',  # 'episodes',\n        # 10 episodes per update\n        batch_size=32,\n        # # Every 10 episodes\n        frequency=10\n    ),\n    memory=dict(\n        type='latest',\n        include_next_states=False,\n        capacity=50000\n    ),\n    # DistributionModel\n    distributions=None,\n    entropy_regularization=0.0,  # None\n    # PGModel\n\n    baseline_mode='states',\n    baseline=dict(type='custom', network=baseline_spec),\n    baseline_optimizer=dict(\n        type='multi_step',\n        optimizer=dict(\n            type='adam',\n            learning_rate=(1e-4)  # 3e-4\n        ),\n        num_steps=5\n    ),\n    gae_lambda=0,  # 0\n    # PGLRModel\n    likelihood_ratio_clipping=0.2,\n    # PPOAgent\n    step_optimizer=dict(\n        type='adam',\n        learning_rate=(1e-4)  # 1e-4\n    ),\n    subsampling_fraction=0.2,  # 0.1\n    optimization_steps=10,\n    execution=dict(\n        type='single',\n        session_config=None,\n        distributed_spec=None\n    )\n)\n\n```\n\n### Train and Validate\n```python\n    train_runner = Runner(agent=agent, environment=environment)\n    test_runner = Runner(\n        agent=agent,\n        environment=test_environment,\n    )\n\n    train_runner.run(episodes=100, max_episode_timesteps=16000, episode_finished=episode_finished)\n    print(\"Learning finished. Total episodes: {ep}. Average reward of last 100 episodes: {ar}.\".format(\n        ep=train_runner.episode,\n        ar=np.mean(train_runner.episode_rewards[-100:]))\n    )\n\n    test_runner.run(num_episodes=1, deterministic=True, testing=True, episode_finished=print_simple_log)\n```\n\n### Configuring Agent\n```python\n## you can stack layers using blocks provided by tensorforce or define ur own...\ndef create_network_spec():\n    network_spec = [\n        {\n            \"type\": \"flatten\"\n        },\n        dict(type='dense', size=32, activation='relu'),\n        dict(type='dense', size=32, activation='relu'),\n        dict(type='internal_lstm', size=32),\n    ]\n    return network_spec\n\ndef create_baseline_spec():\n    baseline_spec = [\n        {\n            \"type\": \"lstm\",\n            \"size\": 32,\n        },\n        dict(type='dense', size=32, activation='relu'),\n        dict(type='dense', size=32, activation='relu'),\n    ]\n    return baseline_spec\n```\n\n### Running \n[Verbose] While training or testing, \n* environment will print out (current_tick , # Long, # Short, Portfolio)\n  \n[Portfolio]  \n* initial portfolio starts with 100*10000(krw-won)     \n* reflects change in portfolio value if the agent had invested 100% of its balance every time it opened a position.       \n  \n[Reward] \n* simply pct earning per trade.    \n\n### Inital Result\n\n#### Portfolio Value Change, Max DrawDown period in Red\n![trade](https://github.com/miroblog/tf_deep_rl_trader/blob/master/portfolio_change.png)  \n\n* portfolio value 1000000 -\u003e 1586872.1775 in 56 days\n\nNot bad but the agent definitely needs more\n* training data and \n* degree of freedom  (larger network)\n  \nBeaware of overfitting ! \n\n## Authors\n\n* **Lee Hankyol** - *Initial work* - [tf_deep_rl_trader](https://github.com/miroblog/tf_deep_rl_trader)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiroblog%2Ftf_deep_rl_trader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiroblog%2Ftf_deep_rl_trader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiroblog%2Ftf_deep_rl_trader/lists"}