Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Live algorithm could not restore work automatically during the last maintenance #13

Open
4 tasks
IlshatGaripov opened this issue Oct 16, 2021 · 0 comments
Open
4 tasks

Live algorithm could not restore work automatically during the last maintenance #13

IlshatGaripov opened this issue Oct 16, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@IlshatGaripov
Copy link

IlshatGaripov commented Oct 16, 2021 •
edited
Loading

Expected Behavior

In live mode the algorithm, if it cannot connect to the broker, works normally, trying to reconnect, does not stop.

Actual Behavior

During the last maintenance BITFINEX
photo_2021-10-12_14-03-21

Got the following logs before the algorithm went down:

2021-10-13T11:44:56.1902910Z ERROR:: WebSocketClientWrapper.OnError(): WebSocketClientWrapper.OnError(): (IsOpen:False, State:Closed): wss://api.bitfinex.com/ws/2: The server returned status code '429' when status code '101' was expected. System.Net.WebSockets.WebSocketException (0x80004005): The server returned status code '429' when status code '101' was expected.
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at QuantConnect.Extensions.SynchronouslyAwaitTask(Task task) in D:\5 repos\Lean\Common\Extensions.cs:line 2760
   at QuantConnect.Brokerages.WebSocketClientWrapper.HandleConnection() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 216
2021-10-13T11:44:58.5303524Z ERROR:: WebSocketClientWrapper.OnError(): WebSocketClientWrapper.OnError(): (IsOpen:False, State:Closed): wss://api.bitfinex.com/ws/2: The server returned status code '503' when status code '101' was expected. System.Net.WebSockets.WebSocketException (0x80004005): The server returned status code '503' when status code '101' was expected.
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at QuantConnect.Extensions.SynchronouslyAwaitTask(Task task) in D:\5 repos\Lean\Common\Extensions.cs:line 2760
   at QuantConnect.Brokerages.WebSocketClientWrapper.HandleConnection() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 216
2021-10-13T11:45:00.0105808Z TRACE:: Brokerage.PerformCashSync(): Sync cash balance
2021-10-13T11:45:00.1446599Z ERROR:: Brokerage.PerformCashSync(): Error in GetCashBalance: System.Exception: BitfinexBrokerage.GetCashBalance: request failed: [405] Not Allowed, Content: <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
, ErrorMessage: 
   at QuantConnect.Brokerages.Bitfinex.BitfinexBrokerage.GetCashBalance() in D:\5 repos\Lean\Brokerages\Bitfinex\BitfinexBrokerage.cs:line 283
   at QuantConnect.Brokerages.Brokerage.PerformCashSync(IAlgorithm algorithm, DateTime currentTimeUtc, Func`1 getTimeSinceLastFill) in D:\5 repos\Lean\Brokerages\Brokerage.cs:line 344
2021-10-13T11:45:00.1447890Z TRACE:: Brokerage.PerformCashSync(): No cash balances available, cash sync not performed
2021-10-13T11:45:00.1449188Z TRACE:: Brokerage.PerformCashSync(): Sync cash balance
2021-10-13T11:45:00.2747895Z TRACE:: Brokerage.PerformCashSync(): No cash balances available, cash sync not performed
2021-10-13T11:45:00.5415248Z TRACE:: WebSocketClientWrapper.HandleConnection(wss://api.bitfinex.com/ws/2): Connecting...
2021-10-13T11:45:00.6654368Z ERROR:: WebSocketClientWrapper.OnError(): WebSocketClientWrapper.OnError(): (IsOpen:False, State:Closed): wss://api.bitfinex.com/ws/2: The server returned status code '503' when status code '101' was expected. System.Net.WebSockets.WebSocketException (0x80004005): The server returned status code '503' when status code '101' was expected.
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at QuantConnect.Extensions.SynchronouslyAwaitTask(Task task) in D:\5 repos\Lean\Common\Extensions.cs:line 2760
   at QuantConnect.Brokerages.WebSocketClientWrapper.HandleConnection() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 216
2021-10-13T11:45:01.0103609Z TRACE:: Brokerage.PerformCashSync(): Sync cash balance
2021-10-13T11:45:01.1454456Z ERROR:: Brokerage.PerformCashSync(): Error in GetCashBalance: System.Exception: BitfinexBrokerage.GetCashBalance: request failed: [405] Not Allowed, Content: <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
, ErrorMessage: 
   at QuantConnect.Brokerages.Bitfinex.BitfinexBrokerage.GetCashBalance() in D:\5 repos\Lean\Brokerages\Bitfinex\BitfinexBrokerage.cs:line 283
   at QuantConnect.Brokerages.Brokerage.PerformCashSync(IAlgorithm algorithm, DateTime currentTimeUtc, Func`1 getTimeSinceLastFill) in D:\5 repos\Lean\Brokerages\Brokerage.cs:line 344
2021-10-13T11:45:01.1455631Z TRACE:: Brokerage.PerformCashSync(): No cash balances available, cash sync not performed
2021-10-13T11:45:01.1456603Z TRACE:: Brokerage.PerformCashSync(): Sync cash balance
2021-10-13T11:45:01.2892237Z TRACE:: Brokerage.PerformCashSync(): No cash balances available, cash sync not performed
2021-10-13T11:45:02.0106157Z TRACE:: Brokerage.PerformCashSync(): Sync cash balance
2021-10-13T11:45:02.1417846Z TRACE:: Brokerage.PerformCashSync(): No cash balances available, cash sync not performed
2021-10-13T11:45:02.3294704Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 10/13/2021 11:45:02 AM UTC. Context: AlgorithmManager.Run System.Exception: The maximum number of attempts for brokerage cash sync has been reached.
   at QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler.ProcessSynchronousEvents() in D:\5 repos\Lean\Engine\TransactionHandlers\BrokerageTransactionHandler.cs:line 605
   at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer, ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler alphas, CancellationToken token) in D:\5 repos\Lean\Engine\AlgorithmManager.cs:line 320
   at QuantConnect.Lean.Engine.Engine.<>c__DisplayClass10_2.<Run>b__9() in D:\5 repos\Lean\Engine\Engine.cs:line 342
2021-10-13T11:45:02.6721601Z TRACE:: WebSocketClientWrapper.HandleConnection(wss://api.bitfinex.com/ws/2): Connecting...
2021-10-13T11:45:02.7073007Z TRACE:: LiveTradingDataFeed.Exit(): Start. Setting cancellation token...
2021-10-13T11:45:02.7078360Z TRACE:: BaseDataExchange(CustomDataExchange) Stopping...
2021-10-13T11:45:02.7079267Z TRACE:: LiveTradingDataFeed.Exit(): Exit Finished.
2021-10-13T11:45:02.7080014Z TRACE:: Engine.Run(): Sending runtime error to user...
2021-10-13T11:45:02.7087599Z TRACE:: Config.Get(): Configuration key not found. Key: transaction-log - Using default value: 
2021-10-13T11:45:02.7094254Z TRACE:: StopSafely(): waiting for 'Transaction Thread' thread to stop...
2021-10-13T11:45:02.7096311Z TRACE:: BrokerageTransactionHandler.Run(): Ending Thread...
2021-10-13T11:45:02.7099859Z TRACE:: StopSafely(): waiting for 'RealTime Thread' thread to stop...
2021-10-13T11:45:02.7133012Z TRACE:: BaseDataExchange(CustomDataExchange).ConsumeQueue(): Cancellation requested.  Exiting...
2021-10-13T11:45:02.7914972Z TRACE:: Log: Runtime Error: The maximum number of attempts for brokerage cash sync has been reached. in BrokerageTransactionHandler.cs:line 605
2021-10-13T11:45:02.7989795Z ERROR:: WebSocketClientWrapper.OnError(): WebSocketClientWrapper.OnError(): (IsOpen:False, State:Closed): wss://api.bitfinex.com/ws/2: The server returned status code '503' when status code '101' was expected. System.Net.WebSockets.WebSocketException (0x80004005): The server returned status code '503' when status code '101' was expected.
   at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
   at QuantConnect.Extensions.SynchronouslyAwaitTask(Task task) in D:\5 repos\Lean\Common\Extensions.cs:line 2760
   at QuantConnect.Brokerages.WebSocketClientWrapper.HandleConnection() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 216
2021-10-13T11:45:03.0106531Z TRACE:: LiveTradingRealTimeHandler.Run(): Exiting thread... Exit triggered: True
2021-10-13T11:45:03.0172729Z TRACE:: DataManager.RemoveSubscription(): Removed BTCUSD,BTCUSD,Minute,TradeBar,Trade,Adjusted
2021-10-13T11:45:03.0176160Z TRACE:: DataManager.RemoveSubscription(): Removed BTCUSD,BTCUSD,Minute,TradeBar,Trade,Adjusted,Internal
2021-10-13T11:45:03.0177889Z TRACE:: DataManager.RemoveSubscription(): Removed ETHUSD,ETHUSD,Minute,TradeBar,Trade,Adjusted
2021-10-13T11:45:03.0184381Z TRACE:: DataManager.RemoveSubscription(): Removed QC-UNIVERSE-USERDEFINED-BITFINEX-CRYPTO,QC-UNIVERSE-USERDEFINED-BITFINEX-CRYPTO,Minute,TradeBar,Trade,Adjusted,Internal
2021-10-13T11:45:03.0188931Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): BTCUSD
2021-10-13T11:45:03.0202620Z TRACE:: DataManager.RemoveSubscription(): Removed BTCUSD,BTCUSD,Minute,QuoteBar,Quote,Adjusted
2021-10-13T11:45:03.0213722Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): ETHUSD
2021-10-13T11:45:03.0214894Z TRACE:: DataManager.RemoveSubscription(): Removed ETHUSD,ETHUSD,Minute,QuoteBar,Quote,Adjusted
2021-10-13T11:45:03.0222865Z TRACE:: DefaultAlphaHandler.Exit(): Exiting...
2021-10-13T11:45:03.0226525Z TRACE:: DefaultAlphaHandler.Exit(): Ended
2021-10-13T11:45:03.0242144Z TRACE:: StopSafely(): waiting for 'Result Thread' thread to stop...
2021-10-13T11:45:04.4166105Z ERROR:: Runtime Error: The maximum number of attempts for brokerage cash sync has been reached. in BrokerageTransactionHandler.cs:line 605
 System.Exception: The maximum number of attempts for brokerage cash sync has been reached. in BrokerageTransactionHandler.cs:line 605
 ---> System.Exception: The maximum number of attempts for brokerage cash sync has been reached.
   at QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler.ProcessSynchronousEvents() in D:\5 repos\Lean\Engine\TransactionHandlers\BrokerageTransactionHandler.cs:line 605
   at QuantConnect.Lean.Engine.AlgorithmManager.Run(AlgorithmNodePacket job, IAlgorithm algorithm, ISynchronizer synchronizer, ITransactionHandler transactions, IResultHandler results, IRealTimeHandler realtime, ILeanManager leanManager, IAlphaHandler alphas, CancellationToken token) in D:\5 repos\Lean\Engine\AlgorithmManager.cs:line 320
   at QuantConnect.Lean.Engine.Engine.<>c__DisplayClass10_2.<Run>b__9() in D:\5 repos\Lean\Engine\Engine.cs:line 342
   --- End of inner exception stack trace ---
2021-10-13T11:45:04.4167391Z TRACE:: LiveTradingResultHandler.Run(): Ending Thread...
2021-10-13T11:45:04.4178082Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Starting...
2021-10-13T11:45:04.6456488Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Finished storing results. Start sending...
2021-10-13T11:45:04.6464718Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Ended
2021-10-13T11:45:04.6471395Z TRACE:: Engine.Run(): Disconnecting from brokerage...
2021-10-13T11:45:04.6485290Z ERROR:: WebSocketClientWrapper.Close(wss://api.bitfinex.com/ws/2): System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Net.WebSockets.ClientWebSocket.CloseOutputAsync(WebSocketCloseStatus closeStatus, String statusDescription, CancellationToken cancellationToken)
   at QuantConnect.Brokerages.WebSocketClientWrapper.Close() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 116
2021-10-13T11:45:04.6490260Z TRACE:: WebSocketClientWrapper.OnClose(): Connection closed (IsOpen:False, State:Closed): wss://api.bitfinex.com/ws/2
2021-10-13T11:45:04.6495356Z TRACE:: Engine.Run(): Disposing of setup handler...
2021-10-13T11:45:04.6501350Z TRACE:: BrokerageSetupHandler.Setup(): Found data queue handler to dispose: QuantConnect.Brokerages.Bitfinex.BitfinexBrokerage
2021-10-13T11:45:04.6507475Z TRACE:: Engine.Main(): Analysis Completed and Results Posted.
2021-10-13T11:45:04.6510772Z TRACE:: StopSafely(): waiting for 'Transaction Thread' thread to stop...
2021-10-13T11:45:04.6517065Z TRACE:: Config.GetValue(): close-automatically - Using default value: False
2021-10-13T11:45:04.7930264Z ERROR:: WebSocketClientWrapper.<Connect>b__8_0(): Error in WebSocketClientWrapper connection task: wss://api.bitfinex.com/ws/2:  System.NullReferenceException: Object reference not set to an instance of an object.
   at QuantConnect.Brokerages.WebSocketClientWrapper.HandleConnection() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 204
   at QuantConnect.Brokerages.WebSocketClientWrapper.<Connect>b__8_0() in D:\5 repos\Lean\Brokerages\WebSocketClientWrapper.cs:line 82
2021-10-13T11:45:04.7931660Z TRACE:: WebSocketClientWrapper connection task ended: wss://api.bitfinex.com/ws/2

Potential Solution

Reproducing the Problem

System Information

Checklist

  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues
  • I have provided detailed steps to reproduce the issue
@Martin-Molinero Martin-Molinero added the bug Something isn't working label Oct 18, 2021
@Martin-Molinero Martin-Molinero transferred this issue from QuantConnect/Lean Feb 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Martin-Molinero @IlshatGaripov