Refactor rate limit handling (#280)

* - Refactored retry for rate limits into the base class
- ConsumerProducer is derived from Consumer to simplify code
- Added rate_limit_count metrics for rate limit events

* Add rate limit events to VertexAI and Google AI Studio

* Added Grafana rate limit dashboard

* Add rate limit handling to all LLMs
This commit is contained in:
cybermaggedon 2025-01-27 17:04:49 +00:00 committed by GitHub
parent 26a586034c
commit 0e03bc05a4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
14 changed files with 174 additions and 298 deletions

View file

@ -267,6 +267,7 @@ class Processor(ConsumerProducer):
except Exception as e:
print(type(e))
print(f"Exception: {e}")
print("Send error response...", flush=True)