Cyber MacGeddon
43756d872b
Set dependencies up for the 0.13 branch. Set version=0.0.0 in Makefile
...
to spot build errors.
2024-10-15 00:31:08 +01:00
Cyber MacGeddon
5850b6c136
Merge branch 'release/v0.11' into release/v0.12
2024-10-09 19:38:13 +01:00
cybermaggedon
a711bc1dde
Fix trustgraph broken linkage ( #109 )
2024-10-08 20:33:14 +01:00
cybermaggedon
148092a6af
Fix/lock 0.11 version ( #108 )
...
* - Locked 0.11 packages to 0.11 deps
- Added 'trustgraph' uber-package which installs the rest
- Added dependency to set package versions before building packages
* Bump version
2024-10-04 22:12:39 +01:00
cybermaggedon
dda29bb663
Workflows ( #105 )
...
* Some basic structure for workflows
* Add PyPI publication for 0.12
* Bump version
* Test bundle generation
* Install jsonnet
* Use release action to automate release creation
2024-10-04 17:28:07 +01:00
cybermaggedon
222dc9982c
Feature/azure openai templates ( #104 )
...
* Azure OpenAI LLM templates
* Bump version, fix package versions
* Add azure-openai to template generation
2024-10-04 15:47:46 +01:00
Cyber MacGeddon
adba99f270
Bump version
2024-10-02 22:25:48 +01:00
cybermaggedon
db9ed06b1c
Fix/broken kg extract topics ( #97 )
...
* Add missing kg-extract-topics service
* Bump version
2024-10-02 22:23:00 +01:00
cybermaggedon
14672f7f0e
Fix/processor state specify prom ( #93 )
...
* Provide mean to specify -p prometheus server
* Bump version
2024-10-01 22:14:28 +01:00
Cyber MacGeddon
2e6be5cdce
Bump version
2024-10-01 21:06:07 +01:00
cybermaggedon
56a9ac3ba9
Change LLM latency dashboard to be rate & bump version ( #92 )
2024-10-01 21:04:55 +01:00
cybermaggedon
ef1b8b5a13
Feature/metering dashboard ( #89 )
...
* Bump version
* Added Prom metrics to metering, added dashboard
* Update YAMLs
* Add $ on axis
* Tweak dashboard
2024-10-01 06:46:41 +01:00
cybermaggedon
88a7dfa126
Maint/rename pkg ( #88 )
...
* Rename trustgraph-utils -> trustgraph-cli
* Update YAMLs
2024-09-30 22:20:26 +01:00
cybermaggedon
771d9fc2c7
Make util pathnames have tg- prefix ( #87 )
2024-09-30 21:24:22 +01:00
cybermaggedon
0e4c9c69ee
Add twine upload target ( #86 )
2024-09-30 21:07:18 +01:00
cybermaggedon
c26ada08c2
Fix VertexAI package. Add Python packaging to Makefile. ( #85 )
...
Bump version & generate templates.
2024-09-30 20:50:20 +01:00
cybermaggedon
f00baab1b8
Maint/fix build env ( #84 )
...
* Put README placeholders for packages in place
* Bump version
2024-09-30 19:47:09 +01:00
cybermaggedon
9b91d5eee3
Feature/pkgsplit ( #83 )
...
* Starting to spawn base package
* More package hacking
* Bedrock and VertexAI
* Parquet split
* Updated templates
* Utils
2024-09-30 19:36:09 +01:00
cybermaggedon
3fb75c617b
Maint/auto pkg versions ( #82 )
...
* Remove need to manage setup.py version
* Update YAMLs
2024-09-30 16:38:50 +01:00
cybermaggedon
cdace22ee4
Feature/simpler subpackages ( #81 )
...
* Back to simpler directory structure
* Bump version, update templates
2024-09-30 16:16:20 +01:00
cybermaggedon
f081933217
Feature/subpackages ( #80 )
...
* Renaming what will become the core package
* Tweaking to get package build working
* Fix metering merge
* Rename to core directory
* Bump version. Use namespace searching for packaging trustgraph-core
* Change references to trustgraph-core
* Forming embeddings-hf package
* Reference modules in core package.
* Build both packages to one container, bump version
* Update YAMLs
2024-09-30 14:00:29 +01:00
cybermaggedon
14d79ef9f1
Streamline startup ( #79 )
...
* Separate Prom metrics, different processors as different jobs
* Create producers before consumers, may streamline startup.
* Bump version
* Add Pulsar init command, will replace pulsar-admin invocations.
* Integrate tg-init-pulsar with YAMLs
* Update YAMLs
2024-09-30 12:19:22 +01:00
Cyber MacGeddon
5e8a1520ee
Bump version & update templates
2024-09-30 00:00:36 +01:00
cybermaggedon
74a14639bd
Feature/track processor state ( #78 )
...
* Add a Prom metric to consumers & consumer/producers to track the running
state.
* New script, gets processor state using prometheus
* Bump version, add tg-processor-state to package
* Update templates
2024-09-29 23:50:57 +01:00
cybermaggedon
efc364583b
Fix/graph rag uses wrong prompt ( #77 )
...
* Fix queue name invocation, use correct names, not defaults
* Bump version
* Update templates
2024-09-29 20:38:50 +01:00
Cyber MacGeddon
5951fb4e56
Bump version to 0.11.1
2024-09-29 18:15:34 +01:00
cybermaggedon
fa30544999
Fix/revert template change ( #71 )
...
* Ditched the deploy directory (going away in 0.11) and putting
YAML files in top-dir of Github (for now).
* Update Makefile for the template change
2024-09-29 18:13:34 +01:00
Cyber MacGeddon
e5249c2bac
Bump version
2024-09-29 18:03:32 +01:00
Cyber MacGeddon
1d89e466d5
Docker hub login
2024-09-28 11:55:30 +01:00
Cyber MacGeddon
d6cacd3fb7
Bump version
2024-09-28 11:50:57 +01:00
Cyber MacGeddon
39cf256f5c
Merge branch 'master' into release/v0.10. Includes llamafile and
...
prompt modifications.
2024-09-28 11:26:52 +01:00
Jack Colquitt
9612a11581
Added basic Llamafile integration ( #63 )
...
* Added basic Llamafile integration
* Added llamafile template support
* New templates following llamafile addition
---------
Co-authored-by: Cyber MacGeddon <cybermaggedon@gmail.com>
2024-09-16 16:18:01 +01:00
cybermaggedon
6af86fa09f
Topic templates for extractor ( #62 )
...
* Add topic prompt to templates
* Bump version
* Updated templates
2024-09-15 23:40:37 +01:00
cybermaggedon
0ae6feddb0
Added GCP and Minikube output ( #59 )
...
* Added a config to create Minikube k8s, uses hostpath volumes
* Reworked templater to produce docker compose and minikube output
* Fix config templates
2024-09-09 17:16:50 +01:00
cybermaggedon
f661791bbf
K8s ( #58 )
...
Added templates which produce K8s resources. With the provided GCP wrapper, it works on GCP K8s cluster. This isn't stable enough for other folks to use so will need more piloting before it can be documented and released.
2024-09-07 18:59:38 +01:00
cybermaggedon
375b213a54
Fix replace strings for JSON ( #57 )
...
* Fix replace strings for JSON
* Add JSON markdown preamble/postamble in prompts
* Remove JSON chicanery in LLM handling
* Bump version
2024-09-05 20:49:32 +01:00
cybermaggedon
3445759598
Fix bedrock replace ( #56 )
...
* Fix replace ignoring first output
* Bump version
2024-09-05 20:00:38 +01:00
cybermaggedon
ddd8cc90e5
Fix templates ( #55 )
...
* Fix template import problem
* Bump version
2024-09-05 18:24:51 +01:00
Cyber MacGeddon
6fb118ba95
Bump version
2024-09-05 16:47:49 +01:00
cybermaggedon
65d7f6d261
Refactor templates ( #52 )
...
* Switching from docker compose to abstract form - should be easier to k8s later
* Text loader util
* Recreate templates
2024-09-05 16:40:47 +01:00
cybermaggedon
af5508ff68
Cannot access s error ( #50 )
...
* Fix order of statements error
* Bump version
* Update templates
* Add missing script files
* Added missing DE support init
* Fixed bugs preventing definition extraction from working (#49 ).
2024-09-03 22:10:48 +01:00
cybermaggedon
208c219962
Template rejig ( #48 )
...
* document-rag / graph-rag refactor of templates
* Tweaking the docs and categories
* Clarify triple store vs RAG
* Tweak knowledge graph linkage
* Doc embedding for Qdrant
* Fix document RAG on Qdrant
* Fix templates
* Bump version
* Updated templates
2024-09-03 00:09:15 +01:00
cybermaggedon
f7a30006ad
Make templating work more flexibly ( #44 )
...
* Restructure directory
* Config loading
* Variable override points in JSONNET templates, separate pulsar-manager template
* Bump version
* Tidy chunking
* Simplified prompt overrides
* Update config loader
* Fix recursive chunker template
2024-08-30 17:47:35 +01:00
Cyber MacGeddon
937dad3381
Version to 0.8.0
2024-08-27 23:40:40 +01:00
cybermaggedon
32b087fbf6
Switch Milvus for Qdrant in YAMLs ( #43 )
...
* Qdrant working
* - Fix missing prompt templates
- Bump version
- Add Qdrant to packages
* Switch Milvus for Qdrant in config files
2024-08-27 23:37:24 +01:00
cybermaggedon
e4c4774b5d
Extract rows and apply object embeddings ( #42 )
...
* - Restructured the extract directories
- Added an extractor for 'rows' == a row of a table
- Added a row extractor prompt to prompter.
* Add row support to template prompter
* Row extraction working
* Bump version
* Emit extracted info
* Object embeddings store
* Invocation script
* Add script to package, remove cruft output
* Write rows to Cassandra
* Remove output cruft
2024-08-27 21:55:12 +01:00
cybermaggedon
669aed0f8a
Added doc embedding support ( #41 )
...
* document embedding writer & query
* Added test query for doc embeddings
* Bump version
* Added doc rag prompt
* Document RAG service
2024-08-26 23:45:23 +01:00
cybermaggedon
0159e938a2
Update LLM text-completion duration metric ( #40 )
...
* Added LLM duration metric, better buckets
* Added heatmap to dashboard to replace 95/97/99 chart
* Bump version
2024-08-26 11:46:36 +01:00
cybermaggedon
d0e3fcf019
Catch LLM mismatches ( #39 )
...
* Catch more upstream LLM issues
* Bump version
2024-08-26 10:58:02 +01:00
cybermaggedon
acd60e95ec
Catch llm errors ( #38 )
...
* Catch 'null' output from prompt for some values, presumably this is
caused by an upstream LLM error.
* Bump version
2024-08-26 10:52:39 +01:00
