Commit graph

33 commits

Author SHA1 Message Date
cybermaggedon
88a7dfa126
Maint/rename pkg (#88)
* Rename trustgraph-utils -> trustgraph-cli
* Update YAMLs
2024-09-30 22:20:26 +01:00
cybermaggedon
771d9fc2c7
Make util pathnames have tg- prefix (#87) 2024-09-30 21:24:22 +01:00
cybermaggedon
c26ada08c2
Fix VertexAI package. Add Python packaging to Makefile. (#85)
Bump version & generate templates.
2024-09-30 20:50:20 +01:00
cybermaggedon
f00baab1b8
Maint/fix build env (#84)
* Put README placeholders for packages in place
* Bump version
2024-09-30 19:47:09 +01:00
cybermaggedon
9b91d5eee3
Feature/pkgsplit (#83)
* Starting to spawn base package
* More package hacking
* Bedrock and VertexAI
* Parquet split
* Updated templates
* Utils
2024-09-30 19:36:09 +01:00
cybermaggedon
3fb75c617b
Maint/auto pkg versions (#82)
* Remove need to manage setup.py version

* Update YAMLs
2024-09-30 16:38:50 +01:00
cybermaggedon
cdace22ee4
Feature/simpler subpackages (#81)
* Back to simpler directory structure

* Bump version, update templates
2024-09-30 16:16:20 +01:00
cybermaggedon
f081933217
Feature/subpackages (#80)
* Renaming what will become the core package

* Tweaking to get  package build working

* Fix metering merge

* Rename to core directory

* Bump version.  Use namespace searching for packaging trustgraph-core

* Change references to trustgraph-core

* Forming embeddings-hf package

* Reference modules in core package.

* Build both packages to one container, bump version

* Update YAMLs
2024-09-30 14:00:29 +01:00
cybermaggedon
14d79ef9f1
Streamline startup (#79)
* Separate Prom metrics, different processors as different jobs

* Create producers before consumers, may streamline startup.

* Bump version

* Add Pulsar init command, will replace pulsar-admin invocations.

* Integrate tg-init-pulsar with YAMLs

* Update YAMLs
2024-09-30 12:19:22 +01:00
Cyber MacGeddon
5e8a1520ee Bump version & update templates 2024-09-30 00:00:36 +01:00
cybermaggedon
74a14639bd
Feature/track processor state (#78)
* Add a Prom metric to consumers & consumer/producers to track the running
state.

* New script, gets processor state using prometheus

* Bump version, add tg-processor-state to package

* Update templates
2024-09-29 23:50:57 +01:00
cybermaggedon
efc364583b
Fix/graph rag uses wrong prompt (#77)
* Fix queue name invocation, use correct names, not defaults

* Bump version

* Update templates
2024-09-29 20:38:50 +01:00
cybermaggedon
24b80f4789
Fix/template config vol broken (#72)
* Fix config volume invocation

* Regenerate YAMLs
2024-09-29 18:33:39 +01:00
Cyber MacGeddon
5951fb4e56 Bump version to 0.11.1 2024-09-29 18:15:34 +01:00
cybermaggedon
fa30544999
Fix/revert template change (#71)
* Ditched the deploy directory (going away in 0.11) and putting
YAML files in top-dir of Github (for now).

* Update Makefile for the template change
2024-09-29 18:13:34 +01:00
cybermaggedon
0ae6feddb0
Added GCP and Minikube output (#59)
* Added a config to create Minikube k8s, uses hostpath volumes
* Reworked templater to produce docker compose and minikube output
* Fix config templates
2024-09-09 17:16:50 +01:00
cybermaggedon
f661791bbf
K8s (#58)
Added templates which produce K8s resources.  With the provided GCP wrapper, it works on GCP K8s cluster.  This isn't stable enough for other folks to use so will need more piloting before it can be documented and released.
2024-09-07 18:59:38 +01:00
cybermaggedon
375b213a54
Fix replace strings for JSON (#57)
* Fix replace strings for JSON
* Add JSON markdown preamble/postamble in prompts
* Remove JSON chicanery in LLM  handling
* Bump version
2024-09-05 20:49:32 +01:00
cybermaggedon
3445759598
Fix bedrock replace (#56)
* Fix replace ignoring first output

* Bump version
2024-09-05 20:00:38 +01:00
cybermaggedon
ddd8cc90e5
Fix templates (#55)
* Fix template import problem

* Bump version
2024-09-05 18:24:51 +01:00
Cyber MacGeddon
6fb118ba95 Bump version 2024-09-05 16:47:49 +01:00
cybermaggedon
65d7f6d261
Refactor templates (#52)
* Switching from docker compose to abstract form - should be easier to k8s later
* Text loader util
* Recreate templates
2024-09-05 16:40:47 +01:00
cybermaggedon
af5508ff68
Cannot access s error (#50)
* Fix order of statements error
* Bump version
* Update templates
* Add missing script files
* Added missing DE support init
* Fixed bugs preventing definition extraction from working (#49).
2024-09-03 22:10:48 +01:00
cybermaggedon
208c219962
Template rejig (#48)
* document-rag / graph-rag refactor of templates

* Tweaking the docs and categories

* Clarify triple store vs RAG

* Tweak knowledge graph linkage

* Doc embedding for Qdrant

* Fix document RAG on Qdrant

* Fix templates

* Bump version

* Updated templates
2024-09-03 00:09:15 +01:00
cybermaggedon
f7a30006ad
Make templating work more flexibly (#44)
* Restructure directory
* Config loading
* Variable override points in JSONNET templates, separate pulsar-manager template
* Bump version
* Tidy chunking
* Simplified prompt overrides
* Update config loader
* Fix recursive chunker template
2024-08-30 17:47:35 +01:00
Cyber MacGeddon
937dad3381 Version to 0.8.0 2024-08-27 23:40:40 +01:00
cybermaggedon
32b087fbf6
Switch Milvus for Qdrant in YAMLs (#43)
* Qdrant working

* - Fix missing prompt templates
- Bump version
- Add Qdrant to packages

* Switch Milvus for Qdrant in config files
2024-08-27 23:37:24 +01:00
cybermaggedon
e4c4774b5d
Extract rows and apply object embeddings (#42)
* - Restructured the extract directories
- Added an extractor for 'rows' == a row of a table
- Added a row extractor prompt to prompter.
* Add row support to template prompter
* Row extraction working
* Bump version
* Emit extracted info
* Object embeddings store
* Invocation script
* Add script to package, remove cruft output
* Write rows to Cassandra
* Remove output cruft
2024-08-27 21:55:12 +01:00
cybermaggedon
669aed0f8a
Added doc embedding support (#41)
* document embedding writer & query
* Added test query for doc embeddings
* Bump version
* Added doc rag prompt
* Document RAG service
2024-08-26 23:45:23 +01:00
cybermaggedon
0159e938a2
Update LLM text-completion duration metric (#40)
* Added LLM duration metric, better buckets

* Added heatmap to dashboard to replace 95/97/99 chart

* Bump version
2024-08-26 11:46:36 +01:00
cybermaggedon
d0e3fcf019
Catch LLM mismatches (#39)
* Catch more upstream LLM issues

* Bump version
2024-08-26 10:58:02 +01:00
cybermaggedon
acd60e95ec
Catch llm errors (#38)
* Catch 'null' output from prompt for some values, presumably this is
caused by an upstream LLM error.

* Bump version
2024-08-26 10:52:39 +01:00
cybermaggedon
cea8562ecf
Fix timeouts (#37)
* Fix other timeout default settings

* Add storage-only YAML output

* Bump version
2024-08-25 23:57:30 +01:00