mirror of
https://github.com/trustgraph-ai/trustgraph.git
synced 2026-07-06 03:42:11 +02:00
Tried to add Cohere support
This commit is contained in:
JackColquitt
parent
315b4d81d6
commit
db006a4b7b
24 changed files with 19258 additions and 0 deletions
102
.ipynb_checkpoints/README-checkpoint.md
Normal file
102
.ipynb_checkpoints/README-checkpoint.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
|
||||
# TrustGraph
|
||||
|
||||
## Introduction
|
||||
|
||||
TrustGraph is a true end-to-end (e2e) knowledge pipeline that performs a `naive extraction` on a text corpus
|
||||
to build a RDF style knowledge graph coupled with a `RAG` service compatible with cloud LLMs and open-source
|
||||
SLMs (Small Language Models).
|
||||
|
||||
The pipeline processing components are interconnected with a pub/sub engine to
|
||||
maximize modularity and enable new knowledge processing functions. The core processing components decode documents,
|
||||
chunk text, perform embeddings, apply a local SLM/LLM, call a LLM API, and generate LM predictions.
|
||||
|
||||
The processing showcases the reliability and efficiences of Graph RAG algorithms which can capture
|
||||
contextual language flags that are missed in conventional RAG approaches. Graph querying algorithms enable retrieving
|
||||
not just relevant knowledge but language cues essential to understanding semantic uses unique to a text corpus.
|
||||
|
||||
Processing modules are executed in containers. Processing can be scaled-up
|
||||
by deploying multiple containers.
|
||||
|
||||
### Features
|
||||
|
||||
- PDF decoding
|
||||
- Text chunking
|
||||
- Inference of LMs deployed with [Ollama](https://ollama.com)
|
||||
- Inference of LLMs: Claude, VertexAI and AzureAI serverless endpoints
|
||||
- Application of a [HuggingFace](https://hf.co) embeddings models
|
||||
- [RDF](https://www.w3.org/TR/rdf12-schema/)-aligned Knowledge Graph extraction
|
||||
- Graph edge loading into [Apache Cassandra](https://github.com/apache/cassandra)
|
||||
- Storing embeddings in [Milvus](https://github.com/milvus-io/milvus)
|
||||
- Embedding query service
|
||||
- Graph RAG query service
|
||||
- All procesing integrates with [Apache Pulsar](https://github.com/apache/pulsar/)
|
||||
- Containers, so can be deployed using Docker Compose or Kubernetes
|
||||
- Plug'n'play architecture: switch different LLM modules to suit your needs
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
TrustGraph is designed to be modular to support as many Language Models and environments as possible. A natural
|
||||
fit for a modular architecture is to decompose functions into a set modules connected through a pub/sub backbone.
|
||||
[Apache Pulsar](https://github.com/apache/pulsar/) serves as this pub/sub backbone. Pulsar acts as the data broker
|
||||
managing inputs and outputs between modules.
|
||||
|
||||
**Pulsar Workflows**:
|
||||
- For processing flows, Pulsar accepts the output of a processing module
|
||||
and queues it for input to the next subscribed module.
|
||||
- For services such as LLMs and embeddings, Pulsar provides a client/server
|
||||
model. A Pulsar queue is used as the input to the service. When
|
||||
processed, the output is then delivered to a separate queue where a client
|
||||
subscriber can request that output.
|
||||
|
||||
The entire architecture, the pub/sub backbone and set of modules, is bundled into a single Python package. A container image with the
|
||||
package installed can also run the entire architecture.
|
||||
|
||||
## Core Modules
|
||||
|
||||
- `chunker-recursive` - Accepts text documents and uses LangChain recursive
|
||||
chunking algorithm to produce smaller text chunks.
|
||||
- `embeddings-hf` - A service which analyses text and returns a vector
|
||||
embedding using one of the HuggingFace embeddings models.
|
||||
- `embeddings-vectorize` - Uses an embeddings service to get a vector
|
||||
embedding which is added to the processor payload.
|
||||
- `graph-rag` - A query service which applies a Graph RAG algorithm to
|
||||
provide a response to a text prompt.
|
||||
- `graph-write-cassandra` - Takes knowledge graph edges and writes them to
|
||||
a Cassandra store.
|
||||
- `kg-extract-definitions` - knowledge extractor - examines text and
|
||||
produces graph edges.
|
||||
describing discovered terms and also their defintions. Definitions are
|
||||
derived using the input documents.
|
||||
- `kg-extract-relationships` - knowledge extractor - examines text and
|
||||
produces graph edges describing the relationships between discovered
|
||||
terms.
|
||||
- `loader` - Takes a document and loads into the processing pipeline. Used
|
||||
e.g. to add PDF documents.
|
||||
- `pdf-decoder` - Takes a PDF doc and emits text extracted from the document.
|
||||
Text extraction from PDF is not a perfect science as PDF is a printable
|
||||
format. For instance, the wrapping of text between lines in a PDF document
|
||||
is not semantically encoded, so the decoder will see wrapped lines as
|
||||
space-separated.
|
||||
- `vector-write-milvus` - Takes vector-entity mappings and records them
|
||||
in the vector embeddings store.
|
||||
|
||||
## LM Specific Modules
|
||||
|
||||
- `llm-azure-text` - Sends request to AzureAI serverless endpoint
|
||||
- `llm-claude-text` - Sends request to Anthropic's API
|
||||
- `llm-ollama-text` - Sends request to LM running using Ollama
|
||||
- `llm-vertexai-text` - Sends request to model available through VertexAI API
|
||||
|
||||
## Quickstart Guide
|
||||
|
||||
See [Quickstart on Docker Compose](docs/README.quickstart-docker-compose.md)
|
||||
|
||||
## Development Guide
|
||||
|
||||
See [Development on trustgraph](docs/README.development.md)
|
||||
|
||||
|
||||
|
||||
179
.ipynb_checkpoints/docker-compose-azure-checkpoint.yaml
Normal file
179
.ipynb_checkpoints/docker-compose-azure-checkpoint.yaml
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
|
||||
volumes:
|
||||
cassandra:
|
||||
pulsar-conf:
|
||||
pulsar-data:
|
||||
etcd:
|
||||
minio-data:
|
||||
milvus:
|
||||
|
||||
services:
|
||||
|
||||
cassandra:
|
||||
image: docker.io/cassandra:4.1.5
|
||||
ports:
|
||||
- "9042:9042"
|
||||
volumes:
|
||||
- "cassandra:/var/lib/cassandra"
|
||||
restart: on-failure:100
|
||||
|
||||
pulsar:
|
||||
image: docker.io/apachepulsar/pulsar:3.3.0
|
||||
command: bin/pulsar standalone
|
||||
ports:
|
||||
- "6650:6650"
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- "pulsar-conf:/pulsar/conf"
|
||||
- "pulsar-data:/pulsar/data"
|
||||
restart: on-failure:100
|
||||
|
||||
pulsar-manager:
|
||||
image: docker.io/apachepulsar/pulsar-manager:v0.3.0
|
||||
ports:
|
||||
- "9527:9527"
|
||||
- "7750:7750"
|
||||
environment:
|
||||
SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
|
||||
restart: on-failure:100
|
||||
|
||||
etcd:
|
||||
image: quay.io/coreos/etcd:v3.5.5
|
||||
command:
|
||||
- "etcd"
|
||||
- "-advertise-client-urls=http://127.0.0.1:2379"
|
||||
- "-listen-client-urls"
|
||||
- "http://0.0.0.0:2379"
|
||||
- "--data-dir"
|
||||
- "/etcd"
|
||||
environment:
|
||||
ETCD_AUTO_COMPACTION_MODE: revision
|
||||
ETCD_AUTO_COMPACTION_RETENTION: "1000"
|
||||
ETCD_QUOTA_BACKEND_BYTES: "4294967296"
|
||||
ETCD_SNAPSHOT_COUNT: "50000"
|
||||
ports:
|
||||
- "2379:2379"
|
||||
volumes:
|
||||
- "etcd:/etcd"
|
||||
restart: on-failure:100
|
||||
|
||||
minio:
|
||||
image: docker.io/minio/minio:RELEASE.2024-07-04T14-25-45Z
|
||||
command:
|
||||
- "minio"
|
||||
- "server"
|
||||
- "/minio_data"
|
||||
- "--console-address"
|
||||
- ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minioadmin
|
||||
ports:
|
||||
- "9001:9001"
|
||||
volumes:
|
||||
- "minio-data:/minio_data"
|
||||
restart: on-failure:100
|
||||
|
||||
milvus:
|
||||
image: docker.io/milvusdb/milvus:v2.4.5
|
||||
command:
|
||||
- "milvus"
|
||||
- "run"
|
||||
- "standalone"
|
||||
environment:
|
||||
ETCD_ENDPOINTS: etcd:2379
|
||||
MINIO_ADDRESS: minio:9000
|
||||
ports:
|
||||
- "9091:9091"
|
||||
- "19530:19530"
|
||||
volumes:
|
||||
- "milvus:/var/lib/milvus"
|
||||
restart: on-failure:100
|
||||
|
||||
pdf-decoder:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "pdf-decoder"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
chunker:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "chunker-recursive"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
vectorize:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "embeddings-vectorize"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
embeddings:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "embeddings-hf"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
kg-extract-definitions:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "kg-extract-definitions"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
kg-extract-relationships:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "kg-extract-relationships"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
|
||||
vector-write:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "vector-write-milvus"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
- "-t"
|
||||
- "http://milvus:19530"
|
||||
restart: on-failure:100
|
||||
|
||||
graph-write:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "graph-write-cassandra"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
- "-g"
|
||||
- "cassandra"
|
||||
restart: on-failure:100
|
||||
|
||||
llm:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "llm-azure-text"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
- "-k"
|
||||
- ${AZURE_TOKEN}
|
||||
- "-e"
|
||||
- ${AZURE_ENDPOINT}
|
||||
restart: on-failure:100
|
||||
|
||||
graph-rag:
|
||||
image: docker.io/trustgraph/trustgraph-flow:0.1.16
|
||||
command:
|
||||
- "graph-rag"
|
||||
- "-p"
|
||||
- "pulsar://pulsar:6650"
|
||||
restart: on-failure:100
|
||||
Loading…
Add table
Add a link
Reference in a new issue