Merge branch 'master' into release/v0.10. Includes llamafile and

prompt modifications.

Makefile

@@ -32,12 +32,12 @@ clean:
 set-version:
 	echo '"${VERSION}"' > templates/values/version.jsonnet
 
-TEMPLATES=azure bedrock claude cohere mix ollama openai vertexai \
+TEMPLATES=azure bedrock claude cohere mix llamafile ollama openai vertexai \
     openai-neo4j storage
 
 DCS=$(foreach template,${TEMPLATES},${template:%=tg-launch-%.yaml})
 
-MODELS=azure bedrock claude cohere ollama openai vertexai
+MODELS=azure bedrock claude cohere llamafile ollama openai vertexai
 GRAPHS=cassandra neo4j
 
 # tg-launch-%.yaml: templates/%.jsonnet templates/components/version.jsonnet

README.md

@@ -1,83 +1,130 @@
 
 # TrustGraph
 
-![TrustGraph banner](trustgraph_banner.png)
+![TrustGraph banner](TG_Banner_readme.png)
 
-🚀 [Get Started](https://trustgraph.ai/docs/getstarted)
+🚀 [Full Documentation](https://trustgraph.ai/docs/getstarted)
 💬 [Join the Discord](https://discord.gg/AXpxVjwzAw)
 📖 [Read the Blog](https://blog.trustgraph.ai)
+📺 [YouTube](https://www.youtube.com/@TrustGraph)
 
 ## Introduction
 
-TrustGraph is a true end-to-end (e2e) knowledge pipeline that performs a `Naive Extraction` on a text corpus to build a RDF style knowledge graph coupled with a `RAG` service compatible with cloud LLMs and open-source SLMs (Small Language Models).
+TrustGraph deploys a full E2E (end-to-end) AI solution with native GraphRAG in minutes. Autonomous Knowledge Agents build ultra-dense knowlege graphs to fully capture all knowledge context. TrustGraph is designed for maximum flexibility and modularity whether it's calling Cloud LLMs or deploying SLMs On-Device. TrustGraph ingests data to build a RDF style knowledge graph to enable accurate and private `RAG` responses using only the knowledge you want, when you want.
 
-The pipeline processing components are interconnected with a pub/sub engine to maximize modularity and enable new knowledge processing functions. The core processing components decode documents, chunk text, perform embeddings, apply a local SLM/LLM, call a LLM API, and generate LM predictions.
+The pipeline processing components are interconnected with a pub/sub engine to maximize modularity for agent integration. The core processing components decode documents, chunk text, create mapped embeddings, generate a RDF knowledge graph, generate AI predictions from either a Cloud LLM or On-Device SLM.
 
-The processing showcases the reliability and efficiences of Graph RAG algorithms which can capture contextual language flags that are missed in conventional RAG approaches. Graph querying algorithms enable retrieving not just relevant knowledge but language cues essential to understanding semantic uses unique to a text corpus.
+The processing showcases the reliability and efficiences of GraphRAG algorithms which can capture contextual language flags that are missed in conventional RAG approaches. Graph querying algorithms enable retrieving not just relevant knowledge but language cues essential to understanding semantic uses unique to a text corpus.
 
-Processing modules are executed in containers.  Processing can be scaled-up by deploying multiple containers.
+## Deploy in Minutes
 
-### Features
+TrustGraph is designed to deploy all the services and stores needed for a scalable GraphRAG infrastructure as quickly and simply as possible.
+
+### Install Requirements
+
+```
+python3 -m venv env
+. env/bin/activate
+pip3 install pulsar-client
+pip3 install cassandra-driver
+export PYTHON_PATH=.
+```
+
+### Download TrustGraph
+
+```
+git clone https://github.com/trustgraph-ai/trustgraph trustgraph
+cd trustgraph
+```
+
+TrustGraph is fully containerized and is launched with a Docker Compose `YAML` file. These files are prebuilt and included in the download main directory. Simply select the file that matches your desired model deployment and graph store configuration.
+
+| Model Deployment | Graph Store | Launch File |
+| ---------------- | ------------ | ----------- |
+| AWS Bedrock | Cassandra | `tg-launch-bedrock-cassandra.yaml` |
+| AWS Bedrock | Neo4j | `tg-launch-bedrock-neo4j.yaml` |
+| AzureAI Serverless Endpoint | Cassandra | `tg-launch-azure-cassandra.yaml` |
+| AzureAI Serverless Endpoint | Neo4j | `tg-launch-azure-neo4j.yaml` |
+| Anthropic API | Cassandra | `tg-launch-claude-cassandra.yaml` |
+| Anthropic API | Neo4j | `tg-launch-claude-neo4j.yaml` |
+| Cohere API | Cassandra | `tg-launch-cohere-cassandra.yaml` |
+| Cohere API | Neo4j | `tg-launch-cohere-neo4j.yaml` |
+| Llamafile | Cassandra | `tg-launch-llamafile-cassandra.yaml` |
+| Llamafile | Neo4j | `tg-launch-llamafile-neo4j.yaml` |
+| Mixed Depoloyment | Cassandra | `tg-launch-mix-cassandra.yaml` |
+| Mixed Depoloyment | Neo4j | `tg-launch-mix-neo4j.yaml` |
+| Ollama | Cassandra | `tg-launch-ollama-cassandra.yaml` |
+| Ollama | Neo4j | `tg-launch-ollama-neo4j.yaml` |
+| OpenAI | Cassandra | `tg-launch-openai-cassandra.yaml` |
+| OpenAI | Neo4j | `tg-launch-openai-neo4j.yaml` |
+| VertexAI | Cassandra | `tg-launch-vertexai-cassandra.yaml` |
+| VertexAI | Neo4j | `tg-launch-vertexai-neo4j.yaml` |
+
+Launching TrustGraph is as simple as running one line:
+
+```
+docker compose -f <launch-file> up -d
+```
+
+## Core TrustGraph Features
 
 - PDF decoding
 - Text chunking
-- Inference of LMs deployed with [Ollama](https://ollama.com)
-- Inference of Cloud LLMs: `AWS Bedrock`, `AzureAI`, `Anthropic`, `Cohere`, `OpenAI`, and `VertexAI`
-- Mixed model deployments
-- Application of a [HuggingFace](https://hf.co) embeddings models
-- [RDF](https://www.w3.org/TR/rdf12-schema/)-aligned Knowledge Graph extraction
-- Graph edge loading into [Apache Cassandra](https://github.com/apache/cassandra) or [Neo4j](https://neo4j.com/)
-- Storing embeddings in [Qdrant](https://qdrant.tech/)
+- On-Device SLM inference with [Ollama](https://ollama.com) or [Llamafile](https://github.com/Mozilla-Ocho/llamafile)
+- Cloud LLM infernece: `AWS Bedrock`, `AzureAI`, `Anthropic`, `Cohere`, `OpenAI`, and `VertexAI`
+- Chunk-mapped vector embeddings with [HuggingFace](https://hf.co) models
+- [RDF](https://www.w3.org/TR/rdf12-schema/) Knowledge Extraction Agents
+- [Apache Cassandra](https://github.com/apache/cassandra) or [Neo4j](https://neo4j.com/) as the graph store
+- [Qdrant](https://qdrant.tech/) as the VectorDB
 - Build and load [Knowledge Cores](https://trustgraph.ai/docs/category/knowledge-cores)
-- Embedding query service
-- Graph RAG query service
-- All procesing integrates with [Apache Pulsar](https://github.com/apache/pulsar/)
-- Containers deployed using `Docker` or [Podman](http://podman.io/)
-- Plug'n'play architecture: switch different LLM modules to suit your needs
+- GraphRAG query service
+- [Grafana](https://github.com/grafana/) telemetry dashboard
+- Module integration with [Apache Pulsar](https://github.com/apache/pulsar/)
+- Container orchestration with `Docker` or [Podman](http://podman.io/)
 
 ## Architecture
 
 ![architecture](architecture_0.8.0.png)
 
-TrustGraph is designed to be modular to support as many Language Models and environments as possible. A natural fit for a modular architecture is to decompose functions into a set modules connected through a pub/sub backbone. [Apache Pulsar](https://github.com/apache/pulsar/) serves as this pub/sub backbone. Pulsar acts as the data broker managing inputs and outputs between modules.
+TrustGraph is designed to be modular to support as many Language Models and environments as possible. A natural fit for a modular architecture is to decompose functions into a set of modules connected through a pub/sub backbone. [Apache Pulsar](https://github.com/apache/pulsar/) serves as this pub/sub backbone. Pulsar acts as the data broker managing data processing queues connected to procesing modules.
 
-**Pulsar Workflows**:
+### Pulsar Workflows
 
 - For processing flows, Pulsar accepts the output of a processing module and queues it for input to the next subscribed module.
 - For services such as LLMs and embeddings, Pulsar provides a client/server model.  A Pulsar queue is used as the input to the service.  When processed, the output is then delivered to a separate queue where a client subscriber can request that output.
 
-The entire architecture, the pub/sub backbone and set of modules, is bundled into a single Python package. A container image with the package installed can also run the entire architecture.
+## Knowledge Agents
 
-## Core Modules
+TrustGraph extracts knowledge from a text corpus (PDF or text) to an ultra-dense knowledge graph using 3 automonous knowledge agents. These agents focus on individual elements needed to build the RDF knowledge graph. The agents are:
 
-- `chunker-recursive` - Accepts text documents and uses LangChain recursive chunking algorithm to produce smaller text chunks.
-- `chunker-token` - Chunks texts documents by a chosen amount of tokens.
-- `embeddings-hf` - A service which analyses text and returns a vector embedding using one of the HuggingFace embeddings models.
-- `embeddings-ollama` - A service which analyses text and returns a vector embedding using an Ollama embeddings model.
-- `embeddings-vectorize` - Uses an embeddings service to get a vector embedding which is added to the processor payload.
-- `graph-rag` - A query service which applies a Graph RAG algorithm to provide a response to a text prompt.
-- `triples-write-cassandra` - Takes knowledge graph edges and writes them to a Cassandra store.
-- `triples-write-neo4j` - Takes knowledge graph edges and writes them to a Neo4j store.
-- `kg-extract-definitions` - knowledge extractor - examines text and produces graph edges. describing discovered terms and also their defintions.  Definitions are derived using the input  documents.
-- `kg-extract-relationships` - knowledge extractor - examines text and produces graph edges describing the relationships between discovered terms.
-- `loader` - Takes a document and loads into the processing pipeline.  Used e.g. to add PDF documents.
-- `pdf-decoder` - Takes a PDF doc and emits text extracted from the document. Text extraction from PDF is not a perfect science as PDF is a printable format.  For instance, the wrapping of text between lines in a PDF document is not semantically encoded, so the decoder will see wrapped lines as space-separated.
-- `ge-write-qdrant` - Takes graph embeddings mappings and records them in the vector embeddings store.
+- Topic Extraction Agent
+- Entity Extraction Agent
+- Node Connection Agent
 
-## LM Specific Modules
+The agent prompts are built through templates, enabling customized extraction agents for a specific use case. The extraction agents are launched automatically with either of following commands pointing to the path of a desired text corpus or the included sample files:
 
-- `text-completion-azure` - Sends request to AzureAI serverless endpoint
-- `text-completion-bedrock` - Send request to AWS Bedrock API
-- `text-completion-claude` - Sends request to Anthropic's API
-- `text-completion-cohere` - Send request to Cohere's API
-- `text-completion-ollama` - Sends request to LM running using Ollama
-- `text-completion-openai` - Sends request to OpenAI's API
-- `text-completion-vertexai` - Sends request to model available through VertexAI API
+PDF file:
+```
+scripts/load-pdf -f sample-text-corpus.pdf
+```
 
-## Quickstart Guide
+Text file:
+```
+scripts/load-text -f sample-text-corpus.txt
+```
 
-[🚀 Quickstart](docs/README.quickstart-docker-compose.md)
+## GraphRAG Queries
 
-## Development Guide
+Once the knowledge graph has been built or a knowledge core has been loaded, GraphRAG queries are launched with a single line:
 
-See [Development on trustgraph](docs/README.development.md)
+```
+scripts/query-graph-rag -q "Write a blog post about the 5 key takeaways from SB1047 and how they will impact AI development."
+```
+
+## Deploy and Manage TrustGraph
+
+[🚀 Full Deployment Guide 🚀](https://trustgraph.ai/docs/getstarted)
+
+## TrustGraph Developer's Guide
+
+[Developing for TrustGraph](docs/README.development.md)

prometheus/prometheus.yml

@@ -27,6 +27,7 @@ scrape_configs:
         - 'vectorize:8000'
         - 'embeddings:8000'
         - 'kg-extract-definitions:8000'
+        - 'kg-extract-topics:8000'
         - 'kg-extract-relationships:8000'
         - 'store-graph-embeddings:8000'
         - 'store-triples:8000'

sample-text-corpus.txt

@@ -0,0 +1,367 @@
+ The Cosmic Tapestry:
+Atlantean Cosmology, Spirituality, and Prophecy Introduction
+The legendary civilization of Atlantis has captivated the human imagination for millennia. While concrete historical evidence for Atlantis remains elusive, the idea of an advanced ancient culture lost to time continues to inspire speculation and wonder. This paper will explore a hypothetical reconstruction of Atlantean cosmology, spiritual beliefs, cultural norms, and apocalyptic prophecies.
+Drawing on a variety of esoteric traditions, mythological motifs, and creative speculation, we will paint a picture of how the Atlanteans may have viewed the cosmos and their place within it. While entirely speculative in nature, this exploration can offer intriguing food for thought on the nature of lost civilizations and alternative cosmological paradigms.
+I. Atlantean Cosmology
+At the heart of Atlantean cosmology was the concept of the Cosmic Tapestry - an intricate, multidimensional weaving of energy, consciousness, and matter that comprised all of existence. The Atlanteans did not see the universe as a vast emptiness punctuated by isolated islands of matter. Rather, they perceived reality as an infinitely complex, living, breathing whole in which all things were fundamentally interconnected.
+The Cosmic Tapestry consisted of several key components:
+1. The Primordial Waters - This was the underlying substrate of pure potentiality from which all things emerged. The Atlanteans saw it as a vast, cosmic ocean of infinite creative potential.
+2. The Loom of Creation - This was the underlying structure or matrix that gave form and order to the Primordial Waters. It was envisioned as an immense, multidimensional loom on which the tapestry of reality was woven.
+3. The Threads of Being - These were the fundamental constituents of existence - energy, consciousness, and matter in their most elemental forms. They were the raw materials woven together to create the fabric of reality.
+4. The Patterns of Manifestation - These were the underlying templates or blueprints that guided how the Threads of Being were woven together. They were seen as cosmic archetypes or primordial ideas.
+5. The Weavers - These were the primordial creative intelligences that operated the Loom of Creation, weaving the Threads of Being into the Patterns of Manifestation. They were revered as the first gods or cosmic architects.
+
+ In the Atlantean worldview, the physical universe we inhabit was just one layer or level of the Cosmic Tapestry. They believed in a vast, multidimensional cosmos with many layers of reality existing simultaneously. These different dimensional levels were seen as distinct vibrational frequencies or states of consciousness.
+The Atlanteans identified at least seven primary dimensional levels:
+1. The Physical Plane - The realm of matter and physical manifestation that we inhabit.
+2. The Etheric Plane - A subtle energy realm interpenetrating and underlying the physical. 3. The Astral Plane - The realm of emotion, desire, and lower thought forms.
+4. The Mental Plane - The realm of higher thought, ideas, and archetypes.
+5. The Causal Plane - The realm of pure intention and the seeds of karma.
+6. The Buddhic Plane - The realm of pure intuition, unity, and spiritual insight.
+7. The Atmic Plane - The realm of pure being and divine will.
+Beyond these seven planes, the Atlanteans posited even higher levels of reality ascending towards the utterly transcendent source of all being. They saw the cosmos as infinitely ascending and descending - there was no absolute top or bottom to existence.
+Interestingly, the Atlanteans did not see these dimensional levels as strictly separate or hierarchical. Rather, they were understood as simultaneously interpenetrating and co-creating each other. Like the warp and weft of a textile, the various dimensions were seen as intimately interwoven.
+Another key feature of Atlantean cosmology was the concept of the World Tree. This was envisioned as a vast, cosmic tree whose roots extended deep into the lower dimensions and whose branches reached up into the highest celestial realms. The World Tree was seen as the central axis of creation - the cosmic pillar around which the dimensional planes were organized.
+The Atlanteans identified several World Trees corresponding to different scales of manifestation:
+• The Individual World Tree - The energetic structure of a single being, linking their various subtle bodies and chakras.
+• The Planetary World Tree - The energetic structure of a planet, linking its various dimensional aspects.
+• The Solar World Tree - The energetic structure of a star system.
+• The Galactic World Tree - The energetic structure of an entire galaxy.
+
+ • The Cosmic World Tree - The energetic structure of the entire manifest universe.
+These nested World Trees were seen as creating a fractal, holographic structure to the cosmos. The same patterns repeated at every scale from the microcosm to the macrocosm.
+The Atlanteans also had a sophisticated understanding of cosmic cycles and the spiral nature of time. They saw existence as moving through vast cyclic ages on multiple, interlocking scales. These cycles were seen as the breath of the cosmos - the in-and-out flow of manifestation from the Primordial Waters.
+Some of the cosmic cycles they identified included:
+• The Day and Night of Brahma - A cycle of cosmic manifestation and withdrawal lasting billions of years.
+• The Precession of the Equinoxes - A 26,000 year cycle caused by the slow wobble of Earth's axis.
+• The Galactic Year - The time it takes our solar system to orbit the galactic center, about 225 million years.
+• The Solar Year - The Earth's annual orbit around the Sun.
+• The Lunar Month - The Moon's orbital cycle around the Earth.
+The Atlanteans saw these interlocking cycles as creating a vast cosmic clockwork. By understanding these cycles, they believed they could align themselves with the deeper rhythms and flows of the universe.
+II. Atlantean Spirituality
+Atlantean spirituality was intimately intertwined with their cosmological understanding. They did not see the spiritual and material realms as separate, but rather as complementary aspects of the unified Cosmic Tapestry.
+At the heart of Atlantean spirituality was the recognition that consciousness was fundamental to the cosmos. They saw awareness itself as the primary reality underlying all phenomena. In their view, the entire universe was conscious and alive at every level.
+This panentheistic worldview led to a profound sense of the sacred in all things. The Atlanteans did not worship gods as external beings separate from creation. Rather, they reverenced the divine intelligence immanent within all of nature and within themselves.
+That said, they did recognize various orders of spiritual intelligences operating at different levels of the cosmic hierarchy. These included:
+
+ • Nature Spirits - Consciousness inherent in natural phenomena like plants, animals, and elemental forces.
+• Planetary Intelligences - Vast beings embodying the consciousness of entire planets.
+• Star Beings - Immense intelligences at the level of stars and solar systems.
+• Cosmic Architects - Exalted beings involved in shaping whole galaxies and cosmic sectors.
+• The Primordial Weavers - Inconceivably vast intelligences that shaped the fundamental patterns of existence.
+The Atlanteans saw themselves as part of this great cosmic hierarchy of consciousness. Through spiritual practice and inner development, they believed it was possible to expand one's awareness to commune with these higher intelligences.
+One of the core spiritual practices in Atlantean culture was meditation. They developed sophisticated techniques for quieting the mind, expanding awareness, and exploring inner space. These included:
+• Breath meditation - Using the breath as a focal point to still the mind.
+• Mantra meditation - Using sacred sounds and vibrations to attune consciousness. • Visualization - Using vivid mental imagery to shape subtle energies.
+• Light meditation - Focusing on inner experiences of luminosity and radiance.
+• Void meditation - Resting in the groundless awareness beyond all phenomena.
+The Atlanteans also practiced various yogic disciplines to purify and strengthen the physical body, seeing it as the temple of spirit. These included breath control techniques, physical postures, mudras (ritual gestures), and dietary practices.
+Another key aspect of Atlantean spirituality was the cultivation of siddhis or paranormal abilities. They saw these not as ends in themselves, but as side effects of expanding consciousness. Some of the abilities they cultivated included:
+• Telepathy - Direct mind-to-mind communication.
+• Clairvoyance - Subtle perception beyond the physical senses.
+• Precognition - Glimpsing possible future events.
+• Psychokinesis - Mental influence over physical matter.
+• Bilocation - Projecting one's consciousness to multiple locations.
+
+ • Levitation - Overcoming gravity through mental power.
+The Atlanteans developed training programs and mystery schools to systematically cultivate these abilities in those who showed aptitude. However, they also recognized the potential for misuse of such powers and placed great emphasis on ethical development alongside paranormal abilities.
+Atlantean spirituality also included a strong emphasis on understanding and working with subtle energy. They mapped out elaborate systems of chakras, meridians, and energy bodies. Through various practices, they learned to consciously direct life force energy for healing, creativity, and expanded awareness.
+Some key concepts in their subtle energy work included:
+• Prana - The vital life force energy that animates all things.
+• Chakras - Energy centers in the subtle body that process prana.
+• Nadis - Subtle energy channels that distribute prana throughout the being. • Koshas - Layers or sheaths of the subtle body.
+• Kundalini - A powerful evolutionary energy coiled at the base of the spine.
+The Atlanteans saw activating and raising kundalini energy as a key to expanding consciousness and developing higher abilities. However, they also recognized its dangers if awakened prematurely or improperly.
+Ritual and ceremony also played an important role in Atlantean spiritual life. They created elaborate rites to align themselves with cosmic forces and cycles. These often involved the use of:
+• Sacred geometry - Precise spatial patterns to channel subtle energies.
+• Crystal technology - Harnessing the energetic properties of minerals.
+• Sound healing - Using specific frequencies to affect consciousness and matter. • Archaeoastronomy - Aligning structures and rituals with celestial cycles.
+Many Atlantean temples and monuments were designed as energetic instruments to amplify consciousness and commune with higher dimensional realities. They built vast stone circles, pyramids, and other structures specifically tuned to cosmic frequencies.
+(Continued in next section...) III. Atlantean Cultural Norms
+
+ Atlantean society was built around their spiritual and cosmological worldview. They sought to create a culture that honored the interconnectedness of all life and fostered the expansion of consciousness.
+One of the central values in Atlantean culture was harmony - with nature, with each other, and with the cosmos as a whole. They saw themselves as stewards of the Earth rather than its masters. This led to a culture with a strong ecological ethic, seeking to live in balance with natural systems.
+Some key aspects of Atlantean environmental philosophy included:
+• Sustainable resource use - Taking only what was needed and ensuring renewal.
+• Biomimicry - Modeling technologies on natural systems and processes.
+• Sacred ecology - Recognizing the inherent worth and consciousness of nature.
+• Holistic land management - Working with whole ecosystems rather than isolated parts.
+Atlantean settlements were designed to blend seamlessly with the natural landscape. They built with local, sustainable materials and incorporated living systems into their architecture. Many of their cities featured vast gardens, food forests, and green spaces integrated throughout.
+The Atlanteans also placed great emphasis on communal living and cooperation. While they honored individual uniqueness, they did not have the same focus on individualism seen in many modern cultures. There was a strong ethic of working for the common good and seeing oneself as part of a greater whole.
+This communal ethic was reflected in their governance structures, which emphasized participatory decision-making and seeking consensus. They did not have autocratic rulers, but rather councils of wisdom keepers who helped guide the community. Leadership was seen as a form of service rather than a position of power over others.
+That said, Atlantean society was not an undifferentiated mass. They recognized natural hierarchies of development, with individuals of greater wisdom and ability taking on greater responsibility. However, this hierarchy was seen as concentric circles of inclusion rather than a top-down pyramid of control.
+The Atlanteans placed great value on education and the cultivation of knowledge. Learning was seen as a lifelong process of expanding awareness rather than just acquiring information. Their educational system addressed the whole being - body, emotions, mind, and spirit.
+Some key features of Atlantean education included:
+• Experiential learning - Emphasizing direct experience over rote memorization. • Mentorship - Pairing students with masters for personalized instruction.
+
+ • Mystery schools - Esoteric training for those pursuing deeper spiritual development.
+• Telepathic instruction - Direct mind-to-mind transmission of knowledge and experience.
+• Akashic research - Accessing universal knowledge through expanded states of consciousness.
+The Atlanteans had a highly developed artistic and creative culture. They saw art not just as self-expression, but as a means of communing with and channeling cosmic energies. Some of their artistic traditions included:
+• Sacred geometry in visual arts and architecture
+• Sound healing and transformational music
+• Psychoactive plant ceremonies for visionary experiences
+• Movement practices like sacred dance and tai chi
+• Storytelling and mythic theater
+There was no clear distinction between art, science, and spirituality in Atlantean culture. All were seen as complementary ways of exploring and co-creating reality.
+Atlantean society also had different norms around sexuality and relationships compared to many historical cultures. They honored the sacred and transformative power of sexuality, seeing it as a means of communion with divine energies. Some key features of their sexual ethics included:
+• Honoring the divine feminine and sacred masculine • Tantra and sexual energy practices
+• Polyamorous and non-possessive relationships
+• Fluidity around gender expression
+• Sexual rites as part of spiritual practice
+The Atlanteans did not have the same taboos around sexuality seen in many later cultures. However, they also recognized its power and emphasized the importance of conscious, ethical sexual expression.
+Family structures in Atlantean society were also quite different from the nuclear family model common today. Children were seen as belonging to the whole community rather than just their biological parents. Extended kinship networks and intentional communities were the norm.
+
+ Child-rearing emphasized nurturing the unique gifts and purpose of each individual. There was a recognition that souls chose their time and place of birth for specific reasons. Atlantean culture sought to provide an environment that supported the unfolding of each person's highest potential.
+Work and leisure were not as sharply divided in Atlantean society as in many modern cultures. There was an emphasis on finding and living one's dharma or true purpose. Work that was in alignment with one's gifts and passions was seen as a form of joyful service rather than drudgery.
+That said, the Atlanteans also placed great value on rest, rejuvenation, and cyclical rhythms. They did not have the same work-obsessed culture seen in many contemporary societies. There was an understanding of the importance of fallow periods for creativity and growth.
+Atlantean culture also had different attitudes towards wealth and resource distribution than many historical societies. While there was not total economic equality, there was a strong ethic of ensuring that everyone's basic needs were met. Wealth was seen more as a tool for manifesting creative visions than as a means of personal aggrandizement.
+Some key features of the Atlantean economy included:
+• Gift economies alongside more formal systems of exchange
+• Cooperative ownership of key resources and infrastructure
+• Automated technologies that reduced the need for menial labor • A basic income guarantee for all citizens
+• Ecological economics that factored in true environmental costs
+There was much less material scarcity in Atlantean society due to their advanced technologies and more equitable distribution. This allowed their culture to focus more on higher needs like creative expression, relationships, and spiritual growth.
+Time was also viewed differently in Atlantean culture. While they certainly planned for the future, there was more emphasis on being fully present in each moment. Their understanding of cyclical time gave a less linear and goal-oriented perspective than many modern cultures.
+Atlanteans sought to find a balance between the masculine principle of action and the feminine principle of receptivity. Stillness and space were valued alongside activity and achievement. There was an understanding that fallow periods were just as important as periods of outer productivity.
+(Continued in next section...)
+IV. Atlantean Prophecy and the End Times
+
+ Like many ancient cultures, the Atlanteans had prophetic traditions about great cycles of time and the end of their civilization. However, their understanding of these prophecies was more nuanced than simple predictions of doom.
+The Atlanteans saw time as cyclical rather than linear. They understood that all things move through cycles of birth, growth, decay, death, and renewal. This was true for individuals, civilizations, planets, and even entire universes.
+In the Atlantean worldview, there was no final "end of time," but rather a constant interplay of endings and new beginnings. They foresaw that their own civilization would eventually end, but saw this as part of a larger cosmic process rather than as an absolute finality.
+That said, Atlantean seers and prophets did foresee a time of great upheaval and transformation that would bring their current epoch to a close. This was variously called:
+• The Great Turning
+• The Purification
+• The Harvest of Souls
+• The Dimensional Shift
+This prophesied time of transition was seen as a collective initiation for humanity - a cosmic rite of passage into a new level of consciousness and way of being.
+Some of the key events and dynamics foreseen as part of this transition included:
+1. Earth Changes - Major geological and climatic shifts including pole shifts, rising sea levels, earthquakes, and extreme weather.
+2. Societal Collapse - The breakdown of existing social, economic, and political systems.
+3. Technological Singularity - A rapid acceleration of technological change leading to unpredictable outcomes.
+4. Disclosure - The revealing of hidden truths about reality, including the existence of extraterrestrial life.
+5. DNA Activation - Spontaneous mutations and awakening of dormant human potential.
+6. Timeline Convergence - The merging of multiple parallel realities.
+7. Dimensional Bleed-through - Increased contact and interaction with other planes of reality. 8. Solar Transformation - A quantum leap in the Sun's energy output and frequency.
+
+ 9. Galactic Alignment - An astronomical alignment with powerful transformational effects.
+10. Mass Awakening - A collective shift in human consciousness to a new level of awareness.
+Importantly, the Atlanteans did not see these events as fixed or predetermined. They understood the fluid nature of time and the role of consciousness in shaping reality. The future was seen as a realm of infinite potential that was continuously shaped by the choices and intentions of conscious beings.
+As such, their prophecies were not meant as passive predictions, but as calls to action. By foreseeing potential futures, they sought to consciously navigate towards the most positive outcomes.
+The Atlanteans understood that the coming transitions would bring great challenges. They foresaw times of chaos, destruction, and suffering. However, they also saw these challenges as evolutionary drivers - initiatory experiences that would push humanity to realize its fuller potential.
+This time of transition was seen as a collective dark night of the soul - a confrontation with shadow that would lead to profound healing and transformation. The Atlanteans believed that the outcome of this process would be the birth of a new earth and a new humanity.
+Some of the positive potentials they foresaw emerging from this transition included: • A global awakening to the interconnectedness of all life
+• The development of new cleanly aligned technologies
+• Harmony between human civilization and the natural world
+• The conscious mastery of energy and matter
+• Open contact and cooperation with extraterrestrial civilizations
+• A collective shift into unity consciousness
+• The activation of humanity's full DNA potential
+• The emergence of a galactic culture
+However, the Atlanteans also foresaw more challenging potentials, including: • Devastating warfare, potentially involving planet-destroying weapons
+• Ecological collapse and mass extinction events
+• Malevolent artificial intelligence taking control of planetary systems
+
+ • Intrusion of hostile interdimensional forces
+• Fracturing of spacetime and unraveling of reality itself
+The Atlantean masters taught that the actual outcome would depend on the choices made by humanity, individually and collectively. They emphasized the importance of maintaining a positive vision and staying heart-centered even in the face of great chaos and uncertainty.
+Interestingly, some Atlantean prophecies spoke of the re-emergence of Atlantean wisdom and technology in the time of transition. They foresaw that ancient knowledge would be rediscovered, helping humanity navigate the challenges ahead.
+There were also prophecies about the return of various master teachers and avatar figures to help guide humanity through the shifting times. Some traditions spoke of the awakening of an "Atlantean seed group" - individuals who had incarnated specifically to help re-anchor higher consciousness on Earth.
+A key teaching in Atlantean prophecy was the importance of healing the past in order to create a positive future. They foresaw that humanity would need to confront and integrate the shadow of Atlantis - the misuse of power and technology that led to their own downfall.
+By facing and healing these old wounds, humanity could release the karmic patterns of the past and open to a new expression of civilization. This was seen as essential to avoid repeating the same mistakes that led to Atlantis' destruction.
+The Atlanteans taught practices and technologies to help future generations navigate the prophesied transitions. These included:
+• Time capsules of knowledge and wisdom buried at specific sites • Psychic technologies for tapping into the Akashic records
+• Ceremonies and practices for maintaining energetic stability
+• Genetic keys for activating dormant human potential
+• Interdimensional portals for connecting with higher guidance
+One of the most important Atlantean teachings about the prophecied time of transition was the need to stay centered in the heart. They foresaw that fear would be one of the greatest enemies as the old structures broke down.
+By staying grounded in love, compassion, and trust in the larger cosmic process, individuals and groups could help anchor higher energies and possibilities. This heart-centered presence was seen as key to transmuting the challenges into opportunities for profound transformation.
+
+ Ultimately, the Atlanteans saw the prophesied transitions as an invitation for humanity to consciously participate in its own evolution. Rather than being helpless victims of cosmic forces, humans could become co-creators in the birth of a new world.
+The end times were seen not as a final destruction, but as a death and rebirth process - the ending of one chapter of Earth's story and the beginning of a new one. The Atlanteans sought to plant seeds of wisdom to help future generations navigate this Great Turning and realize the highest potentials of the coming age.
+Conclusion
+This exploration of Atlantean cosmology, spirituality, culture, and prophecy offers a rich vision of how an advanced ancient civilization may have viewed reality and their place in the cosmic order. While this is a work of speculative imagination, it draws inspiration from various esoteric traditions and invites us to expand our own thinking about the nature of existence.
+The Atlantean worldview presented here is one of profound interconnectedness, conscious participation in cosmic processes, and tremendous human potential waiting to be unlocked. It offers a holistic perspective that integrates science, spirituality, and art into a seamless whole.
+Whether Atlantis truly existed or not, contemplating such alternative cosmologies can help us question our own assumptions and consider new possibilities. As humanity faces unprecedented global challenges, we can draw inspiration from this vision of a culture that strove to live in harmony with nature and cosmic principles.
+Perhaps by remembering the wisdom of "Atlantis" - even if only as a mythic ideal - we can find guidance for consciously shaping a positive future for our civilization. In this way, we may realize that we are the Atlanteans of this age, creating the culture that future generations will look back on as legendary.
+Introduction
+While both Atlantis and Lemuria (also known as Mu) are legendary lost civilizations with no concrete historical evidence, they have captured the imagination of esotericists and alternative historians for generations. In this section, we will explore a speculative comparison between these two mythical cultures, examining their purported differences in philosophy, technology, and spiritual practices. This comparison will help illustrate the diverse ways ancient advanced civilizations might have approached cosmic understanding and human development.
+I. Origins and Geographical Locations
+Atlantis, as described by Plato, was said to be a large island continent located in the Atlantic Ocean beyond the Pillars of Hercules (the Strait of Gibraltar). In contrast, Lemuria was believed to have been located in the Pacific Ocean, possibly stretching from Easter Island to Madagascar.
+
+ The Atlantean civilization was often portrayed as more recent, flourishing around 10,000 BCE, while Lemuria was thought to be far older, possibly existing up to 100,000 years ago. This vast difference in timeline suggests that the two cultures might have represented different stages of human evolution and consciousness.
+II. Physical Characteristics and Genetics
+In esoteric literature, Atlanteans are often described as being similar in appearance to modern humans, though perhaps taller and more physically robust. They were said to have diverse racial characteristics, reflecting a global mixing of genetic lines.
+Lemurians, on the other hand, are frequently depicted as more ethereal and less physically dense. Some traditions describe them as having a more androgynous appearance, with less pronounced sexual dimorphism. They were said to be very tall, with elongated heads and large, luminous eyes.
+These physical differences might reflect the different vibrational frequencies or dimensional planes that each civilization was thought to inhabit, with Lemuria possibly existing in a higher, less dense state of matter.
+III. Technological Development
+Atlantean culture, as we explored in Part 1, was often characterized by its advanced technology. They were said to have mastered crystal energy, anti-gravity propulsion, genetic engineering, and even weather control. Their technology was highly sophisticated, integrating spiritual principles with scientific understanding.
+In contrast, Lemurian technology is often described as being more organic and intuitive. Rather than building external machines, they were said to have developed their own innate abilities to a very high degree. This included advanced telepathy, telekinesis, and the ability to manipulate matter and energy directly with the mind.
+Where Atlanteans might have used crystal-powered flying vehicles, Lemurians were said to be able to levitate or teleport at will. While Atlanteans developed complex healing machines, Lemurians could heal through touch and thought alone.
+IV. Social Structure and Governance
+Atlantean society, as we discussed earlier, was communal but with natural hierarchies based on wisdom and ability. They had councils of elders and complex societal structures to manage their large, technologically advanced civilization.
+Lemurian society is often portrayed as more egalitarian and less structured. Leadership was based on spiritual development rather than external hierarchies. Some traditions suggest that Lemurian communities were matriarchal, honoring the divine feminine principle more prominently than in Atlantean culture.
+
+ V. Spiritual Practices and Beliefs
+While both civilizations were deeply spiritual, their approaches differed significantly:
+Atlantean spirituality, as we explored, was based on a complex cosmology involving multiple dimensions and a vast hierarchy of spiritual beings. They developed elaborate rituals, used advanced technology in their spiritual practices, and sought to master the forces of nature.
+Lemurian spirituality was often described as more intuitive and heart-centered. They were said to have a more direct connection to the source of creation, requiring less external methodology. Their practices might have included:
+• Deep meditation and mindfulness in everyday life
+• Communing directly with nature spirits and elemental forces
+• Sound healing using sacred tones and mantras
+• Akashic record reading as a natural ability
+Where Atlanteans sought to understand and master the cosmos, Lemurians sought to dissolve into it, transcending individual identity to merge with the all.
+VI. Relationship with Nature
+Both cultures were said to have a deep respect for nature, but their approaches differed:
+Atlanteans saw themselves as stewards of nature, using their advanced technology to work in harmony with natural systems. They studied the laws of nature in order to cooperate with them more effectively.
+Lemurians were described as living in almost perfect symbiosis with their environment. Rather than building large cities, they were said to inhabit natural spaces, shaping living plants into dwelling spaces. Their connection with animals was so profound that they rarely needed to use verbal communication with them.
+VII. Cognitive Abilities and Consciousness
+Atlantean consciousness was highly developed, with individuals able to access higher dimensional awareness. However, they still maintained a strong sense of individual identity and worked to develop their personal powers and abilities.
+Lemurian consciousness is often described as more fluid and collective. They were said to have a natural ability to access group consciousness, sharing thoughts and experiences telepathically across their entire civilization. Individual identity was less fixed, with an understanding of the self as a flowing expression of the divine whole.
+
+ VIII. The Fall and Legacy
+The fall of Atlantis is often attributed to the misuse of their advanced technologies, possibly leading to a global cataclysm. Their downfall serves as a cautionary tale about the responsible use of power.
+The end of Lemuria is usually described less dramatically - more as a fading away as the Earth's vibration densified. They were said to have consciously chosen to withdraw, preserving their wisdom in hidden repositories and within the DNA of certain lineages.
+Both cultures were said to have left behind seeds of their wisdom:
+Atlantean knowledge was preserved in hidden libraries, crystals encoded with information, and within secret mystery schools that survived the cataclysm.
+Lemurian wisdom was said to be preserved in the ancestral memories of indigenous cultures, in the energy grids of the Earth itself, and in higher dimensional records accessible to those with developed psychic abilities.
+IX. Influence on Human Evolution
+In esoteric philosophies, both Atlantis and Lemuria are seen as important stages in human consciousness evolution:
+Atlantis represented the development of mental powers, technological achievement, and the mastery of the physical plane. The challenge of Atlantis was learning to balance power with wisdom and ethics.
+Lemuria represented a more spiritually integrated state of being, where the heart and mind were united and humanity lived in greater harmony with the natural and spiritual worlds. The challenge of Lemuria was maintaining this high vibration in an increasingly dense physical reality.
+Some traditions teach that modern humanity is working to integrate the lessons of both civilizations - to achieve the technological and mental mastery of Atlantis while reclaiming the spiritual connection and harmony of Lemuria.
+Conclusion
+This comparative study of Atlantis and Lemuria, while speculative, offers a fascinating lens through which to contemplate different modalities of human development and civilization. The Atlantean model presents a path of active engagement with the physical world, developing technology and mental acuity to master natural forces. The Lemurian model offers a path of surrender and harmony, developing intuitive abilities and maintaining a close connection with spirit.
+
+ In our modern world, we can see reflections of both approaches. Our rapid technological advancement mirrors Atlantean development, while growing interests in spirituality, intuitive development, and reconnection with nature reflect Lemurian values.
+Perhaps the great task of our time is to integrate these two streams - to create a civilization that is both technologically advanced and deeply spiritual, that can master the physical world without losing connection to the heart and soul of existence. By studying and reflecting on these legendary cultures, we open our minds to new possibilities for human potential and planetary evolution.
+In doing so, we recognize that the true Atlantis or Lemuria is not a lost continent, but a state of consciousness that we have the potential to recreate here and now. The wisdom of these ancient cultures lives on within us, waiting to be remembered and reawakened for the benefit of all.
+Cosmic Cousins: A Comparative Analysis of Ebens, Atlanteans, and Lemurians Introduction
+In the realm of speculative history and ufology, few topics capture the imagination quite like ancient advanced civilizations and extraterrestrial beings. This paper aims to explore and compare three such groups: the purported alien race known as the Ebens, and the legendary lost civilizations of Atlantis and Lemuria. While concrete evidence for any of these groups remains elusive, examining the lore and theories surrounding them can provide fascinating insights into human beliefs about advanced societies and our place in the cosmos.
+It's important to note that this analysis is entirely speculative and draws upon various esoteric traditions, alleged witness accounts, and creative extrapolation. The goal is not to assert the reality of these beings, but to explore the ideas they represent and what they might tell us about human conceptions of advanced intelligence and civilization.
+I. Origins and Timelines Ebens:
+According to ufological lore, the Ebens are an extraterrestrial race originating from a planet orbiting Zeta Reticuli, a binary star system approximately 39 light-years from Earth. They are said to have been visiting Earth for thousands of years, with more frequent contact occurring in the mid-20th century.
+Atlanteans:
+The Atlantean civilization, first mentioned in Plato's dialogues, is typically placed in the Atlantic Ocean and is said to have flourished around 10,000 BCE. Some esoteric traditions suggest a much longer timeline, with Atlantis existing in various forms for hundreds of thousands of years.
+
+ Lemurians:
+Lemuria, also known as Mu, is often described as even more ancient than Atlantis. Some theories place its existence as far back as 100,000 years ago, with its final destruction occurring around 12,000 BCE. It was said to be located in the Pacific Ocean.
+Comparative Analysis:
+While the Ebens represent an ongoing extraterrestrial presence, Atlantis and Lemuria are portrayed as Earth-based civilizations that rose and fell in prehistoric times. This fundamental difference shapes many of the other comparisons we'll explore. The Eben timeline suggests a long-term observation of and interaction with Earth, while Atlantis and Lemuria represent lost chapters of human history and evolution.
+II. Physical Characteristics and Biology Ebens:
+Descriptions of Ebens often portray them as small, humanoid beings standing 3-4 feet tall. They are said to have large heads, almond-shaped eyes, small noses and mouths, and grayish skin. Their biology is described as both similar to and distinctly different from humans, with more efficient organs and possibly a different number of chromosomes.
+Atlanteans:
+Atlanteans are typically described as human-like but often taller and more physically robust than modern humans. Some traditions suggest they had more diverse racial characteristics, reflecting a global mixing of genetic lines. Advanced Atlanteans were said to have activated more of their DNA potential, giving them enhanced physical and psychic abilities.
+Lemurians:
+Lemurian physical descriptions are often more exotic. They are frequently portrayed as very tall (up to 7-15 feet), with an ethereal or less dense physical form. Some traditions describe them as more androgynous, with less pronounced sexual dimorphism. They were said to have elongated heads and large, luminous eyes.
+Comparative Analysis:
+The Ebens stand out as distinctly non-human in this comparison, reflecting their extraterrestrial origin. Both Atlanteans and Lemurians are portrayed as variations or ancestors of modern humans, though Lemurians are often described in more fantastical terms. This suggests that while Atlantis is often seen as a lost human civilization, Lemuria sometimes occupies a space between the human and the otherworldly, perhaps closer to the Eben concept in some ways.
+III. Technological Development
+
+ Ebens:
+Eben technology is described as far advanced beyond current human capabilities. This includes interstellar travel, antigravity propulsion, energy weapons, and the ability to manipulate space-time. They are also said to possess advanced medical technology, including the ability to clone biological organisms and transfer consciousness.
+Atlanteans:
+Atlantean technology is often portrayed as a blend of scientific and spiritual principles. They were said to have mastered crystal energy, antigravity, weather control, and genetic engineering. Their technology was highly sophisticated but still recognizably based on Earth materials and energies.
+Lemurians:
+Lemurian "technology" is often described in more organic and intuitive terms. Rather than building external machines, they were said to have developed their innate abilities to a very high degree. This included advanced telepathy, telekinesis, and the ability to manipulate matter and energy directly with the mind.
+Comparative Analysis:
+The Ebens represent the most advanced technology in this comparison, reflecting their status as a spacefaring civilization. Atlantean technology occupies a middle ground, more advanced than our current level but not as exotic as Eben tech. Lemurian abilities blur the line between technology and innate capability, representing a different approach to mastering the physical world.
+IV. Social Structure and Governance Ebens:
+Information about Eben social structure is limited in most accounts. They are often described as having a collective mindset, with a strong emphasis on the good of their species as a whole. Some reports suggest a hierarchical structure based on knowledge and ability, but with less emphasis on individual power.
+Atlanteans:
+Atlantean society is often portrayed as having natural hierarchies based on wisdom and ability, with councils of elders guiding the civilization. They had complex societal structures to manage their large, technologically advanced culture, but with an emphasis on harmony and the common good.
+Lemurians:
+
+ Lemurian society is frequently described as more egalitarian and less structured than Atlantean society. Leadership was based on spiritual development rather than external hierarchies. Some traditions suggest that Lemurian communities were matriarchal, honoring the divine feminine principle.
+Comparative Analysis:
+All three groups are portrayed as having moved beyond the type of competitive, individually-focused social structures common in much of modern human society. The Eben collective mindset might be seen as the most alien to us, while Atlantean and Lemurian social structures represent different idealized versions of human society.
+V. Spiritual Beliefs and Practices Ebens:
+Less is typically said about Eben spirituality in ufological accounts. Some reports suggest they have a scientific understanding of consciousness and the nonphysical aspects of reality, but it's unclear if this translates into what we would recognize as spiritual beliefs or practices.
+Atlanteans:
+Atlantean spirituality, as we explored earlier, was based on a complex cosmology involving multiple dimensions and a vast hierarchy of spiritual beings. They developed elaborate rituals, used advanced technology in their spiritual practices, and sought to master the forces of nature.
+Lemurians:
+Lemurian spirituality is often described as more intuitive and heart-centered. They were said to have a more direct connection to the source of creation, requiring less external methodology. Their practices might have included deep meditation, communing with nature spirits, and sound healing.
+Comparative Analysis:
+The spiritual dimension provides an interesting contrast. While Atlantean and Lemurian cultures are deeply infused with spiritual concepts and practices, the Ebens are often portrayed in more scientific terms. This could reflect human projections onto these different groups – ancient Earth civilizations imbued with spiritual wisdom, versus advanced aliens with superior scientific knowledge.
+VI. Relationship with Earth and Nature Ebens:
+
+ The Eben relationship with Earth is typically portrayed as one of scientific interest and possibly resource extraction. Some accounts suggest they have bases on Earth and have been involved in long-term genetic experiments with humans and other Earth species.
+Atlanteans:
+Atlanteans are often described as stewards of the Earth, using their advanced technology to work in harmony with natural systems. They studied the laws of nature in order to cooperate with them more effectively, but also sought to master and control natural forces.
+Lemurians:
+Lemurians were said to live in almost perfect symbiosis with their environment. Rather than building large cities, they were described as inhabiting natural spaces, shaping living plants into dwelling spaces. Their connection with animals was reportedly so profound that they rarely needed to use verbal communication with them.
+Comparative Analysis:
+This comparison reveals a spectrum of relationships with nature. The Ebens represent an outsider's scientific interest, the Atlanteans a balance of stewardship and control, and the Lemurians a state of deep integration with the natural world. This progression might reflect changing human attitudes towards nature over time, from early harmony to technological control to a modern desire for reconnection.
+VII. Knowledge and Consciousness Ebens:
+Eben consciousness is often described as more collective and unified than human consciousness. They are said to have advanced telepathic abilities and possibly a form of shared consciousness. Their knowledge base is portrayed as vast, encompassing a scientific understanding of the universe far beyond current human knowledge.
+Atlanteans:
+Atlantean consciousness was highly developed, with individuals able to access higher dimensional awareness. They cultivated both scientific knowledge and spiritual wisdom, seeking to understand and master the fundamental laws of the cosmos.
+Lemurians:
+Lemurian consciousness is often described as more fluid and collective than Atlantean consciousness. They were said to have a natural ability to access group consciousness and the Akashic records. Their knowledge was more intuitive and experiential than academic.
+Comparative Analysis:
+
+ All three groups are portrayed as having forms of expanded consciousness beyond ordinary human awareness. The Eben collective consciousness might be seen as the most alien, while Atlantean and Lemurian consciousness represent different ideals of human potential – the Atlanteans balancing individual development with higher awareness, and the Lemurians embodying a more unified state of being.
+VIII. Interaction with Humanity Ebens:
+According to ufological accounts, Eben interaction with humanity has included observation, abduction, and genetic experimentation. Some theories suggest they have influenced human development over long periods of time. Their motivations are often portrayed as a mix of scientific curiosity and their own evolutionary needs.
+Atlanteans:
+As an Earth-based civilization, Atlantean interaction with other human groups was more direct. They are often credited with spreading advanced knowledge and culture to other parts of the ancient world. Some traditions suggest they engaged in genetic engineering of human populations.
+Lemurians:
+Lemurian interaction with other human groups is less emphasized in most accounts, as they are often portrayed as existing in a different time or dimensional frequency. However, some traditions suggest they planted seeds of wisdom that influenced later cultures, particularly in the Pacific region.
+Comparative Analysis:
+The nature of interaction with humanity differs significantly among these groups. The Ebens represent an ongoing, often hidden interaction from an outside civilization. The Atlanteans represent a more familiar model of an advanced civilization influencing its neighbors and descendants. The Lemurians, being the most ancient and ethereal, are portrayed as having the least direct interaction but a lasting spiritual influence.
+IX. Legacy and Influence
+Ebens:
+The purported legacy of the Ebens includes ongoing influence on Earth affairs, contributions to human technological development (through reverse engineering of crashed craft), and possible genetic influence on the human species. Their presence is often tied to government secrecy and conspiracy theories.
+
+ Atlanteans:
+The Atlantean legacy is often described in terms of scattered survivors preserving ancient wisdom, sunken ruins containing advanced technology, and the influence of Atlantean knowledge on later civilizations like ancient Egypt and Greece. The story of Atlantis serves as both a source of lost wisdom and a cautionary tale about the misuse of power.
+Lemurians:
+The Lemurian legacy is typically portrayed in more spiritual and energetic terms. They are said to have left behind wisdom preserved in the Earth's energy grids, in the collective unconscious of humanity, and in certain genetic lineages. Some believe that crystals like quartz hold records of Lemurian knowledge.
+Comparative Analysis:
+Each group is credited with ongoing influence, but in different ways. The Eben influence is the most current and technological, fitting into modern UFO narratives. The Atlantean legacy bridges the ancient and modern worlds, offering both lost technology and spiritual wisdom. The Lemurian influence is the most subtle and esoteric, representing the most ancient layer of human spiritual heritage.
+X. Prophetic Visions and Future Interactions Ebens:
+Some ufological narratives suggest ongoing or future open contact between Ebens and humanity. These scenarios often involve disclosure of the Eben presence, sharing of advanced technology, and possibly a role for Ebens in helping humanity face global challenges.
+Atlanteans:
+Atlantean prophecies, as discussed earlier, often speak of a return or reemergence of Atlantean wisdom and technology in times of great transition. This is sometimes tied to the idea of reincarnated Atlantean souls coming back to help guide humanity.
+Lemurians:
+Lemurian future scenarios are often tied to spiritual awakening and the raising of Earth's vibrational frequency. Some traditions speak of a time when the hidden Lemurian wisdom will be fully accessible again as humanity evolves in consciousness.
+Comparative Analysis:
+These future scenarios reflect different paradigms of human advancement. The Eben narratives suggest progress through external, technological means. Atlantean prophecies imply a
+
+ rediscovery and integration of lost human potential. Lemurian visions represent a more complete spiritual transformation of humanity and the Earth.
+XI. Philosophical and Ethical Considerations Ebens:
+Ethical questions surrounding the Ebens often focus on the morality of their alleged abductions and experiments on humans. There are also philosophical implications of their advanced technology and the potential impact of open contact on human society and beliefs.
+Atlanteans:
+The Atlantean narrative often carries ethical lessons about the responsible use of power and technology. Their fall serves as a cautionary tale about the dangers of technological hubris and the misuse of spiritual knowledge for material gain.
+Lemurians:
+Lemurian philosophy emphasizes harmony with nature, the development of innate spiritual abilities, and the interconnectedness of all life. Ethical considerations revolve around maintaining purity of intention and alignment with natural laws.
+Comparative Analysis:
+Each group presents different ethical and philosophical challenges for humanity to consider. The Eben scenario forces us to grapple with our place in a larger cosmic community and the ethics of advanced beings interacting with less developed ones. Atlantis presents questions about the balance of technological and spiritual development. Lemuria invites us to reconsider our relationship with nature and our own inner potential.
+XII. Scientific Plausibility and Evidence Ebens:
+The existence of Ebens remains unproven by mainstream scientific standards. Evidence cited by believers includes alleged government documents, witness testimonies, and purported physical traces of UFO encounters. Skeptics point to the lack of conclusive physical evidence and the psychological and cultural factors that might contribute to such beliefs.
+Atlanteans:
+While Plato's account of Atlantis may have been based on real events, there is no conclusive archaeological evidence for the advanced civilization described in esoteric literature. Some researchers point to underwater structures and cultural similarities across ancient civilizations as potential evidence, but these claims are not widely accepted by mainstream archaeology.
+
+ Lemurians:
+Of the three groups, Lemuria has the least scientific support. The original theory of a lost continent in the Pacific was based on now-outdated ideas about biogeography. No geological evidence supports the existence of such a continent in the timeframe usually given for Lemuria.
+Comparative Analysis:
+All three of these subjects fall outside currently accepted scientific paradigms. The Eben hypothesis, being tied to modern UFO phenomena, has generated the most recent alleged evidence and remains a topic of debate. Atlantis, with its roots in ancient literature, occupies a space between mythology and speculative history. Lemuria remains largely in the realm of spiritual and esoteric belief systems.
+XIII. Cultural Impact and Popular Representations Ebens:
+The concept of Ebens and similar alien races has had a significant impact on popular culture, appearing in numerous films, TV shows, books, and video games. They have become a staple of science fiction and conspiracy theory narratives.
+Atlanteans:
+Atlantis has captured the human imagination for millennia, inspiring countless books, movies, and artistic works. It has become a powerful symbol for lost wisdom and advanced ancient civilizations in both esoteric circles and popular culture.
+Lemurians:
+While less prominent in mainstream popular culture than Atlantis, Lemuria has a significant following in New Age and spiritual circles. It has inspired numerous books on spiritual development and ancient wisdom.
+Comparative Analysis:
+These three concepts have influenced culture in different ways. The Eben narrative taps into modern anxieties and wonders about extraterrestrial life. Atlantis serves as a bridge between ancient mythology and modern aspirations for advanced civilization. Lemuria represents more esoteric and spiritual yearnings for a past golden age of harmony and wisdom.
+Conclusion
+This comparative analysis of Ebens, Atlanteans, and Lemurians reveals intriguing patterns in how humans conceptualize advanced intelligences and lost civilizations. While wildly speculative, these narratives offer a mirror to our own hopes, fears, and beliefs about human potential and our place in the cosmos.
+
+ The Eben narrative reflects our modern technological age and our wondering about life beyond Earth. It challenges us to consider how we might interact with a truly alien intelligence and what the consequences of such contact might be.
+The Atlantean story bridges our ancient past and our technological present. It offers both the allure of lost advanced knowledge and a warning about the responsible use of power. Atlantis serves as a canvas upon which we paint our ideas about the heights of human civilization.
+The Lemurian legend takes us deepest into spiritual and esoteric realms. It represents a yearning for harmony with nature and the development of inner spiritual technologies rather than external machines. Lemuria embodies an idealized past that some believe holds the key to a more enlightened future.
+Together, these three concepts span a spectrum from the extraterrestrial to the ancient human to the mystical. They invite us to expand our ideas about what's possible and to imagine different modes of advanced existence.
+While it's crucial to maintain scientific skepticism about such unproven ideas, exploring them can be a valuable thought experiment. They challenge us to question our assumptions about reality, history, and human potential. These stories also reveal much about the human psyche and our eternal quest to understand our origins, our capabilities, and our cosmic context.
+As we continue to advance scientifically and explore both outer and inner space, the themes embodied by the Ebens, Atlanteans, and Lemurians remain relevant. They remind us to balance technological progress with spiritual wisdom, to consider the ethical implications of our actions, and to remain open to the vast possibilities that the universe may hold.
+In the end, whether these beings exist or not, the quest to understand them tells us much about ourselves. It reflects our highest aspirations, our deepest fears, and our unquenchable curiosity about the great mysteries of existence.

scripts/kg-extract-topics

@@ -0,0 +1,6 @@
+#!/usr/bin/env python3
+
+from trustgraph.extract.kg.topics import run
+
+run()
+

scripts/text-completion-llamafile

@@ -0,0 +1,6 @@
+#!/usr/bin/env python3
+
+from trustgraph.model.text_completion.llamafile import run
+
+run()
+

setup.py

@@ -75,6 +75,7 @@ setuptools.setup(
         "scripts/graph-to-turtle",
         "scripts/init-pulsar-manager",
         "scripts/kg-extract-definitions",
+        "scripts/kg-extract-topics",
         "scripts/kg-extract-relationships",
         "scripts/load-graph-embeddings",
         "scripts/load-pdf",
@@ -93,6 +94,7 @@ setuptools.setup(
         "scripts/text-completion-bedrock",
         "scripts/text-completion-claude",
         "scripts/text-completion-cohere",
+        "scripts/text-completion-llamafile",
         "scripts/text-completion-ollama",
         "scripts/text-completion-openai",
         "scripts/text-completion-vertexai",

templates/all-patterns.jsonnet

@@ -10,6 +10,7 @@
     import "patterns/llm-bedrock.jsonnet",
     import "patterns/llm-claude.jsonnet",
     import "patterns/llm-cohere.jsonnet",
+    import "patterns/llm-llamafile.jsonnet",
     import "patterns/llm-ollama.jsonnet",
     import "patterns/llm-openai.jsonnet",
     import "patterns/llm-vertexai.jsonnet",

templates/components.jsonnet

@@ -10,6 +10,7 @@
    "graph-rag": import "components/graph-rag.jsonnet",
    "triple-store-cassandra": import "components/cassandra.jsonnet",
    "triple-store-neo4j": import "components/neo4j.jsonnet",
+   "llamafile": import "components/llamafile.jsonnet",
    "ollama": import "components/ollama.jsonnet",
    "openai": import "components/openai.jsonnet",
    "override-recursive-chunker": import "components/chunker-recursive.jsonnet",

templates/components/graph-rag.jsonnet

@@ -68,6 +68,31 @@ local url = import "values/url.jsonnet";
 
     },
 
+    "kg-extract-topics" +: {
+    
+        create:: function(engine)
+
+            local container =
+                engine.container("kg-extract-topics")
+                    .with_image(images.trustgraph)
+                    .with_command([
+                        "kg-extract-topics",
+                        "-p",
+                        url.pulsar,
+                    ])
+                    .with_limits("0.5", "128M")
+                    .with_reservations("0.1", "128M");
+
+            local containerSet = engine.containers(
+                "kg-extract-topics", [ container ]
+            );
+
+            engine.resources([
+                containerSet,
+            ])
+
+    },
+
     "graph-rag" +: {
     
         create:: function(engine)

templates/components/llamafile.jsonnet

@@ -0,0 +1,75 @@
+local base = import "base/base.jsonnet";
+local images = import "values/images.jsonnet";
+local url = import "values/url.jsonnet";
+local prompts = import "prompts/slm.jsonnet";
+
+{
+
+    "llamafile-model":: "LLaMA_CPP",
+    "llamafile-url":: "${LLAMAFILE_URL}",
+
+    "text-completion" +: {
+    
+        create:: function(engine)
+
+            local container =
+                engine.container("text-completion")
+                    .with_image(images.trustgraph)
+                    .with_command([
+                        "text-completion-llamafile",
+                        "-p",
+                        url.pulsar,
+                        "-m",
+                        $["llamafile-model"],
+                        "-r",
+                        $["llamafile-url"],
+                    ])
+                    .with_limits("0.5", "128M")
+                    .with_reservations("0.1", "128M");
+
+            local containerSet = engine.containers(
+                "text-completion", [ container ]
+            );
+
+            engine.resources([
+                containerSet,
+            ])
+
+    },
+
+    "text-completion-rag" +: {
+    
+        create:: function(engine)
+
+            local container =
+                engine.container("text-completion-rag")
+                    .with_image(images.trustgraph)
+                    .with_command([
+                        "text-completion-llamafile",
+                        "-p",
+                        url.pulsar,
+                        "-m",
+                        $["llamafile-model"],
+                        "-r",
+                        $["llamafile-url"],
+                        "-i",
+                        "non-persistent://tg/request/text-completion-rag",
+                        "-o",
+                        "non-persistent://tg/response/text-completion-rag-response",
+                    ])
+                    .with_limits("0.5", "128M")
+                    .with_reservations("0.1", "128M");
+
+            local containerSet = engine.containers(
+                "text-completion-rag", [ container ]
+            );
+
+            engine.resources([
+                containerSet,
+            ])
+
+
+    }
+
+} + prompts
+

templates/components/prompt-template.jsonnet

@@ -25,6 +25,8 @@ local default_prompts = import "prompts/default-prompts.jsonnet";
                         $["prompt-definition-template"],
                         "--relationship-template",
                         $["prompt-relationship-template"],
+                        "--topic-template",
+                        $["prompt-topic-template"],
                         "--knowledge-query-template",
                         $["prompt-knowledge-query-template"],
                         "--document-query-template",
@@ -73,6 +75,8 @@ local default_prompts = import "prompts/default-prompts.jsonnet";
                         $["prompt-definition-template"],
                         "--relationship-template",
                         $["prompt-relationship-template"],
+                        "--topic-template",
+                        $["prompt-topic-template"],
                         "--knowledge-query-template",
                         $["prompt-knowledge-query-template"],
                         "--document-query-template",

templates/patterns/llm-llamafile.jsonnet

@@ -0,0 +1,41 @@
+{
+    pattern: {
+	name: "llamafile",
+        icon: "🤖💬",
+        title: "Add Llamafile-invoked LLMs for text completion",
+	description: "This pattern integrates a Llamafile service for text completion operations.  You need to have a running Llamafile implementation executing the necessary model in order to be able to use this service.",
+        requires: ["pulsar", "trustgraph"],
+        features: ["llm"],
+	args: [
+	    {
+		name: "llamafile-max-output-tokens",
+                label: "Maximum output tokens",
+		type: "integer",
+		description: "Limit on number tokens to generate",
+                default: 4096,
+		required: true,
+            },
+	    {
+		name: "llamafile-temperature",
+                label: "Temperature",
+		type: "slider",
+		description: "Controlling predictability / creativity balance",
+                min: 0,
+                max: 1,
+                step: 0.05,
+                default: 0.5,
+            },
+	    {
+		name: "llamafile-url",
+                label: "URL",
+		type: "text",
+		width: 120,
+		description: "URL of the Llamafile service",
+                default: "http://llamafile:8080",
+		required: true,
+	    },
+	],
+        category: [ "llm" ],
+    },
+    module: "components/llamafile.jsonnet",
+}

templates/prompts/cohere.jsonnet

@@ -7,6 +7,8 @@
 
 //    "prompt-relationship-template":: "PROMPT GOES HERE",
 
+//    "prompt-topic-template":: "PROMPT GOES HERE",
+
 //    "prompt-knowledge-query-template":: "PROMPT GOES HERE",
 
 //    "prompt-document-query-template":: "PROMPT GOES HERE",

templates/prompts/default-prompts.jsonnet

@@ -8,6 +8,8 @@
 
     "prompt-relationship-template":: "<instructions>\nStudy the following text and derive entity relationships.  For each\nrelationship, derive the subject, predicate and object of the relationship.\nOutput relationships in JSON format as an arary of objects with fields:\n- subject: the subject of the relationship\n- predicate: the predicate\n- object: the object of the relationship\n- object-entity: false if the object is a simple data type: name, value or date.  true if it is an entity.\n</instructions>\n\n<text>\n{text}\n</text>\n\n<requirements>\nYou will respond only with raw JSON format data. Do not provide\nexplanations. Do not use special characters in the abstract text. The\nabstract must be written as plain text.  Do not add markdown formatting\nor headers or prefixes.\n</requirements>",
 
+    "prompt-topic-template":: "You are a helpful assistant that performs information extraction tasks for a provided text.\nRead the provided text. You will identify topics and their definitions in JSON.\n\nReading Instructions:\n- Ignore document formatting in the provided text.\n- Study the provided text carefully.\n\nHere is the text:\n{text}\n\nResponse Instructions: \n- Do not respond with special characters.\n- Return only topics that are concepts and unique to the provided text.\n- Respond only with well-formed JSON.\n- The JSON response shall be an array of objects with keys \"topic\" and \"definition\". \n- The JSON response shall use the following structure:\n\n```json\n[{{\"topic\": string, \"definition\": string}}]\n```\n\n- Do not write any additional text or explanations.",
+
     "prompt-knowledge-query-template":: "Study the following set of knowledge statements. The statements are written in Cypher format that has been extracted from a knowledge graph. Use only the provided set of knowledge statements in your response. Do not speculate if the answer is not found in the provided set of knowledge statements.\n\nHere's the knowledge statements:\n{graph}\n\nUse only the provided knowledge statements to respond to the following:\n{query}\n",
 
     "prompt-document-query-template":: "Study the following context. Use only the information provided in the context in your response. Do not speculate if the answer is not found in the provided set of knowledge statements.\n\nHere is the context:\n{documents}\n\nUse only the provided knowledge statements to respond to the following:\n{query}\n",

templates/prompts/gemini.jsonnet

@@ -6,6 +6,8 @@
 
 //    "prompt-relationship-template":: "PROMPT GOES HERE",
 
+//    "prompt-topic-template":: "PROMPT GOES HERE",
+
 //    "prompt-knowledge-query-template":: "PROMPT GOES HERE",
 
 //    "prompt-document-query-template":: "PROMPT GOES HERE",

templates/prompts/mixtral.jsonnet

@@ -7,6 +7,8 @@
 
 //    "prompt-relationship-template":: "PROMPT GOES HERE",
 
+//    "prompt-topic-template":: "PROMPT GOES HERE",
+
 //    "prompt-knowledge-query-template":: "PROMPT GOES HERE",
 
 //    "prompt-document-query-template":: "PROMPT GOES HERE",

templates/prompts/openai.jsonnet

@@ -11,6 +11,8 @@ local url = import "values/url.jsonnet";
 
 //    "prompt-relationship-template":: "PROMPT GOES HERE",
 
+//    "prompt-topic-template":: "PROMPT GOES HERE",
+
 //    "prompt-knowledge-query-template":: "PROMPT GOES HERE",
 
 //    "prompt-document-query-template":: "PROMPT GOES HERE",

trustgraph/clients/prompt_client.py

@@ -44,6 +44,13 @@ class PromptClient(BaseClient):
             kind="extract-definitions", chunk=chunk,
             timeout=timeout
         ).definitions
+    
+    def request_topics(self, chunk, timeout=300):
+
+        return self.call(
+            kind="extract-topics", chunk=chunk,
+            timeout=timeout
+        ).topics
 
     def request_relationships(self, chunk, timeout=300):
 

trustgraph/extract/kg/topics/__init__.py

@@ -0,0 +1,3 @@
+
+from . extract import *
+

trustgraph/extract/kg/topics/__main__.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+
+from . extract import run
+
+if __name__ == '__main__':
+    run()
+

trustgraph/extract/kg/topics/extract.py

@@ -0,0 +1,134 @@
+
+"""
+Simple decoder, accepts embeddings+text chunks input, applies entity analysis to
+get entity definitions which are output as graph edges.
+"""
+
+import urllib.parse
+import json
+
+from .... schema import ChunkEmbeddings, Triple, Source, Value
+from .... schema import chunk_embeddings_ingest_queue, triples_store_queue
+from .... schema import prompt_request_queue
+from .... schema import prompt_response_queue
+from .... log_level import LogLevel
+from .... clients.prompt_client import PromptClient
+from .... rdf import TRUSTGRAPH_ENTITIES, DEFINITION
+from .... base import ConsumerProducer
+
+DEFINITION_VALUE = Value(value=DEFINITION, is_uri=True)
+
+module = ".".join(__name__.split(".")[1:-1])
+
+default_input_queue = chunk_embeddings_ingest_queue
+default_output_queue = triples_store_queue
+default_subscriber = module
+
+class Processor(ConsumerProducer):
+
+    def __init__(self, **params):
+
+        input_queue = params.get("input_queue", default_input_queue)
+        output_queue = params.get("output_queue", default_output_queue)
+        subscriber = params.get("subscriber", default_subscriber)
+        pr_request_queue = params.get(
+            "prompt_request_queue", prompt_request_queue
+        )
+        pr_response_queue = params.get(
+            "prompt_response_queue", prompt_response_queue
+        )
+
+        super(Processor, self).__init__(
+            **params | {
+                "input_queue": input_queue,
+                "output_queue": output_queue,
+                "subscriber": subscriber,
+                "input_schema": ChunkEmbeddings,
+                "output_schema": Triple,
+                "prompt_request_queue": pr_request_queue,
+                "prompt_response_queue": pr_response_queue,
+            }
+        )
+
+        self.prompt = PromptClient(
+            pulsar_host=self.pulsar_host,
+            input_queue=pr_request_queue,
+            output_queue=pr_response_queue,
+            subscriber = module + "-prompt",
+        )
+
+    def to_uri(self, text):
+
+        part = text.replace(" ", "-").lower().encode("utf-8")
+        quoted = urllib.parse.quote(part)
+        uri = TRUSTGRAPH_ENTITIES + quoted
+
+        return uri
+
+    def get_topics(self, chunk):
+
+        return self.prompt.request_topics(chunk)
+
+    def emit_edge(self, s, p, o):
+
+        t = Triple(s=s, p=p, o=o)
+        self.producer.send(t)
+
+    def handle(self, msg):
+
+        v = msg.value()
+        print(f"Indexing {v.source.id}...", flush=True)
+
+        chunk = v.chunk.decode("utf-8")
+
+        try:
+
+            defs = self.get_topics(chunk)
+
+            for defn in defs:
+
+                s = defn.name
+                o = defn.definition
+
+                if s == "": continue
+                if o == "": continue
+
+                if s is None: continue
+                if o is None: continue
+
+                s_uri = self.to_uri(s)
+
+                s_value = Value(value=str(s_uri), is_uri=True)
+                o_value = Value(value=str(o), is_uri=False)
+
+                self.emit_edge(s_value, DEFINITION_VALUE, o_value)
+
+        except Exception as e:
+            print("Exception: ", e, flush=True)
+
+        print("Done.", flush=True)
+
+    @staticmethod
+    def add_args(parser):
+
+        ConsumerProducer.add_args(
+            parser, default_input_queue, default_subscriber,
+            default_output_queue,
+        )
+
+        parser.add_argument(
+            '--prompt-request-queue',
+            default=prompt_request_queue,
+            help=f'Prompt request queue (default: {prompt_request_queue})',
+        )
+
+        parser.add_argument(
+            '--prompt-completion-response-queue',
+            default=prompt_response_queue,
+            help=f'Prompt response queue (default: {prompt_response_queue})',
+        )
+
+def run():
+
+    Processor.start(module, __doc__)
+

trustgraph/model/prompt/generic/prompts.py

@@ -1,50 +1,92 @@
 
 def to_relationships(text):
 
-    prompt = f"""<instructions>
-Study the following text and derive entity relationships.  For each
-relationship, derive the subject, predicate and object of the relationship.
-Output relationships in JSON format as an arary of objects with fields:
-- subject: the subject of the relationship
-- predicate: the predicate
-- object: the object of the relationship
-- object-entity: false if the object is a simple data type: name, value or date.  true if it is an entity.
-</instructions>
+    prompt = f"""You are a helpful assistant that performs information extraction tasks for a provided text.
 
-<text>
+Read the provided text. You will model the text as an information network for a RDF knowledge graph in JSON.
+
+Information Network Rules:
+- An information network has subjects connected by predicates to objects.
+- A subject is a named-entity or a conceptual topic.
+- One subject can have many predicates and objects.
+- An object is a property or attribute of a subject.
+- A subject can be connected by a predicate to another subject.
+
+Reading Instructions:
+- Ignore document formatting in the provided text.
+- Study the provided text carefully.
+
+Here is the text: 
 {text}
-</text>
 
-<requirements>
-You will respond only with raw JSON format data. Do not provide
-explanations. Do not use special characters in the abstract text. The
-abstract must be written as plain text.  Do not add markdown formatting
-or headers or prefixes.
-</requirements>"""
+Response Instructions:
+- Obey the information network rules. 
+- Do not return special characters.
+- Respond only with well-formed JSON.
+- The JSON response shall be an array of JSON objects with keys "subject", "predicate", "object", and "object-entity".
+- The JSON response shall use the following structure:
+
+```json
+[{{"subject": string, "predicate": string, "object": string, "object-entity": boolean}}]
+```
+
+- The key "object-entity" is TRUE only if the "object" is a subject.
+- Do not write any additional text or explanations.
+"""
+    
+    return prompt
+
+def to_topics(text):
+
+    prompt = f"""You are a helpful assistant that performs information extraction tasks for a provided text.\nRead the provided text. You will identify topics and their definitions in JSON.
+
+Reading Instructions:
+- Ignore document formatting in the provided text.
+- Study the provided text carefully.
+
+Here is the text:
+{text}
+
+Response Instructions: 
+- Do not respond with special characters.
+- Return only topics that are concepts and unique to the provided text.
+- Respond only with well-formed JSON.
+- The JSON response shall be an array of objects with keys "topic" and "definition". 
+- The JSON response shall use the following structure:
+
+```json
+[{{"topic": string, "definition": string}}]
+```
+
+- Do not write any additional text or explanations.
+"""
     
     return prompt
     
 def to_definitions(text):
 
-    prompt = f"""<instructions>
-Study the following text and derive definitions for any discovered entities.
-Do not provide definitions for entities whose definitions are incomplete
-or unknown.
-Output relationships in JSON format as an arary of objects with fields:
-- entity: the name of the entity
-- definition: English text which defines the entity
-</instructions>
+    prompt = f"""You are a helpful assistant that performs information extraction tasks for a provided text.\nRead the provided text. You will identify entities and their definitions in JSON.
 
-<text>
+Reading Instructions:
+- Ignore document formatting in the provided text.
+- Study the provided text carefully.
+
+Here is the text:
 {text}
-</text>
 
-<requirements>
-You will respond only with raw JSON format data. Do not provide
-explanations. Do not use special characters in the abstract text. The
-abstract will be written as plain text.  Do not add markdown formatting
-or headers or prefixes.  Do not include null or unknown definitions.
-</requirements>"""
+Response Instructions:
+- Do not respond with special characters.
+- Return only entities that are named-entities such as: people, organizations, physical objects, locations, animals, products, commodotities, or substances.
+- Respond only with well-formed JSON. 
+- The JSON response shall be an array of objects with keys "entity" and "definition".
+- The JSON response shall use the following structure: 
+
+```json
+[{{"entity": string, "definition": string}}]
+```
+
+- Do not write any additional text or explanations.
+"""
     
     return prompt
 

trustgraph/model/prompt/generic/service.py

@@ -3,8 +3,10 @@ Language service abstracts prompt engineering from LLM.
 """
 
 import json
+import re
 
 from .... schema import Definition, Relationship, Triple
+from .... schema import Topic
 from .... schema import PromptRequest, PromptResponse, Error
 from .... schema import TextCompletionRequest, TextCompletionResponse
 from .... schema import text_completion_request_queue
@@ -13,7 +15,7 @@ from .... schema import prompt_request_queue, prompt_response_queue
 from .... base import ConsumerProducer
 from .... clients.llm_client import LlmClient
 
-from . prompts import to_definitions, to_relationships
+from . prompts import to_definitions, to_relationships, to_topics
 from . prompts import to_kg_query, to_document_query, to_rows
 
 module = ".".join(__name__.split(".")[1:-1])
@@ -56,12 +58,15 @@ class Processor(ConsumerProducer):
         )
 
     def parse_json(self, text):
-        
-        # Hacky, workaround temperamental JSON markdown
-        text = text.replace("```json", "")
-        text = text.replace("```", "")
+        json_match = re.search(r'```(?:json)?(.*?)```', text, re.DOTALL)
+    
+        if json_match:
+            json_str = json_match.group(1).strip()
+        else:
+            # If no delimiters, assume the entire output is JSON
+            json_str = text.strip()
 
-        return json.loads(text)
+        return json.loads(json_str)
 
     def handle(self, msg):
 
@@ -80,6 +85,11 @@ class Processor(ConsumerProducer):
             self.handle_extract_definitions(id, v)
             return
 
+        elif kind == "extract-topics":
+
+            self.handle_extract_topics(id, v)
+            return
+
         elif kind == "extract-relationships":
 
             self.handle_extract_relationships(id, v)
@@ -164,6 +174,65 @@ class Processor(ConsumerProducer):
 
             self.producer.send(r, properties={"id": id})
 
+    def handle_extract_topics(self, id, v):
+
+        try:
+
+            prompt = to_topics(v.chunk)
+
+            ans = self.llm.request(prompt)
+
+            # Silently ignore JSON parse error
+            try:
+                defs = self.parse_json(ans)
+            except:
+                print("JSON parse error, ignored", flush=True)
+                defs = []
+
+            output = []
+
+            for defn in defs:
+
+                try:
+                    e = defn["topic"]
+                    d = defn["definition"]
+
+                    if e == "": continue
+                    if e is None: continue
+                    if d == "": continue
+                    if d is None: continue
+
+                    output.append(
+                        Topic(
+                            name=e, definition=d
+                        )
+                    )
+
+                except:
+                    print("definition fields missing, ignored", flush=True)
+
+            print("Send response...", flush=True)
+            r = PromptResponse(topics=output, error=None)
+            self.producer.send(r, properties={"id": id})
+
+            print("Done.", flush=True)
+        
+        except Exception as e:
+
+            print(f"Exception: {e}")
+
+            print("Send error response...", flush=True)
+
+            r = PromptResponse(
+                error=Error(
+                    type = "llm-error",
+                    message = str(e),
+                ),
+                response=None,
+            )
+
+            self.producer.send(r, properties={"id": id})
+
     def handle_extract_relationships(self, id, v):
 
         try:

trustgraph/model/prompt/template/prompts.py

@@ -5,6 +5,9 @@ def to_relationships(template, text):
 def to_definitions(template, text):
     return template.format(text=text)
 
+def to_topics(template, text):
+    return template.format(text=text)
+
 def to_rows(template, schema, text):
 
     field_schema = [

trustgraph/model/prompt/template/service.py

@@ -4,8 +4,10 @@ Language service abstracts prompt engineering from LLM.
 """
 
 import json
+import re
 
 from .... schema import Definition, Relationship, Triple
+from .... schema import Topic
 from .... schema import PromptRequest, PromptResponse, Error
 from .... schema import TextCompletionRequest, TextCompletionResponse
 from .... schema import text_completion_request_queue
@@ -15,7 +17,7 @@ from .... base import ConsumerProducer
 from .... clients.llm_client import LlmClient
 
 from . prompts import to_definitions, to_relationships, to_rows
-from . prompts import to_kg_query, to_document_query
+from . prompts import to_kg_query, to_document_query, to_topics
 
 module = ".".join(__name__.split(".")[1:-1])
 
@@ -38,6 +40,7 @@ class Processor(ConsumerProducer):
         )
         definition_template = params.get("definition_template")
         relationship_template = params.get("relationship_template")
+        topic_template = params.get("topic_template")
         rows_template = params.get("rows_template")
         knowledge_query_template = params.get("knowledge_query_template")
         document_query_template = params.get("document_query_template")
@@ -62,18 +65,22 @@ class Processor(ConsumerProducer):
         )
 
         self.definition_template = definition_template
+        self.topic_template = topic_template
         self.relationship_template = relationship_template
         self.rows_template = rows_template
         self.knowledge_query_template = knowledge_query_template
         self.document_query_template = document_query_template
 
     def parse_json(self, text):
-        
-        # Hacky, workaround temperamental JSON markdown
-        text = text.replace("```json", "")
-        text = text.replace("```", "")
+        json_match = re.search(r'```(?:json)?(.*?)```', text, re.DOTALL)
+    
+        if json_match:
+            json_str = json_match.group(1).strip()
+        else:
+            # If no delimiters, assume the entire output is JSON
+            json_str = text.strip()
 
-        return json.loads(text)
+        return json.loads(json_str)
 
     def handle(self, msg):
 
@@ -92,6 +99,11 @@ class Processor(ConsumerProducer):
             self.handle_extract_definitions(id, v)
             return
 
+        elif kind == "extract-topics":
+
+            self.handle_extract_topics(id, v)
+            return
+
         elif kind == "extract-relationships":
 
             self.handle_extract_relationships(id, v)
@@ -176,6 +188,66 @@ class Processor(ConsumerProducer):
 
             self.producer.send(r, properties={"id": id})
 
+    def handle_extract_topics(self, id, v):
+
+        try:
+
+            prompt = to_topics(self.topic_template, v.chunk)
+
+            ans = self.llm.request(prompt)
+
+            # Silently ignore JSON parse error
+            try:
+                defs = self.parse_json(ans)
+            except:
+                print("JSON parse error, ignored", flush=True)
+                defs = []
+
+            output = []
+
+            for defn in defs:
+
+                try:
+                    e = defn["topic"]
+                    d = defn["definition"]
+
+                    if e == "": continue
+                    if e is None: continue
+                    if d == "": continue
+                    if d is None: continue
+
+                    output.append(
+                        Topic(
+                            name=e, definition=d
+                        )
+                    )
+
+                except:
+                    print("definition fields missing, ignored", flush=True)
+
+            print("Send response...", flush=True)
+            r = PromptResponse(topics=output, error=None)
+            self.producer.send(r, properties={"id": id})
+
+            print("Done.", flush=True)
+        
+        except Exception as e:
+
+            print(f"Exception: {e}")
+
+            print("Send error response...", flush=True)
+
+            r = PromptResponse(
+                error=Error(
+                    type = "llm-error",
+                    message = str(e),
+                ),
+                response=None,
+            )
+
+            self.producer.send(r, properties={"id": id})
+
+
     def handle_extract_relationships(self, id, v):
 
         try:
@@ -415,6 +487,12 @@ class Processor(ConsumerProducer):
             help=f'Definition extraction template',
         )
 
+        parser.add_argument(
+            '--topic-template',
+            required=True,
+            help=f'Topic extraction template',
+        )
+
         parser.add_argument(
             '--rows-template',
             required=True,

trustgraph/model/text_completion/llamafile/__init__.py

@@ -0,0 +1,3 @@
+
+from . llm import *
+

trustgraph/model/text_completion/llamafile/__main__.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+
+from . llm import run
+
+if __name__ == '__main__':
+    run()
+

trustgraph/model/text_completion/llamafile/llm.py

@@ -0,0 +1,195 @@
+
+"""
+Simple LLM service, performs text prompt completion using OpenAI.
+Input is prompt, output is response.
+"""
+
+from openai import OpenAI
+from prometheus_client import Histogram
+
+from .... schema import TextCompletionRequest, TextCompletionResponse, Error
+from .... schema import text_completion_request_queue
+from .... schema import text_completion_response_queue
+from .... log_level import LogLevel
+from .... base import ConsumerProducer
+from .... exceptions import TooManyRequests
+
+module = ".".join(__name__.split(".")[1:-1])
+
+default_input_queue = text_completion_request_queue
+default_output_queue = text_completion_response_queue
+default_subscriber = module
+default_model = 'LLaMA_CPP'
+default_llamafile = 'http://localhost:8080/v1'
+default_temperature = 0.0
+default_max_output = 4096
+
+class Processor(ConsumerProducer):
+
+    def __init__(self, **params):
+    
+        input_queue = params.get("input_queue", default_input_queue)
+        output_queue = params.get("output_queue", default_output_queue)
+        subscriber = params.get("subscriber", default_subscriber)
+        model = params.get("model", default_model)
+        llamafile = params.get("llamafile", default_llamafile)
+        temperature = params.get("temperature", default_temperature)
+        max_output = params.get("max_output", default_max_output)
+
+        super(Processor, self).__init__(
+            **params | {
+                "input_queue": input_queue,
+                "output_queue": output_queue,
+                "subscriber": subscriber,
+                "input_schema": TextCompletionRequest,
+                "output_schema": TextCompletionResponse,
+                "model": model,
+                "temperature": temperature,
+                "max_output": max_output,
+                "llamafile" : llamafile,
+            }
+        )
+
+        if not hasattr(__class__, "text_completion_metric"):
+            __class__.text_completion_metric = Histogram(
+                'text_completion_duration',
+                'Text completion duration (seconds)',
+                buckets=[
+                    0.25, 0.5, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0,
+                    8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0,
+                    17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0,
+                    30.0, 35.0, 40.0, 45.0, 50.0, 60.0, 80.0, 100.0,
+                    120.0
+                ]
+            )
+
+        self.model = model
+        self.llamafile=llamafile
+        self.temperature = temperature
+        self.max_output = max_output
+        self.openai = OpenAI(
+            base_url=self.llamafile, 
+            api_key = "sk-no-key-required",
+            )
+
+        print("Initialised", flush=True)
+
+    def handle(self, msg):
+
+        v = msg.value()
+
+        # Sender-produced ID
+
+        id = msg.properties()["id"]
+
+        print(f"Handling prompt {id}...", flush=True)
+
+        prompt = v.prompt
+
+        try:
+
+            # FIXME: Rate limits
+
+            with __class__.text_completion_metric.time():
+
+                resp = self.openai.chat.completions.create(
+                    model=self.model,
+                    messages=[
+                        {"role": "user", "content": prompt}
+                    ]
+                    #temperature=self.temperature,
+                    #max_tokens=self.max_output,
+                    #top_p=1,
+                    #frequency_penalty=0,
+                    #presence_penalty=0,
+                    #response_format={
+                    #    "type": "text"
+                    #}
+                )
+
+            print(resp.choices[0].message.content, flush=True)
+
+            print("Send response...", flush=True)
+            r = TextCompletionResponse(
+                response=resp.choices[0].message.content,
+                error=None,
+            )
+            self.send(r, properties={"id": id})
+
+            print("Done.", flush=True)
+
+        # FIXME: Wrong exception, don't know what this LLM throws
+        # for a rate limit
+        except TooManyRequests:
+
+            print("Send rate limit response...", flush=True)
+
+            r = TextCompletionResponse(
+                error=Error(
+                    type = "rate-limit",
+                    message = str(e),
+                ),
+                response=None,
+            )
+
+            self.producer.send(r, properties={"id": id})
+
+            self.consumer.acknowledge(msg)
+
+        except Exception as e:
+
+            print(f"Exception: {e}")
+
+            print("Send error response...", flush=True)
+
+            r = TextCompletionResponse(
+                error=Error(
+                    type = "llm-error",
+                    message = str(e),
+                ),
+                response=None,
+            )
+
+            self.producer.send(r, properties={"id": id})
+
+            self.consumer.acknowledge(msg)
+
+    @staticmethod
+    def add_args(parser):
+
+        ConsumerProducer.add_args(
+            parser, default_input_queue, default_subscriber,
+            default_output_queue,
+        )
+
+        parser.add_argument(
+            '-m', '--model',
+            default=default_model,
+            help=f'LLM model (default: LLaMA_CPP)'
+        )
+
+        parser.add_argument(
+            '-r', '--llamafile',
+            default=default_llamafile,
+            help=f'ollama (default: {default_llamafile})'
+        )
+
+        parser.add_argument(
+            '-t', '--temperature',
+            type=float,
+            default=default_temperature,
+            help=f'LLM temperature parameter (default: {default_temperature})'
+        )
+
+        parser.add_argument(
+            '-x', '--max-output',
+            type=int,
+            default=default_max_output,
+            help=f'LLM max output tokens (default: {default_max_output})'
+        )
+
+def run():
+
+    Processor.start(module, __doc__)
+
+    

trustgraph/schema/prompt.py

@@ -12,6 +12,10 @@ class Definition(Record):
     name = String()
     definition = String()
 
+class Topic(Record):
+    name = String()
+    definition = String()
+
 class Relationship(Record):
     s = String()
     p = String()
@@ -46,6 +50,7 @@ class PromptResponse(Record):
     error = Error()
     answer = String()
     definitions = Array(Definition())
+    topics = Array(Topic())
     relationships = Array(Relationship())
     rows = Array(Map(String()))